Spark transformations and actions with an execution plan

Spark transformations describe a new dataset and are generally lazy; an action requests a result or write and triggers execution. The physical work depends on the optimized plan, partitioning and data. Counting API calls is not a performance analysis.

A static plan with an honest evidence state

The data-engineering lab records a proposed flow: read Parquet, filter date, select account and amount, group by account, sum amount, then write a partitioned feature table.

from data_engineering_cases import spark_plan_case

result = spark_plan_case()
assert result["action_count"] == 1
assert result["pyspark_executed"] is False
print(result["plan"])

The plan marks groupBy as a shuffle boundary and avoids caching until reuse is measured. PySpark is not installed, so there is no explain output, job time or shuffle metric.

Read plans before tuning

In a real environment, inspect logical and physical plans. Look for filters pushed near reads, column pruning, exchanges, join strategy and repeated scans. Use the Spark UI for task skew, shuffle size, spills and executor failures.

Caching consumes memory and can slow a one-use dataset. Cache only when repeated computation and measurements justify it, and unpersist deliberately. Avoid accidental actions such as repeated counts during debugging.

Verify output semantics

Define the feature grain and write mode. Reconcile counts, unique keys and sums against a small local fixture. An optimized job that duplicates entities is still wrong.

The Data Science course separates documented Spark semantics from executed distributed evidence.

Exercise

Run the plan in a reviewed PySpark environment. Save explain output and Spark UI metrics, then test column pruning and one measured cache decision.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in PySpark joins: identify skew before changing configuration.
• Continue with Airflow DAG design: separate orchestration from business logic.

Reference: Spark RDD programming guide on transformations and actions.