Airflow DAG design: separate orchestration from business logic

An orchestrator should coordinate dependencies, retries, schedules and observability. Embedding all transformation logic inside DAG definitions makes local testing and reuse harder. Put business logic in versioned functions or jobs and let the DAG call them.

Test logic without claiming a DAG run

The local lab executes its daily feature transformation on 500 invented events and records an Airflow-style specification.

from data_engineering_cases import airflow_case

result = airflow_case()
assert result["business_logic_test_passed"] is True
assert result["airflow_dag_executed"] is False
print(result["dag_spec"])

The specification schedules source readiness, feature build, output validation and manifest publication at 03:00 with two retries and no catchup. Airflow is not installed; the lab does not claim parsing, scheduling or task execution.

Make tasks idempotent

Parameterize the data interval. Write versioned or replaceable partitions, validate before publishing and make retry converge. Avoid now() inside business logic when the scheduled interval should define time.

Keep DAG import lightweight. Pass references or small metadata between tasks instead of full datasets. Set timeouts, retry delays, owners and useful alerts. Test business functions locally, DAG structure in an Airflow environment and end-to-end behavior with a staging backend.

Treat backfills explicitly

Catchup and backfill can create large parallel load. Bound concurrent runs, estimate partitions and publish atomically. Preserve previous output for rollback.

The Data Science course connects orchestration to reproducible, independently tested data logic.

Exercise

Implement the four-task specification in an authorized Airflow environment. Add data-interval parameters, idempotency checks and a two-day backfill test, then record the actual DAG-run evidence.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Spark transformations and actions with an execution plan.
• Continue with Data quality checks at ingestion and before training.

Reference: Apache Airflow best practices.