Data Ingestion and Transformation
Pipeline Orchestration and Resilience
CoreRepresent ETL dependencies, retries, state, failure paths, maintenance, and notification delivery in the orchestration surface that owns workflow progress.
Aligned to AWS Certified Data Engineer - Associate (DEA-C01) Version 1.1, verified August 25, 2026.
Why this matters
A chain of scheduled jobs is not automatically a resilient workflow. Recovery depends on knowing which stage succeeded, which work is safe to repeat, and where terminal failure is preserved and reported.
Must Know
- Use Amazon MWAA for managed Apache Airflow DAG compatibility, Step Functions for serverless state machines and service integrations, and Glue workflows for Glue-native job and crawler coordination.
- EventBridge routes events and schedules targets; it does not replace stateful dependency, branch, and compensation logic.
- Design each stage with retries appropriate to the error, idempotent side effects, durable progress, a terminal failure path, and enough context to resume or compensate.
- Serverless workflows still need version control, deployment compatibility, execution visibility, alarms, and a rollback boundary.
- Use SNS to publish notifications to subscribers; use SQS when a worker needs a durable queue of work. Alert transport is not the recovery mechanism.
- Build for performance, availability, scalability, resiliency, and fault tolerance without restarting every successful stage after one downstream failure.
Compare and Distinguish
- MWAA versus Step Functions versus Glue workflows: Airflow DAGs, AWS state machines, and Glue-native coordination express different workflow contracts.
- Orchestration versus EventBridge: dependency and execution state differ from event matching and target delivery.
- SNS versus SQS: publish-subscribe notification differs from durable worker buffering.
- Retry versus compensation: repeating a safe action differs from deliberately undoing or offsetting a completed side effect.
Scenario examples
- An existing Airflow DAG with backfills and Python operators moves to MWAA with dependency packages, networking, and logs preserved.
- A Step Functions execution retries a transient task, catches a terminal error, records context, and sends an SNS alert.
- A Glue workflow coordinates crawler and ETL dependencies while each job remains idempotent.
Exam traps
- Using independent cron schedules where downstream work depends on upstream success.
- Treating a notification as proof that the failed data was recovered.
- Restarting an entire pipeline when only one repeatable stage failed.
- Assuming serverless workflow definitions require no release management.
Key takeaways
- Put dependency and retry logic in the layer that owns execution state.
- Separate workflow control, event routing, notification, and work buffering.
- Preserve context for recovery instead of hiding terminal failure.
How it works
- The orchestrator records each stage transition, invokes targets through an execution role, and applies retry, catch, or terminal-failure behavior.
- Run identifiers and durable outputs allow a failed stage to resume or compensate without repeating successful side effects.
When to use it
- Use MWAA for Airflow-compatible DAG operations, Step Functions for AWS-integrated state machines, and Glue workflows for Glue-native dependencies.
- Use SNS to fan out alerts and SQS to buffer worker tasks; neither replaces the workflow's own recovery state.
Security and governance implications
- Give the orchestrator permission only to invoke the named targets, and give each target its own scoped data permissions.
- Protect execution input, output, and failure details because workflow histories can expose sensitive pipeline context.
Common failure modes and diagnosis
- Locate the first failed transition, then verify target-role authorization, input contract, dependency state, retry classification, and task logs.
- For stuck backfills, compare run parameters, concurrency limits, checkpoints, and prior side effects before restarting work.
More detail
- 1.3.1: Use orchestration services to build workflows for data ETL pipelines (for example, Lambda, EventBridge, Amazon Managed Workflows for Apache Airflow [Amazon MWAA], AWS Step Functions, AWS Glue workflows).
- 1.3.2: Build data pipelines for performance, availability, scalability, resiliency, and fault tolerance.
- 1.3.3: Implement and maintain serverless workflows.
- 1.3.4: Use notification services to send alerts (for example, Amazon SNS, Amazon SQS).
Ready for the quiz?
- Does the workflow require Airflow DAG compatibility, an AWS state machine, or coordination limited to Glue jobs and crawlers?
- Which failures can be retried safely, which require compensation, and what state must be retained for recovery?
Related objectives
- D1.3 — Task 1.3: Orchestrate data pipelines
- 1.3.1 — Use orchestration services to build workflows for data ETL pipelines (for example, Lambda, EventBridge, Amazon Managed Workflows for Apache Airflow [Amazon MWAA], AWS Step Functions, AWS Glue workflows).
- 1.3.2 — Build data pipelines for performance, availability, scalability, resiliency, and fault tolerance.
- 1.3.3 — Implement and maintain serverless workflows.
- 1.3.4 — Use notification services to send alerts (for example, Amazon SNS, Amazon SQS).