Data Operations and Support
Automated Data Processing
CoreOperate workflows, SDK calls, service-native processing, data APIs, preparation tools, Athena, Lambda, and EventBridge as repeatable, observable automation.
Aligned to AWS Certified Data Engineer - Associate (DEA-C01) Version 1.1, verified August 25, 2026.
Why this matters
Automation succeeds only when identity, pagination, retry, idempotency, run state, and failure evidence are maintained. Calling a managed service does not remove these operational boundaries.
Must Know
- Choose MWAA or Step Functions from the workflow model and preserve dependencies, retries, run state, backfills, and failure evidence.
- Troubleshoot managed workflows across scheduler or worker health, dependency packaging, logs, IAM, and VPC connectivity before rewriting task logic.
- Use AWS SDK runtime credential providers and workload roles instead of stored access keys; handle pagination, retryable errors, and terminal errors.
- Process data where it already lives when a service-native Glue, EMR, or Redshift feature satisfies the scale and semantics.
- Maintain data APIs through authentication, authorization, versioned contracts, pagination, throttling, monitoring, and compatibility.
- Use AWS Glue DataBrew or Amazon SageMaker Unified Studio preparation surfaces for governed profiling and repeatable cleaning.
- Use Athena for serverless SQL over cataloged S3 data with partitions, output location, and permissions configured.
- Use Lambda for bounded event-driven processing and make side effects idempotent.
- Use EventBridge rules for matching events and EventBridge Scheduler for time-based invocation, with target role and retry behavior.
Compare and Distinguish
- Automation versus orchestration: invoking work differs from owning dependency and execution state.
- SDK credential chain versus stored keys: temporary workload identity differs from embedded long-term credentials.
- EventBridge rule versus Scheduler: event pattern matching differs from clock-based initiation.
- Athena query versus service-local warehouse processing: S3 serverless SQL differs from Redshift-local workload execution.
Scenario examples
- A bounded historical backfill runs as parameterized workflow executions without changing the regular schedule.
- Lambda writes with a stable operation key so an SDK retry after a timeout does not create a duplicate output.
- An EventBridge rule starts event-driven processing while Scheduler runs a nightly reconciliation.
Exam traps
- Embedding access keys because code uses an SDK.
- Assuming SDK retries provide business idempotency.
- Treating EventBridge rules and Scheduler as one identical resource.
- Running an unbounded job in Lambda because it is serverless.
Key takeaways
- Automate identity, progress, retries, and verification together.
- Choose the service-local processing surface when it meets the requirement.
- Operate backfills as controlled runs, not global schedule edits.
How it works
- An event or schedule invokes a workload role, the operation records durable progress, and retry logic distinguishes transient from terminal errors.
- Outputs are verified against the requested run or watermark before the automation advances or reports success.
When to use it
- Use service-native Glue, EMR, Redshift, or Athena processing when the data location and semantics fit that service.
- Use Lambda for bounded event-driven work and a workflow service when dependencies, backfills, or multi-step state must be owned.
Security and governance implications
- Use temporary workload credentials and limit each automated action to its required APIs and data locations.
- Version and authorize data APIs separately from the automation that calls them, including pagination and quota behavior.
Common failure modes and diagnosis
- For a failed automated call, inspect credential resolution, authorization, pagination token, retry classification, and service response.
- For duplicate or missing output, compare event delivery, operation keys, checkpoints, and the verification step after the write.
More detail
- 3.1.1: Orchestrate data pipelines (for example, Amazon Managed Workflows for Apache Airflow [Amazon MWAA], AWS Step Functions).
- 3.1.2: Troubleshoot Amazon managed workflows.
- 3.1.3: Call SDKs to access Amazon features from code.
- 3.1.4: Use the features of AWS services to process data (for example, Amazon EMR, Amazon Redshift, AWS Glue).
- 3.1.5: Consume and maintain data APIs.
- 3.1.6: Prepare data for transformation (for example, AWS Glue DataBrew and Amazon SageMaker Unified Studio).
- 3.1.7: Query data (for example, Amazon Athena).
- 3.1.8: Use AWS Lambda to automate data processing.
- 3.1.9: Manage events and schedulers (for example, Amazon EventBridge).
Ready for the quiz?
- Which component owns the trigger, execution identity, retry state, and verification of the automated action?
- Can a replay or SDK retry repeat the operation safely without changing the regular schedule or duplicating output?
Related objectives
- D3.1 — Task 3.1: Automate data processing by using AWS services
- 3.1.1 — Orchestrate data pipelines (for example, Amazon Managed Workflows for Apache Airflow [Amazon MWAA], AWS Step Functions).
- 3.1.2 — Troubleshoot Amazon managed workflows.
- 3.1.3 — Call SDKs to access Amazon features from code.
- 3.1.4 — Use the features of AWS services to process data (for example, Amazon EMR, Amazon Redshift, AWS Glue).
- 3.1.5 — Consume and maintain data APIs.
- 3.1.6 — Prepare data for transformation (for example, AWS Glue DataBrew and Amazon SageMaker Unified Studio).
- 3.1.7 — Query data (for example, Amazon Athena).
- 3.1.8 — Use AWS Lambda to automate data processing.
- 3.1.9 — Manage events and schedulers (for example, Amazon EventBridge).