Data Operations and Support
Pipeline Monitoring and Support
CoreCollect audit and application evidence, correlate runs, send actionable notifications, diagnose performance, maintain Glue and EMR, and choose a log-analysis engine by location and query pattern.
Aligned to AWS Certified Data Engineer - Associate (DEA-C01) Version 1.1, verified August 25, 2026.
Why this matters
An alarm reports a symptom; it does not locate the slow stage or preserve an audit. Support decisions require correlated metrics, logs, API history, run identity, and engine-specific evidence.
Must Know
- Extract audit evidence with defined account, Region, event, time, retention, integrity, and access scope rather than exporting arbitrary logs.
- Emit structured run, source, partition, and correlation identifiers across stages so an output can be traced to an execution.
- Alert on actionable or terminal state through a managed notification path; avoid paging repeatedly for every retry.
- Diagnose latency from stage duration, queue lag, throttles, resource saturation, file layout, partition distribution, and recent changes.
- Use CloudTrail for AWS API identity and action evidence; it does not contain application record values or guest metrics.
- Troubleshoot Glue and EMR with job or step logs, dependency state, schema, partition data, resource health, and service metrics.
- Send ephemeral application logs to CloudWatch Logs automatically with structure, retention, access, and secret filtering.
- Use Logs Insights for CloudWatch log analysis, Athena for SQL over archived S3 logs, OpenSearch for its indexed search/analytics model, and EMR where large-scale processing requires it.
Compare and Distinguish
- Logs versus metrics versus CloudTrail: event detail, time-series symptoms, and API audit history answer different questions.
- Alarm versus diagnostic telemetry: detection differs from root-cause evidence.
- CloudWatch Logs Insights versus Athena versus OpenSearch: storage location and ingestion/query model drive the choice.
- Application log versus audit trail: workload context differs from AWS control-plane activity.
Scenario examples
- A latency alarm leads operators to compare per-stage duration, iterator age, throttles, worker metrics, and skew before scaling.
- CloudTrail identifies who changed a catalog permission; structured pipeline logs show which run then failed.
- Recent CloudWatch logs use Logs Insights while partitioned S3 archives use Athena.
Exam traps
- Treating a dashboard alarm as complete traceability.
- Adding workers everywhere when one partition causes the delay.
- Expecting CloudTrail to record application payload contents.
- Leaving logs only on ephemeral workers.
Key takeaways
- Collect evidence with consistent run identity and scope.
- Separate audit, detection, diagnosis, and notification.
- Choose log analysis from data location and query needs.
How it works
- Pipeline stages emit structured identifiers and metrics that can be aligned by run, partition, timestamp, account, and Region.
- Alerts route actionable states to owners, who use service logs and history to isolate the smallest failing boundary.
When to use it
- Use Logs Insights for CloudWatch log groups, Athena for partitioned S3 archives, and OpenSearch for indexed search and dashboards.
- Use CloudTrail when the investigation depends on the actor and API call, and workload logs when it depends on record or stage behavior.
Security and governance implications
- Protect log integrity and access, apply deliberate retention, and remove secrets or unnecessary sensitive payloads before ingestion.
- Separate alert delivery permissions from permission to change the monitored pipeline.
Common failure modes and diagnosis
- For rising duration, compare stage timings, queue lag, throttles, resource saturation, data distribution, and deployment changes.
- For missing evidence, verify emission, permissions, destination, retention, timestamp alignment, and correlation identifiers.
More detail
- 3.3.1: Extract logs for audits.
- 3.3.2: Deploy logging and monitoring solutions to facilitate auditing and traceability.
- 3.3.3: Use notifications during monitoring to send alerts.
- 3.3.4: Troubleshoot performance issues.
- 3.3.5: Use AWS CloudTrail to track API calls.
- 3.3.6: Troubleshoot and maintain pipelines (for example, AWS Glue, Amazon EMR).
- 3.3.7: Use Amazon CloudWatch Logs to log application data (with a focus on configuration and automation).
- 3.3.8: Analyze logs with AWS services (for example, Athena, Amazon EMR, Amazon OpenSearch Service, CloudWatch Logs Insights, big data application logs).
Ready for the quiz?
- Which combination of metrics, logs, API history, and run identifiers can locate the first failing stage?
- Is the chosen log-analysis engine aligned to CloudWatch Logs, archived S3 data, or an indexed search workload?
Related objectives
- D3.3 — Task 3.3: Maintain and monitor data pipelines
- 3.3.1 — Extract logs for audits.
- 3.3.2 — Deploy logging and monitoring solutions to facilitate auditing and traceability.
- 3.3.3 — Use notifications during monitoring to send alerts.
- 3.3.4 — Troubleshoot performance issues.
- 3.3.5 — Use AWS CloudTrail to track API calls.
- 3.3.6 — Troubleshoot and maintain pipelines (for example, AWS Glue, Amazon EMR).
- 3.3.7 — Use Amazon CloudWatch Logs to log application data (with a focus on configuration and automation).
- 3.3.8 — Analyze logs with AWS services (for example, Athena, Amazon EMR, Amazon OpenSearch Service, CloudWatch Logs Insights, big data application logs).