Data Ingestion and Transformation
Streaming and Batch Data Ingestion
CoreSelect and configure streaming, batch, API, schedule, and event-driven ingestion paths with explicit progress, retry, replay, and consumer-isolation behavior.
Aligned to AWS Certified Data Engineer - Associate (DEA-C01) Version 1.1, verified August 25, 2026.
Why this matters
Ingestion correctness begins before transformation. Source contract, arrival pattern, retention, ordering, rate limits, and restart behavior determine whether a pipeline can recover without dropping or duplicating data.
Must Know
- Treat streaming as a retained or continuously delivered record flow with consumer progress; treat batch as bounded work over objects, extracts, or scheduled jobs.
- Choose Kinesis Data Streams when AWS-native retained streams and independent consumers fit; choose Amazon MSK when Kafka compatibility and Kafka operational semantics are required; use Amazon Data Firehose for managed delivery rather than general consumer-owned stream processing.
- Lambda reads Kinesis through an event source mapping in batches. Delivery is at least once, so handlers and downstream writes must tolerate duplicates.
- Incremental batch ingestion needs an explicit progress boundary such as partitions, bookmarks, or a source watermark. A schedule alone does not prove that only new data is read.
- API ingestion must preserve authentication, pagination state, throttling recovery, and a restart checkpoint. A successful first response does not imply that every page was consumed.
- Fan-in consolidates producers behind a compatible contract; fan-out gives independent consumers separate read progress and, where needed, isolated throughput.
- Replay requires retained source data, a known restart position, and idempotent or otherwise controlled output behavior.
- Stateful processing retains cross-record or window state and recovery checkpoints. Stateless record processing can still rely on external durable progress and idempotency.
Compare and Distinguish
- Schedule versus event trigger: a clock starts scheduled work; arrival or state change starts event-driven work.
- Kinesis Data Streams versus Amazon MSK versus Amazon Data Firehose: retained AWS stream primitives, Kafka compatibility, and managed delivery have different consumer and ownership models.
- Fan-in versus fan-out: many producers into one path is different from one stream feeding independent readers.
- Stateful versus stateless processing: window or cross-record state differs from independent record handling.
Scenario examples
- An irregular S3 object arrival starts an idempotent transform immediately, while a separate nightly schedule reconciles missed or delayed objects.
- A fixed-address partner allowlist is satisfied through controlled egress; attaching an IAM role alone would not establish an approved source IP.
- Independent quality and archive consumers read one Kinesis stream using separate checkpoints, and enhanced fan-out is considered when shared read throughput is the bottleneck.
Exam traps
- Assuming an event notification guarantees exactly one invocation.
- Calling a one-time delivery path replayable without retained source records and a restart position.
- Sharing one consumer checkpoint among applications that must progress independently.
- Responding to sustained throttling with unlimited immediate retries.
Key takeaways
- Choose ingestion from the source and recovery contract, not from the word streaming or batch alone.
- Make progress, duplicate handling, retry, and replay explicit.
- Separate schedule, event routing, stream consumption, and notification decisions.
How it works
- A producer writes to a batch or streaming source, and each consumer advances from an explicit object, watermark, shard, or offset boundary.
- Retries resume from durable progress while rate controls and idempotent writes prevent throttling or duplicate delivery from corrupting results.
When to use it
- Use event-driven ingestion for irregular arrivals that need prompt handling, and add scheduled reconciliation when missed or delayed input must be found.
- Use retained streams when consumers require replay or independent positions; use managed delivery when records only need buffering and delivery to supported destinations.
Security and governance implications
- Authorize source reads, stream consumption, checkpoints, and destination writes separately with workload roles.
- Validate network routes, allowlisted egress, source credentials, and service permissions instead of treating any one control as complete connectivity.
Common failure modes and diagnosis
- For missing records, compare producer acceptance, shard or partition position, consumer lag, checkpoint state, and destination acknowledgments.
- For throttling or duplicates, inspect key distribution, concurrency, retry timing, and whether the output operation is idempotent.
More detail
- 1.1.1: Read data from streaming sources (for example, Amazon Kinesis, Amazon Managed Streaming for Apache Kafka [Amazon MSK], Amazon DynamoDB Streams, AWS DMS, AWS Glue, Amazon Redshift).
- 1.1.2: Read data from batch sources (for example, Amazon S3, AWS Glue, Amazon EMR, AWS DMS, Amazon Redshift, AWS Lambda, Amazon AppFlow).
- 1.1.3: Implement appropriate configuration options for batch ingestion.
- 1.1.4: Consume data APIs.
- 1.1.5: Set up schedulers by using Amazon EventBridge, Apache Airflow, or time-based schedules for jobs and crawlers.
- 1.1.6: Set up event triggers (for example, Amazon S3 Event Notifications, EventBridge).
- 1.1.7: Call a Lambda function from Kinesis.
- 1.1.8: Create allowlists for IP addresses to allow connections to data sources.
- 1.1.9: Implement throttling and overcoming rate limits (for example, DynamoDB, Amazon RDS, Kinesis).
- 1.1.10: Manage fan-in and fan-out for streaming data distribution.
- 1.1.11: Describe replayability of data ingestion pipelines.
- 1.1.12: Define stateful and stateless data transactions.
Ready for the quiz?
- Does the source require retained records and independent consumer progress, or only managed delivery to a destination?
- What checkpoint, retry, and duplicate-handling design lets ingestion resume without losing or repeating an uncontrolled side effect?
Related objectives
- D1.1 — Task 1.1: Perform data ingestion
- 1.1.1 — Read data from streaming sources (for example, Amazon Kinesis, Amazon Managed Streaming for Apache Kafka [Amazon MSK], Amazon DynamoDB Streams, AWS DMS, AWS Glue, Amazon Redshift).
- 1.1.2 — Read data from batch sources (for example, Amazon S3, AWS Glue, Amazon EMR, AWS DMS, Amazon Redshift, AWS Lambda, Amazon AppFlow).
- 1.1.3 — Implement appropriate configuration options for batch ingestion.
- 1.1.4 — Consume data APIs.
- 1.1.5 — Set up schedulers by using Amazon EventBridge, Apache Airflow, or time-based schedules for jobs and crawlers.
- 1.1.6 — Set up event triggers (for example, Amazon S3 Event Notifications, EventBridge).
- 1.1.7 — Call a Lambda function from Kinesis.
- 1.1.8 — Create allowlists for IP addresses to allow connections to data sources.
- 1.1.9 — Implement throttling and overcoming rate limits (for example, DynamoDB, Amazon RDS, Kinesis).
- 1.1.10 — Manage fan-in and fan-out for streaming data distribution.
- 1.1.11 — Describe replayability of data ingestion pipelines.
- 1.1.12 — Define stateful and stateless data transactions.