Design High-Performing Architectures
Data Ingestion, Streaming, Transformation, and Analytics
CoreDesign secure batch/stream ingestion, data-lake governance, transformation, processing, querying, and visualization.
Aligned to the current SAA-C03 exam guide, verified August 16, 2026.
Why this matters
A data pipeline should be chosen from arrival cadence, ordering, replay, latency, transformation, governance, and consumption—not from a list of analytics products. The design must keep ingestion, durable storage, catalog/governance, processing, query, and presentation as separate decisions.
Must Know
- Batch ingestion moves bounded datasets periodically; streaming ingestion handles continuing records where processing latency and replay semantics matter.
- Kinesis Data Streams supports custom real-time stream producers/consumers, ordering within a partition, and retained records for multiple consumers/replay according to configuration.
- Amazon Data Firehose manages buffering, optional transformation, and delivery of streaming data to supported destinations with minimal administration; it is delivery-oriented rather than a custom consumer stream.
- S3 is a common durable data-lake storage layer. Lake Formation centralizes fine-grained data-lake governance, while Glue provides a catalog and managed data integration/ETL.
- Athena queries data in S3 with serverless SQL; Redshift is a warehouse; EMR runs distributed frameworks; Amazon Quick provides business intelligence, visualization, and current generative analytics capabilities.
- Columnar formats and partitioned layouts can reduce scanned data and improve analytical query efficiency. Transform based on downstream access, not format fashion.
- DataSync moves files/objects online, Transfer Family provides managed partner/client file-transfer protocols, Storage Gateway provides ongoing hybrid access, and Snow Family moves data offline or at the edge.
- Secure the ingestion identity and endpoint, encrypt transport/storage, govern catalog permissions, and separate producer, transformer, analyst, and administrator roles.
Compare and Distinguish
- Kinesis Data Streams vs Data Firehose: Streams wins for custom consumers, replay, and stream control; Firehose wins for managed buffered delivery to supported destinations.
- Kinesis vs SQS: Kinesis fits ordered partitioned streaming and multiple consumers/replay; SQS fits task buffering where each queued message is processed by a consumer path.
- Glue vs EMR: Glue wins for managed catalog/data integration and serverless ETL; EMR wins when distributed framework/cluster control is required.
- Athena vs Redshift vs EMR vs Quick: query S3 directly, run a warehouse, execute distributed processing, or present/analyze insights.
- Lake Formation vs Glue Data Catalog: Lake Formation governs lake permissions; the Glue catalog stores table/metadata definitions used by analytics services.
- DataSync vs Storage Gateway vs Transfer Family vs Snow Family: online movement, ongoing hybrid interface, managed external file-transfer endpoint, or offline transfer are different requirements.
Scenario examples
- Scenario: Devices generate ordered records that several applications must process and replay. Think: Kinesis Data Streams with a partition key that distributes demand while preserving required order.
- Scenario: Logs need low-operations delivery to S3 for later analysis. Think: Data Firehose handles buffered delivery; a custom stream is unnecessary.
- Scenario: Analysts query curated S3 data occasionally with SQL. Think: Glue catalog plus Athena avoids a continuously running warehouse.
- Scenario: Petabytes must move from a site with inadequate bandwidth. Think: Snow Family satisfies the hard transfer constraint, then ingest into the lake.
- Scenario: Raw CSV queries scan too much data. Think: transform to a suitable columnar, partitioned layout and query only needed partitions.
Exam traps
- Data Firehose is not the same as a replayable custom consumer stream.
- Athena does not turn S3 into an OLTP database.
- Quick visualizes/analyzes data; it is not the ingestion or durable storage layer.
- A Glue crawler/catalog does not by itself grant governed access to all consumers.
- Snow Family is not selected when a practical online path already meets time and cost needs.
- Changing CSV to columnar format does not fix a bad partition strategy or uncontrolled access.
Key takeaways
- Separate ingest, store, catalog/govern, transform, process, query, and visualize.
- Choose batch versus stream from latency and arrival cadence.
- Use Data Streams for consumer control/replay and Firehose for managed delivery.
- Let network feasibility and protocol choose the transfer service.
- Optimize lake layout for actual queries and secure every pipeline stage.
How it works
- Producers write records or transfer files through authenticated endpoints.
- Streams retain ordered partition records for consumers; Firehose buffers and delivers batches to supported destinations.
- Glue catalogs and transforms datasets; Lake Formation applies governed lake access; query/process services consume the prepared data.
- Visualization reads from supported analytical sources rather than replacing them.
When to use it
- Use Data Streams for real-time custom processing, Firehose for low-operations delivery, and S3 for durable lake storage.
- Use Glue for catalog/ETL, Lake Formation for lake governance, Athena for ad hoc S3 SQL, EMR for distributed frameworks, Redshift for warehousing, and Quick for BI.
- Use DataSync online, Transfer Family for managed transfer protocols, Storage Gateway for hybrid access, and Snow when the network is impractical.
Security and governance implications
- Use private endpoints or controlled public ingestion, least-privilege roles, and encryption.
- Separate raw-data write, transformation, catalog administration, and analyst permissions.
- Classify sensitive lake data and monitor access; avoid copying it into unmanaged extracts.
- Protect replay and dead-letter paths because they can contain the original sensitive payload.
Operational and diagnostic signals
- For missing records, trace producer acceptance, partition distribution, consumer checkpoints/retries, and destination errors.
- For slow lake queries, inspect format, partition pruning, file sizing, and scan volume.
- For denied analytics, separate storage policy, KMS permission, catalog permission, and Lake Formation grant.
- For transfer delay, compare source throughput, network capacity, protocol overhead, and destination ingestion.
More detail
- Partition keys balance streaming throughput and define the scope of ordering; poor distribution can create hot partitions.
- Data-lake zones and formats should support governance, retention, reprocessing, and consumer performance.
- The current Amazon Quick product line includes the business-intelligence capabilities formerly recognized through QuickSight terminology; focus on visualization and analytical consumption rather than name trivia.
- Pipeline throughput is limited by the slowest stage, and retries or buffering must prevent that stage from causing data loss.
Ready for the quiz?
- What requirement makes Data Streams better than Firehose?
- When does Glue beat EMR?
- How do Athena, Redshift, EMR, and Quick differ?
- Which transfer clue favors DataSync, Gateway, Transfer Family, or Snow?
- Why do columnar format and partitioning improve some queries?
Related objectives
- D3.5.K1
- D3.5.K2
- D3.5.K3
- D3.5.K4
- D3.5.K5
- D3.5.K6
- D3.5.K7
- D3.5.S1
- D3.5.S2
- D3.5.S3
- D3.5.S4
- D3.5.S5
- D3.5.S6
- D3.5.S7