Data Ingestion and Transformation
Data Transformation and Processing
CoreChoose and operate transformation engines, connectors, formats, integration contracts, container resources, APIs, and bounded LLM processing from workload evidence.
Aligned to AWS Certified Data Engineer - Associate (DEA-C01) Version 1.1, verified August 25, 2026.
Why this matters
A transformation can finish successfully and still be slow, expensive, inconsistent, or unsafe. Data engineers must preserve keys and types while matching the engine and representation to scale and access.
Must Know
- Choose AWS Glue for managed data integration and ETL, Amazon EMR for customizable distributed frameworks, Lambda for bounded event-driven work, and Redshift SQL when processing belongs beside warehouse data.
- Container performance work starts with measured CPU, memory, I/O, backlog, and data distribution. EKS or ECS selection alone does not remove the need for resource requests and scaling.
- JDBC and ODBC connections still require a supported driver, endpoint, network route, authentication secret, and compatible connection properties.
- Multi-source integration needs explicit schemas, join keys, timing or watermark rules, duplicate handling, and a defined merge contract.
- Columnar formats such as Parquet reduce analytical scans through column selection, compression, and partition pruning; renaming a CSV file does not convert its encoding.
- Diagnose slow or failed transformations from job logs, stage metrics, schema changes, dependency state, file layout, and skew before adding capacity.
- A data API needs independent authentication, authorization, pagination, quotas, compatibility, and monitoring rather than direct store credentials.
- Use an LLM for a bounded language-processing step, validate its structured output, retain provenance, and keep deterministic validation outside the model. Training and inference science are outside this role boundary.
Compare and Distinguish
- Glue versus EMR versus Lambda versus Redshift: managed ETL, customizable distributed compute, bounded functions, and warehouse-local SQL differ in runtime and ownership.
- CSV or JSON versus Parquet: interchange-oriented row/text data differs from compressed columnar analytical storage.
- Resource shortage versus skew: uniformly busy workers differ from one partition or key dominating execution.
- Deterministic transformation versus LLM-assisted extraction: predictable rules and generated outputs require different validation boundaries.
Scenario examples
- A recurring Athena workload converts validated CSV into partitioned compressed Parquet and verifies row counts, keys, and types.
- A Spark-compatible transformation needing custom libraries uses EMR, while a short object validation remains event-driven Lambda work.
- Bedrock extracts candidate fields from text; a deterministic schema and range validator quarantines invalid outputs before publication.
Exam traps
- Adding workers before checking whether one key or partition causes skew.
- Treating an HTTP API as governed merely because it hides the database endpoint.
- Assuming format conversion also reconciles semantic differences between sources.
- Using an LLM to replace deterministic validation or author access decisions.
Key takeaways
- Select engine, connector, and format from measured constraints.
- Preserve semantics and validation evidence, not only bytes.
- Keep LLM integration bounded to data processing and validate its output.
How it works
- The pipeline reads through an authenticated connector, applies transformations in the selected engine, validates the resulting contract, and isolates rejected records.
- Operational evidence from stages, partitions, and resources distinguishes an undersized runtime from skew, bad input, or a dependency failure.
When to use it
- Use serverless ETL for managed integration, distributed frameworks for custom large-scale processing, bounded functions for short events, and warehouse SQL for data already in Redshift.
- Use a model-assisted extraction step only when language processing is required and deterministic validation can govern the produced structure.
Security and governance implications
- Keep source credentials, connection secrets, routes, job roles, and output permissions scoped to the transformation path.
- Protect model inputs and outputs, retain provenance, and prevent generated content from making authorization or validation decisions.
Common failure modes and diagnosis
- When a transform slows, compare stage duration, worker use, shuffle, partitions, file sizes, and dominant keys before scaling.
- When output changes unexpectedly, trace source schema, connector properties, type conversion, null handling, and rejected-record evidence.
More detail
- 1.2.1: Optimize container usage for performance needs (for example, Amazon EKS, Amazon ECS).
- 1.2.2: Connect to different data sources (for example, Java Database Connectivity [JDBC], Open Database Connectivity [ODBC]).
- 1.2.3: Integrate data from multiple sources.
- 1.2.4: Optimize costs while processing data.
- 1.2.5: Implement data transformation services based on requirements (for example, Amazon EMR, AWS Glue, Lambda, Amazon Redshift).
- 1.2.6: Transform data between formats (for example, from .csv to Apache Parquet).
- 1.2.7: Troubleshoot and debug common transformation failures and performance issues.
- 1.2.8: Create data APIs to make data available to other systems by using AWS services.
- 1.2.9: Define volume, velocity, and variety of data (for example, structured data, unstructured data).
- 1.2.10: Integrate large language models (LLMs) for data processing.
Ready for the quiz?
- Which engine satisfies the workload's scale, runtime, library, data-locality, and operational-control requirements?
- How will the pipeline prove that keys, types, nulls, and rejected records remain correct after transformation?
Related objectives
- D1.2 — Task 1.2: Transform and process data
- 1.2.1 — Optimize container usage for performance needs (for example, Amazon EKS, Amazon ECS).
- 1.2.2 — Connect to different data sources (for example, Java Database Connectivity [JDBC], Open Database Connectivity [ODBC]).
- 1.2.3 — Integrate data from multiple sources.
- 1.2.4 — Optimize costs while processing data.
- 1.2.5 — Implement data transformation services based on requirements (for example, Amazon EMR, AWS Glue, Lambda, Amazon Redshift).
- 1.2.6 — Transform data between formats (for example, from .csv to Apache Parquet).
- 1.2.7 — Troubleshoot and debug common transformation failures and performance issues.
- 1.2.8 — Create data APIs to make data available to other systems by using AWS services.
- 1.2.9 — Define volume, velocity, and variety of data (for example, structured data, unstructured data).
- 1.2.10 — Integrate large language models (LLMs) for data processing.