Analytics
CoreAWS Glue
Managed data integration spanning catalogs, crawlers, ETL, workflows, monitoring, and quality.
Key points
- Glue combines the Data Catalog, crawlers, connections, serverless ETL jobs, workflows, and data-quality capabilities.
- Catalog visibility, underlying data permissions, network reachability, and job-role permissions remain separate requirements.
Best-known use cases
- Discover data sources and register schemas in a shared Data Catalog.
- Run serverless ETL jobs that clean, join, and convert datasets.
- Apply data quality rules before publishing curated data.
What candidates often confuse it with
- Glue supplies managed ETL and catalog services; EMR supplies more customizable distributed compute.
Key takeaway
Use Glue when the pipeline needs managed discovery, ETL, catalog, workflow, or data-quality features.
Relevant exam tasks
- D1.1 — Task 1.1: Perform data ingestion
- 1.1.1 — Read data from streaming sources (for example, Amazon Kinesis, Amazon Managed Streaming for Apache Kafka [Amazon MSK], Amazon DynamoDB Streams, AWS DMS, AWS Glue, Amazon Redshift).
- 1.1.2 — Read data from batch sources (for example, Amazon S3, AWS Glue, Amazon EMR, AWS DMS, Amazon Redshift, AWS Lambda, Amazon AppFlow).
- D1.2 — Task 1.2: Transform and process data
- 1.2.5 — Implement data transformation services based on requirements (for example, Amazon EMR, AWS Glue, Lambda, Amazon Redshift).
- D1.3 — Task 1.3: Orchestrate data pipelines
- 1.3.1 — Use orchestration services to build workflows for data ETL pipelines (for example, Lambda, EventBridge, Amazon Managed Workflows for Apache Airflow [Amazon MWAA], AWS Step Functions, AWS Glue workflows).
- D2.2 — Task 2.2: Understand data cataloging systems
- 2.2.1 — Use data catalogs to consume data from the data's source.
- 2.2.2 — Build and reference a technical data catalog (for example, AWS Glue Data Catalog, Apache Hive metastore).
- 2.2.3 — Discover schemas and use AWS Glue crawlers to populate data catalogs.
- 2.2.4 — Synchronize partitions with a data catalog.
- 2.2.5 — Create new source or target connections for cataloging (for example, AWS Glue).
- 2.2.6 — Create and manage business data catalogs (for example, Amazon SageMaker Catalog).
- D2.4 — Task 2.4: Design data models and schema evolution
- 2.4.2 — Address changes to the characteristics of data.
- D3.1 — Task 3.1: Automate data processing by using AWS services
- 3.1.4 — Use the features of AWS services to process data (for example, Amazon EMR, Amazon Redshift, AWS Glue).
- D3.3 — Task 3.3: Maintain and monitor data pipelines
- 3.3.6 — Troubleshoot and maintain pipelines (for example, AWS Glue, Amazon EMR).
- D3.4 — Task 3.4: Ensure data quality
- 3.4.1 — Run data quality checks while processing the data (for example, checking for empty fields).
- 3.4.2 — Define data quality rules (for example, DataBrew).
- 3.4.3 — Investigate data consistency (for example, DataBrew).
- 3.4.4 — Describe data sampling techniques.
- 3.4.5 — Implement data skew mechanisms.