Data Store Management
Technical and Business Data Catalogs
CoreBuild, update, consume, connect, and govern technical catalog metadata while separating it from business ownership, discovery, and access workflows.
Aligned to AWS Certified Data Engineer - Associate (DEA-C01) Version 1.1, verified August 25, 2026.
Why this matters
Catalogs connect data locations and schemas to engines and people. Queries fail when partitions, formats, permissions, or connections drift; governance fails when technical metadata is mistaken for ownership and approved meaning.
Must Know
- The AWS Glue Data Catalog stores technical databases, tables, schemas, formats, locations, and partitions that engines such as Athena can reference.
- A catalog table does not by itself grant access to underlying S3 data or Lake Formation-governed tables; metadata and data permissions are separate.
- Use a crawler when controlled schema and partition inference is appropriate. Configure targets, classifiers, role, and update behavior so malformed input does not overwrite trusted metadata.
- Register new partitions through a crawler or explicit partition operations that match the object path scheme. S3 prefixes do not automatically become catalog partitions.
- A Glue connection to a private JDBC source needs compatible connection properties, stored credentials, subnet routing, security groups, and an authorized role.
- A technical catalog serves execution metadata; SageMaker Catalog and Unified Studio surfaces add business descriptions, ownership, domains, projects, subscriptions, and governed discovery.
- Schema changes should be validated before publication, regardless of whether discovery is automated.
Compare and Distinguish
- Technical catalog versus business catalog: machine-readable schema/location metadata differs from business meaning, ownership, and discovery.
- Crawler discovery versus explicit definition: inferred metadata differs from controlled schema contracts.
- Metadata visibility versus data authorization: seeing a table differs from permission to read its data.
- Catalog connection versus network reachability: connection metadata does not create a route or security-group rule.
Scenario examples
- Athena resolves a governed table through Glue Catalog and uses separate Lake Formation and S3 permissions to read it.
- A crawler discovers date partitions but is configured not to replace a trusted schema after a malformed file arrives.
- A domain owner publishes a business data product through a project subscription while engines continue to use technical catalog metadata.
Exam traps
- Assuming object folders are automatically queryable partitions.
- Granting metadata access while omitting governed data permissions.
- Letting crawler inference overwrite a production contract without validation.
- Using a business glossary as the execution schema.
Key takeaways
- Catalog metadata, source connectivity, data authorization, and business governance are separate layers.
- Automate discovery only with controlled update behavior.
- Keep partitions synchronized with actual object layout.
How it works
- Crawlers or explicit operations create and update catalog databases, tables, schemas, and partitions that engines reference.
- Business assets link ownership and approved meaning to technical data while subscriptions and permissions control actual use.
When to use it
- Use a crawler for controlled recurring inference and explicit definitions when the schema contract must be tightly managed.
- Use Glue Data Catalog for engine metadata and business catalog surfaces for domains, owners, projects, and governed discovery.
Security and governance implications
- Authorize catalog updates separately from catalog reads and from access to the underlying S3 or Lake Formation data.
- A private JDBC catalog connection needs stored credentials, routing, security groups, and an authorized role in addition to connection metadata.
Common failure modes and diagnosis
- When a query misses data, compare object paths, registered partitions, table format, schema, and engine permissions.
- When discovery works but reads fail, trace catalog visibility, Lake Formation grants, S3 access, connection credentials, and network reachability.
More detail
- 2.2.1: Use data catalogs to consume data from the data's source.
- 2.2.2: Build and reference a technical data catalog (for example, AWS Glue Data Catalog, Apache Hive metastore).
- 2.2.3: Discover schemas and use AWS Glue crawlers to populate data catalogs.
- 2.2.4: Synchronize partitions with a data catalog.
- 2.2.5: Create new source or target connections for cataloging (for example, AWS Glue).
- 2.2.6: Create and manage business data catalogs (for example, Amazon SageMaker Catalog).
Ready for the quiz?
- Is the requirement executable schema and partition metadata, or business ownership, discovery, and subscription context?
- Which permission or connection layer is still missing when a table is visible but its data cannot be queried?
Related objectives
- D2.2 — Task 2.2: Understand data cataloging systems
- 2.2.1 — Use data catalogs to consume data from the data's source.
- 2.2.2 — Build and reference a technical data catalog (for example, AWS Glue Data Catalog, Apache Hive metastore).
- 2.2.3 — Discover schemas and use AWS Glue crawlers to populate data catalogs.
- 2.2.4 — Synchronize partitions with a data catalog.
- 2.2.5 — Create new source or target connections for cataloging (for example, AWS Glue).
- 2.2.6 — Create and manage business data catalogs (for example, Amazon SageMaker Catalog).