Data Store Management
Data Models, Schema Evolution, and Lineage
CoreDesign store-specific schemas and physical layouts, evolve contracts, convert database schemas, trace lineage, and separate vector generation from vector indexing.
Aligned to AWS Certified Data Engineer - Associate (DEA-C01) Version 1.1, verified August 25, 2026.
Why this matters
A logical field list is not enough. Data engineers must design for each store’s access model, handle compatible and incompatible changes, preserve provenance, and optimize physical layout without conflating adjacent metadata layers.
Must Know
- Design Redshift tables for analytical joins, distribution, sort, compression, and pruning; design DynamoDB keys from request patterns; apply Lake Formation permissions to cataloged data at the governance layer.
- Define schema compatibility and version contracts. Optional additions can be compatible, while type or meaning changes often require consumer migration.
- Use current AWS DMS Schema Conversion to assess and convert database objects, review unsupported or partially converted objects, and apply or export converted schema. Data movement remains a separate DMS operation.
- Lineage records source, transformation, run, and output relationships; catalog metadata records definitions and locations. Link them without treating one as the other.
- Choose partitioning, compaction, compression, sorting, distribution, and indexes from measured filters, joins, file size, and data distribution.
- Vectorization produces embeddings from source content; a vector index organizes existing embeddings for similarity retrieval.
- Bedrock knowledge bases can integrate source ingestion, chunking, embeddings, vector storage, and retrieval configuration without requiring the candidate to train a model.
- Use Apache Iceberg metadata operations for supported schema and partition evolution rather than rewriting consumer-visible folder conventions.
Compare and Distinguish
- Redshift schema versus DynamoDB key model versus Lake Formation: analytical layout, request-key design, and data permission governance differ.
- Schema evolution versus schema conversion: changing an active contract differs from translating database objects between engines.
- Lineage versus catalog: transformation provenance differs from discoverable technical or business metadata.
- Vectorization versus vector index: embedding generation differs from nearest-neighbor organization.
Scenario examples
- A producer adds an optional field under a versioned contract, but a later type change triggers validation and consumer migration.
- A heterogeneous database migration uses DMS Schema Conversion assessment before moving data.
- A knowledge-base pipeline chunks and vectorizes documents, stores metadata with embeddings, and configures a separate retrieval index.
Exam traps
- Assuming schema-on-read removes compatibility obligations.
- Believing data migration automatically converts every database object.
- Using CloudTrail or a catalog location as complete field-level lineage.
- Claiming HNSW generates embeddings.
Key takeaways
- Design logical and physical models for the actual store.
- Version and validate schema change before publication.
- Keep conversion, lineage, vectorization, and indexing boundaries explicit.
How it works
- Producers publish a versioned schema, consumers validate compatibility, and incompatible changes trigger a controlled migration.
- Lineage links source, transformation run, and output while physical evidence guides partition, compression, distribution, or index choices.
When to use it
- Use Redshift table design for analytical joins, DynamoDB key design for request patterns, and Lake Formation at the governance layer.
- Use Iceberg metadata operations for supported table evolution and a vector index only after embeddings have been produced.
Security and governance implications
- Keep lineage and catalog access scoped because provenance can reveal sensitive source and processing relationships.
- Apply Lake Formation, store, and KMS permissions at their respective layers instead of treating schema design as authorization.
Common failure modes and diagnosis
- For a breaking change, compare producer and consumer schema versions, field types, defaults, and rejected records.
- For poor query or retrieval performance, inspect file sizes, partitions, sort or distribution choices, and index behavior using measured workloads.
More detail
- 2.4.1: Design schemas for Amazon Redshift, DynamoDB, and Lake Formation.
- 2.4.2: Address changes to the characteristics of data.
- 2.4.3: Perform schema conversion by using the current AWS DMS Schema Conversion workflow.
- 2.4.4: Establish data lineage by using AWS tools (for example, Amazon SageMaker ML Lineage Tracking and Amazon SageMaker Catalog).
- 2.4.5: Describe best practices for indexing, partitioning strategies, compression, and other data optimization techniques.
- 2.4.6: Describe vectorization concepts (for example, Amazon Bedrock knowledge base).
Ready for the quiz?
- Is the task evolving an active contract, converting database objects, or changing a store-specific physical design?
- Does the evidence describe where data is defined, how it was transformed, or how vectors are organized for retrieval?
Related objectives
- D2.4 — Task 2.4: Design data models and schema evolution
- 2.4.1 — Design schemas for Amazon Redshift, DynamoDB, and Lake Formation.
- 2.4.2 — Address changes to the characteristics of data.
- 2.4.3 — Perform schema conversion (for example, by using AWS SCT and AWS DMS Schema Conversion).
- 2.4.4 — Establish data lineage by using AWS tools (for example, Amazon SageMaker ML Lineage Tracking and Amazon SageMaker Catalog).
- 2.4.5 — Describe best practices for indexing, partitioning strategies, compression, and other data optimization techniques.
- 2.4.6 — Describe vectorization concepts (for example, Amazon Bedrock knowledge base).