Data Operations and Support
Data Quality and Skew
CoreDefine and execute quality rules, investigate consistency, sample with known risk, quarantine and replay defects, and correct uneven data distribution.
Aligned to AWS Certified Data Engineer - Associate (DEA-C01) Version 1.1, verified August 25, 2026.
Why this matters
A successful job can publish invalid data, and a representative average can hide rare but important defects. Quality controls must be measurable, placed in the processing path, and connected to evidence and recovery.
Must Know
- Run quality checks at the processing boundary before curated publication; validate required fields, types, ranges, uniqueness, and contract-specific rules.
- Define rules with measurable thresholds, scope, and explicit pass, quarantine, or fail outcomes.
- Investigate consistency by run and watermark, comparing keys, counts, duplicates, missing records, late arrivals, and transformation logic.
- Choose random, systematic, or stratified sampling from population structure and risk. Convenience samples such as first rows can hide partition-specific defects.
- Data skew is uneven key or partition distribution that creates work imbalance; it is different from uniformly high volume.
- Mitigate skew with semantically safe repartitioning, salting, pre-aggregation, or key redesign and verify the resulting balance.
- Quarantine invalid records with reason, lineage, and original identity; replay corrected records idempotently through the same rules.
Compare and Distinguish
- Quality rule versus operational alarm: data validity differs from service health.
- Sampling error versus data skew: representativeness differs from execution imbalance.
- Quarantine versus silent drop: recoverable evidence differs from data loss.
- Equal row counts versus consistency: totals can match while keys or values differ.
Scenario examples
- Rows missing required identifiers are quarantined before publication and replayed after correction.
- Source and curated totals are reconciled by watermark and key rather than by total row count alone.
- A dominant tenant key is salted with a semantics-preserving second aggregation.
Exam traps
- Treating schema validity as complete quality.
- Sampling only the first or smallest files.
- Adding uniform workers to a hot-key problem.
- Editing curated output directly to repair quarantined records.
Key takeaways
- Make quality measurable and inline.
- Investigate consistency with keys and run boundaries.
- Distinguish population sampling from workload skew.
How it works
- Checks evaluate required fields, types, ranges, uniqueness, and reconciliation at the processing boundary before publication.
- Failed records retain identity, lineage, and reason codes so corrected data can be replayed through the same rules.
When to use it
- Use stratified sampling when important subgroups must be represented and full validation is impractical.
- Use salting, repartitioning, pre-aggregation, or key redesign only when the change preserves the workload's semantics.
Security and governance implications
- Restrict quarantined data because rejected records may contain the same or greater sensitivity as curated output.
- Preserve rule results and correction history as governed evidence without exposing unnecessary record contents.
Common failure modes and diagnosis
- When counts disagree, compare watermark, keys, duplicates, late arrivals, rejected records, and transformation logic by run.
- When one task dominates, inspect key frequencies and partition sizes, then verify that the chosen skew mitigation preserves aggregation results.
More detail
- 3.4.1: Run data quality checks while processing the data (for example, checking for empty fields).
- 3.4.2: Define data quality rules (for example, DataBrew).
- 3.4.3: Investigate data consistency (for example, DataBrew).
- 3.4.4: Describe data sampling techniques.
- 3.4.5: Implement data skew mechanisms.
Ready for the quiz?
- Which rule, population, threshold, and outcome define whether the dataset can be published?
- Does the evidence show a data defect, an unrepresentative sample, or uneven work caused by a dominant key?
Related objectives
- D3.4 — Task 3.4: Ensure data quality
- 3.4.1 — Run data quality checks while processing the data (for example, checking for empty fields).
- 3.4.2 — Define data quality rules (for example, DataBrew).
- 3.4.3 — Investigate data consistency (for example, DataBrew).
- 3.4.4 — Describe data sampling techniques.
- 3.4.5 — Implement data skew mechanisms.