Design data storage solutions
Data Integration and Analytics Architecture
ImportantDesign batch, streaming, orchestration, transformation, and analytical serving paths around latency, data shape, governance, and consumer needs.
Aligned to the current AZ-305 study guide, skills measured as of April 17, 2026, verified August 25, 2026.
Why this matters
A pipeline architecture must distinguish movement from transformation and operational processing from analytical consumption.
Must Know
- Separate data movement and orchestration from the engines that transform, store, and serve the data.
- Choose batch for bounded scheduled work and streaming for continuously arriving data that needs low-latency processing.
- Keep transactional stores isolated from analytical scans when shared workloads would disrupt operations.
- Design source connectivity, lineage, monitoring, governance, and residency across the complete pipeline rather than only at the sink.
Compare and Distinguish
- Azure Data Factory orchestrates data movement and transformation activities; event-streaming services ingest continuous event flows; messaging services coordinate application work.
- Batch processing optimizes bounded datasets and scheduled windows; stream processing handles continuously arriving events with low-latency needs.
- A lake stores diverse data at scale; a warehouse provides governed relational analytical serving; Power BI provides semantic modeling and visualization.
Scenario examples
- Scenario: A factory copies on-premises data nightly and transforms it in a managed Spark engine. Think: Data Factory orchestration with an appropriate integration runtime and transformation service.
- Scenario: Telemetry must be aggregated within seconds. Think: event ingestion plus stream processing and a serving store.
- Scenario: Executives need governed metrics over curated history. Think: an analytical model and reporting layer, not direct queries against production.
Exam traps
- Event Hubs is not a work-queue replacement when per-message settlement is required.
- Data Factory is an orchestrator and integration service, not the default low-latency operational database.
- A visualization tool does not itself create a governed ingestion architecture.
Key takeaways
- Classify latency, ordering, volume, transformation, and consumer requirements.
- Separate ingestion, orchestration, processing, storage, and serving decisions.
- Protect operational systems from analytical workload coupling.
How it works
- A Data Factory trigger creates a pipeline run, activities perform or dispatch work, linked services describe data stores or compute, and the integration runtime supplies the execution and connectivity bridge.
- Copy activities move data between sources and sinks, while transformation activities can dispatch processing to another compute engine; pipeline control flow sequences, branches, loops, and passes parameters between those steps.
- A streaming path appends events to a partitioned ingestion service, lets independent consumers track their progress, and writes processed results to a serving store suited to the required query latency.
Objects and administrative surfaces
- Sources, integration runtime location, network access, orchestration, transformation engine, sink, lineage, and monitoring are separate design surfaces.
- Operational and analytical stores should be separated when analytical scans would disturb transactional workloads.
- Data governance and residency can constrain regions, copies, and service boundaries.
When to use it
- Use Data Factory with the appropriate integration runtime for scheduled hybrid movement and orchestration.
- Use Event Hubs when continuous high-volume telemetry must feed independent consumers.
- Use a governed lake and analytical serving layer when historical reporting must remain isolated from transaction processing.
Security and governance implications
- Authorize each source, integration runtime, transformation, lake zone, and serving layer independently with managed identities where supported.
- Preserve lineage and residency across intermediate data, not only at the final analytical store.
How to validate and revise the design
- When a batch pipeline stalls, inspect source reachability, integration runtime health, credentials, activity dependencies, and sink throttling.
- When streaming results lag or diverge, compare partitioning, consumer position, processing state, replay boundary, and serving-store capacity.
More detail
- The integration runtime location can decide whether a pipeline can reach private or on-premises sources and whether movement satisfies regional and compliance constraints.
- A successful pipeline run proves that configured activities completed, not that the delivered data is semantically complete; reconciliation, idempotent reprocessing, and lineage remain architecture concerns.
- Streaming designs must define partitioning, ordering scope, checkpoint or replay behavior, and treatment of late or duplicate events before choosing throughput capacity.
Ready for the quiz?
- Is the workload bounded batch movement, continuous event ingestion, transformation, analytical serving, or a combination with clear boundaries?
- Which integration runtime, source connectivity, lineage, and governance controls span the path from producer to consumer?
- How will analytical consumers avoid degrading transactional workloads while receiving consistent governed measures?
Related objectives
- D2.3.S1 — Recommend a solution for data integration
- D2.3.S2 — Recommend a solution for data analysis