An end-to-end Azure Data Engineering project built using Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, Azure Synapse Analytics, and Power BI.
The project demonstrates how raw source data can be ingested, stored, transformed, processed through Bronze/Silver/Gold layers, and made available for analytics and reporting.
This project implements a modern cloud-based data engineering pipeline for the AdventureWorks dataset.
The solution follows a Medallion Architecture:
- 🥉 Bronze Layer → Raw/Ingested Data
- 🥈 Silver Layer → Cleaned & Transformed Data
- 🥇 Gold Layer → Business-Ready Data
Azure Data Factory is used for data ingestion and orchestration, Azure Data Lake Storage Gen2 is used as the central data lake, Azure Databricks is used for data transformation, and Azure Synapse Analytics is used for analytical querying.
┌─────────────────────┐
│ Source Data │
│ AdventureWorks │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Azure Data │
│ Factory │
│ Ingestion & ETL │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ ADLS Gen2 │
│ Data Lake │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Azure Databricks │
│ Transformation │
└──────────┬──────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Bronze │ → │ Silver │ → │ Gold │
│ Raw │ │ Cleaned │ │ Business │
│ Data │ │ Data │ │ Ready │
└──────────┘ └──────────┘ └─────┬────┘
│
▼
┌─────────────────────┐
│ Azure Synapse │
│ Analytics │
│ SQL / Analytics │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Power BI │
│ Reporting & │
│ Visualization │
└─────────────────────┘