Medallion Lakehouse on Databricks
A reference implementation of an Azure Lakehouse: Data Factory ingestion, Auto Loader into Delta, bronze/silver/gold modelling, Unity Catalog governance and BI-ready gold tables.
The problem
“Lakehouse” means different things to different teams. This project builds one end to end, so the decisions are concrete rather than slideware.
Scope
- Ingestion: Azure Data Factory copies source extracts into a landing zone; Auto Loader picks them up incrementally.
- Bronze: raw, append-only Delta tables with ingestion metadata.
- Silver: typed, deduplicated and conformed entities, upserted with
MERGE. - Gold: a star schema with clearly defined grain, ready for BI and ML feature tables.
- Governance: Unity Catalog for permissions and lineage.
- Delivery: infrastructure and jobs deployed as code with Databricks Asset Bundles and CI/CD.
Milestones
- Landing zone and ADF pipeline
- Bronze and silver with Auto Loader and
MERGE - Gold star schema and data quality expectations
- Jobs and permissions as code, deployed from CI
- Write-up: cost, performance and the trade-offs I’d change