Open Lakehouse with Apache Iceberg
A vendor-neutral Lakehouse you can run on a laptop: Spark and Apache Iceberg on S3-compatible object storage with a REST catalog, compared side by side with the Databricks approach.
The problem
Open table formats promise a Lakehouse without lock-in. How much work is it to run one yourself, and what do you give up compared with a managed platform?
Scope
- Spark writing Iceberg tables to MinIO through a REST catalog.
- Schema evolution, partition evolution, time travel and table maintenance in practice.
- Querying the same tables from a second engine to prove the “open” part.
- An honest comparison with the Databricks Lakehouse project.
Milestones
- One-command local stack with Docker Compose
- Ingestion and modelling with Spark and Iceberg
- Multi-engine queries
- Write-up: open versus managed Lakehouse