Projects

What I'm building

Each project is a complete, reproducible build with its own repository, a write-up of the architecture and the trade-offs, and an honest status. Planned projects double as my public roadmap.

Planned

CI Loop: closed-loop analytics

The Composite Intelligence idea as a runnable demo: governed BI metrics, ML anomaly detection and forecasting, human-approved actions, and outcomes fed back as data.

  • Python
  • PySpark
  • Delta Lake
  • MLflow
  • Airflow
  • SQL
Planned

Medallion Lakehouse on Databricks

A reference implementation of an Azure Lakehouse: Data Factory ingestion, Auto Loader into Delta, bronze/silver/gold modelling, Unity Catalog governance and BI-ready gold tables.

  • Databricks
  • Delta Lake
  • PySpark
  • Azure Data Factory
  • Unity Catalog
  • Databricks Asset Bundles
Planned

Production ML pipeline with MLflow and Airflow

Taking a model from notebook to production: feature pipelines in Spark, experiment tracking and a model registry in MLflow, batch scoring orchestrated by Airflow, and drift monitoring.

  • Apache Airflow
  • MLflow
  • PySpark
  • scikit-learn
  • Docker
  • GitHub Actions
Planned

Open Lakehouse with Apache Iceberg

A vendor-neutral Lakehouse you can run on a laptop: Spark and Apache Iceberg on S3-compatible object storage with a REST catalog, compared side by side with the Databricks approach.

  • Apache Iceberg
  • Apache Spark
  • MinIO
  • Docker Compose
  • Trino