Why Composite Intelligence: where BI ends and AI begins

AI + BI = CI. Composite Intelligence is the practice of fusing the rigour and trust of Business Intelligence with the adaptability of Artificial Intelligence and the signal discipline of Machine Learning, so organisations can understand, predict and act in a single loop.

A short history of two traditions

Zoom out far enough and BI and AI look like two old rivers that finally meet.

In 1958, IBM researcher Hans Peter Luhn described “business intelligence” as turning records into useful knowledge. Through the 1990s the BI river widened: data warehouses, star schemas, ETL, OLAP cubes, all tools for reaching consistent answers about yesterday and today. The 2000s democratised access with self-service BI and dashboards; the 2010s moved warehouses to the cloud and strengthened governance. The BI ethos became: stable definitions, auditability, explainability.

AI’s river starts in the late 1950s with the perceptron, forks into expert systems in the 1980s, returns to neural networks with backpropagation, then accelerates with deep learning and the Transformer architecture in 2017. Where BI standardised “the truth”, AI pursued adaptation: predicting what comes next, recommending what to do, and increasingly generating language, images and code.

Machine Learning sits between these worlds in a very practical way. It turns historical data into signals: forecasts, propensity scores, anomaly flags, clusters, rankings and evaluation loops. Composite Intelligence treats those signals as governed analytics assets, not isolated experiments.

For decades the rivers ran in parallel. BI delivered trusted numbers but little agency; AI offered agency but often drifted away from governed numbers. Composite Intelligence is the confluence: products that keep BI’s reliability and add the foresight of AI and ML, so a question can become an answer, a recommendation and a measured outcome in one place.

TL;DR

BI excels at trusted numbers and explainable hindsight: what happened, and why.
Machine Learning turns governed history into reusable signals: forecasts, anomalies, scores and model-driven decisions.
AI excels at adaptation and action: what might happen, what should be done, and how work can be assisted or automated.
CI blends them into a closed loop: consistent metrics, data quality, prediction, generation, workflow automation and feedback.

A quick map of the territory

Layer	Questions answered	Typical outputs	Typical tools
BI	What happened? Why?	Metrics, KPIs, dashboards, drill-downs	SQL, dimensional models, semantic layers, BI tools
Machine Learning	What is likely to happen? Which cases need attention?	Forecasts, anomaly signals, scores, segments	scikit-learn, Spark ML, MLflow, feature tables, model monitoring
AI	What should be done? How can work be accelerated?	Summaries, recommendations, agents, copilots	LLMs, retrieval, vector search, orchestration, evaluation tools
CI	Instrument → Model → Decide → Act → Learn	One experience: trusted metrics, ML signals, AI insight and feedback on actions	Lakehouse platforms, BI tools, ML platforms, workflow systems, human-in-the-loop products

Where BI ends, and why that’s a feature

BI’s strengths

Governed metrics. A gross-margin tile means the same thing on every dashboard, so debates focus on the business rather than on whose number is right.
Explainability. BI answers why with drill-downs and dimensions: a KPI can be traced to the segments that moved it, and back to the tables that fed it.
Reliability. Pipelines are tested, lineage is visible and access is controlled. Everyone knows what runs when, and who can see what.

BI’s limits on its own

Static answers. Dashboards are great at showing deltas and drivers, but they rarely propose the next move.
Human bottlenecks. Business questions are translated into SQL by analysts, and every new slice takes time.
Limited foresight. Forecasts and anomaly detection are often bolted on rather than built into the experience.

Where AI begins, and why it needs BI

AI’s strengths

Foresight. Models project likely futures and spot unusual behaviour as it starts.
Synthesis. Language models turn analysis into concise narratives and suggest the next question, even for people who don’t speak SQL or statistics.
Agency. AI can draft an email, open a ticket, adjust a bid or generate an analysis, turning an insight into a first action.

AI’s risks without BI

Shifting numbers. Without governed inputs and metric definitions, models can be confidently wrong.
Opacity. People hesitate to act on an insight they can’t trace back to data they trust.
Sprawl. Prompts, models and experiments multiply quickly; without controls, trust erodes.

The CI loop, an engineer’s version

Instrument  →  Model  →  Decide  →  Act  →  Learn
    ↑                                         │
    └────── BI · ML · lineage · feedback ─────┘

Instrument. Define data models and metrics as code. Add freshness checks and anomaly monitors so the inputs can be trusted.

Model. Use tabular ML for forecasts and scores, and language models for summaries, reasoning and guided exploration. Keep prompts schema-aware and evaluable.

Decide. Show insights next to the KPI that triggered them. Include uncertainty and alternatives, and allow adjustments before anything is committed.

Act. Connect to operational systems such as ticketing, email, pricing, CRM or campaign platforms. Start with suggestions, and log every approval and rejection.

Learn. Write outcomes back: was the recommendation applied, and what changed afterwards? Feed that into evaluation dashboards and future training data.

The loop is the product. Without the Act and Learn steps, you have decorated dashboards, not Composite Intelligence.

Where data engineering fits

None of this works without the platform underneath. Every step of the loop is a data engineering problem: reliable ingestion and modelling for Instrument, feature pipelines and MLOps for Model, low-latency serving for Decide and Act, and event capture plus lineage for Learn. That’s why this site is mostly about Spark, Lakehouse architecture, orchestration and MLOps: they’re how the equation becomes a working system.