The Data Orchestration Maturity Model: A Field Guide
A reference guide for data leaders and engineers navigating the journey from ad hoc scripts to a mature data platform — with emphasis on the inflection point that changes everything.
How to Use This Guide
This article is structured as a reference. Each level of the maturity model is covered in its own section — what defines it, what it feels like from the inside, and what it takes to move beyond it.
If you’re already familiar with L1 and L2, skip ahead to Level 3 — that’s where the paradigm shift happens and where the most significant productivity gains become available.
Level 1: Ad Hoc
For reference. Most organisations have moved past this — or should have.
What it looks like
Scripts run manually or via cron. There is no central scheduler, no alerting, no lineage. Someone wrote a script, put it on a server, and added it to crontab. If it fails, nobody knows until someone notices the data is missing.
The defining characteristic
Complete invisibility. There is no system — just files on a machine.
When you’re here
You’re here if your answer to “where are the pipelines?” is a folder path or someone’s name.
How to move on
Adopt a central scheduler. Almost any scheduler is better than cron. This is a tool change, not a paradigm change.
Level 2: Scheduled
For reference. This is where most data organisations currently sit.
What it looks like
A central scheduler manages job execution. Failure alerts exist. Dependencies between jobs are managed through careful timing — Job A runs at 6am, Job B runs at 7am because A is usually done by then. Someone on the data team knows the implicit dependency map. It might be documented somewhere. It might not.
The defining characteristic
The tool thinks in jobs. A job is a script — it runs, it exits, success means it didn’t crash. That is the entire information model. The scheduler knows what ran. It does not know what that means for your data.
What it feels like from the inside
Three questions become impossible to answer reliably:
- “Did it actually work?” — the job completed, but is the data right? Fresh? Complete? Nobody knows without checking.
- “When will my data be ready?” — there’s a scheduled time, not a data-ready guarantee. If something upstream ran long, the answer is: ask the data team.
- “Can I just run it myself?” — no. Re-running is risky. Raise a ticket. Wait.
The ceiling
You can optimise L2 indefinitely — better alerting, better documentation, more reliable timing — and you will not escape these three questions. They are structural. They are a consequence of modelling pipelines as jobs rather than data.
How to move on
This is not a tool upgrade. It is a paradigm shift. Read on.
Level 3: Orchestrated
This is the inflection point. Everything before this is incremental improvement. Everything after this is compounding return.
The Paradigm Shift
At L2, your mental model starts with execution. What code runs, when it runs, whether it ran.
At L3, your mental model starts with data. Data is a first class citizen. The platform is built around data assets — what they are, what produces them, what depends on them, whether they are fresh. Execution is a means to an end, not the thing you manage.
This is not a subtle difference. It changes everything downstream of it.
What L3 Requires
Five architectural properties — not features, not configuration options. Prerequisites. You either have them or you don’t.
Asset-aware
The platform has a model of your data. Every asset — every table, file, model, or dataset your pipelines produce — is a first class entity in the system. The platform knows what assets exist, who produces them, what depends on them, and whether they are fresh. Not just what code runs.
Example: At L2, renaming a column in an upstream table means you find out something is broken when an analyst’s dashboard goes wrong. At L3, the system knows exactly what depends on that column. The impact is visible before you deploy the change — not after it’s already consumed bad data.
Lineage-native
Dependencies between assets are derived automatically from how the code is written — not documented separately after the fact. This distinction matters enormously. Documented lineage rots. It is always slightly wrong, always slightly out of date. Native lineage is always correct by construction because the graph is the code.
Observable by default
Freshness and quality are properties of every asset automatically. You do not opt in. You do not add a monitoring layer on top. The moment an asset exists, it has observable properties — when it was last materialised, whether it meets its freshness expectation, whether it passed its quality checks.
Example: At L2, a job that writes zero rows because of a subtle filter condition is reported as a success. At L3, an asset check catches this — a row count expectation is defined on the asset and evaluated as part of materialisation. A zero-row result fails the check. The asset is marked unhealthy. Nobody has to notice.
Full CI/CD
Pipeline changes go through version control, automated testing, and a deployment pipeline before reaching production. Nobody edits pipelines directly in production. Changes are reviewable, testable, and reversible. This is not a process nicety — it is what makes the rest of the model reliable.
Safe re-runs
At L2, re-running a pipeline after a failure is a risk assessment. The pipeline was written to do a thing, not to produce a specific deterministic output. Re-running might create duplicate rows, overwrite something it shouldn’t, or produce inconsistent state.
At L3, an asset defines exactly what it produces. Re-materialising it is always safe — the orchestration layer handles it correctly by design. This changes the economics of failure recovery entirely. Incidents stop being stressful manual investigations and start being a managed re-materialisation.
The Productivity Gains
The shift to L3 is where compounding returns begin. Here is what changes for each role:
- Engineers stop rebuilding retry logic, alerting, and lineage in every pipeline. They write business logic. Backfills — which at L2 are emergency scripts written under pressure — become a parameterised re-run of a defined partition.
- Engineering managers recover 20–30% of engineering capacity currently spent on plumbing — custom integrations, one-off monitoring, manual dependency management.
- On-call engineers stop starting every incident from zero. When an asset fails, the graph shows what it affects immediately. Blast radius is known before the investigation begins.
- Data analysts stop maintaining private validation rituals. Freshness and lineage are visible in the platform. They can inspect it themselves instead of asking the data team.
- Product owners stop discovering data failures when their product breaks. Staleness surfaces at the platform layer — before it reaches users.
Level 4: Observable
The organisation knows the state of its data — at all times, without asking.
What Changes at L4
L3 gives you a platform that is observable by default at the asset level. L4 extends that observability across the entire organisation and adds the intelligence to act on it automatically.
Freshness SLAs on every asset
At L3, freshness expectations exist. At L4, they are formal SLAs — defined, monitored, and reported on. Every asset has a committed freshness window. A breach surfaces immediately and automatically, before anyone downstream has consumed stale data.
Example: A financial report that depends on three upstream assets has an SLA of 7am. At L4, if any of those assets breaches its freshness expectation at 6:30am, the alert fires. The data team has thirty minutes to remediate before the SLA breach affects the report. At L2, nobody knows until the CFO opens the dashboard.
End-to-End Lineage
L3 lineage covers what is inside your orchestration platform. L4 lineage is complete — it spans the entire journey of data from source system to final consumer, across every tool and transformation layer.
The distinction matters for two reasons. First, column-level lineage becomes possible — not just “Asset B depends on Asset A” but “this specific column in Asset B is derived from this specific column in Asset A, transformed in this specific way.” Second, impact analysis becomes precise — before deploying any change anywhere in the stack, you can see exactly what it affects downstream.
Automated Remediation
When an asset becomes stale or fails, the platform can re-materialise it automatically — without human intervention. On-call stops being about reacting to failures and starts being about the genuinely novel problems the system cannot handle alone.
ROI Tracked
For the first time, the data platform generates the operational metadata to make its own value measurable. Usage per asset, freshness breach rates, quality check pass rates, time-to-remediation — all observable, all attributable. The data team stops arguing from intuition about platform value and starts arguing from data. That is a different conversation to have with a CFO.
Level 5: Platform
The data platform is a product. Data is democratised across the organisation.
What Changes at L5
L4 makes the data platform fully observable. L5 makes it a product — something other teams build on, with the same reliability expectations they would place on any production system.
Data Contracts with Enforced Quality
A data contract is a formal agreement between the team that produces a dataset and the teams that consume it. It defines schema, freshness, quality guarantees, ownership, and versioning. Critically, the contract is machine-readable and enforced automatically. A producer cannot push a breaking schema change without consumers being notified — or without the system blocking the deployment.
Example: At L2, a producer renames a field and discovers downstream breakage when three dashboards go wrong simultaneously. At L5, the contract defines that field as part of the public interface. A rename triggers a versioning event, consumers are notified, and the change cannot be deployed until the contract is updated and consumers have acknowledged the change.
Universal Self-Service
At L3, engineers are self-service — they can build on each other’s data without raising tickets. At L5, everyone is self-service. An analyst, a product owner, a finance manager can find an asset in the platform, see its lineage, understand its freshness guarantee, review its quality checks, and consume it with full confidence — without involving the data team, without knowing how pipelines work.
This is only possible at L5 because it requires everything underneath it. Without contracts, quality is not guaranteed. Without lineage, provenance is not visible. Without observability, freshness is not trustworthy. Universal self-service is not a feature — it is the consequence of a fully mature platform.
Board-Level KPIs
At L2, nobody puts a pipeline-generated number in front of a board without validating it first. The informal tax on every important number is a manual verification step. At L5, the data is so reliable, so well-governed, so traceable, that leadership makes strategic decisions directly from it. Revenue figures, customer metrics, operational performance — consumed at the top of the organisation without a manual validation step in between.
Composable, Standardised Pipelines
Rather than every team writing their own asset definitions for common patterns, the platform provides reusable building blocks. Ingesting from a new source, loading into the warehouse, running a transformation — these are assembled from proven components, not implemented from scratch. New teams do not start with a blank canvas. The platform accumulates value over time rather than accumulating debt.
Where Most Organisations Go Wrong
The most common mistake is treating the L2 to L3 crossing as a tool upgrade. It is not. You can adopt a new orchestrator and remain firmly at L2 if the thinking does not change with it.
The right tool makes the crossing possible. But the tool only works if the organisation stops thinking about execution and starts thinking about data assets. That shift — from “what code runs when?” to “what data depends on what?” — is the thing that unlocks everything above L2. Without it, you have a more expensive scheduler.
The second most common mistake is underestimating the compounding cost of staying at L2. Every new pipeline built in the job-centric model is another thing to migrate later. Every new team that joins and learns to think in jobs is another change management challenge. The cost of crossing to L3 is the lowest it will ever be.
A Diagnostic: Where Are You?
Use this to assess your current level honestly.
| Signal | Level |
|---|---|
| Pipelines are scripts on a schedule. Failures are discovered by users. | L1 |
| You have a scheduler and failure alerts. You manage dependencies through timing. You cannot reliably answer “is the data ready?” | L2 |
| Your platform knows what data assets exist and what depends on what. Re-runs are safe. Quality is checked automatically. | L3 |
| You have formal freshness SLAs. You can trace any number to its source. Failures remediate automatically. | L4 |
| Teams consume data assets without involving the data team. Contracts enforce quality. Leadership trusts the numbers. | L5 |
Most organisations who believe they are at L3 are at L2. The test is simple: can your platform answer “is this data fresh and correct?” without a human checking? If not, you are at L2.
I help data teams cross the L2 to L3 inflection point — from job-centric scheduling to asset-aware orchestration. If your team is stuck at L2 and ready to make the crossing, let’s talk.