The Ceiling of Job-Centric Thinking: Why Your Data Platform Needs a Paradigm Shift
Most data teams don't have a tooling problem. They have a mindset problem. Here's what that means — and what's on the other side of it.
Most data teams don’t have a tooling problem. They have a mindset problem. Here’s what that means — and what’s on the other side of it.
It always starts small.
One team needs a daily data load. A job gets created. And when I say job, I mean it literally — a script. It runs, it exits. Success means it didn’t crash. That’s the entire information model. That’s all the system knows.
It works. Simple, effective, done.
Then other teams need data. More jobs. More scripts. You start managing timing manually — because the tool has no concept of dependencies. Job A runs at six. Job B runs at seven, because A is usually done by then. Fragile, but functional.
Then one day, Job A runs long.
Job B fires at seven anyway. The scheduler has no idea that Job B was waiting for Job A — it doesn’t model that relationship. It only knows about individual scripts. Job B runs on incomplete data, or fails, or produces something wrong. The scheduler reports success. Nothing looks broken from the outside.
Meanwhile, everything downstream is late. And then the question arrives — from three different people, simultaneously:
“Is the data ready?”
No one can answer.
Not because the team isn’t capable. Because there’s no system that can answer it. The scheduler knows scripts ran. It has no idea whether the data is ready. That’s not a question it was ever designed to answer.
This is the moment every data organisation hits. Not a failure — a ceiling.
The Three Questions That Expose the Ceiling
When a data platform is built around jobs rather than data, three questions become impossible to answer reliably. You’ll recognise them.
“Did it actually work?”
A green tick means the code ran to the end without throwing an error. It says nothing about what the code produced. A job can complete successfully and write zero rows, write duplicate rows, or write data with a subtle corruption — a timestamp comparison that breaks at month boundaries, a filter condition slightly wrong, a join that silently drops records. The scheduler reports success. The data is wrong. Nobody knows until an analyst notices something looks off, or worse, until a stakeholder makes a decision on it.
You’re monitoring the wrong thing. Nobody consumes job status. They consume data.
“When will my data be ready?”
You might have an informal SLA — the data should be ready by 7am. But that expectation lives in a wiki, an email thread, or someone’s head. Nobody is automatically checking whether it was met. If the data isn’t ready by 7am, nobody knows until someone notices.
The answer to “when will my data be ready?” is effectively: ask the data team. That’s a person becoming the monitoring system. Their morning starts with checking whether last night’s runs are healthy and fielding messages from analysts. That’s not engineering. That’s babysitting a system that should be able to speak for itself.
“Can I just run it myself?”
Re-running a pipeline in a job-centric world is risky. It might duplicate data, overwrite something, or break a dependency that wasn’t documented. So the data team becomes a gatekeeper — not because they want to be, but because the system requires it.
When people ask “can I just run it myself?” they’re not being impatient. They’re telling you the data platform isn’t serving them. They’ve lost confidence that the system will deliver without intervention. That’s a trust problem. And it compounds quietly over time.
This Isn’t a Tool Problem. It’s an Organisational Maturity Problem.
The pattern I’ve just described isn’t unique to any organisation. Every organisation that scales on data hits exactly this wall. The symptoms are the same. The sequence is the same.
Teams solving orchestration independently. Data logic fragmented across codebases. No one knowing the full dependency picture. Building new products getting slower because you’re wiring integrations, not building features. Quality issues hiding until they hit production.
The through-line is a single word: visibility.
At this level, you have visibility into individual jobs. You have no visibility into data — what exists, what depends on what, whether it’s healthy, who owns it. Everything above is a consequence of that missing layer.
And here’s what I want to be precise about: the right tool makes the crossing possible, but the tool only works if the thinking changes with it. What’s needed is a platform that’s inherently built to solve these problems from the ground up.
A Model for Where You Are — and Where You’re Going
I use a five-level maturity model to think about this. Here’s where most data organisations sit, and what the path forward looks like.
Most organisations I work with are at Level 2. They have a scheduler, they have failure alerts, and they think in jobs. The step from L2 to L3 is the most consequential crossing in this model — and it’s the one I want to spend the rest of this article on.
The Difference Between a Scheduler and an Orchestrator
These two terms get used interchangeably. They shouldn’t.
A scheduler answers one question: when? Run this job at 6am. Run this job every hour. It fires scripts according to a timetable and reports whether they ran. It has no concept of what those scripts produce, what depends on what, or whether anything downstream is healthy.
An orchestrator answers a different question: what, in what order, and why? It manages the flow of work based on dependencies and state — not just time. It knows that Asset B depends on Asset A, so it waits for A to be healthy before running B. It knows that if A fails, B shouldn’t run. It knows what the data is, not just when the code runs.
The difference isn’t cosmetic. It’s the difference between a timetable and an air traffic controller.
The Paradigm Shift: From Job-Centric to Asset-Centric
Crossing from L2 to L3 requires a shift in how you think about what you’re orchestrating. Not a better scheduler — a different mental model entirely.
A paradigm is the mental model you use to think about a problem — the assumptions so fundamental you don’t even notice you’re making them. A paradigm shift is when those assumptions change completely. Not an improvement on the old way of thinking. A replacement of it.
At L2, your mental model starts with code. What scripts run, when they run, whether they ran. At L3, your mental model starts with data. Data is a first class citizen. The platform is built around it — not the other way around. Same organisation, same engineers, completely different starting point.
The last row in that comparison deserves particular attention: re-runs are safe by design.
At L2, if a pipeline half-fails and you re-run it, you’re taking a gamble. The pipeline was written to do a thing — not to produce a specific, deterministic output. You might get duplicate rows, inconsistent state, overwritten data. You don’t know until you check. Every failure recovery starts with a risk assessment.
At L3, an asset defines exactly what it produces. Re-materialising it is safe — not by convention, not by discipline, not because the engineer who wrote it was careful. By design. The model makes it impossible for the orchestration layer to produce an inconsistent result.
This isn’t harder. It’s easier — because for the first time you have a platform that’s actually designed for what you’re trying to do.
What L3 Actually Requires
Crossing to L3 isn’t a configuration change. It requires an architectural foundation with five properties — and you either have them or you don’t. They’re not features you bolt on later.
Asset-aware. The platform has a model of your data. What assets exist, who produces them, what depends on them, whether they’re fresh. Not just what code runs.
Lineage-native. Dependencies between assets are derived automatically from how the code is written — not documented separately after the fact. Documented lineage rots. Native lineage is always correct by construction. Rename a column in an upstream table at L2, and you find out something is broken when an analyst’s dashboard goes wrong. At L3, the system knows exactly what depends on that column — the impact is visible before you deploy.
Observable by default. Freshness and quality are properties of every asset automatically. You don’t opt in. You don’t add a monitoring layer on top. It’s there from the moment an asset exists.
Built for sharing. A shared asset graph means any team can see what data exists and build on it without a custom integration request. This is what turns the data platform from a service into infrastructure.
Self-service. Teams build without raising tickets. The data team stops being a bottleneck and starts being a platform team — which is where they should be spending their time.
The Hidden Cost of Staying at L2
The costs of remaining at L2 are real. They just don’t show up on a spreadsheet.
Stakeholder trust erodes — not in a single incident, but over months of “that number looked off” and “I validated it before the meeting just in case” and “I keep my own copy because I don’t fully trust the feed.” By the time it surfaces as a problem, it’s already deep.
Decision latency accumulates — every time someone asks “can we trust this number?” that’s an engineer dropping what they’re doing, tracing through logs, validating outputs, and coming back thirty minutes later. Multiply that across the organisation. That’s real capacity disappearing with no visible trace.
Engineer burnout follows — “blast radius is unknowable” is the specific mechanism. When something breaks on-call at L2, you don’t know what it affects until you’ve manually traced every downstream dependency. Every incident starts from zero.
And the migration cost compounds. Every new pipeline built in the old model is another thing to migrate later. Every new team that joins and learns the old way is another change management problem down the line. The cost of crossing today is the lowest it will ever be.
What Changes at L3
The same three questions. Different answers.
The third row is worth pausing on. The entire resolution to a question that at L2 requires a ticket, a queue, a person, and a wait — is the word yes. That’s the crossing in a single character.
Who Feels the Difference
This isn’t abstract. Every role in a data-dependent organisation feels L2 pain differently — and L3 relief differently.
The engineering manager row is worth dwelling on. 20–30% of engineering capacity on plumbing — retry logic, alerting, lineage documentation, manual backfills — is the consistent pattern at L2. That’s not an estimate. It’s what you find when you actually measure it. At L3, that time goes back to building.
The Horizon: What L4 and L5 Deliver
L3 is not the destination. It’s the foundation.
L4 — Observable: the organisation knows the state of its data.
Freshness SLAs on every asset — not informal agreements, but formal expectations enforced automatically. Impact analysis from lineage — before you deploy a change, you can see exactly what it affects downstream. Automated remediation — stale or failed assets re-materialise without human intervention. Failures isolated to specific assets — incident response becomes precise instead of investigative.
L5 — Platform: the data platform is a product.
Data contracts with enforced quality — teams publish and consume assets with formal guarantees on schema, freshness, and quality. All machine-checked. Breaking changes can’t reach consumers silently. Board-level KPIs — the data is so reliable, so well-governed, so observable, that leadership makes strategic decisions directly from it without a manual validation step. Universal self-service — at L3, engineers are self-service. At L5, everyone is. An analyst, a product owner, a finance manager can find an asset, see its lineage, understand its freshness, and consume it with confidence — without asking anyone, without knowing how pipelines work.
None of this is available without L3 underneath it. You cannot have observable data without an asset-aware platform. You cannot have data contracts without lineage. You cannot measure ROI without observability. L3 is what makes all of it reachable.
The Crossing Is Available Now
The platform exists. The architecture is proven. Organisations are crossing from L2 to L3 right now and experiencing exactly the improvements described above — not as theoretical outcomes, but as concrete daily realities.
The question isn’t whether the crossing is worth making. Every organisation that has made it will tell you it is. The question is how long you wait.
And waiting has a compounding cost. Every new pipeline built in the old model is another thing to migrate later. Every new team that joins and learns job-centric thinking is another change management challenge. The cost of crossing today is the lowest it will ever be.
I help data teams cross the L2 to L3 inflection point — from job-centric scheduling to asset-aware orchestration. If your team is asking the three questions above and not getting good answers, let’s talk.