It often starts with one daily data load. Someone writes a script, schedules it, and treats a zero exit code as success. For that first use case, the arrangement may be entirely adequate.
Other teams then ask for data. More scripts appear, and timing starts to stand in for dependency management. Job A runs at six. Job B runs at seven because A is usually finished by then.
Then one day, Job A runs long.
Job B fires at seven anyway. The scheduler has no idea that Job B was waiting for Job A. It doesn’t model that relationship. It only knows about individual scripts. Job B runs on incomplete data, or fails, or produces something wrong. The scheduler reports success. Nothing looks broken from the outside.
Downstream data is now late, and someone asks the question the scheduler cannot answer:
“Is the data ready?”
The team can investigate, but the platform has no direct answer. It knows which scripts ran. It does not know whether their outputs are complete, fresh, or suitable for downstream use.
That is the ceiling of a job-only model. The scheduler does not represent the state the team is being asked about.
Three questions that expose the ceiling
When a data platform is built around jobs rather than data, three questions require manual investigation.
“Did it actually work?”
A green tick means the code ran to the end without throwing an error. It says nothing about what the code produced. A job can complete successfully and write zero rows, write duplicate rows, or write data with a subtle corruption: a timestamp comparison that breaks at month boundaries, a filter condition slightly wrong, a join that silently drops records. The scheduler reports success. The data is wrong. Nobody knows until an analyst notices something looks off, or worse, until a stakeholder makes a decision on it.
You’re monitoring the wrong thing. Nobody consumes job status. They consume data.
“When will my data be ready?”
You might have an informal SLA: the data should be ready by 7am. But that expectation lives in a wiki, an email thread, or someone’s head. Nobody is automatically checking whether it was met. If the data isn’t ready by 7am, nobody knows until someone notices.
The answer to “when will my data be ready?” is often: ask the data team. Their morning starts with checking last night’s runs and fielding messages from analysts. The people operating the platform become the missing data-status layer.
“Can I just run it myself?”
Re-running a pipeline in a job-centric world is risky. It might duplicate data, overwrite something, or break a dependency that wasn’t documented. So the data team becomes a gatekeeper, not because they want to be, but because the system requires it.
When people ask “can I just run it myself?” they are exposing an operating constraint. The platform cannot offer self-service recovery until the team has defined what a safe re-run means for that data.
The limit is in the operating model
I have seen this pattern in organisations with capable teams and well-run schedulers. Different groups build pipelines independently, dependencies spread across codebases, and one or two engineers become the people who know how the pieces fit together. New integrations take longer because each team has to reconstruct that context.
The missing capability is visibility into the data itself: what exists, which declared dependencies connect it, who owns it, and which checks or freshness expectations apply. A different orchestrator can support that model, but installing one does not create the model for you.
A model for the current state
I use a five-level maturity model to separate improvements to scheduling from changes in how the platform represents data.
Many organisations I work with are at Level 2. They have a scheduler and failure alerts, but the system still describes pipelines primarily as jobs. Level 3 adds an explicit asset model.
Scheduler and orchestrator
The terms overlap in everyday use, so the distinction here is about responsibility rather than product category.
A scheduler answers when work should run and reports its execution state. In a job-only setup, the scheduler may not represent the tables, files, or models produced by that work.
An asset-aware orchestrator can also represent those outputs and their declared dependencies. It can use asset state when deciding what to run and show operators which downstream assets may be affected by a failure.
From jobs to assets
Moving from L2 to L3 changes the unit the platform manages. The team keeps running code, but it defines that code in relation to the data it produces.
At L2, the model starts with code: what runs, when it runs, and whether it completed. At L3, the model also names the resulting data assets, their declared dependencies, their owners, and their operating expectations.
The last row in that comparison deserves qualification: re-runs have explicit boundaries.
At L2, if a pipeline half-fails and you re-run it, you’re taking a gamble. The pipeline was written to do a thing, not to produce a specific, deterministic output. You might get duplicate rows, inconsistent state, overwritten data. You don’t know until you check. Every failure recovery starts with a risk assessment.
At L3, an asset and its partitions define a clearer unit of recovery. Re-materialising that unit is safe only when the underlying writes handle repeated execution correctly. The orchestrator can scope and track the run; it cannot make an arbitrary side effect idempotent.
What L3 requires
L3 requires more than enabling an asset view. The team has to make several design and operating choices.
Asset-aware. The platform names important data assets and records who owns them, what produces them, and which expectations apply.
Lineage-native. Asset dependencies are declared with the code and displayed in the graph. That reduces reliance on a separate wiki, but the graph can still be incomplete when dependencies are omitted or work happens outside the orchestrator.
Observable. Materialisations can carry metadata, freshness expectations, and asset checks. The team decides which signals matter and configures them.
Built for sharing. A shared asset graph gives teams a common place to discover data and its declared dependencies. Access controls, documentation, and stable interfaces still determine whether another team can use it safely.
Operated for self-service. Teams define supported paths for discovery, development, and recovery so routine work does not require a ticket.
The hidden cost of staying at L2
The costs of remaining at L2 are real. They fall on capable teams doing exactly what the architecture permits. They just don’t show up on a spreadsheet.
Stakeholder trust erodes. Not in a single incident, but over months of “that number looked off” and “I validated it before the meeting just in case” and “I keep my own copy because I don’t fully trust the feed.” By the time it surfaces as a problem, it’s already deep.
Decision latency accumulates. Every time someone asks “can we trust this number?” an engineer may have to stop, trace through logs, and validate outputs. The delay is small in isolation and expensive when it becomes routine.
On-call work becomes investigative. When something breaks, the engineer has to reconstruct downstream impact from job dependencies, code, and local knowledge before deciding what to re-run.
Migration scope grows. Each new pipeline adds outputs and dependencies that may need to be modeled later. That growing scope has to be weighed against the cost of changing the platform now.
What changes at L3
The same three questions. Different answers.
The third row is deliberately conditional. Self-service is possible only after the team has defined safe recovery boundaries and access controls.
Who feels the difference
The change affects each role differently.
Engineering managers should measure this rather than assume a standard percentage. Useful signals include time spent on manual backfills, one-off monitoring, dependency questions, and recovery work. Those numbers establish whether an asset migration is solving a real capacity problem.
Beyond L3
L4 and L5 extend the model along different axes. A platform can improve operational control without yet offering broad self-service, or build strong product practices while parts of its operational model remain manual.
L4 (Observable): operational maturity of the platform itself. Important assets have service expectations, named owners, and alerts that route breaches to the people responsible. Teams extend lineage across the systems needed for impact analysis and automate recovery only for failure modes they have tested.
L5 (Platform): organisational maturity around the platform. The data platform is a shared product that multiple teams build on.
At L5, producers and consumers use data contracts to record schema, freshness, quality, ownership, and change rules. Supported interfaces let analysts and other consumers discover assets and inspect their operating status without routing every request through the data team. Critical metrics remain subject to the review appropriate to the decision, but their sources and checks are easier to trace.
An asset model gives these practices a common object to attach to, but it is not the only way to implement them. The point of L3 in this model is to make ownership, lineage, quality, and recovery part of the platform’s normal operating vocabulary.
Deciding when to make the change
Not every team needs to move immediately. A stable scheduler may be the right system when job status answers the operational questions the organisation actually asks.
The case for change becomes stronger when engineers repeatedly reconstruct lineage, validate data after green runs, or gate every backfill. At that point, compare the ongoing operating cost with the work of identifying assets, declaring dependencies, and changing deployment and recovery practices.
If these questions are consuming engineering time in your organisation, get in touch.