How to use this guide
Each section describes the operating model at that level, the problems teams tend to encounter, and the capabilities needed to move further.
If you’re already familiar with L1 and L2, skip ahead to Level 3. That is where the platform begins to model data assets alongside execution.
Level 1: Ad hoc
A useful baseline for understanding what a scheduler adds.
What it looks like
Scripts run manually or via cron. There is no central scheduler, no alerting, no lineage. Someone wrote a script, put it on a server, and added it to crontab. If it fails, nobody knows until someone notices the data is missing.
The defining characteristic
The team has little visibility beyond files on a machine and whatever logging each script provides.
When you’re here
You’re here if your answer to “where are the pipelines?” is a folder path or someone’s name.
How to move on
Adopt a central scheduler. Almost any scheduler is better than cron. This is a tool change, not a paradigm change.
Level 2: Scheduled
Teams here can have solid infrastructure and reliable delivery. The constraints come from what the platform models.
What it looks like
A central scheduler manages job execution. Failure alerts exist. Dependencies between jobs are managed through careful timing: Job A runs at 6am, Job B runs at 7am because A is usually done by then. Someone on the data team knows the implicit dependency map. It might be documented somewhere. It might not.
The defining characteristic
The tool treats the job as the primary object: it runs, it exits, and success means the execution completed. The scheduler knows what ran. It does not necessarily know what that means for the resulting data.
What it feels like from the inside
Three questions become difficult to answer reliably:
- “Did it actually work?” The job completed, but is the data right? Fresh? Complete? Nobody knows without checking.
- “When will my data be ready?” There’s a scheduled time, not a data-ready guarantee. If something upstream ran long, the answer is: ask the data team.
- “Can I just run it myself?” No. Re-running is risky. Raise a ticket. Wait.
The ceiling
Better alerting and documentation improve an L2 platform, but they do not give the scheduler a model of the data each job produces. The team still has to connect job status to data health itself.
How to move on
Decide whether the platform should continue to manage jobs or should also model the data products those jobs create.
Level 3: Orchestrated
The platform now treats data assets as managed objects, not only as side effects of jobs.
The change in operating model
At L2, your mental model starts with execution. What code runs, when it runs, whether it ran.
At L3, the model starts with data assets: what they are, what produces them, what depends on them, and whether they meet their operating expectations. Jobs still execute the work, but the platform gives the team a data-level view of the result.
What L3 requires
In this model, L3 combines five properties. Treat them as design goals rather than a checklist that a tool satisfies automatically.
Asset-aware
The platform represents important tables, files, models, and datasets as assets. Teams declare who owns them, how they depend on one another, and which operating expectations apply to them.
Example: At L2, an upstream schema change may first appear as a downstream job failure or a broken dashboard. At L3, declared asset dependencies show which downstream assets need review. Column-level impact still requires column-level lineage; the asset graph does not infer it from arbitrary code.
Lineage-native
Dependencies live with the asset definitions instead of only in a separate wiki. Dagster can build a visible graph from those declarations and use it for execution and navigation. The graph is only as complete as the dependencies the team declares, especially when work happens in external systems.
Observable by default
Assets can carry materialisation metadata, freshness expectations, and asset checks. Teams still have to define the checks and expectations that matter; creating an asset does not make its data correct.
Example: A job can finish successfully after writing zero rows. If the team defines a row-count check on the corresponding asset, Dagster records the failed check alongside the materialisation instead of leaving the team to infer the problem from job logs.
Full CI/CD
Pipeline changes go through version control, automated tests, review, and a deployment path. This is an engineering practice around the orchestrator, not a capability the asset model supplies on its own.
Safe re-runs
At L2, re-running a pipeline after a failure is a risk assessment. The pipeline was written to do a thing, not to produce a specific deterministic output. Re-running might create duplicate rows, overwrite something it shouldn’t, or produce inconsistent state.
Asset boundaries and partitions give the team a smaller, explicit unit to re-run. Safety still depends on the implementation: writes must be idempotent or otherwise handle repeated execution. The platform makes recovery easier to reason about, but it cannot make side-effecting code safe by declaration.
The productivity gains
When these practices are in place, the day-to-day work changes for several roles:
- Engineers can reuse orchestration, metadata, and partitioning conventions instead of rebuilding them in each pipeline. Backfills become planned runs over defined partitions when the underlying writes support them.
- Engineering managers can see where teams still spend time on one-off monitoring, manual dependencies, and recovery work.
- On-call engineers can start from the failed asset and its declared downstream dependencies instead of reconstructing the entire path from logs.
- Data analysts can check freshness and lineage themselves, in the platform, without asking the data team.
- Product owners can receive data-health signals before a stale asset affects the product, provided the team has configured the relevant expectations and alerts.
Level 4: Observable
The organisation has explicit service expectations for important data and can see when they are missed.
What changes at L4
L4 is operational maturity of the platform itself. Teams define freshness targets for important assets, route breaches to the right owners, and use lineage during impact analysis and recovery.
Freshness service levels
At L3, freshness expectations can exist. At L4, the organisation decides which assets need formal service levels, monitors them, and reports breaches. The aim is to detect stale data before a downstream consumer reports it.
Example: A financial report due at 7am depends on three upstream assets. If one misses its 6:30am freshness target, the data team receives an alert before the reporting deadline. In a job-only setup, the report owner may be the first person to notice the stale result.
Cross-system lineage
L3 lineage usually starts with dependencies represented inside the orchestration platform. At L4, teams extend that view across the transformation and consumption tools that matter to them.
Column-level lineage can make impact analysis more precise than an asset-level graph. Its coverage depends on the integrations and metadata available across the stack, so teams should state where the lineage ends rather than assume it is complete.
Automated remediation
Teams automate recovery for failure modes they understand and can reverse safely. On-call engineers then spend less time on routine retries and more time on failures that need diagnosis.
Operational value measured
Materialisation history, freshness breaches, quality-check results, usage, and time to recovery give platform teams evidence about where reliability work is paying off. The metrics are useful only if the team defines them consistently and connects them to outcomes the organisation cares about.
Level 5: Platform
The data platform is operated as a shared product with clear ownership and supported ways to consume data.
What changes at L5
L5 is organisational maturity around the platform. Data contracts, supported self-service paths, and explicit ownership allow multiple teams to build on shared data without routing every change through one central group. L4 and L5 extend along different axes, so a platform can progress in one without completing the other.
Data contracts with enforced quality
A data contract records the schema, freshness expectations, quality rules, ownership, and versioning agreed between producers and consumers. At this level, teams encode the parts they can check automatically and define how breaking changes are reviewed or blocked.
Example: A contract can define a field as part of a dataset’s public interface. When a producer proposes a rename, CI can require a contract update and the organisation’s change process can identify affected consumers before deployment.
Supported self-service
At L3, engineers can discover and build on declared assets. At L5, supported interfaces let analysts and other consumers find data, inspect its owner and operating status, and use it without asking the data team to mediate every request.
Traceable critical metrics
Critical metrics have named owners, traceable sources, and documented quality controls. Leadership can see how a number was produced and which checks it passed, while finance and governance teams retain whatever review the decision requires.
Composable, standardised pipelines
Rather than having every team implement common ingestion and transformation patterns independently, the platform provides reusable components with known configuration and operating behaviour. Teams can still depart from those components when the workload requires it.
Self-service depends on the less visible work underneath it: ownership, lineage, quality controls, documentation, and support. A catalogue alone does not create trust.
Where organisations get stuck
A common mistake is to replace the scheduler while preserving the same job-only model. The new tool may improve deployment or operations, but the team will not gain an asset graph unless it identifies assets and declares their dependencies.
The useful design question is no longer only “what code runs when?” It also becomes “what data does this produce, who owns it, and what depends on it?” An asset-aware tool supports that model, but the team still has to supply the definitions and operating practices.
There is also a cost to postponing the decision. Each new pipeline adds another set of implicit outputs and dependencies that the team may have to model later. That cost should be weighed against the disruption and engineering work of changing the platform now.
A diagnostic
Use this to assess your current level honestly.
| Signal | Level |
|---|---|
| Pipelines are scripts on a schedule. Failures are discovered by users. | L1 |
| You have a scheduler and failure alerts. You manage dependencies through timing. You cannot reliably answer “is the data ready?” | L2 |
| Your platform represents data assets and their declared dependencies. Re-runs have defined boundaries. Relevant quality checks are automated. | L3 |
| Important assets have freshness targets. Lineage supports impact analysis. Routine failure modes have tested recovery paths. | L4 |
| Teams use supported self-service paths. Contracts define producer and consumer expectations. Critical metrics have named owners and traceable sources. | L5 |
One practical test is to ask how the team answers “is this data fresh and correct?” If the answer requires an engineer to inspect several job logs and query the output manually, the platform is still operating mainly at L2.
If your team is deciding whether to move from scheduled jobs to an asset-aware platform, get in touch.