A command-line workflow for compiling analytical models, resolving dependencies and retrying failed work.
This data transformation case study gave the client a command-line workflow. It turns SQL models into warehouse tables and views, in the order that model references describe. Jinja supplies the reusable logic, and YAML holds the configuration.
The engine compiles and runs your chosen models. It then saves the results, so the next run can build on them. Recovery mattered from the very start.
The challenge
A failed run does not leave every model in the same place. Some of them have already finished. Others never started, because a result they depended on was missing. Running the whole project again is often wasteful, yet picking only the failed models is not enough on its own. A retry also needs the settings from the original run, including whether that run asked for a full refresh.
What we built
We tied project loading, dependency selection and execution to a recorded run state. The tool reads the models, the profiles and the shared packages. It resolves their references and builds a plan that places the tests next to the models they check.
A user can select work by comparing the current project state with an earlier one. Deferral lets a run reuse relations that an earlier run already created. Retry is the central part of the workflow. It picks the failed or skipped work out of the saved results, and it leaves the successful nodes alone rather than running them again.
Technical approach
Typed schemas validate the configuration first. Only then does the scheduler walk the dependency graph. Materialization macros carry out the database work, whether that means creating or updating a model's table, its view or another supported form.
Retry reads the run-results artifact first. It restores the original command and its refresh settings. JSON and Parquet artifacts keep that execution record after the terminal session has closed. The next attempt can follow the earlier choices, so nobody has to reconstruct them by hand.
Warehouse connections run on Arrow ADBC drivers. A driver is chosen for its version and target platform. It is downloaded when needed and checked against a SHA-256 digest, which keeps driver distribution separate from the tool. The first download does still need network access.
An optional run cache can help on the paths that support it. Without it, those paths run normally instead.
Project success
Models and tests run in dependency order, and each attempt leaves inspectable results behind. Failed work has a defined way back in. A user selects the relevant nodes, keeps the recorded settings and produces a fresh run record. The client can manage that cycle without treating every interruption as a reason to restart the whole project.
Technology stack
Execution engine
Rust, Tokio
Command-line interface
Clap
Model definitions
SQL, Jinja, YAML
Warehouse connectivity
Arrow ADBC
Run artifacts
JSON, Parquet
Documentation interface
React, Vite, DuckDB-WASM
Other engagements
Recognize the pattern?
If one of these looks like your situation, the discovery call is where we find out how close.
