Pipelines

Sparquet runs one JSON at a time. A real load is usually several: truncate, then bronze, then silver, then a commit. The one thing a single JSON cannot express is the order several of them run in — and that is what a Pipeline is.

A Pipeline is an ordered set of Jobs from the same Workflow. It stores no JSON of its own: every stage holds a reference to a Job, and the JSON is compiled from that Job’s canvas at run time. A stage can therefore never drift from the file it names, and a deleted Job leaves a stage that reports itself as broken instead of a stale copy that still runs.

A Job is one pipeline JSON

The framework keeps its own vocabulary: one JSON is a pipeline — the file the Pipeline class executes, reported as a PipelineResult. In Studio that file is a Job, and the name Pipeline is reserved for a sequence of several of them. A Pipeline with four stages is four framework Pipeline runs, in order, sharing one Spark session.

Build one

1. Create it. In a Workflow, press New pipeline and name it after the load it performs — Nightly cessions, not sequence 2. It opens at /pipelines/:id.

2. Add the stages. The Workflow’s Jobs are listed on the left. Click one to append it at the end of the row, or drag it onto the canvas to place it where you dropped it. The same Job may appear more than once — running one file twice against different upstream data is legitimate — so nothing disappears from the list once used; it shows how many times each Job is already staged.

3. Draw the order. Link a box’s right handle to the next box’s left handle. Nothing is inferred from paths here, deliberately: two stages that share no path at all can still have to run in a fixed order — a truncate before a load — so the order is the one you draw.

4. Drill in when you need to. Opening a box takes you to that Job’s own canvas, with every panel it normally has: inspector, JSON, issues, AI. Come back and the stage already reflects the edit, because the stage was never a copy.

Each box carries the stage’s number in the execution order, what the Job reads, what it writes, how many transformations it applies and whether it has a validations block — enough to recognise a stage without opening it.

How the order is decided

The links you draw define it; ties break alphabetically, so the numbering is stable between renders and machines.

• A loop is refused while you draw it: a sequence with a cycle has no first stage, so it could not run at all.
• A stage nothing links to still runs, at the end, and is flagged as a warning — so that its position in the sequence is deliberate rather than an accident of when you dropped it.

How a stage hands data to the next

Stages do not pass a DataFrame between themselves. They share one Spark session, and a stage simply reads what an earlier one wrote:

• A path or a table. Stage 1 writes bronze.orders; stage 2 reads bronze.orders.
• A temp view. Stage 1 writes a view output; stage 2 reads that view as its input. Nothing touches storage — the view lives in the session both stages share.

There is no extra wiring for this, and none is needed: a Job that already reads the right location is already connected. A link on the canvas sets when a stage runs, not what it receives.

Tip

The Pipeline tab of a Workflow infers exactly these connections from the Jobs’ paths and views, read-only. It is the quickest way to check whether the order you drew matches the way data actually flows.

Run it

The same local runner and the same token as a single Job. The Pipeline posts every stage in order to POST /run/flow/stream, which streams Server-Sent Events, so the panel fills in while Spark works instead of sitting frozen until the end.

What you get	Detail
Per-stage status	each box turns running, succeeded, skipped or failed as the sequence advances
Per-stage results	rows read, rows written, duration and validation results, for every stage
Logs labelled by stage	each line carries the stage that emitted it, stdout and JVM lines included
Preview	up to 50 rows of the last stage’s output — the Pipeline’s result

The run stops at the first failing stage: later stages never start, and the error names the stage that broke. The whole Pipeline takes the same single run lock as one Job, so a second run started while one is in progress gets 409.

What blocks a run

All or nothing, on purpose: a sequence is only meaningful whole, so one broken stage stops the run instead of quietly executing the rest in a shortened order.

Blocker	Fix
A stage points at a Job that no longer exists	delete the stage, or recreate the Job it referenced
A stage’s Job does not compile yet	open it and clear the blocking issues on its canvas
A stage sits in a loop	remove one of the links that closes the loop

What a Pipeline is not

It is a development convenience, like the runner itself — no schedule, no retries, no alerting, no backfill window. In production, hand the same JSON files to Airflow, Dagster, Databricks Workflows or cron, in the same order. What a Pipeline buys you is the ability to run and read that order locally, before it becomes somebody’s 3 a.m. page.