Core concepts

Everything in Sparquet follows from one execution order. Learn it once and the rest of the reference reads itself.

The pipeline document

A pipeline has five parts. Only three are required.

{
  "name": "orders_curated",          // required
  "description": "…",                // optional
  "spark": { "configs": {} },        // optional

  "input":  { … },                   // required
  "transformations": [ … ],          // optional, ordered
  "validations": { … },              // optional
  "output":  { … }                   // required — or "outputs": [ … ]
}

Note

output and outputs are both accepted. When both are present, outputs wins and output is ignored entirely.

Execution order

This sequence is the mental model for the whole framework:

1. {param} substitution — placeholders are replaced in the raw text, before the JSON is parsed.
2. $include expansion — referenced fragments are inlined into transformations.
3. Parse — the document becomes typed configuration; unknown keys are carried along, not rejected.
4. Read the input — one source, through the reader registry. An ingestion_ts column is added automatically.
5. Transformations — applied in array order, with {{runtime}} variables resolved as each one runs.
6. Validations — measured on the transformed DataFrame; the optional report is written here.
7. For each destination — its own transformations, then its column projection, then the write.

Two consequences people trip on:

• Transformations run before validations. A rule always sees the cleaned data, never the raw source.
• Per-destination transformations run after validations. Reshaping for one output never affects what the rules measured, or what another output receives.

Transformations change data. Validations report on it.

You want to…	Use
Remove rows with a null id	filter in transformations
Know how many null ids arrived	not_null in validations
Drop duplicates	drop_duplicates
Fail the run when duplicates exist	unique with on_failure: "fail"

A validation never modifies the DataFrame. That separation is what makes a quality report trustworthy: it describes the data you actually wrote.

Two kinds of placeholder

Sparquet has two substitution mechanisms, and they resolve at different times.

	{param}	{{variable}}
Resolved	before parsing	during execution
Comes from	the params argument of the run	a collect transformation
Typical use	environment, date, feature flags	a key list pushed into a later read
If unresolved	stays literal, no error	stays literal, resolved later if it appears

// {param}: known when you launch the run
{ "type": "filter", "condition": "region = '{region}'" }

// {{variable}}: computed by the pipeline itself
{ "type": "collect", "column": "customer_id", "as": "active" },
{ "type": "join",
  "with": { "format": "delta", "path": "sales.events" },
  "with_transformations": [
    { "type": "filter", "condition": "customer_id IN ({{active}})" }
  ],
  "on": "customer_id" }

See Parameters and variables for the formatting rules of each type.

How a canvas becomes a file

Studio does not store a proprietary format — it compiles the graph. Two rules define the mapping:

1. The shared chain is the main one. The transformations every destination has in common become the top-level transformations. Once the graph forks, each branch becomes that destination’s own transformations.

source → filter → cast ─┬─→ [group_by] → delta        ← group_by belongs to this output
                        └─→ parquet                    ← filter and cast are shared

2. A join or union takes its second source from its second input. The chain feeding that handle becomes with_transformations.

                 ┌── delta(events) → filter → select ──┐   ← with + with_transformations
source → filter ─┴────────────────────────────────────→ join → sink

Everything else is a field on a node. Notes never compile; disabled nodes are skipped.

Where a pipeline runs

The framework detects its environment and adapts the session:

Environment	Behavior
Databricks	Reuses the active session; the spark block is ignored
EMR / Dataproc / Synapse	Builds a session with your configs
Local	Applies master too (local[*] by default)

The session is a process-wide singleton: the first Sparquet instance wins, and later pipelines share it. That is what makes running several pipelines in one job cheap.

Failure model

PipelineResult never raises. A failed run comes back as data:

result = fw.run("pipeline.json")

result.success            # False when something went wrong
result.error              # the message, when it did
result.skipped            # True when stop_if_empty ended the run early
result.rows_read          # rows read from the input
result.rows_written       # rows in the main DataFrame at write time
result.validation_results # one entry per rule

That shape is what lets an orchestrator branch on the outcome without wrapping everything in try.

Next

• The pipeline JSON — every field, with its defaults.
• Transformations — all twenty.
• Guides — end-to-end recipes.