Troubleshooting

The pipeline does not start

The pipeline JSON needs 'output' (object) or 'outputs' (list). Neither key is present. One of them is required.

KeyError: 'format' or KeyError: 'path' An input, an output or a join/union with is missing a required field. input must be complete even when you inject a DataFrame with input_df — the block still has to parse.

Read format 'x' not supported The format is not in the reader registry. Check spelling; binary is read-only (no writer), while kafka reads and writes. Custom formats must be registered before the run.

Nothing was written

result.skipped is True A stop_if_empty fired: there was no data to process. success stays True — this is a no-op, not a failure.

A transformation was silently skipped Its skip_if_false evaluated to empty or false. Remember that False, [] and a missing parameter all become an empty string. Print the substituted document if in doubt.

Wrong or missing data

Columns disappeared select is destructive: everything not listed is gone. group_by drops every column not in by or produced by agg.

A union produced garbage allow_missing_columns defaults to false, and Spark then matches columns by position. Set it to true to match by name.

A join multiplied rows The right side is not unique on the join key. Add distinct in with_transformations, and a unique validation to catch it permanently.

A cast produced nulls Spark returns null instead of raising when a value cannot be cast. Validate with a not_null rule after casting.

An unexpected ingestion_ts column The framework adds it after the read. Drop it if the destination schema is fixed.

{param} appears literally in the data The key was not in params. Unmatched placeholders stay literal by design.

Merge failures

multiple source rows matched (Delta) The incoming DataFrame has more than one row per merge key. Deduplicate before writing, and add a unique validation on those columns.

Unknown save mode: MERGE The Iceberg writer compares the mode case-sensitively. Write merge in lowercase.

Connectors

ClassNotFoundException / No suitable driver The connector package is not on the classpath. Add spark.jars.packages to the pipeline spark.configs, or install it as a cluster library.

Read format 'x' not supported The format is not registered. Custom connectors must be registered before the run — see Extending.

Studio

The canvas will not compile Open the Issues panel. Blocking errors: no destination, an orphan node, a required field empty, two main inputs into one node, a validations node in a divergent branch, transformations on a union’s second input.

The JSON panel is empty The graph does not compile yet — the panel shows the blocking issues instead.

Local runner not detected The service is not running, or the URL in Settings → Local runner does not match. See Running a job.

A run returns 401 The runner requires its token. Copy the one printed in its terminal into Settings, or start it with SPARQUET_STUDIO_TOKEN to pin a stable value.

A run returns 409 Another run is in progress. The service serializes runs to protect the shared Spark session.

My work disappeared Studio stores your Workflows in the browser’s IndexedDB. Clearing site data deletes them, and a different browser or profile has its own copy. Export from Settings → Data, and commit the compiled pipelines to git.

Environment

A local run hangs on Windows Spark needs the Hadoop native shims. Install winutils.exe and hadoop.dll in C:\hadoop\bin and set HADOOP_HOME. WSL2 or Docker avoids it entirely.

JAVA_HOME is not set Spark needs a JDK — 11 or 17 for Spark 3.5, 17+ for Spark 4.

Settings in the spark block had no effect On Databricks the active session is reused and the block is ignored. The session is also a process-wide singleton: the first pipeline that creates it wins.

Debugging technique

Add a debug node where you lost the plot:

{
  "type": "debug",
  "label": "after join",
  "actions": ["count", "print_schema", "show"],
  "transformations": [{ "type": "filter", "condition": "id = 'SUSPECT'" }],
  "show_rows": 20
}

Its nested transformations apply to a throwaway copy, so you can focus the inspection on one record without changing the pipeline.

Still stuck

Open an issue with the pipeline JSON (secrets removed), the error message, and the environment — github.com/VictorPasqualini/sparquet/issues.