Sparquet Studio

Studio is a browser application that edits Sparquet Jobs on a node canvas — a Job being one pipeline JSON. It is not a separate product with its own format: it reads the pipeline JSON and writes the pipeline JSON.

cd sparquet-studio
npm install
npm run dev          # http://localhost:5273

Local first

Workflows, Jobs and Pipelines live in your browser’s IndexedDB. No account, no server, no telemetry. Export the whole workspace as one JSON file.

No lock-in

The canvas compiles to the file the CLI runs. Delete Studio and every Job keeps working.

Documented in place

Every field carries the same notes as this reference, including the behaviors that normally live only in the framework source.

Optional runner

Point it at a small local service and execute a Job — or a whole Pipeline — from the canvas, with counters, validations and a data preview.

The layout

Area	What it does
Palette (left)	Every source, transformation, validator and destination. Drag onto the canvas, or click to append after the last node.
Canvas (center)	The Job as connected nodes. Left to right is execution order.
Inspector (right)	The selected node’s fields, with help, examples and gotchas.
JSON panel	The compiled pipeline JSON, live. Also the import surface.
Issues panel	Lint results as you type, each one linking to the node and field.
Run panel	Execution through the local runner.
Status bar	Node count, validity, save state.

Panels share one tabbed, resizable column on the right — Ctrl/⌘+J, Ctrl/⌘+E, Ctrl/⌘+/ and Ctrl/⌘+Enter switch between them.

Workflows, Jobs and Pipelines

Studio nests three things:

Concept	What it is	Where it lives
Workflow	The container. Usually one per domain: Sales, Billing, CRM.	/workflows/:id
Job	One pipeline JSON, drawn on the node canvas.	/jobs/:id
Pipeline	An ordered set of Jobs that run in sequence.	/pipelines/:id

A Job is one pipeline JSON

The framework keeps its own vocabulary: one JSON is a pipeline — the file the Pipeline class executes, reported as a PipelineResult. In Studio that file is a Job, and the name Pipeline is reserved for a sequence of several of them. A Studio Pipeline with four stages is four framework Pipeline runs, in order, sharing one Spark session.

A Workflow’s screen has two tabs. Jobs lists the Jobs it holds, together with the Pipelines built out of them. Pipeline draws how those Jobs connect — inferred, read-only, from the paths and temp views they share — and is where you start a runnable Pipeline.

All of it lives in your browser. Settings → Data exports everything as a single JSON bundle and imports it back, which is how you move a workspace between machines or share a set of Jobs with a colleague.

Caution

Browser storage is per browser and per profile. Clearing site data deletes your workspace. Commit the compiled JSON to git — it is the artifact that matters, and Studio can reimport any of it.

How the canvas becomes the file

Two rules define the whole mapping:

The shared chain is the main one. Whatever every destination has in common becomes the top-level transformations. After the graph forks, each branch becomes that destination’s own transformations.

A join or union takes its second source from its second input. The chain feeding that handle becomes with plus with_transformations.

Everything else is a node field. Sticky notes never compile; muted nodes are skipped.

Tip

Press Ctrl/⌘+J at any point to see exactly what your drawing produces. If the JSON is not what you expected, the graph is not what you thought — the panel is the fastest way to learn the mapping.

Importing what you already have

Open the JSON panel, switch to Edit, paste a pipeline JSON and click Apply to canvas. Studio lays it out automatically. Unknown transformation types — custom ones you registered — are preserved untouched through import and export.

Verifying a change

Studio ships its own checks:

npm run typecheck   # TypeScript, zero errors expected
npm run test        # unit tests, including compiler round-trips
npm run lint        # eslint
npm run smoke       # end-to-end pass in a real Chrome

The compiler is covered by round-trip tests over the framework’s own example pipelines: import each one, compile it back, and compare. That test is what guarantees the canvas never quietly changes your file.

Next

• The canvas — connections, branching, keyboard control
• The inspector — editing node fields
• AI assistant — generating and modifying Jobs
• Running a job — the local runner
• Pipelines — several Jobs in sequence
• Templates and lessons — the fastest way to learn a feature