Install

Requirements

Python	3.9 or newer
Java	11 or 17 — required by Spark itself (JAVA_HOME must point at it)
PySpark	3.4 or newer, installed as a dependency
Node.js	18.18+ — only for Sparquet Studio

On a managed platform (Databricks, EMR, Dataproc, Synapse) Spark and Java are already there; you only install the framework.

The framework

pip

pip install sparquet

From source

git clone https://github.com/VictorPasqualini/sparquet.git
cd sparquet
pip install -e .

Databricks

Add sparquet to the cluster libraries, or install it in a notebook:

%pip install sparquet
dbutils.library.restartPython()

Verify the install:

from sparquet import Sparquet
print(Sparquet)

Optional extras

Extra	Install	When you need it
Delta Lake (OSS)	pip install "sparquet[delta]"	Delta tables outside Databricks
Iceberg	Spark package on the cluster	Iceberg tables
Kafka	Spark package on the cluster	Publishing to a topic

Connector JARs are declared per pipeline, so a job carries its own dependency:

{
  "spark": {
    "configs": {
      "spark.jars.packages": "io.delta:delta-spark_2.12:3.2.0"
    }
  }
}

Windows

Spark needs the Hadoop native shims to touch the local filesystem on Windows. Download winutils.exe and hadoop.dll, put them in C:\hadoop\bin, and set HADOOP_HOME=C:\hadoop. Without them a local run hangs the first time it writes a file. WSL2 or Docker avoids the problem entirely.

Sparquet Studio

Studio is a static browser app — it needs no server to design pipelines.

1. Clone the repository and install:

   git clone https://github.com/VictorPasqualini/sparquet.git
   cd sparquet/sparquet-studio
   npm install

2. Start it:

   npm run dev

3. Open http://localhost:5273. The first launch seeds a Getting Started Workflow with working Jobs.

To host it, build the static bundle and copy it anywhere:

npm run build      # → dist/
npm run preview    # serve the build locally

The local runner (optional)

Designing needs only the browser. Executing from the canvas needs Spark, so Studio ships a small service you run yourself:

cd sparquet-studio
pip install -r server/requirements.txt
uvicorn server.main:app --port 8787

It prints a token on startup; paste it into Settings → Local runner in Studio. See Running a job for the full procedure.

Next

Write your first pipeline — five minutes, no Studio required.