Extending

Everything built in is registered through the same interfaces you can use. When the language does not cover a case, extend it rather than working around it.

A transformation

import pyspark.sql.functions as F
from sparquet.transform.base import BaseTransformation

class NormalizeText(BaseTransformation):
    """Trims and lowercases a column."""

    def apply(self, df):
        column = self.config.params["column"]
        return df.withColumn(column, F.trim(F.lower(F.col(column))))

fw.register_transformation("normalize_text", NormalizeText)

{ "type": "normalize_text", "column": "email" }

self.config.params holds every JSON key except type and skip_if_false — which means skip_if_false works on your transformation for free.

A validator

import pyspark.sql.functions as F
from sparquet.validation.base import BaseValidator, ValidationResult

class NoFutureDateValidator(BaseValidator):
    def validate(self, df):
        column = self.rule.params["column"]
        failed = df.filter(F.col(column) > F.current_date()).count()
        if failed:
            return ValidationResult("no_future_date", False, f"{failed} future dates", failed)
        return ValidationResult("no_future_date", True)

fw.register_validator("no_future_date", NoFutureDateValidator)

{ "type": "no_future_date", "column": "ordered_at" }

Returning a ValidationResult — rather than raising — is what lets on_failure decide the policy and the report record the outcome.

A connector

from sparquet.io.base import BaseReader, BaseWriter

class ElasticReader(BaseReader):
    def read(self):
        options = {**self.config.options, "es.resource": self.config.path}
        return self.spark.read.format("org.elasticsearch.spark.sql").options(**options).load()

class ElasticWriter(BaseWriter):
    def write(self, df):
        writer = df.write.format("org.elasticsearch.spark.sql").mode(self.config.mode)
        writer.options(**self.config.options).save(self.config.path)

fw.register_reader("elasticsearch", ElasticReader)
fw.register_writer("elasticsearch", ElasticWriter)

{ "format": "elasticsearch", "path": "orders/_doc", "options": { "es.nodes": "es.internal" } }

Note

Reader and writer registries are class-level, shared by every Sparquet in the process. Transformation and validator registries are per-instance, so two frameworks can carry different custom transformations.

Keeping Studio in step

Studio builds its palette, its forms, its linter and the assistant’s prompt from a single catalog in sparquet-studio/src/catalog/. A custom type still works without touching it — unknown nodes are imported, preserved and exported untouched — but it gets no dedicated form and the assistant will not suggest it.

To make it first class, add an entry:

src/catalog/transformations.core.ts

{
  type: 'normalize_text',
  label: 'Normalize text',
  family: 'compute',
  accent: 'transform',
  icon: 'Type',
  summary: 'Trims and lowercases a column.',
  description: 'Applies trim + lower to one column, in place.',
  fields: [
    {
      key: 'column',
      label: 'Column',
      type: 'text',
      required: true,
      placeholder: 'email',
      help: 'Column rewritten in place.',
    },
  ],
  keywords: ['trim', 'lower', 'clean'],
  gotchas: ['Rewrites the column in place — cast it first if the type matters.'],
  examples: [{ title: 'Normalize an email column', json: '{ "type": "normalize_text", "column": "email" }' }],
}

That single entry gives you the palette item, the inspector form, the required-field lint rule and a line in the assistant’s system prompt.

Discovering what a runtime actually has

Registries are dynamic, so the authoritative list lives in the running process. Studio’s local runner exposes it:

curl -s localhost:8787/capabilities | jq

{
  "transformations": ["filter", "select", "…", "normalize_text"],
  "readers": ["parquet", "delta", "…", "elasticsearch"],
  "writers": ["parquet", "delta", "…", "elasticsearch"],
  "validators": ["not_null", "unique", "…", "no_future_date"]
}

Comparing that against the catalog is the quickest way to spot a custom type the editor does not know about yet.

Contributing it back

If the extension is general — a connector for a common database, a transformation every pipeline ends up rewriting — send a pull request. Two things make it mergeable: an example pipeline under examples/ that exercises it, and a catalog entry so Studio ships it too.