Supabase Pipelines adds DuckLake replication in public alpha

Supabase Pipelines, which is in public alpha, now offers a DuckLake replication feature that lets developers mirror Postgres tables into a lakehouse for current‑state analytical queries. The replication creates read‑only snapshots that can be queried with a compatible engine such as DuckDB.

Tables that only receive inserts can be replicated without defining a primary key or replica identity. For tables that receive updates or deletions, a published Postgres row identity must be enabled so that changes can be tracked.

Users can configure the destination in two ways: either let Pipelines use the same Supabase project’s Postgres database as a metadata catalog and a storage bucket in that project as object storage, or supply their own catalog URL and S3‑compatible storage credentials. When using the built‑in option, Pipelines automatically creates the catalog schema and the bucket, and it also generates the necessary connection secrets. Optionally, users can adjust the pool size in advanced settings to control the number of concurrent DuckDB connections.

DuckLake stores table metadata in a Postgres catalog and writes the actual data as Parquet files in object storage under an s3:// path. Because Pipelines does not include a query endpoint, developers must connect a DuckDB or another compatible engine to read the catalog and the Parquet files.

After the initial sync, the pipeline creates current‑state tables, copies rows from the source, and then continuously applies published changes and supported schema modifications. The engine can run between one and six concurrent DuckDB connections, with four connections as the default setting. Column names are converted to lowercase, and DuckDB compares identifiers without ASCII case distinctions, so source schemas should avoid names that differ only by case.

All catalog URLs and storage credentials are kept as secrets and are not returned after the destination is created, protecting access keys from exposure. A unique metadata schema name using only letters, numbers, and underscores must be provided when creating the DuckLake catalog. The replicated tables should be treated as read‑only; writes performed outside Pipelines may conflict with the replication process and background maintenance tasks.