Pangolin

Pangolin is my Postgres-to-Parquet snapshot and analytics export service — it reaches into the databases behind my other services once an hour, dumps their tables as Parquet onto object storage, and lets me query them with DuckDB.

Every service owns its own database and none expose an analytics surface, so a cross-service question used to mean a psql shell against each one by hand. Instead pangolin-worker snapshots nine services’ tables to date-partitioned Parquet on MinIO on a cron, and pangolin-api runs an embedded DuckDB engine that queries those files directly off object storage. A separate publish job republishes a few services’ public data as JSON on a real S3 bucket for static sites and browser widgets.

Pangolin is my Postgres-to-Parquet snapshot and analytics export service — a service that reaches into the Postgres databases behind my other self-hosted services once an hour, dumps their tables out as Parquet, and lands them on MinIO where I can query them with DuckDB.

Why I Built It

Every service in my platform (weevil, owl, magpie, shrike, greyseal, lynx, woodrat, narwhal, rabbit) owns its own Postgres database, and none of them expose any kind of analytics or export surface — the only way to answer a cross-service question like “how many books did I add across weevil and owl last month” was to open a psql shell against each one by hand. I wanted a single place that periodically pulls the current state of everything into a columnar format I can query cheaply and keep around historically, without adding reporting endpoints to nine different services.

Parquet plus DuckDB’s httpfs extension turned out to be the easiest way to get that: pangolin-worker snapshots each service’s tables to date-partitioned Parquet on MinIO on an hourly cron, and pangolin-api runs an embedded DuckDB engine that queries those files directly off object storage on demand.

I want to be upfront that this started as an analytics export, not a backup system, and it still is — see the “Known Limitations” note in the README for why I don’t yet treat these snapshots as disaster recovery.

What It Does

• Runs an hourly snapshot job (SNAPSHOT_CRON) that connects directly to each service’s Postgres database, reads its tables, and writes them out as Parquet files to a private MinIO bucket, partitioned by service, table, and date.
• Covers nine services today: weevil (books), owl (books and papers), magpie (resources and labels), shrike (search index records), greyseal (conversations and messages), lynx (RSS feeds and websites), woodrat (cataloged files), narwhal (products and design docs), and rabbit (projects, sprints, and tickets).
• Exposes an embedded DuckDB query engine (internal/query) over POST /api/v1/query that reads the Parquet files straight off MinIO via httpfs — no separate data warehouse or ETL step to keep in sync.
• Runs a separate, lower-frequency publish job (PUBLISH_CRON) that reads a handful of services’ own APIs (never their databases) and republishes their current public data as per-entity JSON on a real, internet-reachable AWS S3 bucket, for static sites and browser widgets to consume directly.
• Ships as two binaries from one codebase: cmd/api (HTTP server: health, manual snapshot/publish triggers, query, snapshot listing) and cmd/worker (the actual gocron scheduler that runs the snapshot and publish jobs unattended).

Tech Stack

• Backend: Go, net/http, jackc/pgx (Postgres reads)
• Export format: Apache Parquet (parquet-go/parquet-go)
• Object storage: MinIO (private snapshot bucket) and AWS S3 (public publish bucket), both via gocloud.dev/blob
• Query layer: DuckDB (marcboeker/go-duckdb) with the httpfs extension reading Parquet directly from MinIO
• Scheduling: go-co-op/gocron

For a deeper look at how the pieces fit together, see Architecture.