Architecture
🗃️ Woodrat
·
Go Kafka S3 PostgreSQL
System Context
Woodrat is the file cataloging and archiving layer for the joel.holmes.haus platform: it records file metadata, stores the bytes in object storage, and publishes lifecycle events over Kafka. Files are cataloged either through a cmd/woodrat CLI client or directly against the ConnectRPC API. Downstream services subscribe to woodrat.v1.file_pushed by name — the service currently recognizes owl-papers and owl-stacks as live subscribers and accepts owl-books as a target even though no consumer exists for it yet (see lib/woodrat/push/service.go).
C4Context
title System Context — Woodrat within joel.holmes.haus
Person(admin, "Admin / Operator", "Catalogs, archives, and pushes files via the woodrat CLI or API")
Boundary(platform, "joel.holmes.haus Platform") {
System(cli, "woodrat CLI", "Go / Cobra — cmd/woodrat: catalog, archive, push, folder, import, reconcile")
System(woodrat, "Woodrat", "File cataloging, archiving, and distribution service")
System(downstream, "Downstream services", "owl-papers, owl-stacks (live) · owl-books (accepted, no consumer yet)")
}
SystemDb(postgres, "PostgreSQL", "File and folder metadata")
SystemDb(minio, "MinIO / S3", "Blob storage for file contents")
SystemQueue(kafka, "Kafka", "FileCatalogedEvent · FileArchivedEvent · FilePushedEvent")
Rel(admin, cli, "Uses")
Rel(cli, woodrat, "ConnectRPC (FileService / FolderService)")
Rel(woodrat, postgres, "Reads / writes metadata")
Rel(woodrat, minio, "Stores / retrieves blobs")
Rel(woodrat, kafka, "Publishes lifecycle events")
Rel(kafka, downstream, "FilePushedEvent, by target name")
Note: go.mod also lists github.com/maxence-charriere/go-app/v10 (a Go WASM SPA framework), but no code in this repo imports it — there is no admin web UI implemented here today; it may be vestigial from initial scaffolding.
Container Diagram
C4Container
title Woodrat — Internal Containers
Boundary(woodrat, "Woodrat Service") {
Container(cli, "cmd/woodrat", "Go / Cobra", "CLI client — catalog, archive, push, folder, import, reconcile subcommands")
Container(api, "cmd/api", "Go / ConnectRPC", "Serves FileService and FolderService over HTTP/2 (h2c) on :9000")
Container(worker, "cmd/worker", "Go", "Standalone process, currently a no-op placeholder — no Kafka consumers implemented (per code comment: \"MVP\")")
Container(catalogSvc, "catalog.Service", "Go", "CatalogFile · GetFile · ListFiles — creates file records, checksums, MIME sniffing via gabriel-vasile/mimetype")
Container(archiveSvc, "archive.Service", "Go", "ArchiveFile — reads bytes from local disk and stores to object storage")
Container(pushSvc, "push.Service", "Go", "PushFile — validates target, marks file as pushed, publishes FilePushedEvent")
Container(folderSvc, "folder.Service", "Go", "CRUD for Folders; AddFilesToFolder; ArchiveFolder / PushFolder — implements its own per-file upload + publish, reusing only push.ValidateTarget")
Container(reconcileSvc, "reconcile.Service", "Go", "ReconcileUnprocessed — backfills File records for objects already in the bucket (e.g. a raw mc mirror / aws s3 sync) that were never cataloged")
ContainerDb(fileRepo, "lib/repo.FileRepo", "PostgreSQL / squirrel", "files table — uuid, name, checksum, mime_type, origin, folder_uuid, cataloged_at/archived_at")
ContainerDb(folderRepo, "lib/repo.FolderRepo", "PostgreSQL / squirrel", "folders table")
}
SystemDb(postgres, "PostgreSQL", "Persistent metadata store")
SystemDb(minio, "MinIO / S3", "Blob store (gocloud.dev/blob)")
SystemQueue(kafka, "Kafka", "Event bus")
Rel(cli, api, "ConnectRPC")
Rel(api, catalogSvc, "delegates (via catalog.NewCombinedService, which also wraps archiveSvc/pushSvc/reconcileSvc for FileService RPCs)")
Rel(api, folderSvc, "delegates")
Rel(catalogSvc, fileRepo, "CRUD")
Rel(archiveSvc, fileRepo, "reads / updates")
Rel(archiveSvc, minio, "PutObject")
Rel(pushSvc, fileRepo, "updates state")
Rel(pushSvc, kafka, "FilePushedEvent")
Rel(folderSvc, folderRepo, "CRUD")
Rel(folderSvc, fileRepo, "reads / updates, for bulk archive+push")
Rel(folderSvc, minio, "Upload, per file in ArchiveFolder")
Rel(folderSvc, kafka, "FileArchivedEvent / FilePushedEvent, per file")
Rel(reconcileSvc, fileRepo, "creates records for unmatched objects")
Rel(reconcileSvc, minio, "lists / reads unprocessed/ prefix")
Rel(reconcileSvc, kafka, "FileCatalogedEvent")
Rel(fileRepo, postgres, "SQL")
Rel(folderRepo, postgres, "SQL")
Rel(catalogSvc, kafka, "FileCatalogedEvent")
Rel(archiveSvc, kafka, "FileArchivedEvent")
Note: worker (cmd/worker/main.go) starts, logs “no consumers in MVP”, and blocks on a signal — it does not currently consume from Kafka or call into catalogSvc. All event publishing today happens synchronously from cmd/api’s request handlers.
The repository also contains a root main.go + cmd/ package (a separate, older Cobra scaffold using raw gRPC) and cmd/form; these are unfinished generator boilerplate — most subcommands are unimplemented stubs (see cmd/model_cmd.go: “This file is a placeholder”) — and are not part of the working CLI. The real, working CLI is cmd/woodrat.
File Lifecycle
stateDiagram-v2
[*] --> Cataloged : CatalogFile RPC\n(metadata recorded, no bytes yet)
Cataloged --> Archived : ArchiveFile RPC\n(bytes written to MinIO/S3)
Archived --> Pushed : PushFile RPC\n(FilePushedEvent published to Kafka)
Pushed --> [*]
Cataloged --> Pushed : PushFile RPC\n(no server-side guard requires prior archiving —\nsee lib/woodrat/push/service.go PushFile)
Data Model
Reflects lib/repo/migrations/00000001_model.up.sql. The files table has no state column — lifecycle is inferred from archived_at being null/non-null (there’s no separate “pushed” flag persisted; pushes are fire-and-forget events).
erDiagram
folders {
text uuid PK
text name
text description
timestamptz created_at
}
files {
text uuid PK
text name
text mime_type
bigint size_bytes
text checksum
text storage_path
jsonb origin
timestamptz cataloged_at
timestamptz archived_at "nullable"
text_array tags
text description
text folder_uuid FK
}
folders ||--o{ files : contains
The migration also generates fileorigins, filecatalogedevents, filearchivedevents, and filepushedevents tables with full CRUD methods in lib/repo/model.go. These appear to be leftover scaffolding from a code generator — no service in lib/woodrat/ calls them; origin is stored inline as JSONB on files, and events are published to Kafka rather than persisted. They’re omitted above as dead weight.
Environment Variables
| Variable | Required | Description |
|---|
DATABASE_URL | Yes | PostgreSQL connection string |
KAFKA_BROKERS | Yes | Kafka broker address(es) |
WOODRAT_BUCKET_URL | No | gocloud.dev/blob URL (e.g. s3://woodrat?endpoint=minio:9000). If unset, cmd/api starts with archiving disabled — ArchiveFile/ArchiveFolder/ReconcileUnprocessed are unavailable rather than the service failing to start |
OTEL_EXPORTER_OTLP_ENDPOINT | No | OpenTelemetry collector; telemetry is a noop if unset |
WOODRAT_API_URL | No | Used only by the cmd/woodrat CLI to locate the API; defaults to http://localhost:9000 |