Architecture

Status

Narwhal is early-stage. There is no web UI — the only surfaces are a ConnectRPC API, a narwhal CLI, and a background Kafka worker. The data model (products → design docs → system docs) is the real, working core; the broader “personal product catalog” ambitions described in the repo are not built yet.

System Context

Narwhal is one service in a small personal platform. It owns the canonical notion of a “Product” — every other service references a Narwhal product UUID rather than defining its own.

                 ┌──────────────┐
                 │  narwhal CLI │  (cobra: product/doc/system/sync/dump)
                 └──────┬───────┘
                         │ ConnectRPC (HTTP/2, h2c)
                         ▼
┌───────────┐     ┌──────────────┐     ┌────────────┐
│ Postgres  │◄────┤ narwhal api  │────►│   Kafka     │
│ (goose    │     │ cmd/api      │     │ (via        │
│ migrations)│    └──────────────┘     │ archaea/    │
└───────────┘                          │  kafka)     │
                                         └─────┬──────┘
                                               │
                                         ┌─────▼──────┐
                                         │ narwhal     │
                                         │ worker      │
                                         │ cmd/worker  │
                                         └─────┬──────┘
                                               │
                     ┌─────────────────────────┼─────────────────────┐
                     ▼                         ▼                     ▼
              designs/ (markdown          Magpie (search        Rabbit (design
              files on disk)              resource index)       back-references)

• narwhal api (cmd/api): runs goose migrations on boot, then serves ProductService, DocService, and SystemDocService over ConnectRPC/h2c, plus /health and /info.
• narwhal worker (cmd/worker): consumes doc and product change events from Kafka. For docs, it writes a markdown snapshot to DESIGNS_DIR, publishes a magpie.v1.Resource so the doc is searchable, and separately consumes rabbit.v1.Design events to set rabbit_design_uuid back-references on the originating doc.
• narwhal CLI (cmd/narwhal): talks to the API over ConnectRPC (gRPC-Web). Subcommands: product add/list/rekey, doc new/edit/list/show/status/promote, system edit/show, sync, dump.
• Shared library dependencies: archaea (Kafka client/producer/consumer, worker runner, telemetry), magpie (resource/search schema), rabbit (design/ticket schema) — all sibling repos under github.com/holmes89.

Data Model

Three tables, managed by pressly/goose migrations in lib/repo/migrations/:

products — the cross-service project identity.

• uuid (deterministic, derived from slug via lib/narwhal/uid), slug, prefix (short uppercase code), name, description, next_seq, meerkat_project_uuid (added in migration 4, links to the Meerkat project tracker).
• A nil-UUID "unknown" sentinel product exists (via narwhal product rekey) to represent items in other services not yet tied to a real product.

design_docs — the actual unit of work.

• Belongs to a product (product_uuid), has a type (feature, discovery, spike, system) and a status (draft, review, approved, deprecated).
• seq is a per-scope index: unique within the product when discovery_uuid is null, or unique within the parent discovery doc when set (migration 5). This lets a discovery doc spawn its own numbered child docs, separate from the product’s top-level numbering.
• rabbit_design_uuid links a promoted doc to its Rabbit design/ticket counterpart.
• Migration 5 also dropped the earlier human_id/rabbit_human_id text-ID columns in favor of the numeric seq as the doc’s index — an early naming scheme that didn’t survive contact with the discovery hierarchy.

system_docs — one long-lived architecture/system doc per product (product_uuid UNIQUE), edited via narwhal system edit.

ConnectRPC API Surface

Defined in schemas/narwhal/v1/narwhal.proto, generated into lib/schemas/narwhal/v1/.

ProductService: CreateProduct, GetProduct, ListProducts, UpdateProduct.

DocService: CreateDoc, GetDoc, ListDocs (filterable by product/status/type/discovery parent), UpdateDoc, SetDocStatus, PromoteDoc, SetDocRabbitRef (worker-side back-reference setter).

PromoteDoc does not itself create a Rabbit design: it sets the doc’s status to approved and publishes a change event; its rabbit_project/rabbit_sprint_id request fields are currently unused. The actual Rabbit design gets created out-of-band, and the link back is completed asynchronously when the worker consumes a rabbit.v1.Design event and calls SetDocRabbitRef (see runRabbitDesignConsumer in cmd/worker/main.go). A RabbitClient wrapper exists at lib/narwhal/rabbit/client.go with a CreateDesign method, but it is not wired into PromoteDoc or called anywhere in the codebase yet.

SystemDocService: UpsertSystemDoc, GetSystemDoc.

Each service’s Go implementation lives under lib/narwhal/<product|doc|systemdoc>/service.go, backed by a store.go doing raw pgx queries (not yet using squirrel), and optionally wired to a Kafka Publisher via WithPublisher(...).

Markdown Sync

narwhal sync and narwhal dump (lib/narwhal/sync/{load,dump}.go) round-trip design docs between the database and a directory of markdown files with YAML-ish frontmatter (uuid, index, status, type, discovery, rabbit ref, updated_at). The worker also writes this same file layout as a side effect of processing doc events, so designs/<product-slug>/features/<seq>-<slug>.md stays current without running dump manually.

Notable Gaps

• No authentication/authorization on the ConnectRPC API.
• No web UI; all interaction is via CLI or direct RPC calls.
• Store layer uses hand-written pgx SQL rather than a query builder.
• Migration 5’s column drops (human_id, rabbit_human_id) indicate the ID scheme is still settling — expect further schema churn as the discovery-doc hierarchy gets used more.
• PromoteDoc → Rabbit design creation is not actually wired up yet (see above); the existing lib/narwhal/rabbit client is unused.