Architecture

📚 Weevil  ·  Go Kafka PostgreSQL

System Context

C4Context title System Context — Weevil within joel.holmes.haus Person(admin, "Admin", "Tracks personal reading list, searches for and adds books") Boundary(platform, "joel.holmes.haus Platform") { System(ui, "joel.holmes.haus", "Go-app WASM admin SPA") System(weevil, "Weevil", "Book tracking and enrichment service") System(magpie, "Magpie", "Resource hub — receives book resources from Weevil") System(shrike, "Shrike", "Search — indexes book metadata") } SystemDb(postgres, "PostgreSQL", "Books and book metadata (enrichment results)") SystemDb(minio, "MinIO / S3", "Archived Google Books and Open Library JSON responses") SystemQueue(kafka, "Kafka", "weevil.v1.Book · magpie.v1.Resource") System_Ext(googlebooks, "Google Books API", "Book metadata and cover images") System_Ext(openlibrary, "Open Library API", "ISBN-based book metadata") Rel(admin, ui, "Uses") Rel(ui, weevil, "ConnectRPC") Rel(weevil, postgres, "Reads / writes") Rel(weevil, minio, "Archives enrichment API responses") Rel(weevil, kafka, "Publishes weevil.v1.Book") Rel(weevil, googlebooks, "Search + metadata enrichment") Rel(weevil, openlibrary, "ISBN metadata enrichment") Rel(kafka, magpie, "magpie.v1.Resource") Rel(kafka, shrike, "resource events")

Container Diagram

C4Container title Weevil — Internal Containers Boundary(weevil, "Weevil") { Container(api, "cmd/api", "Go / ConnectRPC h2c :9000", "BookService (CRUD + deterministic UUID from ISBN/title) · SearchService (Google Books proxy)") Container(worker, "cmd/worker", "Go / Kafka", "BookConsumer · Enricher goroutine") Container(uiSrv, "cmd/ui", "Go-app WASM :8000", "Browser SPA — book list, create, enrichment view, search") Container(bookSvc, "book.Service", "Go", "Create · Get · List · Update · Delete") Container(searchSvc, "search.Service", "Go", "Stateless proxy to Google Books API") Container(enricher, "enrichment.Enricher", "Go / goroutine", "6-step pipeline: Google Books → Open Library → archive both → persist metadata → magpie (stub)") ContainerDb(bookRepo, "BookRepo + BookMetadataRepo", "PostgreSQL / squirrel", "books · book_metadata (ON CONFLICT DO UPDATE)") } SystemDb(postgres, "PostgreSQL", "") SystemDb(minio, "MinIO / S3", "books//google-books.json · open-library.json") SystemQueue(kafka, "Kafka", "consumer group weevil-service-group") System_Ext(googlebooks, "Google Books API", "") System_Ext(openlibrary, "Open Library API", "") Rel(api, bookSvc, "delegates · publishes weevil.v1.Book to Kafka on Create") Rel(api, searchSvc, "delegates") Rel(kafka, worker, "weevil.v1.Book") Rel(worker, bookSvc, "Persist book from Kafka") Rel(worker, kafka, "Publishes magpie.v1.Resource") Rel(worker, enricher, "Enqueue(EnrichBookRequest)") Rel(enricher, googlebooks, "Search by GoogleID / ISBN / title") Rel(enricher, openlibrary, "Fetch by ISBN") Rel(enricher, minio, "Archive JSON responses") Rel(enricher, bookRepo, "Upsert book_metadata") Rel(bookRepo, postgres, "SQL")

System Overview

Weevil is structured around a classic event-driven backend. The API server publishes book creation events to Kafka; the worker consumes them, persists the book, and enqueues an enrichment request on a buffered channel for asynchronous processing.

graph TD CLI["CLI (weevil)"] Browser["Browser"] UI["cmd/ui :8000\ngo-app WASM SPA"] API["cmd/api :9000\nBookService · SearchService"] Kafka[("Kafka\nweevil.v1.Book")] Worker["cmd/worker\nBookConsumer"] Enricher["Enricher goroutine\nbuffered channel"] PG[("PostgreSQL")] MinIO[("MinIO / S3")] CLI -->|"ConnectRPC"| API Browser --> UI UI -->|"ConnectRPC"| API API -->|"weevil.v1.Book"| Kafka Kafka --> Worker Worker -->|"persist book"| PG Worker -->|"magpie.v1.Resource"| MagpieKafka[("Kafka\nmagpie topic")] Worker -->|"Enqueue()"| Enricher Enricher -->|"Google Books API"| GB["Google Books"] Enricher -->|"Open Library API"| OL["Open Library"] Enricher -->|"archive JSON"| MinIO Enricher -->|"upsert metadata"| PG

Binary Breakdown

cmd/api — API Server

Serves two ConnectRPC services over HTTP/2 (h2c):

  • BookService — CRUD for books. CreateBook generates a deterministic UUID from ISBN or title (MD5-namespaced), publishes the entity to Kafka. GetBook also returns enrichment metadata from book_metadata if available.
  • SearchService — stateless; proxies queries to the Google Books API directly. Returns candidate Book objects without persisting.

Both services are wrapped with CORS middleware (all origins) to support browser access.

Dependency wiring:

psql.Conn → psql.BookRepo → book.BookService → bookgrpc.BookService
         ↘ psql.BookMetadataRepo → bookgrpc.BookService.WithMetadataRepo
kafka.Conn → kafka.Producer[*Book] → bookgrpc.BookService

cmd/worker — Background Worker

Runs two concurrent processes in one binary:

  1. BookConsumer — subscribes to Kafka topic weevil.v1.Book (consumer group weevil-service-group). Persists the book, publishes a magpie.v1.Resource event, and enqueues an enrichment request on the Enricher’s buffered channel.
  2. Enricher — drains the channel in a dedicated goroutine, running the six-step enrichment pipeline for each book.

cmd/ui — Browser SPA

go-app v9 WebAssembly SPA served on port 8000. Calls the API on port 9000 using ConnectRPC.

Routes:

/                     → HomePage
/books                → BookListPage
/books/create         → BookCreatePage
/books/<uuid>         → BookGetPage + enrichment metadata
/searchs/create       → SearchCreatePage
/searchs/<uuid>       → SearchGetPage

Library Packages

lib/weevil/enrichment — Enrichment Pipeline

The Enricher receives EnrichBookRequest values via a buffered channel (default capacity: 100). Each request is processed synchronously in a single goroutine to avoid stampeding the external APIs. The pipeline mirrors the previous Temporal workflow structure:

StepFunctionNotes
1FetchGoogleBooksActivitytries: GoogleID → ISBN-13 → ISBN-9 → title
2FetchOpenLibraryActivityby ISBN-13 then ISBN-9; skipped if no ISBN
3ArchiveGoogleResponseActivityuploads raw JSON to books/<uuid>/google-books.json
4ArchiveOpenLibraryResponseActivityuploads to books/<uuid>/open-library.json
5PersistBookMetadataActivityupserts book_metadata table — required
6SubmitToMagpieActivityTODO: not yet implemented

Steps 1–4 and 6 are non-fatal (logged and skipped on error). Step 5 is required; failure aborts the pipeline for that book. If the queue is full when Enqueue is called, the request is dropped with a warning log — enrichment is best-effort.

lib/client/ — External API Clients

  • google — Google Books REST client using go-resty; supports search by title/author/ISBN and full volume fetch
  • goodreads — Goodreads RSS feed parser using gofeed; paginates and extracts ISBN, author, Goodreads ID
  • openlibrary — Open Library Books API client using net/http; fetches by ISBN, extracts covers, subjects, publishers

lib/repo/psql — PostgreSQL Repositories

  • Conn — wraps database/sql, runs goose migrations on startup
  • BookRepo — implements CRUD; maps proto Book fields to/from SQL
  • BookMetadataRepo — upserts enrichment data; performs ON CONFLICT ... DO UPDATE

lib/storage — Object Storage

BlobClient (alias MinioClient) wraps gocloud.dev/blob backed by an S3-compatible endpoint. Used by enrichment activities to archive raw API responses.

Event Flow: Book Creation

graph TD A["CLI / Browser UI\nCreateBook(data)"] B["cmd/api BookService.CreateBook\nderive UUID from ISBN or title MD5"] C[("Kafka\nweevil.v1.Book")] D["cmd/worker BookConsumer.run()"] E["book.BookService.Create\n→ psql.BookRepo.Create\n→ books table"] F[("Kafka\nmagpie.v1.Resource")] G["enrichment.Enricher\nbuffered channel goroutine"] H["FetchGoogleBooksActivity\n→ Google Books API"] I["FetchOpenLibraryActivity\n→ Open Library API"] J[("MinIO\nbooks/uuid/google-books.json\nbooks/uuid/open-library.json")] K[("PostgreSQL\nbook_metadata table")] A --> B B -->|"publish"| C C --> D D --> E D -->|"publish"| F D -->|"Enqueue()"| G G --> H G --> I H -->|"ArchiveGoogleResponseActivity"| J I -->|"ArchiveOpenLibraryResponseActivity"| J G -->|"PersistBookMetadataActivity"| K

Environment Variables

API Server (cmd/api)

VariableDescription
DATABASE_URLPostgreSQL connection string
KAFKA_BROKERSKafka broker address
GOOGLE_BOOKS_API_KEYOptional; improves Google Books rate limits

Worker (cmd/worker)

VariableDescription
DATABASE_URLPostgreSQL connection string
KAFKA_BROKERSKafka broker address
MINIO_ENDPOINTMinIO/S3 endpoint; archiving skipped if unset
MINIO_ACCESS_KEYMinIO access key
MINIO_SECRET_KEYMinIO secret key
MINIO_BUCKETBucket name (default: weevil)