Architecture

🔍 Shrike  ·  Go Qdrant Ollama Kafka PostgreSQL

System Context

C4Context title System Context — Shrike within joel.holmes.haus Boundary(platform, "joel.holmes.haus Platform") { System(ui, "joel.holmes.haus", "Go-app WASM admin SPA — submits search queries") System(shrike, "Shrike", "Full-text and semantic search indexing service") System(lynx, "Lynx", "Web archiving — publishes TextExtractedEvent after enrichment") System(greyseal, "Grey Seal", "RAG conversations — queries Shrike for context retrieval") System(magpie, "Magpie", "Resource index — publishes Resource events for indexing") } SystemDb(postgres, "PostgreSQL", "Index records and full-text document store") SystemDb(qdrant, "Qdrant", "Vector store — cosine similarity search (768-dim)") SystemExternal(ollama, "Ollama", "Local LLM — nomic-embed-text embeddings") SystemQueue(kafka, "Kafka", "TextExtractedEvent · EnvelopeEvent · ImportRequest · magpie.v1.Resource") Rel(ui, shrike, "ConnectRPC search queries") Rel(greyseal, shrike, "ConnectRPC hybrid search") Rel(lynx, kafka, "Publishes TextExtractedEvent") Rel(magpie, kafka, "Publishes magpie.v1.Resource") Rel(shrike, postgres, "Reads / writes index records") Rel(shrike, qdrant, "Vector upsert / search") Rel(shrike, ollama, "Embed text chunks") Rel(shrike, kafka, "Consumes 4 topics")

Container Diagram

C4Container title Shrike — Internal Containers Boundary(shrike, "Shrike") { Container(api, "cmd/api", "Go / ConnectRPC h2c :9000", "IndexRecordService (CRUD) · SearchService (semantic + keyword + reindex)") Container(worker, "cmd/worker", "Go / Kafka", "4 consumers: TextExtracted · Envelope · Import · Resource") Container(indexSvc, "index_record.Service", "Go", "CRUD for index records") Container(chunker, "lib/chunker", "Go", "512-word overlapping windows, 64-word overlap") Container(embedder, "OllamaEmbedder", "Go / HTTP", "nomic-embed-text via Ollama REST API") Container(vectorStore, "QdrantStore", "Go / HTTP", "Upsert · Search · DeleteByEntity on shrike_context collection") ContainerDb(indexRepo, "IndexRecordRepo + DocumentRepo", "PostgreSQL / squirrel", "indexrecords (tsvector GIN) · documents tables") } SystemDb(postgres, "PostgreSQL", "") SystemDb(qdrant, "Qdrant", "shrike_context collection, 768-dim cosine") SystemExternal(ollama, "Ollama", "http://ollama:11434") SystemDb(minio, "MinIO / S3", "Blob source for ImportConsumer") SystemQueue(kafka, "Kafka", "") Rel(api, indexSvc, "delegates CRUD") Rel(api, chunker, "Reindex path") Rel(api, embedder, "Embed query") Rel(api, vectorStore, "Search / DeleteByEntity") Rel(api, indexRepo, "Keyword search (tsvector)") Rel(worker, indexSvc, "Upsert index records") Rel(worker, chunker, "Chunk full text") Rel(worker, embedder, "Embed chunks") Rel(worker, vectorStore, "Upsert points") Rel(worker, minio, "Download blobs (ImportConsumer)") Rel(worker, kafka, "Consumes 4 topics") Rel(indexRepo, postgres, "SQL") Rel(embedder, ollama, "POST /api/embed") Rel(vectorStore, qdrant, "HTTP REST")

Process Decomposition

Shrike is deployed as two containerised binaries (api and worker) plus an optional browser UI (ui). All three share the same Go module and library code under lib/.

API Server (cmd/api/main.go)

Starts an HTTP/2 server (h2c, port 9000) with:

  • IndexRecordService — ConnectRPC handler wrapping indexRecordService (CRUD)
  • SearchService — ConnectRPC handler with direct access to IndexRecordRepo, DocumentRepo, OllamaEmbedder, and QdrantStore
  • Logging middleware and CORS middleware (all origins allowed)
  • /health endpoint

Runs database migrations on startup (via goose, embedded SQL).

Worker (cmd/worker/main.go)

Long-running process that starts four Kafka consumers as goroutines and blocks until SIGINT/SIGTERM:

  1. TextExtractedConsumer (group shrike-text-indexer) — deserialises TextExtractedEvent, upserts IndexRecord, stores full text in DocumentRepo, and vectorises (chunk → embed → Qdrant upsert).
  2. EnvelopeConsumer (group shrike-envelope) — deserialises EntityEnvelope, skips non-deleted events, deletes the corresponding IndexRecord.
  3. ImportConsumer (group shrike-import) — deserialises ImportRequest, downloads text from S3/MinIO via BlobStore, stores text, upserts IndexRecord, and vectorises.
  4. ResourceConsumer (group shrike-magpie-resource) — deserialises a magpie Resource proto, upserts IndexRecord, stores minimal text, and vectorises.

The worker calls vectorStore.EnsureCollection at startup to create the Qdrant collection shrike_context if it does not exist.

Library Layout (lib/)

lib/
  chunker/          — text chunking (overlapping word windows)
  embedding/        — Embedder interface + OllamaEmbedder implementation
  repo/             — Postgres: Conn, IndexRecordRepo, DocumentRepo, embedded migrations
  schemas/          — generated protobuf Go code (shrike/v1 entities + services)
  shrike/
    index_record/   — IndexRecordService interface + implementation, consumer logic
    import_request/ — ImportConsumer
    resource/       — ResourceConsumer
  storage/          — BlobStore (S3-compatible via gocloud.dev)
  ui/               — go-app WebAssembly UI
  vector/           — Store interface + QdrantStore implementation

Data Flow: Ingest via TextExtractedEvent

graph TD Upstream["Upstream service"] -->|"TextExtractedEvent"| Kafka[("Kafka")] Kafka --> Consumer["Worker: TextExtractedConsumer"] Consumer -->|"1. Upsert fullText"| DocRepo["DocumentRepo\n(documents table)"] Consumer -->|"2. Create IndexRecord"| IdxRepo["IndexRecordRepo\n(indexrecords table)"] Consumer -->|"3. Chunk text\n512 words, 64 overlap"| Chunker["lib/chunker"] Chunker -->|"4. Embed each chunk"| Ollama["OllamaEmbedder\nHTTP → Ollama"] Ollama -->|"5. Upsert vector points"| Qdrant[("QdrantStore\nshrike_context")] DocRepo --> PG[("PostgreSQL")] IdxRepo --> PG

Data Flow: Search Request

graph TD Client -->|"SearchService.Search\nquery · mode · entity_uuids"| API["cmd/api"] API -->|"semantic path (default)"| Embed["OllamaEmbedder.Embed(query)"] Embed -->|"vector search + MatchAny filter"| QSearch["QdrantStore.Search"] QSearch --> SR1["SearchResult{snippet=chunk_text}"] API -->|"keyword path\nmode=keyword or fallback"| KWSearch["IndexRecordRepo.SearchKeyword\ntsvector GIN"] KWSearch --> SR2["SearchResult"]

Data Flow: Reindex

graph TD Client -->|"SearchService.Reindex(uuid)"| API["cmd/api"] API -->|"1. GetFullText"| DocRepo["DocumentRepo (Postgres)"] API -->|"2. DeleteByEntity"| Qdrant[("QdrantStore\nfilter by entity_uuid payload")] DocRepo -->|"fullText"| Chunker["3. chunker.Chunk"] Chunker -->|"chunks"| Ollama["OllamaEmbedder.Embed"] Ollama -->|"4. Upsert new points"| Qdrant

External Dependencies

SystemRoleDefault address
PostgresIndexRecord metadata, full text, tsvector keyword indexdb:5432
QdrantVector storage, cosine similarity searchhttp://qdrant:6333
OllamaText embedding (nomic-embed-text, 768 dims)http://ollama:11434
Kafka / RedpandaEvent bus (four consumer groups)redpanda-0:9092
S3 / MinIOBlob storage for imported text filesS3_ENDPOINT env var

Chunking Strategy

lib/chunker/Chunk(text, size=512, overlap=64) splits text into overlapping word-level windows:

  • Step size = 512 − 64 = 448 words
  • Each window is up to 512 words, with 64-word overlap from the previous chunk
  • Qdrant point IDs are deterministic: uuid.NewSHA1(NameSpaceURL, "<entity_uuid>:<chunk_index>")

Qdrant Collection

  • Collection name: shrike_context
  • Vector size: 768 (matches nomic-embed-text output)
  • Distance metric: Cosine
  • Created/ensured by the worker at startup

Database Migrations

Three goose migrations embedded in lib/repo/migrations/:

  1. 00000001_index_record.up.sql — creates indexrecords table
  2. 00000002_tsvector.up.sql — adds search_vector generated column + GIN index
  3. 00000003_documents.up.sql — creates documents table with FK to indexrecords

Migrations run automatically on API server startup. The worker skips them.