Shrike

Shrike is a unified search service for my self-hosted platform — a single endpoint any service can query instead of each one maintaining its own index.

It sits downstream of the Kafka bus, ingests events from all producers, and builds a dual-backend index: Qdrant for semantic (vector) search using Ollama embeddings, and PostgreSQL for keyword full-text search. When Ollama or Qdrant is unavailable it falls back to keyword search automatically. Any service that needs search talks to Shrike.

Shrike is a unified search and retrieval service — a single endpoint any service in my platform can query instead of each one maintaining its own search index.

Why I Built It

As the number of services grew, so did the need to search across them. I didn’t want every service managing its own embeddings and full-text indexes independently. Shrike sits downstream of the Kafka bus, ingests events from all producers, and builds a shared dual-backend index: Qdrant for semantic (vector) search and PostgreSQL for keyword full-text search. Any service that needs search talks to Shrike rather than its own database.

It also gave me a focused project to work with embedding models and vector databases in a real, albeit personal-scale, pipeline.

What It Does

• Consumes Kafka events from upstream services (Lynx, Weevil, Magpie, and others) and indexes their content automatically.
• Chunks extracted text into overlapping 512-word windows, embeds each chunk via Ollama (nomic-embed-text, 768 dimensions), and inserts vectors into Qdrant.
• Stores entity metadata and full extracted text in PostgreSQL with a tsvector GIN index for keyword search.
• Exposes a ConnectRPC SearchService supporting semantic search (Qdrant), keyword search (Postgres tsvector), and automatic fallback between the two when Ollama or Qdrant is unavailable.
• Provides a GetEntity RPC for fetching a stored record and its full text by UUID.
• Serves a WebAssembly browser UI for browsing and reindexing records.

Tech Stack

• Backend: Go, ConnectRPC, PostgreSQL
• Vector store: Qdrant
• Embedding model: Ollama (nomic-embed-text)
• Messaging: Kafka (Redpanda in local development)
• Frontend: WebAssembly SPA written in Go using go-app

For a deeper look at how the pieces fit together, see ARCH.md.