Problem
It is easy to bolt an LLM onto a document set and get confident-sounding answers with no way to check where they came from.
What I built
A document ingestion pipeline chunks and embeds text into PostgreSQL with pgvector, storing metadata alongside vectors. Retrieval runs through FastAPI with metadata filtering and hybrid retrieval controls, and generation stays citation-preserving, so every answer traces back to the source chunks that produced it, with token/cost logging and a fixed evaluation harness to catch retrieval regressions.
Why it matters
A retrieval system with the guardrails that make it usable in production: citations on every answer, a fixed evaluation set, and per-query cost you can actually see.
Result
Answers are source-linked and auditable. Retrieval quality is regression-tested against a fixed evaluation set on every meaningful change, and cost is tracked per query.
Architecture
- Ingestion: chunking and embedding generation, stored in PostgreSQL/pgvector alongside source metadata
- Retrieval: vector similarity plus metadata filtering, with hybrid retrieval controls
- Citation-preserving generation: responses link back to the exact source chunks used
- Token and cost logging per request
- Fixed evaluation harness re-run against retrieval changes to catch regressions before they ship
Testing & reliability
A fixed evaluation harness runs retrieval quality checks on every meaningful change, catching regressions before they reach generation.
Stack
- Python
- FastAPI
- PostgreSQL
- pgvector
- OpenAI API
- Docker