Retrieval over ten thousand policy documents

The knowledge already existed. It sat in more than 10,000 policy documents, assessments and reports: most of them long, most of them PDFs, and nearly all written for people who already knew the field.

Retrieval over a corpus this size is a coverage problem before it is a model problem. That sentence is the whole project in one line. Almost every hour I lost went to documents the system could not read properly, not to the model being insufficiently clever.

The second complication was the audience. A municipal officer, a researcher and a school student ask about the same material and need three different answers from it, in more than one language. The platform answers in English, Italian, Portuguese or Greek, at one of two reading levels. That last part is a framing decision as much as a technical one, and it shaped the retrieval design more than any benchmark did.

The pipeline reads PDFs, spreadsheets, CSVs, JSON and video. That list is less elegant than it sounds and it was more of the work than anyone expects.

The part people underestimate is vision extraction. In a policy report the figures and tables carry something close to half the meaning. A chart of emissions against a target year is the finding; the paragraph beside it is commentary. Text-only extraction throws the finding away and keeps the commentary, and you do not notice until someone asks a question whose answer only ever existed inside a chart.

Chunking and summarisation sit on top of that. Both are where a corpus stops being a pile of files and starts being something you can retrieve against.

Embeddings and a vector store get you a long way, and for a while they were the whole system. The failure mode shows up on questions that span documents. Plain similarity search treats every passage as an island: it returns the five chunks most like your question, and if the answer is the relationship between two reports rather than a sentence inside either, it cannot see it.

So the platform reasons across a graph. Microsoft GraphRAG does entity and relationship extraction with hierarchical community modelling, which means the system can follow how sources relate instead of ranking them independently. It also makes an answer traceable back to the documents it came from, which for a public-sector audience is not a nice-to-have. An answer nobody can check is not usable in policy work.

Milvus holds the vectors, OpenSearch handles lexical retrieval, and Redis keeps the hot path warm. Hybrid rather than pure semantic search, because policy documents are full of exact strings such as regulation numbers, and embeddings are reliably mediocre at exact strings.

This is where most of my time went. Early responses took around a minute. A minute is not slow, it is broken: nobody asks a follow-up question of a system that takes a minute, and follow-up questions are the entire point of building a conversational interface over a corpus.

Three things got it to roughly ten seconds. Model optimisation, better serving through vLLM, and cross-encoder re-ranking. The re-ranking had a side effect worth naming: it improved the answers rather than merely hurrying them. Retrieving a wider candidate set and then re-ranking it properly beats retrieving a narrow set and trusting the first pass, and it turns out to be cheaper than it sounds because the re-ranker only ever sees a shortlist.

The general lesson I would keep: latency work and quality work are not opposed as often as people assume. Several of the changes that made the system faster also made it more correct, because both problems had the same root cause, which was asking a large model to do work that a smaller, sharper component could do better.

Underneath is a Kubernetes cluster: networking, ephemeral and persistent storage, vLLM serving, Ingress, database configuration and autoscaling. Access goes through Keycloak, because a platform serving several institutions needs identity handled by something that already knows how, not by application code.

Ingestion is event driven. A document lands in MinIO, a pipeline wakes up, and the document is folded into the knowledge base without anyone running a script. That property matters more over time than any single model choice: a corpus nobody has to maintain by hand is a corpus that stays current, and a knowledge platform that goes stale is worse than no platform at all, because people keep trusting it.

Response tracing and dashboards through Langfuse, evaluation through TruLens, and user feedback fed back into fine-tuning. Administrators can see real sessions and real queries rather than a benchmark.

I would argue this is the least glamorous and most load-bearing part of the whole system. Retrieval quality is not one number, and a benchmark score on a public dataset tells you very little about a corpus of European climate policy written across four languages and thirty years. The only honest signal is what real users asked and whether they got something they could use.

Spend your first month on ingestion, not on the model. The quality ceiling of the whole system is set by how well you read the documents, and no amount of prompt engineering recovers a table you never extracted.

Build the evaluation loop before you need it. You cannot reason about whether a change helped if the only instrument you have is your own impression of ten queries you happen to remember.

And decide early who is asking. Two reading levels and four languages sound like a feature list. They are actually a constraint on retrieval, on chunking, and on what a good answer even means, and they are much cheaper to design for at the start than to add at the end.