Every RAG demo has a great accuracy number. Almost none of those numbers mean anything three weeks after launch, because they were measured on the questions the team thought users would ask, not the ones users actually ask.
The fix isn't a better benchmark before launch. It's a harness that keeps scoring the system on real traffic after launch, and tells you which of the three RAG failure modes you're in when the score moves.
Separate the three failures
A wrong answer has exactly three causes: the right passage wasn't retrieved, the right passage was retrieved but the model ignored it, or the source itself is wrong or stale. Score retrieval and generation separately, or every incident review turns into guesswork. Retrieval gets recall-at-k against labeled passages; generation gets faithfulness against the retrieved context; the corpus gets a freshness audit on a schedule.
Build the eval set from traffic, not imagination
Seed with 50 hand-written questions if you must, but the standing rule is: every escalation, every thumbs-down, every support ticket that names the assistant becomes an eval case within the week. Six weeks after launch the majority of your set should be questions no one on the team predicted. That's the point.
Pin the judge, version the set
If an LLM grades your answers, pin its model version and prompt, and re-baseline deliberately when you change either — otherwise judge drift reads as product regression. Version the eval set like code: a score only means something against a named revision of the questions.
Wire it to releases
The harness runs on every change to prompts, chunking, embeddings, or model version, and the release blocks when the score drops past a threshold you chose while calm. A nightly run on sampled production traffic catches the slow drifts: index staleness, corpus rot, traffic shifting toward topics you never covered.
None of this is exotic. It's a few hundred lines of harness code and a discipline about where eval cases come from. It's also the difference between an incident review that ends with a fix and one that ends with a shrug.
If you are weighing whether your corpus can support the feature at all, a diagnostic sprint answers it in a week: we run the hardest examples against your real documents and write down what we find.