Choosing a groundedness metric when no leaderboard covers your corpus, every candidate fails your gates, and the bias you ship has to be documented instead of hidden.
I was two days from rewriting a working system to fix a deficit that did not exist. Cosine similarity was scoring vocabulary reuse, not truth.
My ship gate came back 0 for 14, three passes, zero variance. The thresholds existed only as text inside a prompt, and the LLM reading them could not do arithmetic.
A mocked unit test passed on every commit while the live Cohere call failed on every run. What that gap cost, and the three-layer methodology that closes it.
One day of evaluation produced three null results. Every one was a measurement artifact, and each nearly became a documented fact.
A grid search across 15 RAG configs revealed that chunk size matters more than embedding model, overlap is not optional, and bigger parameters don't mean better recall.
How Instructor, flat schemas, and two-phase validation got me to 100% structured output success across 580 LLM-generated records.
Grid search across 16 RAG configurations reveals embedding model selection drives 26% more retrieval quality than chunk tuning.
My first LLM judge had a 0% failure rate. That meant it was useless. This is the story of calibrating it to actually catch failures, and building a correction loop that took synthetic data failures from 36 to zero.