Choosing a groundedness metric when no leaderboard covers your corpus, every candidate fails your gates, and the bias you ship has to be documented instead of hidden.
I was two days from rewriting a working system to fix a deficit that did not exist. Cosine similarity was scoring vocabulary reuse, not truth.
My ship gate came back 0 for 14, three passes, zero variance. The thresholds existed only as text inside a prompt, and the LLM reading them could not do arithmetic.
A mocked unit test passed on every commit while the live Cohere call failed on every run. What that gap cost, and the three-layer methodology that closes it.
One day of evaluation produced three null results. Every one was a measurement artifact, and each nearly became a documented fact.
The same vector store comparison gave opposite answers two weeks apart. The decision wasn't really about the vectors. It was about how long the system needed to live.
A grid search across 15 RAG configs revealed that chunk size matters more than embedding model, overlap is not optional, and bigger parameters don't mean better recall.
How Instructor, flat schemas, and two-phase validation got me to 100% structured output success across 580 LLM-generated records.
What to do when your team was promised promotions or headcount that never materialized, and you're the new manager holding the bag.
A diagnostic framework for identifying what broke before you arrived and knowing whether your repair is working.