Building a RAG agent from scratch, part 5: eval, and how I actually know it works

Where we are

Over four parts I built the whole thing: chunk the notes, embed them, retrieve the right ones, and an agent loop that answers with citations. Which leaves the question that should have been there from the start. Is any of it good?

That question is eval, and it turned out to be the quiet main character of the entire series. Every surprising moment, the chunker that passed its tests and still produced garbage, the nine-hour model that tied a free one, the "broken" retrieval that was actually a broken test, came from measuring something, not from building it. So the last part is the skill the whole project kept forcing on me.

The metric

Eval sounds heavy. Mine is two numbers.

I wrote a set of questions I'd actually ask my notes, and for each one I labeled the note that answers it. Then, for any version of the system, I ask all the questions and check two things:

• Recall@5: did the right note show up in the top 5 results? Pass or fail per question. If 10 of 12 questions find their note, recall@5 is 10/12.
• MRR: how high did it land? First place scores 1.0, second 0.5, third 0.33, averaged across questions. Recall says "found it," MRR says "found it, and near the top."

That's the whole instrument. A list of questions, a known right answer for each, and two numbers. It fits in about ten lines of code, and it's the thing that told me the truth every time my intuition was wrong.

The ruler has to be honest

The metric is the easy part. The work is building a test that measures the real thing instead of a convenient one.

Part 3 told the long version of how I learned this, so here's the short one. My first eval said retrieval was failing badly. Before I spent a week "fixing" it, I traced each miss, and most of them weren't retrieval failures at all. My questions were phrased to dodge the notes' own words, which made them harder than anything I'd really ask. And each question accepted exactly one correct note, so when the system returned a different note that also answered, I scored it as a miss. I'd built a ruler that was bent in two places.

I rebuilt it with realistic questions and with every note that genuinely answers marked as acceptable. Same system. The score went from looking broken to looking good. The lesson I'd put above every other one in this series: before you improve the thing, make sure the thing measuring it is honest. A bad ruler misleads you, and worse, sends you confidently building the wrong fix.

Eval is what settled every argument

Once the ruler was trustworthy, it ended debates that would otherwise have been vibes.

Should I add hybrid keyword search? I'd assumed yes. The numbers said it didn't beat plain keyword on my notes, so no. Does reranking help? It helped on easy questions and actively hurt on hard ones, which I'd never have guessed. Does HyDE close the gap when a question is worded differently from the note? Yes, measurably. And when a founder and CTO left a comment suggesting I pool several models and separate "did we find the note" from "did we pick it," I didn't have to argue about whether he was right. I ran it. The numbers showed his approach beating everything I'd built on my own, so that's what I shipped.

None of those calls were made by taste. Each one was a number moving, or not moving, on an honest test. That's what eval is for.

What eval can't do

One honest limit. Eval tells you where a problem is, not always how to fix it. One note in my test misses on every method, even pooled. When I traced it, the answer simply isn't sitting in any single retrievable chunk, it's implied across the note rather than stated in one place. No retriever or reranker can select a note that was never cleanly findable. Eval pointed at the exact failure and told me it lives in chunking and content, not retrieval. The fix is a different problem. Knowing which problem you have is most of the value.

The quiet main character

That's why I keep calling eval the main character. It isn't glamorous, and it's the one thing that separated what actually worked from what I assumed would. Build by feel and you get a pile of plausible choices. Measure, honestly, and the right ones announce themselves.

That's the series: five parts, every piece built by hand, nothing I can't explain. I pulled the big lessons from the whole thing, chunking, models, retrieval, the agent loop, and eval, into one short recap, if you want the whole arc in a single read: https://noahjobse.dev/blog/building-a-rag-agent-from-scratch-what-i-learned

Built this in public, one piece at a time, by hand. I wrote every part myself and didn't ship anything I couldn't explain. The code is open source if you want to run it on your own notes.