noah jobse

Building a RAG agent from scratch, part 3: retrieval, and the eval that lied to me

#rag-from-scratch #rag #retrieval
A marble statue slumped forward with its face buried in one hand, the classic facepalm pose

Roughly how most of this one went.

Where we are

Part 1 cut my notes into chunks. Part 2 turned each chunk into a vector, a list of numbers that places it in meaning-space. Part 3 is retrieval: actually searching those vectors to pull back the right note for a question. I thought this would be the short one. The search itself is short, about four lines. Then I tried to find out whether it was any good, and spent far longer than I planned learning that the hardest part of retrieval is knowing whether your search is working at all.

The search is four lines

Here's the whole thing. Every note-vector is stacked into one matrix. A question comes in as its own vector. You multiply, and you sort:

def search(self, q, k=5):
    matrix, metas = self.load_all()   # every note-vector, stacked into one array
    scores = matrix @ q               # one dot product per note, all at once
    top = np.argsort(scores)[::-1][:k]
    return [metas[i] for i in top]

That matrix @ q scores every chunk in my brain against the question in a single operation. Because every vector was scaled to length 1 back in part 2, that dot product is the cosine similarity directly, no division, no fuss. Sort descending, take the top few, done.

That's worth sitting with: people reach for a dedicated "vector database" to do this, and for a few thousand notes it is genuinely a matrix multiply and a sort. I wrote it by hand so I'd know exactly what those products are doing under the hood. No framework, no black box.

And it worked. I asked it a real question and the right note came back at the top. Great. Ship it?

Is it any good?

"The right note came back" for one question I happened to try is not evidence. So I built a tiny evaluation: a set of questions I'd actually ask, each labeled with the note that answers it. Run every question, look at the top 5 results, count how often the right note is in there. That fraction is recall@5. I also tracked MRR, which cares about where in the top 5 it landed (part 2 covered that one).

I ran it. Recall@5 came back at 3 out of 8. Not good. The right note was missing most of the time.

So I started reaching for fixes. Better search, reranking, combining methods. And that's exactly where I almost wasted a week.

My eval was lying to me

Before optimizing against a number, I made myself trace why each miss happened, one question at a time: where did the right note actually rank, and what beat it. The answer was uncomfortable. A lot of my "misses" were not retrieval failures at all.

Two problems, both in the test, not the search:

  1. The questions were unfair. I had deliberately phrased them to avoid the notes' own words, to stress-test the embeddings. Nobody actually asks questions that way. It made the test far harder than real use.
  2. The labels were too strict. Each question accepted exactly one "correct" note. But some facts legitimately live in two or three notes. Retrieving a perfectly good alternative got scored as a miss.

I rebuilt the test: realistic questions, phrased the way I'd really type them, and every note that genuinely answers a question marked as acceptable. Same search, same notes. Recall@5 went from 3 out of 8 to 10 out of 12.

My ruler was bent, and it was about to send me optimizing in the wrong direction. The search had been fine the whole time. If there's one lesson in this whole post, it's this: before you improve the thing, make sure the thing measuring it is honest.

Testing the other way: does phrasing break it?

Now I had the opposite worry. Realistic questions share words with the notes, which is easy mode. What happens when I ask about the same fact using different words than the note uses?

So I made a second test: the same facts, re-asked with synonyms, describing things instead of naming them, sharing as few words with the notes as I could manage. Then I ran plain keyword search (match the actual words) next to meaning search (match the vectors) on both sets:

                   literal questions   paraphrased questions
keyword search          11/12                1/12
meaning search          10/12                4/12

There it is. When the question uses the note's words, dumb keyword search wins. When it doesn't, keyword falls off a cliff, it has no words to match, and meaning search is the only thing left standing. That gap is the entire reason embeddings exist: they match on concept, so they survive you not knowing the magic words. It also means keyword search is a criminally strong, cheap baseline that most people skip straight past. Both things are true.

The paraphrase set became my real target. Meaning search at 4 out of 12 had room to grow, and that's the case embeddings are supposed to be good at.

The obvious fixes that didn't work

Two upgrades everyone reaches for:

Hybrid search (keyword and vectors combined). I tested every sensible way to merge the two ranked lists. The naive merge was worse than either alone, it rewards notes both methods mildly like, and buries the note one method loves. The best merge barely tied keyword. On a small corpus full of distinctive names, hybrid did not earn its complexity.

Reranking (retrieve a wide net, then use a heavier model to reorder the top candidates). On the literal questions it helped, nudged me from 10 to 11. On the paraphrased ones it made things worse, dropping me from 5 to 3.

That second result took me a minute to understand, and then it clicked: a reranker is still a text-matching model, it just reads the question and chunk together instead of as separate vectors. So when the words match, it confirms the match nicely. When they don't, it suffers the same vocabulary gap, and worse, it reorders confidently, shoving the subtly-right answer down in favor of a surface-similar wrong one. Reranking amplifies whatever the vocabulary situation already is. Helpful when words line up, actively harmful when they don't.

Both of these are, at heart, matching text. My hard problem was text that doesn't match. No amount of better text-matching fixes that.

HyDE: translate the question first

The one technique aimed at the actual root cause is HyDE: hypothetical document embeddings. Instead of searching with the question, you ask an LLM to write a short fake answer to it first, then search with that.

It sounds backwards until you see why it works. My paraphrased question uses my words. The fake answer the model writes uses the domain's words, the same vocabulary the real note is likely written in, even if the invented facts are wrong. So you stop matching my-words against note-words and start matching note-dialect against note-dialect.

It moved the number. Paraphrase recall went from 5 out of 12 to 8, and it helped the literal set too, with no regressions. First thing that actually attacked the vocabulary gap instead of doing more matching.

I thought that was the answer. Something better was still sitting in my notifications.

The comment I missed

Back on the part 2 post, a founder and CTO named Scott Stouffer had left a comment. I'd replied to his first note, but he'd posted a follow-up I never saw until this week. His point, roughly: my different models were missing different questions, so instead of picking one, I should pool their candidates, and, crucially, I should keep two scores separate, how many right notes make it into the pooled candidate set ("discovery"), versus how many survive the final reranking step down to five ("selection"). If a note gets discovered but then disappears, the loss is in selection, and that tells you exactly where to look next.

I had gone and reinvented half of this the hard way on my own. So I ran his actual experiment.

Running his experiment

I pooled a few embedding models: take each one's top candidates for a question, union them together. Then measure the two scores he called out.

Discovery was almost solved. The pooled union contained the right note for 12 of 12 literal questions and 11 of 12 paraphrased ones. The models really do miss different things, so together they find nearly everything.

Selection was the bottleneck. When I reranked that rich pool down to five with the cross-encoder, the paraphrase score collapsed from 11 back to 5. The right notes were in the pool and the reranker was throwing them out. Scott's two-number split pointed a finger straight at the culprit: finding the notes was handled, picking them back out was now the hard part.

So I replaced the selector. Instead of the little cross-encoder, I handed the pooled candidates to an actual LLM and asked it to pick the five most relevant. An LLM reads meaning, so it shouldn't collapse on paraphrase the way a text-matcher does.

                              literal      paraphrased
pooled union (discovery)       12/12          11/12
pool + LLM pick (final)        12/12          11/12

Zero drop. The LLM selector kept everything the pool discovered: 12 out of 12, and 11 out of 12 on the hard set, beating HyDE and everything else I'd tried. Discovery by pooling, selection by an LLM. The thing a CTO sketched in a LinkedIn comment beat the best I'd built on my own.

The one it still can't find

Honesty check. On the paraphrased set, one note still misses, and it's instructive: it's not even in the pooled union. No model retrieves it, alone or combined. When I traced it, the answer simply isn't sitting in any cleanly-retrievable chunk, it's a decision that's implied across a note rather than stated in one place. That's a chunking and content problem, which sits outside retrieval's reach. No reranker, no LLM, no pooling can select a note that discovery never found. That's the real floor, and it's a different part 4 problem.

What I actually learned

  • Check your ruler before you optimize. My "3 out of 8 retrieval problem" was mostly a broken eval. Fixing the test, not the search, took me to 10 out of 12. A bad measurement will happily send you building the wrong thing for a week.
  • The popular upgrades are all text-matching. Hybrid and reranking are great when the words line up and useless (or harmful) when they don't. If your real problem is vocabulary mismatch, only transforming the query (HyDE) or reading with an actual language model touches it.
  • Discovery and selection are different problems. Pooling made finding the right note nearly free. Picking it back out was the hard part. Measuring those two separately is what located the fix, and that framing came from someone who'd done it before.
  • Take the expert comment seriously. I went the long way and reinvented a worse version of what Scott handed me in two paragraphs. The humbling, useful move was to stop, run his experiment exactly, and let the numbers settle it.

Part 4 is the agent loop: handing these retrieved notes to a model and letting it decide when to search again, and answer with citations. That part already works. But now I can measure whether what it's searching is any good, which, it turns out, was the whole game.

Building this in public, one piece at a time, by hand. I write every part myself and I don't ship anything I can't explain. Follow along if you want to watch a RAG agent get built from the ground up, mistakes and all.

← all posts