noah jobse

Building a RAG agent from scratch, part 2: embeddings

#rag-from-scratch #rag #embeddings

Where we are

Part 1 was chunking, cutting my notes into pieces. Part 2 is embeddings: turning each of those pieces into a vector so the computer can search by meaning instead of keywords. I figured this part would be quick. Pick the best model, wire it up, move on. It turned into the most surprising thing I've built so far, and the lesson wasn't the one I expected.

What an embedding actually is

An embedding is a list of numbers that places a chunk of text as a point in space. The model is trained so that text with similar meaning lands at nearby points. "deposit held in trust" and "escrow rules" end up close together; "deposit" and "pizza recipe" end up far apart. That's the whole trick that lets you search by meaning.

The one detail that matters later: I have the model return every vector scaled to length 1 (normalized). Once everything's the same length, measuring how similar two chunks are is a plain dot product, no division, no fuss. It also means the whole search later is one matrix multiply, which is the part I actually wanted to write by hand.

In code the wrapper is barely anything. Load the model once, hand back normalized vectors:

class Embedder:
    def __init__(self, model_name="BAAI/bge-small-en-v1.5"):
        self._model = SentenceTransformer(model_name)   # load once, reuse

    def encode(self, texts):
        vecs = self._model.encode(texts, normalize_embeddings=True,
                                  convert_to_numpy=True)
        return vecs.astype("float32")   # length-1 vectors, ready for dot-product search

Running it on my own machine

No API. The embedding model runs locally on my laptop GPU, a little 6GB card. That's part of the point of this project: my notes never leave my machine, and I'm not paying per token to understand my own data. It also set up the mistake that makes this post worth reading.

I didn't want to trust a leaderboard

There are public benchmarks that rank embedding models. The problem is they're scored on generic datasets, and I don't care how a model does on generic data. I care how it does on my notes, with questions phrased the way I'd actually ask them.

So instead of picking a model off a chart, I built a bake-off. Embed my entire brain with five different models, ask a set of real questions, and measure which one actually pulls back the right note. The five:

  • all-MiniLM-L6-v2 (tiny, the old baseline)
  • bge-small (my default, 384 numbers per chunk)
  • bge-base (bigger sibling, 768)
  • Qwen3-Embedding-0.6B (newer, 1024)
  • gte-Qwen2-1.5B (the big one, 1536)

The questions are drawn from my own private notes, so I'm not going to paste them here. Think "what did we decide about X," "who's my contact at Y," "what's the number for Z." For each one I noted which single note actually answers it, so I could check whether that note came back.

How I scored it

Two numbers, both simple once you see them.

Recall@5: did the right note show up anywhere in the top 5 results? It's pass/fail per question. If I ask 8 questions and the right note lands in the top 5 for 3 of them, recall@5 is 3/8. It doesn't care whether the note was ranked 1st or 5th, only that it was there.

MRR (mean reciprocal rank): this one cares about position. For each question you take 1 divided by the rank of the right note (1st = 1.0, 2nd = 0.5, 3rd = 0.33), and average those. It rewards putting the answer at the very top, and barely distinguishes 4th from 5th, which matches how retrieval actually gets used. Recall says "found it," MRR says "found it, and how close to first."

I wrote the metric myself. It's about ten lines. The input is one list of booleans per question, True wherever the right note showed up in the top 5:

def score(per_query_hits, k):
    recall_count = 0
    mrr = 0.0
    for row in per_query_hits:          # one row per question: [False, True, False, ...]
        if any(row):                    # right note landed somewhere in the top k
            recall_count += 1           # recall: presence only
            mrr += 1 / (row.index(True) + 1)   # mrr: 1 / rank of the first hit
    n = len(per_query_hits)
    return {"recall@k": recall_count / n, "mrr": mrr / n}

The results

Eight questions, five models. Here's what came back:

model                 recall@5   MRR       time
-----------------------------------------------
all-MiniLM-L6-v2           1/8   0.13       15s
bge-small                  3/8   0.21       55s
bge-base                   2/8   0.19    2.6 min
Qwen3-Embedding-0.6B       2/8   0.17     63 min
gte-Qwen2-1.5B             3/8   0.38   9 hours (!)

Two things jumped out, and neither was "here's the winner."

Surprise 1: the nine-hour model

See that 9 hours on the bottom row? The 1.5B model technically fit in my 6GB of VRAM, it sat right at the ceiling, about 5.8 of 6.1 gigs. On Windows and WSL, when the GPU's memory is that full, the driver quietly starts spilling data into regular system RAM and shuffling it back and forth over the bus. That's roughly a hundred times slower than staying on the GPU. So the card showed 100% busy the whole time, but most of that was it waiting on memory, not computing. A job that needed about three minutes of actual math took nine hours of thrashing.

Windows Task Manager showing the RTX 4050 at 100% utilization, 81C, Dedicated GPU memory at 5.7 of 6.0 GB, and Shared GPU memory at 4.8 of 15.8 GB

Here it is mid-run. Dedicated GPU memory pinned at 5.7/6.0 GB, so the card is full. The row that matters is Shared GPU memory: 4.8 GB of the model sitting in regular system RAM instead of on the GPU. That spillover, shuffled back and forth over the bus, is the entire nine hours. Utilization reads 100%, which is what fooled me, but it was 100% busy waiting on slow memory, not computing.

I didn't catch it while it ran. I saw "GPU at 100%" and assumed it was working hard. It was working hard at the wrong thing. The tell was the memory sitting pinned at the ceiling the entire time.

Surprise 2: the model barely mattered

Look at the recall column. The best score, across all five models, was 3 out of 8. The free 384-number model (bge-small) tied the giant 1.5B model that took nine hours. The tiny old baseline found 1 of 8. On one easy question the small models got it and both big Qwen models missed it entirely.

I went in expecting to crown a model. What I found is that for my notes, the model isn't the thing holding retrieval back. They all landed somewhere between 1 and 3 out of 8, and swapping a small one for a model 40 times its size bought me essentially nothing on recall.

One honest caveat, and I owe it to a comment on the post: recall only tells half the story. bge-small and the 1.5B both found 3 of 8, but the big model dropped its hits at rank 1 while bge-small left them sitting around rank 5, so its MRR was nearly double. The model does matter for ordering, even if it barely moves how many answers come back. I'm still keeping bge-small, because for a RAG agent recall is the ceiling: if the right note never comes back at all, good ordering can't save it. Fair catch though, and the ranking gap is one I'll watch once reranking is in.

What I actually took away

The embedder isn't the bottleneck. If the best model only finds the right note 3 times out of 8, the fix isn't a fancier embedder. It's the stuff around it: better chunks, a reranking step, searching more than the top 5, maybe mixing in keyword search. That's part 3.

Test on your own data. A leaderboard would have told me to reach for the big model. On my actual notes it was a nine-hour tie with a model I can run in under a minute.

Watch the memory ceiling, not the utilization. "GPU at 100%" told me nothing useful. "GPU memory pinned at the limit" was the real signal, and I'd have saved most of a day if I'd read it right.

A fair caveat: this is 8 scrappy questions, so the exact numbers are directional, not a verdict. I'm not publishing a model ranking off that. But the shape of it (everything's mediocre, the big model gained nothing for its cost, the memory trap is real) is solid enough to act on. I'm keeping bge-small as the default and moving on.

Part 3 is retrieval: actually searching these vectors, and trying to drag that "3 out of 8" up to something I'd trust.

Building this in public, one piece at a time, by hand. I'm writing every part myself and I don't ship anything I can't explain. Follow along if you want to watch a RAG agent get built from the ground up, mistakes and all.

← all posts