noah jobse

What I learned building a RAG agent from scratch

#rag #llm #lessons

I spent a few weeks building a retrieval-augmented-generation agent over my own notes, entirely by hand, no LangChain, no LlamaIndex, no vector database. I wrote it up as a five-part series as I went, mistakes and all. This is the whole thing in one place: what I built, what I got wrong, and the handful of lessons that actually stuck.

The reason I did it this way: I'd used RAG and agent tooling plenty and still couldn't have told you how any of it worked. That's the trap. Lean on a framework, get something that runs, never learn what's underneath. I wanted the opposite. Every piece built by hand and actually understood.

What it is

A tool where I ask my notes a question in plain English and get a cited answer. The pipeline, all hand-written:

notes -> chunk -> embed locally -> store (SQLite) -> retrieve -> agent loop -> cited answer

Chunk each note into sentence-aware pieces. Embed them with local models so the notes never leave my machine. Store the vectors in a SQLite file. Retrieve by pooling several models for candidates and letting an LLM pick the best. Then an agent loop where the model decides when to search and answers with sources. It's open source: [github.com/noahjobse/brainbot-rag](https://github.com/noahjobse/brainbot-rag).

The five parts, if you want the detail:

  1. [Chunking](https://noahjobse.dev/blog/building-a-rag-agent-from-scratch-part-1-chunking)
  2. [Embeddings](https://noahjobse.dev/blog/building-a-rag-agent-from-scratch-part-2-embeddings)
  3. [Retrieval, and the eval that lied to me](https://noahjobse.dev/blog/building-a-rag-agent-from-scratch-part-3-retrieval)
  4. [The agent loop](https://noahjobse.dev/blog/building-a-rag-agent-from-scratch-part-4-agent-loop)
  5. [Eval, and how I know it works](https://noahjobse.dev/blog/building-a-rag-agent-from-scratch-part-5-eval)

The five lessons that stuck

1. Chunking is the underrated lever. The boring plumbing step, slicing notes into pieces, moved quality more than anything else early on. My first chunker passed every test I wrote and still produced garbage, because it kept cutting sentences in half. Passing tests told me the code did what I said. It told me nothing about whether the output was any good.

2. Test on your own data, not a leaderboard. I benchmarked five embedding models on my actual notes instead of trusting a public ranking. The biggest model took nine hours (it quietly spilled past my GPU's memory and thrashed) and tied a free one I run in under a minute. A leaderboard would have sent me straight to the expensive wrong answer.

3. The popular upgrades are text-matching underneath. Hybrid search and reranking are the two things everyone reaches for. On my notes, hybrid didn't beat plain keyword, and reranking helped on easy questions and actively hurt on hard ones. They all match text, so none of them fix the real hard case: asking about something using different words than your note uses. Only transforming the query or using an actual language model touched that.

4. Discovery and selection are different problems. Finding the right note and picking it back out of a pile fail for different reasons and want different fixes. Pooling several models made discovery nearly free (the right note was almost always somewhere in the candidates). The bottleneck moved to selection, and a small reranker was bad at it. Swapping in an LLM to choose fixed it. I only saw this because I measured the two stages separately.

5. Measure before you optimize, and check your ruler first. My biggest "retrieval problem" turned out to be a broken test: unfair questions and over-strict labels made a working system look broken. Fixing the ruler, not the system, took it from looking like a failure to looking good, and nearly saved me a week of optimizing the wrong thing. The highest-leverage work I did all series was on the evaluation, not the code.

Two things underneath all of it

Building is not understanding. I could use RAG before this. I could not explain it. Writing every piece by hand is slower and it's the only thing that actually closed the gap. The parts I'd have let a framework handle, the search math, the agent loop, the eval, are exactly the parts I now understand.

The best improvement came from a comment I could test. A founder and CTO left a note on one of the posts suggesting I pool models and track discovery separately from selection. I'd gone the long way and reinvented a worse version of it on my own. Because I had an honest eval, I didn't have to argue about whether he was right. I ran his experiment, the numbers beat everything I'd built, and that's what shipped. An honest test turns advice into something you can verify instead of debate.

What's next

One thread stays open on brainbot. That single note that misses every method has its answer implied across the whole thing, never written in one place, so there's nothing clean to retrieve. The fix lives on the content side: a short summary written at index time that states out loud what the note only implies, with a pointer back to the original as the source of truth. Retrieve the summary, load the original as evidence. That's the next experiment there, and like the retrieval win, it came from a comment on these posts.

Same approach, harder target. The next thing I'm building from scratch is a small LLM inference server: load a model, write the token-by-token generation loop by hand, add a KV cache, batch requests, and measure throughput and tail latency under load. Building AI infrastructure from the ground up, the same way, piece by piece, measured, in public.

If you've got a folder of markdown notes, the agent runs on them: [github.com/noahjobse/brainbot-rag](https://github.com/noahjobse/brainbot-rag). Thanks for following along.

Built this in public, one piece at a time, by hand. I wrote every part myself and didn't ship anything I couldn't explain.

← all posts