noah jobse

Building a RAG agent from scratch, part 1: chunking

#rag-from-scratch #rag #python

Why I'm doing this

I'm building a little RAG agent, I'm calling it brainbot, that answers questions over my own notes. A few hundred markdown files I've piled up over the years. The catch, and the whole point, is there's no framework. No LangChain, no LlamaIndex. I'm writing the chunking, the embeddings, the search, the agent loop, and the eval myself.

Why? Because I'd used RAG and agent tooling a bunch and still couldn't tell you how any of it actually worked. That's the trap I wanted out of. So much of what gets shipped now is slop, code you pasted in, that happens to run, that you couldn't explain if someone put you on the spot. This is the opposite. Every piece is built by hand and I actually understand it. I don't move on from a part until I can tell you why every line is there, how it flows, and what breaks it. Frameworks are great at making something run and terrible at making you understand it, so the parts that actually matter, I'm writing myself.

It's a series, built in public. The plan:

  1. Chunking (this one)
  2. Embeddings, running on my own machine
  3. Retrieval and vector search
  4. The agent loop
  5. Eval, how you actually know it works

I only lean on a library for the stuff that isn't the point (the embedding model, the API SDK). Anything that teaches me something, I write.

Part 1 is chunking. First real decision you hit, and the one I totally underrated.

Why chunking is the decision that matters

Chunking is how you cut your docs into the pieces you embed and search over. It looks like boring plumbing, and it's the single biggest lever in the whole pipeline.

Here's what I didn't get at first: retrieval can only ever hand you back a chunk. A chunk is the smallest thing the system can pull up. So whatever you decide here sets the ceiling for everything after it. No fancy embedding model or reranker is going to save a fact you cut in half, or pull a clean answer out of a chunk that smushed ten different topics into one blurry average.

It all comes down to size, and it pulls two ways:

  • Too big, and the chunk's embedding is an average of too many ideas. It matches everything a little and nothing well.
  • Too small, and you've got a fragment with no context to answer anything.

What you want is one idea per chunk, sitting safely under the embedder's token limit.

There's a ladder of ways to do this

There's a whole spectrum of ways to chunk, from dumb to smart:

  • Fixed-size: cut every N tokens. Simple, ignores meaning.
  • Structure-aware: split on paragraphs, sections, headings.
  • Sentence-aware: never cut mid-sentence.
  • Semantic: start a new chunk when the topic actually changes.
  • Document-aware: keep tables, code blocks, and lists in one piece.

The higher you go, the better the chunks and the more it costs you. My rule was start dumb, only climb a rung when I could prove I needed to.

V1: the obvious version

My first chunker was the obvious thing. Split on blank lines into paragraphs, pack them up to a word budget, carry a bit of overlap so an idea sitting on a boundary doesn't get lost, and hard-split any paragraph too big to fit.

Passed every test I wrote. No chunk over the limit, no words dropped, overlap there. All green.

And that's the trap. Passing tests tells you the code does what you told it to. It tells you nothing about whether what comes out is any good.

Quick tangent: words aren't tokens

My budget was in words. But the embedder's limit is in tokens, and English runs about 1.3 tokens per word. So a chunk that's comfortably under my word budget can blow right past the model's token limit, and when that happens the model just quietly chops off the end. The tail of that note is gone. Invisible to search, forever.

No error. No warning. Just data silently disappearing. I fixed it by setting the word budget well under the token limit, with a note that the real fix is counting actual tokens. It's the kind of bug you only catch if you go looking, which is kind of the whole theme here.

Testing it on real data

I ran V1 on my actual notes and a messy meeting transcript. The chunks were, honestly, hard to judge by eye. Some looked great. Some started mid-sentence. One was just a file header. The transcript ones were full of timestamp junk.

"Looks fine" isn't a measurement. I couldn't fix what I couldn't see.

Scoring it

So I built a deterministic quality score. One number, 0 to 100, from four cheap signals I could compute without embedding anything:

  • size: are the chunks a reasonable size
  • redundancy: how much the overlap is duplicating
  • boundary: do chunks end on a sentence or get cut mid-thought
  • junk: how many tiny, useless chunks

Deterministic is the point. Same input, same score, so it's a clean before-and-after. I ran it on the whole thing.

brainbot chunk report scoring 85 out of 100, boundary at 41%

85 out of 100. Not bad. But look at boundary: 41%. Most of my chunks were ending in the middle of a sentence. That's the dilution problem for real, a chunk that stops at "the deposit must be held in a separate" is missing the half that says "trust account within three days."

Best part, the score prints suggestions, so it straight up told me which lever to pull.

V2: sentence-aware splitting

The fix was to quit cutting at some random word number and make the sentence the unit instead. Split into sentences, pack whole sentences up to the budget, overlap by whole sentences too. A sentence never gets split across two chunks, unless it's some monster run-on, and even then it falls back to a word split so the size limit still holds.

The sentence splitter is a little regex:

re.split(r"(?<=[.!?])\s+", text)

It splits on whitespace that comes after a ., !, or ?, so each sentence keeps its punctuation. It's naive (trips on "e.g." and "Dr."), but good enough to start, and I wrote the limitation down instead of pretending I'd solved it. That's one rung up the ladder, structure-aware to sentence-aware.

Here's the whole function. It's about forty lines, you can read the whole thing top to bottom:

def split_sentences(text):
    parts = re.split(r"(?<=[.!?])\s+", text)
    return [" ".join(p.split()) for p in parts if p.strip()]


def chunk_markdown(text, source, max_tokens=300, overlap=45):
    text = text.replace("\r\n", "\n").replace("\r", "\n")
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    if not paragraphs:
        return []

    sentences = []
    for para in paragraphs:
        sentences += split_sentences(para)

    chunks = []    # finished chunks, as strings
    current = []   # the sentences in the chunk being built

    def wc(sents):
        return len(" ".join(sents).split())

    for sent in sentences:
        sw = len(sent.split())

        # a single sentence bigger than the budget: fall back to a word split
        if sw > max_tokens:
            if current:
                chunks.append(" ".join(current))
                current = []
            words = sent.split()
            start = 0
            while start < len(words):
                chunks.append(" ".join(words[start:start + max_tokens]))
                if start + max_tokens >= len(words):
                    break
                start += max_tokens - overlap
            continue

        # does the whole sentence still fit in the current chunk?
        if wc(current) + sw <= max_tokens:
            current.append(sent)
        else:
            # close the chunk, seed the next one with trailing whole sentences as overlap
            chunks.append(" ".join(current))
            keep = []
            for s in reversed(current):
                if wc(keep) >= overlap:
                    break
                keep.insert(0, s)
            current = keep + [sent] if wc(keep) + sw <= max_tokens else [sent]

    if current:
        chunks.append(" ".join(current))

    return [Chunk(t, source, i) for i, t in enumerate(chunks)]

Then I ran the score again.

brainbot chunk report scoring 100 out of 100, boundary at 73%

Boundary jumped from 41% to 73%. Redundancy settled into the healthy range. Score hit 100.

What I actually took away

Two things stuck with me.

One, these metrics are proxies. They catch the obvious garbage, but the real test of chunking is downstream: does the right chunk actually come back when I ask a real question? That's recall, and I can't measure it until the retrieval half exists. So I treat the score as a regression check I keep an eye on while I build. Hitting 100 just means the smoke test passed, the real test comes later.

Two, the gap between "my tests pass" and "this is actually good" is where all the real work hides. The score is what turned a vague "eh, seems fine" into "boundary's at 41%, go fix it."

Part 2 is next: embeddings. Turning these chunks into vectors on my own machine, and figuring out what "similar" even means to a model.

Building this in public, one piece at a time, by hand. I'm writing every part myself and I don't ship anything I can't explain. Follow along if you want to watch a RAG agent get built from the ground up, mistakes and all.

← all posts