noah jobse

Building a RAG agent from scratch, part 4: the agent loop

#rag-from-scratch #rag #agents

Where we are

Parts 1 through 3 built the retrieval half: chunk the notes, embed them, and search to pull back the right ones. That gives me a function that takes a question and returns relevant note excerpts. Useful, and still just a search box. Part 4 is the piece that makes it an agent: a loop where a model decides when to search, reads what comes back, and answers in plain language with citations.

This is the part I built the whole project to understand. I'd used agents plenty and still couldn't have told you what the "loop" actually was. Turns out it's about twelve lines.

What makes it an "agent"

The naive version is: I take your question, run the search myself, paste the top results into a prompt, and ask the model to answer. That works, and for a lot of cases it's all you need. But the model is passive. It answers from whatever I happened to hand it, and if my single search missed, it's stuck.

The agent version flips who's in control. I give the model a tool, a function it can choose to call, and let it drive. It decides whether to search, what to search for, whether the results are good enough, and whether to search again with a better query before answering. The model runs the show; my code just executes the tool calls it asks for.

That shift, from "I feed the model context" to "the model asks for context," is the whole idea of an agent.

One tool

I give it exactly one tool, a read-only search:

search_brain(query) -> the most relevant note excerpts, each tagged with its source file

Behind that tool is the retrieval from part 3 (pool a few embedding models for candidates, let an LLM pick the best). The agent doesn't know or care how retrieval works. It knows there's a tool that takes a query and returns notes. That seam matters: I can change retrieval completely without touching the loop.

The loop

Diagram of the agent loop: your question goes to the model, the model calls search_brain and reads the note excerpts that come back, loops to search again if the results are thin, then answers with citations once it has enough.

Here's the heart of it, trimmed to the shape:

messages = [{"role": "user", "content": question}]

while True:
    resp = client.messages.create(model=MODEL, tools=[SEARCH_TOOL], messages=messages)

    if resp.stop_reason != "tool_use":        # the model answered, we're done
        return text_of(resp)

    # the model asked to search: run it, hand the results back, loop
    messages.append({"role": "assistant", "content": resp.content})
    results = [run_search(b.input["query"])
               for b in resp.content if b.type == "tool_use"]
    messages.append({"role": "user", "content": results})

Read it as a conversation that repeats. Each time around, I send the whole conversation so far. The model replies one of two ways. If it answered, I return the text and stop. If it asked to search, I run the search, add the results to the conversation as if they were the next message, and go around again. It keeps going until the model has what it needs and answers.

That's it. That's the loop everyone means when they say "agent loop." A framework would hide this inside a .run() call. Writing it by hand is the point: now I can see exactly where a tool gets called, where results go back in, and what makes the model stop.

What the loop buys you

Because the model drives, it can do things a single search can't:

  • Re-search. If the first results are thin, it rephrases and tries again before answering, inside the same loop, no extra code from me.
  • Say "I don't know." A line in the system prompt tells it to answer only from the notes and to say so plainly when they don't contain the answer. That turns out to matter more than any retrieval trick, because a confident wrong answer is worse than "not in your notes."
  • Cite. Every excerpt comes tagged with its source file, and the prompt tells the model to cite the source for each fact. So the answer comes back with the filenames it leaned on, and I can check it.

The part tutorials skip: hardening

The twelve-line loop is the demo. Making it something I'd actually run every day took a few unglamorous guards:

  • A cap on rounds. while True with a model that keeps calling the tool is an infinite loop and a runaway bill. A hard limit on search rounds fixes that.
  • Handling a weird response. If the model ends its turn without emitting any text (it happens), the naive "grab the text" line crashes. Pull the text defensively and return a clear fallback instead.
  • A tool failure not killing the answer. If a search throws, I feed the error back to the model as the tool result instead of letting it blow up the whole request. The model can then recover or say it couldn't look that up.

None of it is clever, and all of it is the difference between a thing that demos and a thing that works when the inputs get ugly.

What I took away

  • The agent loop is small. The thing I'd treated as a black box is a while loop with a stop condition. Once I'd written it, "agent" stopped being a mystery word.
  • The tool seam is the good design. The loop only knows "a tool that returns notes." That let me swap retrieval strategies all through part 3 without touching the loop once, and it let me test the loop with a fake search, no model or real notes needed.
  • Hardening is most of the real work. The happy path is a dozen lines. The guards around failure, runaway loops, and "I don't know" are what make it usable.

Part 5 is eval: how I actually know any of this works, which, it turns out, was the quiet main character of the whole series.

Building this in public, one piece at a time, by hand. I write every part myself and I don't ship anything I can't explain. Follow along if you want to watch a RAG agent get built from the ground up, mistakes and all.

← all posts