CloudSignal AI

Blog

Retrieval-augmented generation from scratch, in one file

By Ashwin Rajendraprasad Tutorials

RAG is usually taught as a stack of framework abstractions, which hides how simple it is. In one Python file (embed, retrieve, prompt) you’ll build a working retrieval system, and walk away owning the mental model of what every RAG framework is doing under the hood.

1. What RAG is actually for

You have a model that’s good at language and knows nothing about your documents: your policies, your codebase, last quarter’s report. Retrieval-augmented generation is the cheapest fix: instead of retraining the model to memorize your facts, you fetch the relevant text at query time and hand it to the model inside the prompt. The model stays frozen; your knowledge lives outside it, where you can update it by editing a file. That’s the whole pitch, and it’s why RAG is the first thing to reach for when someone says “we need the model to know our stuff.”

2. The whole pattern in one sentence

Embed your documents into vectors, embed the question the same way, retrieve the chunks whose vectors are nearest the question’s, and paste them into the prompt. Five verbs: embed, store, retrieve, augment, generate. The middle three are just arithmetic. Everything a heavyweight framework adds (connectors, caching, rerankers, agents) is scaffolding around this core. Build the core once by hand and the frameworks stop being magic; you’ll know exactly which box you’re configuring.

3. Chunking: the unglamorous step that decides quality

Before anything is embedded, your documents get split into chunks. This is the least glamorous step and the one that most often decides whether retrieval works. Too large, and a chunk dilutes the relevant sentence with noise, so its vector drifts away from any specific question. Too small, and you sever the context that made the sentence meaningful. A plain word-window with a little overlap is a fine place to start:

def chunk(text, size=500, overlap=100):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]

The overlap matters: it keeps a sentence that straddles a boundary from being cut in half and lost to both chunks. Tune size to your content later, but start here, and change it only when your evals (section 6) tell you to.

4. Embeddings and cosine search, from scratch

An embedding turns a piece of text into a vector positioned so that similar meanings sit near each other. “From scratch” here means the search is from scratch (no vector database), not that we train an embedding model; we use an open one. sentence-transformers (Reimers & Gurevych, UKP Lab) with the small all-MiniLM-L6-v2 model runs locally and is more than enough to learn on:

# pip install sentence-transformers==5.6.1 numpy==2.5.1 requests==2.34.2   # needs Python 3.12+
import numpy as np
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

def embed(texts):
    # normalize so a dot product is exactly cosine similarity
    return embedder.encode(texts, normalize_embeddings=True)

def top_k(query, chunks, chunk_vecs, k=3):
    q = embed([query])[0]
    scores = chunk_vecs @ q          # NumPy dot products == cosine, vectors are unit-length
    idx = np.argsort(-scores)[:k]
    return [(chunks[i], float(scores[i])) for i in idx]

That’s the “vector search” the whole industry is built on, in six lines of NumPy. You do not need a vector database to learn RAG, or even to run it for a few hundred chunks. A dot product against a matrix is fast. Databases pay off later, at scale.

5. The grounded prompt (and the hallucination you didn’t fix)

Retrieval found the text; now you instruct the model to answer from that text and nothing else. The instruction is doing real work: it’s the difference between “answer the question” and “answer the question using only what I gave you.” Here we send the grounded prompt to the local endpoint from the local-LLM tutorial, but RAG is vendor-neutral: swap this call for any chat API and nothing else changes.

import requests

PROMPT = """Answer the question using ONLY the context below.
If the answer is not in the context, say "I don't know."

Context:
{context}

Question: {question}
Answer:"""

def answer(question, chunks, chunk_vecs):
    hits = top_k(question, chunks, chunk_vecs, k=3)
    context = "\n\n".join(c for c, _ in hits)
    resp = requests.post(
        "http://localhost:11434/api/chat",
        json={"model": "qwen3:8b",
              "messages": [{"role": "user",
                            "content": PROMPT.format(context=context, question=question)}],
              "stream": False},
        timeout=120,
    )
    return resp.json()["message"]["content"]

Wire the pieces together and the whole system is a dozen lines of driver code:

docs = open("handbook.txt").read()
chunks = chunk(docs)
chunk_vecs = embed(chunks)
print(answer("How many vacation days do new hires get?", chunks, chunk_vecs))

One caveat before you ship this: RAG reduces hallucination, it does not eliminate it. A model handed correct context can still ignore it and answer from its own parametric memory, or blend the two. The “say I don’t know” instruction helps; it is not a guarantee. Grounding lowers the odds of a fabricated answer. It does not lock them to zero.

6. Where it breaks, and when to graduate

Three failure modes account for most bad answers, and none of them are the model’s fault:

  • Retrieval miss. The chunk containing the answer never made the top-k, so the model never saw it. Usually a chunking or embedding problem, not a generation one.
  • Stale context. Your source changed; your vectors didn’t. Re-embed when documents change, or you’ll confidently serve last month’s answer.
  • Ignored context. Covered above. The model answers around the evidence.

You cannot see any of these without measurement, so build an eval set from day one: a couple dozen real questions with known-good answers, checked every time you touch chunk size, k, or the model. This is the single habit that separates a RAG demo from a RAG system.

One more failure, on the security side, and no eval set will catch it:

  • Retrieved chunks are untrusted input. Whatever the retriever pastes into the prompt can carry instructions of its own, so a poisoned or attacker-authored document can steer the answer. That is indirect prompt injection, and the risk compounds the moment answer() is exposed as an agent tool, because the injected text then reaches something that can act on it. Control what enters the corpus, and treat retrieved text as data the model may quote rather than as commands it should follow.

Graduate to a real vector database when you outgrow a matrix in memory (thousands of chunks, metadata filtering, persistence, concurrent users). Reach for a framework when orchestration (rerankers, multi-step retrieval, tool use) becomes the bulk of your code rather than a footnote. Before either, though, the highest-leverage upgrade is usually a reranker: a second, slower model that re-scores your top-k candidates for relevance after the fast vector search narrows the field. Cheap first pass, precise second pass; it fixes more retrieval misses than a bigger database ever will. And when your “documents” are really pages (invoices, slides, scanned forms), look at multimodal RAG, which retrieves over page screenshots using multi-vector retrievers like ColPali (Faysse et al., 2024) and ColQwen2, an approach documented in Hugging Face’s 2025 survey of vision-language models. The pattern is identical to what you just built; only the embedder changes.

From here, you can make this retrieval callable by an agent, which is exactly what the first-MCP-server tutorial does, turning your answer() function into a tool a model can invoke. And when a hand-rolled loop meets a real document set — messy PDFs, contradictory sources, questions your eval never imagined — that gap is where CloudSignal builds RAG that survives contact with real documents.

Sources / further reading

Written by Ashwin Rajendraprasad for CloudSignal AI. The code above is free to reuse.