Context and memory
The model has no memory. Everything it "knows" about your task was pasted in front of it a moment ago. Context engineering is deciding what gets pasted — and what gets thrown away.
- You can budget a context window deliberately instead of stuffing it and hoping.
- You can explain retrieval (RAG) and memory as two different mechanisms that are often confused.
Every call to a model is amnesiac. It does not remember the last call, your last message, or the document it read an hour ago. If something needs to be known, it must be in the context window — the fixed token budget sent with that request.
That budget has to hold everything: the instructions, the goal, the conversation so far, the tool definitions, retrieved documents, tool results, and enough room left for the answer. It is a budget in the real sense — spending it in one place means not spending it in another.
5.1Budget it like money
A sane starting allocation for a working agent looks roughly like this. Drag the window size and watch the tokens and the per-call input cost move.
Fixed percentages are a starting point, not a law. Long conversations should shrink history and grow summary; research tasks should grow retrieval and shrink everything else.
Two effects matter more than the exact split. First, cost scales with every iteration: an eight-step task re-sends the context eight times, so a fat context is not one bill, it's eight. Second, attention dilutes: the more irrelevant text surrounds the useful text, the more often the model latches onto the wrong thing. A million-token window is a capability, not a strategy.
"Just put the whole knowledge base in the prompt" feels safe and is usually the most expensive way to be wrong. You pay for tokens you don't need, you slow every call down, and you bury the two paragraphs that mattered. Retrieve narrowly, then verify.
5.2Retrieval: fetching the right paragraphs
RAG (retrieval-augmented generation) is the standard answer to "the answer is in our documents, not in the model". The pipeline has two halves — one that runs when documents change, and one that runs on every question.
Where beginners get stuck:
- Chunking. Split by meaning (headings, sections, functions), not every 500 characters. Overlap chunks slightly so a sentence cut in half still appears whole somewhere.
- Chunk size vs. context. Small chunks retrieve precisely but lose context; large chunks carry context but match loosely. A common fix is to retrieve small, then expand to the surrounding section.
- Hybrid search. Vector similarity finds paraphrases; keyword search (BM25) finds exact identifiers, part numbers and names. Combining both beats either alone.
- Reranking. Retrieve 50 cheaply, then re-score the best 5 with a stronger model. Usually the single biggest quality win for the money.
- Citations or it didn't happen. Make the answer carry the chunk IDs it used. If the model can't cite, it isn't grounded.
5.3Three kinds of memory
# The three moves every memory system makes
write(memory, event) # capture: cheap, append-only, timestamped
results = search(memory, query) # recall: similarity + recency + importance
context = select(results, budget) # inject: only what fits, with sourcesNotice what memory is not: it is not fine-tuning, and it is not the model remembering. It is a database with a retrieval policy, plus a prompt that says "here is what you knew before". That framing keeps you honest about failure — if memory returns the wrong record, the model will confidently act on it.
5.4When the conversation outgrows the window
Long-running agents must compress. The reliable pattern is hierarchical: keep the last few turns verbatim, keep a rolling summary of everything older, and keep a structured task list that never gets summarized away because it's the actual state.
| Technique | What it does | Watch out for |
|---|---|---|
| Summarise old turns | Replace history with a compact recap | Details vanish — decisions and numbers get lost first |
| Extract, don't summarize | Pull decisions, IDs, constraints into a structured block | Needs a schema; more work up front, far more reliable |
| Keep the task list outside | State lives in code/file, not in prose | If it drifts out of sync, the agent contradicts itself |
| Retrieve on demand | Old context becomes searchable instead of resident | Adds a step and latency; the agent must know to ask |
Take the repetitive task you wrote down in module 01 and answer: what would the agent need to know at step one to do it well? Then mark each item S (stable instructions), T (this task's data) or M (memory from before). Anything you can't classify is probably something you'd be dumping in "just in case".
Your RAG assistant gives vague, half-right answers. Which change is most likely to help?
60-second recap
- No memory, only context. Whatever isn't sent in this request does not exist to the model.
- Budget deliberately: system, tools, memory, history, retrieval, answer room. Cost and attention both scale with what you stuff in.
- RAG is a search problem. Chunking, hybrid search, reranking and citations do the work; the model writes it up.
- Memory is a database with a policy — write, recall, inject. Not fine-tuning, not magic.
- Compress hierarchically when history grows: recent verbatim, older summarized, state kept structurally.