Overview

Context engineering refers to the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference, including all the other information that may land there outside of the prompts. Also including strategies for managing the entire context state(system instructions, tools, Model Context Protocol (MCP), external data, message history, etc)

Context Rot

As the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases. Thus, contexts should be treated as minimal finite resource. Like humans, who have "limited working memory capacity", LLMs have an “attention budget” that they draw on when parsing large volumes of context.

For Effective Context

System Prompt

System prompts should be extremely clear and simple and should include the minimal set of information that outlines the expected behavior. Anthropic recommends organizing prompts into distinct sections

(like `<background_information>`, `<instructions>`, `## Tool guidance`, `## Output description`, etc)

and using techniques like XML tagging or Markdown headers to delineate these sections, although the exact formatting of prompts is likely becoming less important as models become more capable.

Tools

Allow agents to operate with their environment and pull in new, additional context as they work.
Tools should be self-contained, robust to error, and extremely clear with respect to their intended use.

"Just In Time" Context Strategy

Rather than pre-processing all relevant data up front, agents built with the “just in time” approach maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools.
Folder hierarchies, naming conventions, and timestamps all provide important signals that help both humans and agents understand how and when to utilize information.
there's a trade-off: runtime exploration is slower than retrieving pre-computed data. Not only that, but opinionated and thoughtful engineering is required to ensure that an LLM has the right tools and heuristics for effectively navigating its information landscape. Without proper guidance, an agent can waste context by misusing tools, chasing dead-ends, or failing to identify key information.
The hybrid strategy might be better suited for contexts with less dynamic content, such as legal or finance work.

Context engineering for long-horizon tasks

To enable agents to work effectively across extended time horizons, there is a few techniques hat address these context pollution constraints directly: compaction, structured note-taking, and multi-agent architectures.

Compaction

Compaction is the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary.

Structured Note Taking

Structured note-taking, or agentic memory, is a technique where the agent regularly writes notes persisted to memory outside of the context window. These notes get pulled back into the context window at later times such as context reseted.

Multi Agent Architecture

Sub-agent architectures provide another way around context limitations. Rather than one agent attempting to maintain state across an entire project, specialized sub-agents can handle focused tasks with clean context windows. The main agent coordinates with a high-level plan while subagents perform deep technical work or use tools to find relevant information. Each subagent might explore extensively, using tens of thousands of tokens or more, but returns only a condensed, distilled summary of its work (often 1,000-2,000 tokens).

Pick a proper option

Compaction maintains conversational flow for tasks requiring extensive back-and-forth;
Note-taking excels for iterative development with clear milestones;
Multi-agent architectures handle complex research and analysis where parallel exploration pays dividends.

Reference

https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

Context Engineering