LLM Context Engineering in Production: Prompt Assembly, Token Budget, and RAG Context Design
Summary
- Context Engineering is the core capability for LLM production in 2026 — broader and more practical than Prompt Engineering alone
- Token budget is a hard constraint: system prompts, tool definitions, RAG retrieval, and conversation history must be allocated in layers
- Prompt assembly is not string concatenation — it is a prioritized Context Pipeline where each layer has a budget and eviction policy
- The 3 biggest RAG context pitfalls: retrieval noise drowning key info, chunk boundaries breaking semantics, duplicate content wasting tokens
- This article provides a complete solution from theory to Python implementation, including a Token budget calculator and production Context Manager
Table of Contents
- Why Prompt Engineering Is No Longer Enough
- Core Concepts of Context Engineering
- Token Budget: Financial Planning for the Context Window
- Prompt Assembly Pipeline Design
- RAG Context Injection Best Practices
- Context Compression and Eviction Strategies
- Python Production Context Manager
- Interview FAQs and Pitfall Guide
- Summary and Further Reading
Why Prompt Engineering Is No Longer Enough
In 2024, everyone was discussing "how to write better prompts." By 2026, what actually blocks enterprise LLM adoption is not wording — it is how the entire context is organized.
A Real Production Incident
An e-commerce customer service AI saw complaint rates spike in its first week:
| Symptom | Root Cause | Impact |
|---|---|---|
| AI answers unrelated to orders | RAG retrieved 10 historical tickets, drowning out current order context | Irrelevant responses |
| Slower responses after multi-turn chat | Conversation history grew unbounded; truncation after token overflow | Lost user intent |
| Monthly API bill 3x over budget | System prompt resent every call without Prompt Caching | Cost explosion |
None of these can be fixed by "optimizing prompt wording." They all belong to Context Engineering.
Prompt Engineering vs Context Engineering
| Dimension | Prompt Engineering | Context Engineering |
|---|---|---|
| Focus | Wording and format of a single prompt | Organization, budget, and lifecycle of entire context |
| Scope | Simple Q&A, one-shot tasks | Multi-turn chat, RAG, Agents, tool calling |
| Core skills | Role setting, Few-shot, CoT | Token budget, context layering, compression |
| Production complexity | Low | High (caching, retrieval, state management) |
| 2026 interview weight | Basic | High-frequency architecture topic |
In short: Prompt Engineering controls what you say; Context Engineering controls what the model sees.
Core Concepts of Context Engineering
Four Layers of Context
Think of the full LLM input as an "information sandwich":
┌─────────────────────────────────────────────┐
│ Layer 1: System Context │
│ Role definition, behavior constraints, format │
├─────────────────────────────────────────────┤
│ Layer 2: Tool Context │
│ Function Calling defs, MCP tools, Schema │
├─────────────────────────────────────────────┤
│ Layer 3: Knowledge Context │
│ RAG results, knowledge graphs, business docs│
├─────────────────────────────────────────────┤
│ Layer 4: Conversation Context │
│ Multi-turn history, current input, reasoning │
└─────────────────────────────────────────────┘
Each layer has its own token budget and priority. When total tokens approach the window limit, evict from the lowest-priority layer first.
Context Lifecycle
Context is not assembled once — it goes through multiple stages per interaction:
- Assembly: Merge layers by priority
- Validation: Check token total, format compliance, sensitive data filtering
- Dispatch: Call LLM API, leverage Prompt Caching for repeated tokens
- Update: Append model response to conversation layer
- Compaction: Summarize or evict low-value history when conversation grows too long
Token Budget: Financial Planning for the Context Window
Why Token Budgeting Matters
With a 128K context window model:
- System prompt + tool definitions: typically 2K–8K tokens
- RAG with 10 documents at 500 tokens each: 5K tokens
- 50 turns of history at 200 tokens each: 10K tokens
- Output reserve: 4K tokens
That is already 21K–27K tokens. Complex Agent or long-document RAG easily exceeds 50K. Without budget management, you are running blind.
Recommended Budget Allocation (128K Window)
| Layer | Budget Type | Token Cap | Eviction Strategy |
|---|---|---|---|
| System | Fixed | 4K | Never evict; use Prompt Caching |
| Tools | Fixed | 6K | Load dynamically by usage frequency |
| Knowledge | Elastic | 20K | Truncate by relevance score |
| Conversation | Elastic | 30K | Sliding window + summary compression |
| Output reserve | Fixed | 8K | — |
| Safety buffer | Fixed | 10K | Prevent estimation overflow |
Three Token Counting Approaches
Approach 1: tiktoken (recommended)
import tiktoken
def count_tokens(text: str, model: str = "gpt-4o") -> int:
encoding = tiktoken.encoding_for_model(model)
return len(encoding.encode(text))
def budget_check(layers: dict[str, str], max_tokens: int = 128000) -> dict:
usage = {name: count_tokens(content) for name, content in layers.items()}
total = sum(usage.values())
return {
"usage": usage,
"total": total,
"remaining": max_tokens - total,
"overflow": total > max_tokens,
}
Approach 2: Character estimation (fast pre-check)
Chinese ~1.5–2 chars/token, English ~4 chars/token. Use for pre-assembly checks; validate with tiktoken before sending.
Approach 3: Actual usage from API response
Read response.usage after each call for monitoring and dynamic budget adjustment.
Prompt Assembly Pipeline Design
Pipeline Architecture
User Input ──→ [Intent Detection] ──→ [Knowledge Retrieval] ──→ [Context Assembly] ──→ [Budget Check] ──→ LLM API
│ │ │ │
▼ ▼ ▼ ▼
Decide which tools Top-K doc filtering Merge by priority Compress if over limit
Eviction Priority Rules
When token budget is insufficient, trim in this order:
- Oldest conversation messages first (keep last N turns + first system interaction)
- Lowest-scoring knowledge documents (remove by Rerank score ascending)
- Unused tool definitions (load only tools likely needed this turn)
- System layer last (keep fixed for Prompt Caching)
Production Assembler Implementation
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class ContextLayer:
name: str
content: str
priority: int
max_tokens: int
cacheable: bool = False
@dataclass
class ContextBudget:
total_limit: int = 128000
output_reserve: int = 8000
safety_buffer: int = 10000
@property
def available(self) -> int:
return self.total_limit - self.output_reserve - self.safety_buffer
class ContextAssembler:
def __init__(self, budget: ContextBudget):
self.budget = budget
self.layers: list[ContextLayer] = []
def add_layer(self, layer: ContextLayer) -> None:
self.layers.append(layer)
self.layers.sort(key=lambda l: l.priority)
def assemble(self) -> str:
available = self.budget.available
parts: list[str] = []
for layer in self.layers:
tokens = count_tokens(layer.content)
if tokens <= layer.max_tokens and tokens <= available:
parts.append(layer.content)
available -= tokens
elif tokens > layer.max_tokens:
truncated = self._truncate(layer.content, layer.max_tokens)
parts.append(truncated)
available -= layer.max_tokens
return "\n\n---\n\n".join(parts)
def _truncate(self, text: str, max_tokens: int) -> str:
encoding = tiktoken.encoding_for_model("gpt-4o")
tokens = encoding.encode(text)
if len(tokens) <= max_tokens:
return text
return encoding.decode(tokens[:max_tokens])
RAG Context Injection Best Practices
RAG is the most error-prone part of Context Engineering. How retrieved documents are fed to the model directly determines answer quality.
Three Document Injection Formats
Format 1: XML tags (recommended)
<retrieved_documents>
<document source="product_manual_v3.pdf" score="0.92">
Product warranty is 12 months from purchase date...
</document>
<document source="faq_returns.md" score="0.87">
7-day no-reason return requires intact product...
</document>
</retrieved_documents>
Format 2: Numbered citations (good for traceability)
[1] Product warranty is 12 months... (source: product_manual_v3.pdf)
[2] 7-day no-reason return... (source: faq_returns.md)
Answer based on the references above and cite numbers in your response.
Format 3: Direct concatenation (not recommended)
Five Rules for RAG Context
| Rule | Description | Consequence of Violation |
|---|---|---|
| Relevance threshold | Do not inject docs below 0.7 score | Noise increases hallucination |
| Deduplication | Merge chunks from same document | Token waste |
| Metadata preservation | Include source, time, version | No traceability, compliance risk |
| Position priority | Most relevant docs first | Lost in the Middle effect |
| Dynamic Top-K | Adjust K by remaining budget | Hard truncation when over budget |
Lost in the Middle
Stanford research (2023) found recall accuracy drops when relevant info sits in the middle of context. Production mitigations:
- Put top 2 most relevant docs first
- Put secondary docs last
- Place system prompt or tool definitions in the middle
Context Compression and Eviction Strategies
Strategy 1: Sliding Window
Keep last N turns. Recommended N=10–20 depending on per-turn token volume.
Strategy 2: Summary Compression
When conversation exceeds threshold, use a small model to summarize early turns:
SUMMARIZE_PROMPT = """Compress the following conversation into a summary, preserving:
1. User's core request and constraints
2. Confirmed key facts (order ID, product model, etc.)
3. Unresolved issues
Conversation:
{history}
Summary (max 200 words):"""
Strategy 3: Structured State Extraction
For Agent scenarios, maintain structured session state instead of full conversation:
{
"user_intent": "check order logistics",
"entities": {"order_id": "ORD-20260703-8842"},
"resolved": ["order confirmed", "carrier is SF Express"],
"pending": ["estimated delivery time"]
}
Token cost drops from O(turns) to O(1).
Strategy Selection
| Scenario | Recommended Strategy | Reason |
|---|---|---|
| Customer service multi-turn | Sliding window + summary | Balance cost and coherence |
| Agent tool calling | Structured state extraction | Tool results are long; state is lean |
| Code assistant | Sliding window (keep code blocks) | Code should not be summarized |
| Document Q&A | No conversation compression needed | Single-turn; knowledge layer is the bottleneck |
Python Production Context Manager
class ProductionContextManager:
def __init__(
self,
model: str = "gpt-4o",
budget: ContextBudget = ContextBudget(),
):
self.model = model
self.budget = budget
self.assembler = ContextAssembler(budget)
self.conversation_history: list[dict] = []
def build_context(
self,
user_input: str,
system_prompt: str,
retrieved_docs: Optional[list[dict]] = None,
tools: Optional[list[dict]] = None,
) -> str:
self.assembler.layers.clear()
self.assembler.add_layer(ContextLayer(
name="system", content=system_prompt,
priority=0, max_tokens=4000, cacheable=True,
))
if tools:
tools_text = self._format_tools(tools)
self.assembler.add_layer(ContextLayer(
name="tools", content=tools_text,
priority=1, max_tokens=6000,
))
if retrieved_docs:
docs_text = self._format_rag_docs(retrieved_docs)
self.assembler.add_layer(ContextLayer(
name="knowledge", content=docs_text,
priority=2, max_tokens=20000,
))
history_text = self._format_history()
self.assembler.add_layer(ContextLayer(
name="conversation", content=history_text,
priority=3, max_tokens=30000,
))
user_layer = f"<user_query>\n{user_input}\n</user_query>"
self.assembler.add_layer(ContextLayer(
name="current_input", content=user_layer,
priority=4, max_tokens=4000,
))
return self.assembler.assemble()
def _format_rag_docs(self, docs: list[dict]) -> str:
filtered = [d for d in docs if d.get("score", 0) >= 0.7]
filtered.sort(key=lambda d: d["score"], reverse=True)
parts = ["<retrieved_documents>"]
for doc in filtered:
parts.append(
f' <document source="{doc["source"]}" score="{doc["score"]:.2f}">\n'
f' {doc["content"]}\n'
f' </document>'
)
parts.append("</retrieved_documents>")
return "\n".join(parts)
def update_history(self, role: str, content: str) -> None:
self.conversation_history.append({"role": role, "content": content})
if len(self.conversation_history) > 40:
self._compact_history()
def _compact_history(self) -> None:
old = self.conversation_history[:20]
self.conversation_history = self.conversation_history[20:]
summary = f"[Earlier conversation summary: {len(old)} messages condensed]"
self.conversation_history.insert(0, {"role": "system", "content": summary})
Interview FAQs and Pitfall Guide
Five Common Interview Questions
Q1: How does Context Engineering differ from Prompt Engineering?
Context Engineering manages the organization, budget, and lifecycle of the entire input context — system-level design. Prompt Engineering optimizes wording of a single prompt — technique-level optimization. In production, the former determines whether the system runs and at what cost; the latter determines answer quality.
Q2: With 128K context, can we skip compression?
No. 128K is a theoretical limit. In practice consider: cost (per-token billing), latency (longer context = slower inference), attention decay (Lost in the Middle), and multi-user concurrency. Production always needs budget management.
Q3: Should we inject all 10 RAG documents?
No. Apply relevance filtering, deduplication, Rerank, and dynamic Top-K by budget. Noisy documents significantly increase hallucination rate.
Q4: How does Prompt Caching relate to Context Engineering?
Prompt Caching is a cost optimization. Context Engineering decides what to cache. Fixed content like system prompts and tool definitions should be cacheable; conversation and RAG results should not.
Q5: How to manage context in multi-Agent systems?
Each Agent maintains its own Context Manager. Pass structured state (not full conversation history) between Agents. Passing Agent A's full context to Agent B causes exponential token growth.
Production Pitfall Checklist
| Pitfall | Symptom | Fix |
|---|---|---|
| Unbounded conversation history | Slower responses, truncation after overflow | Sliding window + summary |
| RAG noise injection | Answers drift from user question | Relevance threshold + Rerank |
| System prompt re-billing | High API cost | Prompt Caching |
| Full tool definition loading | 8K+ tokens wasted on unused tools | Load tools by intent |
| No token monitoring | Cost overrun discovered late | Log usage metrics per call |
Multi-Model Context Management
Production rarely uses one model. Abstract a TokenCounter interface — never use GPT tiktoken to estimate Claude usage.
Prompt Caching in Production
Mark system prompts and tool definitions as cacheable. With 90% cache hit rate on 10K fixed tokens per request, daily cost can drop ~80%.
Context Quality A/B Testing
Compare control vs variant on: user satisfaction, average token cost, P99 latency. Data-driven context strategy beats intuition.
Agent Context Isolation
Orchestrator passes structured state (order_id, intent) — never full conversation history from other agents. Token cost stays O(1) not O(rounds).
Enterprise Case Study
SaaS customer service platform: 45K→12K avg input tokens, ¥280K→¥90K monthly API cost, wrong-answer rate 23%→6% after Context Engineering rollout.
Advanced Interview Questions
Q6: 1M context window? Lost in the Middle worsens — more window ≠ stuff more content.
Q7: RAG vs full document injection? Fixed docs < 30% budget → full injection; dynamic/large → RAG.
Q8: 8K tokens on tool definitions? Dynamic loading by intent, merge similar tools, short Schema.
Hands-On: Minimal Context Pipeline
Build layered assembly → budget check → auto-truncation in 30 lines. Verify noise docs filtered (score < 0.7), history compressed beyond 40 messages.
Context Engineering vs Fine-Tuning vs Long Context
80% of "model underperformance" is context organization, not model capability. Optimize context first, then model selection, then fine-tuning.
2026 Trends
Context-aware Rerankers, adaptive token budgets, cross-session memory (Mem0/Zep), multimodal context management, context security (prompt injection defense).
Summary and Further Reading
Context Engineering is the dividing line between LLM demos and production. Three essentials: layered context management, strict token budget, intelligent compression and eviction.
Key takeaways:
- Four context layers: system, tools, knowledge, conversation — each with its own budget
- Token budget is a hard constraint; 128K window does not mean unlimited spending
- RAG injection must filter, deduplicate, and sort to avoid Lost in the Middle
- Three compression strategies: sliding window, summary, structured state
- Prompt Caching + Context Pipeline is the cost optimization combo
Related reading:
References:
Try these browser-local tools — no sign-up required →