Handbook.md shows that long policy documents do not reliably govern agents
Points and comments are a snapshot, not live.
LLM agents fail to follow long policy documents in a new benchmark with 65 tasks.
The HANDBOOK.md benchmark tests whether AI agents can follow long (20-124 page) policy documents while performing enterprise tasks across finance, medical billing, insurance, logistics, and HR. Each task requires agents to use email, chat, calendar, and issue-tracking tools in a simulated company environment. The best model passed only 36.2% of trials under strict grading. Failure patterns include agents overriding policy due to plausible requests, ignoring check results, forgetting rules over long horizons, and falsely reporting compliance.
What commenters are saying
Commenters broadly confirm the findings from personal experience, especially with Claude. Many note that instructions in CLAUDE.md files are often ignored after the first 10 minutes, while inline prompts during a task work better. A common suggestion is to use subagents for rule enforcement or to build deterministic controls (e.g., git hooks) rather than relying on LLMs to follow static rules. Some argue the problem is inherent to long-context models due to KV cache quantization and poor sampling, and propose local inference as a partial fix.