Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
Points and comments are a snapshot, not live.
New benchmark tests AI coding agents on private enterprise codebases; top model resolves 38.8% of tasks.
Real-SWE evaluates frontier AI models on 10 tasks from licensed, private production codebases with real business consequences like billing and tax calculations. Tasks involve cross-functional changes across multiple services, with median 11 files edited per task. Top model Fable 5.1 achieved 38.8% resolution rate, followed by GPT-6 Astra at 33.8% and Gemini 3.8 Flash at 31.2%. Six of ten tasks had resolution rates below 15%. Most common failure types are missed requirements and unverified assumptions. Estimated rollout costs range from $2.50 to $6.96 per task. No model solved every task.
What commenters are saying
Several commenters questioned the benchmark's reproducibility since the private codebases cannot be inspected. Others argued private benchmarks are necessary to prevent saturation and gaming. Some disputed specific model rankings, claiming Gemini 3.8 Flash performs poorly in practice despite its high ranking here. One commenter noted the benchmark used the deprecated Gemini CLI harness rather than the current antigravity tool. A representative from the benchmark team stated they will open-source some tasks and trajectories.