What is the best harness for each model?
Points and comments are a snapshot, not live.
When switching models, you should also consider switching the harness for best value.
Runta's FrontierHarness Eval v1.1 tested 22 harnesses × 5 open-weight models × 62 tasks × 4 attempts (27,280 runs, 83B tokens). Cost per pass varies up to 4.4× between harnesses on the same model. The lowest-cost harness changes with the model: Codex CLI ranks 1st with GLM-5.3 but 15th with Kimi-K3. Paying more does not guarantee higher pass rates. Model choice accounts for 93% of cost variation, harness for 2.3%, and the pair effect for 4.9%. Steps per pass (correlation +0.51) predict cost better than prompt size, tool count, or cache hit rate.
What commenters are saying
Commenters largely accept the finding that harness choice matters per model. Several note that steps per pass being the main cost driver is nearly tautological, since runs cost tokens and tokens correlate with steps. One commenter warns that closed-source harnesses that aren't fully editable are an anti-pattern; they advocate for harness-aware post-training and self-evolving harnesses. Another points out that failed runs account for a majority of cost, raising variance concerns. A top comment notes the OP's posts have been flagged for AI writing patterns.