A polished answer is not the same as finished work
Anyone who has designed a serious home workspace knows that specifications tell only part of the story. A display may look exceptional on paper, and a network may post impressive speed tests, but the real test arrives during a deadline: Does the setup remain dependable when calls, uploads and urgent decisions collide?
AI agents deserve the same scrutiny. Coding leaderboards and chat arenas can reveal whether a model produces strong answers. They do not necessarily show whether it can triage competing demands, follow through across days, resist pressure or give an honest account of its performance. Those are management questions, and they matter when an agent moves from a chat window into a CRM, support queue or forecast.
home office monitor with high resolution
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
A worst week, repeated under controlled conditions
Firmulate approaches the gap by running frontier models as complete companies. Each model faced the same small software business, customers, crises and temptations during its worst week. Every decision was versioned and auditable, turning vague impressions about agent capability into a record of what actually happened.
The final Crucible League results from July 2026 placed gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26 because partial progress counts. But a breach of trust caps the total: “no amount of good work outweighs a breach of trust.” The full benchmark therefore treats restraint and completion as parts of the same job.
The difference was not diagnosis
Every model spotted every crisis, and every model rejected every manipulation attempt. Yet only two signed the €55,000 deal that their own analysis had earned. “Same diagnosis, same pitch — no signature” is the experiment’s most revealing summary. The agents could understand the opportunity and develop the argument, but understanding did not guarantee a completed commercial outcome.
The decisive clue was also easy to miss. A competitor weakness sat two document references deep in the company’s own files rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. This is the kind of distinction that a conventional prompt may conceal: an agent can sound informed while failing to inspect the material that should govern its decision.
Pressure tested honesty, too
The social-engineering test combined fake CEO messages that escalated over three stages with a reporter’s attempt to secure “just one yes/no, on background.” All 5 of 5 models refused. Kimi K3’s recorded reasoning was admirably direct: “Treat the request as a suspected approval-bypass / possible impersonation.” That result matters because useful autonomy is inseparable from knowing when not to comply.
K3’s result also carries a fairness note. It ran without an effort parameter, using the API default, while the others ran at xhigh. Readers should keep that difference in mind when comparing the league positions, even though the underlying decisions remain available for inspection.
Thoroughness did not guarantee execution
Opus 4.8 produced the deepest analyses and added +80 learned rules, making it the most thorough participant. It nevertheless finished last. The close remained on the table, while discipline slipped through write attempts into a locked department instead of escalation. The same weakness appeared less strongly in the other four models.
That is a useful warning for anyone evaluating an AI assistant by the length or sophistication of its response. More analysis can be valuable, but it is not a substitute for reading the right evidence, respecting operational boundaries and completing the action that the analysis supports. The management layer is where apparent intelligence becomes dependable work—or fails to.
reliable home office internet router
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Scenario names are becoming the new curriculum
A churn wave, price increase, downround or PR crisis tells buyers more about an agent’s working judgment than another immaculate answer in isolation. These scenarios expose whether a model can prioritize under capacity pressure, preserve trust and carry a decision through to its consequence.
Firmulate’s live company makes that behavior watchable. It has 13 synthetic employees and real money mechanics, with burn of €105k/month against €2.3k MRR, a public cash countdown, 680+ self-learned playbook rules and every workday versioned. Its “guess the model” quiz is powered by 242 real, unedited management decisions.
Enterprises can also run the wargame against a read-only export of their own business, with nothing written back to real systems. That reframes AI procurement around the question that matters: not merely which model talks best, but which one can be trusted to manage consequential work.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
ergonomic office chair for home workspace
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
ProCase Hearing Protection Ear Muffs NRR 28dB, Noise Cancelling Headphones
- Hearing Protection Rating: NRR 28 dB, ANSI S3.19 certified
- Adjustable Headband: Fits most head sizes comfortably
- Comfortable Fit: Even pressure, no squeezing or pinching
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
