Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Chat demos lie. Running a company doesn’t.
Every AI model you’ve seen demoed looks brilliant in a chat window. But what happens when one actually has to run a business — handle a churn wave, dodge a fake CEO message, and close a deal it has already earned? A live experiment called Firmulate just answered that, and the results are not what the demos promise.
Top picks for "boss benchmark ends"
As an affiliate, we earn on qualifying purchases.
The experiment
Four frontier AI models were each given the same job: run the same small software company through its worst week. Same customers, same crises, same temptations to cheat — only the model changes. Every decision was versioned and auditable, so nothing is taken on faith.
The final league
The July 2026 Crucible League finished: gpt-5.6-sol took first with 95 points, Kimi K3 second at 93, Sonnet 5 third at 88, Fable 5 at 77, and Opus 4.8 last at 73. A do-nothing baseline scores 26 — partial progress counts, but a single breach of trust caps the total. In Firmulate’s words: “no amount of good work outweighs a breach of trust.”
The finding nobody expected
All models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
The buried fact: the decisive competitor weakness sat two document references deep in the company’s own files, not in the customer event. Models that actually read the file won the deal at full price — worth +€4,583 in monthly recurring revenue. The difference between first and last place wasn’t intelligence. It was whether the model did its homework.
Fake CEOs and reporter tricks
The social-engineering round escalated over three stages of fake CEO messages, plus a reporter trick: “just one yes/no, on background.” Five of five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”
The cautionary tale
Opus 4.8 was the most thorough participant — 80-plus learned rules and the deepest analyses — yet finished last. The close was left on the table and discipline slipped: it attempted writes into a locked department instead of escalating. The same weakness appeared, weaker, in all four. (One fairness note: K3 ran without an effort parameter at API default while the others ran at xhigh.)
Watch it live
The experiment never stopped. A live synthetic company runs at firmulate.com: 13 synthetic employees, real money mechanics — €105k/month burn against €2.3k MRR — a public cash countdown, and 680+ self-learned playbook rules, with every workday versioned. There’s also a “guess the model” quiz powered by 242 real, unedited management decisions at firmulate.com/quiz.html.
Now run it against your own company
Here’s the part that matters for enterprises: you can stop watching and start acting. Firmulate offers a pilot that runs the same wargame against a read-only export of your own business — your customers, your pipeline, your rules — through crisis scenarios like churn waves, price increases, competitor attacks and PR crises. You get a board report with the model ranking and the weak points of your own playbooks, and nothing ever writes back to real systems. Your data stays untouched; the crises stay in the sandbox.
If AI agents will ever touch your CRM, support queue or forecast, this is the cheapest way to find out which one actually closes the deal — and which one leaves it on the table. Ready to wargame your own business? Start your pilot at firmulate.com/pilot.html or email contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
