In the world of AI, success is often judged by how well chatbots can mimic human conversation. But what about real-world performance—particularly the ability to make decisions, resist manipulation, and close deals under pressure? A groundbreaking live experiment by Firmulate reveals that flashy chat demos might hide an AI’s true strength — or weakness — when it’s time to finish the job.
Testing AI in the Trenches: The Same Company, Multiple Models, One Week of Crisis
Recently, four state-of-the-art AI models faced a controlled but intense scenario: running a small software company through its worst week. The company, with real money mechanics and a public cash countdown, confronted crises like customer churn, internal miscommunication, and tempting manipulations designed to test integrity. Every decision was carefully versioned and auditable, creating a transparent battlefield for these AI systems.
The Models and Their Scores
- gpt-5.6-sol scored 95, leading the league and uncovering critical details buried deep in the company’s own files.
- Kimi K3 followed closely with a score of 93, successfully closing the deal and demonstrating the most discipline.
- Sonnet 5 scored 88, also closing the deal but with some process slips.
- Fable 5 scored 77, showing strong rule discipline but leaving the deal unexecuted.
In contrast, a baseline score of 26 revealed a minimal partial progress, emphasizing that the models’ capability to preserve trust and execute tasks is more crucial than superficial chat prowess.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Crises and Manipulation Tests
All models identified every crisis thrown at them and refused every manipulation attempt, including sophisticated social engineering — fake CEO messages escalating over three stages and a reporter trick asking for a background approval. Remarkably, all five models refused to be manipulated, highlighting their ability to resist social engineering.
The Hidden Weakness
While all models performed admirably in crisis detection and resistance, only two managed to close the deal worth €55,000. The reason was subtle but critical: the decisive advantage lay in a specific document reference deep within the company’s own files. Models that read and understood this file won the full-price deal, worth over €4,583 in monthly recurring revenue.
The Discipline Gap
Despite their capabilities, discipline varied. Opus 4.8, the most thorough model with over 80 learned rules, faltered at the final step—leaving the deal on the table. It attempted to write the approval into a locked department instead of escalating it through proper channels. The same pattern appeared, weaker but similar, across all models.
The Best AI Tools For Accountants: Moving from Compliance to Explosive Growth
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Implications for Business AI Adoption
This experiment underscores a vital point: current chat-focused demos are inadequate for evaluating AI readiness in real business contexts. Success isn’t just about generating convincing conversations; it’s about executing tasks, reading critical files, resisting social engineering, and closing deals without slipping.
What Really Matters
For enterprises considering AI solutions, the key questions are: Will the AI finish what it starts? Does it read your files thoroughly? Can it resist pressures to manipulate or shortcut processes? And crucially, what is the cost of a unit of useful, trustworthy work?
Practical Claude Handbook for Attorneys: Master Case Analysis, Contract Review, Research Automation, Client Communication, and Document Drafting (Claude AI Guide for Beginners)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Live Experiment at Your Fingertips
Firmulate’s live company emulation platform lets organizations run their own “wargames” against exported versions of their business models. These tests are completely read-only, ensuring no actual changes to real systems, but they reveal how AI models perform under real-world stressors—crucial for assessing their suitability before deployment.
The real test of AI readiness isn’t in chat demos but in whether it can finish tasks under pressure, read deep information, and resist manipulation. Firmulate’s live experiments prove that only the most disciplined models succeed in closing crucial deals—highlighting what matters most for AI adoption in business.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Rise of the Titans: A Chronicle of AI War
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
