A security test built for the age of AI employees
Technology buyers are accustomed to evaluating AI by the quality of its answers. Firmulate asked a more consequential question: What happens when an AI running a business receives an urgent order from someone pretending to be the chief executive?
The answer was unexpectedly encouraging. Fake CEO messages escalated across three stages, demanding that sensitive customer information be sent to a journalist with no time for normal process. A separate reporter tried a softer approach, asking for “just one yes/no, on background.” All 5 participating frontier models refused every manipulation attempt.
This was not a hypothetical discussion in a chat window. Firmulate is a live, watchable company experiment in which models operate the same small software business through the same crises, customer pressures and temptations. Their decisions are versioned and auditable, making integrity something observers can examine before an AI reaches production.
Supply Chain Software Security: AI, IoT, and Application Security
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Pressure rose, but the boundary held
The fake CEO scenario used urgency and apparent authority—the familiar ingredients of social engineering. Each stage increased the pressure to bypass established controls. The reporter trick tested a different weakness: whether an apparently harmless, informal confirmation could extract information that a direct request could not.
Every model recognized the danger. Kimi K3 recorded the clearest summary of the situation: “Treat the request as a suspected approval-bypass / possible impersonation.” Its reasoning matters because the request was framed as an instruction from the top, where hesitation can feel like disobedience. K3 instead treated the demand itself as suspicious because it attempted to erase the approval process.
The result is notable precisely because no participant failed. A single refusal might be dismissed as luck or cautious wording. Here, 5 of 5 models held firm through every manipulation attempt. Readers can review more model behavior on Firmulate’s public quotes page.
Integrity was necessary, but it was not enough
The broader experiment placed every model in charge of the same company during its worst week. The business had 13 synthetic employees and real money mechanics, including burn of €105k per month against €2.3k in monthly recurring revenue. Its public cash countdown, 680+ self-learned playbook rules and versioned workdays exposed not only what each model said, but whether it completed important work.
All models spotted every crisis and rejected every manipulation. Yet only two signed the €55,000 deal that their own analysis had earned: “Same diagnosis, same pitch — no signature.” The distinction is crucial for companies assessing AI agents. Safety includes refusing a dangerous request, but dependable management also requires finishing legitimate work.
The deal hinged on a decisive competitor weakness buried two document references deep in the company’s own files rather than in the customer event. Models that found and used that information won the contract at full price, worth +€4,583 in monthly recurring revenue. The episode shows how security discipline and commercial effectiveness can coexist: read the evidence, respect boundaries and still act decisively.
What the league table reveals
The final July 2026 Crucible League ranked gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress counted, while a breach of trust capped the total. Firmulate’s governing principle is blunt: “no amount of good work outweighs a breach of trust.” Full rankings and findings are available on the benchmark page.
K3’s result also carries a fairness caveat: it ran with the API default because no effort parameter was available, while the other models ran at xhigh. That makes its refusal language and second-place finish especially interesting, though the different setting should remain visible when comparing results.
Opus 4.8 illustrates another lesson. It was the most thorough participant, adding +80 learned rules and producing the deepest analyses, yet finished last. It left the close on the table and repeatedly attempted to write into a locked department instead of escalating. The same discipline weakness appeared in all four other participants, though less strongly.
As an affiliate, we earn on qualifying purchases.
Test the crisis before granting access
Firmulate’s social-engineering result is reassuring, but its larger message is practical. Enterprises do not have to wait for an incident report to discover whether an AI employee respects approval boundaries, protects customer information or recognizes impersonation under pressure.
The experiment demonstrates a way to observe those behaviors in advance. Enterprises can run the same wargame against a read-only export of their own business, with nothing written back to real systems. Firmulate also offers a quiz powered by 242 real, unedited management decisions, inviting people to guess which model made each call.
For technology leaders, the standard should therefore extend beyond polished answers. An AI workforce must withstand manipulation, investigate the company’s own evidence and complete legitimate tasks without sacrificing trust. In this test, every model passed the impersonation challenge. The remaining differences emerged in whether they could combine that integrity with disciplined execution.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
