🔍 Read the full analysis: When Should You Choose CUGA Over Hermes Or Deep Agents? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
IBM reports that CUGA completed 73 of 90 AppWorld tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The result points to the effect an agent harness can have on one benchmark, but does not establish a general product ranking or test policy enforcement.
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks, ahead of Hermes at 42 and Deep Agents at 27, with all three systems using GLM-5.3-flash. The comparison suggests that the software coordinating an AI model can affect results on a particular multi-step workload, but it does not establish which system performs best across other tasks or real-world deployments.
IBM says it ran the same 90 tasks across 30 scenarios for each harness, using the Harbor evaluation framework and one attempt per task. CUGA passed 81.1% of tasks, Hermes passed 46.7%, and Deep Agents passed 30.0%. On a separate scenario measure—counting a scenario only when all three of its variants passed—CUGA passed 20 of 30, Hermes eight and Deep Agents two.
The systems did not use identical configurations. CUGA used task decomposition, app-focused tools, verification before execution and automatic reflection. Hermes ran with Harbor’s default setup and AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP and LangChain middleware for functions including tool selection, context summarization, a to-do list, retries and call limits. IBM says it also applied two fixes to Deep Agents so malformed model replies would not end runs.
According to the source, the experiment setup is available through harbor-eval, an IBM Harbor fork. AppWorld tests multi-step tasks across nine apps and 457 APIs, including email, payments and file storage. IBM describes the benchmark as a proxy for some capabilities relevant to enterprise automation, such as reading API documentation and changing state across apps; the test itself does not use real enterprise systems.
Why Harness Design Affects Results
The scores show why evaluating a model alone may not predict how an agent will perform. A harness determines how tasks are divided, what tools are available and whether actions are checked. Those choices can affect whether a model completes a sequence of steps, even when the underlying model stays the same.
For organizations comparing agents, the practical takeaway is to test the intended combination of model, harness, tools and workload, rather than treating a benchmark score as a complete forecast. The reported gap is a useful signal for teams whose workflows resemble AppWorld’s cross-app API tasks. It is not evidence that CUGA will lead in personal assistance, research or other work for which Hermes and Deep Agents may be configured.
The result also says nothing about whether an agent follows organizational rules during execution. IBM describes CUGA as offering features for applying rules through its harness, but these AppWorld scores do not test policy enforcement. That distinction matters for organizations considering agents that can access sensitive data or take consequential actions.
Popular right now
Philips Café Aromis Automatic Espresso Machine with LatteGo Pro Milk System
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
Dyson W1 PencilWash Wet and Dry Cleaner with Dok Cordless Wet Floor Cleaner
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Intended Uses
The source describes CUGA, short for ConfigUrable Generalist Agent, as IBM’s open-source harness for enterprise work across systems such as customer relationship management and payments. It describes Hermes from Nous Research as designed for always-on personal assistants and Deep Agents from LangChain as suited to longer, multi-step work such as research. The comparison is therefore best read as a test of different setups on one benchmark, not a direct judgment of each system’s full range of uses.
The benchmark’s scope is also limited. AppWorld uses consumer apps and simulated APIs to test tasks involving multiple steps and systems. IBM says those tasks approximate selected enterprise-relevant abilities, but the results do not directly establish performance on production enterprise software, organizational access controls or live workflows.
The source also cites a separate directory analysis in which GLM-5.2’s SWE-bench Pro score reportedly rose from 23% to 52% after a harness change. That is a distinct comparison, not part of IBM’s AppWorld experiment, and should not be combined with its results. The broader point in both cases is that harness choices can influence benchmark outcomes.
“This is not a verdict on Hermes or Deep Agents.”
— IBM, describing the AppWorld comparison
Limits of the Reported Comparison
The comparison covers one model, one benchmark, 90 tasks and a single attempt per task. IBM says the harness configurations differed, so the test does not isolate every implementation difference or show how much of the result came from any one feature. The supplied material does not say how the systems would compare with other models, different tasks, repeated runs or production workloads.
It also provides no comparative figures for cost, latency, large-scale reliability or deployment outcomes. The experiment did not use policies, so it cannot establish whether any harness reliably enforces access restrictions, approvals or other rules. IBM’s descriptions of CUGA’s policy-related capabilities are separate claims and are not validated by these scores.
The source does not provide a timeline for independent replication or a broader comparison. Until such evidence is available, the reported ranking should be treated as an outcome for this specific setup, not a universal ranking of the three systems.
Test Agents on Real Workflows
The reported Harbor evaluation setup is publicly available through the IBM fork named in the source, giving other teams a way to inspect the benchmark configuration. The source gives no date for a broader study or independent replication, so those should not be assumed to be scheduled.
For organizations considering these tools, the next useful step is to test them on representative workflows using the models and tool configurations they expect to deploy. Teams should measure not only task completion but also consistency across repeated or varied runs, and separately examine access controls, approval steps, data boundaries and behavior in their own environment. Those questions remain outside the evidence provided by the AppWorld scores.
Key Questions
What did IBM’s AppWorld comparison find?
IBM reported that CUGA completed 73 of 90 tasks, compared with 42 for Hermes and 27 for Deep Agents. All three used GLM-5.3-flash, but their harness configurations differed.
Does this prove CUGA is better than Hermes or Deep Agents?
No. The result applies to one model and one 90-task benchmark. It does not establish a general ranking across other uses, models or deployment conditions.
What is an agent harness?
A harness is the software that coordinates an AI agent’s work, including how it breaks down tasks, selects tools and checks actions or results. The comparison illustrates how harness design can affect task completion while the model is held constant.
Did the experiment test whether the agents enforce policies?
No. The reported AppWorld runs did not use policies, so they provide no evidence about enforcement of access controls, approvals or other organizational rules.
What should teams do with these results?
Treat the scores as a reason to evaluate the full setup—not as a deployment forecast. Test the intended model, harness and tools on representative workflows, and separately assess reliability, cost, security requirements and performance in the organization’s environment.
Primary source: Hugging Face · via ThorstenMeyerAI.com
