🔍 Read the full analysis: Read Ironclad’s Fine Print On OpenAI Training Agents In Your Software on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI described training GPT-6 Astra in hosted copies of contract-management software from Ironclad, using 11 legal, commercial and procurement tasks. Astra met an average 55% of task criteria, while OpenAI’s estimated completion times were simulated and the results do not establish that the agent can safely handle customer workflows without human review.
OpenAI said on October 6 that it trained its GPT-6 Astra model in hosted copies of contract-management software from Ironclad, testing whether an AI agent could carry out multi-step legal, commercial and procurement work. Across 11 tasks, Astra met an average of 55% of the listed criteria; OpenAI cautioned that its time estimates were simulated, and the results do not show that agents can reliably run such workflows without human oversight.
The tasks were selected by Ironclad staff and OpenAI employees who use the product. They included creating nondisclosure agreements, configuring procurement approval processes and changing a reusable contract clause according to a requester’s selected jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes to complete each task.
Each task was assessed against a rubric of 8 to 50 criteria, depending on its complexity. OpenAI reported that GPT-6 Astra met an average 55.0% of criteria, compared with 41.6% for GPT-5.6 Sol in a high-compute setting. An internal model used during Astra’s development reached 63.7%. On one highlighted task, Astra met about 94% of the criteria, but that example does not represent the overall average.
OpenAI said Ironclad supplied hosted product environments for model practice. The company said it created synthetic training tasks from publicly filed contracts in the SEC’s EDGAR database, filtered to remove personal information. It stated that the work did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Contract Workflow Accuracy Matters
The reported 55% is an average share of rubric criteria met, not the percentage of tasks completed successfully. That distinction matters in contract and procurement systems, where a missing requirement can invalidate the practical value of otherwise correct work. A workflow might need Finance approval above a spending threshold, Security review for certain requests and Legal review for nonstandard terms. Missing even one required check could send a purchase through without the intended control.
OpenAI’s post acknowledges that agents can lose track of business rules during multi-step work and says human oversight remains important. The figures do not demonstrate dependable autonomous contract processing, nor do they establish measured savings for Ironclad customers. They do show a test of a model inside specialised business software, where success depends on following detailed rules rather than simply producing plausible text.
The collaboration also has implications for software companies. Training agents on product-specific workflows could make those agents more useful to customers, while exposing where they fail. If agents increasingly become the way customers interact with software, a vendor’s value may depend less on its screens and more on the business rules, records, audit trails and controls that its product maintains. That is an implication of the development, not a result measured by the test.
As an affiliate, we earn on qualifying purchases.
How the Ironclad Test Was Built
OpenAI’s October 6 post was titled “Advancing computer use with Ironclad.” The source material says other AI news trackers misread Ironclad as the name of a hardened agent framework; it is instead a contract-management software company. The reported work focused on training and evaluating a model within that company’s product, rather than announcing a new standalone agent framework.
The test used a small set of 11 selected tasks and criteria tailored to each task. OpenAI estimated the time an experienced user might take, then compared that figure with a simulated agent-time estimate. It also invited a small number of other software companies to bring examples of work current agents cannot reliably complete, along with subject-matter experts, secure testing environments and data that can safely be used for research.
As an affiliate, we earn on qualifying purchases.
What the Results Do Not Establish
The reported averages do not identify, in the provided source material, which criteria Astra missed across all 11 tasks or how often a missed requirement would create a serious operational or legal problem. The score is not a pass rate, and the showcase result of about 94% on one task cannot establish performance across other work.
OpenAI’s comparison of 37.0 minutes for GPT-5.6 Sol and 19.2 minutes for GPT-6 Astra is based on simulated time estimates, not observed customer use. The source does not provide independently verified customer outcomes, a deployment study or evidence that the model can operate Ironclad workflows safely without review. It also does not detail the full security arrangements for partner environments or the terms under which future software partners would share research data.
As an affiliate, we earn on qualifying purchases.
The Next Test for Software Agents
OpenAI said it is seeking a small number of software-company partners to identify challenging workflows for future research. Prospective partners are expected to provide concrete examples where current agents fail, people with deep knowledge of the work, secure test environments and data suitable for research. The source material does not give a schedule for new partnerships or further benchmark results.
For companies considering agents in contract, finance or customer-record systems, the reported test points to practical questions: which exact requirements did the agent miss, who reviews each result, and what controls prevent an incomplete workflow from being treated as finished? Until performance is demonstrated on real deployments with clearly reported error rates and safeguards, the Ironclad results are best read as a research-stage evaluation, not proof of ready-to-run automation.
procurement approval workflow software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is Ironclad in this report?
Ironclad is a contract-management software company, not the name of a new agent framework. OpenAI’s post describes work training and evaluating a model in hosted copies of Ironclad’s product.
What does Astra’s 55% score mean?
It is the average share of rubric criteria met across the selected tasks. It is not the share of tasks completed successfully, and it does not mean the agent delivered 55% of the value of a contract workflow.
Did OpenAI demonstrate customer time savings?
No measured customer savings are established in the source. OpenAI’s reported times—19.2 minutes for Astra and 37.0 minutes for GPT-5.6 Sol—were simulated estimates based on assumed processing and generation speeds.
Can companies use the agent to manage contracts without review?
The results do not establish that. OpenAI’s post says human oversight matters because agents can lose track of business rules, and the reported average score indicates that many listed criteria were not met.
What data did OpenAI say it used?
OpenAI said it built synthetic tasks from publicly filed contracts in the SEC’s EDGAR database and filtered out personal information. It said it did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
