🔍 Read the full analysis: Z.ai’s Legal AI Benchmark Rank: Looking Past The Share Of Tasks Passed on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh among 81 models on Vals AI’s archived Harvey Legal Agent Benchmark leaderboard, with a 19.17% all-criteria task pass rate. The same results show a 92.5% pass rate on individual criteria, but a task fails under the benchmark’s all-or-nothing rule if even one requirement is missed. Z.ai’s claimed 12.5-percentage-point gain over its base model matches the leaderboard figures; broader claims about open-source leadership and real-world legal reliability need more evidence.
Z.ai’s GLM-5.3-FlashX-Legal placed seventh among 81 models on Vals AI’s archived Harvey Legal Agent Benchmark leaderboard, with a 19.17% all-criteria pass rate. The result means the model met every required criterion on about one in five benchmark tasks; it does not mean it passed only one in five individual legal checks, since Vals also reported a 92.5% criteria pass rate.
Z.ai announced the legal model on its AutoClaw blog on October 13, 2026, and says it is built on GLM-5.3-Flash. Its developer documentation describes the system as optimized for legal document understanding, reasoning and task execution, including research, contract review and drafting. Z.ai says the model received legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal workflows.
The benchmark figures come from Vals AI’s archived Version 1 leaderboard, updated October 18. It lists GLM Legal at 19.17%, seventh overall, and the base GLM 5.3 Flash at 6.67%. The difference is 12.50 percentage points, matching Z.ai’s stated improvement. Vals listed the legal model’s cost at $0.15 per test; Z.ai’s pricing page separately lists $0.37 per million input tokens and $1.25 per million output tokens.
The score uses all-pass grading: a task counts as complete only if every rubric criterion passes. Vals says the leaderboard score averages task pass rates from two AI judges. Its leaderboard description gives a 120-task private holdout set, with internet access disabled. The criteria measure details such as facts, conclusions, citations, deadlines, recommendations and formatting. A single missed requirement can therefore make an entire task count as failed.
Why the All-Pass Score Matters
The gap between 19.17% of tasks completed and 92.5% of criteria passed changes how the result should be read. The task score reflects whether a legal work product met every specified requirement, while the criteria score reflects performance on individual checks. Neither number alone establishes how useful the model would be in a law firm, but together they show why a single pass-rate headline can hide important differences in benchmark performance.
For legal teams, a missed deadline, incorrect citation or omitted risk can matter even when most other parts of a draft are sound. The benchmark’s all-pass approach makes incompleteness visible, though it does not by itself measure the practical severity of each miss or how readily a lawyer could correct it. The score is thus a signal for evaluation, not a substitute for human review and citation verification.
Cost is another point of interest, but the leaderboard’s $0.15 per test figure is not a full procurement comparison. It does not, on its own, account for integration, review time, client-data controls or the cost of errors. Firms considering the model would need to examine those factors alongside benchmark performance.
Popular right now
De'Longhi Dinamica Plus Fully Automatic Espresso Machine
Hisense 65-inch U6 Pro Mini-LED 4K UHD Smart Fire TV
Ninja Foodi Dual Zone Smart XL Air Fryer, 10 Qt, 6-in-1, Black (DZ550)
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The benchmark contains 1,250 tasks across 24 legal practice areas. Each task presents an instruction and client matter materials, then asks an agent to produce work for review. Harvey says the tasks are graded against more than 75,000 expert-written, binary criteria.
Harvey’s stated rationale for all-pass grading is that a work product missing even one required element may be materially incomplete. Its public benchmark release includes the task materials, while leaderboard submissions are assessed against a private holdout set. Vals’ archived leaderboard identifies that holdout as 120 tasks and says two AI judges evaluate submissions, with their task pass rates averaged.
The broader benchmark figures provide a caution against treating any one model’s result as a measure of legal competence. In a May 26 baseline, Harvey reported that the best overall all-pass rate among the frontier models it tested was 7.1%, and said the tested models completed fewer than 10% of tasks end to end in aggregate. That was a separate run and is not directly comparable with the later Vals leaderboard.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model.””
— Z.ai, in its model documentation
What the Benchmark Cannot Establish
The results establish performance on a specific benchmark and private holdout, not accuracy across legal work generally. The available leaderboard information does not establish how performance varies by practice area, task difficulty or the type and severity of errors. It also does not show how the model performs when connected to the external legal and company-data services Z.ai recommends for agent use.
Z.ai describes the result as state-of-the-art among open-source models, but the supplied material does not define the comparison set, and it does not establish that the legal model’s weights are publicly available. The claim should therefore be read as Z.ai’s characterization, rather than as a conclusion independently verified by the leaderboard.
A separate four-task spot check described by ThorstenMeyerAI.com found citations to rulings that the author could not locate in responses from the legal model. That small check is not a benchmark result and cannot establish a general error rate. The same account says two Claude models cited real rulings for the opposite of what they held. The source also says the Z.ai pages reviewed described outputs as drafts for legal professionals but did not include a “not legal advice” notice. The available material does not settle how client data is handled, what safeguards apply, or who is responsible for checking each citation before use.
Questions for Legal Buyers
The next useful evidence would be updated leaderboard results, clearer documentation of the comparison set behind Z.ai’s open-source claim, and evaluations that report error types and performance across legal practice areas. Vals’ archived figures are a dated snapshot; later leaderboard changes or benchmark versions could alter both rankings and comparisons.
For law firms assessing the model, the immediate next steps are practical: test representative matters under controlled conditions, check citations and other high-impact details, and review data processing, access controls and service terms before supplying client material. Z.ai’s documentation recommends connecting the model to services for laws, cases and company data, making the security and provenance of those connected sources relevant to any deployment decision.
No further benchmark release or independent legal-practice evaluation is specified in the available material. Until more evidence is published, the confirmed takeaway remains a seventh-place leaderboard result with a low all-pass task rate and a much higher individual-criteria rate—not proof that the system can safely complete legal work without professional review.
Key Questions
What does GLM-5.3-FlashX-Legal’s 19.17% score mean?
On Vals AI’s archived leaderboard, it is the share of benchmark tasks on which the model passed every required criterion. A task with even one criterion missed does not count as a pass.
How can the model have a 92.5% criteria pass rate but a 19.17% task rate?
The figures measure different things. The criteria rate counts individual checks passed, while the task rate counts only tasks for which all checks passed. A model can pass most individual checks while still missing at least one on many tasks.
Did the legal model improve on Z.ai’s base model?
On the archived Vals Version 1 leaderboard, GLM Legal scored 19.17% and GLM 5.3 Flash scored 6.67%. The difference is 12.50 percentage points, consistent with Z.ai’s stated improvement. This is a comparison within that leaderboard, not proof of performance across all legal work.
Does seventh place prove the model is the best open-source legal AI?
No. Seventh place is its ranking among 81 models on the listed leaderboard. Z.ai also claims state-of-the-art performance among open-source models, but the supplied material does not define that comparison group or establish that the legal model’s weights are publicly available.
Can law firms use the model without checking its work?
The benchmark does not establish that. Its results leave room for missed criteria, and a small spot check cited by ThorstenMeyerAI.com reported rulings that the author could not locate. Firms would need to test the system, verify citations and review data-handling terms before using it on client matters.
Source: ThorstenMeyerAI.com
