🔍 Read the full analysis: The Law-Focused AI’s Score On 15 Legal Questions: 8/100 on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A test published by ThorstenMeyerAI.com gave Z.ai’s GLM-5.3 FlashX Legal 8 out of 100 across 15 legal questions, with no points on five hidden tasks. The publisher says the model fully solved none of the tasks and left two unanswered, but discloses that it runs the test and that Claude agents grade answers. The result is a limited spot check, not proof of how the model performs in every legal setting.
GLM-5.3 FlashX Legal, a model Z.ai describes as optimized for professional legal workflows, scored 8 out of 100 on a 15-question legal AI test published November 4 by ThorstenMeyerAI.com. The test’s publisher, Thorsten Meyer, says the model fully solved none of the tasks and scored zero on the five hidden questions; he also discloses that his company runs the test and that Claude agents grade answers.
Lexassay tested models on questions covering German, U.S. and EU law, including legal deadlines, recent changes, court decisions and questions built around false assumptions. The models answered without internet access or tools. GLM-5.3 FlashX Legal produced 21 citations classified as real, alongside 25 that Meyer classified as misattributed and one as invented. The published account says misattributed or invented citations reduce a task’s score.
The model also gave no answer to two German-law questions after using its full 64,000-token thinking budget, according to Meyer. In one example, it treated a challenge to the EU–U.S. Data Privacy Framework as pending, while the test’s account says the EU General Court dismissed the challenge on September 3, 2025. These examples are the publisher’s descriptions of the model’s responses and the grading process.
The same test gave Mistral Large 4 a score of 10. Meyer says it often got the substance partly right but lost points over citations. The highest reported scores were Claude Opus 5.5 at 79, GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the leading model met every criterion on only eight of the ten public questions, according to the article.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citation Errors Matter
The score draws attention to a practical risk: an answer can sound confident and still rely on a citation that is wrong, misattributed or no longer current. That matters especially when a person uses a chatbot to make decisions about court deadlines, claims or legal filings. The article’s opening example describes German appeal deadlines that differ by location because of a public holiday, illustrating how a one-day mistake could affect a case.
The results do not show that every legal AI answer is unreliable, nor do they establish how these models perform when connected to legal databases or supplied with case documents. Lexassay tested models without internet access or tools. Still, the findings offer a reason to treat generated legal answers as a starting point rather than verified advice, and to check cited laws and rulings in authoritative sources.
Popular right now
De'Longhi Dinamica Plus Fully Automatic Espresso Machine
Hisense 65-inch U6 Pro Mini-LED 4K UHD Smart Fire TV
Ninja Foodi Dual Zone Smart XL Air Fryer, 10 Qt, 6-in-1, Black (DZ550)
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay asked each model the same 15 questions: 10 public and five hidden. The hidden questions are scored but not published, according to Meyer, to reduce the chance that later models can learn their answers online. Questions span German, U.S. and EU law, and include prompts designed to test whether a model challenges an incorrect premise rather than accepting it.
Each answer was graded against a reference answer with three to six weighted criteria, and citations were checked against official databases, Meyer says. The test also deducts points when an answer lacks a legal-advice disclaimer or a suggestion to consult a lawyer. Meyer characterizes Lexassay as a spot check, not a strict benchmark: it uses one answer per model per task, and small score differences may reflect noise.
The publisher’s disclosure is relevant to interpreting the rankings. Thorsten Meyer AI runs Lexassay, and Claude agents graded the answers; a Claude model ranks first in the results. The article says the method page and public answers, reference answers and citation checks are available for review. Those disclosures allow readers to assess the process but do not independently resolve whether the grading is unbiased.
“optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution”
— Z.ai, in developer documentation as quoted by Thorsten Meyer
Limits of the Published Scores
The results come from a small set of questions and one answer for each model-task pairing. The publisher warns that small score differences may be noise; the results do not establish how often the models would make similar errors across a larger range of real legal work. The five hidden tasks are not publicly detailed, limiting outside review of those particular scores.
The source material also does not provide independent replication of the grading or a response from Z.ai to the 8/100 result. Because Claude agents graded the answers and a Claude model led the ranking, the disclosed conflict of interest remains relevant. The published method and checks may help readers examine the work, but they are not the same as an external audit.
The test does not evaluate models using web search, legal research tools or supplied documents. It therefore cannot settle how a legal AI product performs when configured differently, or whether a professional workflow changes the outcome. Its score is specific to the models, settings and tasks included in this test.
What Readers Should Check
Meyer advises readers to use AI responses as a starting point, verify every cited law and ruling in the original source, and avoid relying on a chatbot for deadlines or legal decisions. He recommends consulting a lawyer when money, a deadline or an authority is involved. The article says the test’s method page and published answer materials are available so readers can review how scores were assigned.
No follow-up test date or response from Z.ai is provided in the source material. For now, the published result remains a limited comparison under Lexassay’s stated conditions. Further testing across more questions, repeated model runs and tool-enabled settings would be needed to show whether the same patterns hold more broadly.
Key Questions
What score did GLM-5.3 FlashX Legal receive?
It scored 8 out of 100 on Lexassay’s 15 legal questions, according to ThorstenMeyerAI.com. The publisher reports a score of zero on the five hidden tasks.
What did the test ask the models?
The questions covered German, U.S. and EU law, including deadlines, legal changes, court decisions and prompts that contained false assumptions. Models answered without internet access or tools.
How reliable is the 8/100 result?
It is a result from a limited spot check, not a broad measure of every legal use. The publisher says each model answered each task once and warns that small score differences may be noise.
Who graded the answers?
Thorsten Meyer says Claude agents graded the answers against reference answers and that citations were checked against official databases. Meyer also discloses that his company runs Lexassay and that a Claude model leads the test.
Should people rely on these models for legal deadlines?
The test publisher advises readers not to take deadlines from a chatbot. Check the relevant law and court information, and consult a qualified lawyer when a deadline, money or an official authority is involved.
Source: ThorstenMeyerAI.com
