A Better Test Score Didn’t Change My News Pipeline’s AI Model

🔍 Read the full analysis: A Better Test Score Didn’t Change My News Pipeline’s AI Model on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he will keep GPT-6 Luna as the production model for his news pipeline, despite GPT-6 Astra scoring higher in a test of 71 model configurations. In a report published October 13, Meyer said Astra produced the strongest articles, but cost about $0.12 per article, compared with roughly $0.0012 for Luna, making the lower-cost model his choice for routine publishing.

The test ran from September 8 to October 13, 2026, using Meyer’s production prompt and real news sources. Each source was turned into an English and German article, then evaluated by two AI judges: GPT-6.1 Sol and Claude Opus 5.5. Their rubric covered source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with German-language quality also scored.

Astra scored 4.00 out of 5 in each language in the standard test and had perfect source-fidelity scores there. In a larger robustness run, it averaged 3.93 across 12 articles. Luna scored 3.67 in that run, with source fidelity of 4.71 out of 5, and all of its outputs were valid under the pipeline’s repair process. Meyer reported a median production time of 17 seconds per Luna article, compared with about 50 seconds for Astra.

The report also says higher reasoning settings did not produce a reliable quality gain for Luna. At maximum effort, the model took 211 seconds and cost about 1.43 cents per article, versus 17 seconds and about 0.12 cents at the production setting, with scores Meyer considered within the test’s noise. He also said two backup models were removed after producing fabricated quotations or invalid output.

Why the Cheaper Model Stayed

The result illustrates the difference between a model that performs best in a test and one that makes sense for repeated production use. Astra led on writing quality in Meyer’s evaluation, but its reported per-article cost was about 100 times Luna’s. For a high-volume publisher, that gap can shape operating costs, while generation time affects how quickly an article can move through a pipeline.

The findings also point to reliability as a separate concern from average scores. Meyer said the two removed fallback models generated invented quotations, a serious problem for news output because readers may mistake fabricated statements for real sourcing. A pipeline’s backup choices can therefore affect whether a system preserves source fidelity when its primary model is unavailable.

These are results from one publisher’s task and evaluation setup, not a general ranking of AI systems. They do, however, show why organizations using models for publication may need to test them on their own sources, prompts and output checks rather than relying only on broad benchmarks.

Testing Models on Real Articles

Meyer’s comparison focused on a specific job: turning a source into a finished article using the prompt already used by his sites. The two test sources differed substantially in length: one was a 6,340-character product filing, while the other was a 619-character political news item. Meyer said short sources better represent much of his production work, so the test included a robustness run of 10 additional articles, mostly based on thin sources.

The report says the judges both received the original source and scored outputs on a five-point scale. Meyer treated differences below about 0.3 points in the initial two-article comparison as potentially attributable to noise, and said the robustness run offered a more reliable ranking. Scores were reported with 95% bootstrap intervals. An output counted only if it remained usable after the same repair step applied in production.

The report gives examples of rankings that changed across tasks. Claude Opus 5.5, which Meyer described as his coding model, scored 3.46 in English and 3.39 in German in the article test; its robustness score was 3.52. Claude Sonnet 5.5 scored close to Luna, but Meyer estimated its API-equivalent article cost at about $0.036. These comparisons reflect Meyer’s test, rather than independent validation of the models’ broader capabilities.

“For a high-volume task, the right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13 report

Limits of the Model Comparison

The results come from one publisher’s workflow, a limited set of sources and a judging process carried out by two AI systems. The report does not establish how the same configurations would perform with other prompts, subject areas, editorial standards or human editors. Nor does it provide an independent audit of the judges’ scoring or the reported costs.

Meyer says the robustness run used 10 further articles, mostly from short sources, but the report does not establish that this sample represents all routine news assignments. The reported score intervals help describe uncertainty within the test; they do not remove the limits of its scope. The findings also do not show whether the models’ performance or prices will remain stable as services change.

The account identifies fabricated quotations in outputs from the two backup models and says they were removed. It does not describe the full safeguards now used in the route, or provide results for a larger test of the replacement fallback setup. The production choice is Meyer’s decision for his sites, not evidence that Luna is the right model for every news operation.

Production Remains on Luna

According to the report, GPT-6 Luna remains the production model, while GPT-6 Astra is not being used for routine articles because of its higher cost. Meyer said the production configuration uses the “none” reasoning setting, which had the lowest reported cost and time among the six settings tested without a clear quality penalty in his comparison.

The two fallback models were removed from the news route on October 7. The report does not give a detailed account of the current failover arrangement beyond stating that Luna now runs first, and its final description of the backup is incomplete. Further information about replacement fallbacks, continued monitoring and any later benchmark results remains to be reported.

Key Questions

Which model scored highest in Meyer’s test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. Its robustness-run average was 3.93 across 12 articles.

Why did Meyer keep GPT-6 Luna in production?

Meyer said Luna scored 3.67 in the robustness run, produced valid outputs in every tested case and cost roughly $0.0012 per article, compared with about $0.12 for Astra.

Did higher reasoning effort improve Luna’s articles?

Meyer reported no reliable improvement across the six effort settings. Maximum effort took about 211 seconds and cost roughly 1.43 cents per article, while the production setting took 17 seconds and cost about 0.12 cents.

What problems did the fallback models produce?

Meyer said GPT-4.1 nano generated invented quotations and received the lowest source-fidelity score. He reported that local gpt-oss-120b produced no valid articles out of four and also invented named quotes. Both were removed from the route on October 7.

Do these results establish the best AI model for news writing overall?

No. The findings describe one publisher’s prompt, sources, languages and evaluation method. They do not establish how the models would perform across other newsrooms or editorial workflows.

Source: ThorstenMeyerAI.com