📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva LLM, built from scratch with extensive Italian data, underperforms on academic benchmarks despite impressive technical results. This raises questions about the scale of native-language investment needed for country-specific language models.
Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian academic benchmark, despite its extensive Italian training data. This outcome questions assumptions about the relationship between training scale and language-specific performance, making it a significant development in the European sovereign-LLM movement.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research and supercomputing infrastructure, trained models ranging from 350 million to 7 billion parameters. The training data consisted of 2.5 trillion tokens, with roughly half Italian, making it one of Europe’s most ambitious efforts to develop a sovereign language model from scratch.
Despite these efforts, Minerva-3B’s performance on the INVALSI Italian school-exam benchmark was only 4.9%, a near-chance result that contrasts sharply with its impressive technical achievements and the large scale of its Italian data. Researchers noted that while dataset composition is important, the overall size of the dataset and the number of parameters are more critical for handling complex language tasks, suggesting that scale alone may not be sufficient to achieve deep country-specific knowledge.
This finding complicates the narrative that larger, native-language models automatically produce better country-knowledge performance and indicates that the European sovereign-LLM strategy may need to reconsider the scale of investment required for meaningful language-specific expertise.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Enhanced ACT Exam Success: ACT Prep Study Guide with Full-Length Practice Tests, Practice Questions, and Access to the Apex Realistic Enhanced ACT-Style Test Simulator
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
OpenAI Evals Cookbook: Designing Benchmarks for Product‑Grade LLM Features
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-LLM Strategies
The results from Minerva demonstrate that even large-scale, native-language training may not produce the desired depth of country-specific knowledge, raising questions about the effectiveness of current investment levels. This challenges the assumption that simply scaling up data and parameters will lead to better performance on complex, country-specific tasks.
The findings suggest that European efforts to develop sovereign LLMs must consider whether current scale and resource commitments are sufficient or if more targeted, higher-investment approaches are necessary to achieve meaningful language and knowledge mastery. This has broader implications for national AI strategies across Europe, emphasizing the need for realistic benchmarks and expectations.
Background on European Sovereign LLM Development
The European sovereign-LLM movement has seen various approaches, including Portugal’s AMÁLIA model, which extended a multilingual foundation with regional data, and Italy’s Minerva, which trained from scratch on extensive Italian data. Italy’s approach involved significant institutional coordination, including Sapienza University, the FAIR consortium, and CINECA’s supercomputing resources, reflecting a strategic effort to build indigenous AI capabilities.
Prior to Minerva, the common assumption was that larger native-language datasets and parameters would lead to superior performance on country-specific tasks. However, the recent results challenge this view by showing that scale alone may not be enough, especially when it comes to complex academic content and nuanced language understanding.
“The 4.9% score on the INVALSI benchmark reveals that even extensive Italian data and large models do not guarantee deep country-specific knowledge.”
— Research team of Minerva
Unresolved Questions About Scale and Effectiveness
It remains unclear how much larger or more specialized models are needed to achieve meaningful country-specific knowledge in European languages. The results from Minerva are limited to the current parameter scales and dataset sizes, and ongoing research may alter these conclusions. Additionally, the impact of different training methodologies or data curation strategies on performance is still under investigation.
Next Steps for European Language Model Development
The Minerva team and broader European AI community are expected to continue refining their models, experimenting with larger scales, different training approaches, and more targeted data. Further benchmarking on complex, real-world tasks will help determine the optimal investment levels needed for effective country-specific language models. Policymakers and researchers will likely reassess resource allocations and strategic priorities based on these evolving insights.
Key Questions
Why did Minerva perform poorly on the Italian academic benchmark?
Despite extensive Italian data and large model sizes, the results suggest that scale alone is insufficient for deep country-specific knowledge. Factors like data quality, training methodology, and task complexity also play critical roles.
Does this mean native-language models are not worth investing in?
Not necessarily. The findings highlight that current scale levels may be inadequate and that more targeted or larger investments might be required to achieve desired performance levels in specific languages or tasks.
How do these results affect the European sovereign-LLM strategy?
The results suggest that European efforts should reconsider assumptions about scale and focus on developing models that balance size, data quality, and task-specific training to better meet national and linguistic needs.
Will future models overcome these performance limitations?
Ongoing research aims to explore larger models, different training paradigms, and better data curation, which may improve performance. However, the exact scale and approach needed remain uncertain.
Source: ThorstenMeyerAI.com
