🔍 Read the full analysis: How To Scale Indic Pretraining With 2.8 Trillion Tokens on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
NVIDIA and AI4Bharat at IIT Madras say they plan to release Nemotron-pretraining-Indic-v1, a corpus with about 2.8 trillion retained tokens across 22 Indian languages, and the GPU-accelerated pipeline used to create it. The release date, licensing terms and detailed quality results have not been specified.
NVIDIA and AI4Bharat at IIT Madras say they plan to release Nemotron-pretraining-Indic-v1, a corpus containing about 2.8 trillion retained tokens across 22 Indian languages, together with an open pipeline for generating the data. The project could give language-model developers a larger starting resource for Indic-language training, but the organizations have not specified when it will be available or provided independent results showing its effect on model performance.
The announced inventory comprises 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer. Native-script material is included for all 22 languages; romanized text, which represents Indic-language content written in Latin letters, is included for 12. The project describes these as 34 language-script configurations.
The corpus is generated from English documents drawn from 15 Nemotron-CC-v2 subsets, covering web text, question-and-answer material, mathematics and code. The workflow uses NVIDIA’s NeMo Data Designer framework to translate documents, assess translation quality, repair outputs that score poorly and convert some accepted translations into Latin script. The announcement says records retain source identifiers and that checks compare outputs with inputs, including checks intended to preserve equations, tables and code.
The teams say developers will be able to use the corpus directly or adapt the prompts, workflow and quality checks to their own source materials. The source material cautions that records can include translations and script variants derived from shared documents. The token total is therefore not a count of unique source documents, and it should not be confused with the amount of data used in any particular model-training run.
A Larger Starting Point for Indic Models
A large corpus could help address a data constraint for teams building language models in Indian languages. The announced subject mix includes technical content and question-and-answer material, potentially useful for models designed to explain mathematics or programming concepts and respond to user queries. Those are possible uses, not demonstrated outcomes: the announcement offers no independent evaluation of model quality or evidence that the corpus improves downstream performance.
Publishing the generation pipeline could matter as much as publishing the text. If the code, prompts and checks are sufficiently documented, researchers may be able to inspect how translations were produced and adapt the process to other sources. Source identifiers and input-output checks could aid traceability. The actual value of those features will depend on the release materials, terms of use and the results of external review.
The corpus is not described as a substitute for naturally authored Indic-language material. Because its translation workflow starts with English documents, the resulting text may reflect the source material’s topics and framing rather than the full range of local knowledge and cultural context in original writing. The project says naturally authored Indic data remains essential.
Popular right now
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
Roborock Robot Vacuum and Mop with LiDAR Navigation and Self-Empty Dock
iRobot Roomba Plus 405 (G181) 2-in-1 Vacuum and Mop Robot with AutoWash Dock
As an affiliate, we earn on qualifying purchases.
From Sangraha to Synthetic Data
AI4Bharat is an Indian-language research lab at IIT Madras that develops open datasets and models. The announcement points to Sangraha, its open pretraining corpus for 22 Indian languages, as part of the foundation for research on Indic-language models. Nemotron-pretraining-Indic-v1 would add a large collection of translated text and a described method for generating more from selected English sources.
The project says its 22 languages are those listed in India’s Constitution and describes them as the first languages of 97% of Indians, or nearly 1.2 billion people, citing the 2011 Census. That population figure reflects the cited census and is not a current population estimate.
The announcement also reports that continued pretraining for Nemotron 3.5 Super VL used 10 billion Indic tokens from a 12-language subset of the corpus, alongside naturally authored Indic text. That reported training use is separate from the corpus inventory of about 2.8 trillion tokens; the figures refer to different quantities and the corpus count depends on the Gemma-4 tokenizer.
““We do not have enough data for Indian languages.””
— Mitesh Khapra, head of AI4Bharat
Release Terms and Quality Details
The announcement describes the release as planned, but gives no publication date or hosting location. It also does not specify redistribution or adaptation terms, provide per-language token totals, or disclose the complete filtering rules and acceptance thresholds. Those details will affect whether developers can use the resource and how researchers can reproduce or evaluate its contents.
No detailed translation-quality scores or independent assessments are provided. The announcement says the workflow evaluates and repairs translations, but readers do not yet have the results needed to judge fidelity across languages or subject areas. It also offers no performance comparison demonstrating the corpus’s effect on models, or a full account of how synthetic translations and naturally authored Indic text would be combined in future training.
The stated inventory includes accepted rewrites and script variants, and some records may originate from the same source document. As a result, the token count does not measure unique documents or establish that a model has trained on the entire corpus. Those distinctions will remain relevant when the data and documentation are available for review.
Publication and Independent Review
The next stated milestone is publication of the corpus and its generation pipeline. No date has been announced. After release, developers and researchers will be able to examine the data, check the documentation and terms, and assess whether the workflow can be reproduced or adapted to other source collections.
Reviewers will also be able to test translation quality across languages and content types, inspect the per-language distribution and examine the filtering criteria. Such work could clarify what the reported token totals represent in practice and where the corpus is useful or limited. Until those materials and evaluations are public, claims about its potential contribution to Indic-language model performance remain unverified.
Key Questions
What is Nemotron-pretraining-Indic-v1?
It is a planned corpus of about 2.8 trillion retained tokens across 22 Indian languages, which NVIDIA and AI4Bharat say they intend to release with the pipeline used to generate it.
How much of the corpus is romanized?
The announced inventory includes 224.6 billion romanized tokens, compared with 2,601.0 billion native-script tokens. The counts use the Gemma-4 tokenizer; romanized text is included for 12 of the 22 languages.
When will the corpus be available?
No release date has been specified. The announcement describes the corpus and pipeline as planned releases but does not name a publication date or hosting location.
Does the corpus prove better Indic-language model performance?
No such result is provided in the announcement. It reports a corpus inventory and a training use involving a 12-language subset, but gives no independent performance comparison showing the corpus improves model quality.
Is the corpus made entirely from original Indic-language writing?
No. The described workflow translates selected English-language documents and creates some Latin-script variants. The project says naturally authored Indic data remains essential.
Primary source: Hugging Face · via ThorstenMeyerAI.com
