🔍 Read the full analysis: Why Does Fine-Tuning Take Longer Than Its FLOPs Estimate? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face reports that a CatBoost model predicted runtime for 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP-based estimate had a 12.41-hour error, but the comparison comes from one production environment and does not show whether the predictor saves time or money in a live scheduler.
Hugging Face says a CatBoost model predicted the end-to-end duration of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation, as detailed in the original analysis. The best tested FLOP-based estimate had a 12.41-hour error, a difference that could help GPU operators plan workloads, though the result has so far been reported for one corporate production environment.
The jobs ran on shared NVIDIA H200 GPUs and accounted for about 7,382 GPU-hours. Their durations ranged from 1.6 to 172.4 hours. The dataset included model families up to 15 billion parameters, LoRA and full fine-tuning, different dataset sizes, and allocations of one, two or four GPUs. The machines came from a pool of 8 to 32 H200s across one to four nodes, while other training and inference work ran alongside them, illustrating how AI bottlenecks extend beyond the models.
Hugging Face defines runtime as wall-clock time from GPU allocation until the job finishes; time spent waiting in a queue is excluded. The CatBoost model used 24 features, including configuration and model metadata available before dispatch, such as evaluation cadence, checkpoint settings, token volumes per GPU and fine-tuning regime. It also used a FLOP-based estimate as one input, so the comparison is not between metadata and compute in isolation.
In the reported five-fold cross-validation, CatBoost had an R² of 0.98, a 3.52-hour root mean squared error and a median error of 0.51 hours. The FLOP baseline had a 12.41-hour mean absolute error at 30% Model FLOPs Utilization (MFU), the best of the tested settings. Its error rose to 13.01 hours at 45% MFU and 14.62 hours at 60%. Hugging Face says testing a wider range from 25% to 65% did not resolve the baseline’s shortcomings.
Why Fine-Tuning Takes Longer Than FLOPs Say
Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.
What the estimates capture
The target is time from GPU allocation to job completion; queue time is excluded.
The training loop
FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.
The whole job
CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.
Other runtime costs
Setup, data handling, network and storage can add time alongside training.
Prediction error in the tested data
Lower error means predicted duration was closer to observed duration on average.
What this could mean for GPU scheduling
More useful duration estimates could help teams plan scarce GPU capacity.
- GPUs may sit idle until another job starts.
- Operators can misjudge when capacity will be free.
- Other work can be delayed.
- Delivery dates may shift.
How far the result reaches
The preview reports cross-validation on one corporate production environment.
Runs used shared NVIDIA H200 GPUs and varied configurations.
Accuracy on other clusters, software stacks and workload mixes is not established.
The preview reports prediction error, not savings in GPU time or scheduling delays.
Runtime forecasts inform when operators dispatch queued work and how they allocate scarce accelerators. If a job finishes earlier than forecast, GPUs can sit idle until another job starts. If it runs longer, it can delay later work and shift delivery schedules. More accurate estimates could help operators plan around those mismatches, especially in clusters where multiple workloads compete for compute, network and storage resources.
The reported error gap is large within this dataset, but it is a measure of prediction accuracy, not operational savings. The preview does not describe a controlled scheduler deployment or quantify changes in GPU utilization, cost, idle time or delays. Whether improved estimates produce those benefits remains to be tested.
Popular right now

Dyson Airwrap i.d. Multi-Styler with Bluetooth Connectivity

Vitamix VX1 Professional-Grade Blender, 64 oz, Black

De'Longhi Magnifica Duo Fully Automatic Espresso Machine with Milk Frother, Silver
As an affiliate, we earn on qualifying purchases.
What FLOP Estimates Leave Out
A common training estimate approximates dense Transformer compute using parameter count and training-token count, then divides by GPU throughput and an assumed Model FLOPs Utilization rate. MFU represents the share of theoretical peak compute achieved in practice. The source describes typical MFU as ranging broadly from 30% to 60%, reflecting that real training does not run at peak theoretical speed.
More fundamentally, a compute estimate may cover the training loop but not the full job lifecycle. Model loading, tokenization, evaluation, checkpointing and storage can add wall-clock time. Their effects depend on job settings and on shared infrastructure, where network or storage traffic may be contested. Hugging Face says it included scheduled evaluation tokens in the baseline comparison and tested multiple MFU assumptions, rather than relying on one rate.
The study’s target is broader than compute time alone: it measures from GPU assignment through completion. Its approach uses job details known before dispatch to account for factors that a simple FLOP calculation may not capture. That makes the result relevant to scheduling, but it also means the model’s accuracy may depend on how closely another cluster’s configurations and operating conditions resemble the training data.
“Everything is known before dispatch.”
— Hugging Face, in its study preview
How Far the Results Carry
The reported scores come from cross-validation within 272 runs from one production pool. They do not establish how the model performs on independently collected clusters, other GPU types, different software stacks or changing workload mixes. The preview also does not report a later-period test set or a live scheduling trial, so it is not possible to tell from these results alone whether accuracy holds as conditions change.
Hugging Face has not yet made the full paper available, and the preview gives no publication date. Further details about the evaluation design, feature selection and dataset may appear there. It is also unclear how often the predictor would need retraining as jobs and shared-cluster conditions evolve. The evidence supports a finding about prediction error on this dataset, not a general claim that the model will reduce costs or delays elsewhere.
Full Paper and External Tests
Hugging Face says it plans to post the full paper on arXiv, but has not announced when. That paper may provide more detail about the dataset and evaluation methods. For the result to support wider scheduling use, the model would need to be tested on workloads collected in other periods or environments, then evaluated in a live scheduler.
Those tests could establish whether the low cross-validation error carries over beyond the original production pool and whether improved forecasts lead to fewer idle GPU intervals or delays. Until such evidence is available, the reported comparison is best treated as a promising result for this dataset rather than a proven operational gain.
Key Questions
What did Hugging Face report?
It reported that a CatBoost model predicted runtime for 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP estimate had a 12.41-hour error.
What does the runtime measure?
It measures wall-clock time from GPU allocation until the job finishes. Queue waiting time is excluded, while the measured duration can include setup, evaluations, checkpointing and other job activity.
Why can a FLOP estimate miss the full duration?
A FLOP calculation estimates computational work, but end-to-end runtime can also reflect model loading, tokenization, evaluations, checkpoint uploads and competition for shared network or storage resources.
Does the result show that GPU costs will fall?
No. The preview reports prediction errors, not a controlled deployment measuring cost, idle time or delays. Operational savings have not been established.
Has the finding been validated on other clusters?
Not in the material available so far. The reported cross-validation used runs from one corporate production environment; results from other clusters or independently collected periods are not reported.
Primary source: Hugging Face · via ThorstenMeyerAI.com
