Why Does Fine-Tuning Take Longer Than Its FLOPs Estimate?
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Why Does Fine-Tuning Take Longer Than Its FLOPs Estimate? on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face reports that a CatBoost model predicted runtime for 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP-based estimate had a 12.41-hour error, but the comparison comes from one production environment and does not show whether the predictor saves time or money in a live scheduler.

Hugging Face says a CatBoost model predicted the end-to-end duration of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation, as detailed in the original analysis. The best tested FLOP-based estimate had a 12.41-hour error, a difference that could help GPU operators plan workloads, though the result has so far been reported for one corporate production environment.

The jobs ran on shared NVIDIA H200 GPUs and accounted for about 7,382 GPU-hours. Their durations ranged from 1.6 to 172.4 hours. The dataset included model families up to 15 billion parameters, LoRA and full fine-tuning, different dataset sizes, and allocations of one, two or four GPUs. The machines came from a pool of 8 to 32 H200s across one to four nodes, while other training and inference work ran alongside them, illustrating how AI bottlenecks extend beyond the models.

Hugging Face defines runtime as wall-clock time from GPU allocation until the job finishes; time spent waiting in a queue is excluded. The CatBoost model used 24 features, including configuration and model metadata available before dispatch, such as evaluation cadence, checkpoint settings, token volumes per GPU and fine-tuning regime. It also used a FLOP-based estimate as one input, so the comparison is not between metadata and compute in isolation.

In the reported five-fold cross-validation, CatBoost had an R² of 0.98, a 3.52-hour root mean squared error and a median error of 0.51 hours. The FLOP baseline had a 12.41-hour mean absolute error at 30% Model FLOPs Utilization (MFU), the best of the tested settings. Its error rose to 13.01 hours at 45% MFU and 14.62 hours at 60%. Hugging Face says testing a wider range from 25% to 65% did not resolve the baseline’s shortcomings.

At a glance
reportWhen: Study preview published; full paper and…
The developmentHugging Face’s study preview reports that a model using job metadata predicted fine-tuning runtimes more accurately than tested FLOP-based estimates on 272 production jobs.
AI Training · Briefing

Why Fine-Tuning Takes Longer Than FLOPs Say

Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.

1.71 hours
CatBoost mean absolute error
12.41 hours
Best tested FLOP baseline error
272
Production jobs evaluated
7,382
Approximate GPU-hours

What the estimates capture

The target is time from GPU allocation to job completion; queue time is excluded.

Compute

The training loop

FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.

Configuration

The whole job

CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.

Shared systems

Other runtime costs

Setup, data handling, network and storage can add time alongside training.

Prediction error in the tested data

Lower error means predicted duration was closer to observed duration on average.

CatBoost1.71 hours
FLOP baseline, 30% MFU12.41 hours
FLOP baseline, 45% MFU13.01 hours
FLOP baseline, 60% MFU14.62 hours

What this could mean for GPU scheduling

More useful duration estimates could help teams plan scarce GPU capacity.

When jobs finish early
  • GPUs may sit idle until another job starts.
  • Operators can misjudge when capacity will be free.
When jobs run over
  • Other work can be delayed.
  • Delivery dates may shift.

How far the result reaches

The preview reports cross-validation on one corporate production environment.

1
Tested on one pool

Runs used shared NVIDIA H200 GPUs and varied configurations.

2
Generalization remains open

Accuracy on other clusters, software stacks and workload mixes is not established.

3
Operational impact remains unmeasured

The preview reports prediction error, not savings in GPU time or scheduling delays.

What to watchThe full paper is forthcoming; validation on independent runs and a live scheduler trial would show whether the result carries over.
The takeaway
Configuration-aware prediction beat the tested FLOP estimates on these jobs, but teams need broader validation before relying on the same accuracy for their own clusters.

Better Forecasts for Shared GPU Pools

Runtime forecasts inform when operators dispatch queued work and how they allocate scarce accelerators. If a job finishes earlier than forecast, GPUs can sit idle until another job starts. If it runs longer, it can delay later work and shift delivery schedules. More accurate estimates could help operators plan around those mismatches, especially in clusters where multiple workloads compete for compute, network and storage resources.

The reported error gap is large within this dataset, but it is a measure of prediction accuracy, not operational savings. The preview does not describe a controlled scheduler deployment or quantify changes in GPU utilization, cost, idle time or delays. Whether improved estimates produce those benefits remains to be tested.

De'Longhi Magnifica Duo Fully Automatic Espresso Machine with Milk Frother, Silver

De'Longhi Magnifica Duo Fully Automatic Espresso Machine with Milk Frother, Silver

As an affiliate, we earn on qualifying purchases.

What FLOP Estimates Leave Out

A common training estimate approximates dense Transformer compute using parameter count and training-token count, then divides by GPU throughput and an assumed Model FLOPs Utilization rate. MFU represents the share of theoretical peak compute achieved in practice. The source describes typical MFU as ranging broadly from 30% to 60%, reflecting that real training does not run at peak theoretical speed.

More fundamentally, a compute estimate may cover the training loop but not the full job lifecycle. Model loading, tokenization, evaluation, checkpointing and storage can add wall-clock time. Their effects depend on job settings and on shared infrastructure, where network or storage traffic may be contested. Hugging Face says it included scheduled evaluation tokens in the baseline comparison and tested multiple MFU assumptions, rather than relying on one rate.

The study’s target is broader than compute time alone: it measures from GPU assignment through completion. Its approach uses job details known before dispatch to account for factors that a simple FLOP calculation may not capture. That makes the result relevant to scheduling, but it also means the model’s accuracy may depend on how closely another cluster’s configurations and operating conditions resemble the training data.

“Everything is known before dispatch.”

— Hugging Face, in its study preview

How Far the Results Carry

The reported scores come from cross-validation within 272 runs from one production pool. They do not establish how the model performs on independently collected clusters, other GPU types, different software stacks or changing workload mixes. The preview also does not report a later-period test set or a live scheduling trial, so it is not possible to tell from these results alone whether accuracy holds as conditions change.

Hugging Face has not yet made the full paper available, and the preview gives no publication date. Further details about the evaluation design, feature selection and dataset may appear there. It is also unclear how often the predictor would need retraining as jobs and shared-cluster conditions evolve. The evidence supports a finding about prediction error on this dataset, not a general claim that the model will reduce costs or delays elsewhere.

Full Paper and External Tests

Hugging Face says it plans to post the full paper on arXiv, but has not announced when. That paper may provide more detail about the dataset and evaluation methods. For the result to support wider scheduling use, the model would need to be tested on workloads collected in other periods or environments, then evaluated in a live scheduler.

Those tests could establish whether the low cross-validation error carries over beyond the original production pool and whether improved forecasts lead to fewer idle GPU intervals or delays. Until such evidence is available, the reported comparison is best treated as a promising result for this dataset rather than a proven operational gain.

Key Questions

What did Hugging Face report?

It reported that a CatBoost model predicted runtime for 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The best tested FLOP estimate had a 12.41-hour error.

What does the runtime measure?

It measures wall-clock time from GPU allocation until the job finishes. Queue waiting time is excluded, while the measured duration can include setup, evaluations, checkpointing and other job activity.

Why can a FLOP estimate miss the full duration?

A FLOP calculation estimates computational work, but end-to-end runtime can also reflect model loading, tokenization, evaluations, checkpoint uploads and competition for shared network or storage resources.

Does the result show that GPU costs will fall?

No. The preview reports prediction errors, not a controlled deployment measuring cost, idle time or delays. Operational savings have not been established.

Has the finding been validated on other clusters?

Not in the material available so far. The reported cross-validation used runs from one corporate production environment; results from other clusters or independently collected periods are not reported.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

AI And The Unfulfilled Dream Of Working 15 Hours A Week

A November 2026 report revisits Keynes’s 15-hour-week idea as incomes rise but working hours stay near 40 and AI savings have yet to shorten schedules.

Can LLMs Reliably Assess Whether A Research Idea Is Novel?

Ai2 tested six LLMs and found novelty judgments could shift sharply with evaluation choices; retrieval and added reasoning did not consistently help.

SEALSQ And Eco Wave Power Demonstrate Digital Energy Transactions For AI Data Centers

A Jaffa Port demonstration linked wave-energy production data to a digital twin and an auditable transaction, but involved no electricity sale.

Mapping The Detection Gaps In Rhysida’s Attack On Berlin

A Berlin department told rbb its attack-detection system covered 5% of devices; Rhysida later published about 1.4 million stolen files.