Determining Model Size and Training Horizon through Scaling Laws

1. Introduction

How big should the model be, and how long should we train it? A fitted scaling law gives a training-compute optimum, but that result changes when unique data is limited or the model will serve many inference tokens. Converting the resulting recipe into wall-clock time also requires a model of cluster efficiency.

The derivations below answer four questions: what allocation minimizes loss under a training-compute budget; how repeated data changes the effective token count; how much extra training a smaller model needs to match a baseline loss; and how MFU and goodput convert model FLOPs into elapsed time.

1.1. Problem formulation and notation

The approach starts from the parametric loss used in Training Compute-Optimal Large Language Models for a fixed model family. We define:

• $N$: Number of model parameters (model size).
• $D$: Number of training tokens processed: the sum of non-padding tokens in every global batch over all training steps. For fixed-length sequences, this reduces to steps times sequences per global batch times sequence length.
• $C_{\text{train}}$: Available training compute budget, measured in floating point operations (FLOPs) or an equivalent unit. This constrains how large $N$ and $D$ can be (since more parameters or more tokens both consume more compute).
• $M$: Projected number of inference tokens the model will process over its lifetime, including both input and generated tokens. For example, one billion requests averaging 1,000 total tokens each gives $M = 10^{12}$.
• $D_c$: Number of unique tokens in the available training corpus (the size of the dataset). If $D > D_c$, it means the dataset will be repeated to supply that many training tokens.
• Loss function $L(N, D)$: A proxy for model quality after training, given by the scaling law. Lower loss corresponds to a better model. We will use a parametric form informed by the Chinchilla paper:

$$L(N, D) = A N^{-\alpha} + B D^{-\beta} + E$$

where the data and model architecture determine the fitted coefficients $A$, $\alpha$, $B$, $\beta$, and $E$. The first two terms model the empirical power-law improvement with model and data scale. $E$ is the irreducible loss approached as $N, D \to \infty$.

These coefficients can be estimated by fitting the scaling law to a ladder of smaller training runs.

2. The Training-Compute Optimum

2.1. Training compute only

The classic compute-optimal training problem chooses $N$ and $D$ to minimize loss under a fixed training budget. For a dense decoder-only Transformer, a common model-FLOPs approximation is $C_{\text{train}} = \eta_{\text{train}} N D$ with $\eta_{\text{train}} \approx 6$. This approximation does not include every optimizer, communication, or hardware operation.

$$\begin{aligned} & \min_{N, D} \; L(N, D) \\ & \text{s.t.} \quad \eta_{\text{train}} N D = C_{\text{train}} \end{aligned}$$

Substituting $D=C_{\text{train}}/(\eta_{\text{train}}N)$ into the loss and differentiating gives the compute-optimal parameter and token counts:

$$N_{\text{opt}}(C_{\text{train}}) = G \left(\frac{C_{\text{train}}}{\eta_{\text{train}}}\right)^{\frac{\beta}{\alpha+\beta}}, \quad D_{\text{opt}}(C_{\text{train}}) = G^{-1} \left(\frac{C_{\text{train}}}{\eta_{\text{train}}}\right)^{\frac{\alpha}{\alpha+\beta}}, \quad G = \left(\frac{\alpha A}{\beta B}\right)^{\frac{1}{\alpha+\beta}}$$

2.2. Repeated data

If $D>D_c$, the training run repeats data. Following the repeated-data component of Scaling Data-Constrained Language Models, define the unique token count processed as $U=\min(D,D_c)$ and the number of repetitions as $R=D/U-1$. The effective token count is

$$D_{\text{eff}} = U\left[1 + R^{*}\left(1-e^{-R/R^{*}}\right)\right],$$

where $R^{*}$ is fitted from repeated-data experiments. When there is no repetition, $R=0$ and $D_{\text{eff}}=D$. As repetition grows, $D_{\text{eff}}$ approaches $U(1+R^{*})$ even though the processed-token count continues to grow.

Replacing $D$ in the loss term, but not in the training-cost constraint, gives

$$\begin{aligned} & \min_{N, D} \; L\!\left(N, D_{\text{eff}}(D,D_c)\right) \\ & \text{s.t.} \quad \eta_{\text{train}} N D = C_{\text{train}} \end{aligned}$$

This is a simplified use of the cited law. Its full formulation also discounts parameters that are excessive for the available unique data; the optimization above models only diminishing returns from token repetition.

3. Adding Inference Demand

For a dense Transformer with $N$ parameters, the model FLOPs for a forward pass are commonly approximated as $\eta_{\text{inf}}N$ per token, with $\eta_{\text{inf}}\approx2$. Serving $M$ tokens therefore adds approximately $\eta_{\text{inf}}NM$ model FLOPs. As in Beyond Chinchilla-Optimal, this assumes that lifetime inference demand can be estimated before training and does not change with model size at fixed quality.

Two scenarios incorporate inference into the compute-optimality problem.

3.1. Fixed lifetime compute

The cited inference-aware analysis states the problem in the other direction: minimize lifetime compute at a fixed target loss. Under a monotonic loss frontier, we can use the dual view below to minimize loss under a fixed lifetime-compute budget and a known $M$:

$$\begin{aligned} & \min_{N, D} \; L(N, D) \\ & \text{s.t.} \quad \eta_{\text{train}} N D + \eta_{\text{inf}} N M = C \end{aligned}$$

3.2. Extra training required by a smaller model

Start from the training-compute-optimal model as a quality baseline. A smaller model can be trained on more tokens to match its predicted loss. This costs more training compute than the baseline allocation but reduces model FLOPs per inference token.

Set the smaller model to $k_N N_{\text{opt}}$ parameters, where $0<k_N<1$, and train it on $k_D D_{\text{opt}}$ tokens, where $k_D>0$. Requiring it to match the compute-optimal loss gives:

$$L(N_{\text{opt}}, D_{\text{opt}}) = L(k_N N_{\text{opt}}, \; k_D D_{\text{opt}})$$

Solving for $k_D$ gives

$$k_D = \left(1 - (k_N^{-\alpha} - 1) \frac{A N_{\text{opt}}^{-\alpha}}{B D_{\text{opt}}^{-\beta}}\right)^{-\frac{1}{\beta}}$$

This solution exists only when the quantity inside the parentheses is positive. Below that minimum $k_N$, the model term alone exceeds the target loss, so no finite increase in training tokens can match the baseline under this scaling law.

At the compute optimum, $A N_{\text{opt}}^{-\alpha}/(B D_{\text{opt}}^{-\beta}) = \beta/\alpha$. Define the baseline training compute as $C_{\text{opt}}=\eta_{\text{train}}N_{\text{opt}}D_{\text{opt}}$. Substituting the optimum relation, the smaller model’s training compute and overhead are

$$C_{\text{new}} = \eta_{\text{train}} \cdot k_N N_{\text{opt}} \cdot k_D D_{\text{opt}} = k_N k_D \, C_{\text{opt}}$$ $$\rho_{\text{overhead}} = \frac{C_{\text{new}} - C_{\text{opt}}}{C_{\text{opt}}} = k_N \left[1 - \frac{\beta}{\alpha}\left(k_N^{-\alpha} - 1\right)\right]^{-\frac{1}{\beta}} - 1$$

4. From Model FLOPs to Hardware Time

Model FLOPs do not determine elapsed time by themselves. The conversion depends on the hardware ceiling and the fraction of allocated time and throughput that the training job uses.

4.1. Training efficiency

For capacity planning, I keep execution efficiency and job availability separate. Here MFU means model-FLOP throughput during productive training steps divided by the hardware’s peak throughput; it excludes checkpointing, recovery, and input stalls. Goodput is the fraction of allocated wall-clock time spent executing those productive steps. With these definitions, multiplying the two factors does not count downtime twice.

Let $F_{\text{peak}}$ be peak FLOP/s per device and $N_{\text{dev}}$ the number of devices. The cluster’s aggregate ceiling is $S_{\text{cluster}}=F_{\text{peak}}N_{\text{dev}}$. The elapsed training time is

$$T_{\text{wall}} = \frac{C_{\text{train}}}{\rho_{\text{MFU}} \cdot \rho_{\text{goodput}} \cdot S_{\text{cluster}}}.$$

If $T_{\text{wall}}$ is measured in hours, the allocation consumes $N_{\text{dev}}T_{\text{wall}}$ GPU-hours. MFU can depend on model geometry, batch size, sequence length, and parallel layout, so two equal-FLOP training recipes need not take equal time on the same cluster.

4.2. Inference efficiency

The approximation $\eta_{\text{inf}}NM$ is useful for comparing model FLOPs, but it is not a serving-cost model. Measured throughput depends on prompt and generation lengths, batching, KV-cache memory, quantization, latency targets, and parallel layout. A deployment calculation should therefore optimize measured tokens per second per host subject to latency and memory constraints. Training MFU should not be reused as an inference-efficiency estimate.