On the smallness of the large language model scaling exponents

1 Introduction

AI and most notably, transformer-based large language models (LLMs), have taken science and society by storm in the recent past [1–3]. Much of this explosive growth is driven by the “no-wall” finding, namely, the fact that the learning capability of LLM-based chatbots increases with size, whence the mantra “bigger is better” that has driven much of the current leading-edge LLM research.

Let us recapitulate the main ideas behind the no-wall finding.

The scaling law of LLMs is usually expressed in the formwhere L is the loss function, a measure of the departure of the LLM outcome from the desired target, and N stands for data ( D ), number of parameters ( P ), and computational cost ( C ), respectively, each characterized by its constant prefactor A and exponent α [4, 5]. A positive value of the exponent indicates that the loss function decreases for increasing data size so that using larger data sets leads to a closer match to the desired target. Formally, the “wall” corresponds to the opposite regime, in which more data lead to a larger discrepancy between model and target, as formally reflected by a negative exponent. The LLM exponents so far have remained positive across several decades, whence the enthusiastic claim that “there is no wall,” supporting a strategy oriented towards ever larger LLM applications that consume even more energy [6].

L N = A / N α , (1)

While this accomplishment is remarkable under all scientific counts, many researchers have observed that the LLM scaling exponents are very small, typically in the range of 0.05–0.10, pointing to a regime of “diminishing returns” for pre-trained LLMs. In a recent article, two of the present authors pointed out what “diminishing returns” really means in actual practice: an exponent of 0.1 implies that cutting the loss function by a factor of 2 requires 2 10 = 1024 more resources [7]. It was then argued that, while formally wall-free, such a regime is simply unsustainable, whence the need for new directions that attend more to physics-aware insight and compliance with world models than muscular leverage of the number of parameters [8–11].

Clearly, the above conclusions should be tempered on the basis of recent and future technological variants of the basic LLM model, such as sparse/hybrid architectures, retrieval-augmented generation, mixture-of-experts models, quantization, as well as hardware improvements [12]. However, until such variants prove capable of significantly boosting the scaling exponents, the basic observation above retains its scope. The fact that big-tech AI companies are still striving for nuclear power in order to feed their data engines only buttresses the point.

Among the critical feedback spawned by this simple observation, a recurrent one is that the loss function cannot be paralleled to numerical discretization errors. The reason is that the machine learning (ML) procedures do not necessarily aim at sending the loss function to zero but are typically stopped before that limit is approached, usually to forestall overfitting and ensuing problems in generalizing, that is, the ability to reproduce unseen targets [3].

In the following, we argue that this criticism does not appear to affect the conclusion that present-day LLM scaling exponents face a severe sustainability issue.

2 The loss function as a pseudo-metric of accuracy

The approximation error associated with a numerical method employing N degrees of freedom, say the grid discretization of a partial differential equation (PDE), usually follows an asymptotic scaling relation of the formwhere a > 0 is the order of accuracy, and A is a prefactor measuring the “critical” size N c = A 1 / a above which the error starts to show a power-law decay. Most grid methods work around a = 2, while stochastic particle methods, such as Monte Carlo, feature a ∼ 1 / 2. Note that a = 2 means that reducing the error by a factor of 2 takes only 2 more resources ( N ), while with a = 1 / 2, this number is 4, which is generally regarded as poor convergence.

E N = A / N a , (2)

The above relation encodes a basic requirement on any well-posed numerical scheme, called “consistency,” that upon sending N → ∞, the error should vanish to reproduce the original target, generally the analytical solution of a continuum differential equation. Any non-zero value E ∞ ≡ E ( N → ∞ ) (the “pedestal”), or, worse, increasing error at increasing resolution (the “wall”), is regarded as an anomaly, usually caused by some form of ill-posedness of the numerical discretization, typically the breaking of a basic continuum symmetry by the discrete scheme.

The above relation bears a direct analogy to the LLM scaling law [4], whence the simple conclusion that an exponent of α = 0.1 in Equation 1 is simply unsustainable. Claiming that this analogy is flawed because machine learning practices do not concern themselves with the limit L → 0 as N → ∞ is tantamount to saying that consistency, a prime requirement for any well-posed numerical method, is not relevant to machine learning as a scientific discipline.

While it is not difficult to see reasons why one might be happy to settle for “small enough” values of the loss function (the early-stop empirical practice), it remains undeniable that dismissing consistency in favor of ambiguous “small enough” criteria leaves much to be desired in terms of the reliability of the methodology as a systematic scientific method. This is especially true with regard to the ability to generalize to unseen data, which is the essence of true learning, as opposed to mere memorization of the training data. Yet, it is true that matching a given set of data is not equivalent to reproducing the solution of a continuum PDE, so let us accept the loss function as a sort of empirical pseudo-metric with no need to comply with the consistency requirement. In the following, we shall argue that even this lenient perspective does not affect the claim of unsustainability made in [7]. The point is no longer consistency, but efficiency—how fast the error decreases with increasing resources, which is precisely dictated by the actual value of the scaling exponent.

3 The pedestal effect

Recently, a group of authors [13] have built on “Chinchilla scaling” [14] to argue that LLM scaling exponents published by Anthropic and subsequent works are biased by the fact of ignoring a “pedestal” in the scaling laws—the fact that the loss function does not vanish in the “continuum limit” N → ∞.

To appreciate the point, let us begin by casting the loss function in the formwhere we have set x ≡ 1 / N for the sake of convenience so that L 0 = L ( x = 0 ) is the continuum limit and L 1 = L ( x = 1 ) is the large-scale limit N = 1. Note that α > 0 denotes the no-wall regime in which L grows with x = 1 / N.

L x = L 0 + L 1 − L 0 x α , (3)

Clearly, the pedestal L 0 introduces its own exponent, α = 0 so that the “effective” exponent associated with the above relation must necessarily lie between 0 and α. The precise form of the transition depends on the ratio p = L 0 / L 1. To highlight the point, let us rescale L → L / L 1 and write

L x ; p = p x 0 + 1 − p x α . (4)

Next, let us define a “running” scaling exponent associated with a given pedestal p aswhere prime denotes derivative with respect to x. Clearly, the running exponent α p ( x ) returns a constant only in the case of a single-exponent power-law behavior, specifically α p = 0 ( x ) = α and α p = 1 = 0.

α p x = x L ′ x , p L x , p , (5)

Such a function is reported in Figure 1, for p = 0.01 , 0.05 , 0.1 , 0.2, for the case of Chinchilla scaling α ≡ α C ∼ 1 / 3. This figure shows that the transition from the Chinchilla (C) to the “Anthropic” (A), with the α ∼ 0.05 – 0.10 regime, is rather sharp, indicating that it does not take extremely large datasets to transition from “high” (C) to “low” (A) scaling regimes. The above observation highlights the major relevance of the pedestal effect in assessing the scaling performance of LLMs.

The same idea can be formulated by introducing a critical threshold x crit ( p ), below which the low-exponent A regime becomes the dominant one. Based on Equation 4, this occurs under the conditionnamely, whenever

p ≫ 1 − p x α C ,

x ≪ x crit p = p 1 − p 1 / α C . (6)

Hence, for α C = 1 / 3 and switching back to the size N, we obtain

N ≫ N crit p = 1 − p p 3 . (7)

Figure 2 shows the boundary N crit ( p ) between the C and A regimes as a function of p. From this figure, it is clear that it takes very small values of the pedestal for the C regime to be dominant for large datasets. Differently stated, by inverting Equation 8, we obtain

p ≪ p crit N = 1 1 + N 1 / 3 ∼ N − 1 / 3 . (8)

Hence, even for a moderate size dataset with N = 1 0 6, we already have p crit ∼ 1 0 − 2. Because the loss function in most LLM applications decays by about half a decade over several decades in N, the pedestal value is well above 0.01, showing that the Chinchilla exponent is not relevant to the scaling performance of large datasets such as the ones used in modern LLM applications.

4 Theoretical prediction of the scaling exponents

Having pinpointed the importance of the pedestal in assessing the effective scaling properties of LLMs, the natural question is to look for a theoretical explanation of their value. A simple yet quite convincing explanation was provided by Sharma and Kaplan (SK hereafter) [15], whose work lends further credit to the relevance of the A-exponents as a proper measure of LLM scaling performance.

In this article, the authors present a simple and convincing toy theory, supported by experimental data, that the LLM scaling exponents obey the following lower bound:where d is the intrinsic dimension of the manifold, namely, the number of independent coordinates required to describe the manifold where most data reside [16].

α ∼ 4 d , (9)

Note that for many complex applications, d ≪ D, D being the dimension of the embedding space. The above relation is remarkable for its simplicity, elegance, and robustness across a broad ensemble of numerical datasets.

A few comments are in order.

First, note that the SK relation refers to concrete LLM practice; hence, it is fully consistent with the pseudo-metric perspective discussed above.

Second, let us recall that in D dimensions, a grid discretization method of order a features an exponent

α D = a D . (10)

This results from the observation that the volume V ( D ) of a D-dimensional region of space of diameter δ scales like V ( D ) ∼ δ D, and standard (non-adaptive) grid discretization treats all portions of the D-dimensional space on the same footing, regardless of whether they host any interesting information process (typically, they do not). This is the infamous Curse of Dimensionality (CoD) [17].

Third, Equation 10 invites a natural analogy with the SK relation with D = d and a = 4. This means that LLMs can be paralleled to adaptive grid refinement methods with a = 4 in a d-dimensional feature space. By adaptive, we mean methods that place the numerical degrees of freedom “on demand,” that is, where the relevant information is located and not everywhere in a D-dimensional region of space. This ability to spot the relevant low-dimensional manifold where the computational resources should be focused is essential to (partially) tame the CoD, possibly one of the major achievements of LLM research.

Let us expand on the above by revisiting the main ideas behind the SK formula.

5 Sharma–Kaplan model

Let us consider a d-dimensional cube of side 1 and fill it with N ( s ) cubelets of side s. By definition N ( s ) s d = 1 d = 1, whence

N s = s − d . (11)

Next, let us define the loss function as the numerical error associated with a piecewise representation f N ( x ) of a smooth function f ( x ) within the unit side cube

L s = ∫ | f x − f N x | 2 d d x . (12)

Because the function f ( x ) is smooth, there is always a position x ′ in the hypercube, such that f N ( x ) = f ( x ′ ). Hence, by virtue of smoothness, the integrand in the above integral is bounded by g s d 1 / 2, where g is a constant proportional to the gradient of f, and s d 1 / 2 is the diameter of the hypercube. As a result, we obtain L ( s ) ∼ g 2 s 2 d, based on Equation 11,whence α = 2 / d. The same argument as applied to a piecewise linear representation of f ( x ) returns s 4 instead of s 2, whence the sought 4 / d exponent. Essentially, the scaling exponent is dictated by the smoothness (roughness) of the target function, the accuracy of the discretization, and the dimensionality of the intrinsic manifold.

L N ∼ N − 2 / d , (13)

Sharma and Kaplan appropriately evoke a connection between adaptive refinement and fractal sets but refrain from pursuing the analogy in quantitative form.

In the following, we expand such analogy in semi-quantitative terms, building on standard arguments from the fractal theory of fluid turbulence [18].

We hasten to emphasize that we make no claim here of any direct equivalence between LLM’s and fluid turbulence, other than observing that both can be regarded as dynamical systems giving rise to arguably non-smooth signals, despite their very different physical nature [19]. Hence, our purpose is simply to portray a heuristic scenario whereby the smoothness/roughness of the signal implies a direct bearing on the corresponding scaling exponents (Equation 13). More precisely, rough signals lead to smaller scaling exponents than smooth ones, indicating a further aggravation of the smallness problems in the case of non-smooth data applications.

5.1 Scaling exponents of three-dimensional homogeneous incompressible turbulence

Let us consider a turbulent eddy (active degree-of-freedom turbulent flow) of size l. More precisely, let v ( l ) be the increment of the velocity across a turbulent eddy of size l, that is, v ( l ) = | v ( x + l ) − v ( x ) | where x runs over the geometrical region occupied by the fluid (for a homogeneous fluid the dependence on x drops out). Based on Kolmogorov’s 1941 theory [20], the energy supplied to the flow at large scales is transferred to small scales, virtually with zero dissipation, through the process of energy cascade. Large eddies break up into smaller daughter eddies, which further break up into even smaller granddaughter eddies and so on down the line until eddies are sufficiently small for dissipation to take over.

According to K41, the energy flux is the same across scales; hence, the energy dissipation rate is written aswhere we have taken the lifetime of the above eddy as τ ( l ) ∼ l / v ( l ).

ϵ l = v 3 l / l = c o n s t . , (14)

This implies v ( l ) ∼ l 1 / 3, that is, the turbulent flow is not smooth but exhibits a scaling (Hurst) exponent h = 1 / 3, as opposed to a smooth (differentiable) signal featuring h = 1. This alone brings the scaling exponent down by a factor h, that is, α ∼ − 4 h / d, with no need to invoke fractal structures, only the fact that the signal is not differentiable.

The connection to fractals arises from the observation that K41 theory captures many features of turbulence but fails to describe intermittency, i.e., the occurrence, here and there, of abrupt bursts of activity. To account for this feature, one postulates that energy dissipation does not act as a space-filling process but rather occurs on a set of fractal dimension d < D, where D is the dimension of the embedding space, for fluids typically D = 2 or D = 3.

To compute the fractal dimension d, an argument similar to SK is usually adopted [21]. In other words, one introduces the probability β ( l ) that a small sphere (or cubelet) of radius l intercepts a d − d i m e n s i o n a l object within a three-dimensional ( D = 3 ) cube of side 1. Such probability is readily shown to scale aswhere D − d is the co-dimension. Next, one assumes that each time a turbulent eddy of size l breaks down in a series of smaller eddies, only a fraction β ( l ) remains active—amenable to further breakup. Hence, the kinetic energy available to sustain the energy cascade at scale l is E ( l ) = β ( l ) v 2 ( l ).

β l ∼ l D − d ,

Combining this with Equation 14, one obtainsyielding an effective exponentwhere we have used D = 3. This shows that the K41 exponent h = 1 / 3 corresponds to a space-filling dissipation process with d = D = 3, whereas intermittency occurs on a fractal set of dimension d < D = 3 so that h < 1 / 3.

v l ∼ ϵ 0 l 1 / 3 β l − 1 / 3 , (15)

h = 1 / 3 − D − d / 3 = d − 2 3 , (16)

Differently restated, the turbulent energy is a space-filling process. In the K41 picture, so is energy dissipation: dissipation “does not hide;” the intrinsic manifold fills up the entire embedding space. Even so, the velocity signal is reasonably rough, with a scaling exponent h = 1 / 3.

In the case of intermittency, dissipation is no longer space filling, and the velocity signal gets further roughness in proportion to the co-dimension of the fractal set where dissipation takes place, according to Equation 16.

In homogeneous incompressible fluid turbulence, one finds d ∼ 2.8 (the beta model discussed above) corresponding to h ∼ 0.27. However, other models deliver different values, all above 2, because d < 2 would imply h < 0, denoting a singularity in the velocity field, something that cannot happen in a homogeneous incompressible flow.

In passing, it is worth mentioning that turbulence is in fact multifractal, meaning that dissipation inhabits a whole set of different manifolds, each with its own scaling exponent [17, 18].

The implications for LLMs are as follows.

For non-smooth processes, such as turbulence and many other complex dynamical systems with h < 1, the scaling exponents become even smaller, i.e.,

α ∼ 4 h d . (17)

Differently restated, the intrinsic dimensions of non-smooth processes are expected to be magnified by a factor of 1 / h, pointing to a further decrease of the scaling exponents.

On the other hand, LLMs appear to be amazingly efficient in compressing the relevant information to low-dimensional manifolds. While turbulence only “needs” to go from D = 3 to d > 2 in order to pin down the dissipative manifold, LLMs accomplish far larger dimensional compression tasks: the embedding dimension of feature space in modern ChatGPT’s scores in the several thousands, as opposed to an intrinsic dimension d ∼ 100, corresponding to a quite impressive dimensional compression d / D ∼ O ( 1 0 − 2 ).

This ability of LLMs to pin down and represent information under such extreme dimensional compression regimes is indeed quite a remarkable scientific achievement.

6 Conclusion

The Sharma–Kaplan analysis leads to the conclusion that any dataset with d > 40 is bound to feature α < 0.1 and hence is hardly sustainable by any reasonable standard of energy efficiency. This appears to be the case regardless of whether the loss function should/could be paralleled to a numerical discretization error.

As shown in this note, the situation is even aggravated for the case of non-smooth data, as it is typically the case for most complex systems [19].

Hence, notwithstanding the amazing ability of LLMs to spot relevant information hidden in extremely tiny corners of ultra-dimensional feature spaces, the dimension of the intrinsic manifolds nevertheless appears to remain too high to be energetically sustainable. This is why the pursuit of the “bigger is better” avenue, under the drive of the no-wall excitement, is at serious risk of turning into a scarcely sustainable slippery energy slope. Many scientific and technological strategies can be deployed to improve this state of affairs, combining advances in hardware, software, and mathematical modeling as well. In our opinion, a return to foundational physics-aware AI models, the “world models” [11], offers one of the most intellectually attractive and practically promising routes towards sustainable AI.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Andrea Montessori, Roma Tre University, Italy

Reviewed by: Nikolay Hinov, Technical University, Sofia, Bulgaria