arXiv:2406.12907v1 Announce Type: new
Abstract: Kaplan et al. [2020] (`Kaplan’) and Hoffmann et al. [2022] (`Chinchilla’) studied the scaling behavior of transformers trained on next-token language prediction. These studies produced different estimates for how the number of parameters ($N$) and training tokens ($D$) should be set to achieve the lowest possible loss for a given compute budget ($C$). Kaplan: $N_text{optimal} propto C^{0.73}$, Chinchilla: $N_text{optimal} propto C^{0.50}$. This note finds that much of this discrepancy can be attributed to Kaplan counting non-embedding rather than total parameters, combined with their analysis being performed at small scale. Simulating the Chinchilla study under these conditions produces biased scaling coefficients close to Kaplan’s. Hence, this note reaffirms Chinchilla’s scaling coefficients, by explaining the cause of Kaplan’s original overestimation.
Source link
lol
