Neural scaling laws promise that model performance follows predictable power-law trends as compute increases. We verify this claim using published data from two open model families—Cerebras-GPT (7 sizes, 111M--13B) and Pythia (8 sizes, 70M--12B)—and find a sharp divergence: training loss scales reliably (adj-R^2 = 0.
Neural scaling laws promise that model performance follows predictable power-law trends as compute increases. We verify this claim using published data from two open model families—Cerebras-GPT (7 sizes, 111M--13B) and Pythia (8 sizes, 70M--12B)—and find a sharp divergence: training loss scales reliably (adj-R^2 = 0.
Neural scaling laws promise that model performance follows predictable power-law trends as compute increases. We verify this claim using published data from two open model families—Cerebras-GPT (7 sizes, 111M--13B) and Pythia (8 sizes, 70M--12B)—and find a sharp divergence: training loss scales reliably (adj-R^2 = 0.