>10x More Efficient Pretraining

A new pretraining recipe announced by the research team behind Magic’s AI models claims to be more than ten times as compute‑efficient as the leading open‑weight base models currently available. By leveraging algorithmic improvements rather than sheer hardware scale, the team reports that their approach matches the performance of DeepSeek V4 Pro Base while using roughly fifty times fewer floating‑point operations—equivalent to about half the compute budget required for GPT‑3’s pretraining, or an estimated cost of $0.5 million on the GB200 GPU cluster. Scaling the method tenfold to a $4 million compute investment produced a model that outperformed every publicly released open base model on perplexity evaluations, and the authors note that replicating the same capability with DeepSeek’s original recipe would exceed $100 million in cost.

The efficiency gains stem from a systematic series of incremental changes across model architecture, optimizer settings, training objectives, and data curation, each validated through power‑law scaling analyses on a diverse suite of 167 evaluation domains. The team employed a “nano‑GPT” speedrun framework to test ideas quickly, but discovered that improvements effective on tiny models did not always translate to large‑scale systems, while certain features common to many large language models could be removed without harming performance. Their evaluation pipeline included held‑out loss measurements normalized by bits‑per‑byte, cross‑backend log‑probability checks on both GB200 and GB300 hardware, and verification with Fireworks’ in‑house inference engine. To guard against memorization, they constructed evaluation sets using parsers and OCR tools distinct from those used in pretraining, and re‑worded or summarized key research documents with a frontier LLM before inclusion.

Looking ahead, the researchers intend to combine their compute‑efficient pretraining with agentic reinforcement learning and extended context windows to create “superhuman” coding assistants and autonomous AI‑research agents. They have already begun testing the correlation between pretraining loss and downstream RL performance through a short math‑focused RL run using a 16 k token chain‑of‑thought budget, finding promising alignment. By continuously scaling up—running intermediate experiments at one‑tenth hero scale every few weeks and full‑scale hero runs every few months—the team aims to close the gap to frontier models without the need for massive chip inventories, positioning their approach as a viable path for labs lacking access to the hundred‑thousand‑chip resources traditionally required for trillion‑parameter training.

⚡ Effects Interpreter