Writing
Notes on model training, systems, open source, and organizations.
Compute-Optimal Is Not Cluster-Optimal
MOSAIC jointly selects a sparse-MoE architecture, token budget, and parallel layout under a fixed cluster and training window.
Research Problems in Pretraining
A practitioner's account of what pretraining research can predict, where current methods break, and which questions remain open.
Your Org Has the Same Scaling Problem as a Badly Tuned Training Run
AI raised individual throughput but coordination overhead stayed fixed. For many product-engineering orgs, the bottleneck flipped from compute-bound to communication-bound.
On Assessing the Value of a Project
A practical framework for comparing research projects by probability of success, effect size, and weighted reach.
Determining Model Size and Training Horizon through Scaling Laws
Deriving model size and training tokens from a fitted scaling law, then extending the calculation to repeated data, inference demand, and cluster efficiency.
GluonNLP — Deep Learning Toolkit for Natural Language Processing
Why we built GluonNLP to make NLP experiments easier to reproduce, maintain, and reuse.