AI researcher and builder writing about model training, systems, open source, and organizations.https://szha.ai/https://szha.ai/blog/compute-optimal-is-not-cluster-optimal/https://szha.ai/blog/compute-optimal-is-not-cluster-optimal/MOSAIC jointly selects a sparse-MoE architecture, token budget, and parallel layout under a fixed cluster and training window.Wed, 12 Aug 2026 00:00:00 GMTpretrainingscaling-lawsmoesystemshttps://szha.ai/blog/research-problems-in-llm-pretraining/https://szha.ai/blog/research-problems-in-llm-pretraining/A practitioner's account of what pretraining research can predict, where current methods break, and which questions remain open.Wed, 03 Jun 2026 00:00:00 GMTpretrainingscaling-lawsoptimizationmuPresearchhttps://szha.ai/blog/badly-tuned-training-run/https://szha.ai/blog/badly-tuned-training-run/AI raised individual throughput but coordination overhead stayed fixed. For many product-engineering orgs, the bottleneck flipped from compute-bound to communication-bound.Wed, 15 Apr 2026 00:00:00 GMTmanagementaiscalingorganizationshttps://szha.ai/blog/on-assessing-the-value-of-a-project/https://szha.ai/blog/on-assessing-the-value-of-a-project/A practical framework for comparing research projects by probability of success, effect size, and weighted reach.Tue, 20 May 2025 00:00:00 GMTresearchdecision-makinghttps://szha.ai/blog/model-size-and-training-horizon-scaling-laws/https://szha.ai/blog/model-size-and-training-horizon-scaling-laws/Deriving model size and training tokens from a fitted scaling law, then extending the calculation to repeated data, inference demand, and cluster efficiency.Mon, 02 Dec 2024 00:00:00 GMTscalingtrainingcompute-optimalchinchillahttps://szha.ai/blog/gluonnlp-deep-learning-toolkit-for-nlp/https://szha.ai/blog/gluonnlp-deep-learning-toolkit-for-nlp/Why we built GluonNLP to make NLP experiments easier to reproduce, maintain, and reuse.Tue, 24 Jul 2018 00:00:00 GMTnlpdeep-learningmxnetopen-source