Distributed-Training on 111qqz's bloghttps://111qqz.com/en/tags/distributed-training/Recent content in Distributed-Training on 111qqz's blogHugo -- gohugo.ioenhust.111qqz@gmail.com (111qqz)hust.111qqz@gmail.com (111qqz)© 2011-2026 111qqzThu, 22 Oct 2026 19:40:00 +0800从 DDP 到 ZeRO：为什么分布式训练要对参数、梯度和优化器状态分片？https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-10-22-ddp-to-zero-memory-sharding/Thu, 22 Oct 2026 19:40:00 +0800hust.111qqz@gmail.com (111qqz)https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-10-22-ddp-to-zero-memory-sharding/<h2 class="relative group">起因 <div id="起因" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e8%b5%b7%e5%9b%a0" aria-label="Anchor">#</a> </span> </h2> <p><a href="https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-10-22-distributed-training-comm-primitives/" >上一篇</a>搞清楚了 AllReduce、ReduceScatter 和 AllGather 三种通信原语，末尾提到 DDP 和 ZeRO Stage 2 选择了不同的通信原语组合，根本原因是参数更新的责任分配不同。</p>从 AllReduce 到 Ring：分布式训练中的通信原语https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-10-22-distributed-training-comm-primitives/Thu, 22 Oct 2026 19:20:00 +0800hust.111qqz@gmail.com (111qqz)https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-10-22-distributed-training-comm-primitives/<h2 class="relative group">起因 <div id="起因" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e8%b5%b7%e5%9b%a0" aria-label="Anchor">#</a> </span> </h2> <p>之前几篇文章从<a href="https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-08-17-gpu-memory-mental-model/" >模型显存组成</a>一路算到 <a href="https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-10-21-7b-training-memory-ledger/" >7B 模型的训练账本</a>，结论很直接：BF16 mixed precision + AdamW 的 Model States 就有 112 GB，一张 80 GB A100 连放都放不下。<a href="https://111qqz.com/en/post/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/2026-10-21-checkpointing-offloading-tradeoff/" >Checkpointing + Offloading</a> 两个一起用可以勉强塞进去，但训练速度要被 PCIe 传输和额外重计算拖累。</p>