https://111qqz.com/tags/pipeline-parallelism/Recent content in Pipeline-Parallelism on 111qqz的小窝Hugo -- gohugo.iozh-cnhust.111qqz@gmail.com (111qqz)hust.111qqz@gmail.com (111qqz)© 2011-2026 111qqzFri, 23 Oct 2026 19:30:00 +0800https://111qqz.com/2026/10/pipeline-parallelism-engineering-tradeoffs/Fri, 23 Oct 2026 19:30:00 +0800hust.111qqz@gmail.com (111qqz)https://111qqz.com/2026/10/pipeline-parallelism-engineering-tradeoffs/<h2 class="relative group">起因 <div id="起因" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e8%b5%b7%e5%9b%a0" aria-label="锚点">#</a> </span> </h2> <p><a href="https://111qqz.com/2026/10/pipeline-parallelism-gpipe-1f1b/" >上一篇</a>搞清楚了 Pipeline Parallelism 的调度机制：GPipe 先 Forward 后 Backward，1F1B 通过交替调度降低 Activation 暂存压力，但理想条件下 Bubble Time 相同。</p>https://111qqz.com/2026/10/pipeline-parallelism-gpipe-1f1b/Fri, 23 Oct 2026 19:00:00 +0800hust.111qqz@gmail.com (111qqz)https://111qqz.com/2026/10/pipeline-parallelism-gpipe-1f1b/<h2 class="relative group">起因 <div id="起因" class="anchor"></div> <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none"> <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e8%b5%b7%e5%9b%a0" aria-label="锚点">#</a> </span> </h2> <p><a href="https://111qqz.com/2026/10/tensor-parallelism-matrix-split/" >上一篇 Tensor Parallelism</a> 写完之后，我对 TP 的基本原理有了底：把单个算子内部的矩阵乘法拆到多张 GPU 上执行，每张卡只算部分输出，最后通过 AllReduce 聚合。TP 分片的是计算本身。</p>