Sequence-Parallelism on 111qqz的小窝https://111qqz.com/tags/sequence-parallelism/Recent content in Sequence-Parallelism on 111qqz的小窝Hugo -- gohugo.iozh-cnhust.111qqz@gmail.com (111qqz)hust.111qqz@gmail.com (111qqz)© 2011-2026 111qqzThu, 29 Oct 2026 10:00:00 +0800Sequence Parallelism 与 Context Parallelism：为什么沿序列切分还不够？https://111qqz.com/2026/10/sequence-context-parallelism/Thu, 29 Oct 2026 10:00:00 +0800hust.111qqz@gmail.com (111qqz)https://111qqz.com/2026/10/sequence-context-parallelism/<p><a href="https://111qqz.com/2026/10/tensor-parallelism-matrix-split/" >上一篇 TP</a> 写完之后，我对 Megatron 的 Column Parallel + Row Parallel 已经比较清楚了：前一层按输出特征切分，后一层按输入特征切分，两层之间不需要 AllGather，只在最后做一次 AllReduce 把部分和加起来。</p>