小红书开源的BigMac打破多模态训练帕累托前沿
BigMac提出依赖安全的嵌套流水线，兼顾多模态训练吞吐与显存，速度提升最高1.9倍。
多模态大模型训练长期面临计算效率与显存占用的两难：计算高效则显存高，显存高效则速度慢。小红书开源的BigMac通过“依赖安全的嵌套流水线”，在不打断LLM流水线的前提下嵌入编码器和生成器计算，同时提升吞吐并控制显存。训练速度提升1.08-1.9倍，显存稳定。
正文摘录
.jpg) BigMac is a new pipeline parallelism paradigm for native multimodal scenarios. It tackles the difficulty of balancing compute efficiency and memory usage in multimodal LLM training by proposing a dependency-safe nested pipeline: it uses the mature LLM pipeline as the backbone, and orderly embeds encoder and generator computations without disrupting the LLM execution order. This enables efficient multimodal pipeline training without increasing LLM pipeline bubbles and while keeping activation memory bounded. Compared to traditional pipeline parallelism implementations, BigMac further decouples global scheduling from runtime execution in its engineering design, and provides a set of i…