BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training

BigMac introduces a dependency-safe nested pipeline for multimodal LLM training that breaks the usual tradeoff between throughput and activation memory. Existing compute-efficient systems such as Optimus decouple the encoder and generator from the LLM pipeline, but this forces retention of encoder activations across the full pipeline and can also задержle LLM activations until generation completes; memory-efficient systems such as Megatron-DistTrain integrate all modules into one pipeline, but then model and data heterogeneity create bubbles and stall the slowest stage. BigMac keeps the user-specified LLM schedule, including interleaved 1F1B, and nests encoder and generator microbatch units only at dependency-safe points, so the encoder and generator each require only O(1) activation memory while the LLM backbone remains unchanged. It also replaces rigid FSDP all-gather with a one-sided NVSHMEM pull to remove synchronization barriers, and supports distinct context-parallel groups for encoder and LLM. On a 128-GPU Hopper cluster, BigMac speeds up MLLM-Understanding with Qwen3-30B-A3B plus a 1.3B ViT by 1.08×–1.9× over the baselines, and MLLM-Generation with the same backbone plus a 20B MMDiT by 1.5×–1.9×, while keeping peak memory flat as batch size increases and avoiding the out-of-memory failures seen in Optimus. In production on a 345B-parameter MLLM across 1,536 Hopper GPUs and more than 18K iterations, it preserved stable loss dynamics, with average loss deviation below 10^-3 versus Megatron-LM in controlled tests.