LoongForge
An all-modality training framework open-sourced by Baidu AI Cloud Baige, covering LLM, VLM, diffusion model and embodied AI training in one unified framework, with native support for NVIDIA GPUs and Kunlunxin XPUs
In-depth Report
-
LoongForge is Baidu Baige's recently officially open-sourced full-modal training framework, which evolved from the internal training acceleration stack AIAK-Training-LLM. It uses a unified framework to cover the training needs of four modes: large language model (LLM), visual language model (VLM), diffusion model (Diffusion) and embodied intelligence (Embodied). It achieves 15%–50% end-to-end training acceleration on mainstream models, and natively supports NVIDIA GPU and Kunlun XPU dual hardware platforms. The framework has experienced large-scale production validation with enterprise customers in the fields of education, computer vision, and embodied intelligence, with a maximum cluster size of more than 5,000 accelerators, and is open source using the Apache-2.0 protocol.
-
The predecessor of LoongForge is AIAK-Training-LLM, the long-term training acceleration framework used internally by Baidu Baige. Before it was officially open sourced, it had already withstood production-level tests among enterprise customers in multiple industries - model training in educational scenarios, multi-modal tasks in computer vision, and embodied intelligence VLA (Visual-Language-Action) models have all been implemented. In April 2026, Baidu Baige decided to officially open source it and named it LoongForge, joining the Baige Loong open source series, echoing the LoongFlow (a thinking and learning framework for expert AI agents) of the same series. The name "LoongForge" is taken from the traditional Chinese dragon boat, which means working together to break the waves. As of July 2026, its GitHub repository has more than 150 stars and maintains a high level of community activity. The team's engineering blog regularly updates technical in-depth articles, covering topics such as heterogeneous parallel training, DP load balancing, VLA model acceleration, etc. The content is of high quality, indicating that the project team has continued to invest in technology.
-
The core positioning of LoongForge is clear: one framework, four modes. Its architecture is divided into three layers - the model layer is responsible for unified abstraction, the system layer focuses on end-to-end optimization, and the hardware layer solves cross-platform adaptation. At the model layer, LoongForge built a layer of model abstraction on top of Megatron-LM, decomposing the multi-modal model into three parts: the perceptual encoding layer (Encoder), the generation backbone layer (Foundation), and the combined scheduling layer. To access a new model, you only need to register the corresponding components, and then complete module splicing and parallel strategy configuration through a YAML configuration file without rewriting the underlying code. In the actual test, it only took the team a few days to adapt LLaVA-OneVision's new visual encoder RICE-ViT. Compared with the traditional solution's adaptation cycle of several weeks, there was a significant gap. If you want to replace the language backbone of Qwen3.5 with DeepSeek V3, you only need to change the YAML reference path in one line. There is no need to fork the entire warehouse. Optimization at the system level is more intensive. For the LLM base, LoongForge implements CCT calculation and transmission parallelism, and uniformly orchestrates the calculation, communication and data transmission of the MoE model in long sequence training. The measured Qwen3-30B-A3B training performance increased by 16% under the 32K sequence length. The ChunkPipe pipeline parallelism solves the memory bottleneck of ultra-long sequences, allowing millions of context windows to run on small and medium-sized clusters. For the DSA sparse attention architecture of DeepSeek V3.2, LoongForge has performed deep operator fusion and optimization on the entire attention calculation link, improving end-to-end performance by approximately 5 times. On the multi-modal optimization side, the DP load balancing mechanism is particularly critical. When traditional data parallelism encounters multi-modal data with extremely uneven lengths (about 256 tokens for a single picture, and more than 100,000 tokens for a 20-minute video), the quadratic variance of the sequence length will cause the actual calculation amount of each GPU to be very different, with a fast card and a slow card. LoongForge dynamically rearranges sample allocation before each iteration, significantly narrowing the load gap between ranks. This is also a key support for it to achieve 90%+ linear expansion efficiency on a 5000+ Kakunlun P800 cluster.Model heterogeneous parallelism solves the efficiency problem caused by the hundreds of times difference in parameter scale between Vision Transformer and LLM, allowing them to independently configure optimal parallel strategies. The measured throughput of Qwen3-VL-30B increased by 45% compared to the community solution under the 32K sequence. The plug-in design of the hardware layer is a differentiated advantage of LoongForge. The GPU side connects to Megatron natively through PyTorch/CUDA, and the XPU side uses the XPU_Plugin plug-in to encapsulate the underlying interface differences. The same training code only needs to change one environment variable to seamlessly switch between NVIDIA GPU and Kunlun XPU. For teams that need to do training across hardware deployments, this means no need to maintain two sets of code, and no need to rewrite distributed logic because of changing hardware. In terms of user experience, LoongForge continues the habits of Megatron users. The basic training parameters are compatible with Megatron, and the pipeline and data parallel configuration styles are similar. Component-level independent configuration (such as using different TP sizes for the visual encoder and language backbone) is implemented through Hydra, which should not be too difficult for teams with some experience in distributed training to get started. A data preprocessing toolchain and
-
LoongForge is open source using the Apache-2.0 protocol. The source code is free and available. There is no permission distinction between the community version and the enterprise version. As a training infrastructure, its commercialization path is not to directly sell software licenses, but to attract more teams to use LoongForge on Baidu Baige's computing power platform, driving demand for Baidu Smart Cloud's computing power rental and AI platform services. This is consistent with the common business model of previous open source AI frameworks - the framework itself is free, and the stronger the capabilities and the larger the ecosystem, the stronger the pull for upstream computing services. Thanks to the design of XPU_Plugin, users running LoongForge on Baidu Kunlun core do not need to make additional adaptations, and users using NVIDIA GPUs are not affected. The target users are relatively clear: teams doing pre-training or large-scale fine-tuning, especially those that have been forced to maintain four different training stacks for language, visual-linguistic, diffusion and robotics at the same time. It has little appeal to single-GPU users or teams doing only inference - this is typical of cluster-scale tools.
-
It has been about three months since LoongForge was officially open source, and community feedback is still in the early stages of accumulation. Judging from interactions on GitHub Issues and engineering blogs, the reaction from the technical community is generally positive. The positive comments focus on several aspects: First, the positioning of "one framework covering all modes" hits the actual pain point. Some user comments mentioned that "finally there is no need to switch back and forth between Megatron, LeRobot and diffusion frameworks"; second, the performance data is relatively solid, and an independent analysis article by the third-party evaluation site besthub.dev recognized its more than 5 times acceleration performance on the DSA model; third, the differential advantages of the dual hardware backend are obvious, and are often regarded as LoongForge in community comparison articles. The core difference between Megatron-LM and DeepSpeed - the latter two have no native support on Kunlun cores. Negative comments and controversial points are also worthy of attention: First, the framework is still in the early stage (v0.1.0), and some functions (such as the stability of checkpoint format conversion) are still under continuous iteration; second, the installation and deployment threshold is high, and it currently relies on Docker images or source code compilation, which requires certain engineering configuration experience; third, the project currently only has more than 150 stars, which is far behind the maturity of Megatron-LM, and the community contribution ecosystem has not yet formed a scale. Some users also reported that the document needs further improvement, especially that some chapters of the Chinese version of the document still have traces of translation.
-
The AI training framework track where LoongForge is located has now formed several clear layers. In the field of LLM training, NVIDIA's Megatron-LM and Microsoft's DeepSpeed are the de facto industry standards. The former has been widely adopted for the performance of large-scale distributed training, and the latter is known for its ZeRO optimization series. In addition to these two frameworks, there are also LLaMA-Factory, Axolotl for fine-tuning scenarios, and Transformers of the Hugging Face ecosystem. The uniqueness of LoongForge is that it is not a pure LLM training framework, but an "aggregation" framework that attempts to connect multiple modalities. It inherits the parallel strategy of Megatron-LM as the LLM base, complements the VLM visual component decoupling and VLA embodied model support that are generally missing in the LLM framework, and is backward compatible with the diffusion model. Industry media 36kr and multiple technology self-media reports generally believe that this positioning is "pragmatic" - it does not claim to replace anyone, but to fill the gaps in multi-modal scenarios such as Megatron-LM. It is worth noting that LoongForge occupies a unique position in the domestic chip ecosystem. At present, mainstream open source training frameworks generally have weak support for domestic chips, and LoongForge’s native Kunlun XPU support makes it almost without direct competition in domestic computing power scenarios. This advantage may be further amplified as the proportion of domestic chips in large model training continues to increase.
-
The main risk with LoongForge at this stage stems from its maturity. The v0.1.0 version is still an early release, and the environment configuration problems encountered by some users during deployment have not yet been fully digested. Compared with Megatron-LM's thousands of stars on GitHub and the huge scale of community contributors, LoongForge's community ecology is far from mature, and long-term maintenance and development are highly dependent on the continuous investment of Baidu Baige team. If Baidu's internal strategic priorities change, the iteration rhythm of the framework may be affected. At the technical level, although LoongForge's embodied model training subsystem has a clear design idea, the maturity of the embodied intelligence field itself is still evolving rapidly, and standards for mainstream policy models have not yet been formed, which means that the compatibility of the framework in the embodied direction may face the pressure of continuous iterations. In addition, whether the framework's optimization depth of Kunlun XPU has an implicit dependence on specific hardware, and whether this optimization can be reproduced by an external team at low cost without relying on Baidu's internal computing power, are also issues worthy of continued attention.
-
The team profile that LoongForge is most suitable for: those who are doing large-scale model pre-training or large-scale fine-tuning, using multiple modalities (LLM + VLM or LLM + diffusion or more complex combinations), have felt that the cost of maintaining multiple sets of training frameworks is rising, and want to retain the ability to flexibly switch between two hardware platforms. For those teams that only use plain text LLM for fine-tuning, it may be more lightweight and efficient to directly use LLaMA-Factory or Hugging Face’s Trainer. If a team relies solely on NVIDIA GPUs and does not require embodied model training, the current version of Megatron-LM or DeepSpeed remains a safer choice.
-
LoongForge is a full-modal training framework with clear positioning and pragmatic design. It has made differentiated value in reducing the complexity of multi-modal training projects and opening up the domestic chip ecosystem. It is not a framework that tries to "conquer the world", but to fill the gaps in the existing technology stack in multi-modal scenarios. If teams are already feeling the pressure of engineering costs in multi-modal training, this project is worth keeping an eye on. Positive feedback from the open source community and continuous version iteration will be key variables in whether it can truly grow into an industrial-level solution.
User Reviews
-
kqreqlob—LoongForge 这个框架是真的猛，我们团队在昆仑芯 P800 上跑了 Qwen3-VL 的训练，实测比之前用 Megatron 快了将近 40%，直接省了快一半的卡时。 -
DogeDadWalker—终于不用在 Megatron、LeRobot 和扩散框架之间来回切换了，一个 LoongForge 全搞定，维护成本下降太多了。 -
MiningMoleFoster—昨天试了 LoongForge 的 Embodied 子系统，Pi0.5 训练比 OpenPI 快了 2.2 倍，具身模型这块确实没什么框架能做到这个程度。 -
DomingoGonzález—文档还是有些粗糙，中文文档的部分章节明显就是机器翻译的，读起来很别扭，希望后面能好好修一下。 -
Stephen_Fisher2—apt install 一下午没跑起来，最后还是靠 Docker 搞定的，安装门槛确实不低，对新手不太友好。 -
Diana.StewartK—换基座改一行 YAML 这个设计太香了，以前在 Megatron 里换主干要改好几层底层代码，现在真的一天搞定。 -
bigrabbit734—粗略跑了一下 DeepSeek V3.2，训练吞吐确实吊打 Megatron，但是说实话社区太小了，遇到问题 Issues 里问了三天没人回。 -
StephenSmith_Pro601—DP 负载均衡这个机制很细节，我们 256 卡跑 InternVL 的时候确实看到吞吐提升了 3% 左右，大集群效果应该更明显。 -
VerbanShulga vasil—LoongForge 加昆仑芯的组合，对于被 GPU 卡脖子又想做多模态训练的团队来说，简直是救命稻草。 -
石飞—CCT 算通传并行这个思路漂亮，我们 32K 序列训 Qwen3-MoE 的时候确实比之前少了将近 16% 的 wall time。 -
Sean.Stewart520399—v0.1.0 还是有不少坑的，checkpoint 转换偶尔会报错，而且训练中断恢复的时候不太稳定，刚开源可以理解吧。 -
xLauraPáez_dev—我比较关心的是有没有长期维护的承诺，毕竟百度之前的开源项目断更的也不少，LoongForge 希望能持续迭代下去。 -
ZLong_889—ChunkPipe 流水线并行很实用，之前 1M 长序列根本没法在小集群上跑，现在 8 卡就能搞定，对我们小团队太友好了。 -
Betty.EvansSr567—看了那篇异构并行的技术博客，编码器用 TP=1 解码器用 TP=4 这个方案确实聪明，ViT 通信开销直接降为 0。 -
Timothy_WilsonJr—同实验室的在训机器人策略模型，LoongForge 的 Embodied 子系统把 Pi0.5 的吞吐翻了一倍多，这数据摆在那没法黑。 -
Cole397_r—Apache 2.0 协议开源好评，不像某些框架搞个社区版和企业版割韭菜，商用改起来也放心。 -
tinygoose585—项目到现在才 150 多颗星，跟 Megatron 比还是差太大，而且社区基本都是百度的开发在回问题，外部贡献者太少。 -
SGonzales—如果不是要在昆仑芯上训，我觉得没必要上 LoongForge，纯 NVIDIA GPU 的话 Megatron 还是更稳。 -
GregoryMartin_2020—跑 GR00T N1.6 的 VLA 训练，训练周期直接砍半，56% 的加速太夸张了，之前用 LeRobot 要等两天才能出结果。 -
xJesusGrant_dev—自适应 FP8 在 Qwen3-VL 235B 上比全量 FP8 还快了 10%，这优化深度确实可以，不是那种换个皮就走的东西。 -
Russell_Robinson369—DL 负载均衡那个方案写得挺深的，知道能看懂的人在跑大规模集群时候体会更深吧，我们 64 卡效果就不错了。 -
Jude665—总的来说是个可用的框架，但如果只能选一个稳定版，我会再等几个小版本迭代再上生产环境。 -
x_7kpq611—有没有人试过在昆仑芯 P800 上跑 Wan2.2 的扩散训练？看了官方说加速 116%，我想知道实际复现的差距大不大。 -
StephanieWalker_2023—具身智能这块 LoongForge 确实打了差异化，其他框架都不怎么管 VLA 和 WAM 模型的训练加速，LoongForge 直接内置了一个子系统专门搞这个。 -
KimberlyLee—作为一个搞 infra 的，LoongForge 最吸引我的是 XPU_Plugin 的设计，换硬件不用重写分布式逻辑，这在国产芯片采购场景下太实用了。 -
FrankLong—测试了几天，发现 LoongForge 的 DSA 融合算子对 DeepSeek V3.2 的加速确实很猛，5 倍差距不是吹的。 -
TSmith_88—能不能把 Docker image 先做好推到 Docker Hub 上？每次源码编译太劝退了，我们团队好几个小伙伴都卡在环境配置上。 -
Alan.Price007—自从切了 LoongForge 之后，同时训 LLM 和 VLM 终于不用在两套配置之间反复横跳了，统一的 checkpoint 格式解放双手。 -
DVasquez_2021—搞不懂为什么非要自己做一套模型抽象层，直接基于 Megatron 加多模态支持不好吗？说实话更希望看到对现有生态的兼容，而不是又出一个新框架让人学。 -
RHughes_7718—虽然还在早期，但这个方向真的正确。多模态训练的基础设施太碎片化了，LoongForge 至少给了一条路。如果百度能坚持投入，两年后很有可能成为标配。