Logo RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models

Hongzhi Zang1*, Mingjie Wei6,2*, Si Xu3, Yongji Wu5, Zhen Guo3, Yuanqing Wang4,3, Hao Lin3, Peihong Wang2, Liangzhi Shi1, Yuqing Xie1, Zhexuan Xu1, Zhihao Liu7,2, Kang Chen4,2, Wenhao Tang1,
Quanlu Zhang3, Weinan Zhang6, Chao Yu1,2,‡,†, Yu Wang1,†
1Tsinghua University, 2Zhongguancun Academy, 3Infinigence AI, 4Peking University, 5UC Berkeley, 6Harbin Institute of Technology, 7Institute of Automation, Chinese Academy of Sciences
*Equal Contribution †Corresponding Authors: zoeyuchao@gmail.com, yu-wang@tsinghua.edu.cn ‡Project Lead
Paper arXiv Code 🤗 Models View:  -
Overview of RLinf-VLA

Figure 1. Built on a unified interface, RLinf-VLA seamlessly supports diverse VLA architectures, multiple RL algorithms, and various simulators. It provides three GPU allocation modes: collocated, disaggregated, and a novel hybrid mode and speeds up training by 2.27x compared to the baseline. A single unified model achieves 98.11% success on 130 LIBERO tasks and 97.66% on 25 ManiSkill tasks. RLinf-VLA achieves an average 84.63% success on 6 RoboTwin tasks.

Abstract

Recent advances in vision-language-action (VLA) models have motivated the extension of their capabilities to embodied settings, where reinforcement learning (RL) offers a principled way to optimize task success through interaction. However, existing methods remain fragmented, lacking both a unified platform for fair comparison across architectures and algorithms and an efficient system design for scalable training.

To address these challenges, we introduce RLinf-VLA, a unified and efficient framework for scalable RL training of VLA models. RLinf-VLA achieves unification by providing a unified interface that standardizes the integration of diverse VLA architectures, multiple RL algorithms, and heterogeneous simulators, enabling extensibility. To ensure efficiency, the system adopts a flexible resource allocation architecture for rendering, inference, and training workloads in RL pipelines. In particular, for GPU-parallelized simulators, RLinf-VLA introduces a hybrid fine-grained pipeline allocation strategy, yielding a 1.61x–1.88x training speedup.

Using this unified system, models trained with RLinf-VLA demonstrate consistent performance improvements of approximately 20–85% across multiple simulation benchmarks, including LIBERO, ManiSkill, and RoboTwin. Furthermore, we distill a set of training practices for effective RL-based VLA training. We position RLinf-VLA as a foundational system to enable efficient, unified, and reproducible research in embodied intelligence.

Contributions

  • Unified system abstraction. RLinf-VLA provides a unified interface that supports multiple robotic simulators (ManiSkill, LIBERO, RoboTwin), diverse VLA architectures (OpenVLA, OpenVLA-OFT), and reinforcement learning algorithms (PPO, GRPO). The system exposes three execution modes, including a novel hybrid GPU allocation mode, enabling scalable and configurable training across heterogeneous setups.
  • Efficient system and algorithm design. RLinf-VLA introduces hybrid fine-grained pipelining for GPU-parallelized simulators and collocated execution for CPU-parallelized simulators, substantially improving training throughput, with speedups of up to 2.27×. In addition, we incorporate a set of algorithmic optimizations that further enhance training efficiency and stability.
  • Strong empirical performance and generalization. Using a single unified model, RLinf-VLA achieves a 98.11% success rate on 130 LIBERO tasks and 97.66% on 25 ManiSkill tasks. RLinf-VLA also achieves an average success rate of 84.63% on six RoboTwin tasks, realizing an average performance improvement of 63.75% on these tasks, which demonstrates its powerful performance and generalization capabilities in post-training.
  • Open and extensible platform. RLinf-VLA is released as an open-source and actively maintained platform, providing a practical foundation to accelerate, standardize, and scale reinforcement learning research for embodied intelligence.

System Overview

The RL training pipeline (described in Section 3.2) for VLA involves varying resource demands from Training, Generation, and Simulator. Resource bottlenecks depend heavily on the simulator type: CPU-parallelized simulators are typically CPU-bound, using GPUs primarily for rendering and inference, whereas GPU-parallelized simulators execute simulation, rendering, and inference entirely on the GPU. While the latter offers higher throughput, it creates severe contention for GPU memory and compute. To maximize efficient utilization across these diverse setups, flexible and optimized GPU allocation strategies are essential. Our framework supports flexible and easily configurable allocation modes: collocated, disaggregated, and a novel hybrid mode.

Legend
❮ ❯

RLinf-VLA provides three GPU allocation modes to support different training scenarios:

  • Collocated mode: Rendering, inference, and training run on the same GPU, suitable for CPU-parallelized simulators.
  • Disaggregated mode: Rendering, inference, and training run on separate GPUs, providing maximum flexibility.
  • Hybrid mode (without pipelining): In hybrid allocation, components can flexibly select GPUs. A typical configuration is to assign Generation and Simulator to different GPU partitions, while allowing Training to utilize all GPUs. However, the resources remain underutilized.
  • Hybrid mode (with pipelining): A novel allocation strategy that combines the benefits of both collocated and disaggregated modes, achieving optimal resource utilization for GPU-parallelized simulators.

Results

Is RLinf-VLA high-performance?

We evaluate RLinf-VLA on three representative testbeds, LIBERO, ManiSkill and RoboTwin. The results demonstrate that RLinf-VLA achieves approximately 20–85% improvement, highlighting its strong capability to support large-scale multi-task learning.

Table 1. Evaluation results across three simulation benchmarks. Values denote success rates (%).
Key Findings Illustration

Is RLinf-VLA high-efficiency?

We benchmark the framework across both GPU-parallelized and CPU-parallelized simulators, and observe that the optimal configuration improves training throughput, with speedups of up to 1.88×. This finding underscores the necessity of supporting diverse allocation modes. Notably, RLinf-VLA achieves up to 2.27× speedup over existing frameworks.

Figure 1. Evaluation of system efficiency on ManiSkill, LIBERO, and RoboTwin, comparing OpenVLA and OpenVLA-OFT. Throughput (total environment frames per second) improves with increasing pipeline stages.
Throughput for different settings Latency breakdown

What are the actionable practices for applying PPO and GRPO to VLA training?

Through extensive ablation studies, we identify the key factors that govern training performance, offering practical guidelines for effectively deploying RL in VLA settings.

  • Action-level value estimation outperforms chunk-level estimation for PPO with action chunks.
  • Partial reset substantially improves sample efficiency.
  • Trajectory length normalization in GRPO can lead to substantially higher performance.
  • Valid action mask in GRPO improves sample efficiency and avoids redundant updates.
  • Success rate filtering can improve training stability in some settings of GRPO.

BibTeX

@article{zang2025rlinf,
  title     = {RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models},
  author    = {Hongzhi Zang and Mingjie Wei and Si Xu and Yongji Wu and Zhen Guo and Yuanqing Wang and Hao Lin and Peihong Wang and Liangzhi Shi and Yuqing Xie and Zhexuan Xu and Zhihao Liu and Kang Chen and Wenhao Tang and Quanlu Zhang and Weinan Zhang and Chao Yu and Yu Wang},
  journal   = {arXiv preprint arXiv:2510.06710},
  year      = {2025}
}