ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models
Abstract
Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors. Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks. We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL. The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes. Fixed visual features keep the inherited learner and its replay compatible with base-policy updates. Across contact-rich assembly sequences in simulation and on real-robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-robot training time by 31.0\% to 68.5\%.
Community
๐ค ๐ฅ๐ฒ๐ฎ๐น-๐ฟ๐ผ๐ฏ๐ผ๐ ๐ฅ๐. ๐๐ฒ๐ ๐ฑ๐ฒ๐บ๐ผ๐ป๐๐๐ฟ๐ฎ๐๐ถ๐ผ๐ป๐. ๐ก๐ฒ๐ ๐๐ธ๐ถ๐น๐น๐ ๐ถ๐ป ๐บ๐ถ๐ป๐๐๐ฒ๐.
Introducing ๐๐๐๐ฅ๐จ๐: Continual Off-Policy Reinforcement Learning for Robot Foundation Models.
We improve robot foundation model behavior with RL and carry that experience across tasks, helping robots ๐น๐ฒ๐ฎ๐ฟ๐ป ๐ณ๐ฎ๐๐๐ฒ๐ฟ ๐๐ต๐ถ๐น๐ฒ ๐ฟ๐ฒ๐๐ฎ๐ถ๐ป๐ถ๐ป๐ด ๐ฒ๐ฎ๐ฟ๐น๐ถ๐ฒ๐ฟ ๐๐ธ๐ถ๐น๐น๐.
๐ฆพ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐ฑ๐ถ๐ฟ๐ฒ๐ฐ๐๐น๐ ๐ถ๐ป ๐๐ต๐ฒ ๐ฟ๐ฒ๐ฎ๐น ๐๐ผ๐ฟ๐น๐ฑ
For our real-world policies, all RL interaction comes from the physical robot. Each task starts with ๐ฎ ๐๐บ๐ฎ๐น๐น ๐๐ฒ๐ ๐ผ๐ณ ๐ฑ๐ฒ๐บ๐ผ๐ป๐๐๐ฟ๐ฎ๐๐ถ๐ผ๐ป๐ to fine-tune the foundation model.
๐ป ๐๐ผ๐ ๐ฎ๐ฑ๐ฑ๐ถ๐๐ถ๐ผ๐ป๐ฎ๐น ๐๐ฃ๐จ ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด ๐ผ๐๐ฒ๐ฟ๐ต๐ฒ๐ฎ๐ฑ
During RL, the foundation model stays frozen. Only a compact actorโcritic is updated to refine its reference actions.
๐ง ๐ ๐ฎ๐ธ๐ฒ ๐๐ต๐ฒ ๐ฒ๐
๐ฝ๐ฒ๐ฟ๐ถ๐ฒ๐ป๐ฐ๐ฒ ๐ฐ๐ผ๐ป๐๐ถ๐ป๐๐ฎ๐น
We inherit that actorโcritic for the next task and distill earlier behavior on a small fraction of prior-task replay.
โฑ๏ธ ๐ก๐ฒ๐ ๐๐ธ๐ถ๐น๐น๐ ๐ถ๐ป ๐บ๐ถ๐ป๐๐๐ฒ๐ ๐ผ๐ณ ๐ฟ๐ผ๐ฏ๐ผ๐ ๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป
Using ฯ0.5 + FastTD3, subsequent real-world tasks reach the 95% success stopping criterion in about ๐ฐโ๐ญ๐ฎ ๐บ๐ถ๐ป๐๐๐ฒ๐ ๐ผ๐ณ ๐ฟ๐ผ๐ฏ๐ผ๐ ๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป.
Compared with single-task RL:
โข AutoMate assembly: ๐ฏ๐ญ.๐ฌโ๐ฒ๐ด.๐ฑ% ๐น๐ฒ๐๐ ๐ฟ๐ผ๐ฏ๐ผ๐ ๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป
โข NIST assembly: ๐ฏ๐ด.๐ณโ๐ฐ๐ฎ.๐ต% ๐น๐ฒ๐๐ ๐ฟ๐ผ๐ฏ๐ผ๐ ๐ถ๐ป๐๐ฒ๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป
Both methods use the same stopping rule over the latest 20 episodes. These are robot interaction times, including online warmup, excluding demonstration collection and foundation-model fine-tuning.
๐ก๏ธ ๐๐ฎ๐ฟ๐น๐ถ๐ฒ๐ฟ ๐๐ธ๐ถ๐น๐น๐ ๐ฟ๐ฒ๐บ๐ฎ๐ถ๐ป
After each full sequence, the final policies achieve ๐ญ๐ญ๐ด/๐ญ๐ฎ๐ฌ ๐๐๐ฐ๐ฐ๐ฒ๐๐๐ฒ๐ across six prior tasks (๐ต๐ด.๐ฏ% ๐ผ๐๐ฒ๐ฟ๐ฎ๐น๐น). Each task is evaluated separately for 20 trials across the two independently trained suites.
We also evaluate ACCRUE separately in simulation using GR00T + FastSAC.
๐ฌ Watch the attached ๐ฒ๐ฌ-๐๐ฒ๐ฐ๐ผ๐ป๐ฑ ๐ผ๐๐ฒ๐ฟ๐๐ถ๐ฒ๐, then explore more videos and interactive results on the project page.
๐ Paper: https://arxiv.org/abs/2610.18596
๐ Project: https://accrue-rl.github.io/
Get this paper in your agent:
hf papers read 2610.18596 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper