Towards Learning Whole-Body Mobile Manipulation for Grasping Any Object Anywhere

Jason Liu, Changwei Yao, Andrew Wang

Carnegie Mellon University

Project ReportGoogle Slide

Full System Demo

Teaser

Abstract

Generalizable whole-body mobile manipulation is challenging because robots must coordinate navigation, active perception, collision avoidance, and dexterous interaction under partial observability. In this project, we present a fully simulation-based framework for learning mobile dexterous grasping without human demonstrations.

Our key idea is to decompose the task into free-space motion and contact-rich local interaction. A GPU-vectorized whole-body controller coordinates end-effector reaching, active vision, and collision avoidance, while local reinforcement learning policies focus on dexterous contact behavior near the object.

We then distill these controller-guided teachers into a single vision-based student policy that directly controls the mobile base, arm, hand, and active camera from onboard point clouds and proprioception. The resulting policy demonstrates robust whole-body grasping behavior across diverse objects and scenes.

System Overview

Platform

Mobile manipulation platform with a holonomic base, Franka Panda arm, LEAP hand, and active-vision neck camera.

Our 32-DoF platform combines a holonomic mobile base, 7-DoF arm, 16-DoF dexterous hand, and a 6-DoF active-vision neck carrying an Intel RealSense D435 camera, together with onboard Unitree L2 3 LiDAR for scene geometry.

Free-Space Control

Controller

We first solve the free-space phase with a GPU-vectorized whole-body controller. It coordinates end-effector reaching, active camera gaze, and collision avoidance before contact, so learning does not need to rediscover these high-level whole-body behaviors from scratch.

Whole-body controller objectives including reaching, vision gaze control, and collision avoidance.

The controller provides geometric priors for whole-body motion, aligning the end effector, keeping the target in view, and avoiding collisions throughout the approach phase.

Local RL Policy Training

Contact-Rich Interaction

These local teacher policies specialize in short-horizon dexterous interaction near the object. By training the final grasp stage separately, the overall learning problem becomes substantially more stable and data efficient.

Local RL Teacher 1
Local RL Teacher 2
Local RL Teacher 3

WBC Policy Rollout

Data Collection

Use controller-guided free-space motion couples the RL policy to collect data.
Point-cloud observations from onboard depth camera and LiDAR.
During rollout generation, we additionally save depth and LiDAR observations so the later student policy can be distilled from realistic onboard sensing rather than privileged simulator state.

Distillation Pipeline

Method Overview

Visual distillation pipeline showing teacher policy, whole-body controller, point-set abstraction, transformer encoder, and student policy.

Teacher rollouts from the whole-body controller and local dexterous policies are distilled into a single student that acts from onboard point clouds, proprioception, and target conditioning.

Qualitative Results

Final Policy

The final distilled policy exhibits active vision and robust whole-body grasping across both single-object and multi-object scenarios.

Single-Object Active Vision

The robot actively reorients the camera to keep the target visible during approach and pre-grasp alignment.

Multi-Object Picking

A single whole-body policy generalizes across objects and cluttered arrangements while preserving safe motion.

Simulation Ablations

Quantitative Results

Training reward curves comparing the full system, no active vision, and a direct joint-space policy.

Training Reward

The full controller-guided system reaches the highest reward, while removing active vision or directly learning joint-space control leads to worse optimization.

Lifting success curves comparing the full system, no active vision, and a direct joint-space policy.

Lifting Success Rate

Active vision and structured whole-body control both matter: the full system reaches near-perfect lifting success, while direct joint-space learning trails far behind.