Contact Energy Based Hindsight Experience Prioritization

Interactive Robot Demonstration Showcase

Abstract & Key Contributions

Multi-goal robot manipulation tasks with sparse rewards are notoriously difficult for reinforcement learning (RL) algorithms due to the extreme inefficiency in collecting successful experiences. While Hindsight Experience Replay (HER) expedites learning by substituting the desired goal with achieved goals from failed trajectories, standard HER samples transitions uniformly without distinguishing between informative physical interactions and idle, uninformative motions.

In this paper, we address the sample inefficiency of HER and introduce novel force-based and work-based hindsight experience prioritization mechanisms that leverage force/torque sensing in the robot's gripper. Contact-rich transitions—which carry the dense dynamics of manipulator-object interaction—are sampled with higher probability from the replay buffer. Extensive evaluations across challenging MuJoCo benchmarks and real-world Franka Emika Panda experiments demonstrate that our method dramatically accelerates learning and surpasses state-of-the-art baselines.

Contact-Energy Prioritization

Prioritizes episodes based on cumulative contact force and mechanical work between gripper and object, focusing the policy on crucial manipulation events.

Sample-Efficient Replay

Replaces uniform sampling in HER with physically grounded probability distributions, reducing sample complexity across standard benchmarks.

Real-World Sim2Real

Successfully transferred trained policies zero-shot to a physical 7-DOF Franka Emika Panda robot with ArUco marker computer vision tracking.

Methodology & Mathematical Formulation

Prioritizing Replay Buffers via Physical Interaction

When an agent explores sparse-reward manipulation tasks, many sampled trajectories never contact the target object. Sampling these empty trajectories uniformly slows policy convergence. To resolve this, we formulate two metrics to calculate the episode sampling probability $p_{\text{episode}}(e)$:

1. Force-Based Replay Buffer

Force sensors in the left and right gripper fingers measure net applied forces (Coriolis, centripetal, gravitational, and external contact forces). Gravitational forces are compensated using the gripper mass $mg$. The multidimensional force is projected to a scalar and cumulatively summed across timesteps:

$$f(e,t) = \sum_{j=0}^{n} \sum_{i=0}^{t} \left( \lvert f_{\text{left}}^{(j)}(e,i) \rvert + \lvert f_{\text{right}}^{(j)}(e,i) \rvert \right)$$

The episode sampling distribution is given by:

$$p_{\text{episode}}(e) = \frac{\sum_{t=0}^{T} f(e,t)}{\sum_{i_e=0}^{E} \sum_{t=0}^{T} f(i_e,t)}$$

2. Work-Based Replay Buffer

Work-based prioritization incorporates end-effector displacement $\Delta x_i^{(j)}$ to differentiate between static resting forces and active spatial manipulation:

$$w(e,t) = \sum_{j=0}^{n} \left( \sum_{i=0}^{t} \left( \lvert f_{\text{left}}^{(j)}(e,i) \rvert + \lvert f_{\text{right}}^{(j)}(e,i) \rvert \right) \right) \cdot \Delta x_i^{(j)}$$

$$p_{\text{episode}}(e) = \frac{\sum_{t=0}^{T} w(e,t)}{\sum_{i_e=0}^{E} \sum_{t=0}^{T} w(i_e,t)}$$

Figure 1: System architecture of Force-Based and Work-Based Hindsight Experience Prioritization integrated with Actor-Critic RL.

Simulation & Real-World Experiments

MuJoCo Multi-Goal Benchmarks

We evaluated the proposed method against vanilla HER and state-of-the-art baselines on standard 7-DOF manipulation tasks: Pick & Place, Push, and Slide across 7 random seeds.

Figure 2: Multi-goal robot manipulation benchmark environments in MuJoCo: (a) FetchPickAndPlace-v1, (b) FetchPush-v1, (c) FetchSlide-v1.

Quantitative Success Curves

Both force-based and work-based prioritization drastically accelerate learning speed and reach near-100% asymptotic success rate well before vanilla uniform HER:

Figure 3: Learning curves comparing Force-Based and Work-Based HER against standard uniform HER across manipulation benchmarks.

Real-World Deployment (Sim2Real)

The learned policies were directly transferred from simulation to a physical 7-DOF Franka Emika Panda robot. Perception was handled via overhead RGB camera sensing with ArUco fiducial markers for perspective transformation into the robot's base coordinate frame. Motion control commands were sent via the Frankx library, demonstrating robust pick, place, and pushing actions in the physical environment.

BibTeX Citation

If you find this work, code, or data useful in your research, please consider citing our ICRA 2024 paper:

@inproceedings{sayar2024contact,
  title     = {Contact Energy Based Hindsight Experience Prioritization},
  author    = {Sayar, Erdi and Bing, Zhenshan and D'Eramo, Carlo and Oguz, Ozgur S. and Knoll, Alois},
  booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
  year      = {2024},
  pages     = {Yokohama, Japan},
  publisher = {IEEE}
}