Multi-Objective Evolutionary Hindsight Experience Replay for Robot Manipulation Tasks
Reinforcement learning (RL) algorithms often face challenges in efficiently learning effective policies for sparse-reward multi-goal robot manipulation tasks, thus requiring a vast amount of experiences. The state-of-the-art algorithm in the field, Hindsight Experience Replay (HER), addresses this issue by using failed trajectories and replacing the desired goal with hindsight goals. However, HER performs poorly when the desired goal is distant from the initial state. To address this limitation, Hindsight Goal Generation (HGG) has been proposed, which generates a curriculum of goals from already visited states. This curriculum generation is based on a single objective, and does not take obstacles into account.
Here, we make a step forward by proposing Multi-Objective Evolutionary Hindsight Experience Replay (MOEHER), a novel curriculum RL algorithm that reformulates curriculum generation considering multiple objectives and obstacles. MOEHER utilizes NSGA-II to generate a curriculum that is optimized w.r.t. four objectives, namely the \(Q\)-function, the goal-proximity function, and two distance metrics, while simultaneously satisfying constraints on the obstacles. We evaluate MOEHER on four different sparse-reward robot manipulation tasks, with and without obstacles, and compare it with HER and HGG. The results demonstrate that MOEHER surpasses or performs on par with these methods on the tested tasks.
4-Objective Optimization
Concurrently optimizes the \(Q\)-network, goal-proximity potential function, Wasserstein-1 distribution distance, and Euclidean distance using NSGA-II.
Obstacle-Aware Repair
Explicitly enforces physical obstacle boundary constraints and genetic variation repairs, ensuring curriculum goals never venture inside restricted obstacle spaces.
K-Means Pareto Clustering
Applies K-Means clustering directly onto non-dominated solutions to extract diverse, non-redundant curriculum goals matched to rollout episodes.
In multi-goal reinforcement learning with sparse rewards, the agent aims to reach goals sampled from a continuous distribution \(\mathcal{G}\). MOEHER frames curriculum generation as a multi-objective optimization problem over states \(s\):
\(Q\)-Value Maximization
\(f_1(s) = Q_{\phi}(s, m(s), \pi_{\psi}(s, m(s)))\)
Maximizes the expected cumulative reward by evaluating states through the policy \(\pi_{\psi}\) and critic \(Q_{\phi}\), identifying promising states with high learning value.
Goal-Proximity Potential
\(f_2(s) = f_{\theta}^{\pi}(s)\)
Estimates the step progression toward the target via a neural potential function trained on the Kantorovich–Rubinstein duality, preventing states from collapsing into local critic maxima.
Wasserstein-1 Distance
\(f_3(s) = W_1(m(s), \mathcal{G})\)
Minimizes the earth mover's distance between the intermediate goal distribution and the desired goal distribution, ensuring broad exploration along feasible paths.
Euclidean Target Distance
\(f_4(s) = \|m(s) - g\|_2\)
Minimizes the straight-line Euclidean distance between the candidate goal and the current target goal \(g\), anchoring convergence toward task completion.
Obstacle Constraint Validation: Whenever obstacles are present, candidate solutions are checked against obstacle coordinates \([\mathcal{O}^-, \mathcal{O}^+]\). If an intermediate goal falls inside an obstacle, genetic variation operators (binary crossover and polynomial mutation) iteratively resample candidates until all cluster centers are outside the obstacle boundaries.
We benchmarked MOEHER against HER (uniform hindsight replay) and HGG (Hindsight Goal Generation) on 4 simulated 7-DOF Fetch manipulator tasks across 10 independent runs each.
Tasks Without Obstacles
Tasks With Obstacles (Obs)
Statistical Significance (Wilcoxon Rank-Sum vs. HGG)
| Environment | Obstacle Present? | Statistic | \(p\)-value | Significance Status |
|---|---|---|---|---|
| FetchSlideObs | Yes | -3.6556 | 0.00013 | Significant Improvement (p < 0.001) |
| FetchReachObs | Yes | -3.6556 | 0.00013 | Significant Improvement (p < 0.001) |
| FetchPushObs | Yes | -2.4000 | 0.00820 | Significant Improvement (p < 0.01) |
| FetchPickAndPlace | No | -2.3553 | 0.00925 | Significant Improvement (p < 0.01) |
| FetchPickAndPlaceObs | Yes | -1.5045 | 0.06623 | Near-Significant (p ≈ 0.06) |
| FetchReach | No | -0.7460 | 0.22782 | Comparable / On Par |
| FetchSlide | No | 2.5236 | 0.99419 | Comparable / On Par |
| FetchPush | No | 2.7418 | 0.99694 | Comparable / On Par |
Visualization of curriculum goals generated during training. The color gradient from red to magenta illustrates the evolution of intermediate goals across successive training epochs, demonstrating how MOEHER naturally navigates around obstacles without getting trapped.
Task Goal and Object Sampling Areas
@inproceedings{Sayar_2024,
title = {Multi-Objective Evolutionary Hindsight Experience Replay for Robot Manipulation Tasks},
author = {Sayar, Erdi and Iacca, Giovanni and Knoll, Alois},
booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '24)},
pages = {403--411},
year = {2024},
month = {July},
publisher = {ACM},
series = {GECCO '24},
doi = {10.1145/3638529.3654045},
url = {https://doi.org/10.1145/3638529.3654045}
}