NeurIPS 2024 · 38th Conference on Neural Information Processing Systems · Vancouver, Canada

Diffusion-based Curriculum Reinforcement Learning

Erdi Sayar¹ Giovanni Iacca² Ozgur S. Oguz³ Alois Knoll¹
¹ Technical University of Munich (TUM), Germany ² University of Trento, Italy ³ Bilkent University, Turkey
Paper (NeurIPS Proceedings) Direct PDF Code (GitHub) Abstract Maze Benchmarks Robot Tasks Cite
DiCuRL Generative Process & Curriculum Evolution
Animated Diffusion Curriculum Goal Generation in PointUMaze
Reverse Diffusion Process Over Training
Figure 1: Overview of DiCuRL. Left: Real-time animation of the conditional diffusion reverse sampling process generating intermediate goals along the manifold of the U-Maze, progressively bridging initial state to the distant goal area without crossing walls. Right: The reverse diffusion process at successive training milestones (\(t=2\,000\), \(t=19\,000\), \(t=34\,000\)), showing how curriculum goal distributions adaptively evolve from localized exploration to the target goal zone.
Abstract & Key Contributions

Curriculum Reinforcement Learning (CRL) is an approach to facilitate the learning process of agents by structuring tasks in a sequence of increasing complexity. Despite its potential, many existing CRL methods struggle to efficiently guide agents toward desired outcomes, particularly in the absence of domain knowledge.

This paper introduces DiCuRL (Diffusion Curriculum Reinforcement Learning), a novel method that leverages conditional diffusion models to generate curriculum goals. To estimate how close an agent is to achieving its goal, our method uniquely incorporates a \(Q\)-function and a trainable reward function based on Adversarial Intrinsic Motivation (AIM) within the diffusion model. Furthermore, it promotes exploration through the inherent noising and denoising mechanism present in the diffusion models and is environment-agnostic.

This combination allows for the generation of challenging yet achievable goals, enabling agents to learn effectively without relying on domain knowledge. We demonstrate the effectiveness of DiCuRL in three different maze environments and two robotic manipulation tasks simulated in MuJoCo, where it outperforms or matches nine state-of-the-art CRL algorithms from the literature.

Conditional Diffusion Sampling

Captures the true manifold of visited states via continuous denoising, enabling robust exploration across complex geometries without heuristic priors.

Dual \(Q\) & AIM Conditioning

Conditions diffusion reverse steps on both the critic \(Q\)-value (ensuring feasibility) and the learned AIM potential (ensuring directional goal progress).

State-of-the-Art Performance

Outperforms or matches 9 CRL baselines across complex mazes (U-Maze, N-Maze, Spiral-Maze) and 7-DOF continuous Fetch manipulation benchmarks.

Methodology & Mathematical Formulation

DiCuRL generates curriculum goals \(g \in \mathcal{G}_c\) by training a conditional diffusion probabilistic model on previously visited states, conditioned on state-evaluation signals:

$$p_\theta(x_{0:K} \mid c) = p(x_K) \prod_{k=1}^K p_\theta(x_{k-1} \mid x_k, c), \quad \text{where } c = \big[ Q_\phi(s, g), \; r_{\text{AIM}}(s, g) \big]$$

The conditioning vector \(c\) balances two essential curriculum learning criteria:

  • Critic \(Q\)-value \(Q_\phi(s, g)\): Measures policy feasibility. Conditioning on high \(Q\)-values forces the diffusion model to sample goals that are within reach of the agent's current capability.
  • Adversarial Intrinsic Motivation (AIM) Reward \(r_{\text{AIM}}(s, g)\): Formulated via Wasserstein-1 distance minimization between the state visitation distribution and the desired goal distribution: $$r_{\text{AIM}}(s) = -\log\big(1 - D_\psi(s)\big) + \log\big(D_\psi(s)\big)$$ This potential function monotonically increases along optimal trajectories toward the target, continuously pulling the curriculum forward.

Progression of Value and Reward Landscapes Across Training

Timestep 2000
Early Phase (\(t=2\,000\))
Timestep 19000
Mid Phase (\(t=19\,000\))
Timestep 34000
Convergence Phase (\(t=34\,000\))
Continuous Maze Navigation Benchmarks

We benchmarked DiCuRL against 9 state-of-the-art CRL algorithms: CURROT, GRADIENT, OUTPACE, HGG, GoalGAN, ALP-GMM, VDS, PLR, and ACL across 3 standard non-convex continuous maze environments.

PointUMaze Benchmark
PointUMaze
PointNMaze Benchmark
PointNMaze
PointSpiralMaze Benchmark
PointSpiralMaze

Curriculum Goal Sampling Comparison vs. Baselines

Visual comparison of intermediate goals generated by DiCuRL versus leading baselines. DiCuRL strictly adheres to the passable geometric manifold without attempting invalid shortcuts through walls.

PointSpiralMaze (High-Complexity Barrier)
DiCuRL Spiral
DiCuRL (Ours)
OUTPACE Spiral
OUTPACE
GRADIENT Spiral
GRADIENT
HGG Spiral
HGG
PointUMaze
DiCuRL UMaze
DiCuRL (Ours)
OUTPACE UMaze
OUTPACE
GRADIENT UMaze
GRADIENT
HGG UMaze
HGG
7-DOF MuJoCo Robotic Manipulation Benchmarks

To demonstrate scalability to continuous high-dimensional systems, we integrated DiCuRL into PyTorch DDPG for 7-DOF Fetch manipulator benchmarks with substantially expanded desired goal areas.

FetchPush Environment Setup FetchPush Results
FetchPush (Expanded Goal Region)
FetchPickAndPlace Environment Setup FetchPickAndPlace Results
FetchPickAndPlace (3D Grasp & Lift)
Figure 4: Robotic manipulation setups (yellow: object sampling; cyan: desired goal distribution) and test success rate curves comparing DiCuRL against HGG and HER across 5 independent random seeds.
Ablation Studies & Conditioning Analysis
PointSpiralMaze Ablation Curve
Component Ablation: Comparison between Full DiCuRL, DiCuRL without \(Q\)-function guidance, and DiCuRL without AIM reward.
Learned AIM Potential Field
Learned AIM Potential Field: Heatmap showing how the potential function increases strictly along the spiral passage toward the goal.
Without Q Conditioning
Without \(Q\)-Conditioning: Goals are sampled prematurely in distant regions before the agent has acquired the skills to reach them.
Without Reward Conditioning
Without AIM Reward: Goals fail to progress along the spiral manifold, diffusing diffusely into dead ends.
BibTeX Citation
@inproceedings{sayar2024diffusion,
  title     = {Diffusion-based Curriculum Reinforcement Learning},
  author    = {Sayar, Erdi and Iacca, Giovanni and Oguz, Ozgur S. and Knoll, Alois},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS 2024)},
  volume    = {37},
  pages     = {97587--97617},
  year      = {2024},
  doi       = {10.52202/079017-3096},
  url       = {https://proceedings.neurips.cc/paper_files/paper/2024/hash/b0e89a49af1fb2ebea69bfc39df0be4a-Abstract-Conference.html}
}