OPERNEX LEARNING

Learn Agentic AI

A structured, practical learning path from language model foundations to intelligent agentic systems.

6 Learning parts
453 Subtopics
415 Published lessons
RL Methods for LLMs / PPO - Proximal Policy Optimization / Detailed Mechanics: Logits and Policy Updates
Lesson 190 of 415 6 min read
5.7.5

Continuous Action Spaces Extension

Understand the source’s extension of PPO mechanics to continuous action spaces.

What is it?

In continuous action spaces, the policy network predicts distribution parameters rather than discrete token logits. The book describes a Gaussian policy using a predicted mean vector and standard deviation vector. The Gaussian log-probability is then used to calculate the PPO probability ratio and apply the same clipping objective.

Key points

1. Continuous action spaces are important in robotics reinforcement learning. 2. The network predicts a mean vector μ. 3. The network predicts a standard deviation vector σ. 4. Actions are evaluated using a Gaussian probability distribution. 5. Log-probability is calculated using the Gaussian log-PDF. 6. The probability ratio is exp(log π_θ − log π_θold). 7. The same PPO clipping objective is applied. 8. LLMs generally use discrete token action spaces rather than this continuous-action formulation.

Example: A Robot Choosing Movement

Imagine a robot controlling a movement dimension. Predicted mean μ = 0.5 Predicted standard deviation σ = 0.2 The policy represents a Gaussian distribution around the predicted mean. An action is sampled from this distribution. The Gaussian log-probability of the sampled action can be compared between the live and old policies to calculate the PPO ratio.

Real-world application

Continuous-action PPO is useful in robotics, where an agent selects values such as movement speed, steering angle or motor control rather than discrete language tokens.
Key takeaway

PPO supports continuous actions by using Gaussian distribution parameters and the same probability-ratio clipping mechanism.