5.7.5
Continuous Action Spaces Extension
Understand the source’s extension of PPO mechanics to continuous action spaces.
What is it?
In continuous action spaces, the policy network predicts distribution parameters rather than discrete token logits. The book describes a Gaussian policy using a predicted mean vector and standard deviation vector. The Gaussian log-probability is then used to calculate the PPO probability ratio and apply the same clipping objective.
Key points
1. Continuous action spaces are important in robotics reinforcement learning.
2. The network predicts a mean vector μ.
3. The network predicts a standard deviation vector σ.
4. Actions are evaluated using a Gaussian probability distribution.
5. Log-probability is calculated using the Gaussian log-PDF.
6. The probability ratio is exp(log π_θ − log π_θold).
7. The same PPO clipping objective is applied.
8. LLMs generally use discrete token action spaces rather than this continuous-action formulation.
Example: A Robot Choosing Movement
Imagine a robot controlling a movement dimension.
Predicted mean μ = 0.5
Predicted standard deviation σ = 0.2
The policy represents a Gaussian distribution around the predicted mean. An action is sampled from this distribution.
The Gaussian log-probability of the sampled action can be compared between the live and old policies to calculate the PPO ratio.
Real-world application
Continuous-action PPO is useful in robotics, where an agent selects values such as movement speed, steering angle or motor control rather than discrete language tokens.
Key takeaway
PPO supports continuous actions by using Gaussian distribution parameters and the same probability-ratio clipping mechanism.