7.4.4
The Goldilocks Rule: Prompt Difficulty and Re-filtering
Learn the recommended 20–80% prompt pass-rate range and why prompts should be re-filtered every 500 training steps.
What is it?
The Goldilocks rule recommends filtering prompts so that the current model solves approximately 20–80% of them. This gives groups a reasonable chance of containing both successes and failures. The source recommends repeating this filtering every 500 training steps as the model improves.
Key points
Target prompt pass rate: 20–80%.
Filter prompts according to current model capability.
Re-filter every 500 training steps.
As the model improves, previously difficult prompts may become easy.
Maintain a useful mixture of successful and unsuccessful responses.
Keeping Prompt Difficulty at the Right Level
A prompt initially solved by only 10% of responses may be too difficult. Another solved by 100% of responses provides no reward variation. Filtering toward a 20–80% pass rate helps maintain useful learning signals as training progresses.
Real-world application
Measure prompt pass rates during training and refresh the training prompt set every 500 steps, as recommended by the source.
Key takeaway
The Goldilocks rule keeps prompts challenging enough to produce both successful and unsuccessful responses.