3.1.5
Markov Property and Agent-Environment Interaction
Understand the Markov property and the step-by-step interaction between an agent and its environment.
What is it?
The Markov property states that the future state depends on the current state and action rather than the complete history of previous states and actions.
Key points
1. Current state contains relevant information about the past.
2. Next-state distribution depends on current state and action.
3. Agent observes sₜ.
4. Agent selects aₜ using policy π(a|s).
5. Environment transitions to sₜ₊₁.
6. Agent receives reward rₜ and repeats.
Example: Agent-environment interaction
1. Observe: Agent receives state sₜ.
2. Act: Agent selects action aₜ.
3. Transition: Environment moves to sₜ₊₁.
4. Reward: Agent receives rₜ.
5. Repeat: Agent observes the new state and chooses another action.
Real-world application
The Markov property helps developers design state representations for reinforcement learning environments.
Key takeaway
The Markov property represents relevant history through the current state.