Tabular RL + Classic Control
This page turns the tabular RL foundations into a visual intuition builder. Move the sliders to see how discounting, uncertainty, and time pressure change the best route through a small frozen-lake style world.
What changed
With moderate slip and a strong discount, the value of the goal propagates backward through the map and shapes a stable route.
Higher discount lets far-away rewards matter more right now.
When movement is unreliable, danger spreads beyond the holes themselves.
Step penalties decide whether wandering feels acceptable or expensive.
Gridworld
The numbered trace follows the greedy policy implied by the current values.
Policy Trace
Curator Guide
The dynamic-programming notebook is the cleanest point in the archive where reinforcement learning still fits in your head all at once. This demo isolates the value-iteration part of that notebook so users can see how values propagate through a map before later sections replace exact tables with sampling, approximation, and deep networks.
From the coursework
The archived dynamic-programming material walks step-by-step through iterative policy evaluation, recovering q-values from v-values, policy improvement, policy iteration, truncated policy iteration, and finally value iteration on a slippery FrozenLake environment.
Why value iteration first
Value iteration compresses the whole planning loop into repeated Bellman optimality backups. That makes it the best visual bridge from exact tabular reasoning to the approximate value updates you later see in Monte Carlo control, temporal-difference learning, Q-learning, and DQN.
What to notice
The grid is small enough that you can watch the trade-offs directly.
Bridge to later sections
Once the state space gets too large, exact tabular updates stop being practical. But the logic does not disappear. DQN, actor-critic methods, and other later algorithms still revolve around estimating future return and improving actions using that estimate.
Source Trail
Primary coursework notebook for policy evaluation, policy iteration, truncated policy iteration, and value iteration.
source-material/dynamic-programming/Dynamic_Programming.ipynb
Canonical solutions to the same exercises, useful when reconstructing exact update rules.
source-material/dynamic-programming/Dynamic_Programming_Solution.ipynb
The archived custom environment that defines the same 4x4 map and slippery transition idea used here.
source-material/dynamic-programming/frozenlake.py
Shows what changes when the agent stops assuming full transition knowledge and starts learning from sampled episodes.
source-material/monte-carlo/Monte_Carlo.ipynb
Extends the story from exact planning into Sarsa, Q-learning, and Expected Sarsa.
source-material/temporal-difference/Temporal_Difference.ipynb
Compact reference sheet for the value-function and policy-update family used across the foundations material.
source-material/classwork/cheatsheet.pdf
Glossary
A score for how promising a state is if you follow a policy from there onward.
One update that rewrites a state's value using immediate reward plus discounted estimates of what comes next.
The step where you turn current value estimates into better action choices.
The number between 0 and 1 that controls how strongly distant rewards matter today.
The chance that the environment moves you sideways instead of where you intended to go.
The reward or penalty you receive simply for taking another step before the episode ends.