CSE510 Deep Reinforcement Learning (Lecture 18)

Model-based RL framework

Model Learning with High-Dimensional Observations

• Learning model in a latent space with observation reconstruction
• Learning model in a latent space without observation reconstruction
• Learning model in the observation space (i.e., videos)

Naive approach:

If we knew $f(s_t,a_t)=s_{t+1}$, we could use the tools from last week. (or $p(s_{t+1}| s_t, a_t)$ in the stochastic case)

So we can learn $f(s_t,a_t)$ from data, and then plan through it.

Model-based reinforcement learning version 0.5:

1. Run base polity $\pi_0$ (e.g. random policy) to collect $\mathcal{D} = \{(s_t, a_t, s_{t+1})\}_{t=0}^\top$
2. Learn dynamics model $f(s_t,a_t)$ to minimize $\sum_{i}\|f(s_i,a_i)-s_{i+1}\|^2$
3. Plan through $f(s_t,a_t)$ to choose action $a_t$

Sometime, it does work!

• Essentially how system identification works in classical robotics
• Some care should be taken to design a good base policy
• Particularly effective if we can hand-engineer a dynamics representation using our knowledge of physics, and fit just a few parameters

However, Distribution mismatch problem becomes worse as we use more expressive model classes.

Version 0.5: collect random samples, train dynamics, plan

• Pro: simple, no iterative procedure
• Con: distribution mismatch problem

Version 1.0: iteratively collect data, replan, collect data

• Pro: simple, solves distribution mismatch
• Con: open loop plan might perform poorly, esp. in stochastic domains

Version 1.5: iteratively collect data using MPC (replan at each step)

• Pro: robust to small model errors
• Con: computationally expensive, but have a planning algorithm available

Version 2.0: backpropagate directly into policy

• Pro: computationally cheap at runtime
• Con: can be numerically unstable, especially in stochastic domains
• Solution: model-free RL + model-based RL

Final version:

1. Run base polity $\pi_0$ (e.g. random policy) to collect $\mathcal{D} = \{(s_t, a_t, s_{t+1})\}_{t=0}^\top$
2. Learn dynamics model $f(s_t,a_t)$ to minimize $\sum_{i}\|f(s_i,a_i)-s_{i+1}\|^2$
3. Backpropagate through $f(s_t,a_t)$ into the policy to optimized $\pi_\theta(s_t,a_t)$
4. Run the policy $\pi_\theta(s_t,a_t)$ to collect $\mathcal{D} = \{(s_t, a_t, s_{t+1})\}_{t=0}^\top$
5. Goto 2

Model Learning with High-Dimensional Observations

• Learning model in a latent space with observation reconstruction
• Learning model in a latent space without observation reconstruction