CSE510 Deep Reinforcement Learning (Lecture 16)

Deterministic Policy Gradient (DPG)

Deterministic policy gradients [Silver et al., ICML 2014]
- Explicitly learn a deterministic policy.
- $a = \mu_\theta(s)$
Advantages
- Existing optimal deterministic policy for MDPs
- Naturally dealing with a continuous action space
- Expected to be more efficient than learning stochastic policies
  - Computing stochastic gradient requires more samples, as it integrates over both state and action space.
  - Deterministic gradient is preferable as it integrates over state space only.

The objective function is:

J(\theta)=\int_{s\in S} \rho^{\mu}(s) r(s,\mu_\theta(s)) ds

where $\rho^{\mu}(s)$ is the stationary distribution under the behavior policy $\mu_\theta(s)$ .

The policy gradient from the standard policy gradient theorem is:

\nabla_\theta J(\theta) = \mathbb{E}_{s\sim \rho^{\mu}}[\nabla_\theta Q^{\mu_\theta}(s,a)]=\mathbb{E}_{s\sim \rho^{\mu}}[\nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)}]

The formulations up to now can only use on-policy data.

Deterministic policy can hardly guarantee sufficient exploration.

Use a stochastic behavior policy $\beta(a|s)$ . The modified objective function is:

J(\mu_\theta)=\int_{s\in S} \rho^{\beta}(s) Q^{\mu_\theta}(s,\beta(s)) ds

The gradients are:

\begin{aligned} \nabla_\theta J(\mu_\theta) &\approx \int_{s\in S} \rho^{\beta}(s) \nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)} ds\\ &= \mathbb{E}_{s\sim \rho^{\beta}}[\nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)}] \end{aligned}

Importance sampling is avoided in the actor due to the absence of integral over actions.

Importance sampling can also be avoided in the critic.

Gradient TD-like algorithm can be directly applied to the critic.

\mathcal{L}_{critic}(w) = \mathbb{E}[r_t+\gamma Q^w(s_{t+1},a_{t+1})-Q^w(s_t,a_t)]^2

\delta_t=r_t+\gamma Q^w(s_{t+1},a_{t+1})-Q^w(s_t,a_t)

w_{t+1} = w_t + \alpha_w \delta_t \nabla_w Q^w(s_t,a_t)

\theta_{t+1} = \theta_t + \alpha_\theta \nabla_\theta \mu_\theta(s_t) \nabla_a Q^{\mu_\theta}(s_t,a_t)\vert_{a=\mu_\theta(s_t)}

Insights from DQN + Deterministic Policy Gradients


- Use a replay buffer

\mathcal{L}_{critic}(w) = \mathbb{E}[r_t+\gamma Q^w(s_{t+1},a_{t+1})-Q^w(s_t,a_t)]^2

Actor is updated every timestep:

\nabla_a Q(s_t,a;w)|_{a=\mu_\theta(s_t)} \nabla_\theta \mu_\theta(s_t)

Smoothing target updated at every timestep:

w_{t+1} = \tau w_t + (1-\tau) w_{t+1}

\theta_{t+1} = \tau \theta_t + (1-\tau) \theta_{t+1}

Exploration: add noise to the action selection: $a_t = \mu_\theta(s_t) + \mathcal{N}_t$

Batch normalization used for training networks

Overestimation bias is an issue of Q-learning in which the maximization of a noisy value estimate

DDPG:\nabla_\theta J(\theta) = \mathbb{E}_{s\sim \rho^{\mu}}[\nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu_\theta}(s,a)\vert_{a=\mu_\theta(s)}]

Because the slow-changing policy in an actor-critic setting

the current and target value estimates remain too similar to avoid maximization bias.
Target value of Double DQN: $r_t + \gamma Q^w'(s_{t+1},\mu_\theta(s_{t+1}))$

Address overestimation bias:

Double Q-learning is unbiased in tabular settings, but still slight overestimation with function approximation.

y_1 = r + \gamma Q^{\theta_2'}(s', \pi_{\phi_1}(s'))

y_2 = r + \gamma Q^{\theta_1'}(s', \pi_{\phi_2}(s'))

It is possible that $Q^{\theta_2}(s, \pi_{\phi_1}(s)) > Q^{\theta_1}(s, \pi_{\phi_1}(s))$

Clipped double Q-learning:

y_1 = r + \gamma \min_{i=1,2} Q^{\theta_i'}(s', \pi_{\phi_i}(s'))

High-variance estimates provide a noisy gradient.

Techniques in TD3 to reduce the variance:

Update the policy at a lower frequency than the value network.
Smoothing the value estimate: $y=r+\gamma \mathbb{E}_{\epsilon}[Q^{\theta'}(s', \pi_{\phi'}(s')+\epsilon)]$

Update target:

y=r+\gamma \mathbb{E}_{\epsilon}[Q^{\theta'}(s', \pi_{\phi'}(s')+\epsilon)]

where $\epsilon\sim clip(\mathcal{N}(0, \sigma), -c, c)$

Generalizable Episode Memory for Deep Reinforcement Learning
Distributed Distributional Deep Deterministic Policy Gradient
- Distributional critic
- N-step returns are used to update the critic
- Multiple distributed parallel actors
- Prioritized experience replay