Exp 1：原始策略优化（Vanilla Policy Gradient）
Vanilla adj. 一般的，没有新意的；香草的
训练算法总体思路
次要的训练算法集中在 RL_Trainer.run_training_loop 中。通过观察一个循环（iteration）的调用过程，可总结为：
- 收集多个路线,，取得 train_batch_size 个时刻的数据 存在 replay buffer 中
- 上一步齐全完结之后，从 buffer 中采样最近 batch_size 个时刻数据
用数据训练模型
- 先更新策略
- 再更新 baseline （第一节还不须要）
第一、二步是齐全串行的。且通过观察代码实现可知，尽管有 buffer 作数据直达，但 train_batch_size 与 batch_size 定义为相等，每次训练的数据都是上次模型更新之后的。因而仍然是 On-Policy RL。
第一局部次要须要补全 PGAgent.train，这一函数又牵扯到计算 总收益 Q，以及 policy update
两种 \( \hat{Q} \) 的计算：Reward-to-go or Not
留神这里是 \( \hat{Q} \) （Q-hat），是 单个蒙特卡洛采样门路的收益值，而不是 Q-learning 中神经网络给出的 状态-动作对 的预测价值。
Reward-to-go 就是思考因果的 Q 值，t 工夫点的的 Q 不思考 t 之前工夫点的收益。
不思考因果的：求和即可，每个地位的值都和时刻 t 无关，都是一样的。
##################################################### ################## HELPER FUNCTIONS ################# ##################################################### def _discounted_return(self, rewards): """ Helper function Input: list of rewards {r_0, r_1, ..., r_t', ... r_T} from a single rollout of length T Output: list where each index t contains sum_{t'=0}^T gamma^t' r_{t'} """ discounted_sum, discount = 0, 1 for rr in rewards: discounted_sum += discount * rr discount *= self.gamma return [discounted_sum for i in range(len(rewards))]思考 Reward-to-go 的：应用迭代的方法
$$\begin{align}\hat{Q}_{t}&=\sum_{t'=t}^{T} \gamma^{t'-t} * r_{t'}\\ &=\sum_{t'=t+1}^{T} \gamma^{t'-t} * r_{t'}+r_{t}\\ &=\gamma\sum_{t'=t+1}^{T} \gamma^{t'-t-1} * r_{t'}+r_{t}\\ &=\hat{Q}_{t+1}+r_{t}\end{align}$$
而 已知：
$$\hat{Q}_{T}=\sum_{t'=T}^{T} \gamma^{t'-T} * r_{t'}=r_{T}$$
def _discounted_cumsum(self, rewards): """ Helper function which -takes a list of rewards {r_0, r_1, ..., r_t', ... r_T}, -and returns a list where the entry in each index t is sum_{t'=t}^T gamma^(t'-t) * r_{t'} (For Reward-to-go) """ rtg_discounted_q = rewards.copy() for i in range(len(rtg_discounted_q)-2, -1, -1): rtg_discounted_q[i] = self.gamma * (rtg_discounted_q[i+1]) + rewards[i] return rtg_discounted_q策略更新（Policy Updating）
策略优化的数学实质是：通过调整 策略概率模型 的散布，最大化收益的期望值 。
但通过应用 对数求导 的数学技巧，策略优化指标函数从后果上讲，能够认为是 策略对数概率 的加权均匀 ，而权重是 收益值之和。因而收益越高的决策权重越大。（当然这是感性认识，而不是数学实质）
$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^{T} \nabla_{\theta} \log \pi_{\theta}\left(\mathbf{a}_{i, t} \mid \mathbf{s}_{i, t}\right) \hat{Q}_{i, t}^{\pi}$$
class MLPPolicyPG(MLPPolicy): def __init__(self, ac_dim, ob_dim, n_layers, size, **kwargs): super().__init__(ac_dim, ob_dim, n_layers, size, **kwargs) self.baseline_loss = nn.MSELoss() def update(self, observations, actions, advantages, q_values=None): observations = ptu.from_numpy(observations) actions = ptu.from_numpy(actions) advantages = ptu.from_numpy(advantages) self.optimizer.zero_grad() observations, actions, advantages = ptu.from_numpy(observations), ptu.from_numpy(actions), ptu.from_numpy(advantages) action_distribution = self.forward(observations) log_probs = action_distribution.log_prob(actions) loss = -torch.mul(log_probs, advantages).mean() loss.backward() self.optimizer.step() if self.nn_baseline: pass # omitted train_log = { 'Training Loss': ptu.to_numpy(loss), } return train_logExp 2：Neural Network Baselines
Critic 模型用作 Baseline 或者 Critic
这届内容理论是在 Actor-Critic 章节的课程才讲的 （）
Critic 模型指的是一个学习器（比方神经网络），输出是状态（或再加上动作），输入是这个状态（或 状态-动作 对）的价值。
这一节中尽管存在 Critic 神经网络模型，但它是作为 Baseline 应用，所以仍然是 Policy Gradient，而不是 Actor-Critic。分辨的准则是，PG 在策略更新中 Advantage 值的 被减数 仍然是蒙特卡洛采样 \( \hat{Q} \)
Baseline 模型训练
如上文思路，尽管有 buffer 作数据直达，但 train_batch_size 与 batch_size 定义为相等，每次训练的数据都是上次更新之后的。
因而，buffer 只是表象，实质上还是 On-policy RL。On-Policy 假如成立。
Baseline 模型的训练思路是 函数预计：神经网络的特色抽取和拟合能力，使得它可能辨认出不同但类似的状态，从而采取类似的决策。
这种能够认为是 蒙特卡洛法的延申解释。传统蒙特卡洛法须要在 完全相同 的输出上屡次采样（ \( V^{\pi}\left(\mathbf{s}_{t}\right) \approx \frac{1}{N}\sum_{i=1}^{N}\sum_{t^{\prime}=t}^{T} r\left(\mathbf{s}_{t^{\prime}}, \mathbf{a}_{t^{\prime}}\right) \) ），这在大部分的 强化学习环境都是不可能的。
当然这里还额定思考了 收益的工夫递加。
class MLPPolicyPG(MLPPolicy): def __init__(self, ac_dim, ob_dim, n_layers, size, **kwargs): super().__init__(ac_dim, ob_dim, n_layers, size, **kwargs) self.baseline_loss = nn.MSELoss() def update(self, observations, actions, advantages, q_values=None): # Updating Policy (omitted) if self.nn_baseline: ## TODO: update the neural network baseline using the q_values as ## targets. The q_values should first be normalized to have a mean ## of zero and a standard deviation of one. ## Note: You will need to convert the targets into a tensor using ## ptu.from_numpy before using it in the loss assert q_values is not None self.baseline_optimizer.zero_grad() q_values = ptu.from_numpy(q_values) if isinstance(q_values, np.ndarray) else q_values q_mean, q_std = torch.mean(q_values), torch.std(q_values) q_values = (q_values - q_mean).divide(q_std) values = self.baseline(observations) print(values.shape, q_values.shape) b_loss = self.baseline_loss(values, q_values) b_loss.backward() self.baseline_optimizer.step() train_log = { 'Training Loss': ptu.to_numpy(loss), } return train_log引入 Baselines
很简略：
def estimate_advantage(self, obs: np.ndarray, rews_list: np.ndarray, q_values: np.ndarray, terminals: np.ndarray): """ Computes advantages by (possibly) using GAE, or subtracting a baseline from the estimated Q values """ # Estimate the advantage when nn_baseline is True, # by querying the neural network that you're using to learn the value function if self.nn_baseline: values_unnormalized = self.actor.run_baseline_prediction(obs) ## ensure that the value predictions and q_values have the same dimensionality ## to prevent silent broadcasting errors assert values_unnormalized.ndim == q_values.ndim ## TODO: values were trained with standardized q_values, so ensure ## that the predictions have the same mean and standard deviation as ## the current batch of q_values values = values_unnormalized * q_values.std() + q_values.mean() batch_size = obs.shape[0] if self.gae_lambda is not None: pass # TODO else: ## TODO: compute advantage estimates using q_values, and values as baselines advantages = np.zeros(batch_size) for i in range(batch_size): advantages[i] = q_values[i] - values[i] # Else, just set the advantage to [Q] else: advantages = q_values.copy() # Normalize the resulting advantages to have a mean of zero # and a standard deviation of one if self.standardize_advantages: ad_mean, ad_std = np.average(advantages), np.std(advantages) advantages = (advantages - ad_mean) / ad_std return advantagesExp 3：GAE
def estimate_advantage(self, obs: np.ndarray, rews_list: np.ndarray, q_values: np.ndarray, terminals: np.ndarray): """ Computes advantages by (possibly) using GAE, or subtracting a baseline from the estimated Q values """ # Estimate the advantage when nn_baseline is True, # by querying the neural network that you're using to learn the value function if self.nn_baseline: values_unnormalized = self.actor.run_baseline_prediction(obs) assert values_unnormalized.ndim == q_values.ndim values = values_unnormalized * q_values.std() + q_values.mean() batch_size = obs.shape[0] if self.gae_lambda is not None: ## append a dummy T+1 value for simpler recursive calculation values = np.append(values, [0]) ## combine rews_list into a single array rews = np.concatenate(rews_list) ## create empty numpy array to populate with GAE advantage ## estimates, with dummy T+1 value for simpler recursive calculation advantages = np.zeros(batch_size + 1) flatten_rews = np.concatenate(rews_list) for i in reversed(range(batch_size)): ## TODO: recursively compute advantage estimates starting from ## timestep T. ## HINT: use terminals to handle edge cases. terminals[i] ## is 1 if the state is the last in its trajectory, and ## 0 otherwise. if terminals[i]: advantages[i] = flatten_rews[i] - values[i] else: delta = flatten_rews[i] + self.gamma * values[i+1] - values[i] advantages[i] = delta + self.gamma * advantages[i+1] else: advantages = np.zeros(batch_size) for i in range(batch_size): advantages[i] = q_values[i] - values[i] else: advantages = q_values.copy() # Normalize the resulting advantages to have a mean of zero # and a standard deviation of one if self.standardize_advantages: ad_mean, ad_std = np.average(advantages), np.std(advantages) advantages = (advantages - ad_mean) / ad_std return advantages