Statistical reinforcement learning : modern machine learning approaches

摘要

Introduction to Reinforcement Learning Reinforcement Learning Mathematical Formulation Structure of the Book Model-Free Policy Iteration Model-Free Policy Search Model-Based Reinforcement Learning MODEL-FREE POLICY ITERATION Policy Iteration with Value Function Approximation Value Functions State Value Functions State-Action Value Functions Least-Squares Policy Iteration Immediate-Reward Regression Algorithm Regularization Model Selection Remarks Basis Design for Value Function Approximation Gaussian Kernels on Graphs MDP-Induced Graph Ordinary Gaussian Kernels Geodesic Gaussian Kernels Extension to Continuous State Spaces Illustration Setup Geodesic Gaussian Kernels Ordinary Gaussian Kernels Graph-Laplacian Eigenbases Diffusion Wavelets Numerical Examples Robot-Arm Control Robot-Agent Navigation Remarks Sample Reuse in Policy Iteration Formulation Off-Policy Value Function Approximation Episodic Importance Weighting Per-Decision Importance Weighting Adaptive Per-Decision Importance Weighting Illustration Automatic Selection of Flattening Parameter Importance-Weighted Cross-Validation Illustration Sample-Reuse Policy Iteration Algorithm Illustration Numerical Examples Inverted Pendulum Mountain Car Remarks Active Learning in Policy Iteration Efficient Exploration with Active Learning Problem Setup Decomposition of Generalization Error Estimation of Generalization Error Designing Sampling Policies Illustration Active Policy Iteration Sample-Reuse Policy Iteration with Active Learning Illustration Numerical Examples Remarks Robust Policy Iteration Robustness and Reliability in Policy Iteration Robustness Reliability Least Absolute Policy Iteration Algorithm Illustration Properties Numerical Examples Possible Extensions Huber Loss Pinball Loss Deadzone-Linear Loss Chebyshev Approximation Conditional Value-At-Risk Remarks MODEL-FREE POLICY SEARCH Direct Policy Search by Gradient Ascent Formulation Gradient Approach Gradient Ascent Baseline Subtraction for Variance Reduction Variance Analysis of Gradient Estimators Natural Gradient Approach Natural Gradient Ascent Illustration Application in Computer Graphics: Artist Agent Sumie Paining Design of States, Actions, and Immediate Rewards Experimental Results Remarks Direct Policy Search by Expectation-Maximization Expectation-Maximization Approach Sample Reuse Episodic Importance Weighting Per-Decision Importance Weight Adaptive Per-Decision Importance Weighting Automatic Selection of Flattening Parameter Reward-Weighted Regression with Sample Reuse Numerical Examples Remarks Policy-Prior Search Formulation Policy Gradients with Parameter-Based Exploration Policy-Prior Gradient Ascent Baseline Subtraction for Variance Reduction Variance Analysis of Gradient Estimators Numerical Examples Sample Reuse in Policy-Prior Search Importance Weighting Variance Reduction by Baseline Subtraction Numerical Examples Remarks MODEL-BASED REINFORCEMENT LEARNING Transition Model Estimation Conditional Density Estimation Regression-Based Approach Q-Neighbor Kernel Density Estimation Least-Squares Conditional Density Estimation Model-Based Reinforcement Learning Numerical Examples Continuous Chain Walk Humanoid Robot Control Remarks Dimensionality Reduction for Transition Model Estimation Sufficient Dimensionality Reduction Squared-Loss Conditional Entropy Conditional Independence Dimensionality Reduction with SCE Relation to Squared-Loss Mutual Information Numerical Examples Artificial and Benchmark Datasets Humanoid Robot Remarks References Index