代写Machine Learning中的Reinforcement Learning.
Introduction
In this project, we will help Wheelbot determine an optimal policy that will enable it to gather an optimal amount of reward reward (in expectation) in a stochastic but fully-observable environment.
In Project 2, Wheelbot moved deterministically. Telling Wheelbot to move a certain direction resulted, with probability 1, in Wheelbot moving the correct direction. In this project, Wheelbot will move stochastically around its environment. Additionally, Wheelbot will be given a reward signal upon transitioning into a new state. Obstacles of known location will still be present.
Reinforcement Learning
We can formalize this environment as a Markov Decision Process with states S, actions A, transition probabilities T and rewards R. The reward function,
R(s) will give Wheelbot a reward signal for being in state s ∈ S. There will a be a transition probability for each possible combination of a A.
The states of the environment are the grid cells. The environment is fully observable because Wheelbot always knows which grid cell it is in. Wheelbot’s transition model is as follows: Wheelbot makes the “correct” transition with probability 0 ≤ pc ≤ 1 for any action a. The remaining probability mass is divided equally amongst incorrect transitions to neighboring grid cells. For example, if Wheelbot is told to go Up and pc = 0.85, it will transition Up with probability 0.85. With probability 0.05 it will go Right. With probability 0.05 it will go Left. With probability 0.05 it will go Down. Note that care must be taken when considering the edges of the grid. When Wheelbot tries to transition into a wall, it will stay where it is. Along the edges of the environment and in the corners, self-transitions will have non-zero probability, and the transition function will be different than it is in the rest of the grid. To simplify calculations, we will not allow diagonal transitions in this project. Only the actions A = {Up, Down, Left, Right} are valid.
Project Details
Your task in this project is as follows: given a list of known obstacle locations, a goal location, and pc , solve this MDP using the value iteration algorithm to generate an optimal policy π(s) → a that maps states to optimal actions so as to maximize Wheelbot’s expected utility. Wheelbot’s expected utility will be defined in terms of a reward function R(s) that you define. This reward function should be generalizable to any set of obstacle locations and goal a location. You will also need to define the specifics of given the definition of Wheelbot’s transition model above.
Once you have generated an optimal policy, Wheelbot should follow that policy to (hopefully) make it to the goal. Note that, due to the stochastic nature of the environment, it is possible for Wheelbot to run into an obstacle, even when following an optimal policy. Therefore, it is important to test your program with a number of values of pc , including pc = 1, to ensure that this does not happen when the environment becomes deterministic again. Another possible method of testing is to set pc to several different (increasingly large) values and try your program a large number of times for each such value, keeping track of the number of times Wheelbot runs into an obstacle. The environment should be kept the same through all runs. In general, the more deterministic the environment becomes, the less often Wheelbot should run into an obstacle. The environments that we use to test your program will be solvable in the sense that there will be at least one possible path to the goal state from the start state.
Wheelbot will have the same sensors as in Project 2, with the exception that the local obstacle sensor will not be used in this project (as there are not hidden obstacles).
Implementation Details
This project will require you to modify the files Project3.h and Project3.cpp files in the project source code. You are not to modify any other file that is part of the simulator. You can, however, add new files to the project to implement new classes as you see fit. We will test your project by copying your Project3.h and Project3.cpp files, as well as any files you have added to the project, into our simulation environment and running it.
Feel free to use the C++ STL and STD library data structures and containers. Additionally, if you have previously implemented data structures you wish to reuse, please do. However, you must not use anything that trivializes the problem. For instance, do not use a downloaded value iteration algorithm package. You must implement value iteration yourself.
For full credit your R(s) must be valid (capable of solving the problem), and your program must compute and follow the optimal policy relative to this reward function. Importantly, if pc = 1, your robot should follow a shortest path (there may be multiple such paths) to the goal. pc may be adjusted in main.cpp during your testing. Set the discount factor = 0.9 for all testing.
Submission
Please submit your modified Project3.h and Project3.cpp, as well as any additional files you added to complete this project, to Blackboard.