CS 660: Comparative Decision Making Studies 1, third assignment

Given the description of the MDP M,

• Use value iteration to find the optimal policy with no discount and horizon 5 (in other words, find V^5 for all states).
• Use value iteration to find an optimal infinite-horizon, discounted policy.
• Use linear programming to find an optimal infinite-horizon, discounted policy.
• Describe an MDP model that would decide (on an ongoing basis) whether a hypothetical patient, Jane, should take the prescribed medication. (See below.) For this question, specify:
  • The states of the MDP;
  • the actions;
  • where the probabilities would come from;
  • where the rewards would come from.
  You are not expected to actually come up with numeric probabilities or rewards! You are also not expected to find a policy for Jane.

Imagine the following scenario.

Jane Doe is a 34-year-old woman. She has health insurance through her husband's job, and they have three children, ages 3, 5, and 11. She lives in Eastern Kentucky, and has just driven in to the KY Clinic because her hands and her hips hurt. Some days, she can't walk. Some days, she can't dress herself, or cook, or clean her house, because she can't hold anything.

Jane has rheumatoid arthritis. The doctor tells her that there are medicines available, but there is a 0.01% chance that the medication can make her very sick, or kill her. If it doesn't make her sick or kill her, it can stop the disease from getting worse. With probability 0.7, it will have her functioning as well as she does on her good days for about 6 days out of 7, and probably will keep her from having days like her worst days are now.