Work Description

Title: Dataset of test instances of Markov decision processes with multiple models Open Access Deposited
Attribute Value
  • We used Python and C++ scripts to generate problem instances of Markov decision process with multiple models of parameters.
  • This repository includes test instances of infinite-horizon Markov decision processes with multiple models of parameters (i.e., "Multi-model Markov decision processes"). We generated each test instance in the dataset using a Python script. The test instances can be read in using the provided C++ and Python script. See the README for details.
Contact information
Funding agency
  • National Science Foundation (NSF)
Citations to related material
  • Ahluwalia, Steimle, and Denton. "Policy-based branch-and-bound for infinite-horizon Multi-model Markov decision processes". 2020.
Resource type
Last modified
  • 01/24/2020
  • 01/24/2020
To Cite this Work:
Ahluwalia, V., Steimle, L., Denton, B. (2020). Dataset of test instances of Markov decision processes with multiple models [Data set]. University of Michigan - Deep Blue.


Files (Count: 10; Size: 1.35 MB)

Infinite-Horizon MMDPs

This code repository contains the files we used to create the test instances for infinte-horizon MMDPs. These problems were based off of case studies previously published in the literature. We use a medical decision-making problem for the initiation of HIV therapy, a machine maintenance problem, and a set of randomly constructed instances. We also include C++ code and Python code that can read in an MMDP data file such that others can experiment with new methods for solving them. All MMDP files that will be generated with the included scripts are all in the same format and style.

Using the Code

If using a Python file, you can install any necessary dependency, "library_name" via the following the following command in the terminal: "pip install library_name".

1) Random Instances (RI) from Buchholz & Scheftelowitsch (2019)
Creates random MMDP instances of various problem sizes. A set of thirty example 8-model, 8-state, 8-action MMDP files are included in the zip file "Buchholz". When the script is run, it will generate random MMDP instances using the structure described in Buchholz & Scheftelowitsch (2019) that will have the problem sizes designated by the vector "problem_sizes". Each entry in the "problem_sizes" vector will yield 30 distinct MMDP files that have the corresponding number of states, actions, and models. All files will be written to a folder called "Buchholz_Files" which must be created by the user in the working directory. Currently, the 17 different problem sizes specified in Buchholz & Scheftelowitsch (2019) are part of the "problem_sizes" vector. This will yield 30 * 17 = 510 files.
Dependency: numpy
run 'python3'

2) Machine Maintenance Problems from Delage & Mannor (2010)
Creates machine maintenance instances of various model sizes and concentration parameters based off the above paper. There are currently three concentration parameters (1, 5, 9) and three different model sizes (10, 30, 100). This script will generate files to a folder calledl, "Machine_Maintenance" which must be created by the user in the working directory. 30 files will be written to that folder for each concentration parameter and model size. Currently, this will lead to 3 * 3 * 30 = 270 files.
Dependency: numpy.
run 'python3'

3) HIV Therapy Example from Shechter et al. (2008)
Creates an HIV initiation MMDP file with name "HIV_test.txt". An example MMDP file is incldued as "HIV_test_72_models.txt".
Dependency: numpy.
run 'python3'

4) Reading in MMDP Files Using C++
The necessary files are 'main.cpp', 'infinte_horizon_mmdp.h', and 'infinite_horizon_mmdp.cpp'. No output expected.
run 1) 'g++ -Wall -Werror main.cpp infinite_horizon_mmdp.cpp -o out' and then 2) './out' in the terminal

5) Reading in MMDP Files Using Python
Analogous to Section 4 above. No output expected. Dependencies: numpy, os.
Run 'python3'


  • Vinayak S. Ahluwalia

Lauren N. Steimle

Brian T. Denton


This work was supported by the National Science Foundation under grant numbers DGE-1256260 (Steimle) and CMMI-1462060 (Denton); any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.


If using the code in this package, please cite the following article where we discuss these methods:

Ahluwalia, Steimle, and Denton, "Policy-based branch-and-bound for infinite-horizon Multi-model Markov decision processes". 2020.

Download All Files (To download individual files, select them in the “Files” panel above)

Best for data sets < 3 GB. Downloads all files plus metadata into a zip file.

Best for data sets > 3 GB. Globus is the platform Deep Blue Data uses to make large data sets available.   More about Globus