Learning Markov Decision Processes for Model Checking

Abstract

Constructing an accurate system model for formal model verification can be both resource demandingand time-consuming. To alleviate this shortcoming, algorithms have been proposed for automaticallylearning system models based on observed system behaviors. In this paper we extend the algorithmon learning probabilistic automata to reactive systems, where the observed system behavior is inthe form of alternating sequences of inputs and outputs. We propose an algorithm for automaticallylearning a deterministic labeled Markov decision process model from the observed behavior of areactive system. The proposed learning algorithm is adapted from algorithms for learning deterministicprobabilistic finite automata, and extended to include both probabilistic and nondeterministic transitions.The algorithm is empirically analyzed and evaluated by learning system models of slot machines. Theevaluation is performed by analyzing the probabilistic linear temporal logic properties of the systemas well as by analyzing the schedulers, in particular the optimal schedulers, induced by the learnedmodels