Attribute Multiset Grammars for Global Explanations of Activities

Abstract

Recognizing multiple interleaved activities in a video requires implicitly partitioningthe detections for each activity. Furthermore, constraints between activities are impor-tant in finding valid explanations for all detections. We use Attribute Multiset Gram-mars (AMGs) as a formal representation for a domain’s knowledge to encode intra- andinter-activity constraints. We show how AMGs can be used to parse all the observa-tions into ‘feasible’ global explanations. We also present an algorithm for building aBayesian network (BN) given an AMG and a set of detections. The set of labellings ofthe BN corresponds to the set of all possible parse trees. Finding the best explanationthen amounts to finding the maximum a posteriori labeling of the BN. The techniqueis successfully applied to two different problems including the challenging problem ofassociating pedestrians and carried objects entering and departing a building