Multivariate Mutual Information for Audio Video Fusion

Abstract

Infonnation fusion is an essential part of distributed wireless sensor networks as well as perceptual user interfaces. Irrelevant and redundant data severely affect the perfonnance of the infonnation fusion framework. The work presented in this thesis addresses the problem of acceptability of data from multiple sources. A uni-modal infonnation-theoretic framework for face recognition is first developed. Perfonnance of the proposed method is compared with existing principal component analysis based face recognition algorithms. The uni-modal framework is extended to develop a multi-modal algorithm. The algorithm uses multivariate mutual information to validate the acceptability of data from two sources. Unlike the preceding algorithms, the framework does not require any preprocessing, such as automatic face recognition, etc. Moreover, it also does not condition any statistical modeling or feature extraction and learning algorithms to extract the maximum infonnation regions. The various cases containing a single speaker as well as a group of speakers are analyzed. The applicability of the algorithm is tested by incorporating it with Haar like features of Voila and Jones for lip activity detection.