Multimodal information fusion for selected multimedia applications

摘要

The effective interpretation and integration of multiple information content are important for the efficacious utilisation of multimedia in a wide variety of application context. The major challenge in multimodal information fusion lies in the difficulty of identifying the complementary and discriminatory representations from individual channels, and the efficient fusion of the resulting information for the targeted application problem. This paper outlines several multimedia systems that utilise a multimodal approach, and provides a comprehensive review of the state-of-the-art in related areas, including emotion recognition, image annotation and retrieval, and biometrics. Data collected from diverse sources or sensors are employed to improve the recognition or classification accuracy. It is shown that the combination of multimodality information is capable of providing a more complete and effective description of the intrinsic characteristics of the specific pattern, and producing improved system performance than single modality only. In addition, we present a facial fiducial point detection and a gesture recognition system, which can be incorporated into a multimodal framework. The issues and challenges in the research and development of multimodal systems are discussed, and a cutting-edge application of multimodal information fusion for intelligent robotic system is presented.