Automatic speechreading with applications to human-computer interfaces

DOI10.1155/S1110865702206137MaRDI QIDQ1424530zbMATH OpenFDO

Authors Charles C. Broun, Russell M. Mersereau, Mark A. Clements, Xiao-Zheng Zhang

Publication date 16 March 2004

Published in EURASIP Journal on Applied Signal Processing (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1155/S1110865702206137

zbMATH Keywords

hidden Markov modelsMarkov random fieldsspeech recognitionvisual feature extraction

Mathematics Subject Classification ID

Pattern recognition, speech recognition (68T10)

Review text

Summary: There has been growing interest in introducing speech as a new modality into the Human-Computer Interface (HCI). Motivated by the multimodal nature of speech, the visual component is considered to yield information that is not always present in the acoustic signal and enables improved system performance over acoustic-only methods, especially in noisy environments. In this paper, we investigate the usefulness of visual speech information in HCI related applications. We first introduce a new algorithm for automatically locating the mouth region by using color and motion information and segmenting the lip region by making use of both color and edge information based on Markov random fields. We then derive a relevant set of visual speech parameters and incorporate them into a recognition engine. We present various visual feature performance comparisons to explore their impact on the recognition accuracy, including the lip inner contour and the visibility of the tongue and teeth. By using a common visual feature set, we demonstrate two applications that exploit speechreading in a joint audio-visual speech signal processing task: speech recognition and speaker verification. The experimental results based on two databases demonstrate that the visual information is highly effective for improving recognition performance over a variety of acoustic noise levels.

Recommendations

• scientific article; zbMATH DE number 2089646
• scientific article; zbMATH DE number 2089645
• scientific article; zbMATH DE number 1927208
• scientific article; zbMATH DE number 1717105
• Audio-visual speech recognition using MPEG-4 compliant visual features

Cited in

(8)

• A set of neural tools for human-computer interactions: application to the handwritten character recognition, and visual speech recognition problems
• Lip contour extraction from color images using a deformable model
• Automatic lip localization under face illumination with shadow consideration
• Auditory-visual speech recognition error detection using longest common subsequence matching of vowel sequences
• scientific article; zbMATH DE number 2089645 (Why is no real title available?)
• scientific article; zbMATH DE number 2089646 (Why is no real title available?)
• scientific article; zbMATH DE number 1841957 (Why is no real title available?)
• A cascade visual front end for speaker independent automatic speechreading

This page was built for publication: Automatic speechreading with applications to human-computer interfaces

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1424530)