嘴形影像辨識在輔助溝通上的應用

The Application of Mouth Shape Recognition System on Assistive Communication

本研究完成一套影像處理輔助溝通輔具，讓無法出聲者，能利用嘴形變化的特徵影像辨識，達到與人或電腦溝通的目的，可作為往後唇語辨識的先期研究。本研究利用LabVIEW程式軟體完成影像處理及分析演算法，並結合電腦硬體控制，完成即時嘴形影像辨識在輔助溝通功能上的應用。本系統的硬體部分，只需一台簡易網路攝影機、一台桌上型或筆記型電腦及微控制器硬體控制電路模組，可以提供臉部表情辨識，並利用影像辨識結果，控制電腦發音，應用在影音演講系統及輔助溝通輔具上，協助言語失能者(如語障者、腦性麻癖者甚至重障者)發聲或進行家電控制。經實驗證實，不論在實驗室或一般住家不同情境條件下，嘴形影像辨識正確率平均高達90.6%。其次，為驗證本系統功能性與有效性，我們邀請了10位受測者進行以嘴形變化代替發聲，演唱小蜜蜂童謠的實驗，讓受測者以不同的嘴形對應不同音頻的音符(如Do、Re、Mi…等)，唱出(由電腦喇叭出聲)小蜜蜂童謠的旋律，驗證嘴形影像辨識轉電腦樂音輸出之應用控制，實驗結果正確率高達89.93%。最後，再進行以嘴唇開閉次數，控制家電的實驗，亦獲得90.2%的正確率。證明本系統在不同亮度與複雜背景環境的應用需求下，仍具有高的影像辨識率外，系統本身在硬體控制應用上亦具實用性。

An image assistive communication device that can help the people with speaking disabilities communicate with other people or machines by using mouth shape recognition has been implemented in this research. It will be a pilot study of further lip language recognition. Image processing and analysis algorithms have been de-signed on LabVIEW graphics platform in the research. We also utilize PC (Personal Computer) hardware tech-niques to complete the application of real time mouth shape recognition to assistive communication. The hardware of the real time mouth shape recognition system can offer the functions of computer sound control and support the audio-visual speech and assistive communication for the disabled (with speaking disabil-ity, cerebral palsy or serious disabled) according to the results of facial expression recognition, and includes a webcam, a PC/notebook, and the microcontroller circuit modules. The experimental results of the real time mouth shape tracking system showed an average accuracy rate higher than 90.6% both in the laboratory and at home, meeting the requirements of various environmental conditions. Secondly, in order to verify the performance and the efficiency of the developed system, we invited 10 subjects to attend the experimental tests of a children’s ballad (a small bee). The 10 subjects were asked to sing the ballad by changing their mouth shape mapping to the music notes (e.g. Do, Re, Mi …etc.) in silence to verify that the results of real time mouth shape image recognition can be applied to computer sound output control. The accuracy rate of the recognition experi-ment is about 89.93%. Finally, another experiment for home appliance control was designed and executed. The accuracy is about 90.2%. It is proved that the developed system not only has a high accuracy of image recogni-tion under the conditions of various illumination and complex background but also can be applied to hardware control.