隨著高齡化失智人口比例快速增加，失智症成為全球健康重要議題之一。早期識別認知障礙的不同階段，並給予老年人有效的干預和及時護理是相當重要的。傳統的診斷方式往往需要經過一系列繁瑣的檢查步驟，也會耗費許多時間和成本。因此，本研究最主要的目標是想建立一套操作簡單、快速、且低成本的失智篩檢方法，基於視覺與聽覺的資訊來偵測老年人的輕度認知障礙及失智症。我們收集了95位受試者(41位為輕度認知障礙；54為失智症患者)，利用簡易心智狀態問卷調查表(SPMSQ)進行失智評估並錄製影片。我們分析影片中受試者的臉部變化及聲音訊號，萃取視覺與聽覺的資訊並進行預測，最後將兩者分析結果結合，得出受試者是屬於輕度認知障礙還是失智症及MMSE、CASI分數。分析結果顯示，經過適當整合視覺與聽覺資訊，判別輕度認知障礙與失智症的整體準確度可以達到76%。在去除焦慮及憂鬱個案後，整體準確度可以達到88%，MMSE評估數值的關聯性可以達到中度相關。從無其他併發精神症狀的失智案例分析中，透過深度學習技術將單獨視覺(81%)或聽覺(87%)的結果結合，能得到最佳失智症疾病嚴重度判別效果 (準確性達到88%)。此外，我們也觀察到MMSE評估數值的關聯性在女性的結果具有統計顯著性，而男性沒有。

In the aging society, there will be more and more people with dementia worldwide by the year 2030. Early identifying different stages of cognitive impairment is important to provide available intervention and timely care in the elderly. However, it is time-consuming and requires professions such as certified psychologists, which may not be available in some settings. Therefore, developing a time-saving, accessible, non-invasive, and inexpensive method for screening dementia is necessary. This study aims to distinguish participants with mild cognitive impairment (MCI) and those with mild to moderate dementia based on automated video analysis. 95 participants were recruited (MCI, 41; mild to moderate dementia, 54) for the study. The videos were captured from individuals who take the short portable mental status questionnaire (SPMSQ) process. Based on the video recordings, we extract visual and aural signals. Deep learning models are then developed to discriminate MCI from mild to moderate dementia. Correlation analysis between predicted mini-mental state examination (MMSE) and cognitive abilities screening instrument (CASI) scores and ground truth is performed. The experimental results show that the deep learning model combing both visual and aural signals discriminates MCI from mild to moderate dementia with accuracy of 76.0%. The accuracy increases to 88.0% for those excluding depression and anxiety. Significant moderate correlations are found between predicted values and the ground truth and the correlation for those excluding depression and anxiety is strong. Furthermore, female, but not male population exhibits strong correlation.

[1] Ahmed Ashraf and Babak Taati. Automated video analysis of handwashing behavior as a potential marker of cognitive health in older adults. IEEE Journal of Biomedical and Health Informatics, 20(2):682–690, 2015.
[2] American Psychological Association. Diagnostic and statistical manual of mental disorders. American Psychological Association, Washington, DC, 5th edition, 2013.
[3] Aparna Balagopalan, Benjamin Eyre, Frank Rudzicz, and Jekaterina Novikova. To BERT or not to BERT: Comparing speech and language-based approaches for Alzheimer’s disease detection. In Proc. Interspeech 2020, pages 2167–2171, 2020.
[4] Chun-Hung Chang, Chieh-Hsin Lin, and Hsien-Yuan Lane. Machine learning and novel biomarkers for the diagnosis of Alzheimer＇s disease. International Journal of Molecular Sciences, 22(5):2761, 2021.
[5] Jungmi Choi, Boncho Ku, Young Gooun You, Miok Jo, Minji Kwon, Youyoung Choi, Segyeong Jung, Soyoung Ryu, Eunjeong Park, Hoyeon Go, et al. Resting-state prefrontal EEG biomarkers in correlation with MMSE scores in elderly individuals. Scientific Reports, 9(1):1–15, 2019.
[6] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
[7] Jeffrey L Cummings, Michael Mega, Katherine Gray, Susan RosenbergThompson, Daniela Anne Carusi, and Jeffrey Gornbein. The Neuropsychiatric Inventory: comprehensive assessment of psychopathology in dementia. Neurology, 44(12):2308–2314, 1994.
[8] Florian Eyben, Klaus R Scherer, Björn W Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y Devillers, Julien Epps, Petri Laukka, Shrikanth S Narayanan, et al. The geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 7(2):190–202, 2015.
[9] Florian Eyben, Martin Wöllmer, and Björn Schuller. Opensmile: the munich versatile and fast opensource audio feature extractor. In Proceedings of The 18th ACM International Conference on Multimedia, pages 1459–1462, 2010.
[10] Elif Eyigoz, Sachin Mathur, Mar Santamaria, Guillermo Cecchi, and Melissa Naylor. Linguistic markers predict onset of Alzheimer’s disease. EClinicalMedicine, 28:100583, 2020.
[11] Marshal F Folstein, Susan E Folstein, and Paul R McHugh. ”Mini-mental state”. A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12(3):189–198, 1975.
[12] Dragan Gamberger, Nada Lavrač, Shantanu Srivatsa, Rudolph E Tanzi, and P Murali Doraiswamy. Identification of clusters of rapid and slow decliners among subjects at risk for Alzheimer’s disease. Scientific Reports, 7(1):1–12, 2017.
[13] Gregorio González-Alcaide, Mercedes Fernández-Ríos, Rosa Redolat, and Emilia Serra. Research on emotion recognition and dementias: Foundations and prospects. Journal of Alzheimer’s Disease, 82(3):939–950, 2021.
[14] Fasih Haider, Sofia De La Fuente, and Saturnino Luz. An assessment of paralinguistic acoustic features for detection of Alzheimer’s dementia in spontaneous speech. IEEE Journal of Selected Topics in Signal Processing, 14(2):272–281, 2019.
[15] R’mani Haulcy and James Glass. Classifying Alzheimer’s disease using audio and text-based representations of speech. Frontiers in Psychology, 11:624137, 2021.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
[17] M Trinidad Hoyl, Cathy A Alessi, Judith O Harker, Karen R Josephson, Fern M Pietruszka, Maryanne Koelfgen, J Randy Mervis, L Jaime Fitten, and Laurence Z Rubenstein. Development and testing of a five-item version of the geriatric depression scale. Journal of the American Geriatrics Society, 47(7):873–878, 1999.
[18] Tomomichi Iizuka, Makoto Fukasawa, and Masashi Kameyama. Deep-learning-based imaging-classification identified cingulate island sign in dementia with lewy bodies. Scientific Reports, 9(1):1–9, 2019.
[19] Loukas Ilias and Dimitris Askounis. Multimodal deep learning models for detecting dementia from speech and transcripts. Frontiers in Aging Neuroscience, 14, 2022.
[20] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015.
[21] Ker-Neng Lin, Pei-Ning Wang, Chia-Yih Liu, Wei-Ta Chen, Yi-Chung Lee, and Hsiu-Chih Liu. Cutoff scores of the cognitive abilities screening instrument, chinese version in screening of dementia. Dementia and Geriatric Cognitive Disorders, 14(4):176–182, 2002.
[22] Saturnino Luz, Fasih Haider, Sofia de la Fuente, Davida Fromm, and Brian MacWhinney. Alzheimer’s dementia recognition through spontaneous speech: the ADReSS challenge. arXiv preprint arXiv:2004.06833, 2020.
[23] Alex J Mitchell. The clinical significance of subjective memory complaints in the diagnosis of mild cognitive impairment and dementia: a meta-analysis. International Journal of Geriatric Psychiatry: A Journal of The Psychiatry of Late Life and Allied Sciences, 23(11):1191–1202, 2008.
[24] John C Morris. The Clinical Dementia Rating (CDR): current version and scoring rules. Neurology, 43:2412–2414, 1993.
[25] Giuseppe Murdaca, Sara Banchero, Alessandro Tonacci, Alessio Nencioni, Fiammetta Monacelli, and Sebastiano Gangemi. Vitamin D and Folate as predictors of MMSE in Alzheimer’s disease: A machine learning analysis. Diagnostics, 11(6):940, 2021.
[26] Fatemeh Noroozi, Marina Marjanovic, Angelina Njegus, Sergio Escalera, and Gholamreza Anbarjafari. Audio-visual emotion recognition in video clips. IEEE Transactions on Affective Computing, 10(1):60–75, 2017.
[27] Enrico Pellegrini, Lucia Ballerini, Maria del C Valdes Hernandez, Francesca M Chappell, Victor González-Castro, Devasuda Anblagan, Samuel Danso, Susana Muñoz-Maniega, Dominic Job, Cyril Pernet, et al. Machine learning of neuroimaging for assisted diagnosis of cognitive impairment and dementia: a systematic review. Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring, 10:519–535, 2018.
[28] Min Peng, Chongyang Wang, Yuan Gao, Tao Bi, Tong Chen, Yu Shi, and Xiang-Dong Zhou. Recognizing micro-expression in video clip with adaptive key-frame mining. arXiv preprint arXiv:2009.09179, 2020.
[29] Ronald C Petersen. Clinical practice. Mild cognitive impairment. The New England Journal of Medicine, 364(23):2227–2234, 2011.
[30] Eric Pfeiffer. A short portable mental status questionnaire for the assessment of organic brain deficit in elderly patients. Journal of the American Geriatrics Society, 23(10):433–441, 1975.
[31] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015.
[32] Kaoru Sakatani, Katsunori Oyama, and Lizhen Hu. Deep learning-based screening test for cognitive impairment using basic blood test data for health examination. Frontiers in Neurology, 11:588140, 2020.
[33] Utkarsh Sarawgi, Wazeer Zulfikar, Nouran Soliman, and Pattie Maes. Multimodal inductive transfer learning for detection of Alzheimer’s dementia and its severity. arXiv preprint arXiv:2009.00700, 2020.
[34] Li Sun, Jieyuan Zheng, Jiyun Li, and Chen Qian. Exploring MMSE score prediction model based on spontaneous speech.
[35] László Tóth, Ildikó Hoffmann, Gábor Gosztolya, Veronika Vincze, Gréta Szatlóczki, Zoltán Bánréti, Magdolna Pákáski, and János Kálmán. A speech recognition-based solution for the automatic detection of mild cognitive impairment from spontaneous speech. Current Alzheimer Research, 15(2):130–138, 2018.
[36] Yumi Umeda-Kameyama, Masashi Kameyama, Tomoki Tanaka, Bo-Kyung Son, Taro Kojima, Makoto Fukasawa, Tomomichi Iizuka, Sumito Ogawa, Katsuya Iijima, and Masahiro Akishita. Screening of Alzheimer’s disease by facial complexion using artificial intelligence. Aging, 13(2):1765, 2021.
[37] Akihiko Wada, Kohei Tsuruta, Ryusuke Irie, Koji Kamagata, Tomoko Maekawa, Shohei Fujita, Saori Koshino, Kanako Kumamaru, Michimasa Suzuki, Atsushi Nakanishi, et al. Differentiating Alzheimer’s disease from dementia with Lewy bodies using a deep learning technique based on structural brain connectivity. Magnetic Resonance in Medical Sciences, 18(3):219, 2019.
[38] Chonghua Xue, Cody Karjadi, Ioannis Ch Paschalidis, Rhoda Au, and Vijaya B Kolachalama. Detection of dementia on voice recordings using deep learning: a Framingham Heart Study. Alzheimer’s Research & Therapy, 13(1):1–15, 2021