簡易檢索 / 詳目顯示

回結果列表
研究生： 曾華偉
Tseng, Hua-Wei
論文名稱： 基於數位表徵中時序性與型態相關性之多任務自監督學習於躁鬱症狀態預測
Multitask Self-Supervised Learning Based on Temporal and Type Correlation in Digital Phenotyping for Bipolar Disorder State Prediction
指導教授： 吳宗憲
Wu, Chung-Hsien
學位類別： 碩士
Master
系所名稱： 電機資訊學院 - 資訊工程學系
Department of Computer Science and Information Engineering
論文出版年： 2022
畢業學年度： 110
語文別： 英文
論文頁數： 83
中文關鍵詞： 躁鬱症 、數位表徵 、自監督學習 、多任務學習 、深度神經網路 、遞迴神經網路 、狀態預測
外文關鍵詞： Bipolar disorder, Digital phenotyping, Self-supervised learning, Multitask training, Deep neural network, Gated recurrent unit
相關次數： 點閱：51  下載：0 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報

    • 躁鬱症為常見的心理疾病之一且有非常高的復發機率，若無及時治療可能會造成更嚴重的後果，因此為了使躁鬱症患者在病情加重之前能及時接受治療，本論文建立一個預警系統，透過智慧型手機APP收集躁鬱症患者的數位表徵中分別包括各種不同型態之資料(如位置資訊(GPS)、自評量表、每日心情、睡眠時間及多媒體記錄(文字、語音及影像))，利用所收集的資料透過模型來預測躁鬱症狀態，以避免延遲治療。預測躁鬱症狀態定義上面，本論文根據醫生的建議，分別使用看診紀錄、用藥資料以及住院急診紀錄來定義躁鬱症狀態，可分為五個等級，相比於與傳統的復發預測之二元分類更為精準。
    而因為躁鬱症資料收集不易，訓練資料數量通常較少導致深度學習之模型抽取出的特徵不夠強健，本論文利用自監督學習方式更有效的利用資料，從中萃取出更多的資訊。而本論文貢獻在於考量了資料型態之間的關聯性與時序上的關聯性並利用自監督學習的方式將其嵌入於機器中，提升整體表現，我們將此學習方法稱為多任務自監督學習，分別又可分為兩種學習方式型態相關性多任務自監督學習以及時序相關性多任務自監督學習，型態相關性多任務自監督學習是透過遮罩的方式將其中一種型態資料屏蔽，利用其他資料來預測被屏蔽之資料所產生的分類類別，從而學習到資料間的關聯性，而時序相關性多任務自監督學習是利用過去時間段之資料來預測未來之資料所產生的分類類別，來學習資料在時間上的關聯性，在訓練的過程中，本論文同時訓練兩種自監督學習並且互相傳遞模型參數，而後將訓練好的模型參數傳遞給躁鬱症狀態預測模型進行最終的訓練任務。
    在實驗方面，使用了多任務自監督學習後可讓躁鬱症狀態預測準確度從85.1%提升到88.2%，另外在消融實驗中也發現，分別利用兩種多任務自監督學習也都能使總體效能有所提升，在類別相關性多任務自監督學習上提升1.2個百分點，而在時序相關性多任務自監督學習上提升2.1個百分點，綜上所述，類別相關性與時序性在數位表徵資料與躁鬱症狀態預測中確實具有相當的重要性，並都可在躁鬱症狀態預測任務中提升其總體效能。

    Bipolar disorder is one of the most common mental illnesses and has a very high recurrence rate. The patients could be dangerous if they are not treated in time. Therefore, to enable patients with bipolar disorder to receive treatment in time before the situation worsens, this thesis establishes an early warning system. The digital phenotyping of bipolar patients is collected through our smartphone App which includes various types of data (such as location information (GPS), self-rating scales, daily mood, sleep time, and multimedia records (text, voice, and video).). The collected data are used to predict the bipolar disorder state to avoid delay in treatment. According to the doctor's suggestion, this thesis uses medical records, medication data, and hospital emergency records to define the state of bipolar disorder, which can be divided into five levels. Compared to a binary classification of relapse prediction our task is more precise.
    Because it is hard to collect bipolar disorder-related data, the amount of training data is usually small. The features extracted by the deep learning model are not robust enough. Therefore, this thesis utilizes the data more effectively to extract more information through self-supervised learning. The contribution of this thesis is that we consider the correlation between data types and the temporal correlation in time-sequential data and use self-supervised learning to embed them into the machine to improve the overall performance. We call this learning method multitask self-supervised learning. This method can be divided into two types of learning methods: type correlation-based multitask self-supervised learning and temporal-correlation-based multitask self-supervised learning. In correlation-based multitask self-supervised learning, we mask one type of the data and use other types of data to predict the classification categories generated by the masked data. Therefore, it can learn the correlation between the data. Time correlation-based multitask self-supervised learning is created by using the data collected in the past period to predict future data. During the training, we train two types of self-supervised learning methods simultaneously and pass model parameters to each other. Finally, we pass the pre-trained model parameters to the bipolar disorder state prediction model for the final training task.
    In terms of experiments, the use of multitask self-supervised learning can improve the state prediction accuracy of bipolar disorder from 85.1% to 88.2%. In addition, the ablation experiment also found that using two kinds of multitask self-supervised learning can improve the overall performance. The correlation-based multitask self-supervised learning increased by 1.2%, and the temporal correlation-based multitask self-supervised learning increased by 2.1%. Temporal and type correlation are all important factors in bipolar disorder state prediction. Both methods can improve overall performance in the bipolar disorder state prediction task.

    摘要 I Abstract III Table of Contents V List of Tables IX List of Figures XI Chapter 1 Introduction 1 1.1 Background 1 1.2 Motivation 5 1.3 Literature review 7 1.3.1 Bipolar research 7 1.3.2 Features associated with bipolar disorder 8 1.3.3 Methods of predicting bipolar disorder 10 1.3.4 Methods of Deep Learning 10 1.3.5 Self-supervised learning 13 1.4 Problems 14 1.5 Brief description of research methods 15 Chapter 2 Database collection and processing 17 2.1 Participate selection criteria 17 2.2 Collection Process 17 2.3 Introduction and processing of different types of data 19 2.3.1 Location information 20 2.3.2 Self-rating scale 21 2.3.3 Daily valence 22 2.3.4 Sleep times 23 2.3.5 Daily 7 emotions 24 2.3.6 Media record 25 2.3.7 Data labeling of BD state 26 2.4 Data Dimensions and Statistics 28 Chapter 3 Research methods 30 3.1 Emotion classification model 31 3.1.1 Emotion classification model-text 31 3.1.2 Emotion classification model-audio 34 3.1.3 Emotion classification model-video 35 3.2 Imputation methods 37 3.2.1 K-Nearest Neighbor imputation 37 3.2.2 Padding imputation 39 3.3 Data balancing 40 3.3.1 Synthesized Minority Oversampling Technique 41 3.3.2 Random undersampling 42 3.4 Model architecture 43 3.5 Multitask self-supervised learning 45 3.5.1 Multitask self-supervised learning task generation 47 3.5.2 Type correlation-based multitask self-supervised learning 49 3.5.3 Temporal correlation-based multitask self-supervised learning 52 3.6 BD state prediction model 56 Chapter 4 Experiment setup and results 59 4.1 Media emotion classification model 59 4.1.1 Results of text emotion classification model 59 4.1.2 Results of audio emotion classification model 59 4.1.3 Results of image emotion classification model 60 4.2 Experiment setup of BD state prediction task 60 4.3 Evaluation 62 4.4 Results of different input days 63 4.5 Influence of each different type of data 65 4.6 Results of different balance number 67 4.7 Results of GRU 68 4.8 Results of different methods 70 4.9 Results of different SSL methods 71 4.10 Ablation study 73 Chapter 5 Conclusion 75 Reference 78

    1. Howland, M. What Are Bipolar Disorders? 2021; Available from: https://psychiatry.org/patients-families/bipolar-disorders/what-are-bipolar-disorders.
    2. 張勳安. 認識躁鬱症. 2022; Available from: https://wwwv.tsgh.ndmctsgh.edu.tw/unit/10058/17020.
    3. Health, H. chages of bipolar disorder. 2021; Available from: https://www.drugs.com/health-guide/bipolar-disorder-manic-depressive-illness-or-manic-depression.html.
    4. Ferrari, A.J., et al., The prevalence and burden of bipolar disorder: findings from the Global Burden of Disease Study 2013. Bipolar Disord, 2016. 18(5): p. 440-50.
    5. Goodwin, F.K. and K.R. Jamison, Manic-depressive illness: bipolar disorders and recurrent depression. Vol. 2. 2007: Oxford university press.
    6. 精神疾病介紹-2躁鬱症(Bipolar Disorder). 2018; Available from: http://www.vhcy.gov.tw/UnitPage/RowViewDetail?WebRowsID=cfee6253-433f-451e-b5c6-df61f79956a9&UnitID=9560bc3c-9b11-43a8-9cd7-89c676199787&CompanyID=e8e0488e-54a0-44bf-b10c-d029c423f6e7&UnitDefaultTemplate=1.
    7. Causes - Bipolar disorder. 2019; Available from: https://www.nhs.uk/mental-health/conditions/bipolar-disorder/causes/.
    8. population of BD. Available from: https://ourworldindata.org/mental-health.
    9. Dass21 questionnaire. Available from: https://maic.qld.gov.au/wp-content/uploads/2016/07/DASS-21.pdf.
    10. APA_DSM5_Level-2-Mania-Adult.pdf. Available from: https://www.psychiatry.org/File%20Library/Psychiatrists/Practice/DSM/APA_DSM5_Level-2-Mania-Adult.pdf.
    11. Borges-Júnior, R., et al. Forecasting depressive relapse in Bipolar Disorder from clinical data. in 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). 2018. IEEE.
    12. de Siqueira Rotenberg, L., et al., Exploring machine learning to predict depressive relapses of bipolar disorder patients. Journal of Affective Disorders, 2021. 295: p. 681-687.
    13. Vázquez, G.H., et al., Recurrence rates in bipolar disorder: Systematic comparison of long-term prospective, naturalistic studies versus randomized controlled trials. European Neuropsychopharmacology, 2015. 25(10): p. 1501-1512.
    14. 財務委員會審核二零一七至一八年度開支預算管制人員的答覆. 2017. 2017.
    15. Tseng, Y.-C., et al., Associations among smartphone app-based measurements of mood, sleep and activity in bipolar disorder. Psychiatry Research, 2022. 310: p. 114425.
    16. Perez Arribas, I., et al., A signature-based machine learning model for distinguishing bipolar disorder and borderline personality disorder. Translational Psychiatry, 2018. 8(1): p. 274.
    17. 王韋凱, 多重相關性資料補值及多重感測器 Lasso 回歸於躁鬱症疾患評估. 2021.
    18. Grünerbl, A., et al., Smartphone-based recognition of states and state changes in bipolar disorder patients. IEEE journal of biomedical and health informatics, 2014. 19(1): p. 140-148.
    19. Su, H.-Y., et al. Assessment of Bipolar Disorder Using Heterogeneous Data of Smartphone-Based Digital Phenotyping. in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2021. IEEE.
    20. Hong, Q.-B., et al., Exploring Macroscopic and Microscopic Fluctuations of Elicited Facial Expressions for Mood Disorder Classification. IEEE Transactions on Affective Computing, 2019. 12(4): p. 989-1001.
    21. Huang, K.-Y., et al., Detecting unipolar and bipolar depressive disorders from elicited speech responses using latent affective structure model. IEEE Transactions on Affective Computing, 2018. 11(3): p. 393-404.
    22. Su, M.-H., et al., Cell-coupled long short-term memory with $ L $-skip fusion mechanism for mood disorder detection through elicited audiovisual features. IEEE Transactions on Neural Networks and Learning Systems, 2019. 31(1): p. 124-135.
    23. Tomasik, J., et al., A machine learning algorithm to differentiate bipolar disorder from major depressive disorder using an online mental health questionnaire and blood biomarker data. Translational psychiatry, 2021. 11(1): p. 1-12.
    24. Lin, E.C.-l., et al., Development and feasibility testing of a continuous moodsensor smartphone application to construct an early-relapse recognition model for individuals with bipolar disorder. Bipolar Disorders, 2020. 22(S1): p. 118.
    25. Jan, Z., et al., The Role of Machine Learning in Diagnosing Bipolar Disorder: Scoping Review. J Med Internet Res, 2021. 23(11): p. e29749.
    26. Vasu, V. and M. Indiramma, A Survey on Bipolar Disorder Classification Methodologies using Machine Learning. 2020 International Conference on Smart Electronics and Communication (ICOSEC), 2020: p. 335-340.
    27. Valenza, G., et al., Mood states modulate complexity in heartbeat dynamics: A multiscale entropy analysis. EPL (Europhysics Letters), 2014. 107(1): p. 18003.
    28. Orsolini, L., M. Fiorani, and U. Volpe, Digital phenotyping in bipolar disorder: Which integration with clinical endophenotypes and biomarkers? International Journal of Molecular Sciences, 2020. 21(20): p. 7684.
    29. Jadhav, R., et al., Mental Disorder Detection : Bipolar Disorder Scrutinization Using Machine Learning. 2019. 304-308.
    30. Liu, Y.S., et al., Screening for bipolar disorder in a tertiary mental health centre using EarlyDetect: A machine learning-based pilot study. Journal of Affective Disorders Reports, 2021. 6: p. 100215.
    31. Pan, Z., et al., Detecting Manic State of Bipolar Disorder Based on Support Vector Machine and Gaussian Mixture Model Using Spontaneous Speech. Psychiatry Investig, 2018. 15(7): p. 695-700.
    32. Sørensen, T., A. Giraldi, and M. Vinberg, Sexual distress and quality of life among women with bipolar disorder. International Journal of Bipolar Disorders, 2017. 5(1): p. 29.
    33. Schmarje, L., et al., A survey on semi-, self-and unsupervised learning for image classification. IEEE Access, 2021. 9: p. 82146-82168.
    34. Singh, A., N. Thakur, and A. Sharma. A review of supervised machine learning algorithms. in 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom). 2016.
    35. Milligan, G.W. and M.C. Cooper, Methodology review: Clustering methods. Applied psychological measurement, 1987. 11(4): p. 329-354.
    36. Argote, L., et al., Knowledge transfer in organizations: Learning from the experience of others. Organizational behavior and human decision processes, 2000. 82(1): p. 1-8.
    37. Gidaris, S., P. Singh, and N. Komodakis, Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728, 2018.
    38. Zhang, R., P. Isola, and A.A. Efros. Colorful image colorization. in European conference on computer vision. 2016. Springer.
    39. Saeed, A., T. Ozcelebi, and J. Lukkien, Multi-task self-supervised learning for human activity detection. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2019. 3(2): p. 1-30.
    40. Zezario, R.E., et al. Self-Supervised Denoising Autoencoder with Linear Regression Decoder for Speech Enhancement. in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2020.
    41. Bank, D., N. Koenigstein, and R. Giryes, Autoencoders. arXiv preprint arXiv:2003.05991, 2020.
    42. Ragab, M., et al., Self-Supervised Autoregressive Domain Adaptation for Time Series Data. IEEE Transactions on Neural Networks and Learning Systems, 2022.
    43. Hadsell, R., S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). 2006. IEEE.
    44. Chen, T., et al. A simple framework for contrastive learning of visual representations. in International conference on machine learning. 2020. PMLR.
    45. Karam, Z.N., et al., ECOLOGICALLY VALID LONG-TERM MOOD MONITORING OF INDIVIDUALS WITH BIPOLAR DISORDER USING SPEECH. Proc IEEE Int Conf Acoust Speech Signal Process, 2014. 2014: p. 4858-4862.
    46. Gold, A.K. and L.G. Sylvia, The role of sleep in bipolar disorder. Nature and science of sleep, 2016. 8: p. 207.
    47. Fraccaro, P., et al., Digital biomarkers from geolocation data in bipolar disorder and schizophrenia: a systematic review. Journal of the American Medical Informatics Association, 2019. 26(11): p. 1412-1420.
    48. Bauer, M., et al., Using technology to improve longitudinal studies: self‐reporting with ChronoRecord in bipolar disorder. Bipolar Disorders, 2004. 6(1): p. 67-74.
    49. Palmius, N., et al., Detecting Bipolar Depression From Geographic Location Data. IEEE Trans Biomed Eng, 2017. 64(8): p. 1761-1771.
    50. Zhang, S., Nearest neighbor selection for iteratively kNN imputation. Journal of Systems and Software, 2012. 85(11): p. 2541-2552.
    51. Chawla, N.V., et al., SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research, 2002. 16: p. 321-357.
    52. Tahir, M.A., J. Kittler, and F. Yan, Inverse random under sampling for class imbalance problem and its application to multi-label classification. Pattern Recognition, 2012. 45(10): p. 3738-3750.
    53. Jadhav, A., D. Pramod, and K. Ramanathan, Comparison of performance of data imputation methods for numeric dataset. Applied Artificial Intelligence, 2019. 33(10): p. 913-933.
    54. Devlin, J., et al., Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
    55. Vaswani, A., et al., Attention is all you need. Advances in neural information processing systems, 2017. 30.
    56. Quan, C. and F. Ren, A blog emotion corpus for emotional expression analysis in Chinese. Computer Speech & Language, 2010. 24(4): p. 726-749.
    57. Baevski, A., et al., wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 2020. 33: p. 12449-12460.
    58. Jegou, H., M. Douze, and C. Schmid, Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 2010. 33(1): p. 117-128.
    59. Livingstone, S.R. and F.A. Russo, The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PloS one, 2018. 13(5): p. e0196391.
    60. Wen, Z., et al., Distract your attention: multi-head cross attention network for facial expression recognition. arXiv preprint arXiv:2109.07270, 2021.
    61. He, K., et al. Deep residual learning for image recognition. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
    62. Gera, D. and S. Balasubramanian, Landmark guidance independent spatio-channel attention and complementary context information based facial expression recognition. Pattern Recognition Letters, 2021. 145: p. 58-66.
    63. Beretta, L. and A. Santaniello, Nearest neighbor imputation algorithms: a critical evaluation. BMC Med Inform Decis Mak, 2016. 16 Suppl 3(Suppl 3): p. 74.
    64. Hsu, J.-H., Digital Phenotyping-Based Bipolar Disorder Assessment Using Multiple Correlation Data Imputation and Lasso-MLP. 2022.
    65. Dey, R. and F.M. Salem. Gate-variants of gated recurrent unit (GRU) neural networks. in 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS). 2017. IEEE.
    66. clay. 關於 GRU 的圖解概論筆記. 2020; Available from: https://clay-atlas.com/blog/2020/06/02/machine-learning-cn-gru-note/.

    下載圖示 校內：2025-04-30公開
    校外：2025-04-30公開
    QR CODE