近年來，深度學習技術在醫療領域取得了顯著的進展，並為該領域帶來了許多創新應用。然而，在處理時間序列資料時，單一模型通常無法同時有效處理時間序列特徵及捕捉不同病患之間的潛在關聯。
為了解決這些挑戰，本研究提出了一個 Multi-Modal BiLSTM-BERT-GNN 模型架構。該架構結合時間序列模型、語言模型與圖神經網路(Graph Neural Network, GNN)，旨在同時分析時間序列及文本資料，並透過建立病患之間的關聯來提升藥物不良反應(Adverse Drug Reaction, ADR)預測的準確度。具體而言，模型首先針對每位病患，利用 BiLSTM 編碼其生理時間序列特徵；同時，透過預訓練的Bio_ClinicalBERT 模型萃取病歷文本的語意資訊；兩者特徵融合後，以 KNN 建構病患關聯圖，接著運用 GNN 聚合鄰居資訊，最終經由全連接層輸出預測結果。
本研究在真實臨床電子病歷資料集上進行實驗，結果顯示所提多模態模型在準確率、精確率、召回率及 F1 分數等多項指標均優於僅使用單一模態，例如:時間序列、文本模型及未結合圖結構之模型組合。此外，透過調整 KNN 中的鄰居數 k 值，發現適當的 k 能顯著提升模型表現，並且比較了多種 GNN 架構(GCN、GraphSAGE、GAT)，結果指出多模態模型搭配 GraphSAGE 在捕捉病患間隱含關係與預測 ADR 上具有最佳平衡。整體而言，本研究驗證了多模態融合與圖結構建模對於提升 ADR 預測效果的有效性，為臨床決策提供了更為精準與全面的輔助工具。

This study proposes a Multi-Modal BiLSTM-BERT-GNN framework to improve the prediction of Adverse Drug Reactions (ADRs). The model integrates time-series physiological data, clinical text, and patient relationships. BiLSTM encodes physiological signals, while Bio_ClinicalBERT extracts semantic features from medical records. These features are fused and used to construct a patient similarity graph via k-nearest neighbors (KNN). Graph Neural Networks (GCN, GraphSAGE, GAT) are then applied to aggregate neighborhood information for final prediction.
Experiments on real-world electronic health records show that the proposed model outperforms single-modality baselines in accuracy, precision, recall, and F1-score. Among the GNN variants, GraphSAGE achieves the best overall performance. The results demonstrate the effectiveness of multi-modal fusion and graph-based learning for ADR prediction.

Atwood, J., & Towsley, D. (2016). Diffusion-convolutional neural networks. Advances in neural information processing systems, 29.
Beijer, H., & De Blaey, C. (2002). Hospitalisations caused by adverse drug reactions (ADR): a meta-analysis of observational studies. Pharmacy World and Science, 24(2), 46-54.
Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8), 1798-1828.
Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2), 157-166.
Boyd-Graber, J., Hu, Y., & Mimno, D. (2017). Applications of topic models. Foundations and Trends® in Information Retrieval, 11(2-3), 143-296.
Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F., & Sun, J. (2016). Doctor ai: Predicting clinical events via recurrent neural networks. Machine learning for healthcare conference,
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers),
Edwards, I. R., & Aronson, J. K. (2000). Adverse drug reactions: definitions, diagnosis, and management. The lancet, 356(9237), 1255-1259.
Fang, L., Chen, Q., Wei, C.-H., Lu, Z., & Wang, K. (2023). Bioformer: an efficient transformer language model for biomedical text mining. ArXiv, arXiv: 2302.01588 v01581.
Gu, Y., Wang, Y., Zhang, H.-R., Wu, J., & Gu, X. (2023). Enhancing text classification by graph neural networks with multi-granular topic-aware graph. IEEE Access, 11, 20169-20183.
Hamilton, W., Ying, Z., & Leskovec, J. (2017). Inductive representation learning on large graphs. Advances in neural information processing systems, 30.
Haq, H. U., Kocaman, V., & Talby, D. (2022). Mining adverse drug reactions from unstructured mediums at scale. In Multimodal AI in healthcare: A paradigm shift in health intelligence (pp. 361-375). Springer.
Hazell, L., & Shakir, S. A. (2006). Under-reporting of adverse drug reactions. Drug safety, 29(5), 385-396.
Ho, T.-B., Le, L., Thai, D. T., & Taewijit, S. (2016). Data-driven approach to detect and predict adverse drug reactions. Current pharmaceutical design, 22(23), 3498-3526.
Hochreiter, S. (1997). Long Short-term Memory. Neural Computation MIT-Press.
Insalata, B., Schmidt, F., & Vlassov, V. (2024). Multimodal survival prediction using TabTransformer and BioClinicalBERT on MIMIC-III. 2024 IEEE International Conference on Big Data (BigData),
Jeon, E., Kim, Y., Park, H., Park, R. W., Shin, H., & Park, H.-A. (2020). Analysis of adverse drug reactions identified in nursing notes using reinforcement learning. Healthcare Informatics Research, 26(2), 104-111.
Jiang, B., Zhang, Z., Lin, D., Tang, J., & Luo, B. (2019). Semi-supervised learning with graph learning-convolutional networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
Joshi, P., Masilamani, V., & Mukherjee, A. (2022). A knowledge graph embedding based approach to predict the adverse drug reactions using a deep neural network. Journal of Biomedical Informatics, 132, 104122.
Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Ijcai,
Kommu, S., Carter, C., & Whitfield, P. (2024). Adverse drug reactions. In StatPearls [Internet]. StatPearls Publishing.
Kwak, H., Lee, M., Yoon, S., Chang, J., Park, S., & Jung, K. (2020). Drug-disease graph: predicting adverse drug reaction signals via graph neural network with clinical data. Advances in Knowledge Discovery and Data Mining: 24th Pacific-Asia Conference, PAKDD 2020, Singapore, May 11–14, 2020, Proceedings, Part II 24,
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. biometrics, 159-174.
Lardon, J., Abdellaoui, R., Bellet, F., Asfari, H., Souvignet, J., Texier, N., Jaulent, M.-C., Beyens, M.-N., Burgun, A., & Bousquet, C. (2015). Adverse drug reaction identification and extraction in social media: a scoping review. Journal of medical Internet research, 17(7), e171.
Le, Q., & Mikolov, T. (2014). Distributed representations of sentences and documents. International conference on machine learning,
Olsson, S. (1998). The role of the WHO programme on International Drug Monitoring in coordinating worldwide drug safety efforts. Drug safety, 19(1), 1-10.
Pires, T., Schlinger, E., & Garrette, D. (2019). How multilingual is multilingual BERT? arXiv preprint arXiv:1906.01502.
Rumshisky, A., Ghassemi, M., Naumann, T., Szolovits, P., Castro, V., McCoy, T., & Perlis, R. (2016). Predicting early psychiatric readmission with natural language processing of narrative discharge summaries. Translational psychiatry, 6(10), e921-e921.
Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., & Bengio, Y. (2017). Graph attention networks. stat, 1050(20), 10-48550.
Wang, X., Wang, X., & Zhang, S. (2022). Adverse drug reaction detection from social media based on quantum bi-LSTM with attention. IEEE Access, 11, 16194-16202.
Xu, K., Hu, W., Leskovec, J., & Jegelka, S. (2018). How powerful are graph neural networks? arXiv preprint arXiv:1810.00826.
Zhang, M., & Geng, G. (2019). Adverse drug event detection using a weakly supervised convolutional neural network and recurrent neural network model. Information, 10(9), 276.