研究生: |
張舒婷 Chang, Shu-Ting |
---|---|
論文名稱: |
有限臨床與醫療影像資料之配對預訓練 Paired Pre-training for Predictions with Limited Clinical and Medical Image Datasets |
指導教授: |
蘇佩芳
Su, Pei-Fang |
共同指導教授: |
李政德
Li, Cheng-Te |
學位類別: |
碩士 Master |
系所名稱: |
管理學院 - 數據科學研究所 Institute of Data Science |
論文出版年: | 2021 |
畢業學年度: | 109 |
語文別: | 中文 |
論文頁數: | 51 |
中文關鍵詞: | 圖神經網路 、配對預訓練 、微調 、多模態深度學習 |
外文關鍵詞: | Graph Neural Network, Paired Pre-training, Fine-tuning, Multimodal Deep Learning |
相關次數: | 點閱:145 下載:0 |
分享至: |
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報 |
在過去許多研究醫學疾病技術當中,都是集中在分析單一資料平台為主,限制了疾病的預測效果,本研究希望能夠透過多模態(Multimodal) 深度學習的模型,在有限的醫療資料上,同時將臨床、CT 影像以及放射學等不同平台資料做結合分析,來預測乳癌病患是否有骨質疏鬆。其中臨床資料透過類神經網路(NN) 以及圖神經網路(Graph Neural Network, GNN) 架構做分析,為了提取變數之間有用的訊息;影像資料透過卷積神經網路(Convolutional Neural Network, CNN) ResNet50 做訓練;而放射學資料使用類神經網路架構分析。由於本研究的醫學資料樣本數有限,我們提出配對預訓練與微調(Paired Pre-training & Fine-tuning, PairPF) 架構,透過配對的預訓練(Pre-training) 方式,使得模型能學到更好的權重後,再針對我們原始模型架構做權重的微調(Fine-tuning),得到更好的預測效果。研究結果顯示,結合多模態深度學習模型,透過適當的資料合併(Concatenate),有效在預測乳癌病患是否有骨質疏鬆,並且提出配對用來做預訓練的架構,有效使得神經網路模型能學習到重要的特徵。
綜合上述研究結果,與牽涉到單一數據平台相比,多模態深度學習模型可以實現較優的分類效能在分析預測骨質疏鬆上。
In the past, many researches on the technology of medical disease focused on analyzing a single data platform, which limited the predictive effect of the disease. This research hopes to be able to use a multimodal deep learning model on limited medical data. At the same time, the clinical, CT imaging, and radiomics data from different platforms are combined and analyzed to predict whether breast cancer patients have osteoporosis. The clinical data is analyzed through neural network and graph neural network (GNN) architecture to extract useful information between variables; image data is analyzed through convolutional neural network ResNet50 for training; and the radiomics data is analyzed using a neural network architecture. Due to the limited number of medical data samples in this study, we propose a paired pre-training & fine-tuning (PairPF) architecture to enable the model to learn. After getting better weights, we can fine-tune the weights for our original model architecture to get better prediction results. The research results show that combining the multi-modal deep learning model, through appropriate data merging (concatenate), can effectively predict whether breast cancer patients have osteoporosis, and propose a paired architecture for pre-training, which effectively enables the neural network model to be able to learn important characteristics. Based on the above research results, compared with a single data platform, the multi-modal deep learning model can achieve better classification performance in analyzing and predicting osteoporosis.
[1] Abavisani, M., Wu, L., Hu, S., Tetreault, J., and Jaimes, A. Multimodal categorization of crisis events in social media. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020), pp. 14679–14689.
[2] Albawi, S., Mohammed, T. A., and Al-Zawi, S. Understanding of a convolutional neural network. In 2017 International Conference on Engineering and Technology (ICET) (2017), Ieee, pp. 1–6.
[3] Baltruschat, I. M., Nickisch, H., Grass, M., Knopp, T., and Saalbach, A. Comparison of deep learning approaches for multi-label chest x-ray classification. Scientific reports 9, 1 (2019), 1–10.
[4] Carreira, J., and Zisserman, A. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 6299–6308.
[5] Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., and Elhadad, N. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining (2015), pp. 1721–1730.
[6] Chen, L., Zhou, Z., Sher, D., Zhang, Q., Shah, J., Pham, N.-L., Jiang, S., and Wang, J. Combining many-objective radiomics and 3d convolutional neural network through evidential reasoning to predict lymph node metastasis in head and neck cancer. Physics in Medicine & Biology 64, 7 (2019), 075011.
[7] Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., et al. Xgboost: extreme gradient boosting. R package version 0.4-2 1, 4 (2015).
[8] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning (2020), PMLR, pp. 1597–1607.
[9] Dunnmon, J. A., Yi, D., Langlotz, C. P., Ré, C., Rubin, D. L., and Lungren, M. P. Assessment of convolutional neural networks for automated classification of chest radiographs. Radiology 290, 2 (2019), 537–544.
[10] García-Durán, A., and Niepert, M. Learning graph representations with embedding propagation. arXiv preprint arXiv:1710.03059 (2017).
[11] Girshick, R. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision (2015), pp. 1440–1448.
[12] Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. Recent advances in convolutional neural networks. Pattern Recognition 77 (2018), 354–377.
[13] Han, Y., Chen, C., Tewfik, A. H., Ding, Y., and Peng, Y. Pneumonia detection on chest x-ray using radiomic features and contrastive learning. arXiv preprint arXiv:2101.04269 (2021).
[14] He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (2016), pp. 770–778.
[15] Kipf, T. N., and Welling, M. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).
[16] Kwok, D. P.-Y. 台灣精準醫療計畫taiwan precision medicine nitiative, 2019. https://tpmi.ibms.sinica.edu.tw/www/en/precision-medicine/.
[17] LeCun, Y., et al. Lenet-5, convolutional neural networks. URL:http://yann. lecun.com/exdb/lenet 20, 5 (2015), 14.
[18] Li, Z., Cui, Z., Wu, S., Zhang, X., and Wang, L. Fi-gnn: Modeling feature interactions via graph neural networks for ctr prediction. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (2019), pp. 539–548.
[19] Lim, H. K., Ha, H. I., Park, S.-Y., and Han, J. Prediction of femoral osteoporosis using machine-learning analysis with radiomics features and abdomen-pelvic ct: A retrospective single center preliminary study. PloS one 16, 3 (2021), e0247330.
[20] Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (2015), pp. 3431–3440.
[21] Malone, B., Garcia-Duran, A., and Niepert, M. Learning representations of missing data for predicting patient outcomes. arXiv preprint arXiv:1811.04752 (2018).
[22] Muslim, D., Mohd, E., Sallehudin, A., Muzaffar, T. T., and Ezane, A. Performance of osteoporosis self-assessment tool for asian (osta) for primary osteoporosis in postmenopausal malay women. Malaysian orthopaedic journal 6, 1 (2012), 35.
[23] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 8024–
8035.
[24] Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine 1, 1 (2018), 1–10.
[25] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 3 (2015), 211–252.
[26] Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. The graph neural network model. IEEE transactions on neural networks 20, 1 (2008), 61–80.
[27] Simonyan, K., and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[28] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (2015), pp. 1–9.
[29] Van Griethuysen, J. J., Fedorov, A., Parmar, C., Hosny, A., Aucoin, N., Narayan, V., Beets-Tan, R. G., Fillion-Robin, J.-C., Pieper, S., and Aerts, H. J. Computational radiomics system to decode the radiographic phenotype. Cancer research 77, 21 (2017), e104–e107.
[30] Wang, X., Xu, Z., Tam, L., Yang, D., and Xu, D. Self-supervised image-text pre-training with mixed data in chest x-rays. arXiv preprint arXiv:2103.16022 (2021).
[31] Yasaka, K., Akai, H., Kunimatsu, A., Kiryu, S., and Abe, O. Prediction of bone mineral density from computed tomography: application of deep learning with a convolutional neural network. European radiology (2020), 1–9.
[32] Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. arXiv preprint arXiv:2106.04560 (2021).
[33] Zhang, X., Chou, J., and Wang, F. Integrative analysis of patient health records and neuroimages via memory-based graph convolutional network. In 2018 IEEE International Conference on Data Mining (ICDM) (2018), IEEE, pp. 767–776.
[34] Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747 (2020).
[35] Zhou, H.-Y., Yu, S., Bian, C., Hu, Y., Ma, K., and Zheng, Y. Comparing to learn: Surpassing imagenet pretraining on radiographs by comparing image representations. In International Conference on Medical Image Computing and Computer-Assisted Intervention (2020), Springer, pp. 398–407.