深度學習與機器學習在電腦視覺領域的應用在近幾年來獲得了非常快速的進步，而其中多模態的邁維斯磁振造影分析也是其中重要的一個研究項目，由於在醫學項目中病人的稀少性，並且資料需要領域專家標註，構造大型資料集是非常困難的，因此如何利用多模態磁振造影來補足小資料的缺點是一個很重要的研究，在本篇論文中，我們提出了兩個分別基於機器學習以及深度學習的多模態造影分析框架來解決兩個臨床醫學上的重要問題，首先，我們使用灰度共生矩陣作為特徵提取器，並利用支持向量機處理多模態照片的紋理特徵、LightGBM處理病人的臨床數值資料；最後使用隨機森林將兩者特徵融和應用於腦膜瘤復發預測。
我們提出的第二個方法為基於自監督學習的多模態對比學習模型應用於肝癌期別分類。
自監督學習在自然語言處理及電腦視覺中都有廣泛的應用，在此方法中，我們首先利用多模態照片的特性來訓練一個多模態的權重共享對比模型，在下游階段我們更進一步的利用跨模態分組卷積網路融合不同模態的信息。具體來說，在預訓練階段我們隨機選擇兩張同一病人在同樣解剖結構的模態照片作為正樣本，使用Siamese網路最大化兩者特徵向量的相似度，藉由這個輔助任務來訓練出強大的多模態表徵抽取模型。在下游階段為了更進一步利用多模態的特徵，我們提出跨模態分組卷積層，結合3D以及2D的分組卷積層並加入通道打亂機制來有效率的融合多模態特徵。而我們提出的兩個方法分別實驗於台南奇美醫院收集的腦瘤以及肝癌相關病人的邁維斯磁振造影資料集，實驗結果也證實我們提出的架構在兩個臨床醫學問題上均能贏過目前現有作法。

The application of computer vision using deep learning and machine learning has achieved promising results in computer vision, and it also recently received attention in the medical domain. Among these studies, medical imaging analysis using multi-modality learning is one of the important research topics recentlly.
In practice, medical data is hard to collect and more difficult to annotate. Therefore, it is crucial to leverage the multi-modality medical images to make up small datasets. In this paper, we present two approaches to analyze the recurrence in parasagittal and parafalcine meningiomas (PSPF) and liver cancer using machine learning and deep learning respectively. In the first study, we adopt GLCM and SVM to tackle the modality-modality images. Besides, we use lightGBM to extract clinical features. Finally, we combine both imaging and clinical features using ensemble methods.
In the second study, we leverage the data characteristic of multi-modality Magnetic Resonance (MR) images to learn multi-modality contrastive representations.
We first present multi-modality data augmentation (MDA) to adapt contrastive learning to multi-modality learning.
Then, the proposed cross-modality group convolution (CGC) is used to utilize the meaningful multi-modality feature in the downstream fine-tune task.
Specifically, in the pre-training stage, considering different behaviors from each MRI modality with the same anatomic structure, without designing a handcrafted pretext task, we select two augmented MR images from a patient as positive pairs, and then directly maximize the similarity between positive pairs using Simple Siamese networks.
To further exploit multi-modality representation, we combine 3D and 2D group convolution with a channel shuffle operation to efficiently incorporate different modalities of image features.
We evaluate our proposed methods on liver MR images collected from CHEMEI hospital. Experimental results show our proposed frameworks have significant improvement from previous methods.

[1] Mohamed Abuella and Badrul Chowdhury. Random forest ensemble of support vector regression models for solar power forecasting. In 2017 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT), pages 1–5. IEEE, 2017.
[2] Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, et al. Big self-supervised models advance medical image classification. arXiv preprint arXiv:2101.05224, 2021.
[3] Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001.
[4] Chaoyue Chen, Xinyi Guo, Jian Wang, Wen Guo, Xuelei Ma, and Jianguo Xu. The diagnostic value of radiomics-based machine learning in predicting the grade of meningiomas using conventional magnetic resonance imaging: a preliminary study. Frontiers in oncology, page 1338, 2019.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
[6] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020.
[7] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-
training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[9] Stavros I Dimitriadis, Dimitris Liparas, Magda N Tsolaki, Alzheimer’s Disease Neuroimaging Initiative, et al. Random forest feature selection, fusion and ensemble strategy: Combining multiple morphological mri measures to discriminate among healhy elderly, mci, cmci and alzheimer＇s disease patients: From the alzheimer＇s disease neuroimaging initiative (adni) database. Journal of neuroscience methods, 302:14–23, 2018.
[10] Harris Drucker, Chris Burges, Linda Kaufman, Alex Smola, and Vladimir Vapnik. Linear support vector regression machines. Advances in neural information processing systems, 9:155–161, 1996.
[11] Yanghua Fan, Shenzhong Jiang, Min Hua, Shanshan Feng, Ming Feng, and Renzhi Wang. Machine learning-based radiomics predicts radiotherapeutic response in patients with acromegaly. Frontiers in endocrinology, page 588, 2019.
[12] Muhammad Al Fatih Abil Fida, Tohari Ahmad, and Maurice Ntahobari. Variance threshold as early screening to boruta feature selection for intrusion detection system. In 2021 13th International Conference on Information & Communication Technology and System (ICTS), pages 46–50. IEEE, 2021.
[13] Lucas Fidon, Wenqi Li, Luis C Garcia-Peraza-Herrera, Jinendra Ekanayake, Neil
Kitchen, Sebastien Ourselin, and Tom Vercauteren. Scalable multimodal convolutional networks for brain tumour segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 285–293. Springer, 2017.
[14] James Franklin. The elements of statistical learning: data mining, inference and prediction. The Mathematical Intelligencer, 27(2):83–85, 2005.
[15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[16] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to selfsupervised learning. arXiv preprint arXiv:2006.07733, 2020.
[17] Zhe Guo, Xiang Li, Heng Huang, Ning Guo, and Quanzheng Li. Medical image segmentation based on multi-modal convolutional neural network: Study on image fusion schemes. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 903–907. IEEE, 2018.
[18] Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. Journal of machine learning research, 3(Mar):1157–1182, 2003.
[19] Mahdi Hashemi. Enlarging smaller images before inputting into convolutional neural network: zero-padding vs. interpolation. Journal of Big Data, 6(1):1–13, 2019.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
[22] Faruk Ildan, Tahsin Erman, A İskender Göçer, Metin Tuna, Hüseyin Bağdatoğlu, Erdal Çetinalp, and Refik Burgut. Predicting the probability of meningioma recurrence in the preoperative and early postoperative period: a multivariate analysis in the midterm follow-up. Skull base, 17(03):157–171, 2007.
[23] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. PMLR, 2015.
[24] Shweta Jain. Brain cancer classification using glcm based feature extraction in artificial neural network. International Journal of Computer Science & Engineering Technology, 4(7):966–970, 2013.
[25] Jue Jiang, Yu-Chi Hu, Neelam Tyagi, Pengpeng Zhang, Andreas Rimner, Gig S Mageras, Joseph O Deasy, and Harini Veeraraghavan. Tumor-aware, adversarial domain adaptation from ct to mri for lung cancer segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 777–785. Springer, 2018.
[26] Peng-Tao Jiang, Chang-Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Layercam: Exploring hierarchical class activation maps for localization. IEEE Trans- actions on Image Processing, 30:5875–5888, 2021.
[27] Jayashri Joshi and MA Phadke. Feature extraction and texture classification in mri. Energy, 1:0, 2010.
[28] Konstantinos Kamnitsas, Christian Ledig, Virginia FJ Newcombe, Joanna P Simpson, Andrew D Kane, David K Menon, Daniel Rueckert, and Ben Glocker. Efficient multiscale 3d cnn with fully connected crf for accurate brain lesion segmentation. Medical image analysis, 36:61–78, 2017.
[29] Alexander Ke, William Ellsworth, Oishi Banerjee, Andrew Y Ng, and Pranav Ra-
jpurkar. Chextransfer: performance and parameter efficiency of imagenet models for chest x-ray interpretation. In Proceedings of the Conference on Health, Inference, and Learning, pages 116–124, 2021.
[30] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
[31] Ching-Chung Ko, Yang Zhang, Jeon-Hor Chen, Kai-Ting Chang, Tai-Yuan Chen, SherWei Lim, Te-Chang Wu, and Min-Ying Su. Pre-operative mri radiomics for the prediction of progression and recurrence in meningiomas. Frontiers in neurology, 12:599, 2021.
[32] Ron Kohavi and George H John. Automatic parameter selection by minimizing estimated error. In Machine Learning Proceedings 1995, pages 304–312. Elsevier, 1995.
[33] Nikos Komodakis and Spyros Gidaris. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018.
[34] Leon Kopitar, Primoz Kocbek, Leona Cilar, Aziz Sheikh, and Gregor Stiglic. Early detection of type 2 diabetes mellitus using machine learning-based prediction models. Scientific reports, 10(1):1–12, 2020.
[35] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097–1105, 2012.
[36] Kang Li, Lequan Yu, Shujun Wang, and Pheng-Ann Heng. Towards cross-modality medical image segmentation with online mutual knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 775–783, 2020.
[37] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint
arXiv:1312.4400, 2013.
[38] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey on deep learning in medical image analysis. Medical image analysis, 42:60–88, 2017.
[39] Zhenyu Liu, Xiao-Yan Zhang, Yan-Jie Shi, Lin Wang, Hai-Tao Zhu, Zhenchao Tang, Shuo Wang, Xiao-Ting Li, Jie Tian, and Ying-Shi Sun. Radiomics analysis for evaluation of pathological complete response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer. Clinical Cancer Research, 23(23):7253–7262, 2017.
[40] David N Louis, Arie Perry, Guido Reifenberger, Andreas Von Deimling, Dominique Figarella-Branger, Webster K Cavenee, Hiroko Ohgaki, Otmar D Wiestler, Paul Kleihues, and David W Ellison. The 2016 world health organization classification of tumors
of the central nervous system: a summary. Acta neuropathologica, 131(6):803–820, 2016.
[41] Gilles Louppe, Louis Wehenkel, Antonio Sutera, and Pierre Geurts. Understanding variable importances in forests of randomized trees. Advances in neural information processing systems, 26, 2013.
[42] Angel Maillo, Alberto Orfao, Ana B Espinosa, José María Sayagués, Marta Merino, Pablo Sousa, Monica Lara, and María Dolores Tabernero. Early recurrences in histologically benign/grade i meningiomas are associated with large tumors and coexistence
of monosomy 14 and del (1p36) in the ancestral tumor cell clone. Neuro-oncology, 9(4):438–446, 2007.
[43] Xue Ming, Ronald Wihal Oei, Ruiping Zhai, Fangfang Kong, Chengrun Du, Chaosu Hu, Weigang Hu, Zhen Zhang, Hongmei Ying, and Jiazhou Wang. Mri-based radiomics
signature is a quantitative prognostic biomarker for nasopharyngeal carcinoma. Scientific reports, 9(1):1–9, 2019.
[44] P Mohanaiah, P Sathyanarayana, and L GuruKumar. Image texture feature extraction using glcm approach. International journal of scientific and research publications, 3(5):1–5, 2013.
[45] Olivier Morin, William C Chen, Farshad Nassiri, Matthew Susko, Stephen T Magill, Harish N Vasudevan, Ashley Wu, Martin Vallières, Efstathios D Gennatas, Gilmer Valdes, et al. Integrated models incorporating radiologic and radiomic features predict meningioma grade, local failure, and overall survival. Neuro-oncology advances, 1(1):vdz011, 2019.
[46] Nasser M Nasrabadi. Pattern recognition and machine learning. Journal of electronic imaging, 16(4):049901, 2007.
[47] Yae Won Park, Jongmin Oh, Seng Chan You, Kyunghwa Han, Sung Soo Ahn,
Yoon Seong Choi, Jong Hee Chang, Se Hoon Kim, and Seung-Koo Lee. Radiomics
and machine learning may accurately predict the grade and histological subtype in meningiomas using conventional and diffusion tensor imaging. European radiology, 29(8):4068–4076, 2019.
[48] Jérôme Paul, Michel Verleysen, Pierre Dupont, et al. Identification of statistically significant features from random forests. In ECML workshop on Solving Complex Machine Learning Problems with Ensemble Methods, pages 69–80, 2013.
[49] Jenny Pettersson-Segerlind, Abiel Orrego, Stefan Lönn, and Tiit Mathiesen. Long-term 25-year follow-up of surgically treated parasagittal meningiomas. World neurosurgery, 76(6):564–571, 2011.
[50] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74, 1999.
[51] Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
[52] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
[53] Mark R Segal. Machine learning benchmarks and random forest regression. 2004.
[54] Yingying Song, Xueli Jiao, Yuheng Qiao, Xinrui Liu, Yiding Qiang, Zhiyong Liu, and Lin Zhang. Prediction of double-high biochemical indicators based on lightgbm and xgboost. In Proceedings of the 2019 international conference on artificial intelligence and computer science, pages 189–193, 2019.
[55] Hari Sowrirajan, Jingbo Yang, Andrew Y Ng, and Pranav Rajpurkar. Moco-cxr: Moco pretraining improves representation and transferability of chest x-ray models. arXiv preprint arXiv:2010.05352, 2020.
[56] Aiham Taleb, Christoph Lippert, Tassilo Klein, and Moin Nabi. Multimodal self-
supervised learning for medical image analysis. In International Conference on In-
formation Processing in Medical Imaging, pages 661–673. Springer, 2021.
[57] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105–6114. PMLR, 2019.
[58] Simon Tong and Edward Chang. Support vector machine active learning for image retrieval. In Proceedings of the ninth ACM international conference on Multimedia, pages 107–118, 2001.
[59] Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil
Houlsby, Sylvain Gelly, and Mario Lucic. Self-supervised learning of video-induced
visual invariances. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 13806–13815, 2020.
[60] Kuan-Lun Tseng, Yen-Liang Lin, Winston Hsu, and Chung-Yang Huang. Joint sequence learning and cross-modality convolution for 3d biomedical segmentation. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition,
pages 6393–6400, 2017.
[61] R Usha and K Perumal. Svm classification of brain images from mri scans using morphological transformation and glcm texture features. International journal of computational systems engineering, 5(1):18–23, 2019.
[62] Yen Nhi Truong Vu, Richard Wang, Niranjan Balachandar, Can Liu, Andrew Y Ng, and Pranav Rajpurkar. Medaug: Contrastive learning leveraging patient metadata improves representations for chest x-ray interpretation. arXiv preprint arXiv:2102.10663, 2021.
[63] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929–9939. PMLR, 2020.
[64] Joseph Wiemels, Margaret Wrensch, and Elizabeth B Claus. Epidemiology and etiology of meningioma. Journal of neuro-oncology, 99(3):307–314, 2010.
[65] Liping Yang, Panpan Xu, Ying Zhang, Nan Cui, Menglu Wang, Mengye Peng, Chao Gao, and Tianzuo Wang. A deep learning radiomics model may help to improve the prediction performance of preoperative grading in meningioma. Neuroradiology, pages 1–10, 2022.
[66] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In
European conference on computer vision, pages 649–666. Springer, 2016.
[67] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018.
[68] Yang Zhang, Jeon-Hor Chen, Tai-Yuan Chen, Sher-Wei Lim, Te-Chang Wu, Yu-Ting Kuo, Ching-Chung Ko, and Min-Ying Su. Radiomics approach for prediction of recurrence in skull base meningiomas. Neuroradiology, 61(12):1355–1364, 2019.
[69] Yongbei Zhu, Chuntao Man, Lixin Gong, Di Dong, Xinyi Yu, Shuo Wang, Mengjie Fang, Siwen Wang, Xiangming Fang, Xuzhu Chen, et al. A deep learning radiomics model for preoperative grading in meningioma. European journal of radiology, 116:128–134, 2019.