近年來，深度學習逐漸被使用在醫療領域來協助解決臨床問題。其中對於電子病歷的深度學習研究雖逐年增加，但研究方法、任務定義卻相當分歧，且通常僅單靠醫療代碼如診斷碼或藥物編碼這單一種資料類型來學習病人的健康狀況及進行預測，這些問題都限制了深度學習模型在醫療領域的可應用性。

為了解決上述這些問題，我們提出了加護病房病歷編碼器（a ICU record encoder, AICURE），其為編碼器預訓練在各次就診病歷資料上以學習出好的就診病歷向量表達，再針對各個電子病歷預測任務做微調訓練以進行預測。模型的資料集為就診病歷，內容包含醫療代碼、臨床紀錄與病人資訊。我們在病歷編碼器中使用跨領域學習的機制來學習這些不同領域的資料，並導入自然語言處理的概念來處理病史資料（即多次就診病歷）。並且我們依據不同臨床場景設計了四個更符合實際場域狀況的預測任務。

因為預訓練病歷編碼器得到好的就診病歷向量表達，我們的模型可以廣泛應用在多種類型的電子病歷預測任務上，並在此四個任務上都得到突出的表現。此外，透過模型推理過程的視覺分析，病歷編碼器可以提供可解釋的預測結果。最後我們以個案探討的方式呈現此病歷編碼器的表現及性能。

Recently, there have been increasing researches applying deep learning on electronic health records (EHR). However, their methodologies and task definitions are very diverse, and most past works depended solely on medical codes for learning patient's health status and making prediction. These problems limit applicability of deep learning models on medical domain.

To solve difficulties above, we propose AICURE (a ICU record encoder), an encoder pre-trained on each visit record for learning good visit vector and then fine-tuned on each EHR prediction task. Dataset here comprises visit records which contain medical codes, clinical notes and patients' demographics. We adapt cross-modality learning for combining information from different domains, and introduce the concept in natural language processing for learning patient's medical history, i.e. visit record sequence. And we design 4 EHR tasks based on actual clinical scenario situations for more proper definition.

Because our pre-trained AICURE learns good visit vectors, it can be applied to many EHR tasks, and has competitive performances on these 4 tasks. Moreover, our model can provide interpretable predictions by visualizing inference procedure. In the end, we analyze performance and abilities of AICURE by case study.

[1] World Health Organization. Building foundations for eHealth: progress of Member States: report of the WHO lobal Observatory for eHealth. World Health Organization, 2006.
[2] Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F. Stewart, and Jimeng Sun. GRAM: Graph-based attention model for healthcare representation learning. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 787–795, 2017.
[3] Junyuan Shang, Tengfei Ma, Cao Xiao, and Jimeng Sun. Pre-training of graph augmented transformers for medication recommendation. In IJCAI, pages 5953–5959, 2019.
[4] Edward Choi, Mohammad Taha Bahadori, Elizabeth Searles, Catherine Coffey, Michael Thompson, James Bost, Javier Tejedor-Sojo, and Jimeng Sun. Multi-layer representation learning for medical concepts. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 1495–1504, 2016.
[5] Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems, pages 3504–3512, 2016.
[6] Tengfei Ma, Cao Xiao, and Fei Wang. Health-ATM: A deep architecture for multifaceted patient health record representation and risk prediction. In Proceedings of the 2018 SIAM International Conference on Data Mining (SDM), pages 261–269, 2018.
[7] Edward Choi, Zhen Xu, Yujia Li, Michael Dusenberry, Gerardo Flores, Emily Xue, and Andrew Dai. Learning the graphical structure of electronic health records with graph convolutional transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 606–613, Apr. 2020.
[8] Edward Choi, Cao Xiao, Walter Stewart, and Jimeng Sun. MiME: Multilevel medical embedding of electronic health records for predictive healthcare. In Advances in Neural Information Processing Systems, volume 31, pages 4547–4557, 2018.
[9] Benjamin A Goldstein, Ann Marie Navar, Michael J Pencina, and John Ioannidis. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. Journal of the American Medical Informatics Association, 24(1):198–208, 2017.
[10] World Health Organization et al. International classification of diseases: [9th] ninth revision, basic tabulation list with alphabetic index. World Health Organization, 1978.
[11] WHO International Working Group for Drug Statistics Methodology (2004). ATC/DDD classification (final), 2005.
[12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, page 5998–6008, 2017.
[13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, June 2019.
[14] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
[15] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003.
[16] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv: 1301.3781, 2013.
[17] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019.
[18] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237. Association for Computational Linguistics, June 2018.
[19] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32, pages 5753–5763. Curran Associates, Inc., 2019.
[20] Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop of the Association for Computational Linguistics (NAACL), pages 72–78, June 2019.
[21] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020.
[22] Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC-III, a freely accessible critical care database. Scientific data, 3(1):1–9, 2016.
[23] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR 2018, 2017.
[24] Ye Zhang and Byron Wallace. A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. CoRR, abs/1510.03820, 2015.
[25] Larry Medsker and Lakhmi C Jain. Recurrent neural networks: design and applications. CRC press, 1999.
[26] Sebastian Gehrmann, Franck Dernoncourt, Yeran Li, Eric T Carlson, Joy T Wu, Jonathan Welt, John Foote Jr, Edward T Moseley, David W Grant, Patrick D Tyler, et al. Comparing rule-based and deep learning models for patient phenotyping. CoRR, abs/1703.08705, 2017.
[27] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480–1489, 2016.
[28] James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. Explainable prediction of medical codes from clinical text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1101–1111. Association for Computational Linguistics, June 2018.