在台灣現今的健保體系下，醫療費用不斷增加引起政府及社會大眾的關注。醫療費用增加的因素可能包括人口老化、新醫療器材及藥品上市、醫療品質不佳等因素。為了能有效降低醫療保險費用的支出，台灣健保署也制定醫院總額指標來衡量各醫療院所的醫療服務品質，其中也包括以「非計畫性住院案件出院後14天內再住院率」來衡量民眾住院醫療妥善照護狀況，若能有效降低病人再入院率，除了可以降低醫療成本，更可以提高醫療品質，因此本研究提出一個以深度學習模型結合強化學習框架的病人再入院率預測模型，期望可以為醫療產業盡一份心力。
本研究以電子病歷的記載內容為輸入資料，建構住院病人14天內再入院之早期預測模型，預測病人14天內是否會再入院，擷取電子病歷的五種項目分別為入院診斷、出院診斷、主述、治療過程及病程紀錄進行實驗，預測正確率達80%，優於資料集終僅含有病程紀錄或資料集中僅含有其餘四種項目合併之結果。在資料前處理中，本研究使用四種醫療字典將需要的字詞以最大化的方式留下，並刪除較無意義的單詞，以提高模型準確率；因醫療資料集中的每筆資料時間長度不同，因此使用以DQN為框架搭配LSTM的早期預測模型，在全時間段上的預測準確率可達到84%，而在早期預測的部分其預測準確率也有77%，其成效也優於其他機器學習演算法；過往研究中，大部分只給予最終所有病人再入院的模型準確率，並未針對每一位病人提供再入院與非再入院兩類的機率，因此本研究提供專家一份病人再入院比例的指標，期望藉由此指標能夠準確地幫助病人得到最好的照護。

Nowadays, the increasing healthcare costs in Taiwan has aroused concern in the government and the public.
In order to measure the quality of healthcare services and reduce the healthcare costs, The National Health Insurance Administration has set up indications for analyzing healthcare total costs, one of the indications include the “readmission rate within 14 days after leaving the hospital from non-planned inpatient cases”, this indication allows medical professions to observe whether they can provide a proper care in inpatient.
In this study, based on the deep learning model combined with reinforcement learning framework, the prediction of readmission rate within 14 days is built up by taking the electronic patient record as the input data. Due to the variety of factors in data, utilizing DQN as a framework with LSTM has a better outcome than the other algorithms, the accuracy rate of prediction in full-time prediction can reach up to 84%, even up to 77% in early prediction.
However, the majority of early studies only result in the final accuracy rate instead of providing the probability of either readmission or non-readmission in each patient; therefore, this study applies a model of accuracy rate in both probabilities of readmission and non-readmission, which helps the experts to provide better medical care to each patient.

Allahyari, M., Pouriyeh, S., Assefi, M., Safaei, S., Trippe, E. D., Gutierrez, J. B., & Kochut, K. (2017). A brief survey of text mining: Classification, clustering and extraction techniques. arXiv preprint arXiv:1707.02919.
Allaudeen, N., Schnipper, J. L., Orav, E. J., Wachter, R. M., & Vidyarthi, A. R. (2011). Inability of Providers to Predict Unplanned Readmissions. Journal of General Internal Medicine, 26(7), 771-776. doi:10.1007/s11606-011-1663-3
Badea, I., & Trausan-Matu, S. (2013). Text analysis based on time series. Paper presented at the 2013 17th International Conference on System Theory, Control and Computing (ICSTCC).
Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157-166. doi:10.1109/72.279181
Blei, D. M., & Lafferty, J. D. (2006). Dynamic topic models. Paper presented at the Proceedings of the 23rd international conference on Machine learning.
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of machine Learning research, 3(Jan), 993-1022.
Bodenheimer, T. (2005). High and rising health care costs. Part 2: technologic innovation. Annals of internal medicine, 142(11), 932-937.
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the American society for information science, 41(6), 391-407.
Frizzell, J. D., Liang, L., Schulte, P. J., Yancy, C. W., Heidenreich, P. A., Hernandez, A. F., . . . Laskey, W. K. (2017). Prediction of 30-Day All-Cause Readmissions in Patients Hospitalized for Heart Failure: Comparison of Machine Learning and Other Statistical Approaches. JAMA Cardiology, 2(2), 204-209. doi:10.1001/jamacardio.2016.3956
Gao, X. (2018). Deep reinforcement learning for time series: playing idealized trading games. arXiv preprint arXiv:1803.03916.
Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R., & Schmidhuber, J. (2017). LSTM: A Search Space Odyssey. IEEE Transactions on Neural Networks and Learning Systems, 28(10), 2222-2232. doi:10.1109/TNNLS.2016.2582924
Han, B., & Baldwin, T. (2011). Lexical normalisation of short text messages: Makn sens a# twitter. Paper presented at the Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies.
Hsu, C. K. (2018). A Hidden Topic Model for Prediction of crowdfundingCampaigns(Master Thesis, National Cheng-Kung University, Tainan, Republic of China(R.O.C.)). Retrieved from https://hdl.handle.net/11296/2nfg7u
Hartvigsen, T., Sen, C., Kong, X., & Rundensteiner, E. (2019). Adaptive-halting policy network for early classification. Paper presented at the Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
Hasan, O., Meltzer, D. O., Shaykevich, S. A., Bell, C. M., Kaboli, P. J., Auerbach, A. D., . . . Schnipper, J. L. (2010). Hospital readmission in general medicine patients: a prediction model. Journal of General Internal Medicine, 25(3), 211-219.
Heinrich, G. (2005). Parameter estimation for text analysis. Retrieved from
Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735-1780. doi:10.1162/neco.1997.9.8.1735
Hoffman, T. (1999). Probabilistic latent semantic analysis. Paper presented at the proc. of the 15th Conference on Uncertainty in AI, 1999.
Hoffman, M., Bach, F. R., & Blei, D. M. (2010). Online learning for latent dirichlet allocation. Paper presented at the advances in neural information processing systems.
Hofmann, T. (2013). Probabilistic latent semantic analysis. arXiv preprint arXiv:1301.6705.
Huang, B.-Q., Cao, G.-Y., & Guo, M. (2005). Reinforcement learning neural network to the problem of autonomous mobile robot obstacle avoidance. Paper presented at the 2005 International Conference on Machine Learning and Cybernetics.
Huang, H.-S., Liu, C.-L., & Tseng, V. S. (2018). Multivariate time series early classification using multi-domain deep neural network. Paper presented at the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA).
Hussey, P. S., Wertheimer, S., & Mehrotra, A. (2013). The association between health care quality and cost: a systematic review. Annals of internal medicine, 158(1), 27-34.
Jaeger, H., & Haas, H. (2004). Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication. Science, 304(5667), 78. doi:10.1126/science.1091277
Jiang, Z., Xu, D., & Liang, J. (2017). A deep reinforcement learning framework for the financial portfolio management problem. arXiv preprint arXiv:1706.10059.
Jo, Y., Lee, L., & Palaskar, S. (2017). Combining LSTM and latent topic modeling for mortality prediction. arXiv preprint arXiv:1709.02842.
Jozefowicz, R., Zaremba, W., & Sutskever, I. (2015). An empirical exploration of recurrent network architectures. Paper presented at the International conference on machine learning.
Kansagara, D., Englander, H., Salanitro, A., Kagen, D., Theobald, C., Freeman, M., & Kripalani, S. (2011). Risk prediction models for hospital readmission: a systematic review. Jama, 306(15), 1688-1698.
Kerexeta, J., Artetxe, A., Escolar, V., Lozano, A., & Larburu, N. (2018). Predicting 30-day Readmission in Heart Failure using Machine Learning Techniques. Paper presented at the HEALTHINF.
Kern, M. L., Park, G., Eichstaedt, J. C., Schwartz, H. A., Sap, M., Smith, L. K., & Ungar, L. H. (2016). Gaining insights from social media language: Methodologies and challenges. Psychological Methods, 21(4), 507-525. doi:10.1037/met0000091
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Kripalani, S., Theobald, C. N., Anctil, B., & Vasilevskis, E. E. (2014). Reducing hospital readmission rates: current strategies and future directions. Annual review of medicine, 65, 471-485. doi:10.1146/annurev-med-022613-090415
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Paper presented at the Advances in neural information processing systems.
Krumholz, H. M., Parent, E. M., Tu, N., Vaccarino, V., Wang, Y., Radford, M. J., & Hennen, J. (1997). Readmission After Hospitalization for Congestive Heart Failure Among Medicare Beneficiaries. Archives of Internal Medicine, 157(1), 99-104. doi:10.1001/archinte.1997.00440220103013
Le, T. P., Quang, N. D., Choi, S., & Chung, T. (2018). Learning a self-driving bicycle using deep deterministic policy Gradient. Paper presented at the 2018 18th International Conference on Control, Automation and Systems (ICCAS).
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324. doi:10.1109/5.726791
Li, Y. (2017). Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274.
Liang, X., Du, X., Wang, G., & Han, Z. (2019). A deep reinforcement learning network for traffic light cycle control. IEEE Transactions on Vehicular Technology, 68(2), 1243-1253.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., . . . Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.
Lin, C.-H., Chen, W.-L., Lin, C.-M., Lee RN, M.-D., Ko, M.-C., & Li, C.-Y. (2010). Predictors of psychiatric readmissions in the short-and long-term: a population-based study in Taiwan. Clinics, 65(5), 481-489.
Lin, C.-T., & Jou, C.-P. (1999). Controlling chaos by GA-based reinforcement learning neural network. IEEE Transactions on Neural Networks, 10(4), 846-859.
Lin, L.-J. (1992). Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4), 293-321.
Lin, Y.-W., Zhou, Y., Faghri, F., Shaw, M. J., & Campbell, R. H. (2019). Analysis and prediction of unplanned intensive care unit readmission using recurrent neural networks with long short-term memory. PloS one, 14(7), e0218942.
Maass, W., Natschläger, T., & Markram, H. (2002). Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations. Neural Computation, 14(11), 2531-2560. doi:10.1162/089976602760407955
Martinez, C., Perrin, G., Ramasso, E., & Rombaut, M. (2018). A deep reinforcement learning approach for early classification of time series. Paper presented at the 2018 26th European Signal Processing Conference (EUSIPCO).
Mikolov, T. (2012). Statistical language models based on neural networks. Presentation at Google, Mountain View, 2nd April, 80, 26.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., . . . Ostrovski, G. (2015). Human-level control through deep reinforcement learning. nature, 518(7540), 529-533.
Reddy, B. K., & Delen, D. (2018). Predicting hospital readmission for lupus patients: An RNN-LSTM-based deep-learning methodology. Computers in Biology and Medicine, 101, 199-209. doi:https://doi.org/10.1016/j.compbiomed.2018.08.029
Reinhardt, U. E. (2003). Does the aging of the population really drive the demand for health care? Health Affairs, 22(6), 27-39.
Rumshisky, A., Ghassemi, M., Naumann, T., Szolovits, P., Castro, V. M., McCoy, T. H., & Perlis, R. H. (2016). Predicting early psychiatric readmission with natural language processing of narrative discharge summaries. Translational Psychiatry, 6(10), e921-e921. doi:10.1038/tp.2015.182
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized experience replay. arXiv preprint arXiv:1511.05952.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014). Deterministic policy gradient algorithms.
Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction: MIT press.
Sutton, R. S., McAllester, D. A., Singh, S. P., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. Paper presented at the Advances in neural information processing systems.
Tsitsiklis, J. N., & Van Roy, B. (1997). Analysis of temporal-diffference learning with function approximation. Paper presented at the Advances in neural information processing systems.
Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double q-learning. Paper presented at the Proceedings of the AAAI Conference on Artificial Intelligence.
Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., & Freitas, N. (2016). Dueling network architectures for deep reinforcement learning. Paper presented at the International conference on machine learning.
Watkins, C. J., & Dayan, P. (1992). Q-learning. Machine learning, 8(3-4), 279-292.
Wong, E. L. Y., Cheung, A. W. L., Leung, M. C. M., Yam, C. H. K., Chan, F. W. K., Wong, F. Y. Y., & Yeoh, E.-K. (2011). Unplanned readmission rates, length of hospital stay, mortality, and medical costs of ten common medical conditions: a retrospective analysis of Hong Kong hospital data. BMC Health Services Research, 11(1), 149. doi:10.1186/1472-6963-11-149
Yang, C.-L., Yang, C.-Y., Chen, Z.-X., & Lo, N.-W. (2019). Multivariate time series data transformation for convolutional neural network. Paper presented at the 2019 IEEE/SICE International Symposium on System Integration (SII).
Yang, K.-J., Sun, C., Hu, P.-M., Liaw, S.-J., Hu, W.-S., & Chin, H.-K. (2007). Causes of Unplanned Readmission within 14 Days-A Preliminary Study. 台灣家庭醫學雜誌, 17(4), 199-209.
Bodenheimer, T. (2005). High and rising health care costs. Part 2: technologic innovation. Annals of internal medicine, 142(11), 932-937.
Hoffman, T. (1999). Probabilistic latent semantic analysis. Paper presented at the proc. of the 15th Conference on Uncertainty in AI, 1999.
Hussey, P. S., Wertheimer, S., & Mehrotra, A. (2013). The association between health care quality and cost: a systematic review. Annals of internal medicine, 158(1), 27-34.
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Kraemer, H. C. (2014). Kappa coefficient. Wiley StatsRef: Statistics Reference Online, 1-4.
Kripalani, S., Theobald, C. N., Anctil, B., & Vasilevskis, E. E. (2014). Reducing hospital readmission rates: current strategies and future directions. Annual review of medicine, 65, 471-485. doi:10.1146/annurev-med-022613-090415
Li, Y. (2017). Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274.
Martinez, C., Perrin, G., Ramasso, E., & Rombaut, M. (2018). A deep reinforcement learning approach for early classification of time series. Paper presented at the 2018 26th European Signal Processing Conference (EUSIPCO).
Reinhardt, U. E. (2003). Does the aging of the population really drive the demand for health care? Health Affairs, 22(6), 27-39.
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized experience replay. arXiv preprint arXiv:1511.05952.
Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction: MIT press.
Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double q-learning. Paper presented at the Proceedings of the AAAI Conference on Artificial Intelligence.
Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., & Freitas, N. (2016). Dueling network architectures for deep reinforcement learning. Paper presented at the International conference on machine learning.
Watkins, C. J., & Dayan, P. (1992). Q-learning. Machine learning, 8(3-4), 279-292.
Xing, Z., Pei, J., & Philip, S. Y. (2009). Early prediction on time series: A nearest neighbor approach. Paper presented at the Twenty-First International Joint Conference on Artificial Intelligence.