近年來，雖然AI的技術被廣泛運用到各個領域，但是大家依然無法信任AI的結果，尤其是醫療保健，因為AI的運算過程就像是黑盒子，無法解釋其中決策的要素以及運算邏輯。為了解決黑盒子無法解釋的問題，可解釋人工智慧(Explainable AI, XAI)的概念被提出，XAI嘗試利用可視化的方式將AI的運算過程透明化，來增加人們對於AI的信任。在XAI的方法中，以特徵可視化的引導反向傳播(Guided Backpropagation, GBP)、類別活化映射(Class Activation Mapping, CAM)和梯度加權類別活化映射(Gradient-weighted Class Activation Mapping, Grad-CAM)被廣為使用。基於醫療保健在AI的低可信度下，因此本研究主要探討基於心電圖/脈音圖於分類異位搏動/主動脈瓣疾病的演算法開發並利用特徵可視化的方法，讓AI深度學習模型在疾病分類的運算過程透明化。
演算法主要分成疾病分類和特徵可視化兩個部分，1) 在異位搏動分類中，使用來自MIT-BIH 心律不整資料庫的ECG和基於連續小波轉換的ECG時頻圖對正常(Normal)、心房早期搏動(Atrial Premature Contraction, APC)與心室早期搏動(Ventricular Premature Contraction, VPC)進行分類； 而在主動脈瓣的疾病分類中使用NCKUH資料庫的脈音圖 (Pulse Audiogram, PAG)，並使用連續小波轉換的PAG時頻圖針對非瓣膜疾病(Non-valve Disease, NVD)、主動脈瓣關閉不全(Aortic Regurgitation, AR)與主動脈狹窄(Aortic Stenosis, AS)進行分類。2) 在特徵可視化中，使用GBP、CAM和Grad-CAM來可視化疾病分類模型學習到的特徵。本研究使用的深度學習模型為AlexNet、GoogLeNet以及ResNet50，但是基於CAM只能針對具有全局池化層的模型進行可視化，因為會對此三個分類模型進行修改，模型分別為AlexNet_CAM、GoogLeNet_CAM以及ResNet50_CAM。
在結果的部分，分別比較不同模型的準確率以及在不同可視化方法下的結果。利用心臟病醫師臨床診斷的定義與人類感興趣所定義的特徵來找出具有最佳特徵解釋效果的方法。在異位搏動和主動脈瓣疾病的分類中，CAM平均在所有熱圖中具有最好的特徵解釋，準確率分別為 99.26% 和 98.86%。在此研究中，GBP的熱圖比較類似於邊緣偵測，Grad-CAM則是無法呈現與人類定義相符的特徵，所以CAM的特徵解釋效果最佳。以ECG作為輸入圖像進行疾病辨識的結果中，CAM可以得到與臨床診斷相符的特徵，例如，具有P波與T波的Normal、不明顯P波的APC以及寬QRS複合波的VPC。在時頻圖中，CAM得到與人類定義相符的可視化特徵。
根據前述結果，本研究的結果驗證深度學習在計算過程中學習到的特徵與臨床以及人類定義相符，達到深度學習運算過程透明化的效果，藉此增加大家對於AI結果的可信度。

In recent years, although AI has been widely used in various fields, people still cannot trust the results of AI, especially in medical care. Because the computing process of AI is similar to a black box that difficult to explain the decision and logic. To solve the problem of the black box, the concept of explainable artificial intelligence (XAI) was proposed. XAI attempts to use a visual way to make the calculation process of AI transparent so that it can increase trust from people in AI. Guided backpropagation (GBP), class activation mapping (CAM), and grad-weighted class activation mapping (Grad-CAM) are the famous methods of feature visualization in XAI. Based on the low confidence of AI in health care, this study mainly aims to develop the algorithms based on the ectopic beat/aortic valve disease classification using ECG/Pulse audiogram (PAG) via deep learning models and visualized by feature visualization. And make deep learning models in the calculation process of disease classification transparent.
In the classification of ectopic beats, ECG from the MIT-BIH Arrhythmia Database and ECG spectrogram based on continuous wavelet transform were used for classified Normal, atrial premature contraction (APC), and ventricular premature contraction (VPC). PAG from the NCKUH Database is used in the classification of aortic valve diseases. PAG spectrogram obtained by continuous wavelet transform for classifying the non-valve disease (NVD), aortic regurgitation (AR), and aortic stenosis (AS). 2) In the feature visualization, GBP, CAM, and Grad-CAM were used to visualize features learned by the disease classification model. The deep learning models used in this research are AlexNet, GoogLeNet, and ResNet50. Based on the limitation of CAM, the model must have a global average pooling layer; thus, this research also used the modified CNN model, AlexNet_CAM, GoogLeNet_CAM, and ResNet50_CAM.
The results compared the accuracy of different models and feature visualization methods. Use the definition of physiological characteristics, which are defined by the clinical diagnosis of cardiologists, and human-defined features, which are given by the human interested features, to find the method with the best explanation effect. CAM had the best explanation in the average heat maps of the ectopic beat and AVD classification with 99.26% and 98.86% accuracy, respectively. In this study, the heat map of GBP is similar to edge detection, and Grad-CAM cannot present features that matched the human definition feature, so CAM had the best explanation. The heat map of the ECG signal image, CAM could illustrate the features connected to the physiological characteristics. The P wave, QRS complex, and T wave were demonstrated in the Normal; bizarre P wave in APC; the wide QRS complex and disappeared P wave in VPC. For the spectrogram, CAM could present the features matched to the human-defined feature.
According to the above results, this study verified that the feature learned by CNN which matched the physiological characteristics and human-defined features. Let the deep learning process be transparent, thereby increasing everyone's level of confidence in AI.

[1] “Advantages and Disadvantages of Artificial Intelligence,” 30-Dec-2019. [Online]. Available: https://nexusintegra.io/advantages-disadvantages-artificial-intelligence/. [Accessed: 20-May-2021].
[2] Anurag, “Top 11 Uses and Applications of Artificial Intelligence and ML in Business,” 18-Apr-2020. [Online]. Available: https://www.newgenapps.com/blog/ai-uses-applications-of-artificial-intelligence-ml-business/. [Accessed on: 20-May-2021].
[3] C. Orphanidou, “A Review of Big Data Applications of Physiological Signal Data,” Biophysical reviews, vol. 11, no. 1, pp. 83-87, 2019.
[4] M. Jones, “Why Black Box AI Problem is Bad for Business,” 1-Dec-2020. [Online]. Available: https://techhq.com/2020/12/why-black-box-ai-problem-is-bad-for-business/. [Accessed on: 20-May-2021].
[5] A. Bleicher, “Demystifying the Black Box that is AI,” Scientific American Engineering, 9-Aug-2017. [Online]. Available: https://www.scientificamerican.com/article/demystifying-the-black-box-that-is-ai/. [Accessed on: 20-May-2021].
[6] S. Baum, “Healthcare must Overcome AI’s ‘Black Box’ Problem,” 9-Jan-2019. [Online]. Available: https://medcitynews.com/2019/01/healthcare-must-overcome-ais-black-box-problem/?rf=1. [Accessed on: 20-May-2021].
[7] A. B. Arrieta, N. Díaz-Rodríguez, J. D. Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila and F. Herrera, “Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges Toward Responsible AI,” Information Fusion, vol. 58, pp. 82-115, 2020.
[8] M. Moradi and M. Samwald, “Post-hoc Explanation of Black-box Classifiers using Confident Itemsets,” Expert Systems with Applications, vol. 165, pp. 1-14, 2021.
[9] S. Sattarzadeh, M. Sudhakar, K. N. Plataniotis, J. Jang, Y. Jeong and H. Kim, “Integrated Grad-Cam: Sensitivity-Aware Visual Explanation of Deep Convolutional Networks via Integrated Gradient-Based Scoring,” ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1775-1779, 2021.
[10] P. Linardatos, V. Papastefanopoulos and S. Kotsiantis, “Explainable AI: A Review of Machine Learning Interpretability Methods,” Entropy, vol. 23, no. 1, pp. 1-45, 2021.
[11] S. Rokade, “XAI (Explainable AI) & Top 5 Use Cases,” 16-Apr-2019. [Online]. Available: https://www.linkedin.com/pulse/xai-explainable-ai-top-5-use-cases-shridhan-rokade. [Accessed on: 22-May-2021].
[12] D. Dave, H. Naik, S. Singhal and P. Patel, “Explainable AI Meets Healthcare: A Study on Heart Disease Dataset,” arXiv preprint arXiv:2011.03195, pp. 1-23, 2020.
[13] S. Kaptoge, L. Pennells, D. D. Bacquer, M. T. Cooney, M. Kavousi, G. Stevens, L. M. Riley, S. Savin, T. Khan, S. Altay, P. Amouyel, G. Assmann, S. Bell, Y. Ben-Shlomo, L. Berkman, J. W. Beulens, C. Björkelund, M. Blaha, D. G. Blazer, T. Bolton, R. B. Beaglehole, H. Brenner, E. J. Brunner, E. Casiglia, P. Chamnan, Y. H. Choi, R. Chowdry, S. Coady, C. J. Crespo, M. Cushman, G. R. Dagenais, R. B. D. Sr, M. Daimon, K. W. Davidson, G. Engström, I. Ford, J. Gallacher, R. T. Gansevoort, T. A. Gaziano, S. Giampaoli, G. Grandits, S. Grimsgaard, D. E. Grobbee, V. Gudnason, Q. Guo, H. Tolonen, S. Humphries, H. Iso, J. W. Jukema, J. Kauhanen, A. P. Kengne, D. Khalili, W. Koenig, D. Kromhout, H. Krumholz, T. Lam, G. Laughlin, A. M. Ibañez, T. W. Meade, K. G. M. Moons, P. J. Nietert, T. Ninomiya, B. G. Nordestgaard, C. O'Donnell, L. Palmieri, A. Patel, P. Perel, J. F.Price, R. Providencia, P. M. Ridker, B. Rodriguez, A. Rosengren, R. Roussel, M. Sakurai, V. Salomaa, S. Sato, B. Schöttker, N. Shara, J. E. Shaw, H. C. Shin, L. A. Simons, E. Sofianopoulou, J. Sundström, H. Völzke, R. B. Wallace, N. J. Wareham, P. Willeit, D. Wood, A. Wood, D. Zhao, M. Woodward, G. Danaei, G. Roth, S. Mendis, O. Onuma, C. Varghese, M. Ezzati, I. Graham, R. Jackson, J. Danesh and E. D. Angelantonio, “World Health Organization Cardiovascular Disease Risk Charts: Revised Models to Estimate Risk in 21 Global Regions,” The Lancet Global Health, vol. 7, no. 10, pp. 1332-1345, 2019.
[14] “Cardiovascular Diseases,” World Health Organization. [Online]. Available: https://www.who.int/health-topics/cardiovascular-diseases/#tab=tab_3. [Accessed on: 23-May-2021].
[15] G. A. Roth, G. A. Mensah, Catherine. O. Johnson, G. Addolorato, E. Ammirati, L. M. Baddour, N. C. Barengo, A. Z. Beaton, E. J. Benjamin, C. P. Benziger, A. Bonny, M. Brauer, M. Brodmann, T. J. Cahill, J. Carapetis, A. L. Catapano, S. S. Chugh, L. T. Cooper, J. Coresh, M. Criqui, N. DeCleene, K. A. Eagle, S. Emmons-Bell, V. L. Feigin, J. Fernández-Solà, G. Fowkes, E. Gakidou, S. M. Grundy, F. J. He, G. Howard, F. Hu, L. Inker, G. Karthikeyan, N. Kassebaum, W. Koroshetz, C. Lavie, D. Lloyd-Jones, H. S. Lu, A. Mirijello, A. M. Temesgen, A.Mokdad, A. E. Moran, P. Muntner, J. Narula, B. Neal, M. Ntsekhe, G. M. d. Oliveira, C. Otto, M. Owolabi, M. Pratt, S. Rajagopalan, M. Reitsma, A. L. P. Ribeiro, N. Rigotti, A. Rodgers, C. Sable, S. Shakil, K. Sliwa-Hahnle, B. Stark, J. Sundström, P. Timpel, I. M. Tleyjeh, M. Valgimigli, T. Vos, P. K. Whelton, M. Yacoub, L. Zuhlke, C. Murray and V. Fuster, “Global Burden of Cardiovascular Diseases and Risk Factors, 1990–2019: Update from the GBD 2019 Study,” Journal of the American College of Cardiology, vol. 76, no. 25, pp. 2982-3021, 2020.
[16] R. N. Fogoros, “How Heart Disease is Treated,” 20-Nov-2019. [Online]. Available: https://www.verywellhealth.com/heart-disease-treatments-overview-1745923. [Accessed on: 23-May-2021].
[17] J. Fletcher, “What to Know about Ectopic Heartbeats,” MedicalNewsToday. [Online]. Available: https://www.medicalnewstoday.com/articles/323202. [Accessed on: 11-Aug-2021].
[18] D. Sullivan, “Aortic Valve Disease,” healthline. [Online]. Available: https://www.healthline.com/health/aortic-valve-disease. [Accessed on: 12-Aug-2021].
[19] R. A. Nishimura, “Aortic Valve Disease,” Circulation, vol. 106, no. 7, pp. 770-772, 2002.
[20] M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks,” arXiv e-prints arXiv: 1311, pp. 818-833, 2013.
[21] K. Simonyan, A. Vedaldi and A. Zisserman, “Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps,” arXiv preprint arXiv:1312.6034, pp. 1-8, 2013.
[22] J. T. Springenberg, A. Dosovitskiy, T. Brox and M. Riedmiller, “Striving for Simplicity: The All Convolutional Net,” arXiv preprint arXiv:1412.6806, pp. 1-14, 2014.
[23] K. Wickstrøm, M. Kampffmeyer and R. Jenssen “Uncertainty Modeling and Interpretability in Convolutional Neural Networks for Polyp Segmentation,” 2018 ieee 28th international workshop on machine learning for signal processing (mlsp). IEEE, pp. 1-6, 2018.
[24] J. Lee, D. Lee, Y. Li and B. Shin, “N-Crop Based Image Division in Deep Learning with Medical Image,” Advanced Multimedia and Ubiquitous Engineering, vol. 590, pp. 213-218, 2019.
[25] B. Zhoa, A. Khosla, A. Lapedriza, A. Oliva and A. Torralba, “Learning Deep Features for Discriminative Localization,” Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
[26] P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, R. L. Ball, C. Lamglotz, K. Shpanskaya, M. P. Lungren and A. Y. Ng, “CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning,” arXiv preprint arXiv:1711.05225, pp. 1-7, 2017.
[27] G. Zdolsek, Y. Chen, H. P. Bögl, C. Wang, M. Woisetschläger and J. Schilcher, “Deep Neural Networks with Promising Diagnostic Accuracy for the Classification of Atypical Femoral Fractures,” Acta Orthopaedica, pp. 1-7, 2021.
[28] S. Camalan, H. Mahmood, H. Binol, A. L. D. Araújo, A. R. Santos-Silva, P. A. Vargas, M. A. Lopes, S. A. Khurram and M. N. Gurcan, “Convolutional Neural Network-Based Clinical Predictors of Oral Dysplasia: Class Activation Map Analysis of Deep Learning Results,” Cancers, vol. 13, no, 6, pp. 1-18, 2021.
[29] A. Yildiz, H. Zan and S. Said, “Classification and Analysis of Epileptic EEG Recordings using Convolutional Neural Network and Class Activation Mapping,” Biomedical Signal Processing and Control, vol. 68, pp. 1-11, 2021.
[30] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh and D. Batra, “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization,” Proceedings of the IEEE international Conference on computer vision, pp. 618-626, 2017.
[31] J. H. Moon, D. Y. Lee, W. C. Cha, M. J. Chung, K. -S. Lee, B. H. Cho and J. H. Choi, “Automatic Stenosis Recognition from Coronary Angiography using Convolutional Neural Networks,” Computer methods and programs in biomedicine, vol. 198, pp. 1-11, 2021.
[32] E. Hata, C. Seo, M. Nakayama, K. Iwasaki, T. Ohkawauch and J. Ohya, “Classification of Aortic Stenosis using ECG by Deep Learning and its Analysis Using Grad-CAM,” 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 1548-1551, 2020.
[33] H. Makimoto, M. Höckmann, T. Lin, D. Glöckner, S. Gerguri, L. Clasen, J. Schmidt, A. Assadi-Schmidt, A. Bejinariu, P. Müller, S. Angendohr, M. Babady, C. Brinkmeyer, A. Makimoto and M. Kelm, “Performance of A Convolutional Neural Network Derived from An ECG Database in Recognizing Myocardial Infarction,” Scientific reports, vol. 10, no. 1, pp. 1-9, 2020.
[34] T. Takata, M. Takechi, A. Furukawa, J. Iwasawa, A. Kawamura, T. Taniguchi and Y. Tamura, “Explainable Artificial Intelligence Model for Diagnosis of Atrial Fibrillation using Holter Electrocardiogram Waveforms,” International Heart Journal, vol. 62, no. 3, pp. 534-539, 2021.
[35] T. J. Jun, H. M. Nguyen, D. Kang, D. Kim, D. Kim and Y. H. Kim, “ECG Arrhythmia Classification using A 2-D Convolutional Neural Network,” arXiv preprint arXiv:1804.06812, pp. 1-22, 2018.
[36] M. Degirmenci, M. A. Ozdemir, E. lzci and A. Akan, “Arrhythmic Heartbeat Classification using 2D Convolutional Neural Networks,” IRBM, pp. 1-12, 2021.
[37] H. Alquran, A.M. Alqudah, I. Abu-Qasmieh, A. Al-Badarneh and S. Almashaqbeh, “ECG Classification using High Order Spectral Estimation and Deep Learning Techniques,” Neural Network World, vol. 29, no. 4, pp. 207-219, 2019.
[38] J. Huang, B. Chen, B. Yao and W. He, “ECG Arrhythmia Classification using STFT-Based Spectrogram and Convolutional Neural Network,” IEEE Access, vol. 7, pp, 92871-92880, 2019.
[39] D. Kucharski, D. Grochala, M. Kajor and E. Kańtoch, “A Deep Learning Approach for Valve Defect Recognition in Heart Acoustic Signal,” International Conference on Information Systems Architecture and Technology, vol. 655, pp. 3-14, 2017.
[40] C. Yang, N. D. Aranoff, P. Green and N. Tavassolian, “Classification of Aortic Stenosis Using Time-Frequency Features From Chest Cardio-Mechanical Signals,” IEEE Transactions on Biomedical Engineering, vol. 67, no. 6, pp. 1672-1683, 2019.
[41] D. M. Clark, V. J. Plumb, A. E. Epstein and G. N. Kay, “Hemodynamic Effects of An Irregular Sequence of Ventricular Cycle Lengths During Atrial Fibrillation,” Journal of the American College of Cardiology, vol. 30, pp. 1039-1045, 1997.
[42] R. Mittal, J. H. Seo, V. Vedula, Y. J. Choi, H. Liu, H. H. Huang, S. Jain, L. Younes, T. Abraham and R. T. George, “Computational Modeling of Cardiac Hemodynamics: Current Status and Future Outlook,” Journal of Computational Physics, vol. 305, pp. 1065-1082, 2016.
[43] H. Ouyang, J. Tian, G. Sun, Y. Zou, Z. Liu, H. Li, L. Zhao, B. Shi, Yu. Fan, Yi. Fan, Z. L. Wang and Z. Li, “Self‐Powered Pulse Sensor for Antidiastole of Cardiovascular Disease,” Advanced Materials, vol. 29, pp. 1-10, 2017.
[44] W. Liu, X. Fang, Q. Chen, Y. Li and T. Li, “Reliability Analysis of An Integrated Device of ECG, PPG and Pressure Pulse Wave for Cardiovascular Disease,” Microelectronics Reliability, vol. 87, pp. 183-187, 2018.
[45] J. Y. Chen, C. C. K. Lin, C. W. Lin, F. M. Yu, K. J. Li and L. M. Tsai, “Development of Radial Artery Pulse Audiogram Sensing System for Fast Detection of Atrial Fibrillation and Pulse Amplitude Variation,” IEEE Access, vol. 8, pp. 178770-178781, 2020.
[46] C. W. Lin, Y. Chang, C. C. K. Lin, L. M. Tsai and J. Y. Chen, “Development of An AI-based Non-invasive Pulse AudioGram Monitoring Device for Arrhythmia Screening,” 2017 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT), pp. 40-43, 2017.
[47] Y. Chang, “Development of An AI-based Arrhythmia and Structural Heart Disease Classification Algorithm based on Pulse AudioGram,” M. S. thesis, department of biomedical engineering, NCKU, Tainan, Taiwan (R.O.C), 2018.
[48] Y. L. Xie, “Evaluation of Deep Learning and Image Feature Extraction in Automatic Cardiovascular Disease Recognition based on the Time-Frequency Transformation of ECG and Pulse-audiogram” M. S. thesis, department of biomedical engineering, NCKU, Tainan, Taiwan (R.O.C), 2019.
[49] J. S. Karnewar and M. V. Sarode, “The Combined Effect of Median and FIR Filter in Pre-processing of ECG Signal using Matlab,” International journal of computer applications, pp. 30-33, 2013.
[50] S. Raj and K. C. Ray, “A Personalized Arrhythmia Monitoring Platform,” Scientific reports, vol. 8, no. 1, pp. 1-11, 2018.
[51] W. Nam and G. Y. Han, “Investigations of Local Pressure Drop Fluctuation Signals in Annular Type Fluidized Bed Photoreactor by Continuous Wavelet Transform,” Korean Journal of Chemical Engineering, vol. 22, no. 6, pp. 964-968, 2005.
[52] Z. M. Zin, S. H. Salleh, S. Daliman and M. D. Sulaiman, “Analysis of Heart Sounds based on Continuous Wavelet Transform,” Proceedings. Student Conference on Research and Development, 2003. SCORED 2003. IEEE, pp. 19-22, 2003.
[53] A. Krizhevsky, I. Sutskever and G. E. Hinton, “Imagenet Classification with Deep Convolutional Neural Networks,” Advances in neural information processing systems, vol. 25, pp. 1097-1105, 2012.
[54] Y. Lecun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
[55] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, “Going Deeper with Convolutions,” Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
[56] K. He, X. Zhang, S. Ren and J. Sun, ‘Deep Residual Learning for Image Recognition,” Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[57] T. T. Wong, “Performance Evaluation of Classification Algorithms by k-fold and Leave-One-Out Cross Validation,” Pattern Recognition, vol. 48, no.9, pp. 2839-2846, 2015.
[58] A. Koul, C. Becchio and A. Cavallo, “Cross-Validation Approaches for Replicability in Psychology,” Frontiers in psychology, vol. 9, pp. 1-4, 2018.
[59] R. Draelos, “CNN Heat Maps: Gradients vs. DeconvNets vs. Guided Backpropagation,” 6-Oct-2019. [Online]. Available: https://glassboxmedicine.com/2019/10/06/cnn-heat-maps-gradients-vs-deconvnets-vs-guided-backpropagation/. [Accessed on: 20-Jun-2021].
[60] S. Sattarzadeh, M. Sudhakar, K. N. Plataniotis, J. Jang, Y. Jeong and H. Kim “Integrated Grad-Cam: Sensitivity-Aware Visual Explanation of Deep Convolutional Networks via Integrated Gradient-Based Scoring,” ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp. 1775-1779, 2021.
[61] S. H. Khan, M. Hayat, M. Bennamoun, F. A. Sohel and R. Togneri, “Cost-Sensitive Learning of Deep Feature Representations from Imbalanced Data,” IEEE transactions on neural networks and learning systems, vol. 29, no. 8, pp. 3573-3587, 2017.
[62] S. Mehta, F. Fernandez, C. Villagrán, S. Niklitschek, J. Ávila, R. Botelho, A. Frauenfelder, D. Vieira, M. Ceschim, S. Merchant, A. Vaid, G. Pinto, I. Vallenilla, G. P. D. Nogal, L. Prieto, M. Luna, D. B. Daher, L. Pisana, and J. Lee, “Application of Novel, Class Activation Maps (CAM) in the Development of Artificial Intelligence-guided, Single and 12-lead ECG to Detect ST-Evaluation Myocardial Infarction,” Journal of the American College of Cardiology, vol. 75, no. 11, pp. 3474-3474, 2020.
[63] S. Vijayarangan, B. Murugesan, R. Vignesh, S. Preejith, J. Joseph and M. Sivaprakasam, “Interpreting Deep Neural Networks for Single-Lead ECG Arrhythmia Classification,” 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 300-303, 2020.
[64] T. V. Craenendonck, B. Elen, N. Gerrits and P. D. Boever, “Systematic Comparison of Heatmapping Techniques in Deep Learning in the Context of Diabetic Retinopathy Lesion Detection,” Translational vision science & technology, vol. 9, no. 2, pp. 1-10, 2020.