結核病是全球十大死亡原因之一，經統計2018年有超過一千萬個新增病例，以及超過一百四十萬人因此而喪失生命。結核病主要是經由飛沫傳染，在未開發與發展中國家由其嚴重，超過八成案例高度集中在情況最為嚴俊的三十個國家。結核病的檢驗方法主要包含皮膚測試、胸部Ｘ光檢查、顯微鏡鏡檢、分子測試與培養等。其中以一般光顯微鏡鏡檢最為廣泛使用，其方法的優點是快速、方便製作、價格便宜以及可經由一般顯微鏡直接進行觀查。但缺點是人工觀察非常耗時、準確率偏低、以及高度仰賴醫檢人員經驗而容易造成觀察者間誤差等問題。值得注意的是，結核病是可以治癒的，所以早期發現早期治療對病人是很有幫助的。因此，如何提供一套自動、快速且準確的鏡檢方法成為非常重要的研究課題。
本研究針對一般光顯微鏡抗酸性染色結核菌痰液抹片影像，進行自動結核菌辨識。在一個一千倍油鏡下所看到的視野中，能看到的結核菌是非常細小的。目前己有很多研究針對面積小的物件進行偵測，但對於背景變異度高的顯微鏡影像，要達到很高的準確率仍然是非常困難的。為了克服以上問題，本研究提出兩階段式結核菌辨識方法，首先針對影像中的結核菌進行偵測，接著對偵測到的候選物進行是否為結核菌的分類。
本方法第一階段使用了改良更快速區域基礎類神經網路來進行目標物體偵測，第二階段分類器加入了多種新型卷積神經網路應用技術，相對原始方法在準確率上有顯著的提升。同時，亦提出了加強訓練方法使整體準確度提升，以及讓輸出之分類期望值更接近現實的常態分佈，而不會出現極端的錯誤期望值。本研究針對三個醫學中心所提供的抹片影像共超過三十萬張，進行訓練與測試。其中系統的驗證結果，在影像的敏感度與特異度分別高達0.91、0.99，在針對目標物的評估，本系統之召回率與精確率更分別高達0.86、0.84。
經過大量的影像資料驗證，證實兩階段方法對於辨識影像中細小物件有著非常高的穩定性與準確性。而且在加入多種新型卷積神經網路應用技術後，對系統效能有明顯提升，另外，系統在經過加強式訓練後，能給出更接近現實的常態預測分數。本研究系統對於鏡檢之實際應用將有著很大的幫助，未來將進行更大量、更多跨資料集與肺外檢體之測試與驗證。

Tuberculosis (TB) is one of the top ten causes of death in the world. According to the Global TB report, there were more than 10 million new cases and more than 1.4 million deaths on this disease in 2018. Tuberculosis is an airborne disease, and it is severe in undeveloped and developing countries. Over 80% of cases existed in the 30 high TB burden countries. There are many diagnosis methods for TB, for example: skin test, chest X-ray examination, sputum smear microscopic, molecular and culture tests. Among these methods, the conventional light sputum smear microscopy is the most widely applied. The advantages of this method are fast, low cost, convenient to apply and the pathologist can examine through a general microscope. However, the disadvantages are time-consuming, low accuracy, high dependence on the pathologist experience which is likely to cause inter- or intra observer variances. It is worth to note that TB can be cured, so that early detection and early treatment is very important. Because of this, how to provide an automatic, fast and accurate microscopic examination method has become a very important research topic.
In this dissertation, an automatic method for Mycobacterium tuberculosis (MTB) identification from acid-fast bacteria smear microscopy images is proposed. From the field of view of a thousand times oil lens, the MTB can be seen are very small. Numerous studies had investigated the detection of small objects, but due to large background variation of microscopy images, to achieve high accuracy is still a highly challenging task. In order to overcome this, a two-stage MTB identification method is proposed which first detects the MTB candidates from the image, and then classifies them finely.
A refined Faster Regional-based convolutional neural network (RFR-CNN) is applied as the first stage for target object detection. The second stage classifier incorporates a series of advance application technologies of convolutional neural networks, which can further improve the performance from the original one. In addition, a novel reinforcement training strategy is proposed to improve the system accuracy, and to make the prediction output probability more realistic, which is more close to the normal distribution, so as to prevent high confidence scores on false predictions. In this study, more than 300,000 microscopy images provided by three medical centers are evaluated. From the experimental results, the average sensitivity and specificity of our system are 0.91 and 0.99, respectively, and the recall and precision are 0.86 and 0.84, respectively.
After evaluation of a large amount of images, the results show that our proposed two-stage method achieves high reliability and accuracy for the identification of MTB from microscopy images. Moreover, after integrating the advance application technologies of convolutional neural networks, the system performance has been significantly improved. On the other hand, the proposed reinforcement training strategy let the system predicts more normally. The false samples concentrate on the probability of 0.5 (50%) which is convenient for double checking and further improvement and analysis. All of these can help to the practical application of sputum smear microscopy. In the future, more data and cross-medical centers datasets will be evaluated thoroughly, and extra-pulmonary tuberculosis microscopy images will also be studied.

[1] WHO, "Global TB report," WHO, 2019.
[2] Das, P. K., Ganguly, S. B., & Mandal, B., "Sputum smear microscopy in tuberculosis: It is still relevant in the era of molecular diagnosis when seen from the public health perspective," Biomedical and Biotechnology Research Journal, 2019.
[3] Momenzadeh, M., Vard, A., Talebi, A., Mehri Dehnavi, A., & Rabbani, H, "Computer‐aided diagnosis software for vulvovaginal candidiasis detection from Pap smear images," Microscopy research and technique, 2018.
[4] MoradiAmin, M., Memari, A., Samadzadehaghdam, N., Kermani, S., & Talebi, A., "Computer aided detection and classification of acute lymphoblastic leukemia cell subtypes based on microscopic image analysis," Microscopy research and technique, 2016.
[5] Rachmad, A., Chamidah, N., & Rulaningtyas, R., "Mycobacterium tuberculosis identification based on colour feature extraction using expert system," Annals of Biology, 2020.
[6] Díaz-Huerta, J. L., del Carmen Téllez-Anguiano, A., Gutiérrez-Gnecchi, J. A., Colin-González, O. Y., Zavala-Santoyo, F. L., & Arellano-Calderón, S., "Image preprocessing to improve acid-fast bacilli (AFB) detection in smear microscopy to diagnose pulmonary tuberculosis," in International Conference on Electronics, Communications and Computers (CONIELECOMP), 2019.
[7] Shah, M. I., Mishra, S., Sarkar, M., & Sudarshan, S. K., "Automatic detection and classification of tuberculosis bacilli from camera-enabled smartphone microscopic images," in 2016 Fourth International Conference on Parallel, Distributed and Grid Computing (PDGC), 2016.
[8] Kang, M., Leng, X., Lin, Z., & Ji, K., "A modified faster R-CNN based on CFAR algorithm for SAR ship detection," in International Workshop on Remote Sensing with Intelligent Processing (RSIP), 2017.
[9] Eggert, C., Zecha, D., Brehm, S., & Lienhart, R., "Improving small object proposals for company logo detection," in Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, 2017.
[10] Zhang, J., Hu, H., Chen, S., Huang, Y., & Guan, Q., "Cancer cells detection in phase-contrast microscopy images based on Faster RCNN," in 9th International Symposium on Computational Intelligence and Design (ISCID), 2016.
[11] Xu, Y., Li, Y., Wang, Y., Liu, M., Fan, Y., Lai, M., … Chang, C., "Gland instance segmentation using deep multichannel neural networks," IEEE Transactions on Biomedical Engineering, p. 2901–2912, 2017.
[12] Cao, C., Wang, B., Zhang, W., Zeng, X., Yan, X., Feng, Z., ... & Wu, Z., "An improved faster R-CNN for small object detection," IEEE Access, pp. 106838-106846, 2019.
[13] Kant, S., & Srivastava, M. M., "Towards automated tuberculosis detection using deep learning," in 2018 IEEE Symposium Series on Computational Intelligence (SSCI), 2018.
[14] López, Y. P., Costa Filho, C. F., Aguilera, L. M. R., & Costa, M. G. F., "Automatic classification of light field smear microscopy patches using convolutional neural networks for identifying Mycobacterium tuberculosis," in 2017 CHILEAN Conference on Electrical, Electronics Engineering, Information and Communication Technologies (CHILECON), 2017.
[15] Hu, M., Liu, Y., Zhang, Y., Guan, T., & He, Y., "Automatic detection of tuberculosis bacilli in sputum smear scans based on subgraph classification," in 2019 International Conference on Medical Imaging Physics and Engineering (ICMIPE), 2019.
[16] El-Melegy, M., Mohamed, D., ElMelegy, T., & Abdelrahman, M., "Identification of tuberculosis bacilli in ZN-stained sputum smear images: a deep learning approach," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019.
[17] LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P., "Gradient-based learning applied to document recognition," in Proceedings of the IEEE, 1998.
[18] Krizhevsky, A., Sutskever, I., & Hinton, G. E., "Imagenet classification with deep convolutional neural networks," in Advances in Neural information Processing Systems (NIPS), 2012.
[19] Simonyan, K., & Zisserman, A., "Very deep convolutional networks for large-scale image recognition," in arXiv preprint arXiv:1409.1556., 2014.
[20] He, K., Zhang, X., Ren, S., & Sun, J., "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
[21] Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K., "Aggregated residual transformations for deep neural networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
[22] wikimedia.org, "https://upload.wikimedia.org," 2020. [Online]. Available: https://upload.wikimedia.org/wikipedia/en/8/83/VGG_neural_network.png. [Accessed 18 6 2020].
[23] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C., "SSD: single shot multibox detector," in Proceedings of the European Conference on Computer Vision (ECCV), 2016.
[24] Redmon, J., Divvala, S., Girshick, R., & Farhadi, A., "You only look once: Unified, real-time object detection," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[25] Redmon J., & Farhadi., A., "YOLO9000: better, faster, stronger," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[26] Redmon J., & Farhadi, A., "YOLOv3: An incremental improvement," in arXiv preprint arXiv:1804.02767, 2018.
[27] Bochkovskiy, A., Wang, C. Y., & Liao, H. Y. M., "YOLOv4: optimal speed and accuracy of object detection," in arXiv preprint arXiv:2004.10934., 2020.
[28] Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Doll´ar, P., "Focal loss for dense object detection," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
[29] Girshick, R., Donahue, J., Darrell, T. & Malik, J., "Rich feature hierarchies for accurate object detection and semantic segmentation," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[30] R. Girshick, "Fast R-CNN," in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
[31] Ren, S., He, K., Girshick, R., & Sun, J., "Faster R-CNN: towards real-time object detection with region proposal networks," in Advances in Neural Information Processing Systems (NIPS), 2015.
[32] Dai, J., Li, Y., He, K., & Sun. J., "R-FCN:object detection via region-based fully convolutional networks," in Advances in Neural Information Processing Systems(NIPS), 2016.
[33] Lin, T.-Y., Doll´ar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S., "Feature pyramid networks for object detection," in CVPR, 2017.
[34] Kuok, C. P., Horng, M. H., Liao, Y. M., Chow, N. H., & Sun, Y. N., "An effective and accurate identification system of Mycobacterium tuberculosis using convolution neural networks," Microscopy research and technique, 2019.
[35] F. &. K. V. Yu, "Multi-scale context aggregation by dilated convolutions," arXiv preprint arXiv:1511.07122, 2015.
[36] Woo, S., Park, J., Lee, J. Y., & So Kweon, I., "Cbam: convolutional block attention module," in Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[37] Hariharan, B., Arbel´aez, P., Girshick, R., & Malik. J., "Hypercolumns for object segmentation and fine-grained localization," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[38] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z., "Rethinking the inception architecture for computer vision," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[39] mc.ai, "How to reduce the variance of deep learning models in Keras using model averaging ensembles," 20 12 2018. [Online]. Available: https://mc.ai/how-to-reduce-the-variance-of-deep-learning-models-in-keras-using-model-averaging-ensembles/.
[40] Chang, H. S., Learned-Miller, E., & McCallum, A., "Active bias: training more accurate neural networks by emphasizing high variance samples," in Advances in Neural Information Processing Systems, 2017.
[41] Wang, Y., Peng, T., Duan, J., Zhu, C., Liu, J., Ye, J., & Jin, M., "Pathological image classification based on hard example guided CNN," IEEE Access, 2020.
[42] Luo, Y., Wong, Y., Kankanhalli, M., & Zhao, Q., "G-Softmax: improving intraclass compactness and interclass separability of features," IEEE transactions on neural networks and learning systems, pp. 31(2), 685-699, 2019.
[43] Zhao, Y., Lin, F., Liu, S., Hu, Z., Li, H., & Bai, Y., "Constrained-focal-loss based deep learning for segmentation of spores," IEEE Access, pp. 7, 165029-165038, 2019.
[44] Cui, Y., Jia, M., Lin, T. Y., Song, Y., & Belongie, S., "Class-balanced loss based on effective number of samples," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 2019.
[45] Qu, F., Liu, J., Liu, X., & Jiang, L., "A multi-fault detection method with improved triplet loss based on hard sample mining," IEEE Transactions on Sustainable Energy., 2020.
[46] Wang, K., Zhang, D., Li, Y., Zhang, R., & Lin, L., "Cost-effective active learning for deep image classification," IEEE Transactions on Circuits and Systems for Video Technology, pp. 27(12), 2591-2600, 2016.
[47] Xu, X., Liu, L., Liu, W., Wang, M., & Hu, R., "Person re-identification via active hard sample mining".arXiv preprint arXiv:2004.04912..
[48] Nartey, O. T., Yang, G., Wu, J., & Asare, S. K., "Semi-supervised learning for fine-grained classification with self-training," IEEE Access, pp. 8, 2109-2121, 2019.
[49] Shao, W., Sun, L., & Zhang, D., "Deep active learning for nucleus classification in pathology images," in 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, D.C., 2018.
[50] Liao, Y. M., Kuok, C. P., & Sun, Y. N., "Mycobacterium tuberculosis detection using refined faster R-CNN," in The 30th IPPR Conference on Computer Vision, Graphics, and Image Processing (CVGIP 2017), Nantou, 2017.
[51] 黃靜瑜, 楊美玲, 蔡佩芳, 蔡佩珍, 郭振鵬, 何俊逸, 孫永年, "利用卷積神經網路建構非結核性桿菌顯微影像識別," in 第十四屆華人檢驗醫學學術研討會, 台北市, 2019.
[52] 曾正如, 楊美玲, 黃靜瑜, 蔡佩芳, 蔡佩珍, 郭振鵬, 孫永年, "利用卷積神經網路建構結核桿菌顯微影像識別技術," in 台灣微生物學會第二十二屆第三次會員大會學術研討會, 台北市, 2018.
[53] 行政院衛生署疾病管制局, "行政院衛生署疾病管制局出版的結核菌檢驗手冊".
[54] Cai, Z., & Vasconcelos, N., "Cascade r-cnn: Delving into high quality object detection," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.