近年來，深度學習技術在醫學影像分析上大放異彩，例如：卷積神經網路(convolutional neural network)。然而，當一個類神經網路持續學習新任務且不能忘記先前學過的事情時，它的性能會急遽的下滑。有一些方法被提出來解決深度學習模型劇烈忘記(catastrophic forgetting)的問題，但大部分都是處理影像分類問題。相較之下，先前的著作很少討論終身學習(continual learning)應用在語意分割(semantic segmentation)。除此之外，極少或是沒有論文去探討醫學影像分割的終生學習問題。在這篇論文中，我們為終身學習應用在醫學影像切割上的問題建立了一個可靠的架構，透過提出的新策略來訓練attention masks和自動正規化(regularization)類神經網路的參數空間可有效的減輕深度學習模型劇烈忘記的問題。具體來說，attention masks可以透過梯度下降同時和任務一起學習，並保留了先前學到的知識。除此之外，正規化的機制為未來要學習的任務保留了足夠的類神經網路特徵空間。我們的方法是端到端(end-to-end)的訓練並可以應用在任何的前饋(feed-forward)類神經網路。我們利用了四個公開的資料集來驗證這一個架構，包含了腰椎(xVertSeg challenge of MICCAI'16)的電腦斷層(Computed Tomography)影像以及腰椎、心臟(right ventricle segmentation challenge of MICCAI'12)和大腦(multimodal brain tumor segmentation challenge of MICCAI'19)的磁振造影(Magnetic Resonance Imaging)影像。此外，我們也探討了不同任務在特徵空間參數共享的情形以及在類神經網路的參數空間有限的情況下會對效能造成的影響。我們的終身學習模型得到的Dice相似係數只比傳統上每件任務各別訓練一個模型的結果低0.3% ~ 6%。實驗的結果證明了我們的終生學習於多模態影像的多種器官切割的方法是有效的，我們預期這技術上的進展會對臨床上的應用非常有價值。

Recent progress in deep learning techniques, such as convolutional neural network, shows promising results in medical image analysis. However, performance of a neural network for previously learned tasks drops dramatically while continually learning new tasks even if not totally forgetting them. Several approaches are proposed to address catastrophic forgetting of deep learning models, mostly for image classification. In contrast, continual learning for semantic segmentation is rarely studied in previous works. In addition, little or no studies investigate into continual learning of medical image segmentation. In this study, we establish a reliable framework for continual learning of medical image segmentation. The proposed new training strategy for attention masks and the automatic regularization for parameter space in a neural network successfully alleviate catastrophic forgetting problems of deep learning models. Specifically, attention masks preserve information for previously learned tasks and can be learned concurrently for each task through gradient descent optimization. In addition, the regularization mechanism preserves sufficient capacity in feature space of a model for future tasks. Our approach is trained end-to-end and can be adapted to any feed-forward neural network.We evaluate our framework on four public medical image datasets including CT images of lumbar vertebrae (xVertSeg challenge of MICCAI'16) and MR images of lumbar vertebrae, heart (right ventricle segmentation challenge of MICCAI'12) and brain (multimodal brain tumor segmentation challenge of MICCAI'19). Moreover, we investigate weight-sharing in feature space among tasks and, also, how reduced capacity in a network potentially impact performance of learning multiple tasks. Our method achieves Dice metric only 0.3% ~ 6% lower than that of individually trained models. The results demonstrate effectiveness of our approach in continual learning of multi-organ segmentation from multi-modality images, which is valuable in clinical applications.

1. French, R.M., Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1999. 3(4): p. 128-135.
2. McCloskey, M. and N.J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem, in Psychology of learning and motivation. 1989, Elsevier. p. 109-165.
3. Denil, M., et al., Predicting parameters in deep learning. arXiv preprint arXiv:1306.0543, 2013.
4. Cermelli, F., et al. Modeling the background for incremental learning in semantic segmentation. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
5. Michieli, U. and P. Zanuttigh. Incremental learning techniques for semantic segmentation. in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2019.
6. Kirkpatrick, J., et al., Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017. 114(13): p. 3521-3526.
7. Li, Z. and D. Hoiem, Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 2017. 40(12): p. 2935-2947.
8. Lopez-Paz, D. and M.A. Ranzato, Gradient episodic memory for continual learning. arXiv preprint arXiv:1706.08840, 2017.
9. Serra, J., et al., Overcoming Catastrophic Forgetting with Hard Attention to the Task, in Proceedings of the 35th International Conference on Machine Learning, D. Jennifer and K. Andreas, Editors. 2018, PMLR: Proceedings of Machine Learning Research. p. 4548--4557.
10. Shin, H., et al., Continual learning with deep generative replay. arXiv preprint arXiv:1705.08690, 2017.
11. Yoon, J., et al., Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547, 2017.
12. Zenke, F., B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. in International Conference on Machine Learning. 2017. PMLR.
13. Ke, Z., B. Liu, and X. Huang, Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks. Advances in Neural Information Processing Systems, 2020. 33.
14. Ke, Z., et al. Continual Learning with Knowledge Transfer for Sentiment Classification. in Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases. 2020.
15. Petitjean, C., et al., Right ventricle segmentation from cardiac MRI: a collation study. Medical image analysis, 2015. 19(1): p. 187-202.
16. Bakas, S., et al., Segmentation Labels for the Pre-operative Scans of the TCGA-GBM collection. 2017.
17. Bakas, S., et al., Segmentation Labels for the Pre-operative Scans of the TCGA-LGG collection. 2017.
18. Bakas, S., et al., Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific data, 2017. 4(1): p. 1-13.
19. Bakas, S., et al., Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint arXiv:1811.02629, 2018.
20. Menze, B.H., et al., The multimodal brain tumor image segmentation benchmark (BRATS). IEEE transactions on medical imaging, 2014. 34(10): p. 1993-2024.
21. Chen, L.-C., et al., Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 2017. 40(4): p. 834-848.
22. Jégou, S., et al. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. in Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017.
23. Zhao, H., et al. Pyramid scene parsing network. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
24. Long, J., E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
25. Ronneberger, O., P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. in International Conference on Medical image computing and computer-assisted intervention. 2015. Springer.
26. Sekuboyina, A., et al., A localisation-segmentation approach for multi-label annotation of lumbar vertebrae using deep nets. arXiv preprint arXiv:1703.04347, 2017.
27. Dong, H., et al. Automatic brain tumor detection and segmentation using u-net based fully convolutional networks. in annual conference on medical image understanding and analysis. 2017. Springer.
28. Jha, D., et al. Doubleu-net: A deep convolutional neural network for medical image segmentation. in 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS). 2020. IEEE.
29. Zhou, Z., et al., Unet++: A nested u-net architecture for medical image segmentation, in Deep learning in medical image analysis and multimodal learning for clinical decision support. 2018, Springer. p. 3-11.
30. Zhou, Z., et al., Unet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE transactions on medical imaging, 2019. 39(6): p. 1856-1867.
31. Ibtehaz, N. and M.S. Rahman, MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Networks, 2020. 121: p. 74-87.
32. Parisi, G.I., et al., Continual lifelong learning with neural networks: A review. Neural Networks, 2019. 113: p. 54-71.
33. Zhou, G., K. Sohn, and H. Lee. Online incremental feature learning with denoising autoencoders. in Artificial intelligence and statistics. 2012. PMLR.
34. O'Reilly, R.C. and K.A. Norman, Hippocampal and neocortical contributions to memory: Advances in the complementary learning systems framework. Trends in cognitive sciences, 2002. 6(12): p. 505-510.
35. McClelland, J.L., B.L. McNaughton, and R.C. O'Reilly, Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 1995. 102(3): p. 419.
36. Baweja, C., B. Glocker, and K. Kamnitsas, Towards continual learning in medical imaging. arXiv preprint arXiv:1811.02496, 2018.
37. Ma, J., et al., AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem? arXiv preprint arXiv:2010.14808, 2020.
38. Zhou, X., et al. Performance evaluation of 2D and 3D deep learning approaches for automatic segmentation of multiple organs on CT images. in Medical Imaging 2018: Computer-Aided Diagnosis. 2018. International Society for Optics and Photonics.
39. van Rooij, W., et al., Deep learning-based delineation of head and neck organs at risk: geometric and dosimetric evaluation. International Journal of Radiation Oncology Biology Physics, 2019. 104(3): p. 677-684.
40. Xu, X., et al., Efficient multiple organ localization in ct image using 3d region proposal network. IEEE transactions on medical imaging, 2019. 38(8): p. 1885-1898.
41. Hsu, H.-C., Right Ventricle Segmentation from Cine Cardiac MRI Using Deep Learning, in Institute of Computer Science and Information Engineering. 2019, National Cheng Kung University.
42. Soltaninejad, M., et al., Automated brain tumour detection and segmentation using superpixel-based extremely randomized trees in FLAIR MRI. International journal of computer assisted radiology and surgery, 2017. 12(2): p. 183-203.
43. Chu, C., et al., Fully automatic localization and segmentation of 3D vertebral bodies from CT/MR images via a learning-based method. PloS one, 2015. 10(11): p. e0143327.
44. Sled, J.G., A.P. Zijdenbos, and A.C. Evans, A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE transactions on medical imaging, 1998. 17(1): p. 87-97.
45. Tustison, N.J., et al., N4ITK: improved N3 bias correction. IEEE transactions on medical imaging, 2010. 29(6): p. 1310-1320.
46. Bakker, B. and T. Heskes, Task clustering and gating for bayesian multitask learning. 2003.
47. Kingma, D.P. and J. Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
48. Loshchilov, I. and F. Hutter, Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
49. Goyal, P., et al., Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
50. Paszke, A., et al., Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.