根據我國衛生福利部統計,癌症長年居於國人死因之首，於是癌症相關的併發症也是近年研究的重要方向之一，癌症骨轉移即是一種癌症中後期的併發症，當癌細胞轉移至骨頭組織，會開始破壞患者的骨頭組織，後續會造成患者容易骨疼、骨質疏鬆、血鈣濃度升高影響腎功能的種種危害。
於是即早偵測出骨轉移的病灶就十分重要，目前臨床核醫科醫師的診斷方法為參考病人的骨掃描影像，骨掃描影像不像其他CT影像需要高輻射的潛在風險，是最具有經濟效益的方法。臨床上醫師會利用病人以前所拍攝的骨掃描影像做為參考，以此來確定是癌症骨轉移而非病人骨折或是人為損傷和感染的良性病灶。
過去相關於骨轉移在深度學習的研究大多沒有考慮到這一項醫師的工作流程，基於骨掃描影像非特異性的問題，本研究提出一種融合不同時期病人拍攝的骨掃描影像，參考其中病人在空間上與間格時間隱含的資訊，並可以使用在類似需要相近圖像資訊的任務上的空間與時序關係注意力模組，讓物件偵測模型可以從病人先前拍攝的影像中獲得應該有的資訊，另一方面，我們設計了一項額外的相似感知損失函數，來解決當兩張圖片差距多大的時候，資訊無法被好好利用的問題。
本研究在成大醫院的資料上進行了許多實驗。首先我們在分類任務上確信了使用病人先前骨掃描影像可以幫助提升模型效能。接著在病灶偵測的實驗中，我們在不同物件偵測模型上驗證了我們提出的空間與時序注意力模組對於框出病灶與辨別的能力是有顯著提升的。最後我們融合了前人使用過的方法，確保我們的提出的方法是有效且可以相輔相成的，並使模型最後達到可以輔助核醫科醫師標註病人病灶的成果。

Cancer has consistently ranked as the leading cause of death worldwide. Consequently, research focusing on cancer-related complications has become a vital area of investigation in recent years. One such complication is bone metastasis in cancer, which occurs in the later stages of cancer. When cancer cells spread to the bone tissue, they disrupt the patient’s bone structure, resulting in various adverse effects, including bone pain, osteoporosis, and elevated blood calcium levels, which can negatively impact kidney function.
Early detection of bone metastasis is of paramount importance. Currently, clinical nuclear medicine physicians rely on bone scintigraphy images for diagnosis. Unlike other imaging methods that carry potential risks associated with high radiation exposure, bone scintigraphy images provide a cost-effective approach. However, they have inherent limitations. The imaging principle relies on the uptake of radioactive isotopes by cells, enabling only the observation of regions with active osteoblast accumulation. Consequently, these scans do not definitively indicate whether the observed condition is cancer bone metastasis or a benign lesion caused by fractures, injuries, or infections. In such cases, physicians refer to the patient’s previous bone scintigraphy images to make a differential diagnosis.
Previous research on deep learning for bone metastasis has often overlooked the clinical workflow. Given the non-specific nature of bone scintigraphy images, this study proposes a framework that integrates information from bone scintigraphy images taken at different time points. By incorporating the spatial and temporal relationships through a Spatial-Temporal Attention (STA) module, which is applicable to tasks that necessitate similar image information from the patient's previous images. Additionally, a novel loss function, called Similarity-Aware Loss, is designed to enhance attention to image pairs with considerable disparities, thereby addressing the challenge of effectively utilizing information when there is a substantial time difference or interval between two images.
In our study, we conducted numerous experiments using data from a major hospital. Firstly, we confirmed the effectiveness of utilizing a patient's previous bone scan images to improve model performance in classification tasks. Next, in lesion detection experiments, we validated the significant improvements in lesion localization and discrimination capabilities achieved by our proposed Spatial-Temporal Attention (STA) Module in various object detection frameworks. Finally, we integrated our proposed approach with existing methods to ensure effectiveness and synergy, ultimately creating a model that can assist nuclear medicine physicians in annotating patient lesions.

Cheng, D.-C., Hsieh, T.-C., Yen, K.-Y., & Kao, C.-H. (2021). Lesion-based bone metastasis detection in chest bone scintigraphy images of prostate cancer patients using pre-train, negative mining, and deep learning. Diagnostics, 11(3), 518.
Cheng, D.-C., Liu, C.-C., Hsieh, T.-C., Yen, K.-Y., & Kao, C.-H. (2021). Bone metastasis detection in the chest and pelvis from a whole-body bone scan using deep learning and a small dataset. Electronics, 10(10), 1201.
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017). Deformable convolutional networks. Proceedings of the IEEE international conference on computer vision,
De Vos, B. D., Berendsen, F. F., Viergever, M. A., Staring, M., & Išgum, I. (2017). End-to-end unsupervised deformable image registration with a convolutional neural network. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3,
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., & Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
Eldele, E., et al., Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112, 2021.
Feichtenhofer, C., Pinz, A., & Zisserman, A. (2017). Detect to track and track to detect. Proceedings of the IEEE international conference on computer vision,
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition,
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition,
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.
Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. Proceedings of the IEEE conference on computer vision and pattern recognition,
Huang, Z., Pu, X., Tang, G., Ping, M., Jiang, G., Wang, M., Wei, X., & Ren, Y. (2022). BS-80K: The first large open-access dataset of bone scan images. Computers in Biology and Medicine, 151, 106221.
Lin, Q., Cao, C., Li, T., Cao, Y., Man, Z., & Wang, H. (2021). Multiclass classification of whole‐body scintigraphic images using a self‐defined convolutional neural network with attention modules. Medical Physics, 48(10), 5782-5793.
Lin, Q., Li, T., Cao, C., Cao, Y., Man, Z., & Wang, H. (2021). Deep learning based automated diagnosis of bone metastases with SPECT thoracic bone images. Scientific Reports, 11(1), 4223.
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition,
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision,
Liu, Y., Yang, P., Pi, Y., Jiang, L., Zhong, X., Cheng, J., Xiang, Y., Wei, J., Li, L., & Yi, Z. (2021). Automatic identification of suspicious bone metastatic lesions in bone scintigraphy using convolutional neural network. BMC medical imaging, 21(1), 1-9.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF international conference on computer vision,
Luo, C., Yang, W., Huang, P., & Zhou, J. (2019). Overview of image matching based on ORB algorithm. Journal of Physics: Conference Series,
Papandrianos, N., Papageorgiou, E. I., & Anagnostis, A. (2020). Development of convolutional neural networks to identify bone metastasis for prostate cancer patients in bone scintigraphy. Annals of Nuclear Medicine, 34, 824-832.
Pi, Y., Zhao, Z., Xiang, Y., Li, Y., Cai, H., & Yi, Z. (2020). Automated diagnosis of bone metastasis based on multi-view bone scans using attention-augmented deep neural networks. Medical Image Analysis, 65, 101784.
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 779–788.
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.
Tan, Q., Ye, M., Yang, B., Liu, S., Ma, A. J., Yip, T. C.-F., Wong, G. L.-H., & Yuen, P. (2020). Data-gru: Dual-attention time-aware gated recurrent unit for irregular multivariate time series. Proceedings of the AAAI Conference on Artificial Intelligence,
Tian, Z., Shen, C., Chen, H., & He, T. (2019). Fcos: Fully convolutional one-stage object detection. Proceedings of the IEEE/CVF international conference on computer vision,
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2023). YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017). Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE conference on computer vision and pattern recognition,
Yang, J., Liu, S., Li, Z., Li, X., & Sun, J. (2022). Real-time object detection for streaming perception. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
Zhao, Z., Pi, Y., Jiang, L., Xiang, Y., Wei, J., Yang, P., Zhang, W., Zhong, X., Zhou, K., & Li, Y. (2020). Deep neural network based artificial intelligence assisted diagnosis of bone scintigraphy for cancer bone metastasis. Scientific Reports, 10(1), 17046.
Zhou, Y., Yu, H., & Shi, H. (2021). Study group learning: Improving retinal vessel segmentation trained with noisy labels. Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24,