針灸與穴位按摩是傳統中醫常用的療法，根據病患不同症狀針對相對應的穴位做針灸或穴位按摩來緩解病患症狀。由於穴位點的數量龐大，並且穴位點有多樣化、複雜性與專一性，除非經過專業訓練，人們很難記得每個穴位點的位置以及對應的治療用途。而生理結構的限制，人們也難以自行對於背部穴位進行按摩。我們提出一套系統能夠透過相機拍攝人體背部照片後交由穴位預測演算法自動預測穴道位置，並利用深度相機取得穴道點在空間中的位置，並將其座標轉換給機械手臂，使機械手臂能自動定位到穴位點。
透過學習傳統中醫歸納的穴位尋找方法，我們的提出新的背部穴位預測方法，利用背部特徵標記點和脊椎平均模型適應到不同的體型找出脊椎骨位置，在藉由我們建立的脊椎—穴位參考模型來推算穴道位置。系統也結合uArm Swift Pro機械手臂，以展示自動穴道定位應用。

Acupuncture therapy is one of the main forms of treatment in Traditional Chinese Medicine (TCM). Based on different patient symptoms, needling or massage is applied to the corresponding acupuncture points to relieve symptoms. However, given the large number of acupoints and the complexity of their specificity, it is difficult for one to remember the location and function of each acupoint without professional training. Due to the limitation of physiological structures, it is difficult for people to massage the acupoints on the back by themselves. In this work, we propose a system that can estimate acupoint location using a back image, where the 3D coordinates of an acupoints can be retrieved in the real world using a depth camera, and the coordinates can be transformed into the robot arm space so that the robot arm can automatically localize the acupoint.
In our work, through the acupoint localization method summarized in TCM, we propose an approach for back acupoint estimation by leveraging a mean back model consisting of landmark points and spine locations. We build a spine-acupoint relation model that records the relative distance between an acupoint and its reference vertebra. Also, the uArm Swift Pro is incorporated into the system to automatically localize acupoints on the back.

[1] C. Y. Wu, “3D Ashi Point Localization of Back on the Vision-based Massage Machine,” Master’s dissertation, National Cheng Kung University, 2010
[2] S. C. Lin, “Arm Design and Control of A Mechanical Massage System, Master’s dissertation,” National Cheng Kung University, 2012
[3] L. Hu, Y. Wang, J. Zhang, Y. Cui, L. Ma, J. Jiang, L. Fang, and B. Zhang, “A massage robot based on Chinese massage therapy”, Industrial Robot, Vol. 40 No. 2, pp. 158-172. 2013
[4] H. Gao, S. Lu, T. Guo, J. Tan. “Vision-integrated physiotherapy service robot using cooperating two arms.” International Journal on Smart Sensing and Intelligent Systems. 2014
[5] W. Si, G. Srivastava, Y. Zhang and L. Jiang, “Green Internet of Things Application of a Medical Massage Robot With System Interruption,” in IEEE Access, vol. 7, pp. 127066-127077, 2019
[6] Y. Liu, P. Zhou, B. Ma, D. Deng and W. Wang, “The Improvement in Design and Control of a Massage Robot on Human Back,” 2019 IEEE 2nd International Conference on Automation, Electronics and Electrical Engineering (AUTEEE), 2019
[7] F. Eyssautier, https://capsix-robotics.com/iyu-pro/
[8] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014
[9] S. Ren, K. He, R. Girshick, and J. Sun. “Faster R-CNN: Towards real-time object detection with region proposal networks.” In NIPS, 2015.
[10] J. Dai, Y. Li, K. He, and J. Sun. “R-FCN: Object detection via region-based fully convolutional networks.” In NIPS, 2016.
[11] K. He, G. Gkioxari, P. Dollar, and R. Girshick. “Mask r-cnn.” arXiv:1703.06870, 2017.
[12] A. Bochkovskiy, C. Y. Wang, and H. Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” arXiv:2004.10934, 2020.
[13] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. Reed, “SSD: Single shot multibox detector,” arXiv:1512.02325, 2015.
[14] Y. Wu and Q. Ji, “Facial landmark detection: A literature survey,” IJCV, 2017.
[15] J. X. Zhang, “Localization of Foot Acupoints on a Smartphone using Augmented Reality”, Unpublished master’s dissertation, National Cheng Kung University, 2020
[16] Z. Zhang, P. Luo, C. Loy, and X. Tang. “Facial landmark detection by deep multi-task learning.” In European Conference on Computer Vision, 2014
[17] M. Dantone, J. Gall, G. Fanelli, and L. V. Gool. “Real-time facial feature detection using conditional regression forests.” In IEEE conference on computer vision and pattern recognition. 2012
[18] H. Yang, and I. Patras. “Privileged information-based conditional regression forest for facial feature detection.” In IEEE international conference and workshops on automatic face and gesture recognition. 2013
[19] X. Cao, Y. Wei, F. Wen, and J. Sun. “Face alignment by explicit shape regression.” International Journal of Computer Vision, 2014
[20] S. Ren, X. Cao, Y. Wei, and J. Sun. “Face alignment at 3000 FPS via regressing local binary features.” In IEEE conference on computer vision and pattern recognition (CVPR), 2014
[21] V. Kazemi, and J. Sullivan. “One millisecond face alignment with an ensemble of regression trees.” In IEEE conference on computer vision and pattern recognition (CVPR). 2014
[22] M. Sundaram, G. Nayagam. “A survey on Real time Object Detection and Tracking Algorithms.” International Journal of Applied Engineering Research. 2015
[23] J. Athanesious, P. Suresh, “Systematic Survey on Object Tracking Methods in Video,” International Journal of Advanced Research in Computer Engineering & Technology (IJARCET). October 2012
[24] J. Athanesious, P. Suresh. “Implementation and Comparison of Kernel and Silhouette Based Object Tracking.” International Journal of Advanced Research in Computer Engineering & Technology. March 2013
[25] J. Athanesious J, Mr. P. Suresh. “Implementation and Comparison of Kernel and Silhouette Based Object Tracking,” International Journal of Advanced Research in Computer Engineering & Technology. March 2013
[26] A. Balasundaram, S. A. Kumar, S. M. Kumar. “Optical Flow Based Object Movement Tracking,” International Journal of Engineering and Advanced Technology (IJEAT). October 2019
[27] W. Sederberg, and S. R. Thomas “Free-form deformation of solid geometric models”. In Proceedings of ACM SIGGRAPH. 1986
[28] T. BEIER and S. NEELY. “Feature-based image metamorphosis”. In SIGGRAPH: Proceedings of the 19th annual conference on Computer graphics and interactive techniques. 1992
[29] S. Schaefer, T. McPhail and J. Warren. “Image deformation using moving least squares.” In ACM SIGGRAPH. 2006
[30] S. Y. Lee, and K. Y. Chwa. “Image metamorphosis using snakes and free-form deformations”. In: In SIGGRAPH: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 1995.
[31] K. G. KOBAYASHI and K. OOTSUBO. “t-ffd: freeform deformation by using triangular mesh.” In SM: Proceedings of the eighth ACM symposium on Solid modeling and applications, 2003
[32] H. Ling and D. W. Jacobs, “Deformation invariant image matching,” Tenth IEEE International Conference on Computer Vision (ICCV’05), 2005
[33] T. Ma and L. J. Latecki, “From partial shape matching through local deformation to robust global shape similarity for object detection,” In CVPR 2011
[34] D. Sýkora, J. Dingliana, and S. Collins. “As-rigid-as-possible image registration for hand-drawn cartoon animations.” In NPAR, 2009
[35] W. Sederberg, and S. R. Thomas “Free-form deformation of solid geometric models”. In Proceedings of ACM SIGGRAPH, 1986
[36] T. BEIER and S. NEELY. “Feature-based image metamorphosis”. In SIGGRAPH: Proceedings of the 19th annual conference on Computer graphics and interactive techniques, 1992
[37] P. C. Chung and C. Y. Wu. “3D Ashi Point Localization of Back on the Vision-based Massage Machine.” National Cheng Kung University, 2010
[38] H. Jiang, J. Starkman, C. H. Kuo, and M. C. Huang. “Acu Glass: Quantifying Acupuncture Therapy using Google Glass.” Proceedings of the 10th EAI International Conference on Body Area Networks, 2015.
[39] K. C. Lan, M. C. Hu, Y. Z. Chen, and J. X. Zhang “The Application of 3D Morphable Model (3DMM) for Real-time Visualization of Acupoints on a Smartphone,” in IEEE Sensors Journal, 2020
[40] X. M. Yang, Y. Ye, Y. Xia, X. Z. Wei, Z. Y. Wang, H. M. Ni, Y. Zhu, and L. Y. Xu. “A precise and accurate acupoint location obtained on the face using consistency matrix pointwise fusion method.” Journal of Traditional Chinese Medicine, 2015.
[41] G. S. Lee, “Acupressure through the Assistance of a Robot Arm”, Unpublished master’s dissertation, National Cheng Kung University, 2018
[42] J. Kittler, P. Huber, Z.-H. Feng, G. Hu, and W. Christmas. “3D Morphable Face Models and Their Applications.” In International Conference on Articulated Motion and Deformable Objects, 2016.
[43] X. X. Lu. “A Review of Solutions for Perspective-n-Point Problem in Camera Pose Estimation.” J. Phys. 2018
[44] H. Gao, S. Lu, T. Wang, C. Liu, B. Kang, Y. Ji, and H. Bi, “Research and development of Chinese medical massage robot.” Jiqiren/Robot. 2011
[45] Tsai, R., & Lenz, R. “A new technique for fully autonomous and efficient 3D robotics hand/eye calibration.” IEEE Trans. Robotics Autom.1989
[46] COCO dataset. https://cocodataset.org/#home
[47] LabelImg. https://github.com/tzutalin/labelImg
[48] YOLOv4 source code. https://github.com/AlexeyAB/darknet
[49] dlib. http://dlib.net/
[50] Moving-Least-Squares. https://github.com/Jarvis73/Moving-Least-Squares
[51] VOC2010. http://host.robots.ox.ac.uk/pascal/VOC/voc2010/
[52] J. Zhang, K. Gao, K. Fu, P. Cheng, “Deep 3D Facial Landmark Localization on position maps,” Neurocomputing, 2020
[53] F. M. Sukno, J.L. Waddington, P.F. Whelan. “3-D Facial Landmark Localization With Asymmetry Patterns and Shape Regression from Incomplete Local Features.” IEEE Trans Cybern, 2015
[54] P. Perakis, G. Passalis, T. Theoharis and I. A. Kakadiaris, “3D Facial Landmark Detection under Large Yaw and Expression Variations,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
[55] X. Xiong and F. De la Torre, “Supervised Descent Method and Its Applications to Face Alignment,” 2013 IEEE Conference on Computer Vision and Pattern Recognition, 2013
[56] A. Noccaro, L. Raiano, G. Di Pino and D. Formica, “Evaluation of Hand-Eye and Robot-World Calibration Algorithms for TMS Application”, 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob), 2018
[57] S. Kramer, K. Winterhalter, G. Schober, U. Becker, B. Wiegele, D.F. Kutz, F.P. Kolb, D. Zaps, P.M. Lang, D. Irnich. “Characteristics of electrical skin resistance at acupuncture points in healthy humans.” J Altern Complement Med. May, 2009
[58] OpenCV. https://docs.opencv.org/3.3.1/d7/d8b/tutorial_py_lucas_kanade.html
[59] WHO Regional Office for the Western Pacific. “WHO Standard Acupuncture Point Locations in the Western Pacific Region. Manila: World Health Organization”, 2008
[60] Yibian. http://yibian.hopto.org/
[61] Z. Zhang, P. Luo, C. C. Loy, and X. Tang. “Learning deep representation for face alignment with auxiliary attributes.” TPAMI, 2016
[62] G. Trigeorgis, P. Snape, M. A. Nicolaou, E. Antonakos, and S. Zafeiriou. “Mnemonic descent method: A recurrent process applied for end-to-end face alignment.” In CVPR, 2016
[63] J. Lv, X. Shao, J. Xing, C. Cheng, and X. Zhou. “A deep regression architecture with two-stage re-initialization for high performance facial landmark detection.” In CVPR, 2017
[64] A. Bulat and Y. Tzimiropoulos. “Convolutional aggregation of local evidence for large pose face alignment.” In BMVC, 2016
[65] J. Yang, Q. Liu, and K. Zhang. “Stacked hourglass network for robust facial landmark localisation.” In CVPR Workshop, 2017
[66] A. Newell, K. Yang, and J. Deng. “Stacked hourglass networks for human pose estimation.” In ECCV, 2016
[67] J. Deng, G. Trigeorgis, Y. Zhou, and S. Zafeiriou. “Joint multi-view face alignment in the wild.” arXiv preprint, arXiv:1708.06023, 2017
[68] S. Zafeiriou, G. Trigeorgis, G. Chrysos, “J. Deng, and J. Shen. The menpo facial landmark localisation challenge: A step towards the solution.” In CVPR Workshop, 2017
[69] W. Wu, C. Qian, S. Yang, Q. Wang, Y. Cai, and Q. Zhou. “Look at Boundary: A Boundary-Aware Face Alignment Algorithm.” In CVPR, 2018
[70] X. Chu, W. Ouyang, H. Li, and X. Wang. “Structured feature learning for pose estimation.” In CVPR, 2016.
[71] W. Yang, W. Ouyang, H. Li, and X. Wang. “End-to-end learning of deformable mixture of parts and deep convolutional neural networks for human pose estimation.” In CVPR, 2016.
[72] C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic. “300 faces in-the-wild challenge: The first facial landmark localization challenge.” In ICCV Workshop, 2013
[73] X. P. Burgos-Artizzu, P. Perona, and P. Dollar. “Robust face landmark estimation under occlusion.” In ICCV, 2013
[74] X. Cao, Y. Wei, F. Wen, and J. Sun. “Face alignment by explicit shape regression.” IJCV, 2014
[75] J. Lv, X. Shao, J. Xing, C. Cheng, and X. Zhou. “A deep regression architecture with two-stage re-initialization for high performance facial landmark detection.” In CVPR, 2017
[76] S. Ren, X. Cao, Y. Wei, and J. Sun. “Face alignment at 3000 FPS via regressing local binary features.” In CVPR, 2014
[77] W. Wu and S. Yang. “Leveraging intra and inter-dataset variations for robust face alignment.” In CVPR Workshop, 2017
[78] S. Xiao, J. Feng, J. Xing, H. Lai, S. Yan, and A. A. Kassim. “Robust facial landmark detection via recurrent attentive refinement networks.” In ECCV, 2016
[79] X. Xiong and F. D. la Torre. “Supervised descent method and its applications to face alignment.” In CVPR, 2013
[80] J. Zhang, S. Shan, M. Kan, and X. Chen. “Coarse-to-fine auto-encoder networks (cfan) for real-time face alignment.” In ECCV, 2014
[81] S. Zhu, C. Li, C. C. Loy, and X. Tang. “Face alignment by coarse-to-fine shape searching.” In CVPR, 2015
[82] X. Zhu, Z. Lei, X. Liu, H. Shi, and S. Z. Li. “Face alignment across large poses: A 3d solution.” In CVPR, 2016
[83] T. F. Cootes, G. J. Edwards and C. J. Taylor. “Active appearance models.” IEEE Transactions on Pattern Analysis and Machine Intelligence. 2001
[84] X. Zhu and D. Ramanan, “Face detection, pose estimation, and landmark localization in the wild,” 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI. 2012
[85] D. Cristinacce, and T. F. Cootes. “Feature detection and tracking with constrained local models.” In British Machine Vision Conference. 2006.
[86] J. Kim, D. I. Kang. “Positioning Standardized Acupuncture Points on the Whole Body Based on X-Ray Computed Tomography Images.” Medical acupuncture. 2014
[87] T. Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár. “Focal Loss for Dense Object Detection.” In CVPR, 2018
[88] H. Law, J. Deng. “CornerNet: Detecting Objects as Paired Keypoints.” In CVPR, 2019
[89] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, Q. Tian. “CenterNet: Keypoint Triplets for Object Detection.” In CVPR, 2019
[90] Z. Tian, C. Shen, H. Chen, T. He. “FCOS: Fully Convolutional One-Stage Object Detection.” In CVPR, 2019
[91] M. Tan, R. Pang, Q. V. Le. “EfficientDet: Scalable and Efficient Object Detection.” In CVPR, 2020
[92] Q. Zhao, T. Sheng, Y. Wang, Z. Tang, Y. Chen, L. Cai, H. Ling. “M2Det: A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network.” In CVPR, 2018
[93] C. Y. Wang, I. H. Yeh, H.Y. M. Liao. “You Only Learn One Representation: Unified Network for Multiple Tasks.” In CVPR, 2021
[94] I. Lundberg, M. Björkman and P. Ögren, “Intrinsic camera and hand-eye calibration for a robot vision system using a point marker,” 2014 IEEE-RAS International Conference on Humanoid Robots, 2014
[95] M. i. Ribeiro. “Introduction to Kalman Filtering.” 2000
[96] Q. H. Bai. “Analysis of Particle Swarm Optimization Algorithm.” 2010
[97] I. J. Cox, S.L. Hingorani. “An efficient implementation of Reid’s multiple ypothesis tracking algorithm and its evaluation for the purpose of visual tracking.” in IEEE Transactions on Pattern Analysis and Machine Intelligence. Feb. 1996
[98] D. Comaniciu, R. Visvanathan, and M. Peter. “Kernel-based object tracking.” IEEE Transactions on pattern analysis and machine intelligence. 2003
[99] S. Avidan. “Support Vector Tracking.” Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR, 2001
[100] H. Tao, H. S. Sawhney and R. Kumar. “Object tracking with Bayesian estimation of dynamic layer representations,” in IEEE Transactions on Pattern Analysis and Machine Intelligence. 2002
[101] Q. Xiaoping, Z. Qiheng, O. Yimin and M. Jiaguang, “A Method for Object Tracking using Shape Matching,” 2006 IEEE Workshop on Signal Processing Systems Design and Implementation, 2006
[102] R. Patel. “Contour Based Object Tracking.” International Journal of Computer and Electrical Engineering. 2012
[103] V. Spruyt, A. Ledda and W. Philips, “Sparse optical flow regularization for real-time visual tracking,” 2013 IEEE International Conference on Multimedia and Expo (ICME), 2013
[104] J.W. Frymoyer. “Back pain and sciatica.” N Engl J Med. 1988
[105] G. Wadlow, E. Peringer. “Retrospective survey of patients of practitioners of traditional Chinese acupuncture in the UK.”" Complementary therapies in medicine. 1996
[106] D. C. Cherkin, K.J. Sherman, A.L. Avins et al. “A Randomized Trial Comparing Acupuncture, Simulated Acupuncture, and Usual Care for Chronic Low Back Pain.” Arch Intern Med. 2009
[107] H.A.M. Daanen, F.B. Ter Haar. “3D whole body scanners revisited.” Displays. 2013
[108] Imgaug. https://github.com/aleju/imgaug
[109] Imglab. https://github.com/NaturalIntelligence/imglab
[110] PhotoScape X. http://x.photoscape.org/
[111] uArm API. https://github.com/uArm-Developer/uArm-Python-SDK
[112] Numpy. https://numpy.org/
[113] RealSense D415. https://www.intelrealsense.com/compare-depth-cameras/
[114] uArm Swift Pro. https://www.ufactory.cc/pages/uarm
[115] W. C. Chao & M. Y. Wang. “Using the finger cun of acupuncture-point location as benchmark to estimate body dimensions,” Journal of the Chinese Institute of Industrial Engineers, 2010
[116] pyttsx3. https://pypi.org/project/pyttsx3/
[117] SpeechRecognition. https://pypi.org/project/SpeechRecognition/
[118] xARM. https://shop.qts.tw/products/xarm-5-lite-協作型多軸機械手臂
[119] L. N. Leng, H. J. Ma, & D. W. Si. “A morphometric study of the thoracolumbar spine spinous process and lamina space in the Chinese.” Folia morphologica, 80(3), 665–674. 2021
[120] S. Y. Luo, “Adaptive Trajectory Generation via Visual and Voice Feedback for Automated Robotic Therapeutic Massage Applications.” Unpublished master’s dissertation, National Taiwan University, 2018
[121] X. Z. Ma et al. “Rethinking Pseudo-LiDAR Representation,” Computer Vision and Pattern Recognition (CVPR), 2020
[122] RealSense API. https://dev.intelrealsense.com/docs/projection-in-intel-realsense-sdk-20