中醫針灸與穴位按摩是中醫常用的療法，根據病患不同症狀可以針對對應的穴位做針灸或穴位按摩來緩解病患症狀。由於有龐大的穴位點的數量，並且穴位點有多樣化、複雜性與專一性，除非經由一段時間的專業訓練，常人很難記得每個穴位點的位置以及對應的治療用途。我們系統藉由擴增實境，穴位點會顯示在人體的輸入影像上，相比傳統使用穴位點量測電阻值，我們的系統透過軟體的方式，使用已有的智慧型手機，不需要額外的硬體支出。在輕微症狀如踝關節痛、足底痛，透過我們的系統能幫助病人快速定位穴位點用於按摩達到緩解症狀，而不需中醫專家的幫助。
我們的提出新的穴位預估系統以足部為範例，利用足部特徵標記點和3D模型達到可以適應不同足型與足部角度。系統也被實作在Android 平台上，以展示真實的穴位定位應用。

Acupoint therapy is one of the main modalities of treatment in Traditional Chinese Medicine (TCM). Based on different patient symptoms, needling or massage is applied to the corresponding acupuncture points to relieve the symptoms. However, given the large number of acupoints and the complexity of their specificity, it is difficult for one to remember the location and function of each acupoint without extensive training. In this work, through the use of augmented reality (AR), the acupuncture points can be displayed directly on the image of the human body. Compared to existing acupoint probe devices that work by measuring skin conductivity, our solution does not require any additional hardware and is purely software-based. In this paper, we propose an approach for foot acupoint localization by leveraging the landmark points utilizing a 3D model. In the case of mild symptoms (e.g. ankle pain, plantar pain), with the aid of our proposed system, the patient can quickly locate the corresponding acupuncture points for the application of massage to relieve his/her symptoms without the help of TCM physicians.

[1] Statista: https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/
[2] Wiki: https://en.wikipedia.org/wiki/List_of_countries_by_smartphone_penetration
[3] Tensorflow: https://www.tensorflow.org/lite/models/object_detection/overview
[4] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” arXiv preprint arXiv:2004.10934, 2020.
[5] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. Reed, “SSD: Single shot multibox detector,” arXiv:1512.02325, 2015.
[6] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. CoRR, abs/1804.02767, 2018.
[7] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861, 2017.
[8] Y. Wu and Q. Ji, “Facial landmark detection: A literature survey,” IJCV, pp. 1–28, 2017.
[9] S. A. K. Tareen and Z. Saleem, "A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK," 2018 International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), Sukkur, 2018, pp. 1-10, doi: 10.1109/ICOMET.2018.8346440.
[10] Lowe, D.: Distinctive image features from scale-invariant keypoints, cascade filtering approach. IJCV 60, 91–110 (2004)
[11] Bay H., Tuytelaars T., Van Gool L. (2006) SURF: Speeded Up Robust Features. In: Leonardis A., Bischof H., Pinz A. (eds) Computer Vision – ECCV 2006. ECCV 2006. Lecture Notes in Computer Science, vol 3951. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11744023_32
[12] Alcantarilla P.F., Bartoli A., Davison A.J. (2012) KAZE Features. In: Fitzgibbon A., Lazebnik S., Perona P., Sato Y., Schmid C. (eds) Computer Vision – ECCV 2012. ECCV 2012. Lecture Notes in Computer Science, vol 7577. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33783-3_16
[13] P. F. Alcantarilla et al., "Fast explicit diffusion for accelerated features in nonlinear scale spaces", British Machine Vision Conference, 2013.
[14] E. Rublee et al., "ORB: An efficient alternative to SIFT or SURF", IEEE International Conference on Computer Vision, pp. 2564-2571, 2011.
[15] S. Leutenegger et al., "BRISK: Binary robust invariant scalable keypoints", IEEE International Conference on Computer Vision, pp. 2548-2555, 2011.
[16] Wiki: https://en.wikipedia.org/wiki/3D_projection
[17] Xiao Xin Lu 2018 J. Phys.: Conf. Ser. 1087 052009
[18] W. SEDERBERG, PARRY, and S. R. “Free-form deformation of solid geometric models”. In: In Proceedings of ACM SIGGRAPH 1986, ACM Press (2019), pp. 151–160.
[19] T. BEIER and S. NEELY. “Feature-based image metamorphosis”. In: In SIGGRAPH ’92: Proceedings of the 19th annual conference on Computer graphics and interactive techniques, ACM Press (), pp. 35–42.
[20] Schaefer S, McPhail T and Warren J 2006 Image deformation using moving least squares ACM SIGGRAPH 2006 Papers.
[21] Pau-Choo Chung and Chia-YuWu. 3D Ashi Point Localization of Back on the Vision-based Massage Machine.
[22] Haotian Jiang, James Starkman, Chia-Hung Kuo, and Ming-Chun Huang. Acu Glass: Quantifying Acupuncture Therapy using Google Glass. Proceedings of the 10th EAI International Conference on Body Area Networks, pages 7–10, 2015.
[23] K. -C. Lan, M. -C. Hu, Y. -Z. Chen, J. -X. Zhang and J. -X. Zhang, "The Application of 3D Morphable Model (3DMM) for Real-time Visualization of Acupoints on a Smartphone," in IEEE Sensors Journal, doi: 10.1109/JSEN.2020.3022958.
[24] J. Kittler, P. Huber, Z.-H. Feng, G. Hu, and W. Christmas. 3D Morphable Face Models and Their Applications. In International Conference on Articulated Motion and Deformable Objects, pages 185–206. Springer, 2016.
[25] N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), San Diego, CA, USA, 2005, pp. 886-893 vol. 1, doi: 10.1109/CVPR.2005.177.
[26] Fischler, M. and Bolles, R. 1981. Random sample consensus: A paradigm for model fitting with application to image analysis and automated cartography. Commun. Assoc. Comp. Mach., 24:381-395.
[27] M. Norouzi, D. M. Blei, and R. R. Salakhutdinov. Hamming distance metric learning. In NIPS 25, 2012.
[28] DeTone, D.; Malisiewicz, T.; and Rabinovich, A. 2016. Deep image homography estimation. arXiv preprint arXiv:1606.03798.
[29] Sense 3D: https://www.3dsystems.com/applications/3d-scanning
[30] Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2879–2886
[31] Tzimiropoulos, G., Pantic, M.: Optimization problems for fast aam fitting in-the-wild. In: IEEE International conference on Computer Vision, pp. 593–600
[32] Belhumeur, P., Jacobs, D., Kriegman, D., Kumar, N.: Localizing parts of faces using a consensus of exemplars. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(12), 2930–2940
[33] Wu, Y., Ji, Q.: Discriminative deep face shape model for facial point detection. International Journal of Computer Vision 113(1), 37–53
[34] Xiong, X., De la Torre Frade, F.: Supervised descent method and its applications to face alignment. In: IEEE International Conference on Computer Vision and Pattern Recognition
[35] X.X. Lu A review of solutions for perspective-n-point problem in camera pose estimation Journal of Physics: Conference Series, 1087, IOP Publishing (2018), p. 052009
[36] Z.-Q. Zhao, P. Zheng, S.-T. Xu, and X. Wu, ‘‘Object detection with deep learning: A review,’’ IEEE Trans. Neural Netw. Learn. Syst., vol. 30,no. 11, pp. 3212–3232, Nov. 2019.
[37] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[38] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
[39] J. Dai, Y. Li, K. He, and J. Sun. R-FCN: Object detection via region-based fully convolutional networks. In NIPS, 2016.
[40] K. He, G. Gkioxari, P. Dollar, and R. Girshick. Mask r-cnn. arXiv:1703.06870, 2017.
[41] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
[42] Zicong J., Liquan Z., Shuaiyang L., Yanfei J. (2020). Real-time object detection method based on improved YOLOv4-tiny. arXiv:2011.04244v2.
[43] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, and A. Rabinovich, “Going deeper with convolutions,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[44] Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu. Object Detection with Deep Learning: A Review. arXiv e-prints, page arXiv:1807.05511, Jul 2018.
[45] Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001). Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6), 681–685.
[46] Zhang, Z., Luo, P., Loy, C., & Tang, X. (2014). Facial landmark detection by deep multi-task learning. In European Conference on Computer Vision, Part II, pp. 94–108.
[47] Android App for Real-time Face Landmark Detection: https://github.com/Adityasiwan007/facial-landmarks-recognisation
[48] S. Grewenig, J. Weickert, C. Schroers, and A. Bruhn. Cyclic Schemes for PDE-Based Image Analysis. Technical Report 327, Department of Mathematics, Saarland University, Saarbrücken, Germany, March 2013.
[49] S. Grewening, J. Weickert, and A. Bruhn. From box filtering to fast explicit diffusion. In Proceedings of the DAGM Symposium on Pattern Recognition, pages 533–542, 2010.
[50] Kun-Chan Lan, Min-Chun Hu and Guan-Sheng Lee. Acupressure through the Assistance of a Robot Arm. NCKU. 2018.
[51] Wewow photography. https://24h.pchome.com.tw/prod/QAAR0J-A90092BEA
[52] Free Video to JPG Converter. https://www.dvdvideosoft.com/products/dvd/Free-Video-to-JPG-Converter.htm
[53] OpenCV. https://opencv.org/
[54] X. Zhu and D. Ramanan, "Face detection, pose estimation, and landmark localization in the wild," 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, 2012, pp. 2879-2886, doi: 10.1109/CVPR.2012.6248014.
[55] Dantone,M., Gall, J., Fanelli, G.,&Gool, L. V. (2012). Real-time facial feature detection using conditional regression forests. In IEEE conference on computer vision and pattern recognition.
[56] Yang, H., & Patras, I. (2013). Privileged information-based conditional regression forest for facial feature detection. In IEEE international conference and workshops on automatic face and gesture recognition, pp. 1–6.
[57] ORB (Oriented FAST and Rotated BRIEF). https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_orb/py_orb.html
[58] Cao, X., Wei, Y., Wen, F., & Sun, J. (2014). Face alignment by explicit shape regression. International Journal of Computer Vision, 107, 177–190.
[59] Ren, S., Cao, X., Wei, Y., & Sun, J. (2014). Face alignment at 3000 FPS via regressing local binary features. In IEEE conference on computer vision and pattern recognition (CVPR), pp. 1685–1692.
[60] Kazemi, V., & Sullivan, J. (2014). One millisecond face alignment with an ensemble of regression trees. In IEEE conference on computer vision and pattern recognition (CVPR), pp. 1867–1874.
[61] Sun, Y., Wang, X., & Tang, X. (2013). Deep convolutional network cascade for facial point detection. In IEEE conference on computer vision and pattern recognition, pp. 3476–3483.
[62] Zhang, Z., Luo, P., Loy, C. C., & Tang, X. (2016). Learning deep representation for face alignment with auxiliary attributes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(5), 918–930.
[63] Ranjan, R., Patel, V. M., & Chellappa, R. (2016). Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. CoRR arXiv:1603.01249.
[64] D. Eggert and K. Bowyer, "Perspective projection aspect graphs of solids of revolution: an implementation," [1991 Proceedings] Workshop on Directions in Automated CAD-Based Vision, Maui, HI, USA, 1991, pp. 44-53, doi: 10.1109/CADVIS.1991.148756.
[65] C. Chang, M. Hu, W. Cheng and Y. Chuang, "Rectangling Stereographic Projection for Wide-Angle Image Visualization," 2013 IEEE International Conference on Computer Vision, Sydney, NSW, 2013, pp. 2824-2831, doi: 10.1109/ICCV.2013.351.
[66] Yu, M., Liu, L., & Shao, L. (2015). Kernelized multiview projection. arXiv:1508.00430.
[67] H. M. Choi, H. Kang and Y. Hyun, "Multi-View Reprojection Architecture for Orientation Estimation," 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea (South), 2019, pp. 2357-2366, doi: 10.1109/ICCVW.2019.00289.
[68] L. Xiaowu and X. Qian, "The axonometric projection in mechanical drawing," 2011 International Conference on Consumer Electronics, Communications and Networks (CECNet), Xianning, China, 2011, pp. 816-820, doi: 10.1109/CECNET.2011.5768559.
[69] L. Xiaowu and Y. Changzhi, "The axonometric projection in teaching practice," 2011 International Conference on Consumer Electronics, Communications and Networks (CECNet), Xianning, China, 2011, pp. 2715-2719, doi: 10.1109/CECNET.2011.5768521.
[70] R. T. Behrens and L. L. Scharf, "Signal processing applications of oblique projection operators," in IEEE Transactions on Signal Processing, vol. 42, no. 6, pp. 1413-1424, June 1994, doi: 10.1109/78.286957.
[71] Wirawan, K. A. Meraim, H. Maitre and P. Duhamel, "Blind multichannel image restoration using oblique projections," Sensor Array and Multichannel Signal Processing Workshop Proceedings, 2002, Rosslyn, VA,USA, 2002, pp. 125-129, doi: 10.1109/SAM.2002.1191013.
[72] CHWA LEE S.-Y. and K.-Y. “Image metamorphosis using snakes and free-form deformations”. In: In SIGGRAPH ’95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, ACM Press (), pp. 439–448.
[73] K. G. KOBAYASHI and K. OOTSUBO. “t-ffd: freeform deformation by using triangular mesh”. In: In SM ’03: Proceedings of the eighth ACM symposium on Solid modeling and applications, ACM Press (),pp. 226–234.
[74] Haibin Ling and D. W. Jacobs, "Deformation invariant image matching," Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, Beijing, 2005, pp. 1466-1473 Vol. 2, doi: 10.1109/ICCV.2005.67.
[75] T. Ma and L. J. Latecki, "From partial shape matching through local deformation to robust global shape similarity for object detection," CVPR 2011, Providence, RI, 2011, pp. 1441-1448, doi: 10.1109/CVPR.2011.5995591.
[76] Sýkora, D., Dingliana, J., and Collins, S. 2009. As-rigid-as-possible image registration for hand-drawn cartoon animations. In NPAR '09, 25--33.
[77] Cristinacce, D., & Cootes, T. F. (2006). Feature detection and tracking with constrained local models. In British Machine Vision Conference.
[78] Saragih, J. M., Lucey, S., & Cohn, J. F. (2011). Deformable model fitting by regularized landmark mean-shift. International Journal of Computer Vision, 91(2), 200–215.
[79] Vahid Kazemi and Josephine Sullivan. One millisecond face alignment with an ensemble of regression trees. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1867–1874, 2014.
[80] Sense 3D Scanner System. https://support.3dsystems.com/s/article/Sense-Scanner?language=en_US
[81] LabelImg. https://github.com/tzutalin/labelImg
[82] Raccoon Detector Dataset. https://github.com/datitran/raccoon_dataset
[83] The ElementTree XML API. https://docs.python.org/3/library/xml.etree.elementtree.html
[84] Pandas. https://pandas.pydata.org/
[85] Tensorflow. https://github.com/tensorflow/tensorflow
[86] TensorFlow Detection Model. https://github.com/tensorflow/models
[87] Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S. (2016). Face alignment across large poses:A3D solution. In IEEE conference on computer vision and pattern recognition. Las Vegas, NV.
[88] Jourabloo, A., & Liu, X. (2016). Large-pose face alignment via CNN-based dense 3D model fitting. In IEEE conference on computer vision and pattern recognition. Las Vegas, NV.
[89] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
[90] c. Harris and M. Stephens. A combined corner and edge detector. In Alvey Vision Conference, pages 147-151, 1988.
[91] M. Calonder, V. Lepetit, C. Strecha, and P. Fua. Brief: Binary robust independent elementary features. In In European Conference on Computer Vision, 2010.
[92] Python random function. https://docs.python.org/3.6/library/random.html
[93] Android Studio. https://developer.android.com/studio
[94] Paint 3D. https://www.microsoft.com/zh-tw/p/paint-3d/9nblggh5fv99?activetab=pivot:overviewtab
[95] Blender. https://www.blender.org/
[96] Real Time pose estimation of a textured object. https://docs.opencv.org/master/dc/d2c/tutorial_real_time_pose.html
[97] 3D Projection. https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html#projectpoints
[98] Moving-Least-Squares. https://github.com/Jarvis73/Moving-Least-Squares
[99] BFMatcher. https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html
[100] Feature Matching + Homography. https://docs.opencv.org/master/d1/de0/tutorial_py_feature_homography.html
[101] Visual Object Classes Challenge 2010 (VOC2010). http://host.robots.ox.ac.uk/pascal/VOC/voc2010/
[102] N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), San Diego, CA, USA, 2005, pp. 886-893 vol. 1, doi: 10.1109/CVPR.2005.177.
[103] DescriptorMatcher function. https://docs.opencv.org/master/d3/da1/classcv_1_1BFMatcher.html
[104] Yibian. http://yibian.hopto.org/
[105] Hong Gu amnd Hankang Wang Jinli Chen. “Mathematical analysis of main-to-sidelobe ratio after pulse compression in pseudorandom code phase modulation CW radar”. In: 2008 IEEE Radar Conference. May 2008, pp. 1–5. DOI: 10 . 1109 / RADAR . 2008 . 4720884.
[106] D. S. Bolme, J. R. Beveridge, B. A. Draper and Y. M. Lui, "Visual object tracking using adaptive correlation filters," 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, 2010, pp. 2544-2550, doi: 10.1109/CVPR.2010.5539960.
[107] PhotoScape X. http://x.photoscape.org/
[108] Class TrackerMOSSE. https://docs.opencv.org/3.4/javadoc/org/opencv/tracking/TrackerMOSSE.html
[109] IBM Cloud Annotations. https://github.com/cloud-annotations