近年來，以遠程光體積描記法(remote photoplethysmography, rPPG) 為基礎的心率測量方法逐漸盛行，只要簡單地運用相機或網路攝像頭等遠程感測器便能進行非接觸式的心率評估。然而，現存方法大多使用正臉影像作為原始訊號來源，只能在固定臉部角度、位置的情況下進行心率測量；此外，測量時也會要求受測者主動對齊臉部的感興趣區域(region of interest, ROI)。在現實應用上較難滿足這樣的實驗環境，同時也很大程度地限制了受測者的自由。因此，為了降低心率測量時的侷限性，本研究提出基於臉部關鍵點影像的心率評估方法，其主要架構為臉部關鍵點偵測與
rPPG 訊號處理。

根據相關文獻，大面積的臉部皮膚區域(如臉頰、額頭) 擁有著豐富的光體積描記(photoplethysmography, PPG) 訊息，而這些訊息有助於後續的心率值評估。本研究採用了開源的81 點人臉對齊模型來進行臉部關鍵點偵測，並利用這些關鍵點來定義多角度(pose) 的臉部ROI；在測量時，除了能避開與PPG 訊號不相關的區域，臉部ROI 更能依據受測者臉型進行自動定義。運用人臉對齊模型多角度偵測的特性，在角度範圍內(−55◦ ∼ 55◦) 的臉部影像皆有對應的ROI 定義，如此便能達到多角度的心率測量。此外，本研究也定義了主要與次要的ROI 區域，旨在降低臉部部分遮擋對心率測量所帶來的影響。最後，我們讓自定義的臉部ROI 進行動態更新，讓受測者得以在頭部旋轉或平移時進行心率測量，更進一步提高測量時的靈活度。

本研究共進行了三個實驗，並使用平均絕對誤差(mean absolute error, MAE) 與均方根誤差(root-mean square error, RMSE) 作為指標進行心率測量的準確度評估。第一個實驗使用了不同的ROI 組合來進行心率測量，藉此探討皮膚面積對於測量準確度的影響；於實驗結果中，我們發現越大面積的ROI 組合確實會有更準確的心率測量效果，其中以正臉之ROI 組合(Ri,Rii,Riii) 表現最佳。接著，本研究結合了自定義的主要與次要ROI，對不同角度下的臉部影像進行心率測量與探討；於實驗結果中，0◦ 與±35◦ 的表現較佳，而±20◦ 與±50◦ 的表現較差。儘管如此，不同角度時的測量皆有著MAE 低於3bpm 的優異表現，達成了「多角度」臉部影像的心率測量效果。最後一個實驗則是讓本研究自定義的ROI 進行動態更新，並將其用於頭部平移或旋轉時的心率測量。在實驗結果中，其中頭部旋轉時的表現較為穩定，而頭部平移時的平均表現較佳；無論是頭部平移或旋轉時的心率測量皆有著MAE 低於4bpm 的表現，這說明了動態ROI 在有著較高靈活度的同時也能保有一定的測量準確度。從三個實驗的結果來看，研究方法成功運用了多角度、動態的臉部影像來進行心率評估，讓受測者在較低侷限性的情況下也能獲得準確的心率值。

In this study, we propose a method based on images of facial landmarks to improve heart rate measurement, which is combined with facial landmark detection and remote photoplethysysmography (rPPG). Facial landmark detection provides key points of face, which helps us to define the region of interest (ROI). Then the rPPG method uses variations in pixel values of the skin area for signal extraction. Finally, we convert the obtained photoplethysysmography (PPG) signal into the corresponding heart rate signal.

Most of the existing methods use the frontal face image as the original signal source. It would impose greater restrictions on the subject and leads to inconvenience in real-life application. In order to improve the limitation, this study uses the face alignment model to define the ROI. For images close to the frontal face, the PPG information is mainly derived from the forehead area. Otherwise, for large-pose facial images, the PPG information is extracted from the cheek area. In this way, all facial images within the angle range have corresponding ROI definitions and then multi-pose heart rate measurement can be performed.

In experiment phase, we use mean absolute error (MAE) and root-mean square error (RMSE) as indicators to evaluate the accuracy of heart rate measurement. Our result shows that the measurements at all selected pose have the performace with MAE lower than 3bpm. Moreover, we also let self-defined ROI update dynamically for heart rate measurement when the head translates or rotates in yaw. The result shows that the measurements in both situations have the performance of MAE lower than 4bpm.

[1] C. Chen, A. Dantcheva, and A. Ross, “Automatic facial makeup detection with application in face recognition,” in 2013 International Conference on Biometrics (ICB), pp. 1–8, 2013.
[2] A. B. Ashraf, S. Lucey, J. F. Cohn, T. Chen, Z. Ambadar, K. M. Prkachin, and P. E. Solomon, “The painful face–pain expression recognition using active appearance models,”Image and Vision Computing, vol. 27, no. 12, pp. 1788–1796, 2009.
[3] H. Mo, L. Liu, W. Zhu, S. Yin, and S. Wei, “Face alignment with expression- and pose-based adaptive initialization,” IEEE Transactions on Multimedia, vol. 21, no. 4, pp. 943–956, 2019.
[4] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 1, pp. I–I, 2001.
[5] D. E. King, “Dlib-ml: A machine learning toolkit,” Journal of Machine Learning Research, vol. 10, no. 60, pp. 1755–1758, 2009.
[6] Y. Liu, A. Jourabloo, W. Ren, and X. Liu, “Dense face alignment,” 2017.
[7] X. Xiong and F. De la Torre, “Supervised descent method and its applications to face alignment,” in 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539, 2013.
[8] S. Zhu, C. Li, C. C. Loy, and X. Tang, “Face alignment by coarse-to-fine shape searching,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4998–5006, 2015.
[9] Y. Sun, X. Wang, and X. Tang, “Deep convolutional network cascade for facial point detection,” in 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3476–3483, 2013.
[10] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, “Facial landmark detection by deep multitask learning,” in Computer Vision – ECCV 2014 (D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, eds.), (Cham), pp. 94–108, Springer International Publishing, 2014.
[11] G. Trigeorgis, P. Snape, M. A. Nicolaou, E. Antonakos, and S. Zafeiriou, “Mnemonic descent method: A recurrent process applied for end-to-end face alignment,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4177–4187, 2016.
[12] X. Zhu, X. Liu, Z. Lei, and S. Z. Li, “Face alignment in full pose range: A 3d total solution,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, p. 78–92, Jan 2019.
[13] A. Sherstinsky, “Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,” Physica D: Nonlinear Phenomena, vol. 404, p. 132306, 2020.
[14] R. Caruana, “Multitask learning,” Machine Learning, vol. 28, pp. 41–75, 2004.
[15] V. Blanz and T. Vetter, “Face recognition based on fitting a 3d morphable model,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 9, pp. 1063–1074, 2003.
[16] X. Zhu, Z. Lei, J. Yan, D. Yi, and S. Z. Li, “High-fidelity pose and expression normalization for face recognition in the wild,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 787–796, 2015.
[17] A. Bulat and G. Tzimiropoulos, “How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks),” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[18] A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for human pose estimation,” in Computer Vision – ECCV 2016 (B. Leibe, J. Matas, N. Sebe, and M. Welling, eds.), (Cham), pp. 483–499, Springer International Publishing, 2016.
[19] J. Shen, S. Zafeiriou, G. G. Chrysos, J. Kossaifi, G. Tzimiropoulos, and M. Pantic, “The first facial landmark tracking in-the-wild challenge: Benchmark and results,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, December 2015.
[20] S. Zafeiriou, G. Trigeorgis, G. Chrysos, J. Deng, and J. Shen, “The menpo facial landmark localisation challenge: A step towards the solution,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2116–2125, 2017.
[21] C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, “A semi-automatic methodology for facial landmark annotation,” in 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 896–903, 2013.
[22] N. Sun, M. Garbey, A. Merla, and I. Pavlidis, “Imaging the cardiovascular pulse,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 2, pp. 416–421 vol. 2, 2005.
[23] S. Y. Chekmenev, A. A. Farag, W. M. Miller, E. A. Essock, and A. Bhatnagar, Multiresolution Approach for Noncontact Measurements of Arterial Pulse Using Thermal Imaging, pp. 87–112. London: Springer London, 2009.
[24] W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmographic imaging using ambient light.,” Opt. Express, vol. 16, pp. 21434–21445, Dec 2008.
[25] M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation.,” Opt. Express, vol. 18, pp. 10762–10774, May 2010.
[26] M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE Transactions on Biomedical Engineering, vol. 58, no. 1, pp. 7–11, 2011.
[27] T. Pursche, J. Krajewski, and R. Moeller, “Video-based heart rate measurement from human faces,” in 2012 IEEE International Conference on Consumer Electronics (ICCE), pp. 544–545, 2012.
[28] S. Kwon, H. Kim, and K. S. Park, “Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone,” in 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2174–2177, 2012.
[29] G. Balakrishnan, F. Durand, and J. Guttag, “Detecting pulse from head motions in video,” in 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3430–3437, 2013.
[30] M. Rubinstein et al., Analysis and visualization of temporal variations in video. PhD thesis, Massachusetts Institute of Technology, 2014.
[31] X. Li, J. Chen, G. Zhao, and M. Pietikäinen, “Remote heart rate measurement from face videos under realistic situations,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271, 2014.
[32] R. Stricker, S. Müller, and H.-M. Gross, “Non-contact video-based pulse rate measurement on a mobile service robot,” in The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pp. 1056–1062, 2014.
[33] S. Xu, L. Sun, and G. K. Rohde, “Robust efficient estimation of heart rate pulse from video,” Biomed. Opt. Express, vol. 5, pp. 1124–1135, Apr 2014.
[34] Z. Wang, X. Yang, and K.-T. Cheng, “Accurate face alignment and adaptive patch selection for heart rate estimation from videos under realistic scenarios,” PloS one, vol. 13, no. 5, p. e0197275, 2018.
[35] X. Niu, S. Shan, H. Han, and X. Chen, “Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation,” IEEE Transactions on Image Processing, vol. 29, pp. 2409–2423, 2020.
[36] Y. Liu, B. Qin, R. Li, X. Li, A. Huang, H. Liu, Y. Lv, and M. Liu, “Motion-robust multimodal heart rate estimation using bcg fused remote-ppg with deep facial roi tracker and pose constrained kalman filter,” IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1–15, 2021.
[37] C. Wang, T. Pun, and G. Chanel, “A comparative survey of methods for remote heart rate detection from frontal face videos,” Frontiers in Bioengineering and Biotechnology, vol. 6, p. 33, 2018.
[38] R. Irani, K. Nasrollahi, and T. B. Moeslund, “Improved pulse detection from head motions using dct,” in 2014 International Conference on Computer Vision Theory and Applications (VISAPP), vol. 3, pp. 118–124, 2014.
[39] S. M. Sahin, Q. Deng, J. Castelo, and D. Lee, “Non-contact heart rate monitoring from face video utilizing color intensity,” Journal of Multimedia Information System, vol. 8, no. 1, pp. 1–10, 2021.
[40] S. Benedetto, C. Caldato, D. C. Greenwood, N. Bartoli, V. Pensabene, and P. Actis, “Remote heart rate monitoring-assessment of the facereader rppg by noldus,” PloS one, vol. 14, no. 11, p. e0225592, 2019.
[41] A. Gudi, M. Bittner, and J. van Gemert, “Real-time webcam heart-rate and variability estimation with clean ground truth for evaluation,” Applied Sciences, vol. 10, no. 23, 2020.
[42] M. Lewandowska, J. Rumiński, T. Kocejko, and J. Nowak, “Measuring pulse rate with a webcam —a non-contact method for evaluating cardiac activity,” in 2011 Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 405–410,
2011.
[43] D. Datcu, M. Cidota, S. Lukosch, and L. Rothkrantz, “Noncontact automatic heart rate analysis in visible spectrum by specific face regions,” in Proceedings of the 14th International Conference on Computer Systems and Technologies, CompSysTech ’13, (New
York, NY, USA), p. 120–127, Association for Computing Machinery, 2013.
[44] C. Tang, J. Lu, and J. Liu, “Non-contact heart rate monitoring by combining convolutional neural network skin detection and remote photoplethysmography via a low-cost camera,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1390–13906, 2018.
[45] N. Ibrahim, R. Tomari, W. N. W. Zakaria, and N. Othman, “Analysis of non-invasive video based heart rate monitoring system obtained from various distances and different facial spot,” Journal of Physics: Conference Series, vol. 1049, p. 012003, jul 2018.
[46] P. Gupta, B. Bhowmick, and A. Pal, “Mombat: Heart rate monitoring from face video using pulse modeling and bayesian tracking,” 2020.
[47] Sahindrakar, “Improving motion robustness of contact-less monitoring of heart rate using video analysis,” 2011.
[48] S. Zaunseder, A. Heinke, A. Trumpp, and H. Malberg, “Heart beat detection and analysis from videos,” in 2014 IEEE 34th International Scientific Conference on Electronics and Nanotechnology (ELNANO), pp. 286–290, 2014.
[49] G. De Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Transactions on Biomedical Engineering, vol. 60, no. 10, pp. 2878–2886, 2013.
[50] S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Selfadaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 2396–2404, 2016.
[51] W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan, “Color-distortion filtering for remote photoplethysmography,” in 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp. 71–78, IEEE, 2017.
[52] Z. Zhang and H. Zha, “Nonlinear dimension reduction via local tangent space alignment,” in International Conference on Intelligent Data Engineering and Automated Learning, pp. 477–481, Springer, 2003.
[53] L. K. Mestha, S. Kyal, B. Xu, L. E. Lewis, and V. Kumar, “Towards continuous monitoring of pulse rate in neonatal intensive care unit with a webcam,” in 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 3817–3820, 2014.
[54] Y.-P. Yu, B.-H. Kwan, C.-L. Lim, S.-L. Wong, and P. Raveendran, “Video-based heart rate measurement using short-time fourier transform,” in 2013 International Symposium on Intelligent Signal Processing and Communication Systems, pp. 704–707, 2013.
[55] N. E. Ruben, “Remote heart rate estimation using consumer-grade cameras,” 2015.
[56] H. Monkaresi, R. A. Calvo, and H. Yan, “A machine learning approach to improve contactless heart rate monitoring using a webcam,” IEEE Journal of Biomedical and Health Informatics, vol. 18, no. 4, pp. 1153–1160, 2014.
[57] L. Tarassenko, M. Villarroel, A. Guazzi, J. Jorge, D. A. Clifton, and C. Pugh, “Noncontact video-based vital sign monitoring using ambient light and auto-regressive models,” Physiological Measurement, vol. 35, pp. 807–831, mar 2014.
[58] A. Osman, J. Turcot, and R. E. Kaliouby, “Supervised learning approach to remote heart rate estimation from facial videos,” in 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–6, 2015.
[59] M. Villarroel, J. Jorge, C. Pugh, and L. Tarassenko, “Non-contact vital sign monitoring in the clinic,” in 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG 2017), pp. 278–285, 2017.
[60] A. Ni, A. Azarang, and N. Kehtarnavaz, “A review of deep learning-based contactless heart rate measurement methods,” Sensors, vol. 21, no. 11, p. 3719, 2021.
[61] Y. Qiu, Y. Liu, J. Arteaga-Falconi, H. Dong, and A. E. Saddik, “Evm-cnn: Real-time contactless heart rate estimation from facial video,” IEEE Transactions on Multimedia, vol. 21, no. 7, pp. 1778–1787, 2019.
[62] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1800–1807, 2017.
[63] M. Paracchini, M. Marcon, F. Villa, F. Zappa, and S. Tubaro, “Biometric signals estimation using single photon camera and deep learning,” Sensors, vol. 20, no. 21, 2020.
[64] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention, pp. 234–241, Springer, 2015.
[65] W. Chen and D. McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 349–365, 2018.
[66] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[67] A. Reiss, I. Indlekofer, P. Schmidt, and K. Van Laerhoven, “Deep ppg: large-scale heart rate estimation with convolutional neural networks,” Sensors, vol. 19, no. 14, p. 3079, 2019.
[68] E. Lee, E. Chen, and C.-Y. Lee, “Meta-rppg: Remote heart rate estimation using atransductive meta-learner,” in European Conference on Computer Vision, pp. 392–409, Springer, 2020.
[69] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in Proceedings of the IEEE international conference on computer vision, pp. 4489–4497, 2015.
[70] R. Špetlík, V. Franc, and J. Matas, “Visual heart rate estimation with convolutional neural network,” in Proceedings of the british machine vision conference, Newcastle, UK, pp. 3–6, 2018.
[71] Z. Yu, W. Peng, X. Li, X. Hong, and G. Zhao, “Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 151–160, 2019.
[72] Z. Yu, X. Li, and G. Zhao, “Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks,” arXiv preprint arXiv:1905.02419, 2019.
[73] J. Hou, X. Wu, F. Yu, and Y. Jia, “Multimedia event detection via deep spatial-temporal neural networks,” in 2016 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6, 2016.
[74] T. Luguev, D. Seuß, and J.-U. Garbas, “Deep learning based affective sensing with remote photoplethysmography,” in 2020 54th Annual Conference on Information Sciences and Systems (CISS), pp. 1–4, 2020.
[75] O. Perepelkina, M. Artemyev, M. Churikova, and M. Grinenko, “Hearttrack: Convolutional neural network for remote video-based heart rate monitoring,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops
(CVPRW), pp. 1163–1171, 2020.
[76] F. Bousefsaf, A. Pruski, and C. Maaoui, “3d convolutional neural networks for remote pulse rate measurement and mapping from facial video,” Applied Sciences, vol. 9, no. 20, p. 4364, 2019.
[77] S.-Q. Liu and P. C. Yuen, “A general remote photoplethysmography estimator with spatiotemporal convolutional network,” in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 481–488, 2020.
[78] S. Bobbia, R. Macwan, Y. Benezeth, A. Mansouri, and J. Dubois, “Unsupervised skin tissue segmentation for remote photoplethysmography,” Pattern Recognition Letters, vol. 124, pp. 82–90, 2019.
[79] V. Kazemi and J. Sullivan, “One millisecond face alignment with an ensemble of regression trees,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874, 2014.
[80] P. Werner, A. Al-Hamadi, S. Walter, S. Gruss, and H. C. Traue, “Automatic heart rate estimation from painful faces,” in 2014 IEEE International Conference on Image Processing (ICIP), pp. 1947–1951, 2014.
[81] K. Zheng, K. Ci, J. Cui, J. Kong, and J. Zhou, “Non-contact heart rate detection when face information is missing during online learning,” Sensors (Basel, Switzerland), vol. 20, 2020.
[82] Y. Sun, S. Hu, V. Azorin Peris, R. Kalawsky, and S. Greenwald, “Noncontact imaging photoplethysmography to effectively access pulse rate variability,” Journal of biomedical optics, vol. 18, p. 61205, 06 2013.