研究生: |
葉桐 Yeh, Tung |
---|---|
論文名稱: |
PnP-MediaPipe 方法在無人機人機互動系統上之應用 Performance of PnP-MediaPipe Method for Human Drone Interaction System |
指導教授: |
蘇芳慶
Su, Fong-Chin |
學位類別: |
碩士 Master |
系所名稱: |
工學院 - 生物醫學工程學系 Department of BioMedical Engineering |
論文出版年: | 2024 |
畢業學年度: | 112 |
語文別: | 英文 |
論文頁數: | 69 |
中文關鍵詞: | 無人機人機互動 、無標記 、人體姿態預測 、MediaPipe |
外文關鍵詞: | Human-Drone interaction, Markerless, Pose estimation, MediaPipe |
相關次數: | 點閱:42 下載:0 |
分享至: |
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報 |
現在的世界正慢慢走向高齡化的社會,研究預估在2050年老年人口的比率會來到15.9%,而到了2100年老年人口的比率則會來到22.4%。而隨著高齡化社會的到來,老化給人帶來的影響會越來越普遍。老化是人不可避免的過程,而氧化、代謝壓力與細胞壓力適應受損是人體老化的機轉,這些機轉會使得神經元變得更脆弱,而導致得到神經疾病的機率上升,隨著老化人的神經細胞與肌肉細胞都會逐漸退化,最終導致感知、運動與認知功能的下降。而研究也指出在影響身體機能的因素中與老化同等重要的是生活的模式,如果生活模式中有規律的運動可以降低老化帶來的影響。
而無人機人機互動是一種輔助運動的方式,它可以對使用者進行個人化且即時的互動進而提升運動的動機,而且無人機可以在三維的空間中高自由度的移動,可以給使用者視覺空間感知的訓練。現在無人機的發展已經有小型且安全的無人機適合在室內中使用,現在的社會有很多的因素會成為老人外出運動的阻礙,例如在新冠疫情過後有很多長輩會傾向待在家裡以防受到疾病傳染,或是有些住在都市地區的長輩們住在沒有電梯的公寓裏面,他們從家裡走樓梯到戶外的過程就是可能跌倒的風險因子,於是乎一個在室內運動的方式變得很有用。
現在已經有許多無人機人機互動的系統,其中有最高自由度與精度的是無人機與動作捕捉系統結合的系統,但這樣的系統需要很多的訓練才能使用,其它的系統包括使用紅外線源當作參考點或是用無線電訊號與參考點來定位,這些系統需要一些標記但使用上相對簡單但在精度上也較低,隨著深度學習的發展現在人體姿態辨識已經十分成熟,於是有系統結合的這樣的功能來做互動,這樣的好處是不需要放任何的標誌在身上,但它的發展目前還在二維的訊息。本研究的目的就是開發一個無人機人機互動系統,且此系統不需要標記與校正並可以進行三維的互動。
本研究研發的方法結合了MediaPipe BlazePose和PnP方法。MediaPipe BlazePose是一個深度學習模型可以使用單張影像並預測影像中人的姿態,其中預測的姿態再結合了人體動作模型後可以輸出三維的人體姿態預測,PnP方法是線性相機模型中的一個應用,其功用是輸入數個點的三維座標,並輸入一個相機看這幾個點的二維座標,再輸入這個相機的校正矩陣就可以得到這個相機相對於三維座標系的旋轉矩陣與轉移矩陣,本研究的方法結合了這兩項技術以得到無人機與人之間的關係。
這個方法測出的結果分成兩個部分,一個部分是使用網路鏡頭另一部分是使用無人機鏡頭,使用網路鏡頭有約130 mm的誤差,但從中可以發現這個方法距離的限制而且還有一些意外的雜訊,至於使用無人機的結果則有約300 mm的誤差,其原因可能是解析度不足、傳輸造成的誤差或是相機本身的雜訊。
本篇研究研發出一個新方法讓無人機可以用方便的方法測量無人機與使用者之間的關係,未來可以透過提升硬體效能的方式或是使用及時濾波的方法來提升效能,至於如何互動的方法也待設計,但本研究提供了一個具潛力的方法供未來的研究者參考。
The world’s aging population is growing rapidly nowadays. Studies estimate that the global aged population proportion will reach 15.9% in 2050 and 22.4% in 2100. Along with an aging society, the effects of aging will become more common. Aging is an inevitable process. Oxidative and metabolic stress, and impaired cellular stress adaptation, are mechanisms of aging that make neurons vulnerable to degeneration, thereby raising the probability of neural diseases. With aging, neural cells, muscles, and connective tissues gradually degenerate, eventually leading to the deterioration of sensory, motor, and cognitive functions. Studies also show that lifestyle is equally important as aging affects bodily functions. Maintaining a lifestyle with exercise can reduce the effects of aging.
Human-drone interaction (HDI) can be used as a supportive method for exercise. It can give the user individualized and instant responses, which can improve motivation. Drones can move in 3D spaces with a high degree of freedom and provide the user with visuospatial perception. Drones nowadays have developed dramatically; they can be small and safe for indoor use. Many factors in modern society can prevent the elderly from going outside. For instance, after COVID-19, many elderly people still stay home to prevent infection. In some urban areas, elderly individuals living in apartments without elevators face the risk of falling when using stairs. Therefore, technologies for exercise and training at home are future trends.
Many HDI systems have already been developed. one with the highest degree of freedom and accuracy is to integrate drones with motion capture systems. However, such systems require extensive training prior to popular use. Other systems utilize infrared sources as reference points or radio signals for localization. These systems require some markers to be attached to users, but they are relatively simple, though their accuracy can be improved. With the development of deep learning models, new models for human pose estimation are available. Some studies attempt to use these in HDI systems. The benefit of this kind of system is markerless and easy to use.
This study proposes a method to integrate MediaPipe BlazePose and the PnP method. MediaPipe BlazePose is a deep learning model using a single image to estimate the human pose. When the learning model is integrated with a human pose model, it can recognize the 2D image and output 3-dimensional position data. The PnP method is an application of the linear camera model. With the PnP method, the translation matrix and rotation matrix of a camera can be obtained from the 3D position of several points and their 2D position in the image. This study integrates these two techniques to estimate the relationship between the user and the drone.
A webcam and a drone camera are used to capture images, respectively. to the findings show a measured error of around 130 mm using a webcam and 300 mm for the drone camera. Unexpected noise is one of the factors to cause measurement errors. Also, insufficient resolution, data transmission errors, or drone camera noise cause bigger distance estimation errors.
This study develops a new method to estimate the relative position and orientation between a drone and a user. However, the accuracy of this method can be improved by upgrading the hardware capabilities or applying real-time filters to the data. The method of interaction also awaits further design, but this study provides a method with potential.
[1] M. Sander, B. Oxlund, A. Jespersen, A. Krasnik, E. L. Mortensen, R. G. J. Westendorp, and L. J. Rasmussen, “The challenges of human population ageing,” Age and Ageing, vol. 44, no. 2, pp. 185-187, 2014.
[2] D. Gu, K. Andreev, and M. E. Dupre, “Major Trends in Population Growth Around the World,” China CDC Weekly, vol. 3, no. 28, pp. 604-613, 2021.
[3] M. P. Mattson, and T. Magnus, “Ageing and neuronal vulnerability,” Nature Reviews Neuroscience, vol. 7, no. 4, pp. 278-294, 2006.
[4] G. D'Antona, M. A. Pellegrino, R. Adami, R. Rossi, C. N. Carlizzi, M. Canepari, B. Saltin, and R. Bottinelli, “The effect of ageing and immobilization on structure and function of human skeletal muscle fibres,” vol. 552, no. 2, pp. 499-511, 2003.
[5] E. Carmeli, H. Patish, and R. Coleman, “The Aging Hand,” The Journals of Gerontology: Series A, vol. 58, no. 2, pp. M146-M152, 2003.
[6] M. Izquierdo, R. A. Merchant, J. E. Morley, S. D. Anker, I. Aprahamian, H. Arai, M. Aubertin-Leheudre, R. Bernabei, E. L. Cadore, M. Cesari, L. K. Chen, P. De Souto Barreto, G. Duque, L. Ferrucci, R. A. Fielding, A. García-Hermoso, L. M. Gutiérrez-Robledo, S. D. R. Harridge, B. Kirk, S. Kritchevsky, F. Landi, N. Lazarus, F. C. Martin, E. Marzetti, M. Pahor, R. Ramírez-Vélez, L. Rodriguez-Mañas, Y. Rolland, J. G. Ruiz, O. Theou, D. T. Villareal, D. L. Waters, C. W. Won, J. Woo, B. Vellas, and M. F. Singh, “International Exercise Recommendations in Older Adults (ICFSR): Expert Consensus Guidelines,” The Journal of nutrition, health and aging, vol. 25, no. 7, pp. 824-853, 2021.
[7] U. Ekelund, H. A. Ward, T. Norat, J. a. Luan, A. M. May, E. Weiderpass, S. J. Sharp, K. Overvad, J. N. Østergaard, and A. Tjønneland, “Physical activity and all-cause mortality across levels of overall and abdominal adiposity in European men and women: the European Prospective Investigation into Cancer and Nutrition Study (EPIC),” The American journal of clinical nutrition, vol. 101, no. 3, pp. 613-621, 2015.
[8] E. L. Cadore, A. Casas-Herrero, F. Zambom-Ferraresi, F. Idoate, N. Millor, M. Gómez, L. Rodriguez-Mañas, and M. Izquierdo, “Multicomponent exercises including muscle power training enhance muscle mass, power output, and functional outcomes in institutionalized frail nonagenarians,” AGE, vol. 36, no. 2, pp. 773-785, 2014.
[9] E. F. Binder, K. B. Schechtman, A. A. Ehsani, K. Steger‐May, M. Brown, D. R. Sinacore, K. E. Yarasheski, and J. O. Holloszy, “Effects of exercise training on frailty in community‐dwelling older adults: results of a randomized, controlled trial,” Journal of the American Geriatrics Society, vol. 50, no. 12, pp. 1921-1928, 2002.
[10] J. E. Morley, “Pharmacologic Options for the Treatment of Sarcopenia,” Calcified Tissue International, vol. 98, no. 4, pp. 319-333, 2016.
[11] L. F. Biazus-Sehn, F. B. Schuch, J. Firth, and F. de Souza Stigger, “Effects of physical exercise on cognitive function of older adults with mild cognitive impairment: A systematic review and meta-analysis,” Archives of Gerontology and Geriatrics, vol. 89, pp. 104048, 2020.
[12] M. Izquierdo, J. E. Morley, and A. Lucia, "Exercise in people over 85," British Medical Journal Publishing Group, 2020.
[13] W. K. Abdelbasset, “Stay Home: Role of Physical Exercise Training in Elderly Individuals’ Ability to Face the COVID-19 Infection,” Journal of Immunology Research, vol. 2020, pp. 1-5, 2020.
[14] M. Noseworthy, L. Peddie, E. J. Buckler, F. Park, M. Pham, S. Pratt, A. Singh, E. Puterman, and T. Liu-Ambrose, “The Effects of Outdoor versus Indoor Exercise on Psychological Health, Physical Health, and Physical Activity Behaviour: A Systematic Review of Longitudinal Trials,” International Journal of Environmental Research and Public Health, vol. 20, no. 3, pp. 1669, 2023.
[15] T. Wang, Y. Gan, S. D. Arena, L. T. Chitkushev, G. Zhang, and R. Rawassizadeh, “Advances for Indoor Fitness Tracking, Coaching, and Motivation: A Review of Existing Technological Advances,” IEEE Systems, Man, and Cybernetics Magazine, vol. 7, no. 1, pp. 4-14, 2021.
[16] E. Graether, and F. Mueller, "Joggobot."
[17] F. F. Mueller, and M. Muirhead, "Jogging with a Quadcopter."
[18] M. Avila, M. Funk, and N. Henze, "DroneNavigator."
[19] N. Maclean, P. Pound, C. Wolfe, and A. Rudd, “A critical review of the concept of patient motivation in the literature on physical rehabilitation,” Soc Sci Med, vol. 50, no. 4, pp. 495-506, 2000.
[20] P. Dias, R. Silva, P. Amorim, J. Lains, E. Roque, I. S. F. Pereira, F. Pereira, and B. S. Santos, “Using Virtual Reality to Increase Motivation in Poststroke Rehabilitation,” IEEE Computer Graphics and Applications, vol. 39, no. 1, pp. 64-70, 2019.
[21] G. A. Hromas, and R. M. Bauer, “Changes in visuospatial, visuoperceptual, and navigational ability in ageing,” Cognitive changes of the ageing brain, pp. 54-67, 2019.
[22] J. La Delfa, M. A. Baytas, R. Patibanda, H. Ngari, R. A. Khot, and F. F. Mueller, "Drone Chi: Somaesthetic Human-Drone Interaction."
[23] S. G. Zwaan, and E. I. Barakova, "Boxing against drones."
[24] M. A. Baytas, D. Çay, Y. Zhang, M. Obaid, A. E. Yantaç, and M. Fjeld, "The Design of Social Drones."
[25] O. I. Abiodun, A. Jantan, A. E. Omolara, K. V. Dada, N. A. Mohamed, and H. Arshad, “State-of-the-art in artificial neural network applications: A survey,” Heliyon, vol. 4, no. 11, 2018.
[26] M. Saqib, S. Daud Khan, N. Sharma, and M. Blumenstein, "A study on detecting drones using deep convolutional neural networks."
[27] Z. Marinov, S. Vasileva, Q. Wang, C. Seibold, J. Zhang, and R. Stiefelhagen, "Pose2Drone: A Skeleton-Pose-based Framework for Human-Drone Interaction."
[28] E. H. Kadhim, and A. T. Abdulsadda, “Mini Drone Linear and Nonlinear Controller System Design and Analyzing,” Journal of Robotics and Control (JRC), vol. 3, no. 2, pp. 212-218, 2022.
[29] M. R. Kaplan, A. Eraslan, A. Beke, and T. Kumbasar, "Altitude and Position Control of Parrot Mambo Minidrone with PID and Fuzzy PID Controllers."
[30] J. M. Villadangos, J. Urena, M. Mazo, A. Hernandez, F. Alvarez, J. J. Garcia, C. De Marziani, and D. Alonso, "Improvement of ultrasonic beacon-based local position system using multi-access techniques."
[31] M. Emimi, M. Khaleel, and A. Alkrash, “The current opportunities and challenges in drone technology,” Int. J. Electr. Eng. and Sustain., pp. 74-89, 2023.
[32] M. M. Quamar, B. Al-Ramadan, K. Khan, M. Shafiullah, and S. El Ferik, “Advancements and Applications of Drone-Integrated Geographic Information System Technology—A Review,” Remote Sensing, vol. 15, no. 20, pp. 5039, 2023.
[33] D. Odijk, N. Nadarajah, S. Zaminpardaz, and P. J. G. Teunissen, “GPS, Galileo, QZSS and IRNSS differential ISBs: estimation and application,” GPS Solutions, vol. 21, no. 2, pp. 439-450, 2017.
[34] S. Zaminpardaz, K. Wang, and P. J. G. Teunissen, “Australia-first high-precision positioning results with new Japanese QZSS regional satellite system,” GPS Solutions, vol. 22, no. 4, 2018.
[35] T. S. Chu, A. Chua, E. Sybingco, and M. A. Roque, “A Performance Analysis on Swarm Drone Loco Positioning System for Two-Way Ranging Protocol,” International Journal of Automation and Smart Technology, vol. 11, no. 1, pp. 2244-2244, 2021.
[36] A. Taffanel, B. Rousselot, J. Danielsson, K. McGuire, K. Richardsson, M. Eliasson, T. Antonsson, and W. H\"onig, “Lighthouse Positioning System: Dataset, Accuracy, and Precision for UAV Research,” arXiv pre-print server, 2021-04-23, 2021.
[37] V. Bazarevsky, I. Grishchenko, K. Raveendran, T. Zhu, F. Zhang, and M. Grundmann, “BlazePose: On-device Real-time Body Pose tracking,” arXiv pre-print server, 2020-06-17, 2020.
[38] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, Ming, J. Lee, W.-T. Chang, W. Hua, M. Georg, and M. Grundmann, “MediaPipe: A Framework for Building Perception Pipelines,” arXiv pre-print server, 2019-06-14, 2019.
[39] H. Xu, E. G. Bazavan, A. Zanfir, W. T. Freeman, R. Sukthankar, and C. Sminchisescu, "Ghum & ghuml: Generative 3d human shape and articulated pose models." pp. 6184-6193.
[40] I. Jubran, F. Fares, Y. Alfassi, F. Ayoub, and D. Feldman, "Newton-PnP: Real-time Visual Navigation for Autonomous Toy-Drones."