簡易檢索 / 詳目顯示

回結果列表
研究生： 張健新
Chang, Chien-Hsin
論文名稱： 融合人類運動特徵之增強式學習過程於雙足機器人跑步與跳躍動作產生器與控制器之設計與實現
Design and Implementation of Running and Jumping Motion Generation and Corresponding Controllers for Biped Robot by Integrating Reinforcement Learning with Human Motion Features
指導教授： 李祖聖
Li, Tzuu-Hseng S.
學位類別： 碩士
Master
系所名稱： 電機資訊學院 - 電機工程學系
Department of Electrical Engineering
論文出版年： 2021
畢業學年度： 109
語文別： 英文
論文頁數： 89
中文關鍵詞： 雙足機器人 、機器人動作產生器 、增強式學習 、動作捕捉系統
外文關鍵詞： Biped Robot, Robot Motion Generation, Reinforcement Learning, Motion Capture System
相關次數： 點閱：196  下載：0 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報

    • 本論文研究目標係為雙足機器人建立適合跑步與跳躍的運動產生器，接著針對穩定的跑步步態以及不同的跳躍目標，設計基於深層神經網路的控制器，並於訓練控制器的增強式學習過程中融入了人類的運動特徵。本論文首先探討如何設計雙足機器人機構，討論的內容包含雙腳自由度的安排、機構材料的選用、機構設計以及重量的分配。於機器人系統架構中，致動系統採用伺服馬達，感測器系統包含深度攝影機、慣性量測元件以及重量感測元件，以電腦作為控制系統中心接收與處理感測器資訊並下達動作指令，透過整合以上系統實現雙足機器人跑步與跳躍的運動產生器和控制器。本論文提出以質量中心軌跡作為雙足機器人的跳躍動作產生器，此動作產生器可以依據機器人的初始狀態以及達成不同任務所需的起跳狀態，並考量運動過程的時間，產生適合的質心軌跡。跑步動作產生器使用基於線性倒單擺模型的可參數化步態規劃方式，透過提高質量中心的移動速度以縮短運動週期，使雙足機器人可以做到跑步的敏捷步態。訓練基於深度神經網路的跑步與跳躍控制器的過程，加入了人類動作資訊於設計強化學習演算法的獎勵函數。最後，實驗結果於Pybullet模擬器與實體機器人上皆顯示本論文提出之動作產生器與控制器能有效讓雙足機器人完成跳躍和跑步的動作。

    The purpose of this thesis is devoted to establish running and jumping motion generation for the biped robot. The deep neural network based controllers are designed for running motion stabilization and different objective jumping motion. This work incorporates the human motion features in the reinforcement learning process. First of all, this thesis introduces the mechanism, hardware and control system of the biped robot. The Linear Inverted Pendulum Model (LIPM) is utilized to generate running pattern. The running pattern can be mapped to robot joints through the inverse kinematics. The Center of Mass (CoM) trajectory is generated through the polynomial equation for jump motion. Then, the Jacobian Matrix is adopted to track the trajectory through the velocity control. Different phases of jump motion are examined. In order to make the robot’s motion perform like a human, we combine the human motion features with the reward function while training the deep neural network controller by using reinforcement learning. After training, the controllers can help the robot stable while running and accomplishing different ways of jump motion. We implement the motion generations and controllers on the simulation and the real biped robot. Three experiments are performed on real biped robot, including implementation of running stabilization, different ways of jump motion, and different postures during the flight phase of jump motion. Experimental results demonstrate that the proposed motion generations and controllers are successfully achieved on the real robot.

    Abstract I Acknowledgement III Content IV List of Figures VIII List of Tables XI Chapter 1 Introduction 1 1.1 Motivation 1 1.2 Related Works 3 1.3 Thesis Organization 5 Chapter 2 Biped Robot Prototype 7 2.1 Introduction 7 2.2 Mechanical Design of Biped Robot 8 2.2.1 The Configuration of Biped Robot 9 2.2.2 Mechanism Design of Biped Robot 12 2.2.3 Structure Design of Biped Robot 13 2.3 Hardware of Biped Robot 15 2.3.1 HT-03 Servo Motor 15 2.3.2 Dynamixel All-in-one Smart Actuator 17 2.3.3 Camera 19 2.3.4 Inertial Measurement Unit 20 2.3.5 Pressure Sensor 21 2.3.6 Arduino Nano 22 2.3.7 Jetson Xavier NX 23 2.3.8 Power Supply 25 2.4 System Architecture of Biped Robot 26 Chapter 3 Running or Jumping Motion Generation for Biped Robot 27 3.1 Introduction 27 3.2 Linear Inverted Pendulum Model for Running 28 3.2.1 Linear Inverted Pendulum Model 28 3.2.2 Double-link LIPM 31 3.2.3 Zero Moment Point 33 3.3 Different Phases of Jumping Motion 35 3.3.1 Squatting Phase 36 3.3.2 Lifting Off Phase 36 3.3.3 Flight Phase 37 3.3.4 Landing Phase 37 3.4 CoM Trajectory Generation of Jump Motion 37 3.4.1 Spherical Coordinate System 38 3.4.2 Radial Distance Function 38 3.4.3 Polar Angle and Azimuthal Angle of Trajectory 40 3.5 Introduction to Jacobian Matrix 40 3.5.1 Kinematics Chain 41 3.5.2 Forward Kinematics 42 3.5.3 Conventional Jacobian Matrix 44 3.5.4 Kinetic Jacobian Matrix 47 Chapter 4 Running and Jumping Feedback Control System for Biped Robot 52 4.1 Introduction 52 4.2 Running Stabilization Feedback Control System 53 4.2.1 Running Stabilization System 53 4.2.2 Running Stabilization Controller Learning Process 55 4.3 Controllable Jumping Feedback System 56 4.3.1 Controllable Jumping System 56 4.3.2 Controllable Jumping Controller Learning Process 57 4.4 Reinforcement Learning Algorithm 58 4.4.1 Actor-Critic 59 4.4.2 Deep Deterministic Policy Gradient 60 4.4.3 Twin Delayed Deep Deterministic Policy Gradient 62 Chapter 5 Alpha Testing 65 5.1 Introduction 65 5.2 Implementation of Human Motion Data Analysis 66 5.2.1 Introduction to Motion Capture Tool 66 5.2.2 Motion Capture Data Collection and Processing 68 5.2.3 Feature Extraction for Reinforcement Learning 70 5.3 Introduction to Simulation Environment 71 5.4 Simulation and Experimental Results 72 5.4.1 Simulation I: Comparison of the Kinetic Jacobian and Conventional Jacobian Matrix 73 5.4.2 Simulation II: Orientational Variation of Ankle While Jumping 74 5.4.3 Simulation III: Integrating Reinforcement Learning with Human Motion Features for Running Motion 74 5.4.4 Simulation IV: Integrating Reinforcement Learning with Human Motion Features for Jumping Motion 77 5.4.5 Experiment I: Implementation of Running Stabilization Controller on Biped Robot 79 5.4.6 Experiment II: Implementation of Vertical and Forward Jumping Motion on Biped Robot 80 5.4.7 Experiment III: Comparison of Different Robot Postures During the Flight Phase of Jumping Motion 81 Chapter 6 Conclusions and Future Work 81 6.1 Conclusions 82 6.2 Future Work 83 References 85

    [1]Y. Kim, Y. Lee, S. Lee, J. Kim, H. S. Kim, and T. W.Seo, “STEP: A new mobile platform with 2-DOF transformable wheels for service robots,” IEEE/ASME Trans. Mechatronics, vol. 25, no. 4, pp. 1859–1868, Aug.2020, doi: 10.1109/TMECH.2020.2992280.
    [2]“THORMANG3 > THORMANG 3 | ROBOTIS.” http://en.robotis.com/shop_en/item.php?it_id=905-0020-100.
    [3]A. Bouton, C. Grand, and F. Benamar, “Design and control of a compliant wheel-on-leg rover which conforms to uneven terrain,” IEEE/ASME Trans. Mechatronics, vol. 25, no. 5, pp. 2354–2363, Oct.2020, doi: 10.1109/TMECH.2020.2973752.
    [4]“Defense Advanced Research Projects Agency.” https://www.darpa.mil/.
    [5]“Shakey the Robot - SRI International.” https://www.sri.com/hoi/shakey-the-robot/.
    [6]“Atlas® | Boston Dynamics.” https://www.bostondynamics.com/atlas.
    [7]“FIRA RoboworldCup Official Website.” https://www.firaworldcup.org/VisitorPages/default.aspx?itemid=3.
    [8]“ROBOTIS OP.” https://emanual.robotis.com/docs/en/platform/op/getting_started/.
    [9]“NAO Power V6 Standard Edition.” https://www.robotlab.com/store/nao-power-v6-standard-edition.
    [10] “Robots — Agility Robotics.” https://www.agilityrobotics.com/robots#digit.
    [11] “Spot | Boston Dynamics.” https://www.bostondynamics.com/spot.
    [12] “ASIMO by Honda | The World’s Most Advanced Humanoid Robot.” https://asimo.honda.com/.
    [13] G. Kenneally, A. De, and D. E. Koditschek, “Design principles for a family of direct-drive legged robots,” IEEE Robot. Autom. Lett., vol. 1, no. 2, pp. 900–907, Jul.2016, doi: 10.1109/LRA.2016.2528294.
    [14] S. Seok et al., “Design principles for energy-efficient legged locomotion and implementation on the MIT Cheetah robot,” IEEE/ASME Trans. Mechatronics, vol. 20, no. 3, pp. 1117–1129, Jun.2015, doi: 10.1109/TMECH.2014.2339013.
    [15] H. Q. Vu, X. Yu, F. Iida, and R. Pfeifer, “Improving energy efficiency of hopping locomotion by using a variable stiffness actuator,” IEEE/ASME Trans. Mechatronics, vol. 21, no. 1, pp. 472–486, Feb.2016, doi: 10.1109/TMECH.2015.2428274.
    [16] X. Liu, A. Rossi, and I. Poulakakis, “A switchable parallel elastic actuator and its application to leg design for running robots,” IEEE/ASME Trans. Mechatronics, vol. 23, no. 6, pp. 2681–2692, Dec.2018, doi: 10.1109/TMECH.2018.2871670.
    [17] F. Guenther, H. Q. Vu, and F. Iida, “Improving legged robot hopping by using coupling-based series elastic actuation,” IEEE/ASME Trans. Mechatronics, vol. 24, no. 2, pp. 413–423, Apr.2019, doi: 10.1109/TMECH.2019.2893235.
    [18] H. K. Shin and B. K. Kim, “Energy-efficient gait planning and control for biped robots utilizing vertical body motion and allowable ZMP region,” IEEE Trans. Ind. Electron., vol. 62, no. 4, pp. 2277–2286, 2015, doi: 10.1109/TIE.2014.2360152.
    [19] R. C. Luo and C. C. Chen, “Biped walking trajectory generator based on three-mass with angular momentum model using model predictive control,” IEEE Trans. Ind. Electron., vol. 63, no. 1, pp. 268–276, Jan. 2016, doi: 10.1109/TIE.2015.2477062.
    [20] J. Oh, O. Sim, B. Cho, K. K. Lee, and J. H. Oh, “Online Delayed Reference Generation for a Humanoid Imitating Human Walking Motion,” IEEE/ASME Trans. Mechatronics, vol. 26, no. 1, pp. 102–112, Feb.2021, doi: 10.1109/TMECH.2020.3002396.
    [21] B. Ugurlu and A. Kawamura, “ZMP-based online jumping pattern generation for a one-legged robot,” IEEE Trans. Ind. Electron., vol. 57, no. 5, pp. 1701–1709, May 2010, doi: 10.1109/TIE.2009.2032439.
    [22] B. Ugurlu and A. Kawamura, “On the backward hopping problem of legged robots,” IEEE Trans. Ind. Electron., vol. 61, no. 3, pp. 1632–1634, 2014, doi: 10.1109/TIE.2013.2259776.
    [23] K. Ishihara, T. D. Itoh, and J. Morimoto, “Full-Body Optimal Control Toward Versatile and Agile Behaviors in a Humanoid Robot,” IEEE Robot. Autom. Lett., vol. 5, no. 1, pp. 119–126, Jan. 2020, doi: 10.1109/LRA.2019.2947001.
    [24] A. A. Saputra, J. Botzheim, I. A. Sulistijono, and N. Kubota, “Biologically Inspired Control System for 3-D Locomotion of a Humanoid Biped Robot,” IEEE Trans. Syst. Man, Cybern. Syst., vol. 46, no. 7, pp. 898–911, Jul. 2016, doi: 10.1109/TSMC.2015.2497250.
    [25] J. Wang, W. Qin, and L. Sun, “Terrain adaptive walking of biped neuromuscular virtual human using deep reinforcement learning,” IEEE Access, vol. 7, pp. 92465–92475, 2019, doi: 10.1109/ACCESS.2019.2927606.
    [26] A. Xi and C. Chen, “Walking control of a biped robot on static and rotating platforms based on hybrid reinforcement learning,” IEEE Access, vol. 8, pp. 148411–148424, 2020, doi: 10.1109/ACCESS.2020.3015506.
    [27] “RoboCup Humanoid League | Official website of the RoboCup Humanoid League.” https://humanoid.robocup.org/.
    [28] Y. Ding and H. W. Park, “Design and experimental implementation of a quasi-direct-drive leg for optimized jumping,” in IEEE International Conference on Intelligent Robots and Systems, Dec. 2017, vol. 2017-September, pp. 300–305, doi: 10.1109/IROS.2017.8202172.
    [29] “ROBOTIS.” https://www.robotis.com/ (accessed Jun. 10, 2021).
    [30] “DYNAMIXEL All-in-one Smart Actuator.” https://www.robotis.us/dynamixel/.
    [31] “MTi_1_Series_dk_user_manual.” https://www.xsens.com/hubfs/Downloads/Manuals/MTi_1_Series_dk_user_manual.pdf.
    [32] “SPECIFICATION RSP-2000-12 RSP-2000-24 MODEL DC VOLTAGE RATED CURRENT CURRENT RANGE.” Accessed: Jun.20, 2021. [Online]. Available: https://www.meanwell.com/serviceDisclaimer.aspx.
    [33] “480W 15A 0 36V Adjustable Switching Power Supply HJS 480 0 36 DC voltage regulated SMPS with Digital Display 110/220VAC|Switching Power Supply| - AliExpress.” https://www.aliexpress.com/i/4000060344208.html.
    [34] T.-H. S.Li, Y. F. Ho, P. H. Kuo, Y. T. Ye, and L. F. Wu, “Natural walking reference generation based on double-link LIPM gait planning algorithm,” IEEE Access, vol. 5, pp. 2459–2469, 2017, doi: 10.1109/ACCESS.2017.2669209.
    [35] W.-C. Chen, Kicking balance control of humanoid robot and its application to multi-goal game, Master Thesis, National Cheng Kung University, Tainan, Taiwan, 2016.
    [36] D. J. Braun, J. E. Mitchell, and M. Goldfarb, “Actuated dynamic walking in a seven-link biped robot,” IEEE/ASME Trans. Mechatronics, vol. 17, no. 1, pp. 147–156, Feb.2012, doi: 10.1109/TMECH.2010.2090891.
    [37] B. Espiau and R. Boulic, “On the computation and control of the mass center of articulated chains.” Accessed: Jun.25, 2021. [Online]. Available: https://hal.inria.fr/inria-00073210.
    [38] R. S. Sutton, D. Mcallester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Proc. Neural Information Processing Systems 1999, Available:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.6.696&rep=rep1&type=pdf
    [39] T. P.Lillicrap et al., “Continuous control with deep reinforcement learning.” Accessed: Jun. 17, 2021. [Online]. Available: https://goo.gl/J4PIAz.
    [40] S. Fujimoto, H. VanHoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” 2018. Accessed: Jun. 17, 2021. [Online]. Available: https://github.com/.
    [41] “Quality motion capture in one simple suit.” https://www.rokoko.com/products/smartsuit-pro.

    下載圖示
    2026-09-02公開
    QR CODE