隨著工業 4.0 的推進，製造業對於自動化系統的彈性需求日益增加，傳統機械手臂示教依賴人工操作，難以應對未知環境或非結構化之工件。本研究旨在開發一套整合「立體視覺」與「機械手臂路徑規劃」的自主化自動化系統，專注於解決 CNC 工具機在無預先建模環境下的自動上下料作業問題，使機械手臂在無預先離線編程或是人工教導的情況下，也能夠自主感知環境並建立上下料路徑。在建立環境地圖方面，本計畫採用六軸協作型手臂搭配深度相機進行手眼校正與環境掃描。針對點雲數據處理，引入體素網格降採樣技術以提升運算效率，並結合 KNN、RANSAC、切片投影與 DBSCAN 演算法進行特徵提取，精確識別工具機門板與障礙物。上下料路徑規劃方面，利用 RRT* 以及Bi-RRT演算法在空間中生成無碰撞路徑，並透過碰撞檢測確保移動安全。本研究最終建立了從 PyBullet 物理模擬轉移到實體控制器的技術，成功將上下料路徑規劃結果轉譯為手臂控制器可讀取之 G-code 。實驗結果證實，本系統能使機械手臂有效識別未知環境特徵並生成安全路徑，建立了一套機械手臂自主建立工作環境點雲地圖並規劃上下料路徑之系統，20分鐘內可完成環境感知與路徑規劃，使機械手臂能自主完成從環境重建到實體加工的完整任務。

With the advancement of Industry 4.0, the manufacturing sector's demand for flexible automation systems has increasingly grown. Traditional robotic arm teaching relies heavily on manual operation, making it difficult to adapt to unknown environments or unstructured workpieces. This study aims to develop an autonomous automation system that integrates stereo vision with robotic path planning, specifically focusing on solving the automated loading and unloading (machine tending) of CNC machine tools in model-free environments. This approach enables the robotic arm to autonomously perceive its surroundings and generate operational paths without the need for prior offline programming or manual teaching.
For environment mapping, this project utilizes a 6-DOF collaborative robot equipped with a depth camera to perform hand-eye calibration and environmental scanning. In terms of point cloud data processing, voxel grid downsampling is introduced to enhance computational efficiency. This is combined with K-Nearest Neighbors (KNN), Random Sample Consensus (RANSAC), slice projection, and Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithms for feature extraction, allowing for the precise identification of machine tool doors and obstacles. For path planning, RRT* and Bidirectional RRT (Bi-RRT) algorithms are employed to generate collision-free paths in the workspace, with continuous collision detection ensuring motion safety.Ultimately, this study established a framework to transfer the PyBullet physics simulation to a physical controller, successfully translating the machine tending path planning results into G-code readable by the robotic controller. Experimental results demonstrate that the proposed system effectively identifies unknown environmental features and generates safe trajectories. The developed system empowers the robotic arm to autonomously construct point cloud maps of its workspace and plan machine tending paths. The entire process—from environmental perception to path generation—can be completed within 20 minutes, enabling the robot to autonomously execute the complete workflow from 3D environment reconstruction to physical machining operations.

[1] A. Abdalwhab, G. Beltrame, S. E. Kahou, and D. St-Onge, "Learning multi-agent multi-machine tending by mobile robots," arXiv preprint arXiv:2408.16875, 2024.
[2] O. Heimann, S. Niemann, A. Vick, C. Schorn, and J. Krüger, "Mobile Machine Tending with ROS2: Evaluation of system capabilities," in 2023 IEEE 28th International Conference on Emerging Technologies and Factory Automation (ETFA), 2023: IEEE, pp. 1–4.
[3] H. Huang, L. Zheng, Z. Liang, M. Yuan, and R. Shu, "Design and Analysis of Loading and Unloading Robot for CNC Machine," presented at the 2023 5th International Symposium on Robotics & Intelligent Manufacturing Technology (ISRIMT), 2023.
[4] L.-C. Tung , Shu-Hung Chen, Yu-Cheng Ye, Lin-Yin Chen, Yu-Rou Tu, Yen-Wen Lai, "Implementation of Multi-process Automatic Loading and Unloading Production System," presented at the 2023 International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan), 2023.
[5] L. Wang et al., "Safety optimization of grab unloaders based on machine vision and 3D coordinate system reconstruction," Sci Rep, vol. 15, no. 1, p. 36559, Oct 21 2025,
[6] J. Niu and J. Liu, "Research on CNC Machine Tool Industrial Robot Based on Computer Artificial Intelligence Technology," presented at the 2023 IEEE 6th International Conference on Automation, Electronics and Electrical Engineering (AUTEEE), 2023.
[7] Y. Zhang and H. Wang, "Redundancy-Based Motion Planning with Task Constraints for Robot Manipulators," Sensors, vol. 25, no. 6, Mar 19 2025.
[8] H. Zhu, Y. Wang, D. Huang, W. Ye, W. Ouyang, and T. He, "Point cloud matters: Rethinking the impact of different observation spaces on robot learning," Advances in Neural Information Processing Systems, vol. 37, pp. 77799–77830, 2024.
[9] H. Duan, P. Wang, Y. Huang, G. Xu, W. Wei, and X. Shen, "Robotics dexterous grasping: The methods based on point cloud and deep learning," Frontiers in Neurorobotics, vol. 15, p. 658280, 2021.
[10] L. Zhou, G. Wu, Y. Zuo, X. Chen, and H. Hu, "A comprehensive review of vision-based 3D reconstruction methods," Sensors, vol. 24, no. 7, p. 2314, 2024.
[11] C. Cheng, Z. Liu, and M. H. Ang Jr, "3D Reconstruction Techniques in the Manufacturing Domain: Applications, Research Opportunities and Use Cases," arXiv preprint arXiv:2604.28064, 2026.
[12] R. B. Rusu and S. Cousins, "3d is here: Point cloud library (pcl)," in 2011 IEEE international conference on robotics and automation, 2011: IEEE, pp. 1–4.
[13] H. Cao, D. Chen, Z. Zheng, Y. Zhang, H. Zhou, and J. Ju, "Fast point cloud registration method with incorporation of RGB image information," Applied Sciences, vol. 13, no. 8, p. 5161, 2023.
[14] Y. Zhou and O. Tuzel, "Voxelnet: End-to-end learning for point cloud based 3d object detection," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4490–4499.
[15] H. Zhou, P. Ping, Q. Shi, and H. Chen, "An adaptive two-dimensional voxel terrain mapping method for structured environment," Sensors, vol. 23, no. 23, p. 9523, 2023.
[16] D. Wu, Z. Chen, T. Tan, Z. Yang, S. Wang, and J. Guo, "A Method for Point Cloud Segmentation and Registration-based Localization of Highly-Reflective Scattered Workpieces," in 2025 12th International Forum on Electrical Engineering and Automation (IFEEA), 2025: IEEE, pp. 262–266.
[17] G. Liang, X. Cui, D. Yuan, L. Zhang, and R. Yang, "An Improved Point Cloud Filtering Algorithm Applies in Complex Urban Environments," Remote Sensing, vol. 17, no. 8, 2025.
[18] "<paper_5c4d6db2805d5ab3973ad7dd62fe004a.pdf>."
[19] R. B. Rusu, Z. C. Marton, N. Blodow, M. Dolha, and M. Beetz, "Towards 3D point cloud based object maps for household environments," Robotics and Autonomous Systems, vol. 56, no. 11, pp. 927–941, 2008.
[20] 吳佳沅, "以點雲分割技術進行焊道辨識與機械手臂研磨路徑規劃," 2025.
[21] P. E. Hart, N. J. Nilsson, and B. Raphael, "A formal basis for the heuristic determination of minimum cost paths," IEEE transactions on Systems Science and Cybernetics, vol. 4, no. 2, pp. 100–107, 1968.
[22] M. Likhachev, G. J. Gordon, and S. Thrun, "ARA*: Anytime A* with provable bounds on sub-optimality," Advances in neural information processing systems, vol. 16, 2003.
[23] D. Dolgov, S. Thrun, M. Montemerlo, and J. Diebel, "Path planning for autonomous vehicles in unknown semi-structured environments," The international journal of robotics research, vol. 29, no. 5, pp. 485–501, 2010.
[24] J. Schulman et al., "Motion planning with sequential convex optimization and convex collision checking," The International Journal of Robotics Research, vol. 33, no. 9, pp. 1251–1270, 2014.
[25] L. E. Kavraki, P. Svestka, J.-C. Latombe, and M. H. Overmars, "Probabilistic roadmaps for path planning in high-dimensional configuration spaces," IEEE transactions on Robotics and Automation, vol. 12, no. 4, pp. 566–580, 1996.
[26] R. Bohlin and L. E. Kavraki, "Path planning using lazy PRM," in Proceedings 2000 ICRA. Millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065), 2000, vol. 1: IEEE, pp. 521–528.
[27] B. Ichter, J. Harrison, and M. Pavone, "Learning sampling distributions for robot motion planning," in 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018: IEEE, pp. 7087–7094.
[28] M. Kalakrishnan, S. Chitta, E. Theodorou, P. Pastor, and S. Schaal, "STOMP: Stochastic trajectory optimization for motion planning," in 2011 IEEE international conference on robotics and automation, 2011: IEEE, pp. 4569–4574.
[29] N. Ratliff, M. Zucker, J. A. Bagnell, and S. Srinivasa, "CHOMP: Gradient optimization techniques for efficient motion planning," in 2009 IEEE international conference on robotics and automation, 2009: IEEE, pp. 489–494.
[30] M. Zucker, N Ratliff, M Pivtoraiko, M Klingensmith, "Chomp: Covariant hamiltonian optimization for motion planning," The International journal of robotics research, vol. 32, no. 9-10, pp. 1164–1193, 2013.
[31] M. Elbanhawi and M. Simic, "Sampling-based robot motion planning: A review," Ieee access, vol. 2, pp. 56–77, 2014.
[32] A. Sepehri and A. M. Moghaddam, "A motion planning algorithm for redundant manipulators using rapidly exploring randomized trees and artificial potential fields," IEEE Access, vol. 9, pp. 26059–26070, 2021.
[33] N. M. Amato, O. B. Bayazit, L. K. Dale, C. Jones, and D. Vallejo, "OBPRM: An obstacle-based PRM for 3D workspaces," in Proceedings of the third workshop on the algorithmic foundations of robotics on Robotics: the algorithmic perspective: the algorithmic perspective, 1998, pp. 155–168.
[34] J. J. Kuffner and S. M. LaValle, "RRT-connect: An efficient approach to single-query path planning," in Proceedings 2000 ICRA. Millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065), 2000, vol. 2: IEEE, pp. 995–1001.
[35] L. E. Kavraki, P. Svestka, J.-C. Latombe, and M. H. Overmars, "Probabilistic roadmaps for path planning in high-dimensional configuration spaces," IEEE transactions on Robotics and Automation, vol. 12, no. 4, pp. 566–580, 2002.
[36] J. Liu, H. J. Yap, and A. S. M. Khairuddin, "Review on motion planning of robotic manipulator in dynamic environments," Journal of Sensors, vol. 2024, no. 1, p. 5969512, 2024.
[37] S. LaValle, "Rapidly-exploring random trees: A new tool for path planning," Research Report 9811, 1998.
[38] L. Ye, J. Duan, Z. Yang, X. Zou, M. Chen, and S. Zhang, "Collision-free motion planning for the litchi-picking robot," Computers and electronics in agriculture, vol. 185, p. 106151, 2021.
[39] S. Karaman and E. Frazzoli, "Sampling-based algorithms for optimal motion planning," The international journal of robotics research, vol. 30, no. 7, pp. 846–894, 2011.
[40] J. D. Gammell, S. S. Srinivasa, and T. D. Barfoot, "Informed RRT*: Optimal sampling-based path planning focused via direct sampling of an admissible ellipsoidal heuristic," in 2014 IEEE/RSJ international conference on intelligent robots and systems, 2014: IEEE, pp. 2997–3004.
[41] D. Soleymanzadeh, I. Lopez-Sanchez, H. Su, Y. Li, X. Liang, and M. Zheng, "Towards generalist neural motion planners for robotic manipulators: Challenges and opportunities," IEEE Transactions on Automation Science and Engineering, 2026.
[42] M. G. Tamizi, M. Yaghoubi, and H. Najjaran, "A review of recent trend in motion planning of industrial robots," International Journal of Intelligent Robotics and Applications, vol. 7, no. 2, pp. 253–274, 2023.
[43] X. He, Y. Zhou, H. Liu, and W. Shang, "Improved RRT*-connect manipulator path planning in a multi-obstacle narrow environment," Sensors, vol. 25, no. 8, p. 2364, 2025.
[44] A. Yershova, L. Jaillet, T. Siméon, and S. M. LaValle, "Dynamic-domain RRTs: Efficient exploration by controlling the sampling domain," in Proceedings of the 2005 IEEE international conference on robotics and automation, 2005: IEEE, pp. 3856–3861.
[45] C. Urmson and R. Simmons, "Approaches for heuristically biasing RRT growth," in Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003)(Cat. No. 03CH37453), 2003, vol. 2: IEEE, pp. 1178–1183.
[46] X. Yan, T. Xiong, D. Xiang, D. Zhao, and Z. Li, "Obstacle Avoidance Path Planning for Robotic Manipulators Based on an Improved Bidirectional RRT* Algorithm," in 2025 7th International Conference on Robotics, Intelligent Control and Artificial Intelligence (RICAI), 2025: IEEE, pp. 99–103.
[47] D. Soleymanzadeh, I. Lopez-Sanchez, H. Su, Y. Li, X. Liang, and M. Zheng, "Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunities," IEEE Transactions on Automation Science and Engineering, vol. 23, pp. 4488–4531, 2026.
[48] J. Zhang, "Research on Automatic Loading and Unloading Path Design and Fixture Design for Industrial Robots," presented at the 2024 International Conference on Artificial Intelligence and Digital Technology (ICAIDT), 2024.
[49] " Milind R. Shinde, <Designing_a_suitable_robotic_arm_for_load.pdf>."
[50] Y. Liu , Chunlai He, Weijun Wang, Wei Sun, Shujian Liu, Jiyong Fan, Huarui Liu, "Loading and Unloading Robot Inverse Kinematics Solution Optimization Algorithm," presented at the 2024 8th International Conference on Robotics, Control and Automation (ICRCA), 2024.
[51] X. Liu, T. Luo, J. Wang, and Y. Ma, "Path Planning of Loading and Unloading Robots Based on Optimal Energy Consumption," presented at the 2023 2nd International Conference on Futuristic Technologies (INCOFT), 2023.
[52] J. Zhao, X. Zhu, and T. Song, "Serial Manipulator Time-Jerk Optimal Trajectory Planning Based on Hybrid IWOA-PSO Algorithm," IEEE Access, vol. 10, pp. 6592–6604, 2022.
[53] R. Y. Tsai and R. K. Lenz, "A new technique for fully autonomous and efficient 3 d robotics hand/eye calibration," IEEE Transactions on robotics and automation, vol. 5, no. 3, pp. 345–358, 1989.
[54] M. Muja and D. G. Lowe, "Scalable nearest neighbor algorithms for high dimensional data," IEEE transactions on pattern analysis and machine intelligence, vol. 36, no. 11, pp. 2227–2240, 2014.
[55] K. Khoshelham and S. O. Elberink, "Accuracy and resolution of kinect depth data for indoor mapping applications," sensors, vol. 12, no. 2, pp. 1437–1454, 2012.
[56] Alexander Sabov, Jörg Krüger,"< Identification and Correction of Flying Pixels in Range Camera Data.pdf>."
[57] J. L. Bentley, "Multidimensional binary search trees used for associative searching," Communications of the ACM, vol. 18, no. 9, pp. 509–517, 1975.
[58] T. Cover and P. Hart, "Nearest neighbor pattern classification," IEEE transactions on information theory, vol. 13, no. 1, pp. 21–27, 1967.
[59] H. Hoppe, T. DeRose, T. Duchamp, J. McDonald, and W. Stuetzle, "Surface reconstruction from unorganized points," in Proceedings of the 19th annual conference on computer graphics and interactive techniques, 1992, pp. 71–78.
[60] P. J. Besl and N. D. McKay, "Method for registration of 3-D shapes," in Sensor fusion IV: control paradigms and data structures, 1992, vol. 1611: Spie, pp. 586–606.
[61] M. A. Fischler and R. C. Bolles, "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography," Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981.
[62] R. Schnabel, R. Wahl, and R. Klein, "Efficient RANSAC for point‐cloud shape detection," in Computer graphics forum, 2007, vol. 26, no. 2: Wiley Online Library, pp. 214–226.
[63] Z. Qiao, H. Hu, W. Shi, S. Chen, Z. Liu, and H. Wang, "A registration-aided domain adaptation network for 3D point cloud based place recognition," in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021: IEEE, pp. 1317–1322.
[64] J. Zhao, Y. Cui, X. Niu, X. Wang, Y. Zhao, M. Guo, and R. Zhang, "Point cloud slicing-based extraction of indoor components," The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 48, pp. 103–108, 2022.
[65] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, "A density-based algorithm for discovering clusters in large spatial databases with noise," in kdd, 1996, vol. 96, no. 34, pp. 226–231.
[66] J. Guo, L. Fu, M. Jia, K. Wang, and S. Liu, "Fast and robust bin-picking system for densely piled industrial objects," in 2020 Chinese Automation Congress (CAC), 2020: IEEE, pp. 2845–2850.
[67] E. Coumans and Y. Bai, "Pybullet, a python module for physics simulation for games, robotics and machine learning," ed, 2016.