近年來，隨著工業自動化的快速發展，工廠中機械手臂的需求增加。對於控制機械手臂的要求也在不斷變化。從依賴手動操作，到使用傳統的電腦視覺方法，隨著人工智慧越來越備受關注，研究學者越來越多地使用深度學習的方法來控制機器人手臂執行特定任務。然而，這些方法存在一些限制，包括繁重和耗時的數據標註過程，以及在不同環境中遇到未知物體時準確性降低。此外，這些方法通常在訓練過程中需要專業人員進行手動參數調整。
為了克服這些缺點，本研究提出了基於深度強化學習的機械手臂抓取方法。此方法不僅能夠實現機械手臂的持續自我學習，不需要大量的數據標註，還能解決深度強化學習中的一個很大的挑戰，也就是收斂速度緩慢和學習時間長。為了達成這一目標，本研究採用了改良版的對決雙重深度Q網路 (Dueling Double Deep Q Network)，並在網絡架構中引入了自注意機制 (Self-attention)以增強網路的效能。巧妙地結合了深度學習中的目標檢測網絡，使得機器人手臂能夠同時學習物體抓取和目標物體分類，只需要少量的人工標註數據。另外，該方法還加快了深度強化學習網絡的訓練和收斂速度。多次實驗證明，該方法可以提高抓取成功率，並在訓練過程中更快地達到一定的準確性。
總而言之，本研究提出的演算法不僅克服了傳統方法的不足之處，還提高了機械手臂的夾取性能，為機械手臂夾取任務提供了一個新的方向。

In recent years, there has been a significant increase in the demand for industrial robotic arms, particularly with the rapid development of industrial automation. The requirements for controlling robotic arms have also been constantly evolving. From relying solely on manual operation to using traditional computer vision methods, and now with the rise of artificial intelligence, researchers are increasingly exploring deep learning methods to control robotic arms for specific tasks. However, these methods have limitations, including the labor-intensive and time-consuming data annotation process and reduced accuracy when encountering unknown objects in different environments. Moreover, these methods often require manual parameter adjustments by experts during the training process.
To overcome these limitations, this study proposes a deep reinforcement learning-based approach for robotic arm grasping tasks. This method not only enables continuous learning of the robotic arm in an unsupervised method, without the need for data annotation but also addresses one of the major challenges of deep reinforcement learning, which is slow convergence and long learning times. To achieve this goal, an improved version of the Dueling Double Deep Q Network is employed, along with self-attention mechanisms within the network architecture to enhance its performance. Additionally, this method cleverly combines object detection networks from deep learning, allowing the robotic arm to simultaneously learn object grasping and target object classification using a small amount of annotated data. Furthermore, this approach accelerates the training and convergence speed of the deep reinforcement learning network. Multiple experiments have demonstrated that this method can improve grasping success rates and achieve satisfactory performance more rapidly during the training process.
In summary, the proposed method in this study not only overcomes the limitations of traditional approaches but also enhances the control performance of robotic arms, presenting broad prospects for industrial applications.

[1]: Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., & Funkhouser, T. (2018, October). Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 4238-4245). IEEE.
[2]: Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.
[3]: Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., ... & Hassabis, D. (2015). Human-level control through deep reinforcement learning. nature, 518(7540), 529-533.
[4]: Van Hasselt, H., Guez, A., & Silver, D. (2016, March). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 30, No. 1).
[5]: Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., & Freitas, N. (2016, June). Dueling network architectures for deep reinforcement learning. In International conference on machine learning (pp. 1995-2003). PMLR.
[6]: Kroemer, O., Niekum, S., & Konidaris, G. (2021). A review of robot learning for manipulation: Challenges, representations, and algorithms. The Journal of Machine Learning Research, 22(1), 1395-1476.
[7]: Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018). Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5), 421-436.
[8]: Kumra, S., & Kanan, C. (2017, September). Robotic grasp detection using deep convolutional neural networks. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 769-776). IEEE.
[9]: Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized experience replay. arXiv preprint arXiv:1511.05952.
[10]: Mohammed, M. Q., Chung, K. L., & Chyi, C. S. (2020). Pick and place objects in a cluttered scene using deep reinforcement learning. Int. J. Mech. Mechatron. Eng, 20(04), 50-57.
[11]: Ren, D., Ren, X., Wang, X., Digumarti, S. T., & Shi, G. (2021, September). Fast-Learning Grasping and Pre-Grasping via Clutter Quantization and Q-map Masking. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 3611-3618). IEEE.
[12]: Hundt, A., Killeen, B., Greene, N., Wu, H., Kwon, H., Paxton, C., & Hager, G. D. (2020). “good robot!”: Efficient reinforcement learning for multi-step visual tasks with sim to real transfer. IEEE Robotics and Automation Letters, 5(4), 6724-6731.
[13]: Xu, K., Yu, H., Lai, Q., Wang, Y., & Xiong, R. (2021). Efficient learning of goal-oriented push-grasping synergy in clutter. IEEE Robotics and Automation Letters, 6(4), 6337-6344.
[14]: Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.
[15]: Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 580-587).
[16]: Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708).
[17]: Yang, Y., Ni, Z., Gao, M., Zhang, J., & Tao, D. (2021). Collaborative pushing and grasping of tightly stacked objects via deep reinforcement learning. IEEE/CAA Journal of Automatica Sinica, 9(1), 135-145.
[18]: Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009, June). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248-255). Ieee.
[19]: Rohmer, E., Singh, S. P., & Freese, M. (2013, November). V-REP: A versatile and scalable robot simulation framework. In 2013 IEEE/RSJ international conference on intelligent robots and systems (pp. 1321-1326). IEEE.
[20]: Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431-3440).
[21]: Redmon, J., & Angelova, A. (2015, May). Real-time grasp detection using convolutional neural networks. In 2015 IEEE international conference on robotics and automation (ICRA) (pp. 1316-1322). IEEE.
[22]: Boularias, A., Bagnell, J., & Stentz, A. (2015, February). Learning to manipulate unknown objects in clutter by reinforcement. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 29, No. 1).