近年來，由於電腦視覺方法不斷精進、CPU計算能力大幅提升且影像擷取裝置售價持續降低，視覺伺服技術逐漸被應用於各類型之工業用機械手臂，使得生產製造更加具有彈性。然而，將視覺伺服技術應用於搭載機械手臂之全向型自走車系統之相關研究卻仍是方興未艾。搭載機械手臂之自走車系統可運用於產線間之整合、倉儲管理或大賣場之顧客協助等，其重要性與日俱增。因此，本論文著重在將視覺伺服技術應用於搭載關節型機械手臂之全向型自走車系統KUKA youBot，期能導引搭載機械手臂之自走車系統至目標位置後進行物件夾取。傳統基於影像之視覺伺服技術(IBVS)架構容易因為偵測到影像特徵點位於影像平面邊緣時，攝影機快速移動使得目標物離開攝影機視野範圍，導致整個系統發散。為改善上述問題並提升收斂速度，本論文採用Q-learning與IBVS切換式控制器以及利用Q-learning調變增益之IBVS架構進行實作。由於Q-learning屬於機器學習中的增強式學習，能在未來實驗中根據環境的反饋資訊持續改善，整個系統兼具強健性以及環境適應能力。

Due to facts such as improved computer vision techniques, increasing computation power of CPU, low cost image capturing devices, the idea of visual servoing has been gradually adopted in various types of robot manipulators such that the manufacturing process can be even more flexible. It is not surprising that most manufacturing powerhouses of industrial robot manipulators have focused on developing computer vision modules for their robot manipulators. However, the idea of applying visual servoing techniques to a mobile robot system equipped with a robot arm is still a developing research topic. Since a mobile robot equipped with a robot arm can provide more flexibility in industrial manufacturing and Logistics/Warehouse management so as to combine different production lines, its importance cannot be overlooked. Therefore, this thesis aims at developing visual servoing techniques for a mobile robot equipped with a 5-axis robot manipulator. In general, problems of instability always exist in the Image-Based Visual Servoing (IBVS) structure. One important research direction is how to keep the visual features within the field of view of the camera. In order to cope with the divergence problem and increase the speed of response, Q-learning is taken for IBVS gain adjustment. Q-learning is a model-free reinforcement learning algorithm, and makes learning agents interact with the environment to learn optimal policy. The Q-learning algorithm enables the robot system to learn and continuously improve its policy online. In this thesis, an IBVS control scheme using Q-learning to tune the gain was developed on a mobile robot system equipped with a manipulator. The control scheme provides high flexibility, robustness and effective performance.

[1] https://zh.wikipedia.org/wiki/%E8%87%AA%E5%8A%A8%E5%8C%96%E6%8A%80%E6%9C%AF
[2] T. Howard, M. Pivtoraiko, R. A. Knepper, and A. Kelly, “Model-Predictive Motion Planning: Several Key Developments for Autonomous Mobile Robots”, IEEE Trans. on Robot. Automat., vol.21, no. 1, pp.64-73, Mar. 2014.
[3] S. Hutchinson, G. D. Hager, and P. I. Corke, “Visual servo control – Part I : Basic approaches, ” IEEE Trans. on Robot. Automat., vol. 13, no. 4, pp. 82-90, Dec. 2006.
[4] S. Hutchinson, G. D. Hager, and P. I. Corke, “A tutorial on visual servo control, ” IEEE Trans. on Robot. Automat., vol. 12, no. 5, pp. 651-670, Oct. 1996.
[5] J. Yuan, “Local SVD inverse of robot Jacobians,” Robotica, vol. 19, pp. 79-86, Jan. 2001.
[6] R.S.Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, 1998.
[7] J. Hill and W. T. Park, “Real time control of a robot with a mobile camera,” in Proc. 9th ISIR, Washington, U.S.A., Mar. 1979, pp. 233 -246.
[8] P. I. Croke and S. A. Hutchinson, “A new partitioned approach to image-based visual servo control,” IEEE Trans. Robot. Automat., vol. 15, no.2, pp.238-250, Apr.1999.
[9] E. Malis and F. Chaumette, “2 1/2 D visual servoing with respect to unknown objects through a new estimation scheme of camera displacement,” Int. J. of Comput. Vision, vol. 37, no.1, pp. 79-97, Jun. 2000.
[10] T. Hamel and R. Mahony, “Visual servoing of an under-actuated dynamic rigid-body system: an image-based approach,” IEEE Trans. Robot. Automat., vol. 18, no. 2, pp. 187-189, Aug. 2002.
[11] W. J. Wilson, C. C. Williams Hulls, and G. S. Bell, “Relative end-effector control using Cartesian position based visual servoing,” IEEE Trans. Robot. Automat., vol.12, no. 237, pp.684-696, Aug. 2002.
[12] F. Chaumette, “Potential problems of stability and convergence in image-based and position- based visual servoing,” Lecture Notes in Control and Information Sciences, Springer, German, 1998, vol. 237, pp. 66-78.
[13] Q. Fang, P. M. Meaney, and K. D. Paulsen, “Singular value analysis of the Jacobian matrix in microwave image reconstruction,” IEEE Trans. Anten.Propo., vol.54, no. 8, pp. 2371-2380, Aug. 2006.
[14] E. Malis, F. Chaumette, and S. Boudet, “2½D visual servoing,” IEEE Trans. Robot. Automat., vol. 15, pp. 238-250, Apr. 1999.
[15] F. Chaumette and E. Malis,” 2 1/2 D visual servoing: a possible solution to improve image-based and position-based visual servoing,” in Proc. of the IEEE Conf. on Robot. Automat., San Francisco, CA, 2000, pp. 630-635.
[16] P. I. Croke and S. A. Hutchinson, “A new hybrid image-based visual servo control scheme,” in Proc. IEEE Conf. Decision and Control, Sydney, NSW, 2000, pp.2521-2526.
[17] N. Gans and S. Hutchinson, “An asymptotically stable switched system visual controller for eye in hand robots,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots Systems, Las Vegas, NV, Oct. 2003, pp. 735–742.
[18] A. Muis and K. Ohnishi, “Eye-to-hand approach on eye-in-hand configuration within real-time visual servoing,” IEEE/ASME Trans. Mechatron., vol. 10, no. 4, pp. 404-410, Sep. 2005.
[19] R. Horaud and F. Dornaika, “Hand-eye calibration,” Int. J. Robot .Res, vol. 14, no. 3, pp. 195-210, Jun. 1995.
[20] K. H. Strobl and G. Hirzinger, “Optimal hand-eye calibration,” in Proc. IEEE/RSJ Int. Conf. IROS, Beijing, China, Oct. 2006, pp. 4647-4653.
[21] C.C. Wang, “Extrinsic calibration of a vision sensor mounted on a robot,” IEEE Trans. Robot. Automat., vol. 8, no.2, pp. 161-175, Apr. 1992.
[22] F. Dornaika and R. Horaud, “Simultaneous robot-world and hand-eye calibration,” IEEE Trans. Robot. Automat., vol. 14, no. 4, pp. 617-622, Aug. 1998.
[23] F. Chaumette, S. Boukir, P. Bouthemy, and D. Juvin, “Structure from controlled motion,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 18, no. 5, pp. 492–504, 1996
[24] A. De Luca, G. Oriolo, and P. R. Giordano, “Feature depth observation for image-based visual servoing: Theory and experiments,” Int. J. Robot. Res., vol. 27, no. 10, pp. 1093–1116, 2008.
[25] P. Y. Oh and P. K. Allen, “Visual servoing by patertitioning degrees of freedom,” IEEE Trans. Robot. Automat., vol. 17, no.1, pp.1-27, Feb. 2001.
[26] C. K. Wang, M. Y. Cheng, C. H. Liao, C. C. Li, C. Y. Sun, and M. C. Tsai, “Desing and implementation of a multi-purpose real-time pan-tilt visual tracking system,” in Proc. of the IEEE Conf. on Control Applications, Taipei, Taiwan, Sep. 2004, pp. 107-1084,.
[27] I. Siradjuddin, L. Behera, T. M. McGinnity, and S. Coleman, “Image-Based Visual Servoing of a 7-DOF Robot Manipulator Using an Adaptive Distributed Fuzzy PD Controller,” IEEE/ASME Trans. on Mechatronics, vol. 19, no. 2, pp. 512-523, Mar. 2013.
[28] C. Cai, N. Somani, and A. Knoll, “Orthogonal Image Features for Visual Servoing of a 6-DOF Manipulator With Uncalibrated Stereo Cameras,” IEEE Trans. Robot., vol.32, no. 2, pp.452-461. Mar. 2016.
[29] N. Guenard, T. Hamel, and R. Mahony, “A practical visual servo control for an unmanned aerial vehicle,” IEEE Trans. Robot.,vol. 24, pp.331-340, Apr. 2008.
[30] J. Chen, W. E. Dixon, M. Dawson, and M. McIntyre, “Homograpy-based tracking control of a wheeled mobile robot,” IEEE Trans. Robot., vol. 22, pp. 406-415, Oct. 2003.
[31] M. Marshall, M. Matthews, A. P. Hu, G.McMurray, and H. Lipkin, “Uncalibrated visual servoing for intuitive human guidance of robots,” in Proc. of the IEEE Conf. on Robot. Automat., pp. 4463-4468, Jun. 2012.
[32] https://zh.wikipedia.org/wiki/%E4%BA%BA%E5%B7%A5%E6%99%BA%E8%83%BD
[33] S. M. Weiss and C. A. Kulikowski, in Proc. Computer Systems That Learn, USA, San Francisco:Morgan Kaufmann,1991.
[34] E. Alpaydın, Introduction to Machine Learning. England, London: The MIT Press, 2010.
[35] A. L. Samuel, “Some Studies in Machine Learning Using the Game of Checkers,” IBM Journal of Research and Development, no. 3, Jul. 1959.
[36] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification 2nd Edition, New York, Wiley, 2001.
[37] T. Mitchell, Machine Learning, McGraw Hill, 1997.
[38] S.Weiss and C. Kulikowski, Computer Systems That Learn, Morgan Kaufmann, 1991.
[39] S. B. Kotsiantis, “Supervised Machine Learning: A Review of Classification Techniques”, in Proc. of the Computer Engineering Conf. on Emerging Artificial Intelligence Applications, pp.3-24, Jul. 2007.
[40] G. Hinton, and T.J. Sejnowski, Unsupervised Learning and Map Formation: Foundations of Neural Computation, MIT Press, 1999.
[41] 蔡孟璇，運用辦監督式學習法於小樣本分類，博士論文，國立成功大學工業與資訊管理學系，2012。
[42] Gosavi and Abhijit, Simulation-based Optimization: Parametric Optimization Techniques and Reinforcement, Springer. 2003.
[43] 林敬斌，使用增強式學習法改善一個臺灣的股價指數期貨當沖交易系統，碩士論文，國立臺灣大學資訊工程學系，2009。
[44] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine Learning, vol.8, no. 3, pp.229-256, May 1992.
[45] S. F. Desouky and H. M. Schwartz, “A novel technique to design fuzzy logic controller using Q-learning and genetic algorithm in the pursuit-evation game,” in Proc. IEEE Conf. Sys. Man and Cybernetics, U.S.A., San Antonio, Oct. 2009.
[46] V. Gullapalli, J. A. Franklin, and H. Benbrahim, “Acquiring robot skills via reinforcement learning,” IEEE Control System, vol.14, pp.13-24, Feb. 1994.
[47] C. Chunlin and C. Zonghai, “Reinforcement learning for mobile robot: From reaction to deliberation,” Systems Engineering and Electronics, vol. 16, no. 1, pp.611-617, Sep. 2005.
[48] N. Mohajerin, M.B. Menhaj, and A. Doustmohammadi, “A Reinforcement Learning Fuzzy Controller for the Ball and Plate system,” in Proc. of the IEEE Conf. on Fuzzy System, Barcelona, Spain, Jul. 2010, pp.1-8.
[49] G. Zhao, C. Zhao, and D. Wang, “Tensor product model transformation based integral sliding mode control with reinforcement learning strategy,” in Proc. of 33rd Chinese Control Conference, China, Jul. 2014, pp.77-82.
[50] B. Luo, H. N. Wu, and T. Huang, “Off-Policy Reinforcement Learning for Control Design,” IEEE Trans. on Cybernetics, vol. 45, no. 1, pp.65-76, Jan. 2015.
[51] G. Tesauro, Temporal difference learning and TD-gammon, Communications of the ACM, Mar. 1995.
[52] 賴怡玲，使用增強式學習法建立台灣股價期貨當沖交易決策，碩士論文，國立臺灣大學資訊工程學系，2009。
[53] J. S. Campbell, S. N. Givigi, and H. M. Schwartz, “Multiple Model Q-Learning for Stochastic Asynchronous Rewards,” Intelligent and Robotic Systems, vol. 81, no. 3, pp 407–422, Mar. 2016.
[54] T. M. Marin and T. Duckettin, “Fast Reinforcement Learning for Vision-guided Mobile Robots,” in Proc. IEEE Int. Conf. Robot. Automat., Barcelona, Spain, Apr. 2005.
[55] J. Cunha, R.Serra, N. Lau, L. S. Lopes, and Antonio J.R. Neves, “ Batch reinforcement learning for robotic soccer using the Q-Batch update-rule,” Intell Robot Syst, vol. 80, no. 3, pp. 385-399, Jan. 2015.
[56] A. Konar, I. G. Chakraborty, S. J. Singh, L. C. Jain, and Atulya K. Nagar, “A deterministic improved Q-Learning for path planning of a mobile robot,” IEEE Trans. on System, man, and cybernetics : systems, vol. 43, no. 5, Sep. 2013.
[57] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., New York, NY: Cambridge Univ. Press, 2004.
[58] 林潔君，基於視覺之工業用機械手臂物件夾取研究，碩士論文，國立成功大學電機工程學系，2015。
[59] 李哲良，基於影像之視覺伺服應用於循跡控制研究，碩士論文，國立成功大學電機工程學系，2016。
[60] K. S. Fu et al., “Robot arm kinematics,” in Robotics: Control, Sensing, Vision and Intelligence, ch2, pp.12-81, New York, NY:McGraw-Hill, 2006.
[61] M.W. Spong, S. Jutchinson, and M. Vidyasagar, Robot Modeling and Control, Hoboken, NJ:Wiley, 2006.
[62] Y. Motai and A. Kosaka, “Hand–eye calibration applied to viewpoint selection for robotic vision,” IEEE Trans. Ind. Electro., vol. 55, no. 10, pp. 3731-3741, Mar. 2008.
[63] 江宗錡，六軸關節型機械手臂之手眼校正研究，碩士論文，國立成功大學電機工程學系，2014。
[64] S. Remy et al., “Hand-eye calibration,” in Proc, IEEE/RSJ Int. Conf. Intelligent Robots and Syst., Grenoble, 1997, pp.1057-1065.
[65] F. Dornaika and R. Horaud, “Simultaneous robot-world and hand-eye calibration,” IEEE Trans. Robot. Automat., vol.14, no.4, pp.617-622, Aug. 1998.
[66] K. Levenberg, “A method for the solution of certain non-linear problems in least squares,” Quart. Appl. Math., vol.2, no.2, pp.164-168, Jan. 1944.
[67] D. W. Marquardt, “An algorithm for least-squares estimation of nonlinear parameters,” J. Soc. For Ind. & Appl. Math., vol.11, no.2, pp.431-441, 1963.
[68] Standford open course, “Introduction to Robotics”.
[69] J. J. Craig, Introduction to Robotics : Mechanics and Control, Addison-Wesley, 1989,
[70] L. Sciavicco and B. Siciliano, Modeling and Control of Robot Manipulars, Springer Verlag, 1996.
[71] 吳如峰，工業用六軸機械手臂之基於影像視覺伺服架構研究，碩士論文，國立成功大學電機工程學系，2015。
[72] S. Sharma and G. K. Kraetzschmar, “Unified closed form inverse kinematics for the KUKA youBot,” in Proc. of the IEEE Conf. on Robot., German, May 2012.
[73] S. Chiaverini, B. Siciliano, and O. Egeland, “Review of the damped least-squares inverse kinematics with experimentson an industrial robot manipulator,” IEEE Trans. Ind. Electron., vol. 40, no. 1, pp.23-36, Feb.1993.
[74] https://en.wikipedia.org/wiki/Singular_value_decomposition
[75] http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node21.html
[76] Y. Wang, H. Lang, and C. W. de Silva, “A Hybrid Visual Servo Controller for Robust Grasping by Wheeled Mobile Robots,” IEEE/ASME Trans. on Mechatronics, vol. 15, no. 5, Oct. 2010.
[77] KUKA, KUKA youBot User Manual, version 0.86.1, Aug. 2012.
[78] R. Bischoff, U. Huggenberger, and E. Prassler , “ KUKA youBot - A Mobile Manipulator for Research and Education,” in Proc. IEEE Int. Conf. Robot. Automat., Shanghai, China, Aug. 2011, pp.1-4.
[79] http://wiki.ros.org/
[80] À. Santamaria-Navarro and J. Andrade-Cetto, “Uncalibrated image-based visual servoing,” in Proc. IEEE Int. Conf. Robot. Automat., Karlsruhe, Germany, May 2013, pp.5247-5252.
[81] K. Watanabe, Y. Iwatani, and K. Hashimoto, “Image-based visual PID Control of a micro helicopter using a stationary camera,” Intelligent & Robotic Systems, vol. 78, no.2-3, pp. 381-393, Jan. 2008.
[82] D. Lee, T. Ryan, and H. J. Kim, “Autonomous landing of a VTOL UVA on a moving platform using image-based visual servoing,” in Proc. IEEE Int. Conf. Robot. Automat., Minnesota, USA, May 2012, pp.974-976.
[83] H. Shi, X. Li, K. S. Hwang, Wei Pan, and Genjiu Xu, “Decoupled Visual Servoing with Fuzzy Q-Learning,” IEEE Transactions on Industrial Informatics (Early access articles), Oct. 2016.