簡易檢索 / 詳目顯示

回結果列表
研究生： 陳亭潔
Chen, Ting-Jie
論文名稱： 基於視覺語言模型之高階語意至低階控制：結合代理人架構與數位孿生之居家輔助機器人系統
A Vision-Language-Model-Based High-Level Semantic Request to Low-Level Control Using Agent Architecture and Digital Twin for Robot-Assisted Home
指導教授： 蘇文鈺
Su, Wen-Yu
學位類別： 碩士
Master
系所名稱： 敏求智慧運算學院 - 智慧運算碩士學位學程
MS Degree in Intelligent Computing
論文出版年： 2026
畢業學年度： 114
語文別： 英文
論文頁數： 98
中文關鍵詞： 視覺語言模型 、居家服務機器人 、代理人架構 、數位孿生 、Unity 、ROS2
外文關鍵詞： Vision-Language Model, home-service robot, agent architecture, digital twin, Unity, ROS2
相關次數： 點閱：3  下載：0 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報

    • 本論文旨在設計與實作一套以視覺語言模型驅動之代理人架構為核心的居家服務機器人系統，透過工作流程將使用者的高階自然語言請求轉換為機器人可執行之低階控制指令，使家中使用者能以直覺且自然的方式，委派機器人完成日常任務。

    在系統架構方面，本研究使用 LangGraph 與 A2A（Agent-to-Agent）協議，將推論代理人及感知、導航與控制等功能模組組織為工作流程節點。VLM 驅動的推論代理人不直接產生低階控制指令，而是根據任務狀態與觀察結果選擇後續節點，再由 ROS2 控制模組發布對應命令。為降低居家資料外傳的隱私風險，系統採用可於本地端運行之開源視覺語言模型完成主要推論與決策。

    為驗證系統可行性，本研究於 Unity 建立的室內數位孿生環境中，以「移動抓取」作為代表性示範任務。情境涵蓋目標物不在機器人初始視野或遭受遮擋等情況。為處理上述挑戰，系統結合環境相機提供的全域資訊與車載相機的即時觀察，由推論代理人調整後續行動，完成從自然語言指令到移動抓取的閉環流程。

    在 200 次消融實驗結果顯示，完整系統的成功率最高，達 90%；移除 VLM 觀察決策、候選停靠點排序及物體尺寸估計後，成功率分別降至 78%、44% 與 68%，顯示各模組皆有助於提升任務成功率。

    This thesis designs and implements a home-service robot agentic system based on a Vision-Language Model (VLM)-driven agent architecture, which converts a user’s high-level semantic request, expressed in natural language, into low-level control commands executable by the robot through a workflow, enabling users at home to instruct the robot to do daily tasks in an intuitive and natural way.

    In terms of system architecture, we use LangGraph and the A2A (Agent-to-Agent) protocol to organize the reasoning agent and functional modules for perception, navigation, and control as workflow nodes. The VLM-based reasoning agent does not directly output low-level control commands; instead, it decides which workflow node should be triggered next based on the current task state and observations, and ROS2 control modules then issue the corresponding commands. To reduce the privacy risks of transmitting household data externally, the system adopts an open-source vision-language model that can run on a local device to perform the main reasoning and decision-making processes.

    To verify the feasibility of the system, we use mobile manipulation as a representative demonstration task in an indoor digital twin environment developed with Unity. The demonstration setting includes cases in which the target object lies outside the robot’s initial field of view or is occluded. To address these challenges, the system combines global information provided by environmental cameras with real-time observations from the onboard camera, allowing the reasoning agent to adjust subsequent actions and complete a closed-loop process from natural-language instructions to mobile manipulation.

    Experimental results from 200 ablation trials show that the full system attains the highest success rate of 90%. Removing VLM observation and decision-making, candidate parking-location ranking, or object size estimation reduces the success rate to 78%, 44%, and 68%, respectively, indicating that all three modules contribute to improving the task success rate.

    中文摘要 i Abstract ii 致謝 iv Contents v List of Tables viii List of Figures ix 1 Introduction 1 2 Related Work 3 2.1 Robots for Home Service 3 2.1.1 VLMs for Task Understanding 3 2.1.2 Home Service Robots and Mobile Manipulation 6 2.2 AI Agents and Workflows 8 2.2.1 LangChain and LangGraph 8 2.2.2 A2A 9 2.3 Robot System Development Framework 11 2.3.1 Unity 11 2.3.2 ROS2 11 2.3.3 SLAM 12 2.3.4 Nav2 12 2.3.5 pros System 13 2.4 Visual Perception and 3D Object Localization 14 2.4.1 Camera Calibration 14 2.4.2 YOLO 15 2.4.3 SAM 16 2.4.4 Norfair 16 2.4.5 Object Localization 16 2.5 Grasp Pose Generation and Manipulation Feasibility 17 2.5.1 GraspGen for Grasp Pose Generation 17 2.5.2 Inverse Kinematics 18 2.5.3 PyBullet 18 3 This Work 19 3.1 Overview 19 3.1.1 System Overview 19 3.1.2 Workflow State and Communication Architecture 26 3.2 Node Introduction 28 3.2.1 LLM-Based Task Classification and General Chat 28 3.2.2 Target Instance Selection 29 3.2.3 Size Estimation and goal_pose Selection 32 3.2.4 Navigation with Position and Rotation Constraints 39 3.2.5 VLM-Based Reasoning Agent 40 3.2.6 Grasp Pipeline and Goal Pose Switching 43 3.2.7 Execution Log 50 4 Experiments 52 4.1 Unity Scene 52 4.1.1 Virtual Robot 53 4.1.2 Cameras 54 4.1.3 Initial Placement of the Robot and Objects 55 4.2 Workflow Execution Platform 57 4.2.1 System Deployment and Computing Platform 57 4.2.2 Model Selection 59 4.3 Experimental Setup 60 4.3.1 Ablation Experiment Design 60 4.3.2 Deployment Comparison Experiment Design 65 4.3.3 Measurement of System Performance 65 4.4 Experimental Results 67 4.4.1 Results 67 4.4.2 Analysis 73 5 Conclusions and Future Works 80 5.1 Conclusions 80 5.2 Limitations 81 5.3 Future Works 81 References 83

    [1] S. Tan, D. Zhou, X. Shao, J. Wang, and G. Sun, “Language-Conditioned Open-Vocabulary Mobile Manipulation with Pretrained Models,” Jul. 23, 2025, arXiv: arXiv:2507.17379. doi: 10.48550/arXiv.2507.17379.
    [2] F.-C. Hsu, Y.-H. Hsieh, and W.-Y. Su, “Opening the Black Box: A Simulation Framework for Observability and Multi-Camera Perception in Smart Homes,” in Proc. 28th Int. Conf. Human-Computer Interaction (HCII 2026), LNCS, vol. 16736, Montreal, Canada, Jul. 26–31, 2026, accepted.
    [3] “Workflows and agents,” Docs by LangChain. Accessed: Jun. 27, 2026. [Online]. Available: https://docs.langchain.com/oss/python/langgraph/workflows-agents
    [4] “Overview - A2A Protocol.” Accessed: Jun. 27, 2026. [Online]. Available: https://a2a-protocol.org/latest/specification/
    [5] M. Ahn et al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances,” Aug. 16, 2022, arXiv: arXiv:2204.01691. doi: 10.48550/arXiv.2204.01691.
    [6] V. Bhat, A. U. Kaypak, P. Krishnamurthy, R. Karri, and F. Khorrami, “Grounding LLMs For Robot Task Planning Using Closed-loop State Feedback,” Adv. Robot. Res., p. e202500072, Nov. 2025, doi: 10.1002/adrr.202500072.
    [7] R. Wang, Z. Yang, Z. Zhao, X. Tong, Z. Hong, and K. Qian, “LLM-based Robot Task Planning with Exceptional Handling for General Purpose Service Robots,” May 24, 2024, arXiv: arXiv:2405.15646. doi: 10.48550/arXiv.2405.15646.
    [8] H. Singh, R. J. Das, M. Han, P. Nakov, and I. Laptev, “MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulation,” Aug. 25, 2025, arXiv: arXiv:2411.17636. doi: 10.48550/arXiv.2411.17636.
    [9] S. Yenamandra et al., “HomeRobot: Open-Vocabulary Mobile Manipulation,” Jan. 10, 2024, arXiv: arXiv:2306.11565. doi: 10.48550/arXiv.2306.11565.
    [10] P. Liu et al., “DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation,” May 29, 2025, arXiv: arXiv:2411.04999. doi: 10.48550/arXiv.2411.04999.
    [11] “LangChain overview,” Docs by LangChain. Accessed: Jul. 30, 2026. [Online]. Available: https://docs.langchain.com/oss/python/langchain/overview
    [12] S. Garg, “LangGraph vs. LangChain: What’s the difference?” Medium, Aug. 4, 2025. [Online]. Available: https://garg-shelvi.medium.com/langgraph-vs-langchain-whats-the-difference-d78f4544728b. [Accessed: Jul. 31, 2026].
    [13] @pramodchandrayan, “LangChain tutorial series: Understanding LangGraph—Part 5,” Artificial Intelligence in Plain English, Jul. 20, 2025. [Online]. Available: https://ai.plainenglish.io/langchain-tutorial-series-understanding-langgraph-part-5-2f123b191bc2. [Accessed: Jul. 31, 2026].
    [14] “Unity: Develop, Deploy, and Grow | The World’s Leading Game Engine,” Unity. Accessed: Jun. 27, 2026. [Online]. Available: https://unity.com
    [15] Y.-S. Tseng, “Coordinated Control of Multiple Robots Using ROS2 and a Unity-Based Digital Twin Environment,” M.S. thesis, Inst. Comput. Sci. Inf. Eng., National Cheng Kung Univ., Tainan, Taiwan, 2025.
    [16] “Interfaces (topics, services, actions) — ROS 2 Documentation: Rolling documentation.” Accessed: Jun. 27, 2026. [Online]. Available: https://docs.ros.org/en/rolling/Concepts/Basic/Interfaces-Topics-Services-Actions.html
    [17] S. Macenski and I. Jambrecic, “SLAM Toolbox: SLAM for the dynamic world,” J. Open Source Softw., vol. 6, no. 61, p. 2783, 2021, doi: 10.21105/joss.02783.
    [18] “Navigation Concepts — Nav2 1.0.0 documentation.” Accessed: Jun. 27, 2026. [Online]. Available: https://docs.nav2.org/concepts/index.html
    [19] “OpenCV: Camera Calibration.” Accessed: Jun. 27, 2026. [Online]. Available: https://docs.opencv.org/4.13.0/dc/dbb/tutorial_py_calibration.html
    [20] G. Jocher, J. Qiu, M. Liu, S. Lyu, F. C. Akyon, and M. E. Kalfaoglu, “Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models,” Jun. 02, 2026, arXiv: arXiv:2606.03748. doi: 10.48550/arXiv.2606.03748.
    [21] “Train a Model,” Roboflow Docs. Accessed: Jul. 19, 2026. [Online]. Available: https://docs.roboflow.com/train/train
    [22] A. Kirillov et al., “Segment Anything,” Apr. 05, 2023, arXiv: arXiv:2304.02643. doi: 10.48550/arXiv.2304.02643.
    [23] Tryolabs, “Norfair: Real-time Multi-Object Tracking Library,” GitHub. Accessed: Jun. 27, 2026. [Online]. Available: https://github.com/tryolabs/norfair
    [24] A. Murali et al., “GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training,” Jul. 17, 2025, arXiv: arXiv:2507.13097. doi: 10.48550/arXiv.2507.13097.
    [25] “Bullet Real-Time Physics Simulation | Home of Bullet and PyBullet: physics simulation for games, visual effects, robotics and reinforcement learning.” Accessed: Jun. 27, 2026. [Online]. Available: https://pybullet.org/wordpress/
    [26] “Foxglove: Multimodal data platform for Physical AI,” Foxglove. Accessed: Jun. 27, 2026. [Online]. Available: https://foxglove.dev/

    QR CODE