本文提出一種基於多代理人強化學習的資源分配框架（MARL-RA），針對混合可見光通訊（VLC）與 Wi-Fi 網路，旨在最大化系統總傳輸量（throughput），同時確保可靠的服務覆蓋。每個用戶設備（UE）被建構為一個獨立的強化學習代理人，能根據自身的觀測結果自主選擇使用 VLC 或 Wi-Fi。所提出的框架整合了階層式資源分配機制與全域獎勵結構，以促進代理人間的合作行為與高效的頻譜利用。此外，我們設計了一個三階段的 VLC 分配演算法，兼顧傳輸量、公平性與用戶滿意度，並對處於中斷狀態（outage）的用戶提供動態 Wi-Fi 回退機制。
模擬結果顯示，在使用者密集的情境下，MARL-RA 顯著優於現有基準方法。在狹窄視場角（FoV）條件下，傳統方法常面臨嚴重的連線中斷問題，而我們的方法能有效判斷適合連接 Wi-Fi 的用戶，並在不產生壅塞的情況下分配 VLC 資源，實現零中斷，且相較於 MCRAIC 提升高達 131.7% 的系統傳輸效能。在高 FoV 環境中，儘管 VLC 覆蓋範圍增加會導致嚴重的跨區干擾與訊號劣化，我們的方法仍能自適應地將壅塞的 VLC 用戶導向 Wi-Fi，進一步維持低中斷率與高公平性。上述結果驗證了 MARL-RA 在多樣部署環境中的穩定性、自適應性與可擴展性。

This thesis presents a Multi-Agent Reinforcement Learning-based Resource Allocation (MARL-RA) framework for hybrid visible light communication (VLC) and Wi-Fi networks, aiming to maximize system throughput while ensuring reliable service coverage. Each UE is modeled as an independent reinforcement learning agent that autonomously selects between VLC and Wi-Fi based on local observations. The proposed framework integrates a hierarchical resource allocation mechanism and a global reward structure to promote cooperative behavior and efficient spectrum utilization. Additionally, a three-stage VLC allocation algorithm is designed to account for throughput, fairness, and user satisfaction, followed by a dynamic Wi-Fi fallback for UEs in outage.
Simulation results demonstrate that MARL-RA significantly outperforms benchmark methods in dense user scenarios. Under narrow Field of View (FoV) conditions, where traditional methods suffer severe connectivity loss, our approach effectively identifies UEs better suited for Wi-Fi and allocates VLC resources without congestion, resulting in zero outage and up to 131.7% improvement in system throughput over MCRAIC. In high FoV settings, where increased VLC coverage leads to severe inter-cell interference and signal degradation, our method adaptively offloads users from congested VLC bands to Wi-Fi, maintaining low outage and high fairness. These results validate the robustness, adaptability, and scalability of MARL-RA in diverse deployment environments.

[1] T. Cogalan and H. Haas, "Why Would 5G Need Optical Wireless Communications?," 2017 IEEE 28th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications, Montreal, Quebec, Canada, pp. 1-6, 2017.
[2] P. Dwivedy, V. Dixit and A. Kumar, "A Survey on Visible Light Communication for 6G: Architecture, Application and Challenges," 2023 International Conference on Computer, Electronics & Electrical Engineering & their Applications (IC2E3), Srinagar Garhwal, India, 2023, pp. 1-6.
[3] H. Haas, L. Yin, Y. Wang and C. Chen, "What is LiFi?," in Journal of Lightwave Technology, vol. 34, no. 6, pp. 1533-1544, 15 Mar.15, 2016.
[4] M. Uysal and H. Nouri, "Optical wireless communications — An emerging technology," 2014 16th International Conference on Transparent Optical Networks (ICTON), Graz, Austria, 2014, pp. 1-7.
[5] J. L. Henao, N. Guerrero González and J. C. García-Álvarez, "Three-Dimensional Spatial Coding Based on Hybrid Pulse Width and Pulse Position Modulation for Simultaneous Data Transmission and Brightness Control on RGB LED Arrays," 2019 IEEE Colombian Conference on Communications and Computing (COLCOM), Barranquilla, Colombia, 2019, pp. 1-6.
[6] S. Ahmed, T. Yeap, S. Ahmed and I. Kiringa, "Ambient Light Impact on Power Reception in Outdoor VLC: A Distance-Based Analysis," 2024 International Symposium on Networks, Computers and Communications (ISNCC), Washington DC, DC, USA, 2024, pp. 1-7.
[7] S. P. Dash, "Performance Analysis for a Narrow-FoV VLC System with Random User Orientation," 2023 International Conference on Microwave, Optical, and Communication Engineering (ICMOCE), Bhubaneswar, India, 2023, pp. 1-4.
[8] Z. Zeng, M. Dehghani Soltani, Y. Wang, X. Wu and H. Haas, "Realistic Indoor Hybrid WiFi and OFDMA-Based LiFi Networks," in IEEE Transactions on Communications, vol. 68, no. 5, pp. 2978-2991, May 2020.
[9] X. Wu, M. D. Soltani, L. Zhou, M. Safari and H. Haas, "Hybrid LiFi and WiFi Networks: A Survey," in IEEE Communications Surveys & Tutorials, vol. 23, no. 2, pp. 1398-1420, Secondquarter 2021.
[10] T. T. Nguyen, N. D. Nguyen and S. Nahavandi, "Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications," in IEEE Transactions on Cybernetics, vol. 50, no. 9, pp. 3826-3839, Sept. 2020.
[11] Y. S. Nasir and D. Guo, "Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networks," in IEEE Journal on Selected Areas in Communications, vol. 37, no. 10, pp. 2239-2250, Oct. 2019.
[12] A. Dorri, S. S. Kanhere and R. Jurdak, "Multi-Agent Systems: A Survey," in IEEE Access, vol. 6, pp. 28573-28593, 2018.
[13]A. Burton, S. Zvanovec, Z. Ghassemlooy, P. A. Haigh, I. Darwazeh and H. L. Minh, "Investigation of WDM VLC Using Standard 5 mm RGB LEDs," 2018 11th International Symposium on Communication Systems, Networks & Digital Signal Processing (CSNDSP), Budapest, Hungary, 2018, pp. 1-6.
[14] X. Liu, Z. Na, Y. Wang and T. S. Durrani, "Joint Resource Allocation for a Novel OFDM-Based Multicolor VLC Network," in IEEE Networking Letters, vol. 3, no. 3, pp. 100-104, Sept. 2021.
[15] H. Liu et al., "Multi-Cell Resource Allocation Mechanism Based on Interference Control in Indoor Multicolor VLC-WiFi Heterogeneous Networks," in IEEE Transactions on Network and Service Management, vol. 21, no. 5, pp. 5707-5717, Oct. 2024.
[16] X. Bao, X. Gu, W. Zhang and H. -H. Chen, "User-Centric Quality-of-Experience Optimization and Scheduling of Multicolor LEDs in VLC Systems," in IEEE Systems Journal, vol. 13, no. 3, pp. 2275-2284, Sept. 2019.
[17] Natasha Shrestha, Muhammad Sohail, Charusluk Viphavakit, Poompat Saengudomlert and W. S. Mohammed, "Demonstration of visible light communications using RGB LEDs in an indoor environment," ECTI-CON2010: The 2010 ECTI International Confernce on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology, Chiang Mai, Thailand, 2010, pp. 1159-1163.
[18] V. K. Papanikolaou, P. D. Diamantoulakis, P. C. Sofotasios, S. Muhaidat and G. K. Karagiannidis, "On Optimal Resource Allocation for Hybrid VLC/RF Networks With Common Backhaul," in IEEE Transactions on Cognitive Communications and Networking, vol. 6, no. 1, pp. 352-365, Mar. 2020.
[19] V. K. Papanikolaou, P. D. Diamantoulakis and G. K. Karagiannidis, "User Grouping for Hybrid VLC/RF Networks With NOMA: A Coalitional Game Approach," in IEEE Access, vol. 7, pp. 103299-103309, 2019.
[20] H. Zhang, W. Ding, F. Yang, J. Song and Z. Han, "Resource Allocation in Heterogeneous Network With Visible Light Communication and D2D: A Hierarchical Game Approach," in IEEE Transactions on Communications, vol. 67, no. 11, pp. 7616-7628, Nov. 2019.
[21] M. S. Demir, S. M. Sait and M. Uysal, "Unified Resource Allocation and Mobility Management Technique Using Particle Swarm Optimization for VLC Networks," in IEEE Photonics Journal, vol. 10, no. 6, pp. 1-9, Dec. 2018, Art no. 7908809.
[22] T. Wan, L. Luo-Kun, Z. Xia and J. Chun-Xiao, "A Resource Allocation Algorithm Combined with Optical Power Dynamic Allocation for Indoor Hybrid VLC and Wi-Fi Network," 2016 8th International Conference on Computational Intelligence and Communication Networks (CICN), Tehri, India, 2016, pp. 21-27.
[23] L. Yang, X. Zheng, X. Ma and Y. Tian, "Optimization of Resource Allocation Algorithm for Visible Light and WiFi Heterogeneous Networks," 2023 6th International Conference on Communication Engineering and Technology (ICCET), Xi'an, China, 2023, pp. 55-61.
[24] Q. Mao, F. Hu and Q. Hao, "Deep Learning for Intelligent Wireless Networks: A Comprehensive Survey," in IEEE Communications Surveys & Tutorials, vol. 20, no. 4, pp. 2595-2621, Fourthquarter 2018.
[25] T. Verma, A. Raza, S. Shrivastava, A. Kumar, D. P. Kothari and U. D. Dwivedi, "A Novel On-Policy DRL-Based Approach for Resource Allocation in Hybrid RF/VLC Systems," in IEEE Transactions on Consumer Electronics, vol. 71, no. 1, pp. 550-560, Feb. 2025.
[26] J. Kong, Z. -Y. Wu, M. Ismail, E. Serpedin and K. A. Qaraqe, "Q-Learning Based Two-Timescale Power Allocation for Multi-Homing Hybrid RF/VLC Networks," in IEEE Wireless Communications Letters, vol. 9, no. 4, pp. 443-447, Apr. 2020.
[27] N. Chen, L. Yang, X. Liang, J. Zhang and Y. Zhang, "Dynamic Resource Allocation for Indoor VLC/WiFi Heterogeneous Network Based on Q-learning," 2024 16th International Conference on Communication Software and Networks (ICCSN), Ningbo, China, 2024, pp. 154-158.
[28] T. Fath and H. Haas, "Performance Comparison of MIMO Techniques for Optical Wireless Communications in Indoor Environments," in IEEE Transactions on Communications, vol. 61, no. 2, pp. 733-742, Feb. 2013.
[29] B. S. Ciftler, A. Alwarafy and M. Abdallah, "Distributed DRL-Based Downlink Power Allocation for Hybrid RF/VLC Networks," in IEEE Photonics Journal, vol. 14, no. 3, pp. 1-10, Jun. 2022, Art no. 8632510.
[30] B. Bai, J. Deng, X. Jin, B. Su and T. Shang, "Hypergraph Based Resource Allocation Algorithms for VLC Network With Heterogeneous LEDs and Varied Users," in IEEE Photonics Journal, vol. 13, no. 4, pp. 1-10, Aug. 2021, Art no. 7300810.
[31] C. Jenil and R. K. Jeyachitra, “Green indoor optical wireless communication systems: Pathway towards pervasive deployment,” Digit. Commun. Netw., vol. 7, no. 3, pp. 410–444, Sep. 2021.
[32] T. Komine and M. Nakagawa, "Fundamental analysis for visible-light communication system using LED lights," in IEEE Transactions on Consumer Electronics, vol. 50, no. 1, pp. 100-107, Feb. 2004.
[33] H. Liu, Z. Lin, Y. Chen and P. Xin, "Elite User Clustering-Based Indoor Heterogeneous VLC Interference Management and Sub-Channel Allocation Strategy," in IEEE Access, vol. 8, pp. 43582-43591, 2020.
[34] R. Ahmad, M. D. Soltani, M. Safari and A. Srivastava, "Reinforcement LearningBased Near-Optimal Load Balancing for Heterogeneous LiFi WiFi Network," in IEEE Systems Journal, vol. 16, no. 2, pp. 3084-3095, Jun. 2022.
[35] G. Ma, R. Parthiban and N. Karmakar, "An Adaptive Handover Scheme for Hybrid LiFi and WiFi Networks," in IEEE Access, vol. 10, pp. 18955-18965, 2022.
[36] A. M. Alenezi and K. A. Hamdi, "Federated Reinforcement Learning With Knowledge Transfer for Network Selection in Hybrid WiFi-VLC Networks," in IEEE Transactions on Artificial Intelligence, vol. 5, no. 6, pp. 2921-2932, Jun. 2024.
[37] S. Shao et al., "Design and analysis of a visible-light-communication enhanced WiFi system," in Journal of Optical Communications and Networking, vol. 7, no. 10, pp. 960-973, Oct. 2015.
[38] A. B. M. Adam, X. Y. Wan, and Z. Q. Wang, “User scheduling and power allocation for downlink multi-cell multi-carrier NOMA systems,” Digit. Commun. Netw., vol. 9, no. 1, pp. 252–263, Mar. 2023.
[39] J. D. Zhang and S. H. Wang, “Noise filtering of optical communication systems based on two modulated quantum numbers,” J. Liaocheng Univ. (Nat. Sci. Ed.), vol. 37, no. 1, pp. 23–28, Feb. 2024.
[40] R. Jain, D. Chiu, and W. Hawe, “A quantitative measure of fairness and discrimination for resource allocation in shared systems,” tech. rep., Digital Equipment Corporation, DEC-TR-301, 1984.