隨著網路規模的快速增長，傳統路由協定因其對流量動態變化的適應能力有限，難以滿足多樣化的服務品質需求。有許多研究嘗試將機器學習應用在路由問題上，一些方法以監督式學習來優化路由策略，但監督式學習仰賴標記好的數據集；相比之下，強化學習不須標記資料，而是透過環境反覆互動、嘗試錯誤來學習最優策略。
本文中提出一個基於多智能體強化學習的軟體定義網路智能路由方法，我們讓一個智能體為一個源目標對決定路由路徑，透過神經網路架構的設計，以及注意力機制來學習其他智能體造成的影響，讓智能體間達到更好的合作。我們在不同拓撲中使用真實及模擬的流量進行了評估，結果表明了我們的方法可以達成良好的負載平衡。

With the rapid growth of network scale, traditional routing protocols face challenges in meeting diverse quality of service demands due to their limited adaptability to dynamic traffic changes. Numerous studies have attempted to apply machine learning to routing problems. Some approaches use supervised learning to optimize routing strategies; however, supervised learning relies on labeled datasets. In contrast, reinforcement learning does not require labeled data and learns optimal strategies through repeated interactions with the environment and trial and error.
This thesis proposes a multi-agent reinforcement learning based intelligent routing method for software-defined networks. We designate one agent to decide the routing path for each source-destination pair. Through the design of the neural network architecture and the use of an attention mechanism, the influences of other agents are learned, achieving better cooperation among the agents. We evaluated our method using real and simulated traffic in different topologies. The results indicate that our method can achieve good load balancing.

[1] D. Kreutz, F. M. V. Ramos, P. E. Veríssimo, C. E. Rothenberg, S. Azodolmolky, and S. Uhlig, “Software-defined networking: A comprehensive survey,” Proceedings of the IEEE, vol. 103, no. 1, pp. 14–76, 2015.
[2] J. S. Marcus, “The economic impact of internet traffic growth on network operators,” SSRN Electronic Journal, Jan. 2014. [Online]. Available: https://doi.org/10.2139/ssrn.2531782. [Accessed: July, 2024].
[3] S. Faezi,A. Shirmarz, “A Comprehensive Survey on Machine Learning using in Software Defined Networks (SDN),“ Human-Centric Intelligent Systems, vol. 3, no. 3, pp. 312–343, Jun. 2023.
[4] S. Chaudhary and R. Johari, “ORuML: Optimized routing in wireless networks using machine learning,” International Journal of Communication Systems, vol. 33, no. 11, p. e4394, 2020.
[5] P. Sun, Y. Hu, J. Lan, L. Tian, and M. Chen, “TIDE: Time-relevant deep reinforcement learning for routing optimization,” Future Generation Computer Systems, vol. 99, pp. 401–409, Oct. 2019.
[6] B. Chen, D. Zhu, Y. Wang and P. Zhang, ”An Approach to Combine the Power of Deep Reinforcement Learning with a Graph Neural Network for Routing Optimization,“ Electronics, vol. 11, no. 3, pp. 368, 2022.
[7] D. M. Casas-Velasco, O. M. C. Rendon, and N. L. S. da Fonseca, “Intelligent routing based on reinforcement learning for software-defined networking,” IEEE Transactions on Network and Service Management, vol. 18, no. 1, pp. 870-881, Mar. 2021.
[8] A. Modi, R. Shah, K. Jain, R. Verma, R. Shorey and H. Saran., ” Multi-Agent Packet Routing (MAPR): Co-Operative Packet Routing Algorithm with Multi-Agent Reinforcement Learning,” 15th International Conference on COMmunication Systems & NETworkS, pp.722-730, 2023.
[9] S. S. Bhavanasi, L. Pappone, and F. Esposito, ”Dealing With Changes: Resilient Routing via Graph Neural Networks and Multi-Agent Deep Reinforcement Learning,” IEEE Transactions on Network and Service Management, VOL. 20, NO. 3, pp. 2283-2294, Sep 2023.
[10] C. Liu, M. Xu, Y. Yang, and Nan Geng, ” DRL-OR: Deep Reinforcement Learning-based Online Routing for Multi-type Service Requirements,” IEEE Conference on Computer Communications, pp. 1-10, May. 2021.
[11] C. Liu, P. Wu , M. Xu, Y. Yang, and Nan Geng, ”Scalable Deep Reinforcement Learning-Based Online Routing for Multi-Type Service Requirements,” IEEE Transactions on Parallel and Distributed Systems , VOL. 34, NO. 8, pp. 2337-2351 Aug 2023.
[12] D. M. Casas-Velasco, O. M. C. Rendon and N. L. S. da Fonseca, “ DRSIR: A Deep Reinforcement Learning Approach for Routing in Software-Defined Networking,” IEEE Transactions on Network and Service Management, vol. 19, no. 4, pp 4807-4820 Dec. 2022.
[13] Tsai-Hsiu Cheng, “Intelligent Routing via Deep Reinforcement Learning in Software Defined Network,” Master Thesis, National Cheng Kung University, Tainan, Taiwan, R.O.C, pp. 1-50, Sep. 2022.
[14] S. Iqbal and F. Sha, ”Actor-Attention-Critic for Multi-Agent Reinforcement Learning,” Proceedings of the 36th International Conference on Machine Learning, pp. 2961-2970, 2019.
[15] F. Christianos, G. Papoudakis, A. Rahman and S. V. Albrecht, ” Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing,” Proceedings of the 38th International Conference on Machine Learning , pp. 1989-1998, 2021.
[16] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel and I. Mordaych, “Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments,” Advances in Neural Information Processing Systems, vol. 30, pp. 6379–6390, 2017
[17] Y. Yang, R. Luo, M. Li, W, Zhang, J. Wang, “Mean Field Multi-Agent Reinforcement Learning,” Proceedings of the 35th International Conference on Machine Learning, vol. 80,pp. 5571-5580 Jul. 2018.
[18] D. D. Clark, C. Partridge, J. C. Ramming, and J. T. Wroclawski, “A knowledge plane for the internet,” Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications, pp. 3-10, 2003.
[19] A. Mestres, A. Rodriguez-Natal, J. Carner, P. Barlet-Ros, E. Alarcn, M. Sol, V. Munts-Mulero, D. Meyer, S. Barkai, M. J. Hibbett, G. Estrada, K. Ma’ruf, F. Coras, V. Ermagan, H. Latapie, C. Cassar, J. Evans, F. Maino, J. Walrand and A. Cabellos, “Knowledge-defined networking,“ SIGCOMM Computer Communication Review, vol. 47, no. 3, pp. 2–10, 2017.
[20] E. Schuitema, L. Busoniu, R. Babuska and P. Jonker, “Control Delay in Reinforcement Learning for Real-Time Dynamic Systems: A Memoryless Approach,” IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3226–3231, Nov. 2010.
[21] L. A. Shalabi and Z. Shaaban, “Normalization as a preprocessing engine for data mining and the approach of preference matrix,” International conference on dependability of computer systems, pp. 207-214, May 2006.
[22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, “Attention Is All You Need,” Advances in Neural Information Processing Systems, vol. 30, pp. 6000–6010, 2017.
[23] V. Mnih et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529-533, 2015.
[24] T. Yuan, W. D. R. Neto, C. E. Rothenberg, K. Obraczka, C. Barakat, and T. Turletti, “Dynamic controller assignment in software defined Internet of vehicles through multi-agent deep reinforcement learning,” IEEE Transactions on Network and Service Management, vol. 18, no. 1, pp. 585–596, Mar. 2021.
[25] P. T. Kirstein, “European international academic networking: A 20 year perspective,“ TERENA Networking Conference, pp. 1-18 2004.
[26] R. L. S. De Oliveira, A. A. Shinoda, C. M. Schweitzer, and L. R. Prete, “Using Mininet for emulation and prototyping software defined networks,” IEEE Colombian Conference on Communications and Computing, pp. 1-6,2014.
[27] Ryu Controller, [Online]. Available: https://github.com/faucetsdn/ryu. [Accessed: July, 2024].
[28] S. Uhlig, B. Quoitin, J. Lepropre, and S. Balon, “Providing public intradomain traffic matrices to the research community,” SIGCOMM Computer Communication Review, vol. 36, no. 1, pp. 83–86, 2006.
[29] P. Tune and M. Roughan, “Spatiotemporal traffic matrix synthesis,” SIGCOMM '15: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, pp. 579–592, Aug. 2015.
[30] A. Sherstinsky, “Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,” Phys. D, Nonlinear Phenomena, vol. 404, pp. 1-43 Mar. 2020.