傳統的路由最佳化方法取決於精心設計的算法，這些算法缺乏適應動態流量和服務要求的質量。隨著人工智能的發展，越來越多的研究集中在使用機器學習來解決路由最佳化問題。與監督式學習相比，強化學習不需要標記好的數據集，而是通過反覆與環境的互動中的試錯與學習。在本文中，我們提出了一種在軟體定義網路中進行智能路由的新方法，我們使用深度強化學習和SDN控制器的優勢，例如全球視圖，集中控制，以實現SDN的智能路由。我們使用真實和模擬的流量來評估我們的算法，結果表明我們的方法在這兩種情況下優於其他演算法，例如DRSIR，RSIR和Dijkstra演算法的變形。此外，我們使用不同的拓撲來評估我們的方法，結果表明我們的演算法也比其他演算法更好。評估結果證明，我們的方法可以良好的利用網路資源並減少網路延遲和損失。

Routing optimization techniques used in the past relied on well meticulously designed algorithms, which lack of adaptation to dynamic traffic and quality of service requirements. With the development artificial intelligence, more and more research focus on using machine learning to solve the problem of routing optimization. Compared with supervised learning, reinforcement learning does not require labeled dataset. Reinforcement learning is a trial and error process with the environment. In this research, we propose a novel approach for intelligent routing in Software Defined Network. We use deep reinforcement learning and the advantage of SDN controller, such as global view, centralized control to achieve intelligent routing in Software Defined Network. We use real and synthetic traffic to evaluate our algorithm, and the results show that our approach outperforms the other algorithms such as DRSIR, RSIR, and the Dijkstra algorithm's variant in both conditions. In addition, we use different topologies to evaluate our approach and the result shows the algorithm is also better than the other algorithms. The result of evaluation proves that our method can make good use of the network's bandwidth and reduce the network delay and loss.

[1] B. A. A. Nunes, M. Mendonca, X. -N. Nguyen, K. Obraczka and T. Turletti, “A survey of software-defined networking: Past, present, and future of programmable networks”, IEEE Communications surveys & tutorials 16.3 (2014), pp. 1617-1634, 2014.
[2] McKeown, Nick and Anderson, Tom and Balakrishnan, Hari and Parulkar, Guru and Peterson, Larry and Rexford, Jennifer and Shenker, Scott and Turner, Jonathan, “OpenFlow: enabling innovation in campus networks”, ACM SIGCOMM computer communication review 38.2 (2008), pp. 69-74, 2008.
[3] J. S. Marcus. “The economic impact of internet traffic growth on network operators”, Available at SSRN: https://ssrn.com/abstract=2531782, 2014.
[4] R. A. Guerin, A. Orda and D. Williams, “ QoS routing mechanisms and OSPF extensions”, GLOBECOM 97. IEEE Global Telecommunications Conference.1997, pp. 1903-1908 vol.3.
[5] Abhishek Verma, Neha Bhardwaj, “A Review on Routing Information Protocol (RIP) and Open Shortest Path First (OSPF) Routing Protocol”, International Journal of Future Generation Communication and Networking. Vol. 9, No. 4 (2016), pp. 161-170.
[6] Albert Mestres, Alberto Rodriguez-Natal, Josep Carner, Pere Barlet-Ros, Eduard Alarcón, Marc Solé, Victor Muntés-Mulero, David Meyer, Sharon Barkai, Mike J. Hibbett, Giovani Estrada, Khaldun Ma'ruf, Florin Coras, Vina Ermagan, Hugo Latapie, Chris Cassar, John Evans, Fabio Maino, Jean Walrand, and Albert Cabellos, “Knowledge-defined networking”, ACM SIGCOMM Computer Communication Review 47.3 (2017), pp. 2–10, 2017.
[7] D. Kreutz, F. M. V. Ramos, P. E. Veríssimo, C. E. Rothenberg, S. Azodolmolky, and S. Uhlig, “Software-defined networking: A comprehensive survey,” Proc. IEEE, vol. 103, no. 1, pp. 14–76, Jan. 2015.
[8] S. Chaudhary and R. Johari, “Oruml: Optimized routing in wireless networks using machine learning,” IJCS, pp.43-58, 2020.
[9] N. Kato, Z. M. Fadlullah, B. Mao, F. Tang, O. Akashi, T. Inoue, and K. Mizutani, “The deep learning vision for heterogeneous network traffic control: Proposal, challenges, and future perspective,” IEEE Wireless Communications, vol. 24, no. 3, pp. 146–153, 2016.
[10] S.-C. Lin, I. F. Akyildiz, P. Wang, and M. Luo, “Qos-aware adaptive routing in multi-layer hierarchical software defined networks: a reinforcement learning approach,” in IEEE SCC, 2016, pp. 25–33.
[11] S. Sendra, A. Rego, J. Lloret, J. M. Jimenez, and O. Romero, “Including artificial intelligence in a routing protocol using software defined networks,” in IEEE ICC Workshops, 2017, pp. 670–674.
[12] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Barath, “A brief survey of deep reinforcement learning,” IEEE Signal Processing Magazine, vol. 34, pp. 26–38, 2017.
[13] L. Zhao, J. Wang, J. Liu, and N. Kato, “Routing for crowd management in smart cities: A deep reinforcement learning perspective,” IEEE Communications Magazine, vol. 57, no. 4, pp. 88–93, 2019.
[14] C. Wang, H. Wang, and W. Wang, “A two-hops state-aware routing strategy based on deep reinforcement learning for leo satellite networks,” Electronics, vol. 8, no. 9, p. 920, 2019.
[15] D. M. Casas-Velasco, O. M. C. Rendon, and N. L. S. da Fonseca, “Intelligent routing based on reinforcement learning for software-defined networking”, IEEE Transactions on Network and Service Management, vol. 18, no. 1, pp. 870–881, March, 2021.
[16] D. M. Casas-Velasco, O. M. C. Rendon and N. L. S. da Fonseca, “ DRSIR: A Deep Reinforcement Learning Approach for Routing in Software-Defined Networking”, IEEE Transactions on Network and Service Management, pp 1-12, 2021.
[17] A. Mestres, J. Carner, Pere Barlet-Ros, E. Alarcón, M. Solé, Victor Muntés-Mulero, D. Meyer, Sharon Barkai, M. J. Hibbett, Giovani Estrada, K. Ma'ruf, Florin Coras, Vina Ermagan, H.Latapie, C. Cassar, J. Evans, F. Maino, J. Walrand, “Knowledge-defined networking”, ACM SIGCOMM Computer Communication Review 47.3 (2017), pp. 2–10, 2017.
[18] H. Yao, T. Mai, X. Xu, P. Zhang, M. Li and Y. Liu, “ NetworkAI: An Intelligent Network Architecture for Self-Learning Control Strategies in Software Defined Networks”, in IEEE Internet of Things Journal, vol. 5, no. 6, pp. 4319-4327, Dec. 2018.
[19] Wai-xi Liu, “Intelligent Routing based on Deep Reinforcement Learning in Software-Defined Data-Center Networks,” 2019 IEEE Symposium on Computers and Communications (ISCC), 2019, pp. 1-6.
[20] Q. Fu, E. Sun, K. Meng, M. Li and Y. Zhang, “Deep Q-Learning for Routing Schemes in SDN-Based Data Center Networks,” in IEEE Access, vol. 8, pp. 103491-103499, 2020.
[21] G. Stampa, M. Arias, D. Sanchez-Charles, V. Muntes-Mulero and A. Cabellos, “A deep-reinforcement learning approach for software-defined networking routing optimization”, in 4th International Conference on Computer Science and Application Engineering October 2020, pp. 1-5.
[22] C. Yu, J. Lan, Z. Guo, and Y. Hu, “Drom: Optimizing the routing in software-defined networks with deep reinforcement learning”, IEEE Access, vol. 6, pp. 64 533–64 539, 2018.
[23] P. Sun, Y. Hu, J. Lan, L. Tian, and M. Chen, “Tide: Time-relevant deep reinforcement learning for routing optimization”, Future Generation Computer Systems, vol. 99, pp. 401–409, 2019.
[24] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez and Y. Tassa, "Continuous control with deep reinforcement learning", Computer Science, vol. 8, no. 6, 2015.
[25] L. Liao and V. C. Leung, “Lldp based link latency monitoring in software defined networks,” in IEEE CNSM, 2016, pp. 330–335.
[26] L. Al Shalabi and Z. Shaaban, “Normalization as a preprocessing engine for data mining and the approach of preference matrix,” in IEEE DepCoS, 2006, pp. 207–214.
[27] P. T. Kirstein, “European international academic networking: A 20 year perspective.” in TERENA Networking Conference, pp. 1-18, 2004.
[28] Mills, David L., and Hans-Werner Braun. "The NSFNET backbone network." Proceedings of the ACM workshop on Frontiers in computer communications technology, pp. 191-196, 1987.
[29] R. L. S. De Oliveira, C. M. Schweitzer, A. A. Shinoda, and L. R. Prete, “Using mininet for emulation and prototyping software-defined networks,” in IEEE COLCOM, 2014, pp. 1–6.
[30] S. Uhlig, B. Quoitin, J. Lepropre, and S. Balon, “Providing public intradomain traffic matrices to the research community,” SIGCOMM Computer Communication Review, vol. 36, no. 1, pp. 83–86, 2006.
[31] P. Tune and M. Roughan, “Spatiotemporal traffic matrix synthesis,” in SIGCOMM. ACM, 2015, pp. 579–592.