ReSQ: Reinforcement Learning-Based Queue Allocation in Software-Defined Queuing Framework

Ilora Maity

¹ Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg,

Luxembourg (email: [email protected])

² Information Technology and Electrical Engineering, University of Oulu, Oulu, Finland (email: [email protected])

³ Department of Computer and Information Security, Sejong University, Seoul, South Korea

^* Corresponding author

With the evolution of 5G networks, the demand for Ultra-Reliable Low Latency Communications (URLLC) services is increasing. Software-Defined Networking (SDN) offers flexible network management to prioritize URLLC services coexisting with best-effort traffic. Utilizing the network programmability feature of SDN, Software-Defined Queueing (SDQ) framework selects the optimal output port queue on forwarding devices and routing path for incoming traffic flows to provide deterministic Quality of Service (QoS) support required for URLLC traffic. However, in the existing SDQ framework, the selections of optimal queue and path are done manually by observing the traffic type of each incoming flow, the available bandwidth of each potential routing path, and the status of output port queues of each forwarding device on each potential routing path. The static allocations of path and queue for each flow are inefficient to provide a deterministic QoS guarantee for a high volume of incoming traffic which is typical in 5G networks. The limited buffer availability on the forwarding devices is another constraint regarding optimal queue allocation that ensures an end-to-end (E2E) delay guarantee. To address these challenges, in this paper, we extend the SDQ framework by automating queue management with a reinforcement learning (RL)-based approach. The proposed queue management approach considers diverse QoS demands as well as a limited buffer on the forwarding devices and performs prioritized queue allocation. Our approach also includes a hash-based flow grouping to handle a high volume of traffic having diverse latency demands and a path selection mechanism based on available bandwidth and hop count. The simulation result shows that the proposed scheme ReSQ reduces the QoS violation ratio by 10.45% as compared to the baseline scheme that selects queues randomly.

SDN, Reinforcement Learning, QoS, Queue Allocation, Deterministic QoS, URLLC

Ilora Maity, and Tarik Taleb (2022). ReSQ: Reinforcement Learning-Based Queue Allocation in Software-Defined Queuing Framework. Journal of Networking and Network Applications, Volume 2, Issue 4, pp. 143–152. https://doi.org/10.33969/J-NaNA.2022.020402.

[1] J. Prados-Garzon, T. Taleb, and M. Bagaa, “LEARNET: Reinforcement Learning Based Flow Scheduling for Asynchronous Deterministic Net-works,” in IEEE ICC, 2020, pp. 1–6.

[2] M. Khoshnevisan, V. Joseph, P. Gupta, F. Meshkati, R. Prakash, and P. Tinnakornsrisuphap, “5G Industrial Networks With CoMP for URLLC and Time Sensitive Network Architecture,” IEEE Journal on Selected Areas in Communications, vol. 37, no. 4, pp. 947–959, 2019.

[3] A. Montazerolghaem and M. H. Yaghmaee, “Load-Balanced and QoS-Aware Software-Defined Internet of Things,” IEEE Internet of Things Journal, vol. 7, no. 4, pp. 3323–3337, 2020.

[4] J. W. Guck, A. Van Bemten, and W. Kellerer, “DetServ: Network Models for Real-Time QoS Provisioning in SDN-Based Industrial En-vironments,” IEEE Transactions on Network and Service Management, vol. 14, no. 4, pp. 1003–1017, 2017.

[5] D. L. C. Dutra, M. Bagaa, T. Taleb, and K. Samdanis, “Ensuring End-to-End QoS Based on Multi-Paths Routing Using SDN Technology,” in IEEE GLOBECOM, 2017, pp. 1–6.

[6] A. N. Abbou, T. Taleb, and J. Song, “A Software-Defined Queuing Framework for QoS Provisioning in 5G and Beyond Mobile Systems,” IEEE Network, vol. 35, no. 2, pp. 168–173, 2021.

[7] D. Ghosal, S. Shukla, A. Sim, A. V. Thakur, and K. Wu, “A Re-inforcement Learning Based Network Scheduler for Deadline-Driven Data Transfers,” in IEEE GLOBECOM, 2019, pp. 1–6.

[8] J. Zhang, M. Ye, Z. Guo, C.-Y. Yen, and H. J. Chao, “CFR-RL: Traffic Engineering With Reinforcement Learning in SDN,” IEEE Journal on Selected Areas in Communications, vol. 38, no. 10, pp. 2249–2259, 2020.

[9] X. Guo, H. Lin, Z. Li, and M. Peng, “Deep-Reinforcement-Learning-Based QoS-Aware Secure Routing for SDN-IoT,” IEEE Internet of Things Journal, vol. 7, no. 7, pp. 6242–6251, 2020.

[10] D. M. Casas-Velasco, O. M. C. Rendon, and N. L. S. da Fonseca, “Intelligent Routing Based on Reinforcement Learning for Software-Defined Networking,” IEEE Transactions on Network and Service Management, vol. 18, no. 1, pp. 870–881, 2021.

[11] J. Yan, H. Zhang, Q. Shuai, B. Liu, and X. Guo, “HiQoS: An SDN-based multipath QoS solution,” China Communications, vol. 12, no. 5, pp. 123–133, 2015.

[12] S. Tomovic, N. Prasad, and I. Radusinovic, “SDN control framework for QoS provisioning,” in Telecommunications Forum (TELFOR), 2014, pp. 111–114.

[13] inSpace, “Queuing Mechanism in Modern Switches,” Tech. Rep., Accessed on: February 24, 2021. [Online]. Avail-able: https://blog.ipspace.net/2014/05/queuing-mechanisms-in-modern-switches.html

[14] J. Daly, V. Bruschi, L. Linguaglossa, S. Pontarelli, D. Rossi, J. Tollet,

E. Torng, and A. Yourtchenko, “TupleMerge: Fast Software Packet Processing for Online Packet Classification,” IEEE/ACM Transactions on Networking, vol. 27, no. 4, pp. 1417–1431, 2019.

[15] W. Li, T. Yang, O. Rottenstreich, X. Li, G. Xie, H. Li, B. Vamanan, D. Li, and H. Lin, “Tuple Space Assisted Packet Classification With High Performance on Both Search and Update,” IEEE Journal on Selected Areas in Communications, vol. 38, no. 7, pp. 1555–1569, 2020.

[16] P. Tehrani, F. Restuccia, and M. Levorato, “Federated Deep Rein-forcement Learning for the Distributed Control of NextG Wireless Networks,” in IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), 2021, pp. 248–253.

[17] S. Knight, H. X. Nguyen, N. Falkner, R. Bowden, and M. Roughan, “The Internet Topology Zoo,” IEEE Journal on Selected Areas in Communications, vol. 29, no. 9, pp. 1765–1775, Oct. 2011.

[18] I. Maity, S. Misra, and C. Mandal, “ETHoS: Energy-Aware Traffic Engineering for Sustainable Hybrid SDN,” IEEE Transactions on Sustainable Computing, doi: 10.1109/TSUSC.2022.3164571.

[19] I. Maity, A. Mondal, S. Misra, and C. Mandal, “CURE: Consistent Update with Redundancy Reduction in SDN,” IEEE Transactions on Communications, vol. 66, no. 9, pp. 3974–3981, Sep. 2018.

[20] A. Nasrallah, A. S. Thyagaturu, Z. Alharbi, C. Wang, X. Shao, M. Reisslein, and H. ElBakoury, “Ultra-Low Latency (ULL) Networks: The IEEE TSN and IETF DetNet Standards and Related 5G ULL Research,” IEEE Communications Surveys & Tutorials, vol. 21, no. 1, pp. 88–145, 2019.

[21] A. Mondal, S. Misra, and I. Maity, “Buffer Size Evaluation of Open-Flow Systems in Software-Defined Networks,” IEEE Systems Journal, vol. 13, no. 2, pp. 1359-1366, Jun. 2019.

[22] M. Rahouti, K. Xiong, Y. Xin and N. Ghani, “A Priority-Based Queueing Mechanism in Software-Defined Networking Environments,” IEEE Annual Consumer Communications & Networking Conference (CCNC), 2021, pp. 1-2.

[23] E. Li, F. He, L. Zhao and X. Zhou, “A SDN-based Traffic Band-width Allocation Method for Time Sensitive Networking in Avionics,” IEEE/AIAA Digital Avionics Systems Conference (DASC), 2019, pp. 1-7.

[24] Y. Goto, H. Masuyama, B. Ng, W. K. G. Seah and Y. Takahashi, “Queueing Analysis of Software Defined Network with Realistic Open-Flow–Based Switch Model,” IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunica-tion Systems (MASCOTS), 2016, pp. 301-306.