Deguang Wang*, Mingze Wang, Tao Zhang*, Shujie Yang,and Changqiao Xu
Deguang Wang, Tao Zhang
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, 100876, China
*Corresponding author
With the continuous increase of end users and types of services, the scale of the network has shown explosive growth, which has brought tremendous pressure and challenges to network data transmission. How to achieve high-quality data transmission has become a core issue. Single-path transmission has been difficult to meet the above requirements. The concurrent multipath transfer extension for stream control transmission protocol (CMT-SCTP), which supports multipath and independent data streams, can solve this problem. However, the current transmission path assessment scheme has too large granularity to make full use of the resources of the transition zone. Most studies ignore the different requirements of different services, a single transmission strategy, and the lack of an intelligent dynamic adjustment mechanism. Therefore, we designed a QCMT(Q-learning based CMT-SCTP) scheduling method. This method considers the multi-dimensional characteristics of the path and the characteristic preferences of different services, periodically evaluates and trains the reinforcement learning model for service adaptation, and makes scheduling decisions dynamically. Experimental results show that dynamic scheduling based on path parameters and service preferences can reduce message delay and improve network throughput.
Quality evaluation, QoS, SCTP, Concurrent Multipath Transfer, Q-learning
Deguang Wang, Mingze Wang, Tao Zhang, Shujie Yang,and Changqiao Xu (2021). An Adaptive CMT-SCTP Scheme: A Reinforcement Learning Approach. Journal of Networking and Network Applications, Volume 1, Issue 4, pp. 170–178. https://doi.org/10.33969/J-NaNA.2021.010404.
[1] Shahid Mumtaz, Josep Miquel Jornet, Jocelyn Aulin, Wolfgang H Gerstacker, Xiaodai Dong, and Bo Ai. Terahertz communication for vehicular networks. IEEE Transactions on Vehicular Technology, 66(7), 2017.
[2] Anastasia Yastrebova, Ruslan Kirichek, Yevgeni Koucheryavy, Aleksey Borodin, and Andrey Koucheryavy. Future networks 2030: Architecture & requirements. In 2018 10th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), pages 1–8. IEEE, 2018.
[3] Ping Dong, Bin Song, Hongke Zhang, and Xiaojiang Du. Improving on-board internet services for high-speed vehicles by multipath transmission in heterogeneous wireless networks. IEEE Transactions on Vehicular Technology, 65(12):9493–9507, 2016.
[4] Shumayla Yaqoob, Ata Ullah, Muhammad Akbar, Muhammad Imran, and Mohsen Guizani. Fog-assisted congestion avoidance scheme for internet of vehicles. In 2018 14th International Wireless Communica-tions & Mobile Computing Conference (IWCMC), pages 618–622. IEEE, 2018.
[5] Mohammad Mahdi Tajiki, Behzad Akbari, Mohammad Shojafar, Seyed Hesomodding Ghasemi, Mahdi Latifi Barazandeh, Nader Mokari, Luca Chiaraviglio, and Michael Zink. Cect: computationally efficient congestion-avoidance and traffic engineering in software-defined cloud data centers. Cluster Computing, 21(4):1881–1897, 2018.
[6] Changqiao Xu, Tianjiao Liu, Jianfeng Guan, Hongke Zhang, and Gabriel-Miro Muntean. Cmt-qa: Quality-aware adaptive concurrent multipath data transfer in heterogeneous wireless networks. IEEE transactions on mobile computing, 12(11):2193–2205, 2012.
[7] Jiyan Wu, Bo Cheng, Ming Wang, and Junliang Chen. Energy-aware concurrent multipath transfer for real-time video streaming over heterogeneous wireless networks. IEEE Transactions on circuits and systems for video technology, 28(8):2007–2023, 2017.
[8] Zhanpeng Li, Mangui Liang, and Hongyu Liu. A selection method of path for concurrent multipath transfer based on network layer. In 2020 International Conference on Computer Engineering and Application (ICCEA), pages 559–564. IEEE, 2020.
[9] Lal Pratap Verma, Neelaksh Sheel, and Chandra Shekhar Yadev. Con-current multipath transfer using delay aware scheduling. In Innovations in Computational Intelligence and Computer Vision, pages 247–255. Springer, 2021.
[10] Honghao Gao, Yueshen Xu, Yuyu Yin, Weipeng Zhang, Rui Li, and Xinheng Wang. Context-aware qos prediction with neural collaborative filtering for internet-of-things services. IEEE Internet of Things Journal, 7(5):4532–4542, 2019.
[11] Lianyong Qi, Yi Chen, Yuan Yuan, Shucun Fu, Xuyun Zhang, and Xiaolong Xu. A qos-aware virtual machine scheduling method for energy conservation in cloud-based cyber-physical systems. World Wide Web, 23(2):1275–1297, 2020.
[12] Wenbo Zhang, Yue Liu, Guangjie Han, Yongxin Feng, and Yuntao Zhao. An energy efficient and qos aware routing algorithm based on data classification for industrial wireless sensor networks. IEEE Access, 6:46495–46504, 2018.
[13] T Jayasri and M Hemalatha. Link quality estimation for adaptive data streaming in wsn. Wireless Personal Communications, 94(3):1543–1562, 2017.
[14] Zeeshan Ansar and Waltenegus Dargie. Adaptive burst transmission scheme for wsns. In 2017 26th International Conference on Computer Communication and Networks (ICCCN), pages 1–7. IEEE, 2017.
[15] Babangida Isyaku, Kamalrulnizam Abu Bakar, Mohd Soperi Mohd Za-hid, Eman H Alkhammash, Faisal Saeed, and Fuad A Ghaleb. Route path selection optimization scheme based link quality estimation and critical switch awareness for software defined networks. Applied Sciences, 11(19):9100, 2021.
[16] Miguel L Bote-Lorenzo, Eduardo G´omez-S´anchez, Carlos Mediavilla-Pastor, and Juan I Asensio-P´erez. Online machine learning algorithms to predict link quality in community wireless mesh networks. Computer Networks, 132:68–80, 2018.
[17] Lal Pratap Verma, Varun Kumar Sharma, and Mahesh Kumar. New delay-based fast retransmission policy for cmt-sctp. International Journal of Intelligent Systems and Applications, 10(3), 2018.
[18] Nasim Arianpoo and Victor CM Leung. A smart fairness mechanism for concurrent multipath transfer in sctp over wireless multi-hop networks. Ad Hoc Networks, 55:40–49, 2017.
[19] Chengxiao Yu, Wei Quan, Deyun Gao, Yuming Zhang, Kang Liu, Wen Wu, Hongke Zhang, and Xuemin Shen. Reliable cybertwin-driven concurrent multipath transfer with deep reinforcement learning. IEEE Internet of Things Journal, 8(22):16207–16218, 2021.
[20] Changqiao Xu, Tao Zhang, Xiaohui Kuang, Zan Zhou, and Shui Yu. Context-aware adaptive route mutation scheme: A reinforcement learn-ing approach. IEEE Internet of Things Journal, 8(17):13528–13541, 2021.
[21] Tao Zhang, Xiaohui Kuang, Zan Zhou, Hongquan Gao, and Changqiao Xu. An intelligent route mutation mechanism against mixed attack based on security awareness. In 2019 IEEE Global Communications Conference (GLOBECOM), pages 1–6. IEEE, 2019.
[22] Tao Zhang, Changqiao Xu, Bingchi Zhang, Xiaohui Kuang, Yue Wang, Shujie Yang, and Gabriel-Miro Muntean. Dq-rm: Deep reinforcement learning-based route mutation scheme for multimedia services. In 2020 International Wireless Communications and Mobile Computing (IWCMC), pages 291–296. IEEE, 2020.
[23] Lyndon Ong, John Yoakum, et al. An introduction to the stream control transmission protocol (sctp). Technical report, RFC 3286 (Informational), May, 2002.
[24] Shruti Saini and Ansgar Fehnker. Evaluating the stream control trans-mission protocol using uppaal. arXiv preprint arXiv:1703.06568, 2017.
[25] Imtiaz Ali Halepoto, Muhammad Sulleman Memon, Nazar Hussain Phulpoto, Ubaidullah Rajput, and Muhammad Yaqoob Junejo. On the use of multipath transmission using sctp. IJCSNS, 18(4):58, 2018.
[26] M Tuexen, R Stewart, P Natarajan, J Iyengar, N Ekiz, T Dreibholz, M Becke, and P Amer. Load sharing for the stream control transmission protocol (sctp). IETF Draft, Individual Submission, Draft-Tuexen-Tsvwg-Sctp-Multipath-16, 2018.
[27] Johan Eklund, Karl-Johan Grinnemo, and Anna Brunstrom. Using multiple paths in sctp to reduce latency for signaling traffic. Computer Communications, 129:184–196, 2018.
[28] Joohyun Shin, Thomas A Badgwell, Kuang-Hung Liu, and Jay H Lee. Reinforcement learning–overview of recent progress and implications for process control. Computers & Chemical Engineering, 127:282–294, 2019.
[29] Martin Greguri´c, Miroslav Vuji´c, Charalampos Alexopoulos, and Mladen Mileti´c. Application of deep reinforcement learning in traffic signal control: An overview and impact of open traffic data. Applied Sciences, 10(11):4011, 2020.
[30] Kaiqing Zhang, Zhuoran Yang, and Tamer Bas¸ar. Multi-agent rein-forcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321–384, 2021.
[31] Yaodong Yang and Jun Wang. An overview of multi-agent rein-forcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583, 2020.
[32] Zichang He and Wen Jiang. An evidential markov decision making model. Information Sciences, 467:357–372, 2018.
[33] David W Hosmer and Stanley Lemeshow. Confidence interval estimation of interaction. Epidemiology, pages 452–456, 1992.
[34] Sang Gyu Kwak and Jong Hae Kim. Central limit theorem: the cornerstone of modern statistics. Korean journal of anesthesiology, 70(2):144, 2017.
[35] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279–292, 1992.
[36] Francisco S Melo. Convergence of q-learning: A simple proof. Institute Of Systems and Robotics, Tech. Rep, pages 1–4, 2001.
[37] Thomas Dreibholz. Multi-path transport with omnet++ and the inet framework. 2017.
[38] Hilal H Nuha and Sidik Prabowo. Tcp congestion window analysis of twitter with exponential model. In 2018 6th International Conference on Information and Communication Technology (ICoICT), pages 61–65. IEEE, 2018.