DRL-based Energy-Efficient Trajectory Planning for Multiple UAVs under Centralized Control

Limei Peng

¹Applied Data Science, Noroff University College, Kristiansand, Norway

²School of Computer Science and Engineering, Kyungpook National University, Daegu, South Korea

³Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada

*Corresponding author

Non-terrestrial networks (NTNs), comprising unmanned aerial vehicles (UAVs) with varying battery and computational capacities, are key technologies for implementing 5G and beyond (B5G) as well as 6G. However, the integration and orchestration of different NTN layers remain underexplored. This paper investigates a two-layer NTN architecture, featuring a high-altitude platform (HAP) with either satellites or high-performance UAVs, and low-altitude UAVs (LAUs) with limited capacity, tasked with collecting data from terrestrial Internet-of-Things (IoT) nodes. We delve into the dynamics of the NTN, focusing on a framework where multiple capacity-constrained LAUs are coordinated by a centralized HAP. The proposed research involves devising optimized trajectories for these cooperative LAUs, under HAP guidance, to boost energy efficiency in data collection. The proposed work tackles this challenge by developing two integer linear programming (ILP) optimization models and introducing a novel algorithm named collaborative multi-agent energy-efficient trajectory design and data collection (CoMETD). The proposed CoMETD, operating within the HAP, leverages a deep reinforcement learning (DRL)-based dueling double deep Q-learning network (D3QN) to dynamically plan multi-LAU trajectories, eliminating the need for prior knowledge of IoT node locations. The effectiveness of the proposed algorithm is validated through extensive simulations, where its performance is compared with contemporary state-of-the-art methods.

Deep reinforcement learning, UAVs, energy efficiency, data collection, trajectory planning, HAP, aerial computing

Shahnila Rahim, Limei Peng, and Pin-Han Ho (2024). DRL-based Energy-Efficient Trajectory Planning for Multiple UAVs under Centralized Control. Journal of Networking and Network Applications, Volume 4, Issue 3, pp. 118–128. https://doi.org/10.33969/J-NaNA.2024.040303.

[1] H. Mei and L. Peng, “On multi-robot data collection and offloading for space-aerial-surface computing,” IEEE Wireless Communications, vol. 30, no. 2, pp. 90–96, 2023.

[2] M. Li, N. Cheng, J. Gao, Y. Wang, L. Zhao, and X. Shen, “Energy-efficient uav-assisted mobile edge computing: resource allocation and trajectory optimization,” IEEE Transactions on Vehicular Technology, vol. 69, no. 3, pp. 3424–3438, 2020.

[3] G. K. Kurt, M. G. Khoshkholgh, S. Alfattani, A. Ibrahim, T. S. Darwish, M. S. Alam, H. Yanikomeroglu, and A. Yongacoglu, “A vision and framework for the high altitude platform station (haps) networks of the future,” IEEE Communications Surveys & Tutorials, vol. 23, no. 2, pp. 729–779, 2021.

[4] S. Rahim and L. Peng, “Intelligent space-air-ground collaborative com-puting networks,” IEEE Internet of Things Magazine, vol. 6, no. 2, pp. 76–80, 2023.

[5] S. Rahim, M. M. Razaq, S. Y. Chang, and L. Peng, “A reinforcement learning-based path planning for collaborative uavs,” in Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, pp. 1938–1943, 2022.

[6] Q. Wu, Y. Zeng, and R. Zhang, “Joint trajectory and communication design for multi-uav enabled wireless networks,” IEEE Transactions on Wireless Communications, vol. 17, no. 3, pp. 2109–2121, 2018.

[7] B. Zhu, E. Bedeer, H. H. Nguyen, R. Barton, and J. Henry, “Joint cluster head selection and trajectory planning in uav-aided iot networks by reinforcement learning with sequential model,” IEEE Internet of Things Journal, 2021.

[8] S. Rahim, L. Peng, S. Chang, and P.-H. Ho, “On collaborative multi-uav trajectory planning for data collection,” Journal of Communications and Networks, vol. 25, no. 6, pp. 722–733, 2023.

[9] O. Esrafilian, R. Gangula, and D. Gesbert, “Learning to communicate in uav-aided wireless networks: Map-based approaches,” IEEE Internet of Things Journal, vol. 6, no. 2, pp. 1791–1802, 2019.

[10] M. Alzenad, A. El-Keyi, F. Lagum, and H. Yanikomeroglu, “3-d place-ment of an unmanned aerial vehicle base station (uav-bs) for energy-efficient maximal coverage,” IEEE Wireless Communications Letters, vol. 6, no. 4, pp. 434–437, 2017.

[11] Y. Yuan, L. Lei, T. X. Vu, S. Chatzinotas, and B. Ottersten, “Actor-critic deep reinforcement learning for energy minimization in uav-aided networks,” in 2020 European Conference on Networks and Communi-cations (EuCNC), pp. 348–352, 2020.

[12] H. Ahmadinejad and A. Falahati, “Forming a two-tier heterogeneous air-network via combination of high and low altitude platforms,” IEEE Transactions on Vehicular Technology, vol. 71, no. 2, pp. 1989–2001, 2021.

[13] A. Masood, T.-V. Nguyen, T. P. Truong, and S. Cho, “Content caching in hap-assisted multi-uav networks using hierarchical federated learning,” in 2021 International Conference on Information and Communication Technology Convergence (ICTC), pp. 1160–1162, 2021.

[14] D. T. Hua, D. S. Lakew, and S. Cho, “Drl-based energy efficient com-munication coverage control in hierarchical hap-lap network,” in 2022 international conference on information networking (ICOIN), pp. 359–362, IEEE, 2022.

[15] A. H. Arani, P. Hu, and Y. Zhu, “Haps-uav-enabled heterogeneous networks: A deep reinforcement learning approach,” arXiv preprint arXiv:2303.12883, 2023.

[16] B. Jabbari, Y. Zhou, and F. Hillier, “Simple random walk models for wireless terminal movements,” in 1999 IEEE 49th Vehicular Technology Conference (Cat. No. 99CH36363), vol. 3, pp. 1784–1788, IEEE, 1999.

[17] H. He, S. Zhang, Y. Zeng, and R. Zhang, “Joint altitude and beamwidth optimization for uav-enabled multi-user communications,” IEEE Com-munications Letters, vol. 22, no. 2, pp. 344–347, 2017.

[18] Q. Ren, O. Abbasi, G. K. Kurt, H. Yanikomeroglu, and J. Chen, “Caching and computation offloading in high altitude platform station (haps) assisted intelligent transportation systems,” IEEE Transactions on Wireless Communications, vol. 21, no. 11, pp. 9010–9024, 2022.

[19] S. S. Khodaparast, X. Lu, P. Wang, and U. T. Nguyen, “Deep reinforce-ment learning based energy efficient multi-uav data collection for iot networks,” IEEE Open Journal of Vehicular Technology, vol. 2, pp. 249–260, 2021.

[20] C. You and R. Zhang, “3d trajectory optimization in rician fading for uav-enabled data harvesting,” IEEE Transactions on Wireless Commu-nications, vol. 18, no. 6, pp. 3192–3207, 2019.

[21] H. Cao, G. Yu, and Z. Chen, “Cooperative task offloading and dispatch-ing optimization for large-scale users via uavs and hap,” in 2023 IEEE Wireless Communications and Networking Conference (WCNC), pp. 1–6, IEEE, 2023.

[22] Z. Jia, M. Sheng, J. Li, D. Niyato, and Z. Han, “Leo-satellite-assisted uav: Joint trajectory and data collection for internet of remote things in 6g aerial access networks,” IEEE Internet of Things Journal, vol. 8, no. 12, pp. 9814–9826, 2020.

[23] J. Ren, Z. Chai, and Z. Chen, “Joint spectrum allocation and power control in vehicular communications based on dueling double dqn,” Vehicular Communications, vol. 38, p. 100543, 2022.

[24] D. J. Birabwa, D. Ramotsoela, and N. Ventura, “Multi-agent deep reinforcement learning for user association and resource allocation in integrated terrestrial and non-terrestrial networks,” Computer Networks, vol. 231, p. 109827, 2023.

[25] “The promise and challenges of airborne wind energy.” https://physicsworld.com/a/thepromise-and-challenges-of-airborne-wind-energy/, 2022-04-26.