Spectrum Allocation for Covert Communications in Cellular-Enabled UAV Networks: A Deep Reinforcement Learning Approach

Xinzhe Pi

¹ School of Systems Information Science, Future University Hakodate, Hakodate, Hokkaido, 041-8655, Japan

² School of Computer and Information Engineering, Chuzhou University, Chuzhou, 239000, China

³ MOSAIC Lab (www.mosaic-lab.org), Espoo 02150, Finland

^*Corresponding author

This paper investigates the covert communications via spectrum allocations in a cellular-enabled unmanned aerial vehicle (UAV) network consisting of a base station (BS), UAVs, ground users (GUs), and a warden, where warden attempts to detect the transmission from a target GU to a UAV receiver. We formulate the spectrum allocation as an optimization problem with the constraints of covertness performance requirement and the qualities of service (QoS) of cellular communications. This is a nonlinear and nonconvex problem, which is generally challenging to be solved. Thus, we propose a deep reinforcement learning (DRL) approach to solve it. Under such an approach, we first model the multi-agent DRL environment in such networks. Then we define the state, action, reward and interaction mechanism of the DRL environment. Finally, a DRL algorithm is presented for learning the optimal policy of spectrum allocation.

Covert communication, cellular-enabled UAV network, spectrum allocation, deep reinforcement learning, multi-agent reinforcement learning

Xinzhe Pi, Bin Yang (2022). Spectrum Allocation for Covert Communications in Cellular-Enabled UAV Networks: A Deep Reinforcement Learning Approach. Journal of Networking and Network Applications, Volume 2, Issue 3, pp. 107–115. https://doi.org/10.33969/J-NaNA.2022.020302.

[1] M. Mozaffari, W. Saad, M. Bennis, Y.-H. Nam, and M. Debbah, “A Tutorial on UAVs for Wireless Networks: Applications, Challenges, and Open Problems,” IEEE Communications Surveys Tutorials, vol. 21, no. 3, pp. 2334–2360, 2019.

[2] B. A. Bash, D. Goeckel, and D. Towsley, “Square root law for commu-nication with low probability of detection on AWGN channels,” in 2012 IEEE International Symposium on Information Theory Proceedings, 2012, pp. 448–452.

[3] Y. Jiang, L. Wang, H. Zhao, and H.-H. Chen, “Covert Communications in D2D Underlaying Cellular Networks With Power Domain NOMA,” IEEE Systems Journal, vol. 14, no. 3, pp. 3717–3728, 2020.

[4] X. Zhou, S. Yan, J. Hu, J. Sun, J. Li, and F. Shu, “Joint Optimization of a UAV’s Trajectory and Transmit Power for Covert Communications,” IEEE Transactions on Signal Processing, vol. 67, no. 16, pp. 4276–4290, 2019.

[5] H.-M. Wang, Y. Zhang, X. Zhang, and Z. Li, “Secrecy and Covert Communications Against UAV Surveillance via Multi-Hop Networks,” IEEE Transactions on Communications, vol. 68, no. 1, pp. 389–401, 2020.

[6] A. Sheikholeslami, M. Ghaderi, D. Towsley, B. A. Bash, S. Guha, and

D. Goeckel, “Multi-Hop Routing in Covert Wireless Networks,” IEEE Transactions on Wireless Communications, vol. 17, no. 6, pp. 3656–3669, 2018.

[7] S. Yan, S. V. Hanly, and I. B. Collings, “Optimal Transmit Power and Flying Location for UAV Covert Wireless Communications,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 11, pp. 3321–3333, 2021.

[8] S. Yan, S. V. Hanly, I. B. Collings, and D. L. Goeckel, “Hiding unmanned aerial vehicles for wireless transmissions by covert com-munications,” in ICC 2019 - 2019 IEEE International Conference on Communications (ICC), 2019, pp. 1–6.

[9] X. Jiang, Z. Yang, N. Zhao, Y. Chen, Z. Ding, and X. Wang, “Resource Allocation and Trajectory Optimization for UAV-Enabled Multi-User Covert Communications,” IEEE Transactions on Vehicular Technology, vol. 70, no. 2, pp. 1989–1994, 2021.

[10] N. C. Luong, D. T. Hoang, S. Gong, D. Niyato, P. Wang, Y.-C. Liang, and D. I. Kim, “Applications of Deep Reinforcement Learning in Communications and Networking: A Survey,” IEEE Communications Surveys Tutorials, vol. 21, no. 4, pp. 3133–3174, 2019.

[11] X. Zhang, Z. Lin, B. Ding, B. Gu, and Y. Han, “Deep Multi-Agent Reinforcement Learning for Resource Allocation in D2D Communica-tion Underlaying Cellular Networks,” in 2020 21st Asia-Pacific Network Operations and Management Symposium (APNOMS), 2020, pp. 55–60.

[12] J. Cui, Y. Liu, and A. Nallanathan, “Multi-Agent Reinforcement Learning-Based Resource Allocation for UAV Networks,” IEEE Trans-actions on Wireless Communications, vol. 19, no. 2, pp. 729–743, 2020.

[13] C. H. Liu, Z. Chen, J. Tang, J. Xu, and C. Piao, “Energy-Efficient UAV Control for Effective and Fair Communication Coverage: A Deep Reinforcement Learning Approach,” IEEE Journal on Selected Areas in Communications, vol. 36, no. 9, pp. 2059–2070, 2018.

[14] K. Chi, Z. Chen, K. Zheng, Y.-h. Zhu, and J. Liu, “Energy Provision Minimization in Wireless Powered Communication Networks With Network Throughput Demand: TDMA or NOMA?” IEEE Transactions on Communications, vol. 67, no. 9, pp. 6401–6414, 2019.

[15] Z. Chen, K. Chi, K. Zheng, Y. Li, and X. Liu, “Common Throughput Maximization in Wireless Powered Communication Networks With Non-Orthogonal Multiple Access,” IEEE Transactions on Vehicular Technology, vol. 69, no. 7, pp. 7692–7706, 2020.

[16] X. Liao, J. Si, J. Shi, Z. Li, and H. Ding, “Generative Adversarial Network Assisted Power Allocation for Cooperative Cognitive Covert Communication System,” IEEE Communications Letters, vol. 24, no. 7, pp. 1463–1467, 2020.

[17] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” in Advances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014, pp. 2672–2680.

[18] B. Kim, Y. E. Sagduyu, K. Davaslioglu, T. Erpek, and S. Ulukus, “How to Make 5G Communications ”Invisible”: Adversarial Machine Learning for Wireless Privacy,” in 2020 54th Asilomar Conference on Signals, Systems, and Computers, 2020, pp. 763–767.

[19] T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. Foer-ster, and S. Whiteson, “QMIX: Monotonic Value Function Factorisa-tion for Deep Multi-Agent Reinforcement Learning,” arXiv preprint arXiv:1803.11485, 2018.

[20] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments,” 2020.