Federated Continual Learning based on Central Memory Rehearsal

Bin Yu

¹School of Computer Science and Technology, Xidian University, Xi’an, Shaanxi, 710126, China

*Corresponding author

Federated learning, hailed as a transformative approach, fosters collaborative and secure acquisition of a unified model within the domain of Industrial Internet of Things (IIoT). This innovative paradigm enables multiple clients to collectively contribute to model training while preserving data privacy, leveraging the coordination of a central server. In the real world, most smart edge devices of the IIoT are always confronted with considerable data in the form of sequential data streams. However, current federated learning models suffer a sharp drop in performance when dealing with sequential data, which is called catastrophic forgetting. Consequently, A crucial obstacle encountered in practical implementations of federated learning revolves around the need to address the issue of catastrophic forgetting, thus enabling it to acquire and retain knowledge across multiple tasks, akin to human capabilities. In this paper, we propose a novel framework, called Federated Central Memory Rehearsal (FedCMR), which is inspired by the rehearsal method of continual learning. Specifically, the Generator model, trained by the central server, is tasked with the responsibility of creating the pseudo data (Central Memory) associated with previous tasks. The pseudo data refers to synthetic data generated by the model, which serves to mimic the data from older tasks. This synthetic data is crucial for rehearsal-based learning, allowing local models to retain knowledge from earlier tasks even when only the current task’s data is available for training. Upon the arrival of a new task, the local client mixes a small amount of pseudo data with the local dataset for training, with the aim of maintaining the knowledge of old tasks (Rehearsal). Upon the completion of training for the current task by each local client, they proceed to upload their respective local models and the sampled data from the current task, fortified with differential privacy noise, to the central server. Subsequently, the server consolidates the collected local models, crafting a novel global model. Additionally, it generates a limited amount of synthetic data representing past tasks, which is then disseminated to each client for secure and collaborative training purposes. Experimental results demonstrate that FedCMR overcomes catastrophic forgetting while realizing privacy preserving and reducing communication costs.

Federated Learning, Privacy Preserving, Sensitive and Private Information

Chen Zhang, Tielin Huang, Wenjie Mao, Hang Bai, and Bin Yu (2024). Federated Continual Learning based on Central Memory Rehearsal. Journal of Networking and Network Applications, Volume 4, Issue 2, pp. 81–93. https://doi.org/10.33969/J-NaNA.2024.040204.

[1] Jakub Koneˇcn`y, H Brendan McMahan, Felix X Yu, Peter Richt´arik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.

[2] Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge-Based Systems, page 106775, 2021.

[3] H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication-efficient learning of deep networks from decen-tralized data. In Artificial intelligence and statistics, pages 1273–1282, 2017.

[4] Weixi Wang, Fan He, Yulei Li, Shengjun Tang, Xiaoming Li, Jizhe Xia, and Zhihan Lv. Data information processing of traffic digital twins in smart cities using edge intelligent federation learning. Information Processing & Management, 60(2):103171, 2023.

[5] Wenshuo Wang, Xu Li, Xiuqin Qiu, Xiang Zhang, Vladimir Brusic, and Jindong Zhao. A privacy preserving framework for federated learning in smart healthcare systems. Information Processing & Management, 60(1):103167, 2023.

[6] Zhaohua Zheng, Yize Zhou, Yilong Sun, Zhang Wang, Boyi Liu, and Keqiu Li. Applications of federated learning in smart cities: recent advances, taxonomy, and open challenges. Connection Science, 34(1):1–28, 2022.

[7] Jan Philipp Albrecht. How the gdpr will change the world. Eur. Data Prot. L. Rev., 2:287, 2016.

[8] Ittai Dayan, Holger R Roth, Aoxiao Zhong, Ahmed Harouni, Amilcare Gentili, Anas Z Abidin, Andrew Liu, Anthony Beardsworth Costa, Bradford J Wood, Chien-Sung Tsai, et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nature medicine, 27(10):1735–1743, 2021.

[9] Viktor Losing, Barbara Hammer, and Heiko Wersing. Incremental on-line learning: A review and comparison of state of the art algorithms. Neurocomputing, 275:1261–1274, 2018.

[10] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learn-ing survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.

[11] Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In International Conference on Machine Learning, pages 12073–12086. PMLR, 2021.

[12] Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10164–10173, 2022.

[13] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017.

[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.

[15] Syreen Banabilah, Moayad Aloqaily, Eitaa Alsayed, Nida Malik, and Yaser Jararweh. Federated learning review: Fundamentals, enabling technologies, and future applications. Information Processing & Man-agement, 59(6):103061, 2022.

[16] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429–450, 2020.

[17] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50–60, 2020.

[18] Zeinab Teimoori, Abdulsalam Yassine, and M Shamim Hossain. A secure cloudlet-based charging station recommendation for electric vehi-cles empowered by federated learning. IEEE Transactions on Industrial Informatics, 2022.

[19] Xu Cheng, Fan Shi, Yongping Liu, Jiehan Zhou, Xiufeng Liu, and Lizhen Huang. A class-imbalanced heterogeneous federated learning model for detecting icing on wind turbine blades. IEEE Transactions on Industrial Informatics, 2022.

[20] Qing Han, Shusen Yang, Xuebin Ren, Peng Zhao, Cong Zhao, and Yimeng Wang. Pcfed: Privacy-enhanced and communication-efficient federated learning for industrial iots. IEEE Transactions on Industrial Informatics, 2022.

[21] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual learning with neural networks: A review. Neural Networks, 113:54–71, 2019.

[22] Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12:1–207, 2018.

[23] Martial Mermillod, Aur´elia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: Investigating the continuum from catastrophic for-getting to age-limited learning effects. Frontiers in psychology, page 504, 2013.

[24] Dong Li, Shulin Liu, Furong Gao, and Xin Sun. Continual learning classification method with new labeled data based on the artificial immune system. Applied Soft Computing, 94:106423, 2020.

[25] Qiubing Ren, Heng Li, Mingchao Li, Ting Kong, and Runhao Guo. Bayesian incremental learning paradigm for online monitoring of dam behavior considering global uncertainty. Applied Soft Computing, 143:110411, 2023.

[26] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.

[27] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.

[28] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.

[29] Fernando E Casado, Dylan Lema, Roberto Iglesias, Carlos V Regueiro, and Sen´en Barro. Collaborative and continual learning for classification tasks in a society of devices. arXiv e-prints, pages arXiv–2006, 2020.

[30] Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10143–10153, 2022.

[31] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.

[32] Yann LeCun, L´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.

[33] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.

[34] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.