隨著人工智慧與行動裝置的快速發展，以及對資料隱私保護需求的提升，聯邦學習憑藉其既能保護隱私又能協同訓練模型的特性，被視為一種有前景的分散式學習技術。聯邦學習透過頻繁與中央伺服器傳輸模型資訊並進行聚合，取代傳統的敏感性資料交換，這使得通訊變得尤為重要，然而，傳統框架在大量裝置環境下會使中央伺服器的工作量迅速增加。為應對此挑戰，本篇論文提出一個旨在提升通訊效率的循序知識蒸餾之聯邦學習框架，該框架利用裝置對裝置通訊以減少中央伺服器與裝置之間的上傳次數，並提出一種基於截止時間的模型壓縮上傳策略，使用腦浮點數 (brain floating point) 作為壓縮的數值格式，顯著降低資料傳輸量及中央伺服器網路處理工作量；為了應對資料異質性，裝置選擇策略根據裝置的資料分布選擇下一個進行循序知識蒸餾學習的裝置，以減輕資料非獨立同分布對訓練造成的負面影響；針對網路與計算資源異質性，裝置在分享模型時會執行裝置對裝置通訊方向引導，有效利用上傳速度較快的裝置優勢，讓更多裝置參與訓練。模擬結果顯示，與傳統框架相比，提出的框架顯著降低傳輸量，並減少裝置向伺服器上傳模型的次數，同時提高準確度，此外，循序知識蒸餾聯邦學習框架在各種程度的非獨立同分布場景及裝置數量龐大的環境中均展現出卓越的性能。

With the rapid advancement of artificial intelligence (AI) and mobile devices, along with the increasing demand for data privacy protection, federated learning (FL) is considered a promising distributed learning technique due to its ability to both privacy protection and collaboration with model training. FL replaces sensitive data exchange with frequent communication and aggregation of model information with the central server, which leads to the importance of communication. However, traditional frameworks escalate the workload on the central server rapidly in environments with a large number of devices. To address this challenge, this thesis presents a sequential knowledge distillation federated learning (SKDFL) framework, with the design of communication efficiency. SKDFL leverages device-to-device (D2D) communication to reduce the number of uploads between the central server and devices. A deadline-driven upload strategy is proposed to reduce devices’ data transfer size and to alleviate central server’s processing workload by using the brain floating-point (Bfloat16) as a numerical format for compression. To address data heterogeneity, the device selection strategy selects the next device for sequential knowledge distillation learning (SKDL) based on the data distribution across devices, thus it mitigates the negative impact of non-independent and identically distributed (non-IID) data on training. To address network and computational resource heterogeneity, the process of device model sharing incorporates with the direction guidance of D2D communication (DGDC). The DGDC leverages the advantages of devices with faster upload data rates to involve more devices in training. Simulation results demonstrate that the proposed framework significantly reduces the number of uploads and data transfer size, while improving accuracy compared with traditional frameworks. Moreover, SKDFL yields superior performance in various degrees of non-IID scenarios and environments with a massive number of devices.

[1] J. P. Albrecht, “How the GDPR will change the world,” Eur. Data Protection Law Rev., vol. 2, pp. 287–289, 2016.
[2] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), vol. 54, pp. 1273–1282, April 2017.
[3] L. Li, Y. Fan, M. Tse, and K.-Y. Lin, “A review of applications in federated learning,” Computers & Industrial Engineering, vol. 149, Nov. 2020, Art. no. 106854.
[4] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent Systems and Technology, vol. 10, no. 2, pp. 1–19, Jan. 2019.
[5] S. Augenstein et al., “Generative Models for Effective ML on Private, Decentralized Datasets,” International Conference on Learning Representations (ICLR), 2020, pp. 1–26.
[6] X. Zhang, Y. Liu, J. Liu, A. Argyriou, and Y. Han, “D2D-assisted federated learning in mobile edge computing networks,” IEEE Wireless Communications and Networking Conference, 2021, pp. 1–7.
[7] Y. Liu, L. Su, C. Joe-Wong, S. Ioannidis, E. Yeh, and M. Siew, “Cache-Enabled federated learning systems,” in Proceedings of the Twentyfourth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, 2023, pp. 1–11.
[8] T. Nishio and R. Yonetani, “Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge,” ICC 2019 - 2019 IEEE International Conference on Communications (ICC), Shanghai, China, 2019, pp. 1-7.
[9] M. A. Hidayat, Y. Nakamura and Y. Arakawa, “Efficient and Secure: Privacy-Preserving Federated Learning for Resource-Constrained Devices,” in 2023 24th IEEE International Conference on Mobile Data Management (MDM), Singapore, 2023, pp. 184-187.
[10] A. Imteaj, U. Thakker, S. Wang, J. Li and M. H. Amini, “A Survey on Federated Learning for Resource-Constrained IoT Devices,” IEEE Internet of Things Journal, vol. 9, no. 1, pp. 1-24, 1 Jan.1, 2022.
[11] K. M. Ahmed, A. Imteaj and M. H. Amini, “Federated Deep Learning for Heterogeneous Edge Computing,” 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, CA, USA, 2021, pp. 1146-1152.
[12] H. Nguyen, H. Warrier, and Y. Gupta, “A novel approach for federated learning with Non-IID data,” International Conference on Soft Computing & Machine Intelligence (ISCMI), 2022, pp. 62-67.
[13] F. Sattler, S. Wiedemann, K. -R. Müller and W. Samek, “Robust and Communication-Efficient Federated Learning from Non-i.i.d. Data,” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 9, pp. 3400-3413, Sept. 2020.
[14] H. Wu and P. Wang, “Node Selection Toward Faster Convergence for Federated Learning on Non-IID Data,” IEEE Transactions on Network Science and Engineering, vol. 9, no. 5, pp. 3099-3111, 1 Sept.-Oct. 2022.
[15] J. Xu, S. -L. Huang, L. Song and T. Lan, “Live Gradient Compensation for Evading Stragglers in Distributed Learning,” IEEE INFOCOM 2021 - IEEE Conference on Computer Communications, Vancouver, BC, Canada, 2021, pp. 1-10.
[16] J. Lee, H. Ko and S. Pack, “Adaptive Deadline Determination for Mobile Device Selection in Federated Learning,” IEEE Transactions on Vehicular Technology, vol. 71, no. 3, pp. 3367-3371, March 2022.
[17] M. Zhang, Z. Cai, D. Liu, R. Jin, G. Zhu and C. Zhong, “Joint Compression and Deadline Optimization for Communication-Efficient Federated Edge Learning,” 2023 IEEE Globecom Workshops (GC Wkshps), Kuala Lumpur, Malaysia, 2023, pp. 521-526.
[18] W. Y. B. Lim et al., “Federated learning in mobile edge networks: A comprehensive survey,” IEEE Communications Surveys & Tutorials, vol. 22, no. 3, pp. 2031–2063, 3rd Quart., 2020.
[19] Y. Shi, W. Li , X. Chang, T. Yang, Y. Sun, and A. Y. Zomaya, “On enabling collaborative non-intrusive load monitoring for sustainable smart cities,” Sci Rep, vol. 13, no. 1, p. 6569, Dec. 2023.
[20] N. Shibahara, M. Koibuchi and H. Matsutani, “Performance Improvement of Federated Learning Server using Smart NIC,” 2023 Eleventh International Symposium on Computing and Networking Workshops (CANDARW), Matsue, Japan, pp. 165-171, 2023.
[21] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015, arXiv:1503.02531. [Online]. Available: https://arxiv.org/abs/1503.02531.
[22] S. Kullback and R. A. Leibler, “On information and sufficiency,” The Annals of Mathematical Statistics, vol. 22, no. 1, pp. 79–86, 1951.
[23] Intel, “Bfloat16 Hardware Numerics Definition White Paper”.
[24] N. Burgess, J. Milanovic, N. Stephens, K. Monachopoulos and D. Mansell, “Bfloat16 Processing for Neural Networks,” 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), Kyoto, Japan, pp. 88-91, 2019.
[25] D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, N. Jammalamadaka, J. Huang, H. Yuen, J. Yang, J. Park, A. Heinecke, E. Georganas, S. Srinivasan, A. Kundu, M. Smelyanskiy, B. Kaul, and P. Dubey, “A study of BFLOAT16 for deep learning training,” 2019, arXiv:1905.12322. [Online]. Available: https://arxiv.org/abs/1905.12322.
[26] W. Liu, L. Chen, Y. Chen and W. Zhang, "Accelerating Federated Learning via Momentum Gradient Descent," IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 8, pp. 1754-1766, 1 Aug. 2020.
[27] A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, “FedPAQ: A communication-efficient federated learning method with periodic averaging and quantization,” International Conference on Artificial Intelligence and Statistics (AISTATS), 2020, pp. 2021–2031.
[28] Y. Xu, Y. Liao, H. Xu, Z. Ma, L. Wang and J. Liu, “Adaptive Control of Local Updating and Model Compression for Efficient Federated Learning,” IEEE Transactions on Mobile Computing, vol. 22, no. 10, pp. 5675-5689, 1 Oct. 2023.
[29] S. Ji, W. Jiang, A. Walid and X. Li, “Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning,” IEEE Intelligent Systems, vol. 37, no. 2, pp. 27-34, 1 March-April 2022.
[30] F.-H Tseng and Y.-H Huang, “FedBF16-Dynamic: Communication-Efficient Federated Learning with Adaptive Transmission,” IEEE INFOCOM 2024 Workshop on Integrating Edge Intelligence and Large Model in Next Generation Networks (IEILM), Vancouver, Canada, May 20, 2024, pp.1-6.
[31] M. Duan, D. Liu, X. Chen, R. Liu, and Y. Tan, “Self-balancing federated learning with global imbalanced data in mobile systems,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 1, pp. 59–71, Jul. 2021.
[32] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” Machine Learning and Systems, vol. 2, pp. 429–450, 2020.
[33] S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh, “SCAFFOLD: Stochastic controlled averaging for federated learning,” International Conference on Machine Learning (ICML), pp. 5132–5143, 2020.
[34] “Evolved Universal Terrestrial Radio Access (E-UTRA); Radio Frequency (RF) system scenarios,” 3GPP TR 36.942 version 13.0.0, 2016.
[35] Y. Lu, X. Chen, Y. Zhang and Y. Chen, "Cost-Efficient Resources Scheduling for Mobile Edge Computing in Ultra-Dense Networks," IEEE Transactions on Network and Service Management, vol. 19, no. 3, pp. 3163-3173, Sept. 2022.
[36] W. Zhang, X. Wang, P. Zhou, W. Wu and X. Zhang, "Client Selection for Federated Learning with Non-IID Data in Mobile Edge Computing," IEEE Access, vol. 9, pp. 24462-24474, 2021.
[37] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” University of Toronto, Toronto, Ontario, Tech. Rep. 0, 2009. [Online]. Available: https://www.cs.toronto.edu/kriz/learningfeatures-2009-TR.pdf.
[38] Q. Li, B. He and D. Song, "Model-Contrastive Federated Learning," 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 10708-10717.
[39] B. Gong, T. Xing, Z. Liu, W. Xi and X. Chen, "Towards Hierarchical Clustered Federated Learning with Model Stability on Mobile Devices," IEEE Transactions on Mobile Computing, vol. 23, no. 6, pp. 7148-7164, June 2024.
[40] J. Zhang, L. Chen, Y. Chen, X. Chen and G. Wei, "Hierarchically Federated Learning in Wireless Networks: D2D Consensus and Inter-Cell Aggregation," IEEE Transactions on Machine Learning in Communications and Networking, vol. 2, pp. 442-456, 2024.