CN117708877B - Personalized federal learning method and system for hybrid multi-stage private model - Google Patents

Abstract

The invention provides a personalized federation learning method and a personalized federation learning system for a hybrid multi-stage private model, comprising the following steps: downloading a global model and a global prototype through a client, and initializing a local model, a personalized model and the local prototype; training a local model by using local data through a global model and a prototype regular term, carrying out weighted mixing on the local model and a historical local model, and training a personalized model by using the local data; the historical local model and the personalized model are weighted and mixed to generate a new personalized model, and then a local prototype is generated through the personalized model; the client uploads the local model and the local prototype to the server; the server aggregates the individual models and prototypes to generate global models and global prototypes, which are then distributed to the individual clients. The method introduces the concept of prototype learning, aggregates the local prototypes of different clients to obtain the global prototype, enhances the performance of the historical local model through regularization terms, and ensures the stability and accuracy of the model.

Description

Personalized federated learning method and system for hybrid multi-stage private models

Technical Field

The present invention relates to the field of federated learning and distributed computing, and in particular to a personalized federated learning method and system of a hybrid multi-stage private model.

Background technique

In supervised learning, the generalization ability of the model depends on the scale of labeled data. Although our world has a huge amount of data created every day, they have the following characteristics:

(1) Distributed: Federated learning is applied to various heterogeneous data sources, which can come from different devices, geographical locations, network conditions, etc. These heterogeneous data may have different distribution characteristics and data types. Data from different devices or users are usually not independent and identically distributed (Non-IID), that is, their data distributions may be very different. This inconsistency increases the challenges of model training in a distributed environment.

(2) Privacy: Because of privacy, data sharing between ends (or data centers) becomes difficult. At the same time, laws and regulations on data privacy are becoming more and more stringent, such as the General Data Protection Regulation issued by the European Union. In particular, data privacy has received a lot of attention in medical scenarios. Federated learning is usually performed on local devices and involves user privacy data. Therefore, it is crucial to ensure that user privacy is protected during data processing and transmission on local devices.

(3) High data transmission costs: The transmission cost of massive data is high. Transmitting large amounts of data involves network communication overhead, especially on large-scale data sets. In some cases, data transmission may result in expensive costs.

In dealing with these characteristics, the federated learning research field has been working hard to develop new algorithms and technologies to ensure effective model training in a distributed environment while protecting user privacy, reducing data transmission costs, and processing heterogeneous and non-independent and identically distributed data. With the deepening of research, federated learning is expected to become an effective method for processing large-scale, heterogeneous and sensitive data. Under the Non-IID setting, the optimization direction of each data center is inconsistent, resulting in poor generalization ability of the global model. Considering that the starting point of each participant is just to have a model with stronger performance for local data, personalized federated learning is to solve the problems of slow convergence on highly heterogeneous (non-IID) data, poor performance and lack of personalization of the model for local tasks or data sets in federated learning.

Patent document CN117035057A discloses a personalized federated learning method based on model and data distillation, with the following steps: construct a local model on the client, including a shared encoder and a private decoder. The client trains the local model based on the private data set, uploads the model parameters of the shared encoder to the server, calculates the output logits of the public data set based on the local model and uploads the logits to the server, the server performs weighted aggregation on the logits and the shared encoder model parameters of multiple clients, obtains global logits and multiple global encoder model parameters, each client downloads multiple global encoder models, updates multiple local shared encoder model parameters in the client, downloads global logits, and participates in the training of the decoder in the form of knowledge distillation. However, this patent cannot completely solve the current technical problems, nor can it meet the needs of the present invention.

Summary of the invention

In view of the defects in the prior art, the purpose of the present invention is to provide a personalized federated learning method and system of a hybrid multi-stage private model.

The personalized federated learning method of the hybrid multi-stage private model provided by the present invention includes:

Step S1: download the global model and global prototype through the client, and initialize the local model, personalized model and local prototype;

Step S2: Using local data to train a local model through a global model and a prototype regularization term, weighting and mixing the local model with the historical local model, and using local data to train a personalized model;

Step S3: weighted mixing of the historical local model and the personalized model to generate a new personalized model, and then generating a local prototype through the personalized model;

Step S4: The client uploads the local model and the local prototype to the server and waits for a new round of model parameters;

Step S5: The server aggregates the various models and prototypes to generate a global model and a global prototype, and then distributes them to various clients.

Preferably, step S2 comprises:

The global class prototype representing the global data feature distribution is obtained by calculation. A regularization term is introduced during local model training to correct the deviation of local training. The expression is:

Among them, L ₂ represents the Euclidean distance; l _i represents the cross entropy loss function; Represents the global class prototype of the j-th type of data; represents the local class prototype of client i; w is the local model parameter; x is the input data; y is the true label of the input data x; C is the image category prototype representation; λ is a hyperparameter that controls the penalty weight of the regularization term;

The local model is then weightedly mixed with the historical local model to preserve the global information, expressed as:

Among them, history represents the historical local model after accumulating each round of local models wi _,t ; t represents the iteration round; β is a hyperparameter that controls the model mixing ratio; _wi,t represents the local model parameters of the tth round of the i-th client.

Preferably, step S3 comprises:

In order to achieve personalization by mixing local information with global information, the historical local model and the personalized model are weighted and mixed to obtain a new round of personalized model, which is expressed as:

in, is a personalized model, which is used for local training to generate the next round of p _i,t+1 and local class prototypes; t is the number of iterations; μ is a hyperparameter that controls the model mixing ratio; the personalized model is trained with local data for one round to obtain p _i,t , and this model is mixed with the historical local model to obtain a new round of personalized model

Preferably, the model p is divided into a representation layer and a prediction layer, expressed as:

p∶＝[ ^pr + ^pd ]

z＝g(x； ^pr )

y＝h(x；p ^d )

Among them, p ^r and p ^d represent the representation layer parameters and prediction layer parameters of the model respectively; z represents the output after the input x passes through the representation layer; y represents the output after the input x passes through the prediction layer; function g represents the result after the input data passes through the model representation layer p ^r ; function h represents the result after the input data passes through the prediction layer p ^d ;

Define ^Cj to represent the prototype of the jth class in the image input. For client i, the prototype of the jth class is expressed as That is, the vector average of the j-th type of data after passing through the representation layer is expressed as:

Where D _i,j represents the j-th category of data in the data set of client i;

In the model training phase, the _L2 distance is used to measure the difference between the model and the prototype in predicting the input x. The expression is:

y= _argminj ||g(x; ^pr ) ^-Cj || ₂ .

Preferably, step S5 comprises:

For the global prototype of class j, collect the local prototypes of class j from each client, and obtain the global prototype of class j by aggregating the local prototypes of different clients. The expression is:

in, represents the set of clients with data of type j, _Sj is the amount of data of type j;

At the same time, local models of different clients are collected and aggregated to obtain the global model, which is expressed as:

Where N is the total number of clients.

The personalized federated learning system of the hybrid multi-stage private model provided by the present invention includes:

Module M1: Download the global model and global prototype through the client, and initialize the local model, personalized model and local prototype;

Module M2: Use local data to train a local model through a global model and prototype regularization term, weightedly blend the local model with the historical local model, and use local data to train a personalized model.

Module M3: weighted mixing of the historical local model and the personalized model to generate a new personalized model, and then generate a local prototype through the personalized model;

Module M4: The client uploads the local model and local prototype to the server and waits for a new round of model parameters;

Module M5: The server aggregates the various models and prototypes to generate a global model and a global prototype, which are then distributed to various clients.

Preferably, the module M2 comprises:

The global class prototype representing the global data feature distribution is obtained by calculation. A regularization term is introduced during local model training to correct the deviation of local training. The expression is:

Among them, L ₂ represents the Euclidean distance; l _i represents the cross entropy loss function; Represents the global class prototype of the j-th type of data; represents the local class prototype of client i; w is the local model parameter; x is the input data; y is the true label of the input data x; C is the image category prototype representation; λ is a hyperparameter that controls the penalty weight of the regularization term;

The local model is then weightedly mixed with the historical local model to preserve the global information, expressed as:

Among them, history represents the historical local model after accumulating each round of local models wi _,t ; t represents the iteration round; β is a hyperparameter that controls the model mixing ratio; _wi,t represents the local model parameters of the tth round of the i-th client.

Preferably, the module M3 comprises:

In order to achieve personalization by mixing local information with global information, the historical local model and the personalized model are weighted and mixed to obtain a new round of personalized model, which is expressed as:

in, is a personalized model, which is used for local training to generate the next round of p _i,t+1 and local class prototypes; t is the number of iterations; μ is a hyperparameter that controls the model mixing ratio; the personalized model is trained with local data for one round to obtain p _i,t , and this model is mixed with the historical local model to obtain a new round of personalized model

Preferably, the model p is divided into a representation layer and a prediction layer, and the expression is:

p∶＝[ ^pr + ^pd ]

z＝g(x； ^pr )

y＝h(x；p ^d )

Among them, p ^r and p ^d represent the representation layer parameters and prediction layer parameters of the model respectively; z represents the output after the input x passes through the representation layer; y represents the output after the input x passes through the prediction layer; function g represents the result after the input data passes through the model representation layer p ^r ; function h represents the result after the input data passes through the prediction layer p ^d ;

Define ^Cj to represent the prototype of the jth class in the image input. For client i, the prototype of the jth class is expressed as That is, the vector average of the j-th type of data after passing through the representation layer is expressed as:

Where D _i,j represents the j-th category of data in the data set of client i;

In the model training phase, the _L2 distance is used to measure the difference between the model and the prototype in predicting the input x. The expression is:

y= _argminj ||g(x; ^pr ) ^-Cj || ₂ .

Preferably, the module M5 comprises:

For the global prototype of class j, collect the local prototypes of class j from each client, and obtain the global prototype of class j by aggregating the local prototypes of different clients. The expression is:

in, represents the set of clients with data of type j, _Sj is the amount of data of type j;

At the same time, local models of different clients are collected and aggregated to obtain the global model, which is expressed as:

Where N is the total number of clients.

Compared with the prior art, the present invention has the following beneficial effects:

(1) The present invention proposes a personalized federated learning framework based on a hybrid multi-stage private model of prototype learning. This framework not only greatly improves the performance of the model in an environment where data is not independent and identically distributed, but also provides a customized personalized model for each participating client. This model has strong adaptability to local data.

(2) Compared with the existing personalized federated learning algorithm, the present invention introduces a hybrid multi-stage private model method to achieve personalization, and adds prototype learning as a regular term to correct the offset of each client. This mechanism can effectively transfer local information and global information to each other without exchanging the original data;

(3) Compared with existing personalized federated learning algorithms, this method can achieve higher performance with fewer iterations. In addition, as a federated learning framework, this method can also be used in multiple fields, realizing distributed machine learning while protecting data privacy.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

Fig. 1 is a flow chart of the present invention;

FIG. 2 is a diagram showing the operation steps of the present invention.

Detailed ways

The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

Example 1

As shown in Figures 1 and 2, the embodiment of the present invention provides a personalized federated learning method of a hybrid multi-stage private model based on prototype learning, introduces prototype learning as a regular term, and implements personalized federated learning through hybrid model parameters. The specific process is as follows:

Step S1: download the global model and global prototype through the client, and initialize the local model, personalized model and local prototype;

Step S2: Using local data to train a local model through a global model and a prototype regularization term, weighting and mixing the local model with the historical local model, and using local data to train a personalized model;

Specifically, in step S2:

The global class prototype representing the global data feature distribution is obtained by calculation, and a regularization term is introduced during local model training to correct the deviation of local training:

Among them, L ₂ represents the Euclidean distance, _li represents the cross entropy loss function, Represents the global class prototype of the j-th type of data, Represents the local class prototype of client i.

The local model is then weightedly blended with the historical local model to preserve the global information:

Step S3: weighted mixing of the historical local model and the personalized model to generate a new personalized model, and then generating a local prototype through the personalized model;

Specifically, in step S3:

In order to achieve personalization by mixing local information with global information, we consider weighting and mixing the historical local model and the personalized model to obtain a new round of personalized model, which is expressed as:

in, are personalized model parameters, which will be used for local training to generate the next round of p _i,t+1 and local class prototypes, where t is the number of iterations.

The deep learning model can be divided into a representation layer and a prediction layer, and the expression is:

p∶＝[ ^pr + ^pd ]

z＝g(x； ^pr )

y＝h(x；p ^d )

Where p ^r and p ^d represent the representation layer parameters and prediction layer parameters of the model respectively, z represents the output after the input x passes through the representation layer, and y represents the output after the input x passes through the prediction layer. Define C ^j to represent the prototype of the jth class in the image input. Then for a specific client i, its prototype of the jth class can be expressed as That is, the vector average of the j-th type of data after passing through the representation layer, that is:

Among them, Di _,j represents the j-th category of data in the dataset of client i. With the above definition, the _L2 distance can be used to measure the difference between the model and the prototype for the prediction of input x during the model training phase. The expression is:

y＝argmin _j || g(x； ^pr ) ^-Cj || ₂

Step S4: The client uploads the local model and the local prototype to the server and waits for a new round of model parameters;

Step S5: the server aggregates the various models and prototypes to generate a global model and a global prototype, and then distributes them to various clients;

Specifically, in step S5:

For the global prototype of class j, collect the local prototypes of class j from each client, and obtain the global prototype of class j by aggregating the local prototypes of different clients. The expression is:

in, It represents the set of clients with data of type j, and _Sj is the amount of data of j.

At the same time, the local models of different clients are collected and aggregated to obtain the global model, which is expressed as:

Example 2

The present invention also provides a personalized federated learning system for a hybrid multi-stage private model. The personalized federated learning system for the hybrid multi-stage private model can be implemented by executing the process steps of the personalized federated learning method for the hybrid multi-stage private model, that is, those skilled in the art can understand the personalized federated learning method for the hybrid multi-stage private model as a preferred implementation of the personalized federated learning system for the hybrid multi-stage private model.

The personalized federated learning system of the hybrid multi-stage private model provided by the present invention includes: module M1: downloading the global model and the global prototype through the client, and initializing the local model, the personalized model and the local prototype; module M2: using local data to train the local model through the global model and the prototype regularization term, weighted mixing the local model with the historical local model, and training the personalized model with the local data; module M3: weighted mixing the historical local model and the personalized model to generate a new personalized model, and then generating the local prototype through the personalized model; module M4: the client uploads the local model and the local prototype to the server, and waits for a new round of model parameters; module M5: the server aggregates the various models and prototypes to generate the global model and the global prototype, and then distributes them to each client.

The module M2 comprises:

The global class prototype representing the global data feature distribution is obtained by calculation. A regularization term is introduced during local model training to correct the deviation of local training. The expression is:

Among them, L ₂ represents the Euclidean distance; l _i represents the cross entropy loss function; Represents the global class prototype of the j-th type of data; represents the local class prototype of client i; w is the local model parameter; x is the input data; y is the true label of the input data x; C is the image category prototype representation; λ is a hyperparameter that controls the penalty weight of the regularization term;

The local model is then weightedly mixed with the historical local model to preserve the global information, expressed as:

Among them, history represents the historical local model after accumulating each round of local models wi _,t ; t represents the iteration round; β is a hyperparameter that controls the model mixing ratio; _wi,t represents the local model parameters of the tth round of the i-th client.

The module M3 comprises:

In order to achieve personalization by mixing local information with global information, the historical local model and the personalized model are weighted and mixed to obtain a new round of personalized model, which is expressed as:

in, is a personalized model, which is used for local training to generate the next round of p _i,t+1 and local class prototypes; t is the number of iterations; μ is a hyperparameter that controls the model mixing ratio; the personalized model is trained with local data for one round to obtain p _i,t , and this model is mixed with the historical local model to obtain a new round of personalized model

The model p is divided into a representation layer and a prediction layer, and the expression is:

p∶＝[ ^pr + ^pd ]

z＝g(x； ^pr )

y＝h(x；p ^d )

Among them, p ^r and p ^d represent the representation layer parameters and prediction layer parameters of the model respectively; z represents the output after the input x passes through the representation layer; y represents the output after the input x passes through the prediction layer; function g represents the result after the input data passes through the model representation layer p ^r ; function h represents the result after the input data passes through the prediction layer p ^d ;

Define ^Cj to represent the prototype of the jth class in the image input. For client i, the prototype of the jth class is expressed as That is, the vector average of the j-th type of data after passing through the representation layer is expressed as:

Where D _i,j represents the j-th category of data in the data set of client i;

In the model training phase, the _L2 distance is used to measure the difference between the model and the prototype in predicting the input x. The expression is:

y= _argminj ||g(x; ^pr ) ^-Cj || ₂ .

The module M5 comprises:

For the global prototype of class j, collect the local prototypes of class j from each client, and obtain the global prototype of class j by aggregating the local prototypes of different clients. The expression is:

in, represents the set of clients with data of type j, _Sj is the amount of data of type j;

At the same time, local models of different clients are collected and aggregated to obtain the global model, which is expressed as:

Where N is the total number of clients.

Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims (6)

1. A personalized federal learning method of a hybrid multi-stage private model, comprising:

Step S1: downloading a global model and a global prototype through a client, and initializing a local model, a personalized model and the local prototype;

Step S2: training a local model by using local data through a global model and a prototype regular term, carrying out weighted mixing on the local model and a historical local model, and training a personalized model by using the local data;

Step S3: the historical local model and the personalized model are weighted and mixed to generate a new personalized model, and then a local prototype is generated through the personalized model;

step S4: uploading the local model and the local prototype to a server by the client, and waiting for a new round of model parameters;

step S5: the server aggregates each model and prototype to generate a global model and a global prototype, and then distributes the global model and the global prototype to each client;

The step S2 includes:

The global model representing the global data feature distribution is obtained through calculation, a regular term is introduced during local model training, the offset of local training is corrected, and the expression is as follows:

Wherein L ₂ represents the euclidean distance; l _i denotes the cross entropy loss function; A global class prototype representing class j data; Representing a local class prototype of the client i; w is a local model parameter; x is input data; y is the real label of the input data x; c is the image class prototype representation; lambda is a super parameter, and the punishment weight of the regular term is controlled;

the local model is then weighted mixed with the historical local model to preserve global information expressed as:

Wherein, history represents the history local model after accumulating each round of local model w _i,t; t represents the iteration round; beta is a super parameter, and the mixing proportion of the model is controlled; w _i,t denotes the local model parameters of the ith client's nth round;

the step S3 includes:

In order to realize individuation by mixing local information and global information, a historical local model and individuation model are weighted and mixed to obtain a new round of individuation model, and the expression is as follows:

Wherein, Generating a p _i,t+1 and a local prototype of the next round for the personalized model for local training; t is the iteration round; mu is a super parameter, and the mixing proportion of the model is controlled; training the personalized model by using local data for one round to obtain p _i,t, and mixing the model with the historical local model to obtain a new round of personalized model

2. The personalized federal learning method of hybrid multi-phase private models according to claim 1, wherein the model p is divided into a representation layer and a prediction layer, expressed as:

p∶＝[p^r+p^d]

z＝g(x；p^r)

y＝h(x；p^d)

Wherein p ^r and p ^d represent the representation layer parameters and the prediction layer parameters of the model, respectively; z represents the output after the presentation layer for input x; y represents the output after passing through the prediction layer for input x; the function g represents the result of the input data after passing through the model representation layer p ^r; the function h represents the result of the input data after passing through the prediction layer p ^d;

Defining C ^j to represent a prototype of the j-th class in the image input, then for client i, its prototype of the j-th class is represented as I.e., vector average of j-th class data after passing through the presentation layer, the expression is:

wherein D _i,j represents the j-th class of data in the dataset of client i;

In the model training stage, the prediction difference of the model and the prototype for the input x is measured by using the L ₂ distance, and the expression is:

y＝argmin_j||g(x;p^r)-C^j||₂。

3. the personalized federal learning method of hybrid multi-phase private models according to claim 2, wherein step S5 comprises:

for the global prototype of the class j, collecting local prototypes of the class j from each client, and obtaining the global prototype of the class j by aggregating the local prototypes of different clients The expression is:

Wherein, Representing a client set related to j-class data, S _j being the data quantity of j;

meanwhile, collecting local models of different clients, and obtaining a global model by aggregation, wherein the expression is as follows:

Wherein N is the total number of clients.

4. A personalized federal learning system for a hybrid multi-stage private model, comprising:

Module M1: downloading a global model and a global prototype through a client, and initializing a local model, a personalized model and the local prototype;

module M2: training a local model by using local data through a global model and a prototype regular term, carrying out weighted mixing on the local model and a historical local model, and training a personalized model by using the local data;

Module M3: the historical local model and the personalized model are weighted and mixed to generate a new personalized model, and then a local prototype is generated through the personalized model;

module M4: uploading the local model and the local prototype to a server by the client, and waiting for a new round of model parameters;

Module M5: the server aggregates each model and prototype to generate a global model and a global prototype, and then distributes the global model and the global prototype to each client;

The module M2 includes:

The global model representing the global data feature distribution is obtained through calculation, a regular term is introduced during local model training, the offset of local training is corrected, and the expression is as follows:

Wherein L ₂ represents the euclidean distance; l _i denotes the cross entropy loss function; A global class prototype representing class j data; Representing a local class prototype of the client i; w is a local model parameter; x is input data; y is the real label of the input data x; c is the image class prototype representation; lambda is a super parameter, and the punishment weight of the regular term is controlled;

the local model is then weighted mixed with the historical local model to preserve global information expressed as:

Wherein, history represents the history local model after accumulating each round of local model w _i,t; t represents the iteration round; beta is a super parameter, and the mixing proportion of the model is controlled; w _i,t denotes the local model parameters of the ith client's nth round;

the module M3 includes:

In order to realize individuation by mixing local information and global information, a historical local model and individuation model are weighted and mixed to obtain a new round of individuation model, and the expression is as follows:

Wherein, Generating a p _i,t+1 and a local prototype of the next round for the personalized model for local training; t is the iteration round; mu is a super parameter, and the mixing proportion of the model is controlled; training the personalized model by using local data for one round to obtain p _i,t, and mixing the model with the historical local model to obtain a new round of personalized model

5. The personalized federal learning system for hybrid multi-phase private models according to claim 4, wherein the model p is divided into a representation layer and a prediction layer, expressed as:

p∶＝[p^r+p^d]

z＝g(x；p^r)

y＝h(x；p^d)

Wherein p ^r and p ^d represent the representation layer parameters and the prediction layer parameters of the model, respectively; z represents the output after the presentation layer for input x; y represents the output after passing through the prediction layer for input x; the function g represents the result of the input data after passing through the model representation layer p ^r; the function h represents the result of the input data after passing through the prediction layer p ^d;

Defining C ^j to represent a prototype of the j-th class in the image input, then for client i, its prototype of the j-th class is represented as I.e., vector average of j-th class data after passing through the presentation layer, the expression is:

wherein D _i,j represents the j-th class of data in the dataset of client i;

In the model training stage, the prediction difference of the model and the prototype for the input x is measured by using the L ₂ distance, and the expression is:

y＝argmin_j||g(x;p^r)-C^j||₂。

6. the personalized federal learning system for hybrid multi-phase private models according to claim 5, wherein the module M5 comprises:

for the global prototype of the class j, collecting local prototypes of the class j from each client, and obtaining the global prototype of the class j by aggregating the local prototypes of different clients The expression is:

Wherein, Representing a client set related to j-class data, S _j being the data quantity of j;

meanwhile, collecting local models of different clients, and obtaining a global model by aggregation, wherein the expression is as follows:

Wherein N is the total number of clients.