2.1.

Horizontal federation

Horizontal federation (also referred to as Horizontal FL in Figure 1) focuses on overlapping data features, that is, the data features of different participants are aligned but their samples are different. Let’s take an example of two banks, the banks’ data features are overlapping but their users rarely overlap. Horizontal federation is also called sample-division federation. The rows in Figure 1 represent samples’ ID and the columns represent data features. So it is data-feature-alignment-based federation.

Figure 1.

Horizontal federation.

2.2.

Vertical Federation

Vertical federation (also referred to as Vertical FL in Figure 2) focuses on overlapping samples, that is, the samples of different participants are aligned but their data features are different. Let’s take an example of a bank and an e-commerce platform, their user samples are overlapping but their transactions are different. Under the circumstances, the bank and the e-supermarket can offer a more specific portrait of consumers together, promoting their products to them according to their spending ability. So it is also called feature-division-based federation.

Figure 2.

Vertical federation.

2.3.

Federal migration learning

Federal Migration Learning focuses on federated problems of heterogeneous data, where the data samples and data features rarely overlap (as shown in Figure 3). Related research has made slow progress and will become a hot topic in the future.

Figure 3.

Federated migration.

4.1.

Model overview

There are two main roles of FedAvg algorithm in the horizontal federated learning implemented in this paper: client and server. The main function of the server is to aggregate the local models uploaded by the selected clients⁷. The main function of the client is to receive the instructions sent by the server and the global model, and use the local data to train the local model. The structure is shown in Figure 4 as follows:

Figure 4.

Client and server.

Furthermore, CIFAR-10 image data set is classified by means of horizontal federation, and ResNet-18 model is adopted.

4.2.

Server

The server functions to aggregate the local models uploaded by the selected clients in horizontal federation.

When it comes to a full-featured federation framework, the server involves much more complicated functions, for example, network monitoring of every client node, sending reconnecting signals to failure nodes and so on and so forth.

Our experiment is simulated locally, not involving network communication details and troubleshooting, so information about such functional details is not discussed, but only the model aggregation function is mentioned^8-9.

First, Server, a server class is defined, where the main functions in the class are included as following.

4.2.2.

Definition of model aggregation function.

As we mentioned earlier, the main function of the server is to aggregate the models. Therefore, after defining the constructor, we need to define the model aggregation function in the class, and update the global model by receiving the model uploaded by the client and using FedAvg algorithm.

where, G^t represents the global model after the Tth round of aggregation, represents the model after the local update of the i th client in the t + 1 th round, and G^t + 1 represents the global model after the t + 1 th round of aggregation.

4.3.

Client

The client mainly functions to receive the instructions sent by the server and the global model, and use the local data to train the local model in horizontal federation. Similarly to the server mentioned above, when it comes to a full-featured federation framework, the client involves many complicated functions, for example, the necessity to consider whether the local resources (CPU, memory, etc.) meet the training needs, the current network interruption and training due to external factors^10-12, etc.

Here, we only emphasize the local model training details of the client. First, Client, a client class is defined, where the two main functions in the class are included as following.

(1) Definition of the constructor

In the constructor, the responsibilities of the client include:

First, to copy the configuration information to the client;

Then, to get the model according to the model information as configurated. Usually the model parameters are delivered to the client from the server and the client overwrites the local model with the global model;

Finally, to configure the local training data. In this case, we get the CIFAR-10 data set through the datasets module of torchvision and then divide it according to the client ID. Different clients have different sub-data sets, and there is no intersection between them.

(2) Definition of the local training function of the model

Loss function is selected after the selection of classification model. Here we use Softmax crossover

Entropy loss function, which is in the form of:

where, n is the sample size on device j, y_j,i is the true label of j th sample of device i th, and ŷ_j,i is the prediction label of j th sample of device i th, which is actually related to the parameter w of classification model. k is the number of categories, or the dimension of label vector. Next it is necessary to aggregate and average the loss function on each device in federated learning.

4.4.

Aggregation

When the configuration file, server class and client class are all defined, we combine this information together. First, read the profile information. Next, we will define a server object and several client objects respectively to simulate the horizontal federation training scene. At every iteration, the server randomly selects some clients to participate in iterative training. The selected clients call the local training interface “local_train” for local training. Finally, the server calls the model aggregation function “model aggregate” to update the global model. The codes are shown in Figure 5 as follows:

Figure 5.

Partial aggregation code.

4.5.

Comparison of the effects of federated learning and centralized training of Resnet 18 on CIFAR-10

Training results are shown in Figure 6.

Figure 6.

Comparison of effects between federated learning and centralized training.

Federal training configuration: There are 10 client devices (no_models=10), 5 of which are randomly selected to participate in training (k=5) in each round, and the number of local training iterations is 3 (local_epochs=3) and the number of global iterations is 20 (global_epochs=20).

Centralized training configuration: We don’t need to write centralized training code separately, and only have to modify the federated learning configuration to make it equivalent to centralized training. Specifically, we can set the client device no_models and the number of training devices k selected in each round to 1. This kind of federal training with only one device is equivalent to centralized training. Other parameter configuration information is consistent with federated learning and training.

Training results are shown in Figure 7.

Figure 7.

Comparison between single point training and federated training.

The single-point training in the figure is the result of model training using local data under a certain client.

We can see that the model effect of single-point training (blue bar) is obviously lower than that of federal training (green bar and red bar), which shows that the global distribution characteristics of data cannot be well learned only through the data of a single client, and the generalization ability of the model is poor.

In addition, for the different number of clients (k value) participating in each round of federal training, its performance will be different. The larger the k value, the more clients participating in each round of training, the better its performance will be, but the completion time of each round will be relatively longer.