DE102021108101A1 - System and method for federal learning using anonymized weighting factorization - Google Patents

Abstract

A federal machine learning system includes a global server and client devices. The server receives updates of weighting factor dictionaries and factor strength vectors from the clients and generates a globally updated weighting factor dictionary and a globally updated factor strength vector. A client device selects a set of parameters from a global set of parameters and trains a model using a data set of the client device and the set of selected parameters. The client device sends a client-updated weighting factor dictionary and a client-updated factor strength vector to the server. The client device receives the globally updated weighting factor dictionary and the globally updated factor strength vector and retrains the model using the data set of the client device, the set of parameters selected by the client device and the globally updated weighting factor dictionary and global updated factor strength vector.

Description

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority under 35 USC § 119 (e) of US Provisional Application No. 63 / 033,747 , filed on June 2, 2020, the disclosure of which is incorporated by reference in its entirety.

TECHNICAL AREA

The subject matter disclosed herein relates to federated machine learning. In particular, the subject matter disclosed herein relates to a system and method for federal machine learning.

BACKGROUND

The development of the Internet of Things (IoT), the widespread use of smartphones and the digitization of documents have all contributed to modern systems that generate ever larger quantities of data. The data that is generated can provide extensive information about people, which on the one hand can lead to highly personalized intelligent applications, but on the other hand can also be confidential and should therefore remain private or confidential. Examples of such private information include, but are not limited to, images of faces, typing history, medical records, and survey responses.

SHORT VERSION

An exemplary embodiment provides a client device in a federal machine learning system that may include at least a computing device, a communication interface, and a processor. The processor can be coupled to the at least one computing device and to the communication interface. The processor can select a set of parameters for the client device from a global set of parameters, train a model using a data set of the client device and the set of parameters selected by the client device, the data set starting from an output of the at least one computing device, update a weighting factor dictionary and a factor strength vector after training the model, send a client-updated weighting factor dictionary and a client-updated factor strength vector through the communication interface to a global server, a globally updated weighting factor dictionary and a globally updated factor strength vector received through the communication interface from the global server and the model using the data set of the client device, the group of parameters selected by the client device and retrain the globally updated weighting factor dictionary and globally updated factor strength vector. In one embodiment, the client device can be part of a group of N client devices, where N is an integer. In another embodiment, the processor selects the group of parameters from the global group of parameters using three variation parameters, which may include seed values, and minimizes a difference between supervised learning of the data set and regularization of the selected group of parameters and the global group of parameters. The processor can select the set of parameters from the global set of parameters by receiving the global set of parameters sent from the global server to a first subset of client devices of the N client devices, the client device Part of the first subset of client devices. The client device can receive the globally updated weighting factor dictionary and a globally updated factor strength vector by receiving the globally updated weighting factor dictionary and a globally updated factor strength vector sent by the global server to a second subset of the N client devices, the client devices Device may be part of the second subset of client devices. In yet another embodiment, the processor can send a request through the communication interface to the global server for a current version of the global group of parameters, can update the model using the current version of the global group of parameters, and can update the model using the current version of the global set of parameters has been updated to inference based on the client device's data set.

An exemplary embodiment provides a federal machine learning system that may include a global server and N client devices. The global server can receive updates to weighting factor dictionaries and factor strength vectors from N client devices, where N is an integer, and can generate a globally updated weighting factor dictionary and a globally updated factor strength vector. At least one client device may include at least one computing device, a communication interface, and a processor. The processor can be coupled to the at least one computing device and to the communication interface. The processor can select a set of parameters from a global set of parameters, train a model using a data set of the client device and the set of parameters selected by the client device, a weighting factor dictionary and a factor strength vector after training of the model, send a client-updated weighting factor dictionary and a client-updated factor strength vector through the communication interface, receive the globally updated weight factor dictionary and the globally updated factor strength vector from the global server through the communication interface, and the model using the data set of the client device, the Retrain the group of parameters selected by the client device and the globally updated weighting factor dictionary and globally updated factor strength vector. In one embodiment, the processor can select the group of parameters from the global group of parameters using three variation parameters, which may include seed values, and minimizes a difference between supervised learning of the data set and regularization of the selected group of parameters and the global group of parameters. In another embodiment, the processor may select the set of parameters from the global set of parameters by receiving the global set of parameters sent from the global server to a first subset of client devices of the N client devices where the client device may be part of the first subset of client devices. In another device, the client device may receive the globally updated weighting factor dictionary and a globally updated factor strength vector by receiving the globally updated weighting factor dictionary and a globally updated factor strength vector sent by the global server to a second subset of the N client devices, wherein the client device can be part of the second subset of client devices. In one embodiment, the processor can send a request through the communication interface to the global server for a current version of the global group of parameters, can update the model using the current version of the global group of parameters, and can update the model using the current Version of the global set of parameters has been updated, evaluate to inference based on the client device data set.

An exemplary embodiment provides a method for federal machine learning that may include: selecting, at a client device, a set of parameters from a global set of parameters, the global set of parameters including a weighting factor dictionary and a factor strength vector; Training, at the client device, a model using a data set of the client device and the set of parameters selected by the client device; Updating a weighting factor dictionary and a factor strength vector after training the model; Sending, from the client device to a global server, a client-updated weighting factor dictionary and a client-updated factor strength vector; Receiving at the client device an updated weighting factor dictionary and a globally updated factor strength vector from the global server; and retraining the model on the client device using the data set of the client device, the set of parameters selected by the client device, and the globally updated weighting factor dictionary and the globally updated factor strength vector. In one embodiment, the client device can be part of a group of N client devices, where N is an integer. In another embodiment, selecting the group of parameters from the global group of parameters may include selecting the group of parameters using three variation parameters having seed values; and to minimize a difference between supervised learning of the data set and regularization of the selected group of parameters and the global group of parameters. In yet another embodiment, selecting the set of parameters from the global set of parameters may include the global set of parameters sent from the global server to a first subset of client devices of the N client devices, at the client Device, wherein the client device may be part of the first subset of client devices. In yet another embodiment, receiving the globally updated weighting factor dictionary and a globally updated factor strength vector from the global server at the client device may include receiving at the client device the globally updated weighting factor dictionary and a globally updated factor strength vector generated by the global server to a second subset of the N client devices, which client device may be part of the second subset of client devices. In one embodiment, the method may further include the client device requesting a current version of the global set of parameters from the global server; receiving the current version of the global group of parameters; updating the model using the current version of the global set of parameters; and evaluating the model updated using the current version of the global set of parameters to inference based on the client device data set.

Figure list

• 1 ein funktionales Blockschaltbild für eine beispielhafte Ausführungsform eines Systems für föderales Lernen gemäß des hierin offenbarten Gegenstands darstellt;
• 2A und 2B jeweils funktionale Blockschaltbilder für beispielhafte Ausführungsformen eines globalen Servers und eines Clients gemäß des hierin offenbarten Gegenstands darstellen;
• 3 ein Flussdiagramm für eine beispielhafte Ausführungsform eines Verfahrens für föderales maschinelles Lernen bei einer Client-Vorrichtung gemäß dem hierin offenbarten Gegenstand ist; und
• 4 eine elektronische Vorrichtung darstellt, die eine Funktionalität für föderales maschinelles Lernen gemäß dem hierin offenbarten Gegenstand umfasst.

In the following section, the aspects of the subject matter disclosed herein are described with reference to exemplary embodiments illustrated in the drawings, in which:

• 1 Figure 12 is a functional block diagram for an exemplary embodiment of a federal learning system in accordance with the subject matter disclosed herein;
• 2A and 2 B each depict functional block diagrams for exemplary embodiments of a global server and a client in accordance with the subject matter disclosed herein;
• 3 Figure 10 is a flow diagram for an exemplary embodiment of a method for federal machine learning at a client device in accordance with the subject matter disclosed herein; and
• 4th FIG. 11 illustrates an electronic device that includes federal machine learning functionality in accordance with the subject matter disclosed herein.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the disclosure. However, it will be understood by those skilled in the art that the aspects disclosed can be put into practice without these specific details. In other instances, known methods, procedures, components, and circuits have not been described in detail in order not to obscure the subject matter disclosed herein.

Reference in this specification to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment disclosed herein. Thus, the appearances of the phrase “in an embodiment,” or “in an embodiment,” or “according to an embodiment” (or other terms that have a similar meaning) in different places in this specification need not necessarily refer to the same embodiment. Furthermore, the particular features, structures, or properties can be combined in any suitable manner in one or more embodiments. In this regard, the word "exemplary" as used herein means that it "serves as an example, case study or illustration". Any embodiment described herein as "exemplary" is not to be interpreted as necessarily preferred or advantageous over other embodiments. In addition, the particular features, structures, or properties can be combined in any suitable manner in one or more embodiments. Also, depending on the context of a discussion herein, a singular term may include the corresponding plural forms and a plural term may include the corresponding singular form. Likewise, a term with a hyphen (e.g. “two-dimensional”, “pixel-specific” etc.) can be used partially interchangeably with a corresponding version without a hyphen (e.g. “two-dimensional”, “given”, “pixel-specific” ”Etc.) and a capitalized entry (e.g.“ Counter Clock ”,“ Row Select ”,“ PIXOUT ”etc.) can be used interchangeably with a non-capitalized version (e.g.“ counter clock ”,“ row select ”,“ pixout ”etc.). Such occasional interchangeable uses are not intended to be construed as inconsistent.

Also, depending on the context of a discussion herein, a singular term may include the corresponding plural forms and a plural term may include the corresponding singular form. It is further noted that various figures (including component diagrams) shown and discussed herein are for illustration purposes only and are not drawn to scale. Also, various waveforms and timing diagrams are shown for illustration purposes only. For example, the dimensions of some of the elements may be exaggerated in comparison to other elements for the sake of clarity. Furthermore, where appropriate, reference symbols have been repeated in the figures in order to identify corresponding and / or analogous elements.

The terminology used herein is for the purpose of describing some exemplary embodiments only and is not intended to limit the claimed subject matter. As used herein, the singular forms "a,""an,""an," and "der, die, that" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is further understood that the terms “comprising” and / or “comprising” when used in this specification indicate the presence of given features, integers, steps, operations, elements, and / or components, but not the presence or exclude the addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The terms "first,""second," etc., as used herein, are used to identify nouns they precede and are not intended to imply any order (e.g., spatial, temporal, logical, etc.) unless they are explicit defined as such. Furthermore, the same reference symbols can be used in two or more figures to denote parts, components, blocks, circuits, units or modules which have the same or a similar functionality. However, such a use is only for the purpose of illustration and simple discussion; it does not imply that the construction or engineering details of such components or units are the same in all embodiments or that such commonly referred to as parts / modules are the only way to implement some of the exemplary embodiments disclosed herein.

It will be understood that when an element or layer is referred to as being “on,” “connected to,” or “coupled to” another element or layer, that element or layer is directly on, connected, or coupled to the other element or the other layer, or there may be intermediate elements or layers. In contrast, when an element is referred to as being “directly on,” “directly connected to,” or “directly coupled to” another element or layer, there are no intermediate elements or layers. The same reference numbers refer to the same elements throughout. In the context of the present text, the term “and / or” includes any combination of one or more of the associated bullet points listed.

The terms "first," "second," etc., as used herein, are used to identify nouns they precede and are not intended to imply any order (e.g., spatial, temporal, logical, etc.) unless they are explicit defined as such. Furthermore, the same reference symbols can be used in two or more figures to denote parts, components, blocks, circuits, units or modules which have the same or a similar functionality. However, such a use is only for the purpose of illustration and simple discussion; it does not imply that the construction or engineering details of such components or units are the same in all embodiments or that such commonly referred to as parts / modules are the only way to implement some of the exemplary embodiments disclosed herein.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter belongs. It is also to be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technical field, and not in an idealized or excessively formal sense unless specifically defined herein.

As used herein, the term “module” refers to any combination of software, firmware, and / or hardware configured to provide the functionality described herein in conjunction with a module. For example, software may be embodied as a software package, code, and / or instruction set or instructions, and the term "hardware" as used in an implementation described herein may, for example, individually or in any combination, include an arrangement, a hard-wired circuitry, a programmable circuitry, a state machine circuit and / or firmware on which instructions are stored that can be executed by programmable circuitry. The modules may be embodied together or individually as circuitry that forms part of a larger system, for example, but not limited to, an integrated circuit (IC), on-chip system (SoC), array, etc.

Federal learning has been proposed to provide machine learning, which may keep personalized data private by keeping user data local on each client device and only sharing model updates with a global server. So puts federal Learning represents one possible strategy for training machine learning models on heterogeneous, distributed networks in a manner that preserves privacy.

While a federal machine learning paradigm can provide a way to keep private data private, federal machine learning systems continue to face a number of challenges. For example, a currently used federal machine learning system includes a single global model that is used by each client. However, the one-model approach may not work particularly well with particular subpopulations, as data distribution can be skewed among different clients.

wobei i ein Index für Client-Vorrichtungen ist, N die Anzahl an Clients ist, F_i(θ) eine lokale Objektive Function ist und p_i ≥ 0 die Gewichtung jeder Vorrichtung i ist.To illustrate, imagine N client devices and that the i-th client device _{includes a data distribution D i} that differs from other client devices depending on i. In a traditional federal machine learning environment, a single global model that can be learned can be applied to all N client devices. The traditional or customary approach assumes a multilayer perceptron (MLP) architecture comprising layers l = 1, ..., L and a set of weights θ = {W ^l } _{l = 1: L} , which with shared among all client devices. In order to meet a global objective, a set of weights θ can be learned to minimize or minimize a loss in resource for all clients. For example, a traditional federal machine learning system minimizes the following objective: $$\underset{\theta }{min}ƒ\left(\theta \right)={\displaystyle \sum {}_{i=1}^N{p}_i{\mathrm{F.}}_i\left(\theta \right)}={\mathbb{E.}}_i\left[{\mathrm{F.}}_i\left(\theta \right)\right]$$

where i is an index for client devices, N is the number of clients, F _i (θ) is a local objective function, and p _i ≥ 0 is the weight of each device i.

If there is statistical heterogeneity, however, a universal approach can lead to a global model that delivers poor performance for certain clients. Often times, performance can reflect how closely the local distribution of a given client matches the distribution of the entire population. As a result, the model of this exemplary traditional federal machine learning system can be viewed as less fair to clients who have data characteristics that are less common to the clients.

The subject matter disclosed herein can improve model consistency for federal learning by using Bayesian nonparametric weighting factorization, which can provide a personalized federal learning solution that can achieve higher local model performance on multiple clients.

The federal machine learning system disclosed herein includes at least three improved features compared to traditional federal learning systems. A first improved feature is that the network in which federal learning takes place is divided into two parts. The first part creates server aggregation and the second part is used for client personalization. A second improved feature relates to a reduced amount of data communicated between a global server and client devices. That is, data communication between the global server and the client devices is more efficient because kernel factorization is used in the client devices and only a subset of the parameters used for training are communicated. A third improved feature relates to an additional layer of security provided by kernel factorization and that only a subset of the parameters used for training are communicated.

The federal machine learning system disclosed herein provides a federal learning system that efficiently uses data in a global model to train neural networks factored in N local models. Each client model can be personalized based on a local distribution at the client, with all client models sharing learned components together.

1 FIG. 10 provides a functional block diagram for an exemplary embodiment of a system 100 for federal learning in accordance with the subject matter disclosed herein. The system 100 for federal learning can be a global server 101 and N clients (ie, local devices) 102 ₁ -102 _N include. The global server 101 can be in a single location in the cloud or in distributed locations. The term "global server" as used herein means any server device that is configured to (wired or) with two or more client devices over a wide area network (e.g., the Internet) wireless) and can be any server device configured to communicate directly with two or more client devices in a federal machine learning system. The clients 102 ₁ -102 _N are via a communication link 103 able to communicate with the global server 101 coupled. The communication link 103 can be a wired communication link and / or a wireless communication link.

2A and 2 B each provide functional block diagrams for exemplary embodiments of a global server 101 and a client 102 in accordance with the subject matter disclosed herein. A global server 101 can be a processing device 201 include, such as a central processing unit (CPU) capable of communication with a memory 202 and a communication interface 203 connected is. The memory 202 can comprise a non-volatile and / or volatile memory. The communication interface 203 can be set up to communicate with a network structure such as, but not limited to, the Internet. The communication interface 203 can be a wired and / or a wireless communication interface. Other configurations for the global server 101 are possible. The global server 101 may be configured to provide federal machine learning functionality as described herein. In one embodiment, the federal machine learning functionality provided by the global server 101 provided by one or more modules, which may be any combination of software, firmware, and / or hardware configured to provide the functionality described herein.

One client 102 can be a processing device 251 include, such as a CPU, capable of communicating with a memory 252 , a communication interface 253 and one or more computing devices 254 connected is. The one or more computing devices 254 may include an ability to capture or gather information indicative of movement, one or more images, biometrics and / or disease of a human and / or non-human animal and / or plant, sound, voice, location, Includes, but is not limited to, metadata, application usage (i.e., browsing history), and / or survey responses. In one embodiment, there is at least one computing device 254 a detection device. Other configurations for a client device 102 are possible. One client 102 may be configured to provide federal machine learning functionality as described herein. In one embodiment, the federal machine learning functionality provided by a client 102 provided by one or more modules, which may be any combination of software, firmware, and / or hardware configured to provide the functionality described herein.

für L Schichten aufweist, die basierend auf einer Datenverteilung

trainiert sein können. Jeder Satz an Gewichtungen θ_i kann für die Datenverteilung

jedes Clients i maximal spezifisch sein. Jeder Client hat allerdings typischerweise beschränkte Daten, die für das Training eines vollständigen Modells ohne Überanpassung (Overfitting) unzureichend sein können. Daher skaliert die Gesamtanzahl an Parametern, die auf allen Clients gelernt werden müssen, mit der Anzahl an Clients. Das Lernen von N separaten Modellen kann allerdings nicht von Ähnlichkeiten zwischen Client-Datenverteilungen oder der geteilten Lernaufgabe profitieren. Um Daten effizienter zu nutzen schafft das System 100 für föderales maschinelles Lernen ein Gleichgewicht zwischen einem einzelnen globalen Modell und N lokalen Modellen. Das heißt, jedes Client-Modell kann für die lokale Datenverteilung personalisiert werden, wobei alle Modelle gemeinsam gelernte Komponenten teilen. Dazu wird eine Gewichtungsmatrix ${\theta }_i={\left\{W^{\mathcal{l}}\right\}}_{\mathcal{l}=1}^L$

für einen Client i faktorisiert als: $$W_i^{\mathcal{l}}=W_a^{\mathcal{l}}{\mathrm{\Lambda }}_i^{\mathcal{l}}{W}_b^{\mathcal{l}},\mathcal{l}=\mathrm{1,}\dots ,L$$

$${\mathrm{\Lambda }}_i^{\mathcal{l}}=diag\left({\lambda }_i^{\mathcal{l}}\right)$$

wobei $W_a^{\mathcal{l}}$

und $W_b^{\mathcal{l}}$

Wörterbücher von Rang-1 an Gewichtungsfaktoren sind, die mit Clients geteilt werden können und ${\mathrm{\Lambda }}_i^{\mathcal{l}}$

die diagonale personalisierte Matrix für jeden Client i ist.One client 102i can have a local model that has a weighting matrix ${\theta }_i={\left({\mathrm{W.}}_i^{\mathcal{l}}\right)}_{\mathcal{l}=1}^{\mathrm{L.}}$

for L has layers based on a data distribution

can be trained. Any set of weights θ _i can be used for data distribution

each client i be maximally specific. However, each client typically has limited data that may be insufficient to train a full model without overfitting. Therefore, the total number of parameters that have to be learned on all clients scales with the number of clients. However, learning from N separate models cannot benefit from similarities between client data distributions or the shared learning task. In order to use data more efficiently, the system creates 100 for federal machine learning, a balance between a single global model and N local models. This means that each client model can be personalized for local data distribution, with all models sharing components that have been learned together. A weighting matrix is used for this ${\theta }_i={\left\{{\mathrm{W.}}^{\mathcal{l}}\right\}}_{\mathcal{l}=1}^{\mathrm{L.}}$

for a client i factored as: $${\mathrm{W.}}_i^{\mathcal{l}}={\mathrm{W.}}_a^{\mathcal{l}}{\mathrm{\Lambda }}_i^{\mathcal{l}}{\mathrm{W.}}_b^{\mathcal{l}},\mathcal{l}=\mathrm{1,}...,\mathrm{L.}$$

$${\mathrm{\Lambda }}_i^{\mathcal{l}}=diaG\left({\lambda }_i^{\mathcal{l}}\right)$$

whereby ${\mathrm{W.}}_a^{\mathcal{l}}$

and ${\mathrm{W.}}_b^{\mathcal{l}}$

Rank 1 dictionaries are weighting factors that can be shared with clients and ${\mathrm{\Lambda }}_i^{\mathcal{l}}$

the diagonal personalized matrix for each client is i.

wobei $w_{a,k}^{\mathcal{l}}$

die k^-te Spalte von $W_a^{\mathcal{l}}$

ist, $w_{b,k}^{\mathcal{l}}$

die k-te Zeile von $W_b^{\mathcal{l}}$

ist und ⊗ ein äußeres Produkt bezeichnet. Auf diese Weise geschrieben wird die Interpretation der entsprechenden Paare an Spalten und Zeilen $w_{a,k}^{\mathcal{l}}$

und $w_{b,k}^{\mathcal{l}}$

als Gewichtungsfaktoren ersichtlicher. Die Wörterbücher $W_a^{\mathcal{l}}$

und $W_b^{\mathcal{l}}$

bilden zusammen ein globales Wörterbuch der Gewichtungsfaktoren und ${\lambda }_{\text{i}}^{\mathcal{l}}$

können als Faktor-Scores von Client i angesehen werden. Differenzen hinsichtlich ${\lambda }_{\text{i}}^{\mathcal{l}}$

zwischen Clients erlaubt eine Anpassung des Modells an die Datenverteilung jedes Clients, während ein Teilen der zugrundeliegenden Faktoren $W_a^{\mathcal{l}}$

und $W_b^{\mathcal{l}}$

ein Lernen aus den Daten aller Clients ermöglicht.The factorization can be equivalently expressed as: $${\mathrm{W.}}_i^{\mathcal{l}}={\displaystyle \sum {}_{k=1}^{\mathrm{F.}}}{\lambda }_{i,k}^{\mathcal{l}}\left(w_{a,k}^{\mathcal{l}}\otimes w_{b,k}^{\mathcal{l}}\right)$$

whereby $w_{a,k}^{\mathcal{l}}$

the k ^th column of ${\mathrm{W.}}_a^{\mathcal{l}}$

is, $w_{b,k}^{\mathcal{l}}$

the kth line of ${\mathrm{W.}}_b^{\mathcal{l}}$

is and ⊗ denotes an external product. In this way the interpretation of the corresponding pairs of columns and rows is written $w_{a,k}^{\mathcal{l}}$

and $w_{b,k}^{\mathcal{l}}$

more apparent as weighting factors. The dictionaries ${\mathrm{W.}}_a^{\mathcal{l}}$

and ${\mathrm{W.}}_b^{\mathcal{l}}$

together form a global dictionary of weighting factors and ${\lambda }_{\text{i}}^{\mathcal{l}}$

can be viewed as client i factor scores. Differences regarding ${\lambda }_{\text{i}}^{\mathcal{l}}$

between clients allows the model to be adapted to the data distribution of each client while sharing the underlying factors ${\mathrm{W.}}_a^{\mathcal{l}}$

and ${\mathrm{W.}}_b^{\mathcal{l}}$

enables learning from the data of all clients.

kann als elementweises Produkt gebildet sein: $${\lambda }_i^{\mathcal{l}}=r_i^{\mathcal{l}}\odot b_i^{\mathcal{l}}$$

wobei $r_i^{\mathcal{l}}$

∈ ℝ^F eine Stärke jedes Faktors angibt und $b_i^{\mathcal{l}}$

∈ {0,1}^F ein binärer Vektor ist, der aktive Vektoren angibt. Wie unten beschrieben, ist $b_i^{\mathcal{l}}$

üblicherweise dünnbesetzt, sodass allgemein jeder Client nur einen kleinen Teilsatz der verfügbaren Gewichtungsfaktoren verwendet. Wie hierin verwendet, bezieht sich die Abwesenheit des hochgestellten Zeichens ℓ (z. B. λ_i) auf die gesamte Sammlung auf allen Schichten L, für die eine Faktorisierung durchgeführt wird. Es können Punktschätzungen für W_a, W_b und Faktorstärken r_i gelernt werden.Any client factor score ${\lambda }_i^{\mathcal{l}}$

can be formed as an element-wise product: $${\lambda }_i^{\mathcal{l}}=r_i^{\mathcal{l}}\odot b_i^{\mathcal{l}}$$

whereby $r_i^{\mathcal{l}}$

∈ ℝ ^F indicates a strength of each factor and $b_i^{\mathcal{l}}$

∈ {0,1} ^{F is} a binary vector indicating active vectors. As described below, is $b_i^{\mathcal{l}}$

usually sparsely populated, so that generally each client uses only a small subset of the available weighting factors. As used herein, the absence of the superscript ℓ (e.g., λ _i ) refers to the entire collection on all layers L that are factored. Point estimates for W _a , W _b and factor strengths r _{i can be} learned.

üblicherweise dünnbesetzt und folglich ist λ_i ebenfalls dünnbesetzt, was eine Konsolidierung von zusammenhängendem Wissen fördert, während Interferenz minimalisiert wird. Das heißt, ein Client A sollte fähig sein, globale Faktoren während eines Trainings zu aktualisieren, ohne die Fähigkeit eines Clients B zunichtezumachen, die Aufgabe von Client B durchzuführen. Andererseits sollten Faktoren von verschiedenen Clients wiederverwendet werden. Während Daten nicht-unabhängig und nicht-identisch an Clients verteilt werden können, gibt es häufig einige Ähnlichkeiten oder eine Überlappung der Daten. Die geteilten Faktoren verteilen ein Lernen auf alle Client-Daten, was ein Szenario mit N unabhängigen Modellen vermeidet. Zudem ist in einer verteilten Umgebung, die für föderales maschinelles Lernen in Betracht kommt, die Gesamtanzahl an Knoten selten vordefiniert. Daher sollte ein System in der Lage sein, problemlos erweitert zu werden, sodass es neue Clients aufnimmt, ohne das gesamte Modell neu zu initialisieren. Dieses Merkmal umfasst sowohl ein Erhöhen der Kapazität auf der Server-Seite (falls nötig) als auch eine Initialisierung von neuen Clients.Within the context of federal machine learning with statistical heterogeneity, there are a number of desirable properties that client factor scores should have in common. As mentioned earlier, is $b_i^{\mathcal{l}}$

usually sparse and consequently λ _{i is} also sparse, promoting consolidation of related knowledge while minimizing interference. That is, client A should be able to update global factors during training without negating client B's ability to perform client B's task. On the other hand, factors should be reused by different clients. While data can be distributed non-independently and non-identically to clients, there is often some similarity or overlap in the data. The shared factors distribute learning to all client data, which avoids a scenario with N independent models. In addition, in a distributed environment that is eligible for federal machine learning, the total number of nodes is rarely predefined. Therefore, a system should be able to easily expand to accommodate new clients without having to reinitialize the entire model. This feature includes both increasing the capacity on the server side (if necessary) and initializing new clients.

zu fördern, kann der diagonale Vektor unter Verwendung eines Prozesses regularisiert werden, der dem Indian Buffet Process (IBP) ähnlich ist. Die Aposteriori-Verteilung des diagonalen Vektors kann durch Variationsinferenz gezwungen werden, so nahe wie möglich an einem A-priori-Diagonalvektor zu sein. Die Verwendung eines Bayesschen nichtparametrischen Ansatzes kann es den Daten erlauben, eine Client-Faktor-Zuordnung, eine Faktorwiederverwendung und eine serverseitige Modellexpansion zu diktieren. Eine Stick-Breaking Construction kann mit dem IBP als A-priori-Verteilung für eine Faktorauswahl wie folgt verwendet werden: $$v_{i,k}^{\mathcal{l}}\sim Beta\left(\alpha \mathrm{,1}\right)$$

$${\pi }_{i,k}^{\mathcal{l}}={\displaystyle \prod {}_{k=1}^k}{v}_{i,k}^{\mathcal{l}}$$

$$b_{i,k}^{\mathcal{l}}\sim Bernoulli\left({\pi }_{i,k}^{\mathcal{l}}\right)$$

wobei α ein Hyperparameter sein kann, der eine erwartete Anzahl an aktiven Faktoren und die Rate an neuen Faktoren steuert, die aufgenommen werden, und k den Faktor indiziert.To a low data density on a diagonal personalized matrix ${\mathrm{\Lambda }}_i^{\mathcal{l}}$

To promote, the diagonal vector can be regularized using a process similar to the Indian Buffet Process (IBP). The a priori distribution of the diagonal vector can be constrained by variation inference to be as close as possible to an a priori diagonal vector. Using a Bayesian nonparametric approach can allow the data to dictate client-to-factor mapping, factor reuse, and server-side model expansion. A stick-breaking construction can be used with the IBP as an a priori distribution for a factor selection as follows: $$v_{i,k}^{\mathcal{l}}\sim \mathrm{B.}eta\left(\alpha \mathrm{,1}\right)$$

$${\pi }_{i,k}^{\mathcal{l}}={\displaystyle \prod {}_{k=1}^k}{v}_{i,k}^{\mathcal{l}}$$

$$b_{i,k}^{\mathcal{l}}\sim \mathrm{B.}ernOulli\left({\pi }_{i,k}^{\mathcal{l}}\right)$$

where α can be a hyperparameter that controls an expected number of active factors and the rate of new factors that are ingested, and k indicates the factor.

$$b_i^{\mathcal{l}}=Bernoulli\left({\pi }_i^{\mathcal{l}}\right)$$

$$v_i^{\mathcal{l}}=\sim Kumaraswamy\left(c_i^{\mathcal{l}},d_i^{\mathcal{l}}\right)$$

The a posteriori distribution can be learned _{for the random variables b i} and v _i. Exact inferences of the posteriori can be unsolvable, so variation inference that has a molecular field approximation can be used to determine the active factors for each client device using the following variation distributions, which include the variation parameters (i.e. seed values) {π _i , Learn c _i , d _i } for each requested client using Bayes-by-backprop propagation: $$q\left({\theta }_i^{\mathcal{l}}\right)=q\left(b_i^{\mathcal{l}}\right)q\left(v_i^{\mathcal{l}}\right)$$

$$b_i^{\mathcal{l}}=\mathrm{B.}ernOulli\left({\pi }_i^{\mathcal{l}}\right)$$

$$v_i^{\mathcal{l}}=\sim Kumaraswamy\left(c_i^{\mathcal{l}},d_i^{\mathcal{l}}\right)$$

wobei |

| die Anzahl an Trainingsbeispielen bei Client i ist. Der erste Term schafft eine Label-Überwachung und der zweite Term regularisiert die Aposteriori-Verteilung, um sich nicht zu weit von der IBP A-priori-Verteilung zu entfernen.In order to have a differentiable parameterization, a Kumaraswamy distribution can be used as a substitute for the beta distribution of v _i along with a slight relaxation of the Bernoulli distribution. The objective for every client is to maximize the lower limit of variation: $${\mathrm{L.}}_i={\displaystyle \sum {}_{n=1}^{\left|{\mathcal{D.}}_i\right|}}{\mathbb{E.}}_q\text{log}p\left(y_i^{\left(n\right)}|{\theta }_i{x}_i^{\left(n\right)}\right)-K\mathrm{L.}\left(q\left({\theta }_i\right)\left|\right|p\left({\theta }_i\right)\right)$$

where |

| is the number of training examples at client i. The first term creates label monitoring and the second term regularizes the a posteriori distribution so as not to deviate too far from the IBP a priori distribution.

A molecular field approximation can be used to allow expansion of the second term to: $$\mathrm{R.}={\displaystyle \sum {}_{\mathcal{l}=1}^{\mathrm{L.}}}K\mathrm{L.}\left(q\left(b_i^{\mathcal{l}}\right)\left|\right|p\left(b_i^{\mathcal{l}}|v_i^{\mathcal{l}}\right)\right)+K\mathrm{L.}\left(q\left(v_i^{\mathcal{l}}\right)\left|\right|p\left(v_i^{\mathcal{l}}\right)\right)$$

für E Epochen verwendet, wobei er nicht nur das Gewichtungsfaktorwörterbuch {W_a, W_b } und die Faktorstärken r aktualisiert, sondern auch Variationsparameter (π_i, c_i, d_i des Clients, die steuern, welche Faktoren der Client verwendet. Die Datenverteilung

kann Informationen bezüglich biometrischen Daten, medizinischen Daten, Bilddaten, Standortdaten, Anwendungsverwendungsdaten, thermische Daten, Atmosphärendaten und/oder Audio umfassen.Before the training begins, global weighting factors {W _a , W _b } and factor strengths r can be checked by the server 101 initialized. Once initialized, each round of training begins with {W _a , W _b , r} to a selected subset of the total number of clients 102 is sent. Each selected (sampled) client then trains the model by sharing its private data

for E epochs, updating not only the weighting factor dictionary {W _a , W _b } and the factor strengths r, but also variation parameters (π _i , c _i , d _{i of} the client which control which factors the client uses. The data distribution

may include information related to biometric data, medical data, image data, location data, application usage data, thermal data, atmospheric data, and / or audio.

As soon as a local training is completed, each client sends {W _a , W _b , r} back to the server, but not the variation parameters {π _i , c _i , d _i }, which remain with the client with a data distribution D _i. After the server 101 Updates received from all sampled clients can change the various new values for {W _a , W _b , r} by the server 101 can be aggregated using an averaging step, which in one embodiment may be a simple averaging step. The process then repeats with the server selecting a new subset of clients to be sampled, sending the new updated set of global parameters to the new subset, and so on until a desired number of rounds of communication has taken place. This process is summarized by the pseudo-code from algorithm 1.

When a client 102 entering an evaluation mode, the client can _{request a current version of global parameters {W a} , W _b , r} from the server. If the client has previously been asked about federal training, the local model comprises the aggregated global parameters and the binary vector that was generated by the local variation parameters {π _i , c _i , d _i } of the client. Otherwise the client only uses the aggregated {W _a , W _b , r}. It should be noted that if a client was previously sampled, the last cached copy of the global parameters at the client can be an option if a network connection is not available or too expensive. Usually clients are able to request the most up-to-date parameters.

Data security is one of the central aspects of federal machine learning. Simpler, more traditional methods of training a model can be used if all the data has first been aggregated at a central server. The very real possibility that confidential client data is intercepted during a transmission, or an attacker into the data store of the server 101 intruding is both a serious concern and one reason why the federal machine learning data is on the local device 102 to leave. However, it may not be sufficient just for security purposes leave the data on the client side. Just as data can be compromised in transit or in a central database in non-federal environments, federal training updates are similarly vulnerable. For example, in an exemplary method for federal machine learning, the update includes all parameters of the model. Basically, this can mean that instead of delivering the data immediately, white-box access to the model is given, which makes the model vulnerable to a wide range of malicious activities, including spying on the very data that the federal machine learning tries to protect.

verlassen die lokale Vorrichtung. Daher ist ein Angreifer möglicherweise nicht in der Lage, das Modell vollständig zu rekonstruieren, selbst falls eine Nachricht abgefangen wird, wodurch die Fähigkeit eines Angreifers behindert werden kann, einen Angriff durchzuführen, um die Daten wiederzugewinnen.For the federal machine learning system disclosed herein, clients transmit the entire dictionary of weighting factors {W _a , W _b } and factor strengths r to the server 101 , but not {π _i c _i , d _i }. This keeps the information related to what specific factors a client is using local. That is, neither the client data D _i nor a factor selection ${\mathrm{\Lambda }}_i^{\mathcal{l}}$

leave the local device. Therefore, even if a message is intercepted, an attacker may not be able to fully reconstruct the model, which may hamper an attacker's ability to attack to recover the data.

3 Figure 3 is a flow diagram for an exemplary embodiment of a method 300 for federal machine learning at a client device in accordance with the subject matter disclosed herein. The method starts at 301. Global parameters, ie global weighting factor dictionaries and factor strengths, can be made by the global server 101 initialized and to a selected subset of the total number of clients 102 sent before training begins. at 302 a group of parameters for a client device is selected by the client device from the global group of parameters. In one embodiment, the client uses variation parameters to form the selection of parameters for the client. at 303 the client device trains a model using a data set of the client device and the set of parameters selected by the client device. at 304 After the training, the client device sends a weighting factor dictionary updated by the client and a factor strength vector updated by the client to the global server 101 but not the variation parameters used by the client to form the selection of parameters for the client, or the data set of the client device. The global server 101 may aggregate dictionary components and factor strength vectors updated by the client using an averaging step. The global server 101 can select a new set of clients to be sampled and send the new updated set of global parameters to the new subset of clients. For the exemplary embodiment for the method 300 the client is selected as part of the new subset of clients. at 305 receives the client device from the global server 101 a globally updated weighting factor dictionary and a globally updated factor strength vector. at 306 the client device retrains the set of parameters selected by the client device, the globally updated weighting factor dictionary, and the globally updated factor strength vector based on the client's data set. The method can be continued until a desired number of training epochs has taken place. The method ends at 307.

4th represents an electronic device 400 that includes federal machine learning functionality in accordance with the subject matter disclosed herein. In one embodiment, the electronic device can 400 be a global server operable to provide federal machine learning as disclosed herein. In another embodiment, the electronic device 400 a client device operable to provide federal machine learning as disclosed herein. The electronic device 400 whether a global server or a client device, it can also function as a computing device, personal digital assistant (PDA), laptop computer, mobile computer, web tablet, wireless phone, cell phone, smartphone, digital music player, or wired or wireless electronic device to be embodied, but is not limited to it. The electronic device 400 can be a controller 410 , an input / output device 420 such as, but not limited to, a numeric keypad, a keyboard, a display, a touchscreen display, a camera and / or an image sensor, a memory 430 , an interface 440 , a GPU 450 and an image processing unit 460 include, interconnected by a bus 470 are coupled. The controller 410 may for example comprise at least one microprocessor, at least one digital signal processor, at least one microcontroller or the like. The memory 430 can be set up command code issued by the controller 410 is used or to save user data.

the interface 440 may be configured to include a wireless interface configured to transmit data from a wireless communication network using an RF signal or to receive. The wireless interface 440 may for example comprise an antenna. The electronic device 400 can also be used in a communication system communication interface protocol such as, but not limited to, Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), North American Digital Communications (NADC), Extended Time Division Multiple Access (E- TDMA), Wideband CDMA (WCDMA), CDMA2000, Wi-Fi, Urban Wi-Fi (Muni Wi-Fi), Bluetooth, Digital Enhanced Cordless Telecommunications (DECT), Wireless Universal Serial Bus (Wireless USB), Fast Low-latency Access With Seamless Handoff Orthogonal Frequency Division Multiplexing (Flash-OFDM), IEEE 802.20 , General Packet Radio Service (GPRS), iBurst, Wireless Broadband (WiBro), WiMAX, WiMAX-Advanced, Universal Mobile Telecommunication Service - Time Division Duplex (UMTS-TDD), High Speed Packet Access (HSPA), Evolution Data Optimized (EVDO ), Long Term Evolution - Advanced (LTE-Advanced), Multichannel Multipoint Distribution Service (MMDS), Fifth-Generation Wireless ( 5G) , and so forth.

Embodiments of the subject matter and the acts described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware comprising the structures disclosed in this specification and their structural equivalents, or in combinations of one or more than one thereof . Embodiments of the subject matter described in this specification can be implemented as one or more than one computer program; H. one or more modules of computer program instructions which are encoded on a computer storage medium for execution by a data processing device or for the control of processes of the same. Alternatively or additionally, the program instructions can be encoded on an artificially generated, broadcast signal, e.g. B. a machine-generated electrical, optical or electromagnetic signal that is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer storage medium can be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access storage array or device, or a combination thereof. While a computer storage medium is not a broadcast signal, a computer storage medium can also be a source or destination for computer program instructions encoded in an artificially generated broadcast signal. The computer storage medium may also be, or included within, one or more separate physical components or media (e.g., multiple CDs, floppy disks, or other storage devices). In addition, the operations described in this specification may be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

While this description may contain many specific implementation details, the implementation details should not be construed as limitations on the scope of any claimed subject matter, but rather as descriptions of features that are specific to particular embodiments. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Furthermore, although features have been described above as acting in certain combinations and are originally claimed as such, one or more features of a claimed combination may in some cases be separated from the combination and the claimed combination may refer to a sub-combination or a modification of a Refer to subcombination.

Also, while operations in the drawings are depicted in a particular order, it should not be construed as requiring that operations be performed in the particular order shown or in a sequential order, or that all of the operations depicted be performed in order to achieve desired results. In certain circumstances, multitasking and parallel processing can be beneficial. Furthermore, the separation of various system components in the embodiments described above should not be understood in such a way that such separation is necessary in all embodiments, and it should be recognized that the program components and systems described are generally combined in a single software product can be integrated or can be packaged in several software products.

Thus, certain embodiments of the subject matter have been described herein. Other embodiments are encompassed within the scope of the following claims. In some cases, the actions listed in the claims can be performed in a different order and still achieve the desired results. In addition, the processes illustrated in the accompanying figures do not necessarily require the particular order shown or sequential order in order to achieve desired results. In certain implementations, multitasking and parallel processing can be beneficial.

It will be apparent to a person skilled in the art that the innovative concepts described herein can be modified and varied for a wide range of applications. Accordingly, the scope of claimed subject matter should not be limited to any of the specific exemplary teachings set forth above, but rather is defined by the following claims.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 63/033747 [0001]

Claims (20)

A client device in a federal machine learning system, the client device comprising: at least one computing device; a communication interface; and a processor coupled to the at least one computing device and to the communication interface, the processor: selects a set of parameters for the client device from a global set of parameters, trains a model using a data set of the client device and the group of parameters selected by the client device, the data set being formed on the basis of an output of the at least one computing device, updated a weighting factor dictionary and factor strength vector after training the model, sends a client-updated weighting factor dictionary and a client-updated factor strength vector through the communication interface to a global server, and receives a globally updated weighting factor dictionary and a globally updated factor strength vector through the communication interface from the global server retrained the model using the client device's data set, the set of parameters selected by the client device, and the globally updated weighting factor dictionary and globally updated factor strength vector. Client device after Claim 1 , wherein the client device is part of a group of N client devices, where N is an integer. Client device after Claim 2 wherein the processor selects the group of parameters from the global group of parameters using three variation parameters having seed values and a difference between supervised learning of the data set and regularization of the selected group of parameters and the global group of parameters minimized. Client device after Claim 3 wherein the processor selects the group of parameters from the global group of parameters by receiving the global group of parameters sent from the global server to a first subset of client devices of the N client devices, the client devices Device is part of the first subset of client devices. Client device after Claim 4 wherein the client device receives the globally updated weighting factor dictionary and a globally updated factor strength vector by receiving the globally updated weighting factor dictionary and a globally updated factor strength vector sent by the global server to a second subset of the N client devices, the client Device is part of the second subset of client devices. Client device after Claim 4 , wherein the processor sends a request through the communication interface to the global server for a current version of the global group of parameters, wherein the processor updates the model using the current version of the global group of parameters, and wherein the processor evaluates the model under Using the current version of the global set of parameters, updated to inference based on the client device's record. Client device after Claim 1 wherein the data set includes information relating to at least one of biometric data, medical data, image data, voice data, location data, application usage data, thermal data, atmospheric data, audio data and survey data. A federal machine learning system, comprising: a global server that receives updates to weighting factor dictionaries and factor strength vectors from N client devices, where N is an integer, and which generates a globally updated weighting factor dictionary and a globally updated factor strength vector; and the client devices, wherein at least one client device comprises: at least one computing device, a communication interface, and a processor coupled to the at least one computing device and to the communication interface, the processor: selecting a group of parameters from a global group of parameters, a model using a data set of the client device and the group of parameters determined by the client device is selected, trained, updated a weighting factor dictionary and a factor strength vector after training the model, sending a client-updated weighting factor dictionary and a client-updated factor strength vector through the communication interface, through the communication interface from the global server the globally updated weighting factor dictionary and the receives globally updated factor strength vector, and the model using the client device's data set, the set of parameters selected by the client device, and the globally updated weight gs factor dictionary and the globally updated factor strength vector retrained. Client device after Claim 8 wherein the processor selects the group of parameters from the global group of parameters using three variation parameters having seed values and a difference between supervised learning of the data set and regularization of the selected group of parameters and the global group of parameters minimized. Client device after Claim 9 wherein the processor selects the group of parameters from the global group of parameters by receiving the global group of parameters sent from the global server to a first subset of client devices of the N client devices, the client devices Device is part of the first subset of client devices. Client device after Claim 10 wherein the client device receives the globally updated weighting factor dictionary and a globally updated factor strength vector by receiving the globally updated weighting factor dictionary and a globally updated factor strength vector sent by the global server to a second subset of the N client devices, the client Device is part of the second subset of client devices. Client device after Claim 10 , wherein the processor sends a request through the communication interface to the global server for a current version of the global group of parameters, wherein the processor updates the model using the current version of the global group of parameters, and wherein the processor evaluates the model under Using the current version of the global set of parameters, updated to inference based on the client device's record. Client device after Claim 8 wherein the data set includes information relating to at least one of biometric data, medical data, image data, voice data, location data, application usage data, thermal data, atmospheric data, audio data and survey data. A method for federal machine learning, the method comprising: Selecting, at a client device, a set of parameters from a global set of parameters, the global set of parameters including a weighting factor dictionary and a factor strength vector; Training, at the client device, a model using a data set of the client device and the set of parameters selected by the client device; Updating a weighting factor dictionary and a factor strength vector after training the model; Sending, from the client device to a global server, a client-updated weighting factor dictionary and a client-updated factor strength vector; Receiving at the client device an updated weighting factor dictionary and a globally updated factor strength vector from the global server; and retraining the model on the client device using the data set of the client device, the set of parameters selected by the client device, and the globally updated weighting factor dictionary and globally updated factor strength vector. Procedure according to Claim 14 , wherein the client device is part of a group of N client devices, where N is an integer. Procedure according to Claim 15 wherein selecting the group of parameters from the global group of parameters further comprises selecting the group of parameters using three variation parameters having seed values; and to minimize a difference between supervised learning of the data set and regularization of the selected group of parameters and the global group of parameters. Procedure according to Claim 16 wherein selecting the group of parameters from the global group of parameters further comprises receiving at the client device the global group of parameters sent by the global server to a first subset of client devices of the N client devices wherein the client device is part of the first subset of client devices. Procedure according to Claim 17 wherein receiving the globally updated weighting factor dictionary and a globally updated factor strength vector from the global server at the client device further comprises receiving at the client device the globally updated weighting factor dictionary and a globally updated factor strength vector sent by the global server to a second Subset of the N client devices were sent, the client device being part of the second subset of client devices. Procedure according to Claim 17 , further comprising: the client device requesting a current version of the global set of parameters from the global server; Receiving the current version of the global group of parameters; Updating the model using the current version of the global set of parameters; and evaluating the model updated using the current version of the global set of parameters to inference based on the client device data set. Procedure according to Claim 14 wherein the data set includes information relating to at least one of biometric data, medical data, image data, voice data, location data, application usage data, thermal data, atmospheric data, audio data and survey data.

Images