DE102023200766A1 - Computer-implemented method for generating labels for a data set and use of a labeled data set generated in such a way - Google Patents

Abstract

Computer-implemented method for generating labels for a data set, wherein the data set comprises measured variables of a measured variable sensor as data (1) and a machine learning model for pattern recognition can be trained using the generated labeled data set, the method comprising the steps of: receiving the data set (V1); mapping the data (1) of the data set to codes (2) in a feature space by means of an encoder (3), wherein in the feature space the codes (2) are determined by the respective data (1) in relation to characteristics of the data (1) and the encoder (3) is designed to recognize the characteristics of the data (1) and to map the data (1) accordingly (V2); visualizing the feature space for human perception of information, wherein groups (1.0-1.9) of data (1) with similar characteristics are generated in a dimensionally reduced space (4) by means of dimensionality reduction of the codes (2) (V3); generating labels in groups, wherein a data sample (1) is selected by a human from at least one of the groups (1.0-1.9) of data (1) and a label is assigned to the data sample (1), thus labeling all data samples (1) of this group (1.0-1.9) accordingly (V4).

Description

The invention relates to a computer-implemented method for generating labels for a data set and a use of a labeled data set generated in this way.

The following definitions, descriptions and statements retain their respective meaning for and apply to the entire disclosed subject matter.

Labeled data, i.e. labeled or tagged data that is provided with identifiers, for example in the form of meta information, is of central importance for every machine learning project. In the machine perception of the environment of driving systems, for example in autonomous driving, there is a need for labels, especially detailed labels. Mere object class information such as car or pedestrian is not sufficient. Additional semantic object information such as "door open", "on foot", "conscious", "unconscious", "eye contact" or meta information such as "night", "day", "sun", "rain" are required. This requires labeling with a high throughput, also known as high-throughput labeling.

The US 11,436,428 B2 discloses a method for increasing the data quality of a dataset for semi-supervised machine learning analysis.

The invention was based on the task of how high-throughput labeling can be enabled for training a machine learning model.

The subject matter of the independent and the subordinate claim each solves this problem by generating a latent feature space for data samples of the data set. Codes generated from the data in the feature space are mapped into a dimension-reduced space and clustered in the dimension-reduced space. The clusters are labeled cluster by cluster. The solution according to the invention thus provides high-throughput labeling that enables cluster labeling within the common latent feature space of a model. Extensive labeled data is generated that is required for the autonomous development and use of machine perception models. In combination with a selection tool, high throughput is enabled when labeling classification features.

Advantageous embodiments of the invention emerge from the definitions, the subclaims, the drawings and the description of preferred embodiments.

The computer-implemented method disclosed here generates labels for a data set. The data set includes measured variables of a measured variable sensor as data. Computer-implemented means that the method is carried out at least partially with a programmable device.

Measurement quantities can be, for example, pixel values of an image, such as brightness values, or sound field quantities, amplitudes or frequencies of electrical signals.

A measurement sensor is a technical component, such as a sensor or detector. The data is therefore measured data. The measurement sensor is a real or a simulated measurement sensor. The outputs of the simulated measurement sensor, i.e. the simulated measured data, interact with a person when carrying out the procedure, for example when selecting data samples from a group of data, and thus symbolize an interaction with the external physical reality. The data can therefore be real measured data or simulated measured data.

Using the generated labeled data set, a machine learning model for pattern recognition can be trained. The machine learning model can be an artificial neural network, for example. Pattern recognition includes classification and segmentation. For example, the machine learning model is trained to classify traffic signs in digital images or to semantically segment video recordings of traffic scenes, for example to segment pedestrians and vehicles in video recordings.

One step of the method is receiving the data set. The data set is received by means of a receiving module. The receiving module can be implemented in the form of a software code section, for example in the form of a read command, which, when loaded or executed, reads the data set from a digital memory. The receiving module can also be implemented in the form of hardware, for example in the form of a technical component for receiving and/or reading data.

A further step of the procedure is to map the data of the data set to codes in a feature space using an encoder. After In one aspect, the feature space is a latent feature space. A latent feature space is an embedding of a set of elements within a manifold in which elements with similar characteristics are closer to each other in the latent feature space. The characteristic of a data set is any property of the data set. An example of a characteristic of a data set is a step size between two points in time, a step size between two hulls of a trajectory, a change in direction of a trajectory, a size of a hull that encompasses an object, or the like. In another aspect, the feature space includes cropped objects or, in the case of classification, the entire image as input value. The codes are elements of the feature space, in one aspect of the latent feature space, for example vectors. In the feature space, in one aspect in the latent feature space, the codes are determined by the respective data in relation to characteristics of the data. The encoder is designed to recognize the characteristics of the data and to map the data accordingly. The feature space, in one aspect the latent feature space, can be high-dimensional, for example 128-dimensional.

A further step in the process is visualising the feature space according to an aspect of the latent feature space for human perception of information. By visualising the feature space according to an aspect of the latent feature space, the codes are displayed, namely clustered, in such a way that people are able to understand the groups quickly and efficiently. The groups are also called clusters. The focus here is not on conveying certain content or conveying it in a special way, but on presenting the groups in a way that takes into account the physical conditions of human perception and absorption of information and is aimed at making the perception of the information shown by people in a certain way in the first place, in particular by displaying the groups as separately from one another as possible. Such a presentation, i.e. the step of visualising the feature space according to an aspect of the latent feature space, serves to solve a technical problem using technical means. The groups of data with similar characteristics are generated by dimensionality reduction of the codes in a dimensionally reduced space that is accessible to human perception.

The dimensionality reduction is carried out using a computational model, a statistical method or a trained machine learning model. The computational model, the statistical method or the trained machine learning model determine which dimensions of the high-dimensional feature space can be merged according to an aspect of the latent feature space in order to obtain the dimensionally reduced space.

A further step is the group-wise generation of labels. In this process, a human selects a data sample from at least one of the groups of data. This data sample is assigned a label. All data samples in this group then automatically receive this label. The action of selecting the data sample by the human or user is also referred to as human in the loop. In one aspect, a user interface to the data set, also known as a human machine interface, is provided for selecting the data sample.

For example, all data samples in that group will be automatically selected and labeled with that label using a selection tool, such as a lasso tool or similar masking tool. The lasso tool is software that works on the active layer of an image and is used to trace the edges of a selection by clicking and dragging. The area enclosed by a cursor path remains selected and open to various transformation operations, such as coloring, moving, scaling, cutting, copying, or pasting, until clicked elsewhere in the image, at which point the lassoed selection is merged with the layer from which it was selected.

This means that all data samples in this group are labeled accordingly. This enables high-throughput labeling.

According to one aspect, the data set includes measured variables of at least one environment detection sensor of a driving system. The environment detection sensor can be, for example, a camera sensor, a radar sensor, a lidar sensor, an acoustic sensor, an odor sensor or an acceleration sensor. Environment can be the vehicle exterior or the vehicle interior. According to one aspect, the environment detection sensor can detect or perceive a vehicle exterior, i.e. the external environment of the vehicle. For example, the environment detection sensor is a radar system with a range of several hundred meters in relation to a point of attachment of the radar system on the vehicle. According to another aspect, the environment detection sensor can detect or perceive a vehicle interior, for example a cabin of a shuttle. For example, the environment detection sensor is an interior camera. In this case, the measured variables are unstructured data, for example pixel values, point clouds or audio signals. By coding image recordings from a camera sensor in the feature space, according to one aspect the latent feature space, groups of daytime and nighttime shots are obtained in a dimension-reduced space. If, for example, a daytime shot is selected from the group of daytime shots and manually labeled as a daytime shot, then all shots in the group of daytime shots automatically receive the label daytime shot. This makes labeling more efficient compared to labeling each individual daytime shot. The same applies to the group of nighttime shots.

According to another aspect, the data set includes measured variables that were recorded in a field bus of a driving system. The field bus connects measured variable sensors and actuators to each other or to a control unit. Measured variables are, for example, steering angle and slip angle. Measured variables from a field bus have a field bus-specific format and are an example of structured data. The field bus is, for example, a CAN bus.

According to another aspect, the data set is received from a cloud storage, for example via a radio interface. In a cloud storage, data from several peripheral units can be collected centrally and thus a large volume of data can be made available. For example, data from environment detection sensors of fleet vehicles can be stored in the cloud storage. High-throughput labeling is particularly effective on large amounts of data.

According to another aspect, the encoder is a machine learning model that is trained to extract characteristics from data. For example, the encoder is a backbone of a trained autoencoder or the backbone of a trained artificial neural network, for example the backbone of the ImageNET. By processing data through several layers of trained machine learning models, the data is mapped to a feature space, according to one aspect to a latent feature space. The encoder can be trained on any data.

According to another aspect, the encoder is trained on the dataset, that is, on the received dataset. By training the encoder to generate the feature space, according to an aspect of the latent feature space, on the data to be labeled, the separation of the groups in the dimensionally reduced space is improved.

According to a further aspect, the groups are displayed by means of a display device, for example by means of a monitor. The display device is a technical component of the method.

According to another aspect, the groups of data with similar characteristics are generated in a two-dimensional space. In the two-dimensional space, the groups are particularly quickly and efficiently perceived by a human.

According to a further aspect, the dimensionality reduction is carried out by means of t-distributed stochastic neighbor embedding, abbreviated t-SNE, or uniform manifold approximation and projection, abbreviated UMAP, or another technique from the field of dimensionality reduction, for example nonlinear dimensionality reduction, for example Sammon's mapping, locally-linear mapping or relational perspective map.

t-SNE is a nonlinear dimensionality reduction technique suitable for embedding high-dimensional data into a low-dimensional two- or three-dimensional space for visualization. In particular, each high-dimensional object is modeled by a two- or three-dimensional point, in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by faraway points with high probability.

UMAP is a dimension reduction method that, similar to t-SNE, can be used for visualization, but also for general nonlinear dimension reduction.

According to a further aspect, subgroups are created for at least one of the groups created and labels are created for the subgroups. A data sample from at least one of the subgroups is selected and a label is assigned to the data sample. This means that all data samples of this subgroup are labeled accordingly. Based on the example of groups of daytime shots and nighttime shots, for example, the group of daytime shots can be divided into the subgroups sunshine, cloudy, rain. By labeling one data sample from each of these subgroups, all daytime shots with sunshine are then automatically labeled as daytime shots. labeled with sunshine and accordingly day shot with clouds and day shot with rain.

With the use of a labeled data set generated according to the invention disclosed here, an artificial neural network can be trained for classification and/or segmentation of images or video recordings, for classification of audio data or for pattern recognition in measured variables recorded in a field bus of a driving system.

• 1 ein Ausführungsbeispiel eines Codes,
• 2 ein Ausführungsbeispiel eines Kodierers,
• 3 ein Ausführungsbeispiel einer Projektion von Codes aus einem latenten Merkmalsraum in einen zweidimensionalen Raum und
• 4 ein Ausführungsbeispiel eines erfindungsgemäßen Verfahrens.

The invention is explained by way of example in the following figures. They show:

• 1 an example of a code,
• 2 an embodiment of an encoder,
• 3 an embodiment of a projection of codes from a latent feature space into a two-dimensional space and
• 4 an embodiment of a method according to the invention.

In the figures, identical reference symbols refer to identical or functionally similar parts. For the sake of clarity, the relevant reference parts for the respective understanding are indicated in the respective figures.

1 shows as an example of a data sample 1 of a data set a digital image taken with a camera sensor. An encoder, for example as in 2 shown, extracts the most important features of the original image and creates a code 2 in a latent feature space. The encoder compresses the original image, for example. A decoder creates the original image again from the information of the code 2.

The 2 The encoder shown is, for example, a backbone of an artificial neural network. The data 1 is fed into an input layer of the artificial neural network. After a certain intermediate layer in the artificial neural network, the data 1 received there and processed by the artificial neural network are read out as codes 2.

3 shows a UMAP projection of the codes 2 of the latent feature space of the MNIST dataset. The MNIST dataset contains 28x28x1 pixel images of the digits 0 to 9. For example, the 2 shown encoder 3 maps these images onto the latent feature space. The UMAP projection reduces the dimensionality of the latent feature space to dimension two and the dimension-reduced two-dimensional space 4 shown is obtained. This dimension-reduced space 4 is displayed to a person on a display device 5, for example a monitor. As a result of the UMAP projection, the groups 1.0-1.9 of data are formed in the dimension-reduced space 4. Each of these groups 1.0-1.9 corresponds to a digit class of the MNIST data set. For example, group 1.3 includes the digits 3. High-throughput labeling is achieved by, for example, selecting one of the digits 3 from group 1.3 and labeling it as digit 3. Using a selection tool, all digits in group 1.3 are then automatically labeled as digits 3.

In a process step V1 of the 4 In the embodiment of a method according to the invention shown, data 1 of a data set is received, for example from a cloud storage. In a method step V2, the data 1 is mapped onto codes 2 in a latent feature space by means of the encoder 3. In a method step V3, the latent feature space is visualized as a two-dimensional space 4 on a display device 5 by means of dimensionality reduction, groups of data being generated as a result of the dimensionality reduction. In a method step V4, labels are generated in groups.

Reference symbol

1

data, data sample

1.0-1.0

group

2

code

3

Encoder

4

dimensionally reduced space

5

Display device

V1-V4

Process steps

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• US 11436428 B2 [0004]

Claims (11)

Computer-implemented method for generating labels for a data set, wherein the data set comprises measured variables of a measured variable sensor as data (1) and a machine learning model for pattern recognition can be trained using the generated labeled data set, the method comprising the steps: • Receiving the data set (V1); • Mapping the data (1) of the data set to codes (2) in a feature space using an encoder (3), wherein in the feature space the codes (2) are determined by the respective data (1) in relation to characteristics of the data (1) and the encoder (3) is designed to recognize the characteristics of the data (1) and to map the data (1) accordingly (V2); • Visualizing the feature space for human perception of information, wherein groups (1.0-1.9) of data (1) with similar characteristics are generated in a dimensionally reduced space (4) by means of dimensionality reduction of the codes (2) (V3); • generating labels in groups, whereby a human selects a data sample (1) from at least one of the groups (1.0-1.9) of data (1) and assigns a label to the data sample (1), thereby labeling all data samples (1) of this group (1.0-1.9) accordingly (V4). Procedure according to Claim 1 , wherein the data set comprises measured variables of at least one environment detection sensor of a driving system. Method according to one of the preceding claims, wherein the data set comprises measured variables recorded in a field bus of a driving system. Method according to one of the preceding claims, wherein the data set is received from a cloud storage. Method according to one of the preceding claims, wherein the encoder (3) is a machine learning model trained to extract characteristics from data (1). Procedure according to Claim 5 , where the encoder (3) was trained on the dataset. Method according to one of the preceding claims, wherein the groups (1.0-1.9) are displayed by means of a display device (5). Method according to one of the preceding claims, wherein the groups (1.0-1.9) of data (1) with similar characteristics are generated in a two-dimensional space (4). Method according to one of the preceding claims, wherein the dimensionality reduction is carried out by means of t-distributed stochastic neighborhood embedding or uniform manifold approximation and projection. Method according to one of the preceding claims, wherein subgroups are generated for at least one of the generated groups (1.0-1.9) and labels are generated for the subgroups, wherein a data sample (1) is selected from at least one of the subgroups and a label is assigned to the data sample (1), and thus all data samples (1) of this subgroup are labeled accordingly. Use of a labeled data set generated according to one of the preceding claims for training an artificial neural network for classification and/or segmentation of images or video recordings, for classification of audio data or for pattern recognition in measured variables recorded in a field bus of a driving system.

Images