DE102024108224B3 - Method for the relational storage of object properties of an object, for training artificial intelligence and for generating a class relating to an object, computer program product, storage medium, data carrier signal, devices and vehicle - Google Patents

Abstract

A computer-implemented method for relationally storing object properties 116 of an object 102, 104 comprises a step of obtaining 118 sensor data 172, 174, 176, 178 in which the object 102, 104 is detected, wherein the sensor data 172, 174, 176, 178 are generated by at least one sensor 106, 168, 170 arranged on a vehicle 100. Furthermore, the method 116 comprises a step of detecting and classifying 120 the object 102, 104 using the sensor data 172, 174, 176, 178, wherein the object 102, 104 is assigned a first class and at least one object property is determined, wherein a confidence value 191 is generated. Furthermore, the method 116 comprises a step of requesting 124 an occupant of the vehicle 100 to provide a second class relating to the object 102, 104 if the confidence value 191 is below a first threshold value 192. Furthermore, the method 116 comprises a further step of receiving 126 a first feedback from the occupant. Furthermore, the method 116 comprises a further step of extracting 128 the second class from the first feedback from the occupant. Furthermore, the method 116 comprises a further step of storing 130 a data set 193 comprising at least the second class, the at least one object property, and a derivative of the sensor data 172, 174, 176, 178.

Description

Technical area

The present invention relates to the field of data processing. In particular, the present invention relates to computer-implemented methods for the relational storage of object properties of an object. Furthermore, the present invention relates to a computer-implemented method for training artificial intelligence, computer-implemented methods for generating a class relating to an object, a computer program product, a computer-readable storage medium, a data carrier signal, devices for classifying an object, and a vehicle.

Technical background and task

The detection of objects outside a vehicle is used to enable automated driving. For this purpose, the objects must also be classified in order to calculate appropriate automated driving maneuvers. The technology is also used to transmit information to vehicle occupants. For example, systems are known that detect and classify vehicles in front of the vehicle, which are then displayed using a digital graphic model based on the vehicle's class. A truck is then represented as such and not as a generic vehicle.

Detection and classification are typically performed using artificial intelligence, such as neural networks (NNs). The accuracy with which a NN can detect and classify an object depends on the training data set. If a specific object is missing from this data set, it can only be reliably detected and classified with low probability. If a vehicle occupant, such as the driver, is presented with a misinterpreted vehicle on an in-vehicle display, this can confuse the driver and, in the worst case, provoke an accident.

The DE 10 2018 219 125 A1 relates to a method for classifying objects by means of an automated motor vehicle, wherein the automated motor vehicle has a device for detecting and classifying objects as well as output and input means, wherein the detected but unclassifiable or not clearly classifiable object is brought to the attention of at least one passenger of the automated motor vehicle via the output means, wherein the passenger is requested to classify the detected object, wherein the classifications of the at least one passenger are stored, as well as an automated motor vehicle.

The DE 10 2014 015 075 A1 relates to a method for operating an automated, driverless motor vehicle, in particular a passenger car, wherein sensor data recorded by environmental sensors, comprising at least one camera, of the motor vehicle are evaluated with regard to objects to be taken into account in trajectory planning, which objects can be classified as an obstacle or not an obstacle by means of at least one classifier which evaluates the associated sensor data, wherein in the case of an object which cannot be classified as an obstacle or not an obstacle or cannot be classified with sufficient certainty and/or in the case of at least one object which prevents the motor vehicle from continuing to a current destination, at least one camera image of the corresponding object is recorded with at least one of the at least one camera, transmitted to a portable mobile communication device carried by a user of the motor vehicle and displayed there, an input from a user classifying the object as an obstacle or not an obstacle is received as classification information, the classification information is transmitted back to the motor vehicle and taken into account in the further automatic guidance of the motor vehicle.

The DE 10 2015 007 493 A1 relates to a method for training a decision algorithm used in a control unit of a motor vehicle and based on machine learning, wherein the decision algorithm determines output data to be taken into account for controlling the operation of the motor vehicle and a reliability value describing the reliability of the output data depending on input data describing the current operating state and/or the current driving situation, and was trained before use in the motor vehicle using a basic training data set, wherein if the reliability value falls below a threshold value, the input data underlying the determination of the output data assigned to the reliability value are stored as assessment input data and presented to a human assessor at a later time, after which assessment output data corresponding to the output data are received by an operator input from the assessor and the decision algorithm is trained using an improvement training data set formed from the assessment input data and the assigned assessment output data.

In the context of automated driving, a misinterpretation can lead to an overreaction or underreaction. For example, a safety-relevant object may not be recognized as such and no appropriate evasive maneuver may be initiated. Or a safety-irrelevant object may be classified as safety-relevant and the vehicle may perform an unnecessary maneuver, which may surprise other road users.

It is therefore the object of the present invention to provide computer-implemented methods for the relational storage of object properties of an object, which eliminate at least one of the aforementioned disadvantages. Furthermore, it is the object of the invention to provide a computer-implemented method for training an artificial intelligence, computer-implemented methods for generating a class relating to an object, a computer program product, a computer-readable storage medium, a data carrier signal, devices for classifying an object, and a vehicle.

Disclosure of the invention

The object is achieved according to the invention by the features of the main claims. Advantageous embodiments can be found in the subclaims.

According to a first aspect of the invention, a computer-implemented method for training artificial intelligence comprises a step of obtaining sensor data in which an object is detected. The sensor data is generated by at least one sensor arranged on a vehicle.

The sensor data can be, for example, image data from a camera or radar data from a radar sensor. A vehicle can be, for example, a motor vehicle, a truck, a train, a helicopter, an aircraft, or the like. Object properties are specific pieces of information in the sensor data, typically related to a region within the sensor data. This can be a physical region such as coordinates, or another region, such as a frequency. Examples of object properties are points, lines, or corners—i.e., geometric properties. A relationship between such information, for example, a density of geometric properties or their specific arrangement, is also understood as object properties.

In a further step of the method, the object is detected and classified using the sensor data, whereby a first class is assigned to the object and at least one object property is determined. A confidence value is generated in this step.

This step represents a well-known task in data and image processing. Those skilled in the art are familiar with various methods available to solve this problem. Typically, such a task is solved using a (convolutional) neural network (CNN).

Essentially, a neural network is trained to recognize patterns and features within an image or other data that indicate a specific class. Common methods generate a confidence value that indicates how likely the class has been correctly assigned to the object. The higher the confidence value, the more certain the object is to be assigned to the correct class. Lower values, on the other hand, indicate that the object is not assigned to the correct class. This can happen, for example, if the (C)NN was not trained on the class and the class is therefore unknown. The first class can therefore also indicate an unknown class.

In a further step of the procedure, a passenger of the vehicle is asked to provide a second class concerning the object if the confidence value is below a first threshold.

The first threshold represents a limit below which assignment to a class is considered certain, for example 0.95 or 95%. If the confidence value is below this, the object may have been assigned the wrong class. In this case, a vehicle occupant, such as the driver, is asked to provide the correct class, which corresponds to the second class. The request can be made via the vehicle's audio system, for example, by asking the driver "What kind of object is that in front of us?" A display can also be used to address such a request to the occupant. For example, objects can be shown to the driver as symbols/icons, and the driver taps the appropriate symbol/icon.

In a further step of the process, initial feedback from the occupant is received.

This can be done, for example, via a microphone as audio feedback or by pressure via a touch display.

In a further step of the procedure, the second class is extracted from the first feedback of the occupant.

If the feedback is provided as audio feedback, speech recognition may need to analyze the audio feedback to extract the second class. For example, if the passenger responds "This is a carriage," only the word "carriage" needs to be extracted as the second class.

In a further step of the method, a data set is stored which comprises at least the second class, the at least one object property, a derivative of the sensor data and an identifier of the occupant and/or the vehicle.

The dataset can also contain only a correlation between the second class, the at least one object property, and the derivative of the sensor data. This means that the actual data is not stored together in one data structure. This allows the different components of the dataset to be stored in a distributed manner.

A derivative of the sensor data can be the sensor data itself or a modification of it, for example, compressed sensor data or only the part of the sensor data that contains object properties. This can potentially save storage space.

The identifier allows conclusions to be drawn about the user of the process. Although the identifier is unique, it can be anonymous. For example, this information can be used to exclude occupants or vehicles from the process that intentionally attempt to establish false classes. The occupant can be identified, for example, via voice recognition, their connected mobile device such as a smartphone, or facial recognition using interior cameras, which allows assignment to an existing identifier or the creation of a new one. An identifier for a vehicle can be realized, for example, via a vehicle serial number.

The artificial intelligence is trained with at least the derivative of the sensor data or the object properties as input and with the second class as output.

Machine learning methods, such as a neural network (NN) or a convolutional neural network (CNN), are particularly suitable for this. These can receive sensor data, such as image data, as input. At the same time, the correct class, namely the second class, is known and can be specified as output to the NN or CNN. In a training phase, the weights of the connections between the artificial neurons of the (C)NN are updated using an error function.

• - Auffordern des Insassen des Fahrzeuges, die erste Klasse des Objektes zu bestätigen,
• - Empfangen einer zweiten Rückmeldung des Insassen,
• - Extrahieren eines Indikators aus der zweiten Rückmeldung des Insassen, wobei der Indikator einen Wahrheitswert indiziert, ob das Objekt der ersten Klasse zugehörig ist, und
• - Speichern des Indikators.

Furthermore, the procedure has further steps if the confidence value is above a second threshold:

• - Asking the occupant of the vehicle to confirm the first class of the object,
• - Receiving a second feedback from the occupant,
• - Extracting an indicator from the second feedback of the occupant, where the indicator indicates a truth value whether the object belongs to the first class, and
• - Save the indicator.

For example, the request can be made via the vehicle's audio system, asking the driver, "Is there a carriage in front of us?" The driver's response can be provided via voice feedback or a touch display. Again, it may be necessary to analyze the feedback to extract the indicator. For example, if the passenger answers "Yes, that's a carriage," this can be converted into a binary value of a Boolean variable.

The inmate is only prompted to provide feedback if the object was detected with a confidence above a second threshold. The second threshold is typically higher than the first threshold, for example, 0.99 or 99%. However, if the inmate's response also suggests an incorrectly detected class, the inmate may be attempting to introduce additional or incorrect classes into the process. The indicator can therefore be used to identify such users (trolls) of the process.

The indicator is preferably stored in correlation with the occupant. Storing it in correlation with the underlying data set, particularly the sensor data and the object properties, is also advantageous. This correlation or information can be used to train artificial intelligence more specifically.

Furthermore, the computer-implemented method for training an artificial intelligence comprises a step in which the previously described steps are repeated for a plurality of Insas sen. A variety of indicators are generated and stored.

In a further step of the process, a number of indicators is calculated for each occupant or vehicle. Using the identifier, indicators can be assigned to the occupants or vehicles. In particular, the number indicates how often a specific occupant or vehicle has provided feedback indicating that an object was not correctly classified, despite a high confidence value for the classification.

In a further step of the process, the artificial intelligence is trained only with data sets whose identifiers have a number of indicators below a third threshold. The third threshold defines a limit up to which an occupant or vehicle is still considered trustworthy. For example, above a third threshold of "5," any further feedback from the affected occupant is not used to train the artificial intelligence.

A class relating to an object can be created by first performing a procedure as described above. A plurality of data records are created and stored.

The data records can then be divided into groups, with data records that have similar object properties and/or similar second classes being grouped into a specific group. For example, data records with an identical second class, i.e. the class suggested by the occupants, are grouped together. Data records with similar object properties, such as similar geometric patterns such as points or lines, can also be added to the group. The similarity of such patterns can be calculated, for example, using a geometric deviation. The similarity of second classes can be calculated, for example, using a translation if the majority of occupants come from different countries and therefore suggest different words for the second class, but actually mean the same thing. The same applies to occupants who use synonymous words for the same object class, such as "carriage" and "horse-drawn carriage."

A count is then calculated for each group. In other words, it calculates how many occupants or vehicles have already consistently classified certain object characteristics.

Subsequently, a third class is generated for those groups whose number exceeds a fourth threshold. In other words, a new class should not be generated from a single response. Rather, swarm intelligence should be implemented, meaning that a new class is only generated when multiple responses have consistently classified a previously unknown object. For example, the fourth threshold for this could be "10." Therefore, 10 occupants must consistently classify an unknown object before this class is assigned to the object.

In a further step of the process, the third class is assigned to the data sets in the corresponding groups. This creates a correlation between the occupant feedback, i.e., the class of the previously unknown object, and the data sets underlying the feedback.

According to a second aspect of the invention, a computer program product comprises instructions which, when executed by a computer, cause the computer to perform a method as described above. The computer program product can be written in a programming language, for example, Python or C++.

According to a third aspect of the invention, a computer-readable storage medium has instructions that, when the program is executed by a computer, cause the computer to perform a method as described above. The computer-readable storage medium can be implemented, for example, as an SSD (solid-state disk) or as a flash memory. The computer-readable storage medium can also store other data, for example, sensor data and/or data that is (temporarily) stored during the execution of the method.

According to a fourth aspect of the invention, a data carrier signal transmits the computer program product as described above. The data carrier signal can be transmitted via a cable, for example, via a CAN bus (controller area network) or Ethernet cable, or wirelessly via Wi-Fi, Bluetooth, or the like.

According to a fifth aspect of the invention, a device for classifying an object comprises an evaluation unit configured to execute a method as described above. The evaluation unit may, for example, comprise a processor. The evaluation unit may be a single component or distributed across multiple components. For example, parts of the method may be implemented on a server and other parts are executed on a local processor in a vehicle.

Furthermore, the device has at least one storage unit that is communicatively connected to the evaluation unit. For example, the computer program product described above can be stored on the storage unit. It is also possible for data to be stored in databases.

The storage unit can be a single component or distributed across multiple components. For example, some data can be stored locally in the vehicle, while others can be stored on a server.

Furthermore, the device has at least one sensor which is communicatively connected to the evaluation unit, wherein the sensor generates sensor data in which the object is detected, wherein the at least one sensor is arranged on a vehicle.

In an advantageous embodiment, at least one sensor is a lidar, a radar, or a camera. These sensors are readily and inexpensively available and are usually already installed in many vehicles for other functions.

According to a sixth aspect of the invention, a vehicle comprises a device as described above.

Summary of the characters

• 1: Eine Fahrszene zur Beschreibung des der Erfindung zugrunde liegenden Problems;
• 2: Ein erstes Ausführungsbeispiel eines Verfahrens zur Relationalen Speicherung von Objekt-Eigenschaften;
• 3: Ein zweites Ausführungsbeispiel eines Verfahrens zur Relationalen Speicherung von Objekt-Eigenschaften;
• 4: Ein Ausführungsbeispiels eines Verfahrens zum Trainieren einer künstlichen Intelligenz;
• 5: Ein Ausführungsbeispiel eines Verfahrens zur Erzeugung einer ein Objekt betreffenden Klasse; und
• 6: Eine Vorrichtung zur Klassifizierung eines Objektes.

The invention is explained in more detail below using exemplary embodiments with the aid of figures. The figures show:

• 1 : A driving scene to describe the problem underlying the invention;
• 2 : A first embodiment of a method for relational storage of object properties;
• 3 : A second embodiment of a method for relational storage of object properties;
• 4 : An embodiment of a method for training an artificial intelligence;
• 5 : An embodiment of a method for creating a class relating to an object; and
• 6 : A device for classifying an object.

Detailed description of the characters

1 shows a driving scene to describe the problem underlying the invention.

1 shows the driving scene from the perspective of a driver of a vehicle 100. In front of the vehicle 100 on the road are a first road user 102 and a second road user 104. The first road user 102 is a carriage, the second road user 104 is a motor vehicle.

The vehicle 100 has a camera 106, in whose field of view 108 objects 102, 104 on the road in front of the vehicle 100 can be detected. The vehicle 100 also has a display device 110. An algorithm has recognized and classified the road users 102, 104 from image data 172 of the camera 106 using an object recognition and classification algorithm. Furthermore, a digital model of the road users 102, 104 was created with a first digital image 112 and a second digital image 114, which are displayed on the display device 110.

However, the first road user 102 was incorrectly classified. Instead of a carriage, the first digital image 112 displayed is a truck because the classification algorithm was not trained accordingly. This could confuse the driver or cause an automated vehicle to perform a risky maneuver.

2 shows a first embodiment of a method for relational storage of object properties 116 of an object 102,104.

In a first obtaining step 118, sensor data 172,174,176,178 are obtained in which the object 102,104 is detected.

In a detection and classification step 120, the object 102, 104 is detected, localized, and classified in the sensor data 172, 174, 176, 178. This step can be performed using conventional image processing methods. Typically, a neural network (NN) is trained with corresponding data and a predefined class.

In this step, object properties such as geometric patterns and their distribution are determined for the object 102,104. Furthermore, a class is assigned to the object 102,104, for example, "pedestrian,""bicycle,""tree,""motorvehicle,""truck," and the like. A confidence value is generated, which is a measure This indicates how confident the classification into the class is. The confidence value is typically expressed as a floating-point number such as 0.97 or as a percentage value such as 97%.

In a first decision step 122, it is checked whether the confidence value is below a first threshold 192. If this is not the case (f-branch), a reliable classification is present, and the process 116 simply continues. However, if the confidence value is less than the first threshold 192, a reliable classification is not present. The object 102,104 is considered unknown.

In this case (t-branch), a first prompting step 124 is performed. In this step, an occupant of the vehicle 100 is prompted to provide a second class relating to the object 102, 104. The prompt can be made, for example, via an audio signal in which the occupant is asked, "What type of object is that in front of the vehicle?" The second class is the correct class to which the object 102, 104 belongs.

The occupant's feedback is received in a first receiving step 126. For example, this can again be done via an audio signal. The occupant can, for example, respond, "This is a carriage."

In a first extraction step 128, the second class is extracted from the occupant's feedback. For example, the word "carriage" is isolated from the audio signal and used as the second class.

In a first storage step, a data set 193 is stored, which includes at least the second class, at least one object property, and a derivative of the sensor data 172, 174, 176, 178. A derivative can be the sensor data 172, 174, 176, 178 itself, but it can also be a compressed version or a relevant part of the sensor data 172, 174, 176, 178.

3 shows a second embodiment of a method for relational storage of object properties 116.

Most steps in 3 (see dashed line) are already in 2 explained and will therefore not be discussed again here. In addition, 3 but further steps that will now be explained.

If it is determined in the first decision step 122 that the confidence value is above the first threshold 192, it can be assumed that the object 102, 104 has been classified with sufficient certainty. For example, the first threshold can be 0.95 or 95%.

Subsequently, in a second decision step 134, it is checked whether the confidence value is above a second threshold 194. For example, this is 0.99 or 99%. If this is the case (t-branch), then the object 102, 104 is very reliably classified into the first class. The subsequent steps 136, 138, 140, and 142 are intended to ensure that the method 116 is not intentionally misused by generating incorrect classifications.

To do this, the passenger is first prompted to confirm the first class in a second prompt step 136. This can be implemented, for example, via an audio signal in which the passenger is asked, "Is there a carriage in front of us?"

Subsequently, a second response from the occupant is received in a second receiving step 138. The response could be, for example, "Yes, that is a carriage" or "No, that is a bicycle."

Therefore, in a second extraction step 140, the second feedback from the occupant is subsequently evaluated and assessed to determine whether it is consent or not. This is represented by an indicator, for example, a Boolean variable. The indicator thus indicates a truth value indicating whether the object 102,104 belongs to the first class. The indicator can, for example, have binary states (true/false) or exhibit a value along a spectrum. The indicator therefore stores whether the occupant provides trustworthy answers, because it is to be expected that a positive answer will be given given the high level of confidence.

The indicator is then stored in a second storage step 142. The storage occurs in such a way that the indicator is linked to the identifier of the occupant or the car. In other words, information is generated that links the occupant's response to their identity.

Subsequently, the training step 132 can be designed more specifically by only including data sets 193 of inmates in the training phase of the artificial intelligence if the inmate is classified as trustworthy via the indicator.

4 shows an embodiment of a method for training an artificial intelligence 144.

In a first implementation step 146, the method is 3 carried out continuously (indicated by the circular arrow) for a large number of occupants or vehicles 100. In the process, a corresponding number of indicators are generated and stored.

In a first calculation step 148, a count is calculated for the plurality of indicators. The count indicates how often an inmate has given a trustworthy answer, since the indicator is only generated for objects 102, 104 that have been classified with very high certainty. The count is therefore a measure of the inmate's trustworthiness.

Subsequently, the number is compared with a third threshold 196 in a third decision step 150. If the number is lower than the third threshold 196 (f-branch), all data records 193 originating from the occupant or vehicle 100 can be used to train the artificial network in training step 132. If the number is higher than the third threshold 196, the data records 193 associated with the occupant or vehicle 100 are excluded from training step 132. The third threshold 196 is thus a measure of the extent to which an occupant or vehicle 100 should be considered trustworthy.

5 : An embodiment of a method for creating a class 152 relating to an object.

In a second implementation step 154, the method is 2 carried out continuously (illustrated by the circular arrow) for a large number of occupants or vehicles 100. In the process, a corresponding number of data records 193 are also generated and stored.

In a classification step 156, the data records 193 are divided into groups. The members of a group are characterized by a similarity in their object properties and/or their second classes. For example, a specific arrangement (e.g., distances) of points, lines, or corners can establish a similarity.

In a second calculation step 158, a count is then calculated for each group. The count is thus a measure of how many different occupants or vehicles 100 have transmitted a matching classification for an object 102, 104.

Subsequently, in a fourth decision step 160, it is checked whether the number exceeds a fourth threshold 198. The fourth threshold 198 is a measure of whether sufficient similar feedback is available regarding a class of an object 102, 104. If this is the case (t-branch), a generation step 162 is executed.

In the first generation step 162, a third class is created for the data records in the groups. The third class may, in particular, correspond to the second class, i.e., the class transmitted by the occupants is adopted.

In an assignment step 164, the third class is assigned to the data records 193. Thus, the data records 193 are now provided with a reliable class.

The artificial intelligence can then be trained using the data sets 193 provided with the third class. The sensor data 172, 174, 176, 178 (or a derivative) present in the data sets 193, or the object properties, or a combination thereof, serve as input for the artificial intelligence, with the class present in the data sets 193 serving as the default for the output. In training step 132, the internal properties of the artificial intelligence, for example, weights between artificial neurons, can be adjusted.

6 shows a device 166 for classifying an object 102,104.

The device 166 includes a camera 106, a lidar sensor 168, and a radar sensor 170. The camera 106 generates image data 172, the lidar sensor 168 generates lidar data 174, and the radar sensor 170 generates radar data 176.

The sensor data 172,174,176 generated by these sensors 106,168,170 are very well suited to detecting objects 102,104 because they contain object-characteristic patterns.

Before the sensor data 172,174,176 are further processed, they are in the example of 6 fused to fused sensor data 178. These are then forwarded to a central computer (HPC; English: high performance computer) 180. The HPC 180 has an evaluation unit 182, which in the example of 6 is implemented as a processor 182. In addition, the HPC 180 has a non-volatile memory 184, which is communicatively connected to the processor 182 with the participation of a volatile memory 200.

A computer program product 186 is stored on the non-volatile memory 184, which has instructions which, when executed by the processor 182, cause the processor 182 to perform the method ren 116,144,152 from the 3 to 5 Furthermore, a first database 188 is stored on the non-volatile memory 184, in which, among other things, the fused sensor data 178 can be stored. In addition, a first neural network (NN) 190 is stored on the non-volatile memory 184, which has been trained to detect and classify objects 102, 104. Furthermore, a first threshold value 192 and a second threshold value 194 are stored on the non-volatile memory 184, which are used for executing the methods 116, 144, 152 from the 3 to 5 are necessary.

The device 166 further comprises an external memory 202, which is implemented as a cloud storage 202. This memory comprises a second neural network 204 and a second database 206. Furthermore, a third threshold value 196 and a fourth threshold value 198 are stored on the cloud storage 202, which are used for executing the methods 116, 144, 152 from the 3 to 5 are necessary.

The device 166 is communicatively connected to an audio system 208 of a vehicle 100 in a bidirectional manner.

The computer program product 186 is now executed on the processor 182. The fused sensor data 178 is evaluated by the first NN 188. The object 102, 104 is detected and classified in the fused sensor data 178. Object properties such as geometric patterns are recognized in the fused sensor data 178. The object properties are characteristic of a first class, which is assigned to the object 102, 104. The first NN 188 implicitly learned these characteristics in a previous training phase. The first NN 188 also generates a confidence value 191, which indicates how reliable the classification into the first class was.

A check is now carried out to determine whether the confidence value 191 is below the first threshold value 192, for example, 0.95 or 95%. If this is the case, no reliable classification is available, and an occupant of the vehicle 100 is prompted via the audio system 208 to provide a second class by asking "What kind of object is that in front of the vehicle?" The response is received, and the second class is extracted from it. If the occupant answers, for example, "That's a carriage," "carriage" is extracted as the second class. The second class is then stored as a data record 193 in the first database 186 together with the object properties, the fused sensor data 178, and an identifier. The identifier is a specific value assigned to each occupant or vehicle 100. It ensures that data records 193 can be assigned to a specific source. This is preferably done anonymously.

However, if the confidence value 191 is above the second threshold 194, for example 0.99 or 99%, then a very confident classification is present and the occupant is prompted to provide confirmation by asking "Is this a truck?" If the occupant answers, for example, "Yes, this is a truck" or "No, this is not a truck" or "This is an airplane," an indicator 195 is extracted from this feedback. The indicator 195 is used in the example of 6 as a binary Boolean variable, i.e., with the values "true" or "false." Then, indicator 195 is saved together with the underlying data set 193. It can be added to data set 193, or a reference to data set 193 can be appended to indicator 195.

The data records 193 and indicators 195 in the first database 186 are added wirelessly to the second database 206 in the cloud 202. A variety of feedback from a variety of occupants or vehicles 100 is collected in the cloud 202.

In the cloud 202, a check is now carried out to determine how many data sets 195 of an occupant or a vehicle 100 have an indicator 195 indicating negative feedback. Negative feedback indicates that the occupant or the vehicle 100 is not providing reliable answers, since the classification of the object 102, 104 was very likely correct. If the number of these data sets 195 is above the third threshold, for example, "10," these data sets 195 are not used to train the second NN 204.

Furthermore, grouping is performed in the cloud 202. Data records 193 are assigned to a group if they have similar second classes and similar object properties, i.e., if the plurality of occupants or vehicles 100 have made a matching classification. If the number of data records in a group exceeds a fourth threshold, for example, "20," it can be assumed that the object properties in the data records 193 actually correspond to the second class of the data records 193.

Therefore, a third class is created for the group, which in the example of 6 corresponds to the second. However, the third class can also differ from the second, for example because different languages are used for the second class. were used. The third class is assigned to the corresponding members in the group.

The second NN 204 is now trained with the data sets 193, as long as they were not excluded from training due to an excessive number of indicators. After the training phase, the second NN 204 has implicitly learned to recognize the new third class and can be transmitted back to the vehicle 100. In other words, the first NN 188 is replaced by the second NN 204, or the internal properties such as weights are transmitted to the first NN 188 for updating. The vehicle 100 can now recognize the new, third class and, based on this, create better planning of driving maneuvers or generate more precise representations of digital objects 112, 114 for a display device 110 of the vehicle 100.

List of reference symbols

100

vehicle

102

First road user; carriage

104

Second road user; car

106

camera

108

Field of view

110

Display device

112

First digital image

114

Second digital image

116

Methods for relational storage of object properties

118

First conservation step

120

Detection and classification step

122

First decision step

124

First prompt step

126

First reception step

128

First extraction step

130

First storage step

132

Training step

134

Second decision step

136

Second prompt step

138

Second reception step

140

Second extraction step

142

Second storage step

144

Method for training artificial intelligence

146

First implementation step

148

First calculation step

150

Third decision step

152

Method for creating a class relating to an object

154

Second implementation step

156

Classification step

158

Second calculation step

160

Fourth decision step

162

Generation step

164

Assignment step

166

device

168

Lidar sensor

170

radar sensor

172

Image data

174

Lidar data

176

Radar data

178

Fused sensor data

180

central computer; HPC

182

evaluation unit; processor

184

Non-volatile memory

186

computer program product

188

First database

190

First neural network

191

Confidence value

192

First threshold

193

data set

194

Second threshold

195

indicator

196

Third threshold

198

Fourth threshold

200

Volatile memory

202

External storage; Cloud

204

Second neural network

206

Second database

208

audio system

Claims (7)

Computer-implemented method for training an artificial intelligence (144), comprising the steps of: a) obtaining (118) sensor data (172, 174, 176, 178) in which an object (102, 104) is detected, wherein the sensor data (172, 174, 176, 178) are generated by at least one sensor (106, 168, 170) arranged on a vehicle (100), b) detection and classification (120) of the object (102, 104) using the sensor data (172, 174, 176, 178), wherein a first class is assigned to the object (102, 104) and at least one object property is determined, thereby generating a confidence value (191), c) requesting (124) an occupant of the vehicle (100) to provide a second class relating to the object (102, 104) if the confidence value (191) is below a first threshold value (192), d) receiving (126) a first feedback from the occupant, e) extracting (128) the second class from the first feedback from the occupant, and f) storing (130) a data set (193) which comprises at least the second class, the at least one object property, a derivative of the sensor data (172, 174, 176, 178) and an identifier of the occupant and/or the vehicle (100), wherein the method (116) comprises the following further steps if the confidence value (191) is above a second threshold value (194): g) requesting (136) the occupant of the vehicle (100) to confirm the first class of the object (102, 104), h) receiving (138) a second feedback from the occupant, i) extracting (140) an indicator (195) from the second feedback from the occupant, wherein the indicators (195) indicate truth values as to whether the object (102, 104) belongs to the first class, j) storing (142) the indicator (195), k) carrying out (146) steps g) to j) for a plurality of occupants, thereby generating a plurality of indicators (195) which are stored, l) calculating (148) a number of indicators (195) for each occupant or for each vehicle (100), and m) training (132) the artificial intelligence (190, 204) at least with the derivative of the sensor data (172, 174, 176, 178) or the object properties as input and with the second class as output, characterized in that only those data sets (193) are used for the training for whose identifier the number of indicators (195) is below a third threshold value (196). Computer program product (186), comprising instructions which, when executed by a computer, cause the computer to carry out a method (116, 144, 152) according to Claim 1 to execute. Computer-readable storage medium (184) comprising instructions which, when the program is executed by a computer, cause the computer to perform a method (116, 144, 152) according to Claim 1 to execute. Data carrier signal which the computer program product (186) Claim 2 transmits. Device for classifying an object (166), comprising: a) an evaluation unit (182) which is configured such that it carries out a method (116, 144, 152) according to Claim 1 can carry out, b) at least one memory unit (184,202) which is communicatively connected to the evaluation unit (182), and c) at least one sensor (106,168,170) which is communicatively connected to the evaluation unit (182), wherein the at least one sensor (106,168,170) generates sensor data (172,174,176,178) in which the object (102,104) is detected, wherein the at least one sensor (106,168,170) is arranged on a vehicle (100). Device according to Claim 5 , characterized in that the at least one sensor (106,168,170) is a lidar (168), a radar (170) or a camera (106). Vehicle (100) which has a device (166) according to Claim 5 or 6 has.

Images