CN113971814A - Data processing method and device, electronic equipment and computer storage medium - Google Patents

Abstract

Embodiments of the present invention provide a data processing method, apparatus, electronic device, and computer storage medium. The data processing method includes: obtaining a first data model and a sample image, wherein the first data model includes a machine learning model; and obtaining a multi-modality corresponding to the sample image at least according to the spatiotemporal information of the sample image Supervision data; train the first data model according to at least the sample image and the multimodal supervision data to obtain a second data model. The data processing method can improve the training effect of the data model.

Description

Data processing method, device, electronic device and computer storage medium

technical field

Embodiments of the present invention relate to the field of computer technologies, and in particular, to a data processing method, apparatus, electronic device, and computer storage medium.

Background technique

With the continuous development of artificial intelligence and machine learning technology, the application of machine learning technology in daily life is more and more extensive. Neural network models trained using machine learning techniques can perform some everyday tasks quickly and efficiently. For example, in the field of pedestrian re-identification in video surveillance, the surveillance video can be processed through a trained pedestrian re-identification model, so as to realize functions such as tracking the movement trajectory of a specific person. However, the application scenarios of existing pedestrian re-identification models are relatively limited. For example, a pedestrian re-identification model trained for scene A cannot be directly transferred to scene B.

In order to solve the above problems, in the prior art, the sample images used for scene B training are usually manually labeled to obtain a batch of supervised data for training, and then these manually labeled supervised data are used to train pedestrians suitable for scene B. Re-identify the model. That is to say, there is a common problem in the prior art that machine learning models need to manually label supervised data, resulting in high training costs, high difficulty, and difficulty in generalization.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present invention provide a data processing solution to at least partially solve the above problems.

According to a first aspect of the embodiments of the present invention, a data processing method is provided, including: acquiring a first data model and a sample image, wherein the first data model includes a machine learning model; information to obtain multi-modal supervision data corresponding to the sample image; train the first data model according to at least the sample image and the multi-modal supervision data to obtain a second data model.

According to a second aspect of the embodiments of the present invention, there is provided a data processing method, comprising: obtaining multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image; The multimodal supervision data is used to train the data model to be trained, and the target data model is obtained, wherein the data model to be trained includes a machine learning model.

According to a third aspect of the embodiments of the present invention, a data processing method is provided, including: receiving a model training request for requesting training of a first data model sent by a client by calling a preset training interface; The model training request is obtained, and a sample image and a first data model for training are obtained; the multimodal supervision data corresponding to the sample image is obtained by the data processing method according to the first aspect, and the sample image and the multimodal supervision data are obtained using the sample image and the multimodal supervision data. The modal supervision data trains the first data model.

According to a fourth aspect of the embodiments of the present invention, a data processing apparatus is provided, including: a first acquisition module for acquiring a first data model and a sample image, wherein the first data model includes a machine learning model; The second acquisition module is used to obtain the multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image; the training module is used to obtain the multimodal supervision data corresponding to the sample image at least according to the sample image and the multimodal supervision data, The first data model is trained to obtain a second data model.

According to a fifth aspect of the embodiments of the present invention, there is provided a data processing apparatus, comprising: a third acquisition module, configured to acquire multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image; second A training module, configured to train a data model to be trained according to at least the sample image and the multimodal supervision data, and obtain a target data model, wherein the data model to be trained includes a machine learning model.

According to a sixth aspect of the embodiments of the present invention, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used for storing at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the data processing methods described in the first aspect, the second aspect or the third aspect.

According to a seventh aspect of the embodiments of the present invention, there is provided a computer storage medium on which a computer program is stored, and when the program is executed by a processor, implements the data processing described in the first aspect, the second aspect or the third aspect method.

According to the data processing solution provided by the embodiment of the present invention, the multimodal supervision data is mined from the sample images according to the spatiotemporal information of the sample images, etc., so that the supervision data used for training the first data model does not need to be manually marked. This reduces the difficulty and labor cost in the process of transferring the data model from the source scene to the target scene, and realizes the purpose of training the data model without manually labeling the sample images of the target scene. It uses multimodal supervised data, which can improve the training effect of the data model.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments described in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

FIG. 1a is a flow chart of the steps of a data processing method according to Embodiment 1 of the present invention;

Fig. 1b is a schematic diagram of an example of a scenario in the embodiment shown in Fig. 1a;

Fig. 1c is a schematic diagram of a usage scenario in the embodiment shown in Fig. 1a;

2 is a flow chart of steps of a data processing method according to Embodiment 2 of the present invention;

3 is a flowchart of steps of a data processing method according to Embodiment 3 of the present invention;

Fig. 4a is a flow chart of the steps of a data processing method according to Embodiment 4 of the present invention;

Fig. 4b is a schematic diagram of an example of a scenario in the embodiment shown in Fig. 4a;

5 is a flowchart of steps of a data processing method according to Embodiment 5 of the present invention;

Fig. 6a is a flow chart of the steps of a data processing method according to Embodiment 6 of the present invention;

Fig. 6b is a schematic diagram of a connection between a SaaS platform and a client according to Embodiment 6 of the present invention;

7 is a structural block diagram of a data processing apparatus according to Embodiment 7 of the present invention;

8 is a structural block diagram of a data processing apparatus according to Embodiment 8 of the present invention;

FIG. 9 is a schematic structural diagram of an electronic device according to Embodiment 9 of the present invention.

Detailed ways

In order to make those skilled in the art better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. The embodiments described above are only a part of the embodiments of the present invention, but not all of the embodiments. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the embodiments of the present invention should fall within the protection scope of the embodiments of the present invention.

The specific implementation of the embodiments of the present invention is further described below with reference to the accompanying drawings of the embodiments of the present invention.

Example 1

Referring to FIG. 1 , a flowchart of steps of a data processing method according to Embodiment 1 of the present invention is shown.

In this embodiment, the data processing method can be applied to the training field of the pedestrian re-identification model, so that the pedestrian re-identification model of the source scene can be trained without manually labeling the sample images in the target scene, so that the It can be used in the target scene. The data processing method can be configured on a server (the server includes a server and a cloud) and executed by the server.

Of course, in other embodiments, the data processing method can be applied to other appropriate fields, such as applied to the training field of other neural network models for image processing. Moreover, the data processing method may be executed by any suitable execution subject, such as a terminal device, etc., which is not limited in this embodiment.

Step S102: Obtain a first data model and a sample image.

In the process of reusing the first data model, the source scene may refer to the scene where the training sample used when the first data model is initially trained. For example, if the first data model A is a pedestrian re-identification model trained using surveillance images of city a, the source scene is city a.

Compared with the source scene, the target scene refers to the scene to which the first data model needs to be transferred. If the first data model A needs to be used to re-identify pedestrians in the surveillance video of city b, the target scene is city b. Because the architectural style, clothing style and other factors of city b are different from those of city a, the first data model A is directly transferred to the target scene, and its use effect cannot meet the needs.

The first data model may be any suitable machine learning model (eg, a neural network model). For example, the first data model may be the aforementioned pedestrian re-identification model, or a neural network model for face recognition, or the like. The pedestrian re-identification model refers to a neural network model that can realize the function of retrieving surveillance video for specific pedestrians.

For the first data models with different functions, different types of images may be used as sample images. For example, if the first data model is a pedestrian re-identification model, the sample images may be surveillance images of surveillance videos. The surveillance images include the collected human body images of pedestrians.

Step S104: Obtain multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image.

The spatiotemporal information includes at least the acquisition time and acquisition location of the sample image. For example, when collecting surveillance images through surveillance cameras, record the time stamp of each surveillance image (that is, the acquisition time), and record the latitude and longitude information of each surveillance camera, which is the latitude and longitude information of all surveillance images collected by the surveillance camera. Collection location.

In multimodal supervised data, multimodality refers to the combination of information from multiple modalities, such as the combination of face information and spatiotemporal information. The multimodal supervision data includes first supervision data and/or second supervision data. The first supervised data may be strongly supervised supervised data, and strong supervision means that the accuracy of pseudo-labels established based on multimodal information of these supervised data is very high (eg, 100% or approximately 100%). The second supervised data may be weakly supervised supervised data. Weak supervision means that the accuracy of pseudo-labels established based on multimodal information is relatively low (usually less than 100% or does not meet the condition of approximately 100%).

Those skilled in the art can obtain multimodal supervision data in any suitable manner as required.

Taking the acquisition of strongly supervised multimodal supervised data as an example, the acquisition process is as follows:

For the target sample image Q in the sample image, according to the face information in each sample image, determine the sample image W that contains the same face as the target sample image Q, because the sample image W and the target sample image Q contain the same face, so It can be considered that the sample image W and the target sample image Q contain the same person, so the pseudo-label of the sample image W is a positive sample.

According to the spatiotemporal information of the sample image, determine the sample image E with the same time as the target sample image Q and the distance between the collection positions is greater than the set distance threshold (the set distance threshold can be determined according to needs). It may appear in two places, so it can be considered that the sample image E and the target sample image Q cannot contain the same person, so the pseudo-label of the sample image E is a negative sample, so that strong supervision can be obtained according to the sample image W and the sample image E. Multimodal Supervision Data.

Of course, in other embodiments, the multimodal supervision data may be obtained in other manners, which is not limited in this embodiment.

Step S106: Train the first data model according to at least the sample image and the multimodal supervision data to obtain a second data model.

After obtaining the multimodal supervision data, use the sample images to train the first data model. During the training process, the multimodal supervision data corresponding to the sample images is used as supervision to calculate the loss value, and the first data model is calculated according to the loss value. The parameters in the model are updated to obtain a second data model suitable for the target scene.

The following describes the implementation process of the data processing method in combination with a specific usage scenario:

In this usage scenario, the first data model is a pedestrian re-identification model, and the sample image corresponding to the target scene is a surveillance image in a surveillance video of city b for illustration.

In this usage scenario, the surveillance images in the surveillance video obtained from city b are used as sample images. The first data model may be a pedestrian re-identification model trained based on surveillance images of city a.

As shown in Figure 1b and Figure 1c, in this usage scenario, the data processing method is configured on the server, and the server obtains the first data model and the sample image.

For the sample image, at least according to its spatiotemporal information, the multimodal supervision data of the sample image is obtained.

like,

The total number of sample images is N. For the ith sample image I _i , according to the face information of the sample image I _i , it is determined that the remaining N-1 sample images except the sample image I _i contain the same as the sample image I _i . A sample image of a human face (denoted as I _j ), since the human body images in the sample images containing the same face are _likely to be the same person, the sample image I _j is a positive sample _. The loss function of the output feature information should be small.

由于一个人不可能在同一时刻出现在相距较远(如1公里)的两个位置，因此，相距距离较远的两个样本图像中不会包括相同人，故而两者组成一个负样本对，互为负样本。In addition to obtaining positive samples, it can also be determined according to the collection time and collection position of the sample image I _i that the collection time is the same, but the distance between the collection positions is greater than the set distance threshold.

Since it is impossible for a person to appear at two locations that are far apart (such as 1 km) at the same time, the two sample images that are far apart will not include the same person, so the two form a negative sample pair, are negative samples of each other.

Multi-modal supervision data is formed by positive samples and negative samples, and the first data model can be trained by using the sample images and multi-modal supervision data to obtain a second data model, which can be applied to the monitoring of city b Pedestrian re-identification of images.

The trained second data model can input different test images into the data model. If the two test images contain human body images of the same person, the feature information corresponding to the two output test images will be closer in space. In this way, during pedestrian re-identification, the image of a specific object is input into the data model to obtain its characteristic information, and then the characteristic information of other monitoring images is retrieved by using the characteristic information, and then the monitoring image containing the specific object can be obtained. Find out the movement trajectory of a specific object.

With this embodiment, the multimodal supervision data is mined from the sample images according to the spatiotemporal information of the sample images, so that the supervision data used for training the first data model does not need to be manually marked, thereby reducing the need for data model The difficulty and labor cost in the process of transferring from the source scene to the target scene realizes the purpose of training the data model without manual annotation of the sample images, and because multi-modal supervision data is used during training, Therefore, the effect of training the data model can be improved.

Embodiment 2

Referring to FIG. 2 , a flowchart of steps of a data processing method according to Embodiment 2 of the present invention is shown.

In this embodiment, the data processing method includes the aforementioned steps S102 to S106. Wherein, in this embodiment, an implementation manner of acquiring the first supervision data in the multimodal supervision data is exemplified, and step S104 includes sub-step S1041 and sub-step S1042.

Sub-step S1041: Perform face recognition on the sample image to obtain face information corresponding to the sample image.

For example, face recognition is performed on all the sample images using any existing suitable face recognition technology, and the face information of each sample image is obtained.

Sub-step S1042: Determine the first supervision data corresponding to the sample image according to the face information and the spatiotemporal information of the sample image.

In this embodiment, the first supervision data is strongly supervised multimodal supervision data.

In a specific implementation, sub-step S1042 includes:

Sub-step I: According to the face information of the sample images, a positive sample set composed of sample images containing the same face is determined.

For example, according to the identified face information, determine the sample images belonging to the same person in the sample images, any two of these sample images can form a positive sample pair, denoted as

其中

表示第k对正样本对,I_i表示第i张样本图像，I_j表示第j张样本图像。For example, the sample image I _i and the sample image I _j both include the face A, then the two form a positive sample pair, denoted as

in

Represents the k-th positive sample pair, I _i represents the i-th sample image, and I _j represents the j-th sample image.

其中

表示第k对正样本对,I_n表示第n张样本图像，I_m表示第m张样本图像。Both the sample image I _n and the sample image I _m include the face B, then the two form a positive sample pair, denoted as

in

represents the kth positive sample pair, In represents the _nth sample image, and Im represents the _mth sample image.

These pairs of positive samples form a set of positive samples.

Sub-step II: According to the spatiotemporal information of the sample images, determine a negative sample set composed of sample images that satisfy the set condition.

In a specific implementation, sub-step II can be implemented as: determining a target sample image from the sample image; The sample images whose distances are greater than the set distance threshold are determined as negative sample images; a negative sample set corresponding to the target sample image is determined according to the negative sample images.

The target sample image may be any sample image in the sample images, that is, sub-step II may be performed for each sample image, thereby obtaining a negative sample pair related to it, and further obtaining a negative sample set.

In this embodiment, each sample image has corresponding spatiotemporal information, and the spatiotemporal information includes acquisition time and acquisition location. The acquisition time can be determined from the time stamps of the sample images. The collection position can be determined according to the surveillance camera that collects the sample image. For example, each surveillance camera has a corresponding number and latitude and longitude information.

For the target sample image, the longitude and latitude information of the target monitoring camera can be determined according to the number of the target monitoring camera collected, and then the monitoring cameras whose distance from the target monitoring camera is greater than the set distance threshold (such as one kilometer) can be found. The distance between the collected sample image (denoted as the sample image in set1 for the convenience of description) and the target sample image is greater than the set distance threshold.

这些样本图像包括的人与目标样本图像包含的人不同，因此为目标样本图像的负样本，目标样本图像与任意一个负样本可以组成一个负样本对记作

其可以表示为

其中，

表示第k个负样本对，I_i表示目标样本图像，

表示第n个负样本。Further, from the sample images in set1, it is determined that the sample images whose acquisition time is the same as the acquisition time of the target sample image are recorded as

The people included in these sample images are different from those included in the target sample image, so they are negative samples of the target sample image. The target sample image and any negative sample can form a negative sample pair, which is recorded as

It can be expressed as

in,

represents the kth negative sample pair, I _i represents the target sample image,

represents the nth negative sample.

Since the moving speed of people is limited, an appropriate distance threshold can be set for different moving modes of people (such as cycling, walking, running, etc.), so as to ensure that the negative samples found are different from the people included in the target sample images. .

A set of negative samples corresponding to the target sample image can be determined according to the negative sample pair of each target sample image.

Sub-step III: Determine the first supervision data corresponding to the sample image according to the positive sample set and the negative sample set.

In this embodiment, in order to ensure that after using the data model trained by the first supervision data, when the data model is used, after inputting the human body image of the same person into the data model, the output feature information is relatively similar in space, so that To achieve the tracking of a specific person, for a sample image, use the sample image, a positive sample in the positive sample set of the sample image, and a negative sample in the negative sample set in the sample image to form a corresponding The first monitoring data.

其中，I_j是正样本，

是负样本。因为，这些正样本对和负样本对的可靠度都非常高，准确率接近100％，而且在活动正样本对和负样本对的过程中，除了使用样本图像中的人体图像信息外，还使用了人脸信息和时空信息，所以第一监督数据可以称之为强监督多模态监督数据。For example, for the sample image I _i , the corresponding first supervision data can be expressed as

where I _j is a positive sample,

is a negative sample. Because, the reliability of these positive sample pairs and negative sample pairs is very high, and the accuracy rate is close to 100%, and in the process of active positive sample pairs and negative sample pairs, in addition to using the human image information in the sample images, also use face information and spatiotemporal information, so the first supervision data can be called strong supervision multimodal supervision data.

During training, the triplet loss loss function can be used to train the data model using the first supervised data.

With this embodiment, the multimodal supervision data is mined from the sample images according to the spatiotemporal information of the sample images, so that the supervision data used for training the first data model does not need to be manually marked, thereby reducing the need for data model The difficulty and labor cost in the process of transferring from the source scene to the target scene realizes the purpose of training the data model without manual annotation of the sample images, and because multi-modal supervision data is used during training, Therefore, the effect of training the data model can be improved.

In addition, the use of face information and spatiotemporal information ensures the accuracy of positive samples and negative samples, and ensures that the first supervision data is strong supervision data with an accuracy rate close to 100%, thereby improving the training effect of the data model.

Embodiment 3

Referring to FIG. 3 , a flowchart of steps of a data processing method according to Embodiment 3 of the present invention is shown.

In this embodiment, in this embodiment, the data processing method includes the aforementioned steps S102 to S106. Wherein, step S104 may adopt the aforementioned implementation manner. In this embodiment, an implementation manner of obtaining the second supervision data in the multimodal supervision data is exemplified, that is, obtaining the second supervision data in the step S104 through sub-steps S1043 to S1045 included in step S104. The second monitoring data.

It should be noted that, in this embodiment, step S104 may only include sub-steps S1043 to S1045. Alternatively, sub-steps S1041 to S1045 may also be included. In this case, sub-steps S1043 to S1045 may be executed before, after, or in parallel with sub-steps S1041 and S1042.

Sub-step S1043: Use the first data model to perform feature extraction on the sample image, and obtain feature information for characterizing human body image information.

The first data model may be the aforementioned data model pre-trained in the source scene. Alternatively, it can also be a data model that completes one or more training cycles in the target scene. For example, for the first data model A, in the first training cycle, after it is trained with N sample images, it is considered that the first data model A has completed the first training cycle. For the convenience of description, the first training cycle will be completed. The data model of the training cycle is denoted as A'. In the second training cycle, the training object becomes the first data model A'. That is, in different training cycles, the first data models for feature extraction on the sample images may be different, and may be the data models trained in the previous training cycle.

Sub-step S1044: Sort the feature information to be sorted according to at least the spatiotemporal information of the sample image and the similarity between the feature information and the feature to be sorted, and obtain a target sorting result.

In a specific implementation, sub-step S1044 includes the following sub-steps:

Sub-step A: Determine the target sample image from the sample image.

The target sample image can be any sample image. For each sample image, the corresponding target ranking result can be obtained through sub-step B.

Sub-step B: According to the similarity between the feature information of the target sample image and the feature to be sorted, sort the feature information to be sorted to obtain an initial sorting result.

The feature to be sorted may be feature information of the remaining sample images except the target sample image. For example, there are N sample images, and the ith sample image is the target sample image, and the remaining N-1 sample images are the remaining sample images.

For the target sample image, the similarity can be determined by calculating the distance between the target sample image and each remaining sample image, and then the features to be sorted can be sorted according to the similarity to obtain an initial sorting result. For example, the initial sorting result is that the similarity is sorted from high to low, and the higher the similarity, the higher the probability that the remaining sample image and the target sample image corresponding to the feature to be sorted include the same person.

Certainly, in other embodiments, the features to be sorted may also be sorted in other manners, which is not limited in this embodiment.

Sub-step C: Adjust the initial sorting result at least according to the spatiotemporal information of the target sample image to obtain the target sorting result.

In a specific implementation, sub-step C includes the following processes:

Process C1: According to the similarity between the face information of the target sample image and the face information of the sample images corresponding to the features to be sorted, perform a first adjustment on the initial sorting result to obtain a first pre-adjustment sorting result.

The face information can be obtained by performing face recognition on the target sample image and the remaining sample images. The higher the similarity between the face information of the target sample image and the remaining sample images, the higher the probability that the two images contain the same person, so the initial sorting result is first adjusted (such as re-correction and sorting) according to the face information. ), the obtained first pre-adjustment sorting result can be ranked more accurately, and the remaining sample images that contain the same person as the target sample image can be ranked more advanced.

Process C2: According to the spatiotemporal information of the target sample image and the spatiotemporal information of the sample image corresponding to the feature to be sorted, a second adjustment is performed on the first pre-adjustment sorting result to obtain a second pre-adjustment sorting result.

In addition to adjusting the sorting according to the face information, the sorting can also be adjusted according to the space-time information (that is, the second pre-adjustment sorting results are re-corrected and sorted), so that the space-time information can be used to make up for the lack of face information. Improve sorting accuracy. When adjusting according to the spatiotemporal information, the acquisition time can be the same as the acquisition time of the target sample image, but the order of the features to be sorted for the remaining sample images that are far away from the acquisition position of the target sample image can be moved backward, because these remaining samples The probability that the image contains the same person as the target sample image is low.

Process C3: Perform a third adjustment on the second pre-adjustment sorting result according to the human body attribute information of the target sample image and the human body attribute information of the sample image corresponding to the features to be sorted to obtain the target sorting result.

In order to further improve the accuracy of sorting, in addition to adjusting the sorting based on face information and spatiotemporal information, it can also be adjusted based on human attribute information. The human attribute information includes appearance information, such as whether to wear a hat, hat color, clothes color, height, and so on. According to the human attribute information, it is possible to better sort and adjust the to-be-sorted features whose faces are more ambiguous and cannot be accurately determined according to the spatiotemporal information, so that the accuracy of the target sorting result is higher.

For example, for the target sample image, the faces in some of the remaining sample images are relatively blurred, and it is impossible to determine whether the target sample image and the remaining sample images contain the same person based on the face information. When it is also impossible to accurately determine whether the two include the same person, the second pre-adjustment sorting result can be thirdly adjusted according to the human attribute information (that is, the second pre-adjustment sorting result is re-corrected and sorted) to obtain the target sorting result. Make the target sorting result more accurate.

Sub-step S1045: Determine the target sorting result as the second supervision data.

The target ranking result after the ranking adjustment is used as the second supervision data, that is, the ranking label of the target sample image. Since the accuracy of target ranking results is not 100%, it is called weakly supervised multimodal training data.

Subsequently, the ranking loss function can be used to train the data model with these weakly supervised multimodal supervised data as supervision. In this way, when the trained data model is used, the accuracy of the ranking results for the test images output by the data model is higher, that is, the images with a higher probability of containing the same person as the test image are ranked higher, so that the ranking results can be more accurate. , Quickly determine the movement trajectory of a specific person.

With this embodiment, the multimodal supervision data is mined from the sample images according to the spatiotemporal information of the sample images, so that the supervision data used for training the first data model does not need to be manually marked, thereby reducing the need for data model The difficulty and labor cost in the process of transferring from the source scene to the target scene realize the purpose of training the data model without manually labeling the sample images of the target scene. Supervise the data, thus improving the training of the data model.

In addition, the face information and spatiotemporal information are used to adjust the initial sorting results to obtain target sorting results with more accurate sorting. The ordering of the output of the second data model is more accurate when used.

Embodiment 4

Referring to Fig. 4a, a flowchart of steps of a data processing method according to Embodiment 4 of the present invention is shown.

In this embodiment, the data processing method includes the aforementioned steps S102 to S106. Wherein, step S104 may adopt any of the foregoing implementation manners. In this embodiment, before step S106, the method further includes step S104a and step S104b.

Step S104a: Use the first data model to perform feature extraction on the sample image.

It should be noted that this step can be executed at any appropriate timing before step S106, and it can be executed before, after or in parallel with step S104.

The first data model may be the aforementioned data model pre-trained in the source scene. Alternatively, it can also be a data model that completes one or more training cycles in the target scene. For example, for the first data model A, in the first training cycle, after it is trained with N sample images, it is considered that the first data model A has completed the first training cycle. For the convenience of description, the first training cycle will be completed. The data model of the training cycle is denoted as A'. In the second training cycle, the training object becomes the first data model A'. That is to say, in different training cycles, the data models for feature extraction on the sample images may be different, which may be the first data models trained in the previous training cycle.

Step S104b: Perform clustering processing on the feature information of the sample image to obtain the category to which the sample image belongs, and determine the category to be the single-modality supervision data of the sample image.

Those skilled in the art can use any appropriate clustering method for clustering, such as K-means clustering algorithm and the like. The category obtained through clustering processing is the pseudo-label corresponding to the sample image.

For example, for the feature information of N sample images, perform clustering processing on them to obtain K categories, among which, sample images 1, 2, and 5 belong to the same category, and the category to which they belong (denoted as label1). Then the pseudo-label of sample image 1 is label1, and the pseudo-label is the corresponding single-modal supervision data.

Due to the relatively low reliability of feature information and clustering, the accuracy cannot be guaranteed to be 100%, so it is called weakly supervised supervised data.

Subsequently, the single-modal supervision data can be used to train the data model by using the classification loss function.

After the supervised data is obtained, the first data model can be trained by using the supervised data, so that the trained second data model can be adapted to the target scene. In this embodiment, step S106 includes the following sub-steps:

Sub-step S1061: Perform training weight configuration on the first supervision data, the second supervision data and the single-modal supervision data.

In order to improve the training effect, multiple supervised data can act on the first data model at the same time, so that the effect of the trained first data model is better, in this embodiment, each supervised data can be used for training The process is regarded as a sub-training framework, and the three sub-training frameworks are carried out at the same time. The training of the data model is a multi-task learning, and the three sub-training frameworks are used to update the same first data model at the same time.

In order to ensure that different supervised data have different influences on the training results during this training process, training weights are configured for the first supervised data, the second supervised data and the single-modal supervised data. For example, the training weight of the first supervision data is 0.7, the training weight of the second supervision data is 0.2, and the training weight of the single-modal supervision data is 0.1.

Of course, those skilled in the art can configure any appropriate training weights as required, which is not limited in this embodiment.

Sub-step S1062: Using the sample image, the weight-configured first supervision data, the second supervision data, the single-modal supervision data, and the loss function corresponding to the supervision data, analyze the first data The model is trained on multiple tasks.

For example, a training process in a training cycle is: select one supervision data from the first supervision data, the second supervision data and the single-modal supervision data according to the training weight. If the selected supervision data is the first supervision data, the selected supervision data will be selected. The sample image corresponding to the first supervision data is input into the first data model, and the first supervision data is used as supervision, the loss value is calculated using the triplet loss loss function, and the parameters of the first data model are updated according to the loss value. .

After one training is completed, the process of selecting one supervised data from the first supervised data, the second supervised data and the single-modal supervised data according to the training weight may be repeated. This is repeated until one training cycle is completed. The loss function used in each training corresponds to the selected supervision data. If the selected supervision data is the first supervision data, the loss function is the triplet loss loss function; if the selected supervision data is the second supervision data, the loss function is rankingloss Loss function; if the selected supervised data is unimodal supervised data, the loss function is the classification loss function.

The implementation process of data processing is described below in combination with a specific usage scenario:

In this usage scenario, the data model can be trained in the source scenario, and the first data model obtained by training is used as the initial model. Of course, other existing models can also be used, which is not limited in this usage scenario.

For the target scene to be transferred to, obtain the corresponding sample image, such as the surveillance image in the surveillance video.

As shown in Figure 4b, in this usage scenario, in order to avoid the problems of high cost, high difficulty, and difficulty in generalization in the prior art by manually labeling a batch of sample images for training in new scenarios. The data processing method of this use scene uses the initial model trained in the source scene to mine more information from the sample images of the target scene as supervised data to achieve the purpose of automatically obtaining supervised data, and then use these supervised data to train and update the initial model, so as to obtain a data model with better effect on the target scene.

In this usage scenario, three types of information are mined: the first supervised data with strong supervision, the second supervised data with weak supervision, and the unimodal supervision data. Using these three kinds of supervision data to train the first data model overcomes the problem of using the human body images in the sample images to generate certain pseudo-supervised data to help the training without labeling in the prior art. The accuracy of the data model is too low, and it is difficult to meet the actual use requirements. Moreover, it can overcome the problem of using fake target scene sample images for training and training in the prior art to automatically generate fake target scene body images for training the target scene in the prior art.

Since the supervised data obtained by using multi-modal information is more accurate, it helps to improve the effect of the trained data model, which is better than that of single-modal pseudo-label training.

The following describes the process of acquiring supervised data and using supervised data for training in conjunction with Figure 4b:

As shown in Figure 4b, in this usage scenario, the first data model is a pedestrian re-identification model.

After obtaining the sample image of the target scene, the first aspect mines the strongly supervised first supervised data based on it. The supervised data integrates face information and spatiotemporal information, so the obtained first supervised data is strongly supervised multimodal Monitor data.

Specifically, for the sample images, face recognition is performed on all sample images of the target scene (that is, pedestrian images included in the surveillance video) by using the face recognition technology to obtain face information. Using the face information, find the sample images belonging to the same person among these sample images, and associate them. These sample images belonging to the same person are used to compose positive sample pairs. If the sample image I _i and the sample image I _j contain the same person, the two sample images constitute a positive sample pair.

Since each sample image has the acquisition time and the corresponding number of the surveillance camera that acquired it, each surveillance camera has longitude and latitude information. Therefore, for each sample image, such as the sample image I _i , the number of the corresponding surveillance camera can be determined, and then the latitude and longitude information can be obtained, and all the surveillance cameras that are one kilometer away from the surveillance camera can be found by using the latitude and longitude information. for set1. From all the sample images collected by set1, find the sample images that are consistent with the collection time of the sample image I _i , and these sample images are the sample images that cannot possibly contain the same person as the sample image I _i . Use these sample images to create negative sample pairs, which can be written as

利用triplet loss损失函数，使用训练数据对第一数据模型进行训练。因为，这些样本对的可靠度可以非常高，准确率接近100％，所以称之为强监督多模态监督数据。Using positive sample pairs from face information and negative sample pairs from spatio-temporal information, strong supervision multimodal supervision data is formed. Subsequent combination of sample images and supervised data can form training data, which is expressed as

Using the triplet loss loss function, the first data model is trained using the training data. Because the reliability of these sample pairs can be very high and the accuracy rate is close to 100%, it is called strongly supervised multimodal supervised data.

In a second aspect, second supervised data, ie weakly supervised multimodal supervised data, is mined based on sample images.

Specifically, based on the first data model trained by using the data of the source scene, feature extraction is performed on the sample image to obtain feature information.

The following process is performed on each sample image in the target scene, so as to obtain the corresponding second supervision data.

For example, a target sample image is determined from the target scene, and the other sample images are the remaining sample images. Use the feature information of the target sample image and the to-be-sorted features of the remaining sample images to perform sorting to obtain an initial sorting result.

Using the face information of the target sample image, the initial sorting result is re-corrected and sorted to obtain a new and more accurate first pre-adjusted sorting result.

Using the spatiotemporal information of the target sample image, the first pre-adjustment and sorting results are re-corrected and sorted to obtain new and more accurate second pre-adjustment and sorting results.

Using the human attribute information of the target sample image, the second pre-adjustment sorting result is re-corrected and sorted to obtain a new and more accurate target sorting result.

Take the target ranking result as the ranking label of the target sample image. Subsequently, weakly supervised training data can be established based on target sample images and target ranking results. Since the correct rate of the target ranking result is not 100%, it is called weakly supervised training data, and the ranking loss loss function and the formed training data are used for training.

In the third aspect, we mine weakly supervised unimodal supervised data in the target scene. For example, by using the first data model trained on the source scene, feature extraction is performed on the sample image of the target scene to obtain feature information. These feature information are clustered by clustering method, and the category obtained by clustering is used as supervised data (ie, pseudo-label).

Subsequently, weakly supervised training data can be established by using the pseudo-labels and sample images obtained by clustering. Due to the relatively low feature information and reliable rows belonging to the category, the accuracy cannot be guaranteed to be 100%, so it is called weakly supervised training data. The data model is trained using a classification loss function with weakly supervised training data.

After obtaining supervised data, each type of supervised data is treated as a sub-training framework. The three sub-training frameworks are carried out at the same time, and the whole training process is regarded as a multi-task learning process, that is, three sub-frames are trained at the same time and the same model is updated. Adjust the proportion of the impact of different training frameworks on the data model by training weights. Finally, the required second data model of the new target scene (such as a pedestrian re-identification model) is obtained.

After one training cycle is completed, the first data model of the source scene is replaced by the trained first data model, and the above process is repeated until the required number of training cycles is completed to obtain the second data model.

In the above process, not only the sample images of the target scene (including human body images) are used to train the first data model, but also various modal information such as face information and spatiotemporal information are introduced to extract supervision data and form training data. .

Moreover, the supervised data obtained in various ways are modularized according to the different accuracy rates, divided into three different sub-training frameworks, and integrated with the method of multi-task learning to update and train the first data model together. . Compared with the previous model trained with only one type of supervised data, the effect is better, and it can directly improve the training effect of the data model without manual annotation in the target scene.

With this embodiment, the multimodal supervision data is mined from the sample images according to the spatiotemporal information of the sample images, so that the supervision data used for training the first data model does not need to be manually marked, thereby reducing the need for data model The difficulty and labor cost in the process of transferring from the source scene to the target scene realize the purpose of training the data model without manually labeling the sample images of the target scene. Supervise the data, thus improving the training of the data model.

Embodiment 5

Referring to FIG. 5 , a flowchart of steps of a data processing method according to Embodiment 5 of the present invention is shown.

In this embodiment, the data processing method includes the following steps:

Step S502: Obtain multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image.

Different sample images can be used for different usage requirements. For example, monitoring images can be used as sample images for use in pedestrian re-identification scenarios. For use in object recognition scenarios, images containing different objects can be used as sample images, and so on.

Taking the pedestrian re-identification scene as an example, the time information of the sample image includes at least the collection time and collection location of the sample image. The acquisition time can be the timestamp created when this sample image was taken. The acquisition location may be the location where the image acquisition device that photographed the sample image is located.

In multimodal supervised data, multimodality refers to the combination of information from multiple modalities, such as the combination of face information and spatiotemporal information. The multimodal supervision data includes first supervision data and/or second supervision data. The first supervised data may be strongly supervised supervised data, and strong supervision means that the accuracy of pseudo-labels established based on multimodal information of these supervised data is very high (eg, 100% or approximately 100%). The second supervised data may be weakly supervised supervised data. Weak supervision means that the accuracy of pseudo-labels established based on multimodal information is relatively low (usually less than 100% or does not meet the condition of approximately 100%).

Those skilled in the art may obtain the multimodal supervision data corresponding to the sample image in any appropriate manner, which is not limited in this embodiment.

Taking obtaining the first supervision data as an example, the first supervision data is used to indicate whether the two sample images contain the same target object. Therefore, for the target sample image Q, according to the face information of the target sample image Q and the face information of the remaining sample images, the sample images containing the same face as the target sample image Q can be determined. represents the same person, so the sample images containing the same face and the target sample image Q form a positive sample pair, and the first supervision data of each sample image containing the same face is a positive sample.

Also for the target sample image Q, according to the spatiotemporal information of the target sample image Q and the spatiotemporal information of the remaining sample images, it can be determined that the target sample image Q is collected at the same time, but the distance between the collection positions is greater than the set distance threshold (set The distance threshold can be determined as needed) sample images, because under normal circumstances, it is impossible for the same person to appear in two locations at the same time at a certain moment, therefore, the distance between the collection locations is greater than the set distance threshold. The target sample image Q constitutes a negative sample pair, and the first supervision data of the sample image whose distance between each collection position is greater than the set distance threshold is a negative sample.

In this way, since at least the spatiotemporal information of the sample images is integrated when determining the supervision data, multi-modal supervision data can be obtained, which not only realizes the automatic acquisition of supervision data, but also does not need to rely on manual annotation, reducing the cost of obtaining supervision data. .

Step S504: Train the data model to be trained according to at least the sample image and the multimodal supervision data, and obtain a target data model.

Wherein, the data model to be trained includes a machine learning model. It can be a different form of machine learning model for different usage scenarios. For example, in a pedestrian re-identification scenario, it can be a convolutional neural network model, or a convolutional neural network model combined with an attention mechanism, and so on.

In the process of using the sample images to train the data model to be trained, the multimodal supervision data corresponding to the sample images is used as supervision, and the loss value is calculated, and then the parameters in the data model to be trained are updated according to the loss value to obtain A target data model that meets the needs of the usage scenario.

For example, during training, the sample image Q and the sample image W form a positive sample pair, and the corresponding first supervision data indicates that the sample image W is a positive sample. Then the sample image Q and the sample image W are input into the data model to be trained, the output data of the data model to be trained is obtained, the loss value is calculated according to the first supervision data and the output data, and the data model to be trained is adjusted according to the loss value. parameter. Then input other positive sample pairs into the data model to be trained after parameter adjustment, and repeat until the training termination condition is met, and the target data model is obtained.

Optionally, in this embodiment, the target data model trained for the pedestrian re-identification scene can be applied to the path recognition of the target object. For example, the recognition of the movement path of the target object is achieved through the following steps.

Step S506: Acquire multiple frames of images in the video data, where at least some of the frame images in the multiple frames of images include the target object.

At least part of the frame images in the multi-frame images may be one frame image or more than one frame image. Taking the video data as surveillance video as an example, in the surveillance video, there may be one frame of image or more than one frame of images to capture the target object.

Step S508: Use the target data model to perform object recognition on the multi-frame images.

During object recognition, multiple frames of images are input into the target data model, and the target frame images containing the target object are identified by using the target data model as the recognition result.

Step S510: Determine the movement path of the target object according to the recognition result.

For each target frame image included in the recognition result, the collection positions of each target frame image are combined according to the collection time of each target frame image, thereby forming a moving path of the target object.

For example, the target frame image includes image 1, image 2, image 3 and image 7, the corresponding acquisition times are time t1, time t2, time t3 and time t7 respectively, and the corresponding acquisition positions are P1 position, P2 position, P3 position and The position of P2, the moving path of the target object determined based on the target frame image can be expressed as: P1-P2-P3-P2.

Through this data processing method, the multi-modal supervision data of the sample image can be mined based on the spatiotemporal information of the sample image, so that the supervision data used for training the data model to be trained does not need to be manually labeled, thereby reducing the need for training the target data model. It realizes the purpose of training the data model without manual annotation of sample images, and because multi-modal supervision data is used during training, the training effect of the data model can be improved.

Embodiment 6

Referring to Fig. 6a, a flowchart of steps of a data processing method according to Embodiment 6 of the present invention is shown.

In this embodiment, the data processing method provided by the embodiment of the present invention is described by taking the example of deploying the data processing method on the server (eg, cloud or server or SaaS platform) and training the first data model according to the client's request.

The data processing method of this embodiment includes the following steps:

Step S602: Receive a model training request for requesting to train the first data model and sent by the client by calling a preset training interface.

Taking the method deployed on the SaaS platform as an example, the training interface may be an interface preset by the SaaS platform for receiving a model training request from a client. The interface may be configured in any appropriate form as required, which is not limited in this embodiment.

The first data model may be the model described in any one of the first to fourth embodiments. The model training request may be any suitable form of request.

Step S604: Obtain sample images for training according to the model training request.

The first data model may be deployed on the SaaS platform, or may be acquired by the SaaS platform from a client or a third party through a network, which is not limited in this embodiment.

Taking the deployment of the first data model on the SaaS platform as an example, when obtaining a sample image, when the first data model is trained by receiving a model training request from the client through the SaaS platform, the model training request can be locally obtained from the SaaS platform for training. training sample images. In this case, the SaaS platform stores applicable sample images locally and can obtain them directly, thereby improving the speed and efficiency of model training.

In another feasible manner, when the first data model is trained by receiving a model training request from the client through the SaaS platform, the SaaS platform may collect sample images for training from a third party according to the model training request. Such as obtaining sample images from third-party websites or data interfaces provided by third-party applications through the network. In this case, the SaaS platform obtains sample images from a third party without local storage, which saves the storage resources of the SaaS platform.

In yet another feasible manner, when training the first data model by receiving a training request from a client model through the SaaS platform, the SaaS platform may obtain sample images for training from the client according to the model training request. . In this case, sample images are stored in the client, and the SaaS platform obtains the sample images from the client, and can train a second data model that better meets the needs of the client.

Step S606: Obtain multimodal supervision data corresponding to the sample image, and use the sample image and the multimodal supervision data to train a first data model.

For example, the multimodal supervision data corresponding to the sample image can be obtained by any one of Embodiments 1 to 4, and the first data model can be trained by using the sample image and the multimodal supervision data to obtain the second data model. This will not be repeated here.

Hereinafter, the above process is exemplarily described by taking the first data model deployed on the SaaS platform as an example, as shown in FIG. 6b.

In Figure 6b, the client sends a model training request to the SaaS platform; after the SaaS platform receives the request, the processing device obtains a sample image for training the first data model from the local storage device; the SaaS platform obtains a sample image based on the obtained sample image , and obtain the multimodal supervision data of the sample image at least according to the spatiotemporal information of the sample image. The first data model is trained using the sample images and the multimodal supervision data; after the SaaS platform completes the training of the first data model and obtains the second data model, the SaaS platform sends a training completion message to the client. Subsequently, if the client has requirements, it can send video data or images to be detected to the SaaS platform to obtain the corresponding moving path of the target object.

In the above, the first data model is deployed on the SaaS platform as an example, but those skilled in the art should understand that the solution of this embodiment is also applicable to the case where the first data model is deployed on other forms of servers.

It can be seen that through this embodiment, the first data model and its training are both deployed on the server, and the server trains the first data model according to the client's request, thereby realizing the first data model without requiring client resources or performance. The training of a data model ensures the training effect and efficiency.

Embodiment 7

Referring to FIG. 7 , it shows a structural block diagram of a data processing apparatus according to Embodiment 7 of the present invention.

In this embodiment, the data processing apparatus includes: a first acquisition module 702 for acquiring a first data model and a sample image, wherein the first data model includes a machine learning model; a second acquisition module 704 for at least according to The spatiotemporal information of the sample image, to obtain the multi-modal supervision data corresponding to the sample image; the first training module 706 is used for at least according to the sample image and the multi-modal supervision data. The data model is trained to obtain a second data model.

Optionally, the second obtaining module 704 includes: a face recognition module 7041, configured to perform face recognition on the sample image, and obtain face information corresponding to the sample image; The face information and the spatiotemporal information of the sample image determine the first supervision data corresponding to the sample image.

Optionally, the determining module 7042 is configured to determine a positive sample set consisting of sample images containing the same face according to the face information of the sample images; according to the spatiotemporal information of the sample images, determine the sample images that meet the set conditions. A set of negative samples formed; according to the set of positive samples and the set of negative samples, determine the first supervision data corresponding to the sample images.

Optionally, the determining module 7042 is configured to determine the target sample image from the sample images when the negative sample set composed of the sample images satisfying the set condition is determined according to the spatiotemporal information of the sample images; Among the sample images other than the sample images, the sample images whose acquisition time is the same as that of the target sample image and the distance between the acquisition positions is greater than the set distance threshold value are determined as negative sample images; The set of negative samples corresponding to the target sample image.

Optionally, the determining module 7042 includes: a first feature extraction module 7042a, configured to perform feature extraction on the sample image by using the first data model, and obtain feature information used to characterize human body image information; a sorting module 7042b, using at least according to the spatiotemporal information of the sample images and the similarity between the feature information and the features to be sorted, the feature information to be sorted is sorted to obtain a target sorting result; the supervision and determination module 7042c is used to sort the targets The result is determined as the second supervision data.

Optionally, the sorting module 7042b is used to determine the target sample image from the sample images; according to the similarity between the feature information of the target sample image and the feature to be sorted, sort the feature information to be sorted to obtain an initial sorting result; at least According to the spatiotemporal information of the target sample image, the initial sorting result is adjusted to obtain the target sorting result.

Optionally, the sorting module 7042b is configured to adjust the initial sorting result at least according to the spatiotemporal information of the target sample image, and when obtaining the target sorting result, according to the face information of the target sample image and all According to the similarity between the face information of the sample images corresponding to the features to be sorted, perform a first adjustment on the initial sorting result to obtain a first pre-adjustment sorting result; according to the spatiotemporal information of the target sample image and the According to the spatiotemporal information of the sample images corresponding to the sorting features, a second adjustment is performed on the first pre-adjustment sorting result to obtain a second pre-adjusting sorting result; according to the human attribute information of the target sample image and the sample image corresponding to the to-be-sorted feature The human body attribute information is obtained, and a third adjustment is performed on the second pre-adjustment sorting result to obtain the target sorting result.

Optionally, the apparatus further includes: a second feature extraction module 708, configured to train the first data model in the first training module 706 at least according to the sample images and the multimodal supervision data, Before obtaining the second data model, use the first data model to perform feature extraction on the sample image; the clustering module 710 is configured to perform clustering processing on the feature information of the sample image to obtain the belonging of the sample image. category, and the category to which it belongs is determined as the single-modal supervision data of the sample image.

Optionally, the multimodal supervision data includes first supervision data and second supervision data; the first training module 706 includes: a weight configuration module 7061, which is used for comparing the first supervision data and the second supervision data Perform training weight configuration with the single-modal supervision data; the model first training module 7062 is used to use the sample image, the first supervision data, the second supervision data, the single The modal supervision data and the loss function corresponding to the supervision data are used to perform multi-task training on the data model.

The data processing apparatus in this embodiment is used to implement the corresponding data processing methods in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, for the function implementation of each module in the data processing apparatus of this embodiment, reference may be made to the descriptions of the corresponding parts in the foregoing method embodiments, and details are not repeated here.

Embodiment 8

Referring to FIG. 8 , a structural block diagram of a data processing apparatus according to Embodiment 8 of the present invention is shown.

In this embodiment, the data processing device includes:

A third obtaining module 802, configured to obtain multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image;

The second training module 804 is configured to train a data model to be trained according to at least the sample image and the multimodal supervision data, and obtain a target data model, wherein the data model to be trained includes a machine learning model .

Optionally, the device further includes:

a fourth acquisition module 806, configured to acquire multiple frames of images in the video data, at least some of the frame images in the multiple frames of images include a target object;

an object recognition module 808, configured to perform object recognition on the multi-frame images using the target data model;

The path determination module 810 is configured to determine the moving path of the target object according to the recognition result.

The data processing apparatus in this embodiment is used to implement the corresponding data processing methods in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, for the function implementation of each module in the data processing apparatus of this embodiment, reference may be made to the descriptions of the corresponding parts in the foregoing method embodiments, and details are not repeated here.

The data processing apparatus in this embodiment is used to implement the corresponding data processing methods in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, for the function implementation of each module in the data processing apparatus of this embodiment, reference may be made to the descriptions of the corresponding parts in the foregoing method embodiments, and details are not repeated here.

Embodiment 9

Referring to FIG. 9 , a schematic structural diagram of an electronic device according to Embodiment 6 of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

As shown in FIG. 9 , the electronic device may include: a processor (processor) 902 , a communication interface (Communications Interface) 904 , a memory (memory) 906 , and a communication bus 908 .

in:

The processor 902 , the communication interface 904 , and the memory 906 communicate with each other through the communication bus 908 .

The communication interface 904 is used to communicate with other electronic devices or servers.

The processor 902 is configured to execute the program 910, and specifically may execute the relevant steps in the above-mentioned embodiments of the check code generation method.

Specifically, the program 910 may include program code including computer operation instructions.

The processor 92 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. One or more processors included in the smart device may be the same type of processors, such as one or more CPUs; or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 906 is used to store the program 910 . Memory 906 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

First Embodiment:

The program 910 can be specifically configured to cause the processor 902 to perform the following operations: acquire a first data model and a sample image, where the first data model includes a machine learning model; Corresponding multi-modal supervision data; at least according to the sample image and the multi-modal supervision data, the first data model is trained to obtain a second data model.

In an optional implementation manner, the program 910 is further configured to cause the processor 902 to perform an analysis on the sample image when obtaining multimodal supervision data corresponding to the sample image according to at least the spatiotemporal information of the sample image. Perform face recognition to obtain face information corresponding to the sample image; and determine first supervision data corresponding to the sample image according to the face information and the spatiotemporal information of the sample image.

In an optional implementation manner, the program 910 is further configured to make the processor 902 determine the first supervision data corresponding to the sample image according to the face information and the spatiotemporal information of the sample image, according to The face information of the sample images determines a positive sample set composed of sample images containing the same face; according to the spatiotemporal information of the sample images, determines a negative sample set composed of sample images that meet the set conditions; The sample set and the negative sample set determine the first supervision data corresponding to the sample image.

In an optional implementation manner, the program 910 is further configured to cause the processor 902 to determine the target from the sample images when determining a negative sample set consisting of sample images that satisfy the set condition according to the spatiotemporal information of the sample images sample image; from the sample images other than the target sample image, the sample image whose acquisition time is the same as that of the target sample image and the distance between the acquisition positions is greater than the set distance threshold is determined as a negative sample image; according to The negative sample image determines a negative sample set corresponding to the target sample image.

In an optional implementation manner, the program 910 is further configured to cause the processor 902 to use the first method when obtaining the multimodal supervision data corresponding to the sample image according to at least the spatiotemporal information of the sample image. The data model performs feature extraction on the sample image to obtain feature information used to characterize human body image information; at least according to the spatiotemporal information of the sample image and the similarity between the feature information and the feature to be sorted, the feature to be sorted is The information is sorted to obtain a target sorting result; the target sorting result is determined as the second supervision data.

In an optional implementation manner, the program 910 is further configured to cause the processor 902 to perform the sorting process on the feature information to be sorted according to at least the spatiotemporal information of the sample image and the similarity between the feature information and the feature to be sorted. Sorting, when the target sorting result is obtained, the target sample image is determined from the sample images; according to the similarity between the feature information of the target sample image and the feature to be sorted, the feature information to be sorted is sorted to obtain the initial sorting result; at least according to the Based on the spatiotemporal information of the target sample image, the initial sorting result is adjusted to obtain the target sorting result.

In an optional implementation manner, the program 910 is further configured to make the processor 902 adjust the initial sorting result according to at least the spatiotemporal information of the target sample image, and when obtaining the target sorting result, adjust the sorting result according to the target the similarity between the face information of the sample image and the face information of the sample image corresponding to the feature to be sorted, first adjust the initial sorting result to obtain a first pre-adjustment sorting result; according to the target sample The spatiotemporal information of the image and the spatiotemporal information of the sample image corresponding to the feature to be sorted, perform a second adjustment on the first pre-adjustment sorting result, and obtain a second pre-adjustment sorting result; The human body attribute information of the sample images corresponding to the features to be sorted is performed, and a third adjustment is performed on the second pre-adjustment sorting result to obtain the target sorting result.

In an optional implementation manner, the program 910 is further configured to cause the processor 902 to train the first data model at least according to the sample image and the multimodal supervision data to obtain a second data model Before, use the first data model to perform feature extraction on the sample image; perform clustering processing on the feature information of the sample image to obtain the category to which the sample image belongs, and determine the category as the Single-modal supervised data for sample images.

In an optional implementation manner, the multimodal supervision data includes first supervision data and second supervision data; the program 910 is further configured to cause the processor 902 to perform an operation based on at least the sample image and the multimodal supervision data. , train the first data model, and when the second data model is obtained, perform training weight configuration on the first supervision data, the second supervision data and the single-modal supervision data; use the sample image , The first supervision data, the second supervision data, the single-modal supervision data, and the loss function corresponding to the supervision data, which have been weighted, perform multi-task training on the data model.

Second embodiment:

The program 910 may be specifically configured to cause the processor 902 to perform the following operations: at least according to the spatiotemporal information of the sample image, obtain multimodal supervision data corresponding to the sample image; at least according to the sample image and the multimodal supervision data , train the data model to be trained, and obtain the target data model, wherein the data model to be trained includes a machine learning model.

In an optional implementation manner, the program 910 is further configured to cause the processor 902 to acquire multiple frames of images in the video data, and at least some of the frame images in the multiple frames of images include the target object; The multi-frame images are used for object recognition; according to the recognition results, the moving path of the target object is determined.

Third Embodiment:

The program 910 can specifically be used to cause the processor 902 to perform the following operations: receive a model training request sent by the client by calling a preset training interface for requesting training of the first data model; and obtain the model training request according to the model training request. Sample images used for training; obtain the multimodal supervision data corresponding to the sample images by using the data processing method according to the first embodiment, and use the sample images and the multimodal supervision data to carry out the first data model. train.

In an optional implementation manner, the program 910 is further configured to cause the processor 902 to obtain the sample image locally from the SaaS platform according to the model training request when obtaining the sample image according to the model training request; or, according to the model training request; For the model training request, the SaaS platform obtains sample images from a third party; or, according to the model training request, the SaaS platform obtains sample images from the client.

For the specific implementation of the steps in the program 910, reference may be made to the corresponding descriptions in the corresponding steps and units in the above data processing method embodiments, which are not repeated here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices and modules, reference may be made to the corresponding process descriptions in the foregoing method embodiments, which will not be repeated here.

It should be pointed out that, according to the needs of implementation, each component/step described in the embodiments of the present invention may be split into more components/steps, or two or more components/steps or some operations of components/steps may be combined into New components/steps to achieve the purpose of embodiments of the present invention.

The above-described methods according to embodiments of the present invention may be implemented in hardware, firmware, or as software or computer codes that may be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or implemented by Network downloaded computer code originally stored in a remote recording medium or non-transitory machine-readable medium and will be stored in a local recording medium so that the methods described herein can be stored on a computer using a general purpose computer, special purpose processor or programmable or such software processing on a recording medium of dedicated hardware such as ASIC or FPGA. It is to be understood that a computer, processor, microprocessor controller or programmable hardware includes storage components (eg, RAM, ROM, flash memory, etc.) that can store or receive software or computer code when the software or computer code is executed by a computer, When accessed and executed by a processor or hardware, the data processing methods described herein are implemented. Furthermore, when a general-purpose computer accesses code for implementing the data processing methods shown herein, execution of the code converts the general-purpose computer into a special-purpose computer for executing the data processing methods shown herein.

Those of ordinary skill in the art can realize that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the embodiments of the present invention.

The above embodiments are only used to illustrate the embodiments of the present invention, but not to limit the embodiments of the present invention. Those of ordinary skill in the relevant technical field can make various Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims (17)

1. A data processing method, comprising: acquiring a first data model and a sample image, wherein the first data model includes a machine learning model; Obtaining multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image; The first data model is trained according to at least the sample image and the multimodal supervision data to obtain a second data model. 2. The method according to claim 1, wherein the obtaining multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image, comprising: Perform face recognition on the sample image to obtain face information corresponding to the sample image; First supervision data corresponding to the sample image is determined according to the face information and the spatiotemporal information of the sample image. 3. The method according to claim 2, wherein the determining the first supervision data corresponding to the sample image according to the face information and the spatiotemporal information of the sample image comprises: According to the face information of the sample images, determine a positive sample set composed of sample images containing the same face; According to the spatiotemporal information of the sample images, determine a negative sample set composed of sample images that meet the set conditions; According to the positive sample set and the negative sample set, first supervision data corresponding to the sample image is determined. 4. The method according to claim 3, wherein, according to the spatiotemporal information of the sample images, determining a negative sample set composed of sample images that satisfy a set condition, comprising: Determine the target sample image from the sample image; From the sample images other than the target sample image, the sample image with the same acquisition time as the target sample image and the distance between the acquisition positions is greater than the set distance threshold is determined as a negative sample image; A negative sample set corresponding to the target sample image is determined according to the negative sample image. 5. The method according to claim 1, wherein obtaining multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image, comprising: using the first data model to perform feature extraction on the sample image to obtain feature information for characterizing human body image information; Sorting the feature information to be sorted according to at least the spatiotemporal information of the sample image and the similarity between the feature information and the feature to be sorted to obtain a target sorting result; The target ranking result is determined as the second supervision data. 6. The method according to claim 5, wherein, according to at least the spatiotemporal information of the sample image and the similarity between the feature information and the feature to be sorted, the feature information to be sorted is sorted to obtain the target ranking Results, including: Determine the target sample image from the sample image; According to the similarity between the feature information of the target sample image and the features to be sorted, sort the feature information to be sorted to obtain the initial sorting result; The initial sorting result is adjusted at least according to the spatiotemporal information of the target sample image to obtain the target sorting result. 7. The method according to claim 6, wherein, at least according to the spatiotemporal information of the target sample image, adjusting the initial sorting result to obtain the target sorting result, comprising: According to the similarity between the face information of the target sample image and the face information of the sample image corresponding to the feature to be sorted, perform a first adjustment on the initial sorting result to obtain a first pre-adjustment sorting result; According to the spatiotemporal information of the target sample image and the spatiotemporal information of the sample image corresponding to the feature to be sorted, a second adjustment is performed on the first pre-adjustment sorting result to obtain a second pre-adjustment sorting result; According to the human body attribute information of the target sample image and the human body attribute information of the sample image corresponding to the features to be sorted, a third adjustment is performed on the second pre-adjustment sorting result to obtain the target sorting result. 8. The method according to claim 1, wherein before the first data model is trained to obtain a second data model according to at least the sample images and the multimodal supervision data, the method further comprises: include: performing feature extraction on the sample image using the first data model; The feature information of the sample image is clustered to obtain the category to which the sample image belongs, and the category is determined as the single-modality supervision data of the sample image. 9. The method of claim 8, wherein the multimodal supervisory data comprises first supervisory data and second supervisory data; The said first data model is trained at least according to the sample image and the multimodal supervision data to obtain a second data model, including: performing training weight configuration on the first supervision data, the second supervision data and the single-modal supervision data; Multi-task training is performed on the data model using the sample image, the first supervised data, the second supervised data, the single-modal supervised data, and the loss function corresponding to the supervised data with weights configured. 10. A data processing method, comprising: Obtaining multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image; At least according to the sample image and the multimodal supervision data, the data model to be trained is trained, and the target data model is obtained, wherein the data model to be trained includes a machine learning model. 11. The method of claim 10, wherein the method further comprises: Acquiring multiple frames of images in the video data, at least part of the frame images in the multiple frames of images includes a target object; performing object recognition on the multi-frame images using the target data model; According to the recognition result, the moving path of the target object is determined. 12. A data processing method, comprising: receiving a model training request sent by the client by calling a preset training interface and used for requesting training of the first data model; obtaining sample images for training according to the model training request; Obtain the multimodal supervision data corresponding to the sample image by using the data processing method according to any one of claims 1-9, and use the sample image and the multimodal supervision data to carry out the first data model. train. 13. The method according to claim 12, wherein the obtaining a sample image according to the model training request comprises: Obtain sample images locally from the SaaS platform according to the model training request; or, According to the model training request, sample images from a third party by the SaaS platform; or, According to the model training request, the SaaS platform obtains sample images from the client. 14. A data processing device comprising: a first acquisition module, configured to acquire a first data model and a sample image, wherein the first data model includes a machine learning model; a second obtaining module, configured to obtain multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image; A first training module, configured to train the first data model according to at least the sample image and the multimodal supervision data to obtain a second data model. 15. A data processing device, comprising: a third obtaining module, configured to obtain multimodal supervision data corresponding to the sample image at least according to the spatiotemporal information of the sample image; The second training module is configured to train a data model to be trained according to at least the sample image and the multimodal supervision data, and obtain a target data model, wherein the data model to be trained includes a machine learning model. 16. An electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the data processing method according to any one of claims 1-9, or execute the method according to claim 10 or 11. the data processing method of claim 12 or 13, or perform operations corresponding to the data processing method of claim 12 or 13. 17. A computer storage medium on which a computer program is stored, and when the program is executed by a processor, the data processing method according to any one of claims 1-9, or the method according to claim 10 or 11 is realized. A data processing method, or implementing the data processing method as claimed in claim 12 or 13.

Images