TWI748685B - Method and server for optimizing joint object detection model - Google Patents

Abstract

The present invention discloses a method and server for optimizing a joint object detection model, which is mainly applied to accelerate and effectively produce artificial intelligence (AI) training image annotation data, AI image detection model and AI image classification model. The present invention can effectively reduce the cost of manpower and time in the stages of graphic data managing, screening, labeling and review.

Description

Method and server for optimizing joint object detection model

The present invention relates to a technique for training an object detection model, and particularly relates to a method and server for optimizing a joint object detection model.

In the application of image monitoring, deep learning technology combined with computer vision has become a development trend, and the most widely used is supervised learning, that is, all data in the model training process needs to be correctly labeled in advance. However, to produce a deep learning model with high prediction accuracy, a large number of image annotation training samples need to be collected. At present, most of the methods of image labeling are manual labeling. The labeler needs to select the object range and the object category name associated with the label one by one for the target object in the image data, and review it after labeling to ensure the accuracy of the labeling content. Such labeling The drawing and funding process is time-consuming and labor-intensive.

In the field of image surveillance applications, various artificial intelligence image recognition scenarios and application requirements are increasing. If only a deep learning model trained based on a fixed set of annotation maps, there is a lack of data diversification, and it is often difficult to adapt to the needs of real scenarios and applications. If you need to build an exclusive set of graphics resources according to the needs of the real scene, realize the specific category The classification requires considerable labeling and verification manpower to maintain. If one or more new object feature categories must be added to the existing application model after construction, the cost of adding new annotations to the original training image collection is also considerable.

In addition, most of the conventional automatic labeling methods are to ensure the objectivity and accuracy of the labeling results, but the automatic labeling model needs to screen the labeling results during its training process, and there is still a large amount of manual review of the cost of labeling data. . In addition, the training process for finding the best model mostly relies on existing training data sets. However, in practical applications, it is often necessary to detect or classify different categories according to different environments and needs, and the categories are diverse And it is complicated, so it is necessary to construct or expand the training data set containing the actual data on the spot to realize the classification of specific categories by the applicable model.

In addition, the conventional technology meets the requirement of changing detected objects for models that have reached the target of prediction accuracy. Under the conditions of limited time and cost, it is not easy to inspect all the huge training images and mark new object categories one by one. Then refine the model again to provide a detection model of new object feature combinations for newly changed application requirements. Therefore, in practice, the workaround is to combine the types of objects that need to be detected from the existing object detection models in a serial or parallel manner. However, this can omit the re-labeling and training of the model in the initial stage. Cost, and quickly launch the application, but the disadvantage is that the system resource requirements are high and the operation efficiency is poor. Because the resources of the online system are limited, and the expansion cost of the image processing unit (GPU) inference equipment is high, the maintenance cost of subsequent expansion is also The pressure of frequent customer complaints is not a good strategy for long-term online applications.

In view of this, the present invention provides a method and server for optimizing a joint object detection model, which can be used to solve the above technical problems.

The present invention provides a method for optimizing a joint object detection model, including: obtaining a map database, wherein the map database includes a first unlabeled map asset and a plurality of annotated map asset sets, and each annotated map asset set corresponds to a single category Objects; training multiple single-object detection models and multiple single-object classification models with the multiple annotated image data sets, wherein the multiple annotated image data sets correspond to the multiple single-object detections one-to-one Model, and the plurality of annotated image data sets correspond to the plurality of single object classification models one-to-one; obtain a plurality of designated detection object categories, and obtain a plurality of external object detection models accordingly, wherein each external object The object detection model is used to detect objects belonging to at least one of the plurality of designated detection object categories; in response to determining that each detection object category corresponds to one of the plurality of annotated image data sets, from The plurality of single-object detection models find out a plurality of specific single-object detection models corresponding to the plurality of designated detection object categories; and find out the plurality of specific single-object detection models corresponding to the plurality of designated detection objects in the plurality of annotated image data sets Detect a plurality of specific annotation map data sets of object types, and train a joint object detection model accordingly; use the plurality of specific single object detection models, the plurality of external object detection models, and joint object detection The model detects the first unlabeled image data to generate a plurality of first object detection results, wherein the plurality of first object detection results correspond to a first of the plurality of specified detection object categories Specify the detection object category; use a first single object classification model corresponding to the first specified detection object category among the plurality of single object classification models to predict and classify the detection results of each first object to obtain each first object Detection knot A first object classification result of the result; based on each first object detection result and the corresponding first object classification result, the plurality of first object detection results are adaptively corrected, and the corrected plurality of first object An object detection result is added to a first annotated image data set corresponding to the first specified detection object category in the plurality of annotated image data sets; the multiple single-object detection models are retrained based on the first annotated image data set A first single-object detection model, a first single-object classification model, and a joint object detection model corresponding to the first designated detection object category.

The invention provides a server for optimizing a joint object detection model, which includes a storage circuit and a processor. The storage circuit stores multiple modules. The processor is coupled to the storage circuit and accesses the plurality of modules to perform the following steps: obtain a map database, where the map database includes a first unlabeled map resource and a plurality of labeled map resource sets, each of which is labeled A set corresponds to a single type of object; multiple single-object detection models and multiple single-object classification models are trained with the multiple annotated image data sets, wherein the multiple annotated image data sets correspond to the multiple one-to-one A single-object detection model, and the plurality of label map data sets correspond to the plurality of single-object classification models one-to-one; obtain a plurality of designated detection object categories, and obtain a plurality of external object detections accordingly Model, wherein each external object detection model is used to detect objects belonging to at least one of the plurality of specified detection object categories; in response to determining that each detection object category corresponds to the plurality of annotated image data sets One of them is to find a plurality of specific single-object detection models corresponding to the plurality of designated detection object categories from the plurality of single-object detection models; The plurality of specific annotation map data sets of the plurality of designated detection object categories are used to train a joint object detection model; the plurality of specific single object detection models are used Type, the plurality of external object detection models and the joint object detection model detect the first unlabeled image to generate a plurality of first object detection results, wherein the plurality of first object detection results Corresponding to a first specified detection object category in the plurality of specified detection object categories; using a first single object classification model pair corresponding to the first specified detection object category among the plurality of single object classification models Predictive classification is performed on each first object detection result to obtain a first object classification result of each first object detection result; based on each first object detection result and the corresponding first object classification result, the multiple A first object detection result, and adding the corrected detection results of the plurality of first objects to a first annotation image data set corresponding to the first designated detection object category in the plurality of annotated image data sets; Retraining a first single-object detection model, a first single-object classification model, and a joint object detection model corresponding to the first designated detection object category among the plurality of single-object detection models based on the first annotation image data set .

100: server

102: storage circuit

104: processor

310: Picture Collection

311: Integrate annotated map data collection

GA ~ GH, GN: Annotated image collection

A ₀ ~ E ₀ , N ₀ : Single object detection model

A’~E’,N’: Single object classification model

324: Joint Detection Model

341~343: Detection result

S210~S290: steps

FIG. 1 is a schematic diagram of a server for optimizing a joint object detection model according to an embodiment of the present invention.

FIG. 2 is a flowchart of a method for optimizing a joint object detection model according to an embodiment of the present invention.

Fig. 3 is an application scenario diagram drawn according to an embodiment of the present invention.

Generally speaking, in order to solve the above-mentioned problems faced by the application requirements of artificial intelligence image recognition, the present invention includes the establishment of single-object annotation map resource collection and single-object detection model and single-object classification model mechanism, image prediction and annotation mechanism, and multi-model prediction The result comparison mechanism and other methods are combined to achieve better application goals.

In terms of expanding the features of annotated image collections, the present invention can uniformly store the externally imported annotated image collections after disassembly and fusion and collect the feature annotation content of the detectable object feature categories of the external object detection model.

In terms of multi-model comparison and training efficiency characteristics, the effect of establishing a single-object feature object detection model for predictive labeling and correction retraining is better than the labeling and training effects of labeling multiple object feature maps at once. , And the quality of the artificially reviewed and revised forecast annotation data is better.

In the field application and the combination of target detection model features, the single-object detection model can be established quickly and generate the prediction annotation data of the new case map, which can be flexibly matched with the combination of multiple object features required by the case, and can be further established Combine the training graphics and object detection model of the new case.

In the automatic review feature of the model predicting the object category name, the regional image within the bounding box of the labeled object can be used as the training image. Through different algorithms and parameters, the single-object feature classification model can be trained for real-time labeling of the predicted data. It has the effect of reducing the cost of manpower review.

In the work method and workflow characteristics of the iterative cycle of the refined artificial intelligence object detection model, the above-mentioned characteristics and the use of the multi-model prediction result comparison mechanism have the training model to converge faster, the annotation map data is fast and easy to diversify, and the accuracy is Higher, The advantages of faster prediction speed, saving annotation and review manpower, make the model more efficient, and can also quickly pick out specific object images from the scene images, which can effectively improve the problem of data bias. A detailed description of the present invention will be provided below.

Please refer to FIG. 1, which is a schematic diagram of a server that optimizes a joint object detection model according to an embodiment of the present invention. In different embodiments, the server 100 for optimizing the combined object detection model is, for example, various computer devices and smart devices, but it is not limited to this. As shown in FIG. 1, the server 100 includes a storage circuit 102 and a processor 104.

The storage circuit 102 is, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard disk Disk or other similar devices or a combination of these devices can be used to record multiple codes or modules.

The processor 104 is coupled to the storage circuit 102, and can be a general purpose processor, a special purpose processor, a traditional processor, a digital signal processor, multiple microprocessors, one or more combined digital signal processing The core microprocessor, controller, microcontroller, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), any other types of integrated circuits , State machines, processors based on Advanced RISC Machine (ARM) and similar products.

In the embodiment of the present invention, the processor 104 can access the storage circuit 102 The module and program code recorded in the document are used to implement the method of optimizing the joint object detection model proposed by the present invention. The details of the method are as follows.

Please refer to FIG. 2, which is a flowchart of a method for optimizing a joint object detection model according to an embodiment of the present invention. The method of this embodiment can be executed by the server 100 in FIG. 1. The details of each step in FIG. 2 are described below with the components shown in FIG. 1. In addition, in order to facilitate the understanding of the concept of the present invention, the following will further take FIG. 3 as an example for description, where FIG. 3 is an application scenario diagram drawn according to an embodiment of the present invention.

First, in step S210, the processor 104 can obtain a map database 310, where the map database may include, for example, a plurality of unlabeled map resources and a plurality of labeled map resource sets GA~GH and GN, and the labeled map resource set GA ~GH and GN can respectively correspond to a single type of object. For example, the annotated image resource set GA may only include all kinds of images labeled with "object A" (such as a person) in a bounding box or other labeling methods, and the annotated image resource set GB may, for example, only include All kinds of pictures of "Object B" (such as a locomotive), for example, annotated image collection GC may only include all kinds of images labeled "Object C" (such as a bicycle), and the content of the rest of the annotated image collections should be deduced by analogy , I will not repeat it here.

After that, in step S220, the processor 104 may train multiple single-object detection models and multiple single-object classification models with the multiple annotated image data sets, wherein the multiple annotated image data sets GA~GH and GN one One-to-one correspondence to the plurality of single-object detection models, and one-to-one correspondence to the multiple single-object classification models.

For example, the processor 104 can annotate the image data set GA to train a single-object detection model A ₀ for detecting "object A", and train a single-object classification model for classifying "object A"A'. For another example, the processor 104 may label the image data set B to train the single-object detection model B ₀ for detecting "object B" and train the single-object to classify "object B" Classification model B'. For another example, the processor 104 may label the image data set C to train a single-object detection model C ₀ for detecting "object C", and train a single-object to classify "object C" Classification model C'. In other words, for each annotated image set in the image database 310, the processor 104 can be used to train the corresponding single-object detection model and single-object classification model.

It should be understood that the various models mentioned in the present invention are all artificial intelligence models, and can be implemented using various models preferred by the designer (for example, various neural networks, etc.), but may not be limited thereto.

In step S230, the processor 104 may obtain a plurality of specified detection object categories, and accordingly obtain a plurality of external object detection models ext1~ext3, of which the external object detection models ext1~ext3 can be used to detect those belonging to the A plurality of objects of at least one of the specified detection object types.

In one embodiment, if the user wants to use the joint object detection model trained by the method of the present invention to identify one or more objects in a specific field (such as a road, intersection, etc.), the user The above-mentioned designated detection object type can be set according to the object type to be detected. For example, if the user wants the joint object detection model to detect objects belonging to "Object A", "Object B", "Object C", "Object D" (such as a dog), "Object E" (such as a car), and "Object N" (e.g. ride 士) and other object types, the user can set "Object A", "Object B", "Object C", "Object D", "Object E" and "Object N" to the above specified detection Object category, but not limited to this.

In addition, in one embodiment, the above-mentioned external object detection model may individually have the ability to detect multiple object types at one time. For example, the external object detection model ext1 can detect "object A", "object B", "object K" and "object S". For another example, the external object detection model ext2 can detect "object C", "object G" and "object M", for example. In addition, the external object detection model ext3 can detect "object A", "object D", "object E" and "object X", for example. It can be seen from the above that the external object detection models ext1 to ext3 each have the ability to detect objects belonging to at least one of the plurality of specified detection object categories.

In addition, in some embodiments, the external object detection model selected by the processor 104 in step S230 may also be required to meet one or more conditions, such as "Intersection over the border of the object prediction area and the original labeled area. Union)>Overlap rate target threshold'', ``Average Precision of a single object feature category (Average Precision)>Single object average accuracy target threshold'', ``Model accuracy (accuracy)>Model accuracy target threshold'', ``Model loss function convergence The result value<model loss function target threshold" etc., but it is not limited to this.

After that, in an embodiment, the processor 104 may determine whether each specified detection object category corresponds to one of the plurality of annotated image resources.

In the scenario in Figure 3, due to "object A", "object B", "object C", "object "Item D", "Object E", and "Object N" respectively correspond to the annotated image data sets GA~GH and GN, so the processor 104 can determine that each specified detection object category corresponds to the plurality of annotated image data sets. One of them can be continued in step S240 to find multiple specific single object detection models corresponding to the multiple specified detection object categories from the multiple single object detection models.

In FIG. 3, the processor 104 can detect the single object model A corresponding to "object A", "object B", "object C", "object D", "object E" and "object N" ₀ ~ E ₀ and N _{0 are} used as the above-mentioned specific single-object detection models, but they are not limited to this.

However, in other embodiments, since the user may set a designated detection object category that does not correspond to any labeled map data set (hereinafter referred to as the designated detection object category), the present invention can use the proposed map data update Added mechanism to add annotated image data set corresponding to the reference specified detection object category (hereinafter referred to as reference annotated image data set), and train the corresponding single object detection model (hereinafter referred to as reference single object detection model) and Single object classification model (hereinafter referred to as reference single object detection model).

In one embodiment, when the processor 104 determines that the reference designated detection object category (for example, "bus") among the plurality of designated detection object categories does not correspond to any one of the annotation data sets GA~GH and GN , The processor 104 may add a reference annotated image resource set corresponding to the reference specified detection object category in the image resource library 310. After that, the processor 104 may obtain a plurality of first image data from the image data library 310, and annotate a specific type of object (such as a bus) corresponding to the reference specified detection object category in each first image data to generate Multiple first reference annotations. In some embodiments, the above-mentioned labeling behavior may be manually performed on the first image resource, but it may not be limited to this. after, The processor 104 may train the reference single object detection model and the reference single object classification model based on the above-mentioned first reference annotation image data.

In one embodiment, in response to determining that the reference single object detection model and the reference single object classification model meet the aforementioned target conditions, the processor 104 may add the reference single object classification model to the single object detection In the model, the reference single-object classification model is added to the above-mentioned single-object classification model and the above-mentioned specific single-object detection model.

In another embodiment, it is reflected in the determination that the reference single object detection model and the reference single object classification model do not meet the above-mentioned target conditions, and the image database 310 still has unmarked objects of the above-mentioned specific category (for example, a bus) The processor 104 may be configured to: use the reference single object detection model to annotate each second image to obtain a plurality of second reference annotated images; modify the above-mentioned second reference annotated image , And add the revised second reference annotated image to the reference annotated image collection; retrain the reference single object detection model and reference single object classification model based on the reference annotated image collection. The processor 104 can continue to repeat the foregoing steps until the reference single object detection model and the reference single object classification model meet the foregoing target conditions, but it may not be limited thereto. In some embodiments, the method of modifying each second reference annotation image data may include, but is not limited to: adjusting the position/size of the bounding box used to mark a specific type of object in each second reference annotation image data, and deleting unnecessary duplicate borders. Frame, adjust the border of the area where a certain type of object is located, or adjust the outline of the edge of the object.

For example, in one embodiment, assuming that there is no annotated map resource set GN in the map database 310 originally, the processor 104 may execute the above-mentioned map resource set adding mechanism to add a map corresponding to " Annotated drawing data collection GN of "Object N". After that, the processor 104 may train the single-object detection model N ₀ and the single-object classification model N′ based on the annotated image data set GN. Furthermore, after the single-object detection model N ₀ and the single-object classification model N′ meet the above target conditions individually, the processor 104 may treat the single-object detection model N ₀ as a specific single-object detection model, but it may not be limited to this. .

In step S250, the processor 104 may find a plurality of specific annotated image sets corresponding to the plurality of specified detection object categories among the plurality of annotated image sets GA~GH, GN, and train them accordingly Joint object detection model 324.

In the scenario of FIG. 3, the plurality of specific annotation map resource sets corresponding to the plurality of designated detection object categories include, for example, annotation map resource sets GA to GE, GN, and they can form an integrated annotation map resource set 311. After that, the processor 104 can train the joint object detection model 324 based on the integrated annotated image resource set 311. In this case, the joint object detection model 324 should be able to detect "object A", "object B", "object C", "object D", "object E" and "object N".

In an embodiment of the present invention, the processor 104 may perform the following steps on one or more unlabeled image assets in the image asset library 310. For the convenience of description, the following will describe a first unlabeled image among the unlabeled images, but the present invention may not be limited to this.

Then, in step S260, the processor 104 may use the plurality of specific single object detection models (ie, single object detection models A ₀ ~ E ₀ and N ₀ ), the plurality of external object detection models ext1 ~ext3 and the combined object detection model 324 detect the first unlabeled image to generate multiple first object detection results, wherein the multiple first object detection results correspond to the multiple specified detection results The first specified detection object category in the detection object category.

In FIG. 3, the detection result 341 may include, for example, the detection of the first unlabeled image data by the plurality of specific single-object detection models (ie, single-object detection models A ₀ ~ E ₀ and N ₀ ) the result of. As shown in FIG. 3, the detection result 341 may include the codes A ₀ ~ E ₀ and N ₀ enclosed by boxes, which means that the single object detection models A ₀ ~ E ₀ and N ₀ are respectively shown in the first unlabeled figure. Objects belonging to "Object A", "Object B", "Object C", "Object D", "Object E" and "Object N" are detected in the data, and each detected object can be mapped to The single-object detection model of is marked with bounding boxes or other similar methods, and can have corresponding detection confidence values.

In addition, the detection result 342 may include, for example, the detection result of the first unlabeled image data by the plurality of external object detection models ext1 to ext3. As shown in FIG. 3, the detection result 342 may include the codes A ₁ , B ₁ , C ₂ , A ₃ , D ₃ and E ₃ framed by boxes, where A ₁ , B ₁ represent the external object detection model ext1 in Objects belonging to "Object A" and "Object B" are detected in the first unlabeled image; C ₂ represents the external object detection model ext2 detected objects belonging to "Object C" in the first unlabeled image ; A ₃ , D ₃ and E ₃ represent that the external object detection model ext3 detects objects belonging to "object A", "object D", and "object E" in the first unlabeled image. Similarly, in the first unlabeled image, each object detected by the external object detection model ext1~ext3 can be marked with a bounding box or other similar methods by the corresponding external object detection model, and can have a corresponding The detection confidence value.

In addition, the detection result 343 may include, for example, the detection result of the first unlabeled image data performed by the joint object detection model 324. As shown in Figure 3, the detection result 343 may include the codes A ₄ ~ E ₄ and N ₄ framed by squares, which means that the joint object detection model 324 has detected that it belongs to the "object" in the first unlabeled image. A", "Object B", "Object C", "Object D", "Object E" and "Object N". Similarly, in the first unlabeled image, each object detected by the combined object detection model 324 can be labeled by the combined object detection model 324 with a bounding box or other similar methods, and can have a corresponding detection. Confidence value.

For ease of description, the following assumes that the first designated detection object category is "Object A". In this case, the detection results of the plurality of first objects obtained by the processor 104 in step S260 include, for example, the code A ₀ in the detection result 341, the codes A ₁ and A ₃ in the detection result 342, and the detection results. The code in the test result 343 is A ₄ , and it can each have a corresponding first detection confidence value.

After that, in step S270, the processor 104 may use the first single-object classification model corresponding to the first designated detection object category among the plurality of single-object classification models to predict and classify the detection results of each first object to obtain The first object classification result of each first object detection result.

Continuing from the above example, if the first designated detection object category is assumed to be "Object A", the processor 104 can use the single object classification model A'as the first single object classification corresponding to the first designated detection object category Model. After that, the processor 104 may use the single-object classification model A'to predict and classify the first object detection result to obtain the first object classification result of each first object detection result.

In an embodiment, the processor 104 may use the single-object classification model A′ to predict and classify the code A _{0 in the detection result 341.} For example, the processor 104 may use the single-object classification model A'to predict and classify the image area selected by the single-object detection model A ₀ in the first unlabeled image to obtain a corresponding first classification confidence value as For the first object classification result of this image area, the first classification confidence value can represent the accuracy of the single object detection model A _{0 detection.}

In another embodiment, the processor 104 may use the single-object classification model A′ to predict and classify the code A _{4 in the detection result 343.} For example, the processor 104 may use the single object classification model A'to predict and classify the image area selected by the joint object detection model 324 in the first unlabeled image to obtain a corresponding first classification confidence value as a reference The first object classification result of this image area, where the first classification confidence value can represent the correctness of the joint object detection model 324 detection.

Regarding the codes A ₁ and A ₃ in the detection result 342, the processor 104 can use the single-object classification model A'to execute a similar mechanism to the above to obtain the first object classification result corresponding to each of them. The details are not mentioned here. Go into details.

Afterwards, in step S280, the processor 104 may adaptively correct the detection results of the plurality of first objects based on the detection results of each first object and the corresponding first object classification results, and combine the corrected plurality of first object detection results. The first object detection result is added to the first annotation image data set (for example, annotation image data set GA) corresponding to the first specified detection object category (for example, "object A") in the plurality of annotation image data sets.

In the first embodiment, the first detection confidence value reflecting the determination that the detection result of each first object is higher than the first detection confidence threshold, and the similarity of the detection results of the plurality of first objects to each other is satisfied Similarity conditions, and the detection results of each first object are right The corresponding first classification confidence values are all higher than the first classification confidence threshold, the processor 104 may accordingly determine that the detection results of the plurality of first objects do not need to be corrected, and add the detection results of the plurality of first objects The first annotated image collection (for example, annotated image collection GA).

In some embodiments, the above-mentioned similarity condition may include "the frame overlap rate (Intersection over Union) is greater than the overlap rate difference reference threshold", but it may not be limited thereto.

In the second embodiment, the first detection confidence value reflecting the determination that any one of the first object detection results is not higher than the first detection confidence threshold value is that two of the plurality of first object detection results If the similarity does not satisfy the above-mentioned similarity condition, or the first classification confidence value corresponding to any of the first object detection results is not higher than the first classification confidence threshold, the processor 104 may determine that the plurality of first objects are detected accordingly The result needs to be corrected, and the corrected detection results of the plurality of first objects are added to the first annotation collection (for example, the annotation collection GA).

After that, in step S290, the processor 104 may retrain the first single-object detection model corresponding to the first specified detection object category among the plurality of single-object detection models (for example, single-object The object detection model A ₀ ), the first single object classification model (for example, the single object classification model A′), and the joint object detection model 324.

In some embodiments, the processor 104 may repeatedly execute steps S260 to S290 for other unlabeled images in the image library 310 until the joint object detection model 324 meets one or more target conditions set by the designer.

For example, for a second unlabeled image in the image library 310, the processor 104 may be configured to: use the plurality of specific single object detection models, the plurality of external object detection models, and the joint detecting a second object model for detecting unlabeled map data to generate a plurality of second object detection result (e.g. code detection result B _{0 341,} 342 of the detection result and the detection result of the code B ₁ Code B _{4 in} 343), wherein the detection results of the plurality of second objects correspond to a second specified detection object category (for example, "object B") among the plurality of specified detection object categories; A second single-object classification model (for example, single-object classification model B') corresponding to the second designated detection object category among the plurality of single-object classification models predicts and classifies the detection results of each second object to obtain each second object The second object classification result of the detection result; based on each second object detection result and the corresponding second object classification result, the plurality of second object detection results are adaptively corrected, and the plurality of corrected The second object detection result is added to the second annotated image resource set (for example, annotated image resource set GB) corresponding to the second specified detection object category in the multiple annotated image resource sets; the training institute is retrained based on the second annotated image resource set A second single-object detection model (for example, a single-object detection model B ₀ ), a second single-object classification model, and a joint object detection model corresponding to the second designated detection object category among the plurality of single-object detection models 324 .

In different embodiments, the target conditions to be met by the joint object detection model 324 may include "the number of new plots in the training plot is greater than the number of new plots in the plan", and the "average object prediction area border and original label The area border overlap rate (Intersection over Union) is greater than the target threshold of the overlap rate", "The average precision of each single object feature category (Average Precision) is greater than the single object average accuracy target threshold", "All available objects of the joint object detection model 324 The Mean Average Precision of the detected object feature category is greater than the model average precision target threshold At least one of "value", "model accuracy (accuracy) greater than model accuracy target threshold", "model loss function less than model loss function target threshold", but not limited to this.

In some embodiments, at the initial stage of the training of the joint object model 324, the processor 104 may use the mechanism described in the first embodiment to add the uncorrected detection results of the plurality of first objects when performing step S280 The first annotated image collection (for example, annotated image collection GA), so as to relatively quickly improve the detection capability of the joint object detection model 324.

In addition, in the middle of the training of the joint object model 324, the processor 104 may use the mechanism described in the second embodiment when executing step S280 to add the corrected detection results of the plurality of first objects to the first annotated image Data collection (for example, annotated map data collection GA) to further enhance the detection capability of the joint object detection model 324 by using more diverse training data.

In addition, in the later stage of the training of the joint object detection model 324, because there may be a lack of content in some annotated image collections (for example, annotated image collection GA), the combined object detection model 324 is therefore The detection capability of objects of the category (such as "Object A") is relatively poor. Therefore, in some embodiments, the processor 104 may only request the specific single object detection models, the multiple external object detection models, and the joint object detection model to detect only those belonging to the above when performing step S260. Specify objects of the detection object category to increase the amount of corresponding training data.

For example, assuming that the content of the annotated image collection GA is relatively lacking, the processor 104 may request the corresponding first single-object detection model (ie, single-object detection model A ₀ ) and external object detection model when performing step S260 ext1~ext3 and the joint object detection model 324 detect other unlabeled images, and add relevant detection results to the labeled image collection GA according to the previous teaching. After that, the processor 104 can train the joint object detection model 324 (and the single object detection model A ₀ ) according to the annotated image data set GA to enhance the ability of the joint object detection model 324 to detect "object A", but it may not Limited to this.

In some embodiments, if the processor 104 determines that the joint object detection model 324 has met the aforementioned target conditions, the processor 104 may determine that the training of the joint object detection model 324 has been completed. In this case, the trained joint object detection model 324 can be used to perform "object A", "object B", "object C", "object D", "object E" and "object N" detection, but not limited to this.

In summary, the present invention has at least the following features.

(1) Cumulative labeling results, flexible combination application: multiple existing training labeling image data sets can be separated, classified, and re-assembled for each object feature category to create a labeling data set for each single object feature category, and train Generate single-object detection models corresponding to each single-object feature category. In this way, image annotation results can be accumulated, subsequent update and maintenance costs can be simplified, and model efficiency and combination flexibility for establishing new applications can be increased.

(2) Existing model results can be integrated: For models that do not provide training data sets, predict and label the existing map database through the predictive labeling module. In addition to obtaining the prediction accuracy of each object feature category of the model, It is also possible to reversely verify whether the data in the existing corresponding labeled data set is accurate, and correct the labeled data. When should The model has a feature category of an object that is not labeled in the annotated map data set. According to the characteristics of the present invention, the corrected annotation data of the feature category of the object can be stored in the annotated map data set of the feature category of an object to realize the integration of existing model results .

(3) Use existing models to compare and filter data: Existing machine learning object detection models within or outside the organization can be used to help filter unlabeled images, and can also be used as comparison information for target model prediction and labeling results to help detect and Establish annotated map data set of each single object feature category.

(4) The single-object detection model is used as the prediction comparison target: the advantage of using the AI detection model of single object feature category to compare with the AI detection model of multiple object feature categories, has faster training convergence, quick and easy map data labeling The advantages of diversification, higher accuracy, and faster prediction speed can be the comparison target for the main AI detection model prediction, replacing the initial map data selection, labeling, and review manpower.

(5) Can quickly and flexibly expand the training and labeling image data: when there is a need to expand the feature category of new objects, a small amount of image data can be labeled from the existing image database, and the object detection model of a single object feature category can be created , And then through the iterative method of predictive annotation module screening and expansion of annotation training image resources to complete all new annotation image resources and refined models.

(6) A menu of object feature categories can be provided to automatically generate training images and application models: select the object feature categories to be detected according to the application needs, and initially can quickly combine multiple object features that meet the requirements of the new application from the set of labeled images The training annotated image data of the category, and the required joint object detection model is obtained after training. Subsequent to the object detection model of each single object feature category in the demand combination, the new investment is not marked The map data is predicted, and then the multi-model prediction result comparison mechanism is used to mark the map data expansion and refine the target joint model.

(7) Multi-model prediction result comparison mechanism: The multi-model prediction result comparison mechanism of the present invention can more accurately select the required image data for labeling; through comparison, new images with inaccurate AI model predictions are selected and then put into training. Make training models more efficient; quickly filter the image data of specific objects needed to solve the problem of biased training maps; help reduce the cost of refining the AI object detection model, reduce the iterative training cycle of the target model, and accelerate the commercialization of AI models.

(8) Provide a review mechanism for the object category of the prediction result: by creating an object classification model of a single object feature category, the area image within the object bounding box of the detection result of the object detection model of the same object feature category can be rechecked , Whether it is the correctness of the feature category of the object.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be subject to those defined by the attached patent scope.

S210~S290: steps

Claims (14)

A method for optimizing a joint object detection model, wherein the method is executed by a processor of a server, and the method includes: obtaining an image database, wherein the image database includes a first unlabeled image data and a plurality of annotations Map data sets, each of the labeled map data sets corresponds to a single type of object; training multiple single-object detection models and multiple single-object classification models with these labeled map data sets, wherein the labeled map data sets are one-to-one Corresponding to the single-object detection models, and the labeling data sets correspond to the single-object classification models one-to-one; obtain multiple designated detection object categories, and obtain multiple external object detection models accordingly , Wherein each of the external object detection models is used to detect objects belonging to at least one of the specified detection object categories; in response to determining that each of the detection object categories corresponds to one of the annotation image data sets , From the single-object detection models, find out the specific single-object detection models corresponding to the specified detection object categories; find out the multiple specific detection object categories corresponding to the specified detection object categories in the annotation map data sets A set of specific labeled image data, and training a joint object detection model accordingly; use the specific single object detection models, the external object detection models, and the combined object detection model to the first unlabeled image data Perform detection to generate a plurality of first object detection results, where the first object detection results correspond to a first specified detection object category among the specified detection object categories; use the single object classification A first single object classification model corresponding to the first designated detection object category in the model predicts and classifies the detection results of each first object To obtain a first object classification result of each of the first object detection results; adaptively correct the first object detection results based on each of the first object detection results and the corresponding first object classification results, and Add the corrected detection results of the first objects to a first annotation image data set corresponding to the first designated detection object category in the annotation image data sets; retrain the first annotation image data sets based on the first annotation image data set In the single-object detection model, a first single-object detection model corresponding to the first designated detection object category, the first single-object classification model, and the combined object detection model. The method according to claim 1, wherein in response to determining that a reference designated detection object category in the detection object categories does not correspond to any one of the annotation image data sets, the method further includes: A reference annotated image collection corresponding to the reference specified detection object category is newly added to the annotation image data sets of the database; a plurality of first image resources are obtained from the image library and placed in each of the first image resources Annotate a specific type of object corresponding to the reference designated detection object category to generate a plurality of first reference annotated images; train a reference list object detection model and a reference list based on the first reference annotated images Object classification model; in response to determining that the reference single object detection model and the reference single object classification model meet at least one target condition individually, the reference single object classification model is added to the single object detection models, and the reference single object classification model is added to the single object detection models. The reference single object classification model is added to the single object classification models and the specific single object detection models. The method according to claim 2, wherein it is reflected in determining that the reference list object detection model and the reference list object classification model do not meet the at least one target condition, and the image database still has not been processed for the specific type of object The method further includes: marking each of the second drawing materials with the reference list object detection model to obtain a plurality of second reference marking drawing materials; revising the second reference drawing materials Annotate images, and add the revised second reference annotated images to the reference annotated image collection; retrain the reference list object detection model and the reference list object classification model based on the reference annotated image collection . The method according to claim 1, wherein the image database further includes a second unlabeled image asset, and the method further includes: using the specific single object detection models, the external object detection models, and the The joint object detection model detects the second unlabeled image data to generate a plurality of second object detection results, wherein the second object detection results correspond to a first of the specified detected object categories 2. Specify the detection object category; use a second single object classification model corresponding to the second specified detection object category in the single object classification models to predict and classify the detection results of each second object to obtain each second object A second object classification result of two object detection results; based on each of the second object detection results and the corresponding second object classification results, the second object detection results are adaptively corrected, and the corrected The detection results of some second objects are added to the annotation map data set corresponding to the second designated detection object category Based on the second annotated image data set; retrain a second single object detection model corresponding to the second designated detection object category in the single object detection models based on the second annotated image data set, the second Single object classification model and the combined object detection model. The method according to claim 1, wherein in response to determining that the joint object detection model has met at least one target condition, the method further includes: determining that the training of the joint object detection model has been completed. The method according to claim 1, wherein each of the first object detection results has a corresponding first detection confidence value, and the first object classification result of each of the first object detection results has a first classification confidence The step of adaptively correcting the detection results of the first objects based on the detection results of each of the first objects and the corresponding classification results of the first objects includes: responding to the determination of the detection results of each of the first objects The first detection confidence value is higher than a first detection confidence threshold, the similarity of the detection results of the first objects meets the similarity condition, and the detection result of each first object corresponds to the first category If the confidence values are all higher than a first classification confidence threshold, it is determined that the detection results of the first objects do not need to be corrected, and the detection results of the first objects are added to the first annotation image data set. The method according to claim 1, wherein each of the first object detection results has a corresponding first detection confidence value, and the first object classification result of each of the first object detection results has a first classification confidence The step of adaptively correcting the detection results of the first objects based on the detection results of the first objects and the corresponding classification results of the first objects includes: In response to determining that the first detection confidence value of any of the first object detection results is not higher than a first detection confidence threshold, the similarity of two of the first object detection results does not satisfy the similarity Condition, or the first classification confidence value corresponding to any one of the first object detection results is not higher than a first classification confidence threshold, it is determined that the first object detection results need to be corrected, and the corrected first object detection results An object detection result is added to the first annotation image collection. A server for optimizing a joint object detection model includes: a storage circuit storing a plurality of modules; and a processor, coupled to the storage circuit, accessing the modules to perform the following steps: obtaining a map database , Wherein the image data library includes a first unlabeled image data and a plurality of annotated image data sets, each of which annotated image data set corresponds to a single type of object; training multiple single-object detection models with these annotated image data sets and A plurality of single-object classification models, wherein the labeled image data sets correspond to the single-object detection models one-to-one, and the labeled image data sets correspond to the single-object classification models one-to-one; A designated detection object category, and a plurality of external object detection models are obtained accordingly, wherein each of the external object detection models is used to detect objects belonging to at least one of the specified detection object categories; in response to the determination Each detection object category corresponds to one of the annotation image data sets, and a plurality of specific single object detection models corresponding to the specified detection object categories are found from the single object detection models; in the Some labeled images are concentrated to find out the categories corresponding to the specified detected objects A set of specific annotation maps of, and training a joint object detection model accordingly; use the specific single object detection models, the external object detection models, and the joint object detection model to the first unlabeled The map data is detected to generate a plurality of first object detection results, where the first object detection results correspond to a first specified detection object category among the specified detection object categories; use the lists In the object classification model, a first single object classification model corresponding to the first designated detection object type performs predictive classification on each of the first object detection results to obtain a first object classification of each of the first object detection results Results; based on the detection results of the first objects and the corresponding classification results of the first objects, adaptively correct the detection results of the first objects, and add the corrected detection results of the first objects to the labels The image data set corresponds to a first annotated image data set of the first designated detection object category; based on the first annotated image data set, the single object detection models are retrained to correspond to the first designated detection object category A first single-object detection model, the first single-object classification model, and the combined object detection model of. The server according to claim 8, wherein in response to determining that a reference-designated detected object type among the detected object types does not correspond to any one of the labeled image data sets, the processor is further configured to: Add a reference annotated image resource set corresponding to the reference designated detection object category to the annotated image resource sets in the image database; Obtain multiple first image data from the image database, and mark a specific type of object corresponding to the reference designated detection object category in each of the first image data to generate multiple first reference annotation image data Training a reference single object detection model and a reference single object classification model based on the first reference annotated images; in response to determining that the reference single object detection model and the reference single object classification model meet at least one target condition individually, The reference single object classification model is added to the single object detection models, and the reference single object classification model is added to the single object classification models and the specific single object detection models. The server according to claim 9, wherein it is reflected in determining that the reference list object detection model and the reference list object classification model do not meet the at least one target condition, and the image database is still not targeted for the specific type of object The processor is further configured to: use the reference single object detection model to annotate each of the second image data to obtain a plurality of second reference image data; The second reference annotated image data, and the revised second reference annotated images are added to the reference annotated image resource set; based on the reference annotated image data set, the reference list object detection model and the reference list are retrained Object classification model. The server according to claim 8, wherein the image database further includes a second unlabeled image asset, and the processor is further configured to: use the specific single object detection models and the external object detection Model and the The joint object detection model detects the second unlabeled image data to generate a plurality of second object detection results, wherein the second object detection results correspond to a first of the specified detected object categories 2. Specify the detection object category; use a second single object classification model corresponding to the second specified detection object category in the single object classification models to predict and classify the detection results of each second object to obtain each second object A second object classification result of two object detection results; based on each of the second object detection results and the corresponding second object classification results, the second object detection results are adaptively corrected, and the corrected The detection results of the second objects are added to a second annotated image data set corresponding to the second designated detection object category in the annotated image data sets; the single-object detection models are retrained based on the second annotated image data set A second single-object detection model, the second single-object classification model, and the combined object detection model corresponding to the second designated detection object category in. The server according to claim 8, wherein in response to determining that the joint object detection model has met at least one target condition, the processor is further configured to determine that the training of the joint object detection model has been completed. The server according to claim 8, wherein each of the first object detection results has a corresponding first detection confidence value, and the first object classification result of each of the first object detection results has a first classification A confidence value, and the processor is configured to: reflect that the first detection confidence value of each of the first object detection results is higher than a first detection confidence threshold, and the first object detection results are mutually exclusive The similarity of both If the similarity condition is met, and the first classification confidence value corresponding to each of the first object detection results is higher than a first classification confidence threshold, it is determined that the first object detection results do not need to be corrected, and the first object detection results are An object detection result is added to the first annotation image collection. The server according to claim 8, wherein each of the first object detection results has a corresponding first detection confidence value, and the first object classification result of each of the first object detection results has a first classification A confidence value, and the processor is configured to: respond to determining that the first detection confidence value of any of the first object detection results is not higher than a first detection confidence threshold, the first object detection results The similarity of two of them does not meet the similarity condition, or the first classification confidence value corresponding to any of the first object detection results is not higher than a first classification confidence threshold, then the first object detection results are determined It needs to be corrected, and the corrected detection results of the first objects are added to the first marked image collection.

Images