TWI794132B - System for detecting misidentified objects - Google Patents

Abstract

A system for detecting misidentified objects comprises a client equipment and a servo equipment. The client equipment performs an elementary image comparing program to determine whether a candidate image area comprising a preset image characteristic exists in an image content, and outputs the candidate image area after it is found. The servo equipment is electrically coupled to the client equipment and receives the candidate image area output therefrom. The servo equipment performs an advanced image comparing program to determine whether the preset image characteristic is included in the received candidate image area, and, when the preset image characteristic is not included in the received candidate image area, outputs the received candidate image area to indicate that a misidentified object is found in the candidate image area.

Description

System for Detecting Misidentified Objects

The present invention relates to a system capable of detecting misjudged objects, in particular to a system for automatically detecting misjudged objects based on the most advanced model.

In artificial intelligence object recognition technology, how to reduce the probability of object recognition errors is a very important issue. For this reason, the earliest technology that appeared was to manually inspect the images used in the process of artificial intelligence object recognition, and then compare the results of artificial intelligence object recognition with the results of manual inspection to find out the differences, and finally Then modify the model content of artificial intelligence or the database used in training artificial intelligence according to the difference results obtained from the comparison, thereby increasing the accuracy of artificial intelligence object recognition. However, since all the scenes must be viewed once to find out the misjudgment, once the number of images to be inspected increases, it is impossible to quickly find the problem. It can be seen that this method is very time-consuming and labor-intensive.

In order to avoid the above-mentioned problems, an object recognition method based on a state-of-the-art model (State of the Arts model, SOTA model) is adopted in recent technologies to highly increase the recognition accuracy. After experiments, it was found that this method of object recognition can indeed increase the accuracy of object recognition, but due to the large number of weights used in the recognition process, it will cause the problem of too long calculation time. For example, for a video When performing object recognition operations, it may take tens of times the time of the video to analyze the results. Therefore, it is obvious that using this technique in object recognition still consumes a lot of time.

In order to solve the above problems, an object of the present invention is to provide a system for detecting misjudged objects, which can automatically find a considerable number of misjudged objects in a relatively short period of time, reducing the time spent on finding misjudged objects.

From one point of view, the present invention proposes a system for detecting misjudged objects, which is characterized in that it includes a client device and a server device, wherein the client device is suitable for obtaining image content, and executing a primary image comparison program to compare images in the image. Find whether there are candidate area tiles containing pre-set preset image features in the content, and output the candidate area tiles after finding qualified candidate area tiles; the server device is electrically coupled to the client The device receives the candidate area tiles output by the client device, and judges whether the received candidate area tiles contain the aforementioned preset image features by executing an advanced image comparison program, and the candidate area tiles do not contain When the preset image features are used, the candidate area block is output to indicate that the candidate area block causes object misjudgment. Among them, the accuracy rate of the advanced image comparison program in judging whether a candidate region block contains a preset image feature is greater than the accuracy rate of the primary image comparison program in judging whether the same candidate region block contains the same preset image feature .

In one embodiment, the above-mentioned client device includes an image input device and obtains the above-mentioned image content through the image input device.

In one embodiment, the above-mentioned image input device is a photographic equipment, and the photographic equipment generates the above-mentioned image content through a shooting operation.

In one embodiment, before the above-mentioned server device executes the advanced image comparison program to determine whether the candidate area block contains preset image features, it further executes a feature enhancement comparison program to enlarge the candidate area block and Analyzing the enlarged candidate area tiles to determine whether the enlarged candidate area tiles contain the same preset image feature, wherein only those that are judged not to include the preset image feature by the feature enhancement comparison program The candidate area tiles will be provided to the advanced image comparison program for subsequent operations.

In one embodiment, before the above-mentioned client device outputs the above-mentioned candidate area tiles, it further executes a feature enhancement comparison program to enlarge the candidate area tiles and analyze the enlarged candidate area tiles to determine the Whether the enlarged candidate area tiles also contain the above-mentioned preset image features, wherein only the candidate area tiles that are determined not to include the above-mentioned preset image features by the feature enhancement comparison program will be output by the client device.

In one embodiment, when the above-mentioned server device determines that the received candidate area block does not contain preset image features, the candidate area block is used as a negative sample that does not contain preset image features to train the initial Image comparison program.

By adopting the above-mentioned technology, the system for detecting misjudged objects provided by the present invention can perform preliminary and relatively simple object detection on images first, and then carry out relatively complex and accurate advances on the images of the selected objects Identify operations. In this way, not only can the burden of advanced identification operations be reduced and thus the time required for advanced identification operations can be reduced, but also objects that are easily misjudged during the identification process can be automatically found according to the different results obtained from multiple identification operations before and after , to reduce the manpower requirement of finding misjudged objects manually.

In order for those skilled in the art to understand the following content clearly, it should be explained here that the phrase "electrically coupled" used in this application indicates that signals can be electrically coupled between two objects transfer between. Unless otherwise limited, the way of transmitting electronic signals can be wired or wireless, and the transmission direction of electronic signals can be one-way or two-way.

Please refer to FIG. 1 , which is a circuit block diagram of a system for detecting misjudged objects according to an embodiment of the present invention. As shown in the figure, the system 10 for detecting misjudged objects in this embodiment includes a client device 100 and a server device 150, wherein the client device 100 can be electrically coupled via various electronic signal transmission methods to the server device 150 so that electronic signals can be transmitted between the client device 100 and the server device 150 .

Further, the client device 100 in this embodiment includes a processor 102, a memory module 104 and an image input device 106, wherein the processor 102 is electrically coupled to the memory module 104 and the image input device respectively 106 , and the memory module 104 is also electrically coupled to the image input device 106 . In some embodiments, the image input device 106 is a photographic device capable of generating corresponding image content by photographing a specific area, such as a monitor, a camera, or a video camera. In these embodiments, the shooting range of the image input device 106 may be fixed, manually adjustable by the user, or adjustable by the image input device 106 according to the control of the processor 102 . Depending on the design, the image content generated by the operation of the image input device 106 by shooting a specific area can be processed by the processor 102 before being stored in the memory module 104, or the generated image content can also be firstly stored in the memory module 104. The image input device 106 is directly stored in the memory module 104, and then the processor 102 takes it out from the memory module 104 at an appropriate time and performs necessary processing.

In other embodiments, the image input device 106 is a data transmission device that allows external data to be connected to the client device 100, such as a card reader, a Universal Serial Bus (USB, Universal Serial Bus) connector or RJ-45 connector, etc. By connecting to the image input device 106 , the client device 100 can obtain image content that will be used in subsequent image analysis from an external data source.

After obtaining the required image content through the image input device 106, the client device 100 will execute a preliminary image comparison program to find whether there are tiles containing preset image features set in advance in the image content (later called region proposal tiles). In this embodiment, the primary image comparison program may be stored in the memory module 104 in advance, and read out from the memory module 104 by the processor 102 and executed after the client device 100 is activated. This elementary image comparison program can be a computer program formed on the basis of a deep neural network, which can extract the image features of the target image and compare the extracted image features with the aforementioned preset image features Or, as known to those skilled in the art, the primary image comparison program can also be other computer programs made in any way and having the aforementioned ability to capture and compare image features.

On the other hand, the previously mentioned server device 150 mainly includes a processor 152 and a memory module 154 in this embodiment, and the server device 150 has an electronic signal transmission module to communicate via a suitable communication protocol To receive data transmitted from the client device 100 or transmit data to the client device 100 . Depending on the design, the data received from the client device 100 via the electronic signal transmission module can be directly sent to the processor 152 for the required processing, or can be stored in the memory module 154 first and then stored in the memory module 154. The processor 152 takes it out from the memory module 154 at an appropriate time and processes it. The result obtained after being processed by the processor 152 can be output to the outside through the output interface 156 for the user to know the relevant information. In an embodiment, the server device 150 may have a stronger computing capability than the client device 100 to perform larger and more complex operations.

In order to make the technical content of the present invention more easily understood by those skilled in the art, a more detailed description will be given below with reference to FIG. 1 and FIG. 2 . Wherein, FIG. 2 is a flow chart of the operation of the system for detecting misjudged objects according to an embodiment of the present invention. As shown in FIG. 2 , firstly, the client device 100 will obtain the image content (step S200 ), and then the client device 100 will process the acquired image content by executing a primary image comparison program, including: converting the image The content is divided into more than one image area tiles, and the image features of each image area block in the image content are sequentially extracted, and the extracted image features are compared with the preset image features that have been set before ( step S202). Next, when all the image region tiles in the image content are processed through step S202 and it is found that none of the image region tiles has an image feature that matches the preset image feature, this embodiment will return the process to step S200 In order to obtain the image content that needs to be processed next (that is, the judgment of step S204 is No); in contrast, during the processing of the image area tile in the image content in step S202, it is found that there is an image of a certain image area block When the feature matches the preset image feature (that is, the judgment of step S204 is yes), the client device 100 will determine the subsequent operation mode according to whether all the image content has been processed this time.

When the determination of the aforementioned step S204 is yes, the process will enter into step S206 to confirm whether all the image contents acquired so far have been processed. When all the currently acquired image content has been processed, in addition to sending the image area tiles found in step S204 that meet the preset image characteristics (that is, the aforementioned candidate area tiles) from the client device 100 In addition to sending to the server device 150 (step S208), the client device 100 will return to step S200 to prepare for obtaining the next image content and perform corresponding processing; After completion, in addition to transmitting the candidate area tiles found in step S204 from the client device 100 to the server device 150 (step S208), the client device 100 will return to step S202 to continue to confirm the image content Whether there are other candidate region tiles.

After step S208 is executed, the server device 150 will receive the candidate area tiles sent from the client device 100, and the processor 152 will execute the aforementioned advanced image comparison program at an appropriate time point to compare all the candidate regions. Necessary processing is performed on the received candidate area tiles (step S210 ). Similar to the aforementioned primary image comparison program, the advanced image comparison program can be formed based on a deep neural network, which can extract the image features of the target image and compare the extracted image features with the A computer program for comparing the aforementioned preset image features; or, as known to those skilled in the art, the advanced image comparison program can also be made in any way and has the aforementioned ability to capture and compare image features computer program. Unlike the basic image comparison program, the advanced image comparison program should have a higher accuracy rate than the primary image comparison program when comparing image features, for example, it has more weight items, including More layers of neural networks, or the most advanced model (State of the Arts model, SOTA model) as the basis for program construction, etc. In one embodiment, because the advanced image comparison program consumes more computing resources than the elementary image comparison program, if the advanced image comparison program is executed on the client device 100 with weak computing power, it may be too slow , but it can be executed smoothly on the server device 150 with strong computing capability.

After the candidate area block is processed by the advanced image comparison program in step S210, the server device 152 can use the processing result to determine whether the candidate area block contains the aforementioned preset image features (step S212). Since the advanced image comparison program has a higher accuracy in image feature comparison, some candidate area tiles that were originally judged to contain preset image features by the primary image comparison program may be changed to not Contains preset image features; once there is a discrepancy between the judgment results of the basic image comparison program and the advanced image comparison program, it means that the candidate area block is very likely to be caused by the primary image It is selected from the misjudgment of objects generated by the comparison program. Therefore, once it is found in step S212 that the judgment result of the advanced image comparison program is that the candidate area block does not contain the preset image feature (that is, the judgment result of step S212 is No), then the server device 150 will Corresponding information is output through the output interface 156 to prompt and inform that the candidate area block has a great possibility of making the preliminary image comparison program produce an object misjudgment result (step S214); in contrast, the advanced image is found in step S212 The judging result of the comparison program is also that the candidate region block contains the preset image features (the judgment result of step S212 is Yes), then the server device 150 can consider that the primary image comparison program has a certain value for this candidate region. There is no misjudgment problem in the identification of the block, and therefore the related processing procedure of the block in the candidate area is ended. In one embodiment, when the advanced image comparison program of the server device 150 judges that the received candidate area tile does not contain the preset image feature, the candidate area tile can be further used as the candidate area block that does not include the default image. The negative samples of the features are used to train the deep neural network of the basic image comparison program to improve its accuracy.

By adopting the above-mentioned technology, although the system provided by the present invention still uses an advanced image comparison program with high accuracy but slow recognition speed (such as but not limited to the image recognition program based on the most advanced model), but because of the need The image content judged by the advanced image comparison program is only the candidate area tiles first filtered by the elementary image comparison program instead of all the image content, so the amount of data that needs to be processed by the advanced image comparison program is compared to Compared with the prior art, the amount of data to be processed must be reduced a lot, so the time that must be spent is also naturally much less than the prior art. Therefore, using the system provided by the present invention can obviously reduce the time cost required to find out the images that cause object misjudgment. In addition, by pointing out the tiles that produce different comparison results in the two image comparisons, the user can easily know that these tiles have a high probability of causing misjudgment of the object, in addition to directly using the finally selected tiles to enhance In addition to training the corresponding neural network, even if it needs to be manually screened again, it can also achieve the benefit of reducing labor costs because there are fewer objects to be screened.

In addition to the above-mentioned techniques, the system 10 proposed by the present invention can also execute other image comparison programs between the primary image comparison program and the advanced image comparison program to further reduce the interpretation required by the advanced image comparison program amount of data.

Please refer to FIG. 1 , FIG. 2 and FIG. 3A together, wherein FIG. 3A is a detailed flow chart of executing step S208 according to an embodiment of the present invention. As shown in FIG. 3A , after executing steps S204 and S206 , the client device 100 will further execute a feature enhancement comparison program to process the candidate area tiles found in step S204 (step S208 ). This feature enhancement comparison program can first perform targeted enhancement on the candidate region tiles according to the attributes of the image features to be compared. For example, when the object to be judged is a pedestrian, you can choose to enlarge the block of the candidate area to make the face or human body features easier to be recognized as a part of the feature enhancement comparison program; or, when the object to be judged When the object is in a specific shape, you can choose to enhance the brightness and darkness of the candidate area tiles to make the edge of the object more obvious as a part of the feature enhancement comparison program. In a word, the feature enhancement comparison program can be modified according to requirements, without being limited by the types of image features mentioned in the embodiments.

After the candidate area block is enhanced, the image feature of the enhanced candidate area block is used to compare with the preset image feature. Next, the client device 100 determines a subsequent processing method for the candidate area block according to whether the enhanced candidate area block contains preset image features (step S302 ). The judgment result of step S302 is yes, that is, when the candidate region block is judged by the feature enhancement comparison program to contain the aforementioned preset image features, the feature enhancement comparison program will remove the candidate region map block, so this candidate area block will not be sent to the server device 150 (step S304); on the contrary, the judgment result in step S302 is no, that is, the candidate area block is compared with the aforementioned feature enhancement When the program determines that it does not contain the aforementioned preset image feature, the feature enhancement comparison program will send the candidate area block to the server device 150 (step S306 ).

By adopting the technique shown in FIG. 3A , the number of candidate region tiles sent to the server device 150 can be further reduced. Therefore, although the time spent on running the feature enhancement comparison program may be increased, it can also be reduced at the same time. The amount of data that a high-level image comparison program needs to process and the time it takes.

It should be noted that although the technology of FIG. 3A can be used to reduce the amount of data transmitted from the client device 100 to the server device 150, generally the computing power of the client device 100 is lower than that of the server device 150, so in order To reduce the computing pressure of the client device 100, the aforementioned feature enhancement comparison program can also be changed to run in the server device 150, as shown in FIG. 3B.

Please refer to FIG. 3B , which is a detailed flow chart of executing step S208 in FIG. 2 according to another embodiment of the present invention. As shown in FIG. 3B , after steps S204 and S206 are performed, the selected candidate region tiles are firstly transmitted from the client device 100 to the server device 150 (step S330 ). After the server device 150 receives the block of the candidate area, it will execute the feature enhancement comparison program at an appropriate time to process the block of the candidate area (step S332), including: enhancing the image features of the block of the candidate area in processing, And comparing the image features of the enhanced candidate region tiles with the pre-set preset image features.

Next, the server device 150 determines a subsequent processing method for the candidate region block according to whether the enhanced candidate region block contains preset image features (step S334 ). The judgment result of step S334 is yes, that is, when the feature enhancement comparison program determines that the candidate area block contains the aforementioned preset image features, the feature enhancement comparison program will remove the candidate area block, Therefore, this candidate area block will not be the data to be processed by the advanced image comparison program (step S336); on the contrary, the judgment result in step S334 is no, that is, the candidate area block is enhanced by the feature ratio When the program judges that it does not contain the aforementioned preset image features, the feature enhancement comparison program will provide the candidate area tiles to the advanced image comparison program so that the advanced image comparison program can compare this candidate area Subsequent operations are performed on the block (step S210).

In addition to the above description, as known to those skilled in the art, more enhancements that can exert features similar to those mentioned in the above-mentioned embodiments can be inserted between the primary image comparison program and the advanced image comparison program. For other image comparison programs that compare the effects of the programs, the number of similar image comparison programs to be added can be adjusted according to actual needs, and does not need to be limited by the details described in the above-mentioned embodiments.

In addition, experiments have proved that by adopting the techniques in the above-mentioned embodiments, the amount of data that needs to be processed by the advanced image comparison program can be reduced to about 15%-25% of the original input image content. The number of blocks containing misjudged objects that can be preserved is about 75% to 85% of the number of blocks containing misjudged objects found manually. It can be seen that implementing the technology provided by the present invention can indeed reduce the amount of data to be processed without using manual labor, and can also retain enough error samples for subsequent use.

Overall, by adopting the above-mentioned technology, although the advanced image comparison program with high accuracy but slow recognition speed is still required in the process, the content of the image that needs to be judged by the advanced image comparison program is enlarged The magnitude is reduced, and the advanced image comparison program that consumes more computing resources is executed on the server device with stronger computing power, so the time that must be consumed is naturally much less than that of the prior art. Therefore, using the system provided by the present invention can obviously reduce the time cost required to find out the images that cause object misjudgment. In addition, by pointing out the tiles that produce different comparison results in the two image comparisons, the user can easily know that these tiles have a high probability of causing misjudgment of the object without adding manual processing, and directly convert the final The selected tiles are used to reinforce and train the corresponding neural network. Further, even if the selected blocks need to be manually screened more accurately, the result of reducing the labor cost can be achieved because the objects to be screened are obviously reduced.

10: A system for detecting misjudged objects 100: client device 102, 152: Processor 104, 154: memory module 106: Image input device 150: server end equipment 156: output interface S200～S214: implementation steps of an embodiment of the present invention S300～S306: Execution steps when one embodiment of the present invention executes step S208 S330～S336: Implementation steps when executing step S208 in another embodiment of the present invention

FIG. 1 is a circuit block diagram of a system for detecting misjudged objects according to an embodiment of the present invention. FIG. 2 is a flow chart of the operation of the system for detecting misjudged objects according to an embodiment of the present invention. FIG. 3A is a detailed flowchart of executing step S208 of FIG. 2 according to an embodiment of the present invention. FIG. 3B is a detailed flow chart of executing step S208 in FIG. 2 according to another embodiment of the present invention.

10: A system for detecting misjudged objects

100: client device

102, 152: Processor

104, 154: memory module

106: Image input device

150: server end equipment

156: output interface

Claims (6)

A system for detecting misjudged objects is characterized by comprising: A client device, adapted to obtain an image content, the client device executes a preliminary image comparison program to find whether there is a candidate area block in the image content, and outputs after finding the candidate area block The candidate area block, wherein the candidate area block includes a preset image feature set in advance; and A server device, electrically coupled to the client device to receive the candidate area block output by the client device, the server device judges the received candidate area by executing an advanced image comparison program Whether the block contains the preset image feature, and when the candidate area block does not contain the default image feature, output the candidate area block to indicate that the candidate area block causes object misjudgment, Wherein, the accuracy rate of the advanced image comparison program when judging whether the candidate region block contains the preset image feature is greater than the accuracy rate of the primary image comparison program when judging whether the candidate region block contains the preset image feature Accuracy. The system as claimed in claim 1, wherein the client device includes an image input device to obtain the image content through the image input device. The system according to claim 2, wherein the image input device is a photographic equipment, and the photographic equipment generates the image content through a shooting operation. The system as described in claim 1, wherein the server device further executes a feature enhancement comparison program to enlarge the image feature before executing the advanced image comparison program to determine whether the candidate area block contains the preset image feature and analyzing the enlarged candidate area block to determine whether the enlarged candidate area block contains the preset image feature, wherein only the feature enhancement comparison program is judged not to contain the preset image feature The candidate area block with image features will be provided to the advanced image comparison program for subsequent operations. The system according to claim 1, wherein the client device further executes a feature enhancement comparison program to enlarge the candidate area block and analyze the enlarged candidate area block before outputting the candidate area block to judging whether the enlarged candidate region tile contains the preset image feature, wherein only the candidate region tile that is judged not to include the preset image feature by the feature enhancement comparison program will be sent to the client device external output. The system as described in claim 1, wherein when the server device judges that the received candidate region tile does not contain the preset image feature, use the candidate region tile as the candidate region tile that does not include the preset image feature Negative samples are used to train the basic image comparison program.

Images