TWI883556B - Re-identification device and method - Google Patents

Abstract

A re-identification device includes a storage and a processor. The processor is configured to extract an object image corresponding to an object from an input image. The processor is further configured to calculate a reliability score corresponding to the object of the object image based on the object image. The processor is further configured to calculate visible scores corresponding to key points of the object image based on the object image. The processor is further configured to determine whether to store an image including the object image to the storage based on the reliability score and the visible scores of the object image. The processor is further configured to perform a re-identification step based on the image.

Description

Re-identification device and method

The present disclosure relates to an identification device and method, and more particularly to a re-identification device and method.

Re-identification technology is to classify images containing objects such as human bodies according to the appearance features of the objects, so as to summarize the images corresponding to the same object (for example, a person). In order to realize re-identification technology, it is necessary to collect a large number of images containing objects as identification data.

For example, many multi-recognition technologies use images of people captured by surveillance cameras as recognition data. However, due to factors such as the position of the surveillance camera, shooting angle, shooting distance, image quality, and object occlusion, the person is often blocked, only the back of the person is captured, the person is too small, or the image is blurred, so that the image of the person captured by the surveillance camera cannot clearly present all parts of the person, resulting in poor re-recognition results.

In view of this, how to filter image data suitable for re-identification is a goal that the industry urgently needs to work on.

To solve the above problem, the present disclosure provides a re-identification device, comprising a memory and a processor. The processor is coupled to the memory, and the processor is used to perform the following operations: capturing a first object image corresponding to an object from a first input image; based on the first object image, calculating a confidence score of the first object image corresponding to the object; based on the first object image, calculating a plurality of visual scores of a plurality of key points corresponding to the first object image; based on the confidence score and the visual scores of the first object image, determining whether to store a first image including the first object image in the memory as one of a plurality of images; and performing a re-identification step based on the images stored in the memory.

The present disclosure also provides a re-identification method applicable to an electronic device, wherein the electronic device includes a memory, and the re-identification method includes: capturing a first object image corresponding to an object from a first input image; based on the first object image, calculating a confidence score that the first object image corresponds to the object; based on the first object image, calculating a plurality of visual scores of a plurality of key points corresponding to the first object image; based on the confidence score and the visual scores of the first object image, determining whether to store the first object image in the memory as one of a plurality of images; and performing a re-identification step based on the images stored in the memory.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are intended to provide further explanation of the disclosure as claimed.

In order to make the description of the present disclosure more detailed and complete, reference may be made to the attached drawings and various embodiments described below, in which the same numbers in the drawings represent the same or similar elements.

Please refer to FIG. 1, which is a schematic diagram of a re-identification device 1 in the first embodiment of the present disclosure. As shown in FIG. 1, the re-identification device 1 includes a processor 12 and a memory 14, wherein the processor 12 is coupled to the memory 14. The re-identification device 1 is used to filter images suitable for re-identification data according to the imaging quality of objects in the input image. Please refer to the following description for the specific operation of the re-identification device 1.

In some embodiments, the processor 12 may include a central processing unit (CPU), a graphics processing unit (GPU), a multiprocessor, a distributed processing system, an application specific integrated circuit (ASIC), and/or an appropriate computing unit.

In some embodiments, the memory 14 may include semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), hard disk, and/or optical disk.

First, please refer to FIG. 2 , the processor 12 of the recognition device 1 extracts the object image OBJ corresponding to the object from the input image IMG. As shown in FIG. 2 , after the processor 12 recognizes the person in the input image IMG, it can extract the object image OBJ containing the person.

Furthermore, the processor 12 calculates a confidence score RS of the object corresponding to the object image OBJ based on the object image OBJ, wherein the confidence score RS is used to indicate a confidence level that the object (ie, human body) is included in the object image OBJ.

In some embodiments, the processor 12 may use an image recognition algorithm to identify whether the input image IMG contains an object to extract the object image OBJ, and based on the object image OBJ, calculate a reliability score of the object contained in the object image OBJ as a confidence score RS. In some embodiments, the confidence score RS is a value between 0 and 1, and the higher the confidence score RS, the higher the probability that the object image OBJ contains the object. In some embodiments, the processor 12 may use the YOLOv7 (You Only Look Once version 7) algorithm to complete the operation of extracting the object image OBJ and calculating the confidence score RS.

Next, please refer to FIG. 3 , the processor 12 calculates the visual scores of the key points KP0-KP16 corresponding to the object image OBJ based on the object image OBJ. As shown in FIG. 3 , the processor 12 marks the key points KP0-KP16 based on various parts of the object in the object image OBJ.

In this embodiment, 5 key points KP0-KP4 are key points of the face, and 12 key points KP5-KP16 are key points of the body, wherein the key points of the face include the nose key point KP0, the left eye key point KP1, the right eye key point KP2, the left ear key point KP3 and the right ear key point KP4, and the key points of the body include the left shoulder key point KP1. P5, right shoulder key point KP6, left elbow key point KP7, right elbow key point KP8, left wrist key point KP9, right wrist key point KP10, left waist key point KP11, right waist key point KP12, left knee key point KP13, right knee key point KP14, left ankle key point KP15 and right ankle key point KP16.

In addition, the processor 12 also calculates the visibility scores corresponding to the key points KP0-KP16, and the visibility scores are used to indicate the visibility of each of the key points KP0-KP16 in the object image OBJ.

In some embodiments, the visibility score is a value between 0 and 1, and the higher the visibility score, the higher the probability that the object image OBJ contains key points. For example, if the right foot of the human body in the object image OBJ is blocked, the visibility scores of the corresponding right knee key point KP14 and right ankle key point KP16 are 0; if the object image OBJ only shows the back of the human body, the visibility score of the facial key point is 0.

In addition, if the human body in the object image OBJ is far away from the shooting equipment (e.g., monitor, camera), resulting in a lower resolution of the object image OBJ, or if the image quality is poor and the object image OBJ is blurred, the processor 12 may also calculate a lower visible score for the corresponding key points KP0-KP16. On the other hand, if the front of the human body is shown in the object image OBJ, all parts are not blocked and/or the resolution is high, the processor 12 calculates a higher visible score for the corresponding key points KP0-KP16.

In some embodiments, the visibility scores corresponding to the aforementioned facial key points KP0-KP4 are facial visibility scores, and the visibility scores corresponding to the aforementioned body key points KP5-KP16 are body visibility scores.

In some embodiments, the processor 12 can calculate the visibility score corresponding to the key points KP0-KP16 by the following operations: determining the position of each key point KP0-KP16 in the object image OBJ, wherein each key point KP0-KP16 corresponds to a part of the object; and calculating the visibility score corresponding to each key point KP0-KP16 based on a plurality of pixels at the position of each key point KP0-KP16 in the object image OBJ. In other words, after marking the position of the key points KP0-KP16 in the object image OBJ, the processor 12 further calculates the corresponding visibility score using the pixels around the position of the key points KP0-KP16.

In some embodiments, the processor 12 may use an image recognition algorithm to calculate the confidence scores of the human body parts in the object image OBJ for the key points KP0-KP16 as the visual scores of the key points KP0-KP16. In some embodiments, the processor 12 may use a YOLOv7-Pose algorithm to complete the operations of marking the key points KP0-KP16 and calculating the visual scores corresponding to the key points KP0-KP16.

Next, the processor 12 determines whether to store the image including the object image OBJ in the memory 14 based on the confidence score RS and the visibility score of the object image OBJ. Finally, the processor 12 performs a re-recognition step based on the image stored in the memory 14.

In some embodiments, the processor 12 can determine whether the confidence score RS and the visible score are greater than a threshold (for example, 0.8). If the confidence score RS and the visible score are greater than the threshold, the image containing the object image OBJ is stored in the memory 14 for a re-recognition step. The image containing the object image OBJ can be determined according to actual needs, such as the process of the re-recognition step, the effect of the re-recognition, etc. It can be the object image OBJ itself, the input image IMG, and/or an image block containing the object image OBJ cut out from the input image IMG.

In some embodiments, the processor 12 may further calculate a first total score based on the confidence score RS and the visual score of the object image OBJ, and in response to the first total score being greater than a threshold, store the object image OBJ in the memory 14. For example, the processor 12 adds up the confidence score RS and the visual score and averages them to calculate the first total score, and when the first total score is greater than a threshold (e.g., 0.8), stores the object image OBJ in the memory 14.

In some embodiments, the processor 12 may further calculate the first total score based on the confidence score RS of the object image OBJ, the visibility score of the object image OBJ, and a plurality of weights corresponding to the confidence score RS and the visibility score. Specifically, the confidence score RS and the visibility scores of different parts of the corresponding object each represent different attributes of the object image OBJ, so the re-identification device 1 may set corresponding weights according to the emphasis of different attributes.

For example, if the recognition strength of facial features needs to be improved, the re-identification device 1 can set the weight corresponding to the visual score of the facial key points to a higher value. In this way, compared with the confidence score RS of the object image OBJ and the visual scores of other parts in the object image OBJ, the re-identification device 1 can filter out images with clearer facial features.

In some embodiments, the re-identification device 1 screens and stores the image data in the memory 14 for a re-identification step, wherein the re-identification step includes grouping the image to be identified into a first category of a plurality of identification categories after receiving the image to be identified; and based on the first category, outputting the identified image corresponding to the first category.

Specifically, the re-identification step can group the input images and group the images corresponding to the same object (e.g., the same person) into the same category. Therefore, after the re-identification step receives and groups the images to be identified, it can further output other images (i.e., identified images) that are grouped into the same category as the images to be identified.

In this way, the re-identification step can be applied to track the time and location of the appearance of the same object. For example, after multiple surveillance camera images located in different locations are grouped using the re-identification step, it can be summarized in which surveillance camera screen the same person appears at each time point, and the person's whereabouts can be further obtained.

In some embodiments, the re-identification step further includes training a re-identification model based on the images stored in the memory 14. Specifically, the re-identification device 1 can use the images in the memory 14 as training data to train the re-identification model. The re-identification model can be a clustering model. During the training process, the re-identification model can be based on the features of the objects captured in the images in the training data, and the trained re-identification model can cluster the input images as in the re-identification step in the aforementioned embodiment, and classify the images corresponding to the same object (e.g., the same person) into the same category.

In some embodiments, the processor 12 of the re-identification device 1 further tracks the object from a second input image to capture a second object image corresponding to the object, wherein a capture time of the second input image is later than the capture time of the first input image; based on the second object image, calculates the confidence score of the second object image corresponding to the object; based on the second object image, calculates the visibility scores of the key points corresponding to the second object image; and based on the confidence score and the visibility scores of the second object image, determines whether to store a second image including the second object image in the memory as one of the images.

Specifically, the re-identification device 1 can further receive an input image subsequent to the input image IMG (for example, the next frame in the video), and based on the same operation as determining whether to use the object image OBJ as an image, determine whether to use the second object image captured from the subsequent image for the re-identification step.

In some embodiments, the processor 12 of the re-identification device 1 can track the position of the same object in different frames (i.e., the input image IMG and subsequent images) in the video, and further determine whether to store the frame image to the memory 14. In some embodiments, the processor 12 can use a target tracking algorithm (e.g., DeepSort algorithm) to track the position of the same object in multiple images.

In some embodiments, the re-identification device 1 can also calculate a first total score based on the confidence score RS and the visible score of the object image OBJ; calculate a second total score based on the confidence score RS and the visible score of the subsequent image; and decide to store the image in the memory 14 as the image for the re-identification step based on the first total score and the second total score.

Similar to the operation of calculating the first total score mentioned above, the processor 12 of the re-identification device 1 can calculate the second total score corresponding to the subsequent image through the same operation, and determine the stored image based on the first total score and the second total score, for example: storing both or either the object image OBJ and the second object image in the memory 14.

For example, the processor 12 can extract multiple images (i.e., object images) containing the same person from multiple frames in a video, and then calculate the total score of each image based on the confidence score and the visual score, and finally filter the images based on the total score of each image. For example, the processor 12 can sort the multiple total scores and store one or more images with the highest total score for the re-identification step, or filter the images with a total score exceeding the threshold value (e.g., 0.8) for the re-identification step. In this way, the re-identification device 1 can extract images containing the same person from the video, and filter out images with better imaging quality for the re-identification step.

It should be noted that the above-mentioned embodiment is explained using an object image OBJ contained in the input image IMG as an example. However, in other embodiments, when the input image contains multiple objects, the re-identification device 1 can capture multiple object images through the same operation, and further determine whether to store the multiple object images in the memory 14 for the re-identification step through the same operation.

It should be noted that although the above embodiments use the human body as an example of an object to illustrate the operation of the re-identification device 1, the technology proposed in the present disclosure is not limited to this. In fact, the re-identification device 1 can also be applied to the object images of other types of objects (such as animals) with the same operation.

In summary, the re-identification device 1 proposed in the present disclosure can capture an object image containing an object in an input image, and determine whether to use the object image for the re-identification step based on the imaging quality of the object image. In addition, the re-identification device 1 can also set weights for different parts of the object to filter out images with better imaging quality of specific parts. Furthermore, the re-identification device 1 can also capture multiple object images corresponding to the same object from multiple frames of a video, and filter out object images with better imaging quality for the re-identification step.

The present disclosure also provides a re-identification method 200, which is applicable to an electronic device (e.g., re-identification device 1), wherein the electronic device includes a memory (e.g., memory 14). In some embodiments, the electronic device further includes a processor (e.g., processor 12) for executing the re-identification method 200. As shown in FIG. 4, the re-identification method 200 includes steps S201 to S205.

In step S201, the electronic device captures a first object image corresponding to an object from a first input image. In step S202, the electronic device calculates a confidence score of the first object image corresponding to the object based on the first object image. In step S203, the electronic device calculates a plurality of visual scores of a plurality of key points corresponding to the first object image based on the first object image. In step S204, the electronic device determines whether to store the first object image in the memory as one of a plurality of images based on the confidence score and the visual scores of the first object image. In step S205, the electronic device performs a re-recognition step based on the images stored in the memory.

In some embodiments, step S204 further includes calculating a first total score based on the confidence score and the visual scores of the first object image; and in response to the first total score being greater than a threshold, storing the first object image in the memory.

In some embodiments, the step of calculating the first total score further includes calculating the first total score based on the confidence score of the first object image, the visibility scores of the first object image, and a plurality of weights corresponding to the confidence score and the visibility scores.

In some embodiments, the re-identification method 200 further includes tracking the object from a second input image to capture a second object image corresponding to the object, wherein a capture time of the second input image is later than the capture time of the first input image; based on the second object image, calculating the confidence score that the second object image corresponds to the object; based on the second object image, calculating the visibility scores of the key points corresponding to the second object image; and based on the confidence score and the visibility scores of the second object image, determining whether to store a second image including the second object image in the memory as one of the images.

In some embodiments, step S204 further includes calculating a first total score based on the confidence score of the first object image and the visual scores; calculating a second total score based on the confidence score of the second object image and the visual scores; and determining to store in the memory as the content of the images based on the first total score and the second total score.

In some embodiments, step S203 further includes determining a position of each of the key points in the first object image, wherein each of the key points corresponds to a portion of the object; and calculating one of the visibility scores corresponding to each of the key points based on a plurality of pixels at the position of each of the key points in the first object image.

In some embodiments, the key points include a plurality of facial key points and a plurality of body key points, and the visibility scores include a plurality of facial visibility scores corresponding to the facial key points and a plurality of body visibility scores corresponding to the body key points.

In some embodiments, step S202 further includes calculating, based on the first object image, a confidence score that the first object image includes the object as the confidence score.

In some embodiments, the re-identification step further includes: receiving an image to be identified, grouping the image to be identified into a first category of a plurality of identification categories; and based on the first category, outputting a plurality of identified images corresponding to the first category.

In summary, the re-identification method 200 proposed in the present disclosure can capture an object image containing an object in an input image, and determine whether to use the object image for the re-identification step based on the imaging quality of the object image. In addition, the re-identification device 1 can also set weights for different parts of the object to filter out images with better imaging quality of specific parts. Furthermore, the re-identification device 1 can also capture multiple object images corresponding to the same object from multiple frames of a video, and filter out object images with better imaging quality for the re-identification step.

Although several embodiments are described in detail above as examples, the re-identification device and method proposed in the present disclosure can also be implemented by other systems, hardware, software, storage media or their combination. Therefore, the protection scope of the present disclosure should not be limited to the specific implementation method described in the embodiments of the present disclosure, but should be defined by the scope of the attached patent application.

It is obvious to those with ordinary knowledge in the art to which the present disclosure belongs that various modifications and changes can be made to the structure of the present disclosure without departing from the scope or spirit of the present disclosure. In view of the foregoing, the protection scope of the present disclosure also covers modifications and changes made within the scope of the attached patent application.

1: Re-identification device 12: Processor 14: Memory IMG: Input image RS: Confidence score OBJ: Object image KP0~KP16: Key points 200: Re-identification method S201~S205: Steps

In order to make the above and other purposes, features, advantages and embodiments of the present disclosure more clearly understandable, the attached drawings are described as follows: Figure 1 is a schematic diagram of the re-identification device in the first embodiment of the present disclosure; Figure 2 is a schematic diagram of capturing an object image from an input image and calculating a trust score in the first embodiment of the present disclosure; Figure 3 is a schematic diagram of key points in an object image in the first embodiment of the present disclosure; and Figure 4 is a flow chart of the re-identification method in the second embodiment of the present disclosure.

Domestic storage information (please note in the order of storage institution, date, and number) None Foreign storage information (please note in the order of storage country, institution, date, and number) None

1: Re-identify device

12: Processor

14: Storage

Claims (9)

A re-identification device comprises: a memory; and a processor coupled to the memory, the processor being used to perform the following operations: extracting a first object image corresponding to an object from a first input image; calculating a confidence score of the first object image corresponding to the object based on the first object image; calculating a plurality of visual scores of a plurality of key points corresponding to the first object image based on the first object image; determining whether to store a first image including the first object image in the memory as one of a plurality of images based on the confidence score and the visual scores of the first object image; and performing a re-identification step based on the images stored in the memory; The operation of determining to store in the memory as one of the images further includes: calculating a first total score based on the confidence score of the first object image and the visual scores; and storing the first object image in the memory in response to the first total score being greater than a threshold. A re-identification device as described in claim 1, wherein the operation of calculating the first total score further includes calculating the first total score based on the confidence score of the first object image, the visual scores of the first object image, and a plurality of weights corresponding to the confidence score and the visual scores. The re-identification device as described in claim 1, wherein the processor is further used to perform the following operations: Tracking the object from a second input image to capture a second object image corresponding to the object, wherein a capture time of the second input image is later than the capture time of the first input image; Based on the second object image, calculating the confidence score of the second object image corresponding to the object; Based on the second object image, calculating the visible scores of the key points corresponding to the second object image; and Based on the confidence score and the visible scores of the second object image, determining whether to store a second image including the second object image in the memory as one of the images. The re-identification device as described in claim 3, wherein the operation of determining to store in the memory as one of the images comprises: Calculating a first total score based on the confidence score of the first object image and the visual scores; Calculating a second total score based on the confidence score of the second object image and the visual scores; and Determining the content to be stored in the memory as one of the images based on the first total score and the second total score. The re-identification device as described in claim 1, wherein the operation of calculating the visible scores of the key points corresponding to the first object image further includes: Determining a position of each of the key points in the first object image, wherein each of the key points corresponds to a part of the object; and Based on a plurality of pixels of the position of each of the key points in the first object image, calculating one of the visible scores corresponding to each of the key points. A re-identification device as described in claim 1, wherein the key points include a plurality of facial key points and a plurality of body key points, and the visibility scores include a plurality of facial visibility scores corresponding to the facial key points and a plurality of body visibility scores corresponding to the body key points. The re-identification device as described in claim 1, wherein the operation of calculating the confidence score of the first object image corresponding to the object further includes: Based on the first object image, calculating a confidence score of the object included in the first object image as the confidence score. The re-identification device as described in claim 1, wherein the re-identification step further includes: receiving an image to be identified, grouping the image to be identified into a first category of a plurality of identification categories; and based on the first category, outputting a plurality of identified images corresponding to the first category. A re-identification method is applicable to an electronic device, wherein the electronic device includes a memory, and the re-identification method includes: Capturing a first object image corresponding to an object from a first input image; Based on the first object image, calculating a confidence score that the first object image corresponds to the object; Based on the first object image, calculating a plurality of visual scores of a plurality of key points corresponding to the first object image; Based on the confidence score of the first object image and the visual scores, determining whether to store a first image including the first object image in the memory as one of a plurality of images; and Based on the images stored in the memory, performing a re-identification step; The step of determining to store in the memory as one of the images further includes: Calculating a first total score based on the confidence score of the first object image and the visual scores; and In response to the first total score being greater than a threshold, storing the first object image in the memory.

Images