TWM629362U - Identification system - Google Patents

Abstract

The utility model is an identification system, including a data processing module, a feature extraction module, and a classification module. A person’ s image of a person form a received image data is cut to form an input data by the data processing module. After a feature vector of the input data is captured by the feature extraction module, whether the person is a clerk with uniform or a consumer without uniform is determined based on the feature vector by the classification module and the flow of people can be counted. In the utility model, the problem that the conventional face recognition technology is not suitable for the crowd counting of retail stores can be solved.

Description

ID system

This work is about the technology of image recognition, especially an identity recognition system used to identify the bodies of people.

For a store, the number of visitors represents the popularity of the store, and it can also increase the chance of consumption. Therefore, how to correctly count the number of visitors to the store in a day or a period of time is very important for the store's self-assessment. In addition, in the application of smart retail, it is usually necessary to collect consumer behavior as a basis for improving the service experience of retail. Traditionally, the collection of consumer behavior must be carried out through the collection of questionnaires or identification devices such as counters worn by store clerks to conduct flow of people. count. Today, the most common methods for collecting consumer behaviors are mostly using image recognition technology to count people, identify hot spots, and identify consumers’ gender and age groups. The purpose of store-related information.

When analyzing consumer-related information, the number of consumers who come to the store is the most basic indicator (Metric). However, there are usually both consumers and shop assistants serving consumers in the store, so how to distinguish consumption in the flow of people count In order to obtain the real number of consumers, it is one of the important themes of smart retail at present. In this regard, in order to provide smart retail counts of real consumers, a common method is to use Face Recognition technology to identify shop assistants who have pre-registered Face ID. However, due to the recent impact of the new crown pneumonia (COVID-19) epidemic, people will wear masks in public places, which will affect the recognition effect of face recognition. Furthermore, according to the experiments of the National Institute of Standards and Technology (NIST), the error rate of facial recognition in recognizing the face of a person wearing a mask is between 5% and 50%. Improvements in facial recognition technology have been improved, but face recognition still has the problem of being affected by the angle of the face. Therefore, the conventional face recognition technology is still lacking in identity recognition.

In addition, because the stores of retail stores usually wear uniforms, the clothing recognition (Clothes Recognition) method is used to identify the uniforms to distinguish them. Very demanding. In addition, when training the model, a static image composed of a model or a set of clothing is usually input, but consumers or shop assistants usually present a dynamic situation in the retail store, so the conventional clothing identification method is not suitable for Streaming video recorded by Closed-Circuit Television (CCTV) when people in retail stores are crowded, angled or obscured, or, using Logo Recognition and Optical Character Recognition Recognition) The method of recognizing the logo or words on the uniform also has the same problem as the aforementioned clothing recognition technology, that is, the method of logo recognition or optical character recognition still needs to input a clear and relatively complete image, so The same does not apply to the field of retail stores.

In view of the above problems, how to identify characters in larger-scale streaming videos, especially, even if the characters displayed on the streaming video are relatively small in size or partially obscured, they can still obtain relatively rich Identifying relevant information will become the goal that those skilled in the art are eager to pursue.

In order to solve the above-mentioned problems of the prior art, the present invention discloses an identity recognition system, which includes: a data processing module, which is used for receiving image data, and uses a multi-object tracking algorithm to track the characters in the image data, so as to obtain the information from the image data. In the image data, a plurality of person images are cut out corresponding to the person to form input data; the feature extraction module is coupled with the data processing module, and uses the person re-identification model to extract the feature vector of the person in the input data and a classification module, which is coupled with the feature extraction module, and is used for inputting the feature vector into a classification model to determine the identity of the character.

In one embodiment, the data processing module includes a posture direction estimation model for sampling the plurality of person images in multiple directions, wherein the posture direction estimation model performs direction recognition on the plurality of person images to The input data is formed by filtering out person images in different directions.

In another embodiment, the image of the person in different directions includes the orientation of the person in front, behind, left and/or right.

In another embodiment, the data processing module anonymously fuzzes the faces of the plurality of human images to form the input data.

In another embodiment, the extraction of the feature vector includes obtaining the feature vector of the uniform, hat and/or identification card on the person.

In another embodiment, during the model training process, the data processing module pre-marks the plurality of character images according to whether the character is wearing a uniform or not, so as to form a training sample for the character re-identification model to use the character image. training samples for training.

As can be seen from the above, the identity recognition system of this creation uses the data processing module to cut the characters of the received image data to form the input data, and the feature extraction module extracts the character feature vectors of the characters from the input data. The classification module can classify and count the characters according to the feature vector, so as to judge the body of the characters through uniform identification, so as to achieve the purpose of counting the flow of people.

1: Identity recognition system

11: Data processing module

12: Feature extraction module

13: Classification module

S201~S207: Steps

S501~S507: Process

U, NU: space

Figure 1 is a system architecture diagram of the identity recognition system of this creation.

Figure 2 is a flowchart of the training of the character re-identification model of the identity recognition system of the present creation.

Figure 3 is a schematic diagram of classification training performed by the identity recognition system of the present creation.

4A-4B are schematic diagrams of classification inference performed by the identity recognition system of the present creation.

FIG. 5 is a flow chart of the identity recognition method for implementing the identity recognition system of the present invention.

The technical content of the present creation is described below by means of specific embodiments, and those skilled in the art can easily understand the advantages and effects of the present creation from the content disclosed in this specification. However, the present invention can also be implemented or applied by other different specific implementation forms.

Figure 1 is a system architecture diagram of the identity recognition system created by the author. As shown in the figure, the identity recognition system 1 of the present creation includes a data processing module 11, a feature extraction module 12 and a classification module 13, wherein the data processing module 11 is used to cut out characters from the received image data The image is used to form the input data. After the feature extraction module 12 extracts the feature vector of the input data, the classification module 13 determines the identity of the person in the person image according to the feature vector, such as a consumer or a store clerk, or even a Retail store managers, event promotion models, delivery staff, and people count. In short, through feature vector extraction to make identification judgments, how to extract and identify the characteristics of the characters in the image becomes the key. The identification of a store clerk is described as an example.

The description of the identity recognition system 1 of the present creation is detailed as follows.

The data processing module 11 is used for receiving image data. In one embodiment, the data processing module 11 receives the streaming video recorded by the external closed-circuit television (CCTV) as the image data, or is stored in the database established by the identity recognition system 1 of the present creation or The image data captured by the camera. In other words, the data processing module 11 utilizes the Multiple Object Tracking (MOT) technology, wherein the multiple object tracking technology is specifically a multiple object tracking algorithm, which can target people in the image data (such as consumers or shop assistants) To perform tracking, specifically, the data processing module 11 can display the tracking of the person in the form of frame selection of the person by the multi-object tracking algorithm. After tracking the person through the multi-object tracking technology, the corresponding image data The tracked person is cropped, that is, according to the frame of the selected person. Since the image data is a continuous streaming video, after the person is cropped, multiple person images will be generated to form the input data.

In one embodiment, the data processing module 11 includes a pose and orientation estimation model for multi-sampling a person from a plurality of person images, since a single person usually has multiple types of people after being cropped from the image data. Images are used as samples, that is, various types of human images will contain human images in various directions or angles. Therefore, even after continuous or random sampling, it is possible to obtain samples that are almost repeated in appearance. Therefore, in order to obtain as much as possible The characters face samples in different directions, so this creation adopts the pose orientation estimation (POE) model for further analysis, so that the data processing module 11 first recognizes the orientation of the character through the pose orientation estimation model, and then analyzes the pose orientation estimation model. Multi-directional sampling is carried out on the personal image, and finally only the personal images in different directions are retained, so that the input data is formed by the multi-directional sampling of these personal images, that is, the multi-directional sampling in this creation is for the before and after the character. , left and/or right orientation are sampled, in particular, samples are formed such as the orientation of the character facing forward, rear, left and/or right. Accordingly, by providing a multi-faceted person shadow of a single person image, so as to improve the accuracy of the identification result of the identity identification system 1 of the present creation, and at the same time exclude redundant similar data, which can help reduce the amount of data calculation.

In an embodiment, the data processing module 11 anonymously blurs the faces of the plurality of human images to form the input data, that is, the data processing module 11 processes the human images further comprising: The facial parts in the human image are blurred, so that the identity recognition system 1 does not need to recognize facial features. In this creation, the facial parts of the people in the human image are blurred, so that the feature extraction module 12 is When performing feature vector extraction, it can focus on parts other than the face, such as clothes, hats, identification cards, etc., and even the part of the logo (LOGO) on the clothes, so that the identification effect of the identity recognition system 1 of this creation is more accurate. ; In addition, through the blurring of the faces of the characters, the purpose of protecting the privacy of consumers can be achieved at the same time.

The feature extraction module 12 uses a person re-identification (Person Re-identification) model to extract the feature vector of the person in the input data, wherein the extraction of the feature vector of the person includes obtaining the uniform and LOGO on the person. (For example, the trademark of a retail store, shop emblem, etc.), hats and/or identification cards, etc., feature vectors other than faces that can be used to identify a person (for example, a person of a store clerk or a consumer).

The classification module 13 inputs the feature vector retrieved by the person re-identification model into the classification model, so that the classification model judges whether the person corresponding to the input data is wearing a uniform and their identity according to the feature vector, so as to classify, especially It is the creation system that classifies shop assistants and non-shop assistants (consumers), and can count the flow of people after the classification, so as to achieve the purpose of counting the number of consumers.

Figure 2 is a flowchart of the training of the character re-identification model of the identity recognition system created by the author. As shown in the figure, before the operation of the identity recognition system, the person re-identification model can be trained in advance. In addition to establishing the person re-identification model, the person re-identification model can also be adjusted and improved, so as to achieve the correct and accurate image data. The effect of accurate consumer counting. Its training steps are as follows.

In step S201, data is prepared. This step can not only obtain the pre-stored image data from the database built by the identity recognition system of this creation, but also receive the image data captured by the camera built in the system or the external CCTV device. The object tracking algorithm crops the people in the image data as samples.

In step S202, multi-directional sampling is performed. In this step, a plurality of person images corresponding to a person are identified through the pose direction estimation model to identify the person's direction, so as to obtain the person images with the person facing different directions as samples. In one embodiment, the result of multi-direction sampling can retain samples in front, back, left, and right directions.

In step S203, face anonymous blurring is performed. Since this creation intends to make the character re-identification model focus on the feature extraction of the unique features of the clerk, such as the uniform, LOGO, hat or identification card of the character, the face is anonymized and blurred. It has almost no effect on the recognition ability of person re-identification; furthermore, since identity recognition does not need to recognize facial features, therefore, in order to improve the general recognition of the character re-identification model of this creation in actual fields (such as in retail stores) Therefore, when processing the input data (ie samples), the face can be anonymous and blurred, on the one hand, it can protect the privacy of consumers; Dependence on facial features.

In step S204, the data is marked. This step is to mark the character in the sample with or without uniform as a condition. In short, in order to distinguish whether the sample is wearing a uniform or not, and then estimate whether the character is a clerk or not, this step can be done first. Using the method of marking, the characters in the sample are marked with or without uniforms.

In step S205, feature extraction is performed. In this step, the sample is input into the character re-identification model to extract the feature vector, and the character re-identification model is used to extract the feature vector for the person with or without the uniform, and then the person with the uniform can be obtained. eigenvector of .

In step S206, classify. In this step, the feature vector is input into a classification model such as a Support Vector Machine (SVM), so as to classify the samples into two categories of shop assistants or consumers. In addition, because the number of employees in a retail store is usually far less than the number of consumers, the obtained data is Imbalanced Data, so when training the support vector machine, the class weight setting is used (for example, sklearn.svm. The category weight setting of the balanced mode of SVC) to automatically increase the weight of the clerk category with fewer samples, so as to avoid the tendency of the person re-identification model to identify the person as the consumer category with more samples.

For example, if there are 1,000 samples of consumers and 100 samples of shop assistants in the image data used for training, if all shop assistants are misidentified as consumers, the error rate is only 100/ (1,000+100), about 9.1%, and its error rate may be ignored. For this reason, this creation increases the weight of shop assistants, that is, the ratio of the weight of consumers to the weight of shop assistants is 1:10, so that 1,000 consumers The importance of the 100 clerks is the same. According to this, if all the clerks are misjudged, the error rate is 100*10/(1,000*1+100*10)=50%, in other words, if all the clerks are misjudged , which is equivalent to misjudging half of the data, which increases the severity of the misjudgment. Therefore, this creation adjusts and sets the weight to increase the severity of the model's misjudgment for the sample of shop assistants with a relatively small number of samples, so as to avoid the problem of the extension of the misjudgment situation. In a specific embodiment, the set weight of this creation can be adjusted automatically through the balanced mode, that is, the balanced mode uses the formula: total number of samples/(number of categories * number of samples of this category) to achieve automatic basis The purpose of adjusting the aforementioned weight by the number of samples, for example, following the aforementioned example, the consumer weight=(1,000+100)/(2*1,000)=0.55, and the shop assistant weight=(1,000+100)/(2*100 )=5.5, therefore, consumption The ratio of the operator weight to the clerk weight is 0.55:5.5, which is 1:10, which is the same as the previous example, so that the model imbalance caused by misjudgment can be avoided.

In step S207, the weight is optimized. In this step, the loss value (Loss) is calculated according to the classification result and the ground truth, and the weight (Weight) of the person re-identification model and the classification model is updated to make the Loss smaller, and then the training is repeated until the termination condition is reached. In the category calculation, t is the real category of the sample, which can be 1, -1, to represent the clerk (1) and the consumer (-1) respectively, when the classification model predicts y=0.7, and the real category of the sample The category is t=1, then the loss is: max(0,1-t*y)=max(0,1-1*0.7)=0.3, which means that the closer y is to t, the smaller the loss.

Next, in order to make the character re-identification model of the identity recognition system of this creation can correctly classify after training, the training is carried out as follows.

As shown in Figure 3, it is a schematic diagram of classification training performed by the identity recognition system of this creation. In order to enable the classification model for performing identity recognition to classify correctly after training, the training goal is set to train the full-body feature extraction capability of the person re-identification model and ensure that the following conditions are true: (a) When the clothes are similar and the same person, the characteristics The vectors should be almost the same; (b) when the clothes are similar, the eigenvectors should be similar, even for different individuals; (c) when the clothes are different, the eigenvectors should be dissimilar.

First, the marked samples in uniform (samples 1-3) and samples without uniforms (samples 4-6) are input into the person re-identification model to train the ability of the person re-identification model to capture the whole body features of the person, so that the person can be re-identified The model has the same or nearly the same eigenvectors captured by the same person in similar clothes, similar eigenvectors captured by different individuals in similar clothes, and dissimilar eigenvectors captured by different individuals.

Afterwards, the classification model is trained to classify the feature vectors. Under the aforementioned conditional classification, the feature vectors corresponding to those with uniforms and those without uniforms will fall into the space U (Uniform, that is, with uniforms) and Space NU (Non-Uniform, ie no uniform).

Next, after the classification model is trained, inference can be made, as shown in Figures 4A and 4B, which are schematic diagrams of classification inference by the identity recognition system of the present creation. The identity recognition system of this system samples the input samples in multiple directions and blurs the faces anonymously. The character re-identification model deduces the feature vectors of the input samples. Then, the classification model is used to classify the feature vectors. The uniform sample, as shown in Figure 4A, if the feature vector falls in the space U (Uniform), that is, the character is wearing a uniform, on the contrary, if it falls in the space NU (Non-Uniform), that is, the character is not wearing a uniform, as shown in the figure shown in 4B.

FIG. 5 is a flow chart of an identity recognition method for implementing the identity recognition system of the present invention. As shown in the figure, the identification method of this creation is executed in electronic devices such as computers, servers or cloud systems. Specifically, the identification method of this creation includes the following processes.

In the process S501, image data is received. This process can obtain the stored image data from the database, or receive the image data captured from the set camera or CCTV device.

In the process S502, a person is tracked. This process uses a multi-object tracking algorithm to track a person in the image data, so as to cut out a plurality of person images from the image data corresponding to the person.

In the process S503, multi-directional sampling is performed. In this process, multi-directional sampling is performed on a plurality of cropped person images by using a pose direction estimation model. The plurality of person images sampled in multiple directions form the input data.

In the process S504, the face is anonymously blurred. This process is to anonymously blur the faces of the multiple human images, so that the face parts are blurred and cannot be identified, and the input data is formed accordingly. The purpose is not only to protect personal information, but also to reduce the time for feature extraction Impact.

In the process S505, input data is formed. In this process, the above-mentioned cropped, multi-directional sampling, and anonymously blurred face images are formed into input data.

In the process S506, the feature vector of the character in the input data is extracted by the character re-identification model. The aforementioned feature vector extraction includes vector extraction of the characters' uniforms, logos, hats, and/or identification cards.

In the process S507, according to the feature vector, it is determined whether the character is wearing a uniform and his identity through the classification model. This process is to use the classification model to determine the identity of the person's image. In this creation, it is mainly determined that the person is a consumer or a clerk, and then the flow of people can be counted.

In one embodiment, in step S506, before using the person re-identification model, the person re-identification model may be trained first to obtain the person re-identification model or the subsequent optimization of the person re-identification model. Specifically, the training steps include data preparation, multi-directional sampling, face anonymous blurring, labeling data, feature extraction, classification, and weight optimization, the details of which have been described in Figure 2 and the corresponding paragraphs, so they will not be repeated here.

In one embodiment, the identity recognition system and method of the present creation is applied in a practical application field. In short, training is performed in 9 large-scale retail stores, and the training samples include 5,077 samples without uniforms and 755 samples with Uniform sample, this creation uses the character re-identification model after this training to conduct 1,074 test samples in other retail stores, among which the average precision and recall rate are over 93%. As shown in Table 1 below.

In addition, the present invention also discloses a computer-readable medium, which is applied to a computing device or computer with a processor (eg, CPU, GPU, etc.) and/or memory, and stores instructions, and can utilize the computing device or computer. The computer executes the computer-readable medium through a processor and/or a memory, so as to execute the above-mentioned methods and steps when executing the computer-readable medium.

In one embodiment, the modules, units, devices, etc. of the present invention include a microprocessor and a memory, and algorithms, data, programs, etc. are stored in a memory or a chip, and the microprocessor can load data from the memory. Or algorithms or programs for data analysis or calculation processing. In other words, the identity recognition system of this creation can be executed on electronic devices, such as ordinary computers, tablets or servers, and performs data analysis and calculation after receiving the image data. Designed and constructed on electronic equipment with processors, memory and other components to run on various electronic equipment; in addition, each module or unit of the identity recognition system can also be composed of independent components, such as designed to calculate A device, memory, storage or firmware with a processing unit can be used as components to realize this creation, and related models such as character re-identification model or classification model can also be constructed in software programs, hardware or firmware. render.

To sum up, the identity recognition system of this creation distinguishes consumers and shop assistants by identifying uniforms and related features in retail stores, and extracts feature vectors for the whole body features of the characters through person re-identification, and then uses the features The vector is input into the classification model used to identify uniforms, so as to judge the characters, such as consumers and shop assistants; in addition, in the training process, for the sample Labeling, for example, uniformed and ununiformed, allows the model to learn and recognize common features of shop assistants such as uniforms, hats, badges, etc. Therefore, this creation solves the problems that face recognition cannot be used to identify store staff who have not registered Face ID in the past, and the recognition rate is affected by mask occlusion and face angle. In other words, even if there are new store staff who have not registered in any system, but As long as the new entrant is wearing a clerk uniform, he can still be identified as a clerk by the ID system even when his back is turned away from the camera.

The above-mentioned embodiments are only illustrative, and are not intended to limit the present creation. Anyone skilled in the art can modify and change the above embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the rights of this creation is defined by the scope of the patent application attached to this creation, and shall be included in the disclosed technical content as long as it does not affect the effect and implementation purpose of this creation.

1: Identity recognition system

11: Data processing module

12: Feature extraction module

13: Classification module

Claims (6)

An identity recognition system, comprising: The data processing module is used to receive the image data, and use the multi-object tracking algorithm to track the person in the image data, so as to cut out a plurality of person images corresponding to the person from the image data to form the input data; a feature extraction module, coupled to the data processing module, to extract feature vectors of the person in the input data by using a person re-identification model; and The classification module is coupled to the feature extraction module, and is used for inputting the feature vector into the classification model to determine the identity of the person. The identity recognition system of claim 1, wherein the data processing module includes a posture direction estimation model for sampling the plurality of person images in multiple directions, wherein the posture direction estimation model is used for the plurality of The human image is subjected to direction identification, and the input data is formed by filtering out the human images in different directions. The identity recognition system as claimed in claim 2, wherein the image of the person in different directions includes the front, back, left and/or right directions of the person. The identity recognition system as claimed in claim 1, wherein the data processing module anonymously fuzzes the faces of the plurality of person images to form the input data. The identity recognition system according to claim 1, wherein the extraction of the feature vector includes obtaining the feature vector of the uniform, hat or identification card on the person. The identity recognition system according to claim 1, wherein, in the model training process, the data processing module pre-marks the plurality of character images according to whether the character is wearing a uniform or not, so as to form a training sample for the training sample. The person re-identification model is trained using this training sample.

Images