CN109977876A - Image-recognizing method, calculates equipment, system and storage medium at device - Google Patents

Abstract

The present invention relates to an image recognition method, an image recognition apparatus, a computing device, a computer-readable storage medium and an image recognition system. The image recognition method includes: predicting lighting conditions in a future period based on a plurality of images respectively shot by a camera at multiple consecutive moments; generating imaging parameters to be sent to the camera based on at least the predicted lighting conditions; imaging parameters are sent to the camera so that the camera captures an image to be recognized based on the imaging parameters in the future period; and an object of interest is recognized from the image to be recognized. The method can improve the quality of captured images, thereby improving the accuracy of detection and recognition of objects of interest.

Description

Image recognition method, apparatus, computing device, system and storage medium

technical field

The present invention relates to image recognition technology, in particular to an image recognition method, an image recognition device, a computing device, an image recognition system and a storage medium.

Background technique

Video surveillance and image recognition technologies are widely used in semi-outdoor or outdoor places, such as large conference venues, airports, railway stations and urban roads. In different application scenarios, such as different locations and/or different times, the lighting conditions will change. Even within the same location, the change in light conditions from early morning to late evening was significant. Image post-processing techniques (for example, extracting illumination invariant features, image enhancement based on facial feature models, or other standardized processing methods) cannot fully compensate for the lack of information that occurs during image capture by camera equipment, which affects the accuracy of recognition results. Therefore, how to reduce the influence of complex on-site lighting environment on image recognition results becomes a challenge.

SUMMARY OF THE INVENTION

It would be advantageous to provide a mechanism that can alleviate, alleviate or even eliminate one or more of the above problems.

According to an aspect of the present invention, there is provided an image recognition method, comprising: predicting the lighting conditions of a future period based on multiple images respectively captured by a camera at multiple consecutive times; camera parameters to the camera; initiate sending of the camera parameters to the camera so that the camera captures an image to be identified based on the camera parameters in the future time period; and identify an object of interest from the to-be-identified image .

In some embodiments, the predicting the lighting conditions of the future period includes: sequentially extracting the respective lighting features of the plurality of images by using a residual network; recursively processing the respective lighting features using a temporal recurrent neural network, wherein the A residual network and the temporally recurrent neural network have been trained to learn changes in lighting conditions over time; and the lighting conditions predicted by the output of the temporally recurrent neural network. The multiple times include N times, where N is a positive integer greater than 1. Both the illumination feature of X _k+1 and the processing result of the temporal recurrent neural network on the illumination feature of X _k are input to the temporal recurrent neural network for recursive processing, where X _k represents the An image captured at time k, and X _k+1 represents an image captured at time k+1 among the N time moments, where k is an integer and 1≦k<N.

In some embodiments, the temporal recurrent neural network comprises a long short term memory network.

In some embodiments, the generating the camera parameters to be sent to the camera includes using a residual structure-based neural network to map the predicted lighting conditions to the first version of the camera parameters. The residual structure-based neural network has been trained to learn the mapping of lighting conditions to camera parameters. The first version of the camera parameters may be selected as the camera parameters to be sent to the camera.

In some embodiments, the residual structure-based neural network includes a plurality of concatenated residual structures. The mapping of the predicted lighting conditions to the camera parameters of the first version includes: inputting features characterizing the predicted lighting conditions into the residual structure-based neural network, so that the residual structure-based neural network performs operations , the operations include: reshaping the features characterizing the predicted lighting conditions; convolving the reshaped features; applying an activation function to the convolved features; performing batch normalization on the features after applying the activation function to obtain a normalized feature map; process the normalized feature map using the plurality of concatenated residual structures; and input the processed feature map into a fully connected layer to obtain the first A version of the camera parameters.

In some embodiments, the method further includes: controlling the camera to capture a sample image in real time before capturing the to-be-recognized image; generating a second version of camera parameters based on the sample image; and converting the first version A weighted average is performed between the camera parameters of the second version and the camera parameters of the second version to obtain weighted camera parameters, and the weighted camera parameters may be selected as the camera parameters to be sent to the camera. In a first mode of operation, the first version of the camera parameters is selected as the camera parameters to be sent to the camera. In a second operating mode different from the first operating mode, the weighted imaging parameters are selected as the imaging parameters to be sent to the camera.

In some embodiments, the object of interest includes the subject's face. The identifying the object of interest includes: using a second deep neural network to detect a face region from the to-be-recognized image; and using a third deep neural network to extract facial features from the detected face region in the to-be-recognized image to used to identify the subject.

In some embodiments, the second deep neural network is a shallow convolutional neural network. The detecting the face region from the image to be recognized includes: using the shallow convolutional neural network to generate multiple face windows; obtaining the optimal face window among the multiple face windows through non-maximum suppression as the final detected face window. facial area.

In some embodiments, the method further comprises: using the second deep neural network to detect facial regions from a registered image, wherein the registered image is an electronic photograph corresponding to the identity of the subject; using the third deep neural network extracts facial features from the detected facial regions in the registered image; and compares the facial features extracted from the image to be recognized with the facial features extracted from the registered image to determine whether the subject matches the identity.

In some embodiments, the method further includes: in response to an abnormal event in which the subject does not match the identity, initiating a reporting of an alert message to the system platform.

In some embodiments, the alert message includes the facial features extracted from the to-be-recognized image and the facial features extracted from the registered image. The method also includes: comparing the facial features extracted from the to-be-recognized image and the facial features extracted from the registered images with the feature library of the public security system to obtain the identity information and the identity information of the impostor. Identity information of the impostor.

In some embodiments, the method further includes: using the identity information of the impostor to search in the database of the suspect at large to confirm whether the impostor is the suspect at large.

In some embodiments, the method further includes: in response to the fraudster being identified as a suspect at large, invoking surveillance cameras within a certain geographic range around the scene; recognizing all faces in the images captured by the surveillance cameras; and The obtained facial features are compared with the facial features of the suspect at large, thereby tracking the suspect.

In some embodiments, the method further includes: controlling the camera to capture a sample image in real time before capturing the to-be-recognized image; estimating the depth of the sample image by using a first deep neural network; determining the sample image according to the estimated depth the foreground area and the background area, the brightness of the foreground area, and the brightness of the background area in the sample image; in response to the difference between the brightness of the foreground area and the brightness of the background area exceeding a predetermined range, adjust the adjustment associated with the camera and repeating the steps of real-time shooting, estimating and determining until the brightness of the foreground area and the brightness of the background area in the current sample image are within the predetermined range.

In some embodiments, the estimating the depth of the sample image comprises: resizing the sample image to have a first size; and inputting the resized sample image into the first deep neural network such that the The first deep neural network performs operations, the operations comprising: performing multiple convolutions on the sample images of the first size; and repeatedly processing the convolved sample images to obtain a depth map with the first size . The processing includes upsampling and convolution.

In some embodiments, the at least one adjustable light source associated with the camera includes two adjustable light sources arranged on either side of the camera, respectively. Adjusting the illumination intensity of the at least one adjustable light source associated with the camera includes executing a strategy selected from the group consisting of: two-side light reduction, dark-side fill light, bright-side light reduction, two-side fill light The sum is not adjusted so that the difference between the brightness of the foreground area and the brightness of the background area is within a predetermined range.

In some embodiments, the method further comprises: detecting ambient light with a light sensor; and in response to the detection result of the light sensor changing by less than a threshold within a predetermined time period, setting a locking time period, during which the locking time During the segment period, the illumination intensity of the adjustable light source is not adjusted before the image to be recognized is captured.

According to another aspect of the present invention, an image recognition device is provided, comprising: an illumination prediction module configured to predict illumination conditions in a future period based on a plurality of images respectively captured by a camera at a plurality of consecutive times; camera parameter adjustment a module configured to generate camera parameters to be sent to the camera based at least on the predicted lighting conditions, and to initiate sending of the camera parameters to the camera so that the camera is based on the camera parameters in the future time period photographing an image to be recognized; and an image recognition module configured to recognize an object of interest from the image to be recognized.

According to yet another aspect of the present invention, there is provided a computing device comprising a memory and a processor, the memory configured to store computer program instructions thereon, the computer program instructions when executed on the processor The processor is caused to perform the method as described above.

According to yet another aspect of the present invention, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed on a processor, cause the processor to perform the method as described above.

According to yet another aspect of the present invention, there is provided an image recognition system including: a camera; and the computing device as described above.

In each embodiment, the imaging parameters of the camera are adaptively set by predicting changes in the lighting conditions of the shooting site, so that the quality of the captured images is improved, thereby improving the accuracy of detection and recognition of objects of interest.

Description of drawings

Further details, features and advantages of the present invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

1 shows a flowchart of an image recognition method according to an embodiment of the present invention;

FIG. 2 shows a schematic block diagram of an example system in which the method of FIG. 1 may be applied, according to an embodiment of the present invention;

FIG. 3A shows an exemplary and schematic recursive structure of the illumination prediction module in FIG. 2;

Figure 3B shows a schematic diagram of a lighting prediction module in which the recursive structure is expanded;

FIG. 4 shows an exemplary and schematic illustration of the operations performed by the camera parameter adjustment module in FIG. 2;

Figures 5A and 5B show exemplary and schematic illustrations of operations performed by the image recognition module in Figure 2;

Figure 6 shows an exemplary and schematic illustration of the operation of the deep neural network used by the depth estimation module in Figure 2;

FIG. 7 shows an example configuration of a depth estimation module, camera, and adjustable light source;

Figure 8 shows the general flow of the operation of the depth estimation module;

FIG. 9 illustrates an application scenario in which techniques according to embodiments of the present invention may be applied; and

10 illustrates an example system including an example computing device representative of one or more systems and/or devices that may implement the various techniques described herein.

Detailed ways

Before describing embodiments of the present invention, several terms used herein are explained. These concepts should be known to those skilled in the art of artificial intelligence, and their detailed descriptions are omitted herein for brevity.

1. Long Short-Term Memory (LSTM). The long short-term memory network is a time recurrent neural network, which includes a structure called a cell, each cell includes three gates, which are called the forget gate, the input gate and the output gate. After the information enters the LSTM, the information that does not conform to the algorithm rules is forgotten through the forgetting gate, while the information that conforms to the algorithm rules is retained.

2. Residual network (Residual Network, Resnet). Residual network is a deep convolutional network proposed in 2015, which aims to solve the side effect (degeneration problem) caused by the increase of network depth and improve network performance. Residual networks typically include multiple residual learning structures (also called residual structures), each of which is implemented through a forward neural network and shortcut connections.

3. VGG16. VGGNet is a deep convolutional neural network proposed by the Visual Geometry Group of Oxford University. By repeatedly stacking 3×3 small convolution kernels and 2×2 max pooling layers, a convolutional neural network with a depth of 16 to 19 layers is constructed. VGG16, the 16-layer VGGNet.

4. Parametric Rectified Linear Unit (PReLU) with parameters. Modified linear unit, also known as linear rectification function, is a commonly used activation function in artificial neural networks, usually referring to nonlinear functions such as ramp functions and their variants.

5. Deep Neural Networks (DNN). Deep neural network is a kind of neural network that appeared with the concept of deep learning in 2006. DNN can be understood as a neural network with many hidden layers.

6. Batch Normalization (BN). Batch normalization is a training optimization method proposed in 2015 for faster and more stable training of deep neural networks. More on batch normalization can be found at https://ar×iv.org/abs/1010.03167.

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

1 shows a flowchart of an image recognition method 100 according to an embodiment of the present invention, and FIG. 2 shows a schematic block diagram of an example system 200 in which the method 100 of FIG. 1 may be applied.

The system 200 includes a camera CAM, adjustable light sources ALS1 and ALS2 associated with the camera CAM (eg, mounted on both sides of the camera), an image recognition device 210 , and a system platform (eg, a server SRV and a database DB).

Camera CAMs may be various types of image capture devices, such as digital cameras, webcams, surveillance cameras, and the like. Although the camera CAM is shown in FIG. 2 as being separate from the image recognition device 210 , in some embodiments the camera CAM may be integrated with the image recognition device 210 .

Adjustable light sources ALS1 and ALS2 are optional to provide supplemental lighting to the camera CAM. The light sources ALS1 and ALS2 are adjustable in the sense that their illumination intensity (luminance brightness) can be controlled to increase and decrease. Although two adjustable light sources are shown in FIG. 2, system 200 may include fewer or more adjustable light sources.

The image recognition device 210 includes an illumination prediction module 211, a camera parameter adjustment module 212, an image recognition module 213, a depth estimation module 215, an alarm module 214 and a light sensor SNS, wherein the depth estimation module 215, the alarm module 214 and the light sensor SNS may be optional . Image recognition device 210 may be via any suitable I/O interface (eg, wired or wireless transceiver, not shown in Figure 2) and any suitable communication protocol (eg, TCP/IP, Bluetooth, Zigbee, Wi-Fi) Interacts with camera CAM, adjustable light sources ALS1 and ALS2, server SRV, and database DB.

The system platform includes, for example, a server SRV and a database DB. In this case, the image recognition apparatus 210 may operate as a client. It will be appreciated that the image recognition device 210 and the system platform may also be based on architectures other than client/server, such as a distributed cloud architecture (where the system platform is a cloud) or physically co-located or separate front-office structures ( The system platform is the background).

1 and 2, in step 110, based on _N images X ₁ , X ₂ , X ₃ . positive integer of . This step can be implemented by the illumination prediction module 211 .

The N images X ₁ , X ₂ , X ₃ . . . , X _N may be received from the camera CAM via the I/O interface of the image recognition device 210 . In some embodiments, the illumination prediction module 211 sequentially extracts illumination features of the images X ₁ , X ₂ , X ₃ . . . , X _N using a residual network, and recursively processes the respective illumination features using a temporal recurrent neural network. The temporal recurrent neural network then outputs the predicted lighting conditions. The residual network and the temporal recurrent neural network are pretrained to learn changes in lighting conditions over time. Lighting data for different locations is fed to the lighting prediction module 211 for training. Alternatively, training can be customized for a specific location.

FIG. 3A shows an exemplary and schematic recursive structure of the illumination prediction module 211 . In this example, the illumination prediction module 211 includes a residual network Resnet50 and a long short term memory network LSTM. As known, Resnet50 is a 50-layer residual network, which achieves a good compromise between performance and required computing power. LSTMs are suitable for processing and predicting important events with relatively long intervals and delays in time series. Therefore, such a structure is suitable for dealing with objects that contain more information and change over time, such as lighting conditions.

As shown in Figure 3A, the image _Xk (k is an integer and 1≤k<N) is input to Resnet50, which outputs a high-dimensional (eg, 1000- or 1600-dimensional) illumination feature. This feature is then fed into the LSTM, and the output of the LSTM (which has the same dimensions as the output of Resnet50) is fed into the LSTM again for recursive processing along with the lighting features extracted by Resnet50 from the image _Xk+1 .

Figure 3B shows a schematic diagram of the illumination prediction module 211 in which the recursive structure is expanded. As shown in Figure 3B, the illumination features extracted by _Resnet50 from image X1 are input to the LSTM, and both the output of the LSTM and the illumination features extracted by _Resnet50 from image X2 are input into the LSTM again. Then, both the output of the LSTM and the lighting features extracted by Resnet50 from image _X3 are fed into the LSTM again, and so on. Thus, with the help of the temporal recursive properties of LSTMs, the lighting prediction module 211 outputs features _hN characterizing the lighting conditions for the predicted future time period.

Referring back to Figures 1 and 2, at step 120, camera parameters to be sent to the camera CAM are generated based at least on the predicted lighting conditions. This step can be implemented by the camera parameter adjustment module 212 . In various embodiments, camera parameters may include, for example, exposure value, aperture value, shutter speed, sensitivity, contrast, resolution, saturation, sharpness, filter settings, color space mode, white balance settings, photo aspect ratio and one or more of photo storage formats.

FIG. 4 shows an exemplary and schematic illustration of the operations performed by the camera parameter adjustment module 212 . In this example, the camera parameter adjustment module 212 utilizes a residual structure-based neural network to map the predicted lighting conditions to the camera parameters of the first version. The residual structure-based neural network includes a plurality of concatenated residual structures, but for the convenience of illustration, only one of them is shown.

Each residual structure is implemented by a feedforward neural network and shortcut connections represented by branch branches, as shown in Figure 4. The idea of the residual structure is not to directly fit the identity mapping function H(x)=x, but to convert the function into H(x)=F(x)+x to obtain the residual function F(x)=H(x )-x. As long as F(x)=0, the identity map H(x)=x is formed. For example, if 5 is mapped to 5.1, then F'(5)=5.1 before the residual is introduced, H(5)=5.1 after the residual is introduced, H(5)=F(5)+5, F(5 )=0.1, where F' and F represent the network parameter mapping. The mapping after introducing residuals is more sensitive to changes in the output. For example, if the output changes from 5.1 to 5.2, the map F' increases by 2%, while for the residual structure, if the output changes from 5.1 to 5.2, the map F is from 0.1 to 0.2, an increase of 100%. This way, small changes are highlighted and the effect is better. The residual structure-based neural network may have been trained with large amounts of data to learn the mapping of lighting conditions to camera parameters.

In some embodiments, the lighting conditions (ie, features h _N ) of the predicted future periods from the lighting prediction module 211 are input into the residual structure-based neural network such that the residual structure-based neural network performs Do the following. First, reshape the input feature h _N. For example, 1600-dimensional features are reshaped to 40*40. Then, it is convolved with a convolution kernel of eg 3*3, and an activation function PReLU is applied. Next, batch normalization is performed on the features after applying the activation function to obtain a normalized feature map. The normalized feature maps are then fed into multiple concatenated residual structures. The number of residual structures here can be flexible. Generally, by using more than 4 residual modules, the data can be better fitted. After being processed by the fully connected layer, the camera parameters of the first version are output from the camera parameter adjustment module 212 .

Referring back to Figures 1 and 2, at step 130, sending the camera parameters to the camera CAM is initiated so that the camera CAM captures images to be identified based on the camera parameters in a future period. The sending of the camera parameters to the camera CAM may be initiated by the camera parameter adjustment module 212 and performed by the I/O interface of the image recognition device 210 .

In some embodiments, the camera parameters of the first version are sent directly to the camera CAM via the I/O interface of the image recognition device 210, so that the camera CAM captures images to be recognized based on the camera parameters of the first version in a future period. That is, the imaging parameters used by the camera CAM are simply derived based on the lighting conditions predicted by the lighting prediction module 211 . This mode of operation may be referred to as the fully automatic mode.

In some embodiments, the camera parameters of the first version may also be sent to the camera CAM after further processing. For example, in order to better adapt the camera parameters to the lighting conditions of the scene, the real-time lighting conditions of the scene may be taken into account when determining the camera parameters. Specifically, the camera CAM is controlled to capture a sample image in real time before capturing the image to be recognized, and the imaging parameter adjustment module 212 generates a second version of imaging parameters based on the sample image. This can be achieved, for example, by typical image processing methods. Specifically, the camera parameter adjustment module 212 processes the sample image to obtain various quality indicators, such as brightness, contrast, hue, saturation, sharpness, etc., and generates a second version of camera parameters based on these quality indicators, aiming to improve these quality indicators Quality Index. Then, the camera parameter adjustment module 212 performs a weighted average of the camera parameters of the first version and the camera parameters of the second version to obtain weighted camera parameters. The weighted imaging parameters are sent to the camera CAM via the I/O interface of the image recognition device 210 . This mode of operation may be referred to as a semi-automatic mode because the camera parameters used by the camera CAM are not derived purely based on the lighting conditions predicted by the lighting prediction module 211 .

In step 140, the object of interest is identified from the to-be-identified image captured by the camera CAM. In various embodiments, the identification of the object of interest may be implemented by the image recognition module 213 .

The object of interest may vary with application scenarios. For example, in the application of traffic violation monitoring, the object of interest may be the license plate of a motor vehicle. As another example, in the application of security inspection, the object of interest may be the face of a subject (eg, a human or an animal). Similar to human faces, animal faces also contain many features, such as geometric features based on the shape and geometric relationships of facial organs. Therefore, the technology of face detection and recognition is also applicable to animals, so that in some application scenarios, the technology according to the embodiments of the present invention can be applied to detect and recognize the faces of animals.

5A and 5B show exemplary and schematic illustrations of the operations performed by the image recognition module 213. As shown in Fig. 5A, a deep neural network including multiple convolutional layers, pooling layers, and fully connected layers is used to detect regions of objects of interest (hereinafter referred to as "regions of interest") from the input images from the camera CAM (Fig. 2). (ROI)”). In this example, the deep neural network extracts high-level features from the input image to detect multiple possible ROI regions in the image, and outputs the score of the highest scoring ROI region among the multiple possible ROI regions. For example, a shallow convolutional neural network (CNN) can be used to quickly generate multiple ROI windows, and then obtain the optimal ROI window among multiple ROI windows through non-maximum suppression. The optimal ROI region is provided to another deep neural network shown in FIG. 5B as the detection result. Similar to Figure 5A, the deep neural network of Figure 5B also includes multiple convolutional layers, pooling layers, and fully connected layers. The deep neural network extracts image features from the detected ROI regions for identification.

In embodiments where the object of interest is a motor vehicle license plate, the extracted image features may be recognized as a combination of a string of characters (eg, Chinese characters, letters, and numbers) using typical image recognition techniques. The recognition result (and optionally, the image itself to be recognized) can be sent to the database DB ( FIG. 2 ) via eg the I/O interface of the image recognition device 210 for recording and querying.

In embodiments where the object of interest is the subject's face, the extracted image features may be used for different purposes, depending on different application scenarios. In the following, the operation of the image recognition module 213 is further described by taking the object of interest as a human face as an example.

In some embodiments, the extracted facial features can be provided to the public security system and compared with the facial features in the ID card information database to identify the subject's identity.

In some embodiments, the extracted facial features may be compared with facial features in a specific information database (eg, a database of suspects at large) to determine whether the subject is a suspect at large.

In some embodiments, the deep neural network of Figure 5A also detects facial regions from registered images, which are electronic photographs corresponding to the subject's identity, such as from an ID card held by the subject The electronic photo of the subject, the electronic copy of the paper photo on the certificate held by the subject taken by the camera CAM (Figure 2), the electronic photo that is stored in the database DB (Figure 2) and registered in advance in the identity of the subject, etc. Wait. Both the facial area extracted from the image to be recognized and the facial area extracted from the registered image are processed into two images having the same size. These two images are then sequentially input into the deep neural network of Figure 5B (which can be a shallower convolutional network) in order to extract facial features. It may be sufficient to use several convolutional layers in this deep neural network, since the content of the two extracted face images is relatively simple. This can reduce convolution operations, increase the running rate, and reduce the requirements for hardware resources.

The facial features extracted from the image to be recognized are compared with the facial features extracted from the registered image to determine whether the subject matches the identity. For example, if the similarity between the facial features from the image to be recognized and the facial features from the registered image is less than a certain threshold, it indicates that the subject is inconsistent with the identity displayed (or previously registered in the database) on the document held by the subject. Otherwise, it is indicated that the subject is using a real identity.

In response to an abnormal event in which the subject does not match the identity, an alert message can be initiated to report to the system platform. This can be accomplished by the alerting module 214 in the image recognition device 210 .

Referring back to Figure 2, the alerting module 214 interacts with the system platform (in this example, the server SRV and the database DB) through appropriate I/O interfaces (not shown). The alert module 214 may send an alert message AM to the system platform, the alert message indicating that the subject does not match the identity. The alert message AM may also contain facial features extracted by the image recognition module 213 from the image to be recognized (taken by the camera CAM) and facial features extracted from the registered image (eg, read from the database DB). Alternatively or additionally, the alarm module 214 may issue an alarm in the form of a sound, a text prompt, or the like at the scene of the image recognition device 210 .

The system platform can utilize different strategies to respond to the alert message AM. In some embodiments, the system platform can compare the two facial features contained in the alarm message AM (one is extracted from the image to be recognized and the other is extracted from the registered image) with the feature database of the public security system, and obtains an impostor identity information of user m and identity information of fraudulent user n. Then, the system platform can send a message (eg, a short message, WeChat message) to the impostor m to remind him to cancel or freeze his credentials. The system platform can also use the identity information of the impostor n to search in the database of the fugitive suspect to confirm whether the impostor n is the fugitive suspect. If the fraudulent user n is confirmed as a fugitive suspect, the system platform can report the result to the public security system, and/or take other measures. For example, the system platform calls surveillance cameras within a certain geographical range around the scene to identify all faces in the images obtained by each surveillance camera, and compares the obtained facial features with the facial features of suspect n, thereby realizing the identification of Tracking of suspect n.

It will be appreciated that anomalies where the subject does not match the identity or other anomalies where the subject's identification fails may also be related to the quality of the image to be identified from the camera CAM. Therefore, in response to these abnormal events, the image recognition device 210 can be triggered to enter the semi-automatic mode described above, wherein the camera parameter adjustment module 212 derives weighted camera parameters from the camera parameters of the first and second versions to adjust the real-time lighting of the scene conditions are taken into account. This helps to further improve the quality of images to be recognized that are subsequently captured by the camera CAM.

In some cases, the adjustment of the imaging parameters of the camera CAM (regardless of whether the real-time lighting conditions of the scene are considered) may still not be sufficient to obtain the image to be recognized with the desired quality. Therefore, additional measures can be taken to obtain further improved image quality. For this purpose, the image recognition device 210 is shown in Figure 2 as further comprising a depth estimation module 215 and an adjustable light source(s) ALS.

In some embodiments, the camera CAM is controlled to capture the sample image in real time before capturing the image to be recognized, and the depth estimation module 215 utilizes a deep neural network to perform depth estimation on the sample image from the camera CAM. The sample image used by the depth estimation module 215 may be the same as the sample image used by the camera parameter adjustment module 212 described above. Alternatively, the sample image used by the depth estimation module 215 may be independent of the sample image used by the camera parameter adjustment module 212, eg, in a time period before or after the time when the sample image used by the camera parameter adjustment module 212 was captured. shoot. Based on the estimated depth, the depth estimation module 215 determines foreground (face) regions and background regions in the sample image, and adjusts the illumination intensity of the adjustable light source(s) ALS according to the respective brightness of the foreground and background regions.

FIG. 6 shows an exemplary and schematic illustration of the operation of the deep neural network used by the depth estimation module 215 . The deep neural network is a modified VGG16 network. The operation of depth estimation is described below with reference to FIG. 6 .

First, the sample image captured by the camera CAM is resized to have a first size, eg, 224×224. The sample image includes three color channels, red (R), green (G), and blue (B), so the resized image is represented as 224×224×3.

Then, the resized sample images are input to the modified VGG16 network such that the modified VGG16 network performs the following operations.

In the first stage, multiple convolutions are performed on the 224×224×3 input image, resulting in 14×14×512 feature layers. In the second stage, the 14×14×512 feature layer is upsampled (upscale=4) to form a 56*56*512 feature layer. Then use a (3, 3, 512, 64) convolution kernel to convolve the 56×56×512 feature layer into a 56×56×64 convolutional layer. Upsampling with upscale=4 is performed again to obtain a convolutional layer of 224×224×64. Then a (3, 3, 64, 1) convolution kernel is used to convolve the 224×224×64 convolutional layer into a 224×224×1 feature layer, which represents depth information in gray values, and So it can be called a depth map.

Based on the estimated depth, the depth estimation module 215 performs a so-called "photometric" operation to determine the foreground and background regions, the brightness of the foreground regions, and the brightness of the background regions in the sample image.

Figure 7 shows an example configuration of depth estimation module 215, camera CAM and adjustable light sources ALS1 and ALS2. In this example configuration, two adjustable light sources ALS1 and ALS2 are arranged to the left and right of the camera CAM, respectively, and the depth estimation module 215 selectively adjusts the adjustable light sources according to the brightness of the foreground area and the brightness of the background area Illumination intensity for ALS1 and ALS2.

FIG. 8 shows a general flow of the operation of the depth estimation module 215.

In step S1, the depth estimation module 215 reads the sample image captured by the camera CAM.

At step S2, the depth estimation module 215 performs depth estimation as described above.

In step S3, the depth estimation module 215 determines the foreground area and the background area in the sample image according to the estimated depth. For example, an area with a depth smaller than a threshold is determined as a foreground area, and an area with a depth greater than the threshold is determined as a background area. The depth estimation module 215 further determines the brightness of the foreground regions and the brightness of the background regions. This can include a variety of situations. For example, when the average brightness of the foreground area is greater than the first threshold, the brightness of the foreground area is determined to be strong; otherwise, the brightness of the foreground area is determined to be weak. Further, it can also be determined whether the brightness of the foreground region is locally weak (for example, the overall variance of the brightness of each pixel in the foreground region is greater than the second threshold) or overall weak (for example, the overall variance of the brightness of each pixel in the foreground region is smaller than the second threshold). threshold). When the average brightness of the background region is greater than the third threshold, the brightness of the background region is determined to be strong; otherwise, the brightness of the background region is determined to be weak. The third threshold may or may not be equal to the first threshold.

In step S4, according to the brightness of the foreground area and the brightness of the background area, the depth estimation module 215 selectively adjusts the illumination intensity of the adjustable light sources ALS1 and ALS2. Specifically, in the case where the two adjustable light sources ALS1 and ALS2 are respectively arranged on both sides of the camera CAM, the adjustment strategy may include: light reduction on both sides, fill light on the dark side and/or light reduction on the bright side, fill light on both sides The sum is not adjusted so that the difference between the brightness of the foreground area and the brightness of the background area is within a predetermined range. The predetermined range can be an empirical value and can be set artificially. More generally, if the difference between the brightness of the foreground area and the brightness of the background area is within a predetermined range (ie, the brightness of the foreground area is close to the brightness of the background area), the adjustable light sources ALS1 and ALS2 do not need to be turned on or adjusted. If the difference between the brightness of the foreground area and the brightness of the background area exceeds the lower limit of the predetermined range (ie, the brightness of the background area is significantly greater than the brightness of the foreground area), the illumination intensity of the adjustable light sources ALS1 and ALS2 can be adjusted. If the difference between the brightness of the foreground area and the brightness of the background area exceeds the upper limit of the predetermined range (ie, the brightness of the background area is significantly smaller than that of the foreground area), the illumination intensity of the adjustable light sources ALS1 and ALS2 can be adjusted. In the last case, the phenomenon of "yin and yang faces (half bright and half dark)" is easy to appear, so the corresponding one of the adjustable light sources ALS1 and ALS2 can be adjusted appropriately to brighten the darker parts in the foreground area .

After the adjustment of the illumination intensity is performed, steps S1 , S2 and S3 are repeated until the brightness of the foreground area and the brightness of the background area in the current sample image are within the predetermined range. As a result, the quality of the image to be recognized can be greatly improved, thereby improving the accuracy of face detection and recognition.

In addition, when the ambient lighting varies uniformly (eg, for sunny outdoors), the depth estimation module 215 may set a lock time period during which adjustment of the adjustable light source is not performed before the image to be recognized is captured, in order to improve the system responsiveness. When the ambient light is prone to sudden changes (eg, for a scene with changing interfering light sources such as a concert), the depth estimation module 215 can perform depth estimation, light metering and adjustment of the adjustable light source in real time to ensure the correctness of facial recognition. For this purpose, the camera CAM or the image recognition device 210 may include a light sensor SNS to sense ambient lighting. The depth estimation module 215 can determine whether the ambient light changes uniformly or is prone to sudden changes according to the detection result from the light sensor SNS, and executes a corresponding lighting strategy. The uniformly varying ambient light may be indicated by the detection result of the light sensor SNS varying less than a threshold within a predetermined period of time.

FIG. 9 illustrates an application scenario in which techniques according to embodiments of the present invention may be applied. This application scenario is, for example, a security check for an open-air venue, in which a participant has previously registered a personal photo (ie, a "registration photo", stored in, for example, a database DB) with the conference organizer, and is accepted at the entrance of the open-air venue Security check. The client as shown in FIG. 9 includes a plurality of functional blocks, which respectively perform the functions of the various modules described above with respect to FIGS. 1-8 . The client can be embodied as an embedded device, such as an embedded video surveillance device integrated with or separate from the camera.

Using the "light prediction" function block, the client learns the changing law of lighting in the venue in different weather and time, and predicts the lighting conditions during the admission time period. Using the "Camera Parameter Adjustment" function block, the client automatically adjusts the camera parameters used by the camera CAM during the entry time period according to the predicted lighting conditions to obtain images with improved quality. During the entry time period, attendees undergo a security check at the entrance of the venue to gain access to the venue. The camera CAM takes a sample image of the subject, and the client uses the "depth estimation and metering" function block to evaluate the brightness of the foreground (face) in the sample image, and adjust the light sources ALS1 and ALS2 in real time accordingly to provide supplementary lighting .

After that, the camera CAM takes a photo of the subject in real time and transmits it to the "image recognition" function block, where the photo taken in real time is compared with the registered photos stored in the database DB. If the comparison result exceeds the threshold, it is considered that the subject has entered with a real identity; otherwise, it is considered that the subject has entered with a false identity. In the latter case, the client, via the "alert" function block, can issue an alert on-site that the subject has failed the security check. Clients can also generate alert messages and send them to the server SRV. The server SRV further searches the information of the examinee in the public security data system. If it is found that the subject is a criminal suspect, the relevant information is returned to the public security video surveillance system (for example, the Sky Eye system) to track and monitor the subject.

10 illustrates generally at 1000 an example system including a camera CAM, adjustable light sources ALS1 and ALS2, and an example computing device 1010 representing one or more systems and/or devices that may implement the various techniques described herein . Computing device 1010 may be, for example, a service provider's server, a device associated with a client (eg, a client device), a system on a chip, and/or any other suitable computing device or computing system. The image recognition apparatus 210 described above with respect to FIG. 2 may take the form of the computing device 1010 .

The example computing device 1010 as illustrated includes a processing system 1011 , one or more computer-readable media 1012 , and one or more I/O interfaces 1013 communicatively coupled to each other. Although not shown, computing device 1010 may also include a system bus or other data and command transfer system that couples the various components to each other. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or processor utilizing any of a variety of bus architectures. local bus. Various other examples are also contemplated, such as control and data lines.

Processing system 1011 represents functionality that uses hardware to perform one or more operations. Accordingly, processing system 1011 is illustrated as including hardware elements 1014 that may be configured as processors, functional blocks, and the like. This may include implementation in hardware as an application specific integrated circuit or other logic device formed using one or more semiconductors. The hardware element 1014 is not limited by the materials from which it is formed or the processing mechanism employed therein. For example, a processor may be composed of semiconductor(s) and/or transistors (eg, electronic integrated circuits (ICs)). In such a context, the processor-executable instructions may be electronically-executable instructions.

Computer readable medium 1012 is illustrated as including memory/storage 1015 . Memory/storage 1015 represents memory/storage capacity associated with one or more computer-readable media. Memory/storage 1015 may include volatile media (such as random access memory (RAM)) and/or non-volatile media (such as read only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory/storage 1015 may include fixed media (eg, RAM, ROM, fixed hard drives, etc.) as well as removable media (eg, flash memory, removable hard drives, optical discs, etc.). The computer-readable medium 1012 may be configured in various other ways as described further below.

One or more input/output (I/O) interfaces 1013 represent functionality that allows commands and data to be communicated to and received from computing device 1010 . I/O interface 1013 may be implemented by any suitable communication interface and communication protocol.

Computing device 1010 also includes an image recognition application 1016 . Image recognition application 1016 may be stored in memory/storage device 1015 as computational program instructions. The image recognition application 1016 may, in conjunction with the processing system 1011 , the computer readable medium 1012 and the I/O interface 1013 , implement the full functionality of the various modules of the image recognition device 210 described with respect to FIG. 2 .

Various techniques may be described herein in the general context of software hardware elements or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The terms "module," "function," and "component" as used herein generally refer to software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform independent, meaning that the techniques can be implemented on a variety of computing platforms with a variety of processors.

An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media that can be accessed by computing device 1010 . By way of example and not limitation, computer-readable media may include "computer-readable storage media" and "computer-readable signal media."

"Computer-readable storage medium" refers to media and/or devices capable of persistent storage of information, and/or tangible storage devices, as opposed to mere signal transmissions, carrier waves, or signals per se. Thus, computer-readable storage media refers to non-signal bearing media. Computer-readable storage media includes, for example, volatile and nonvolatile, removable and non-removable media and/or are suitable for storage of information such as computer-readable instructions, data structures, program modules, logic elements/circuits or other data. ) hardware such as a storage device implemented by a method or technology. Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROMs, digital versatile disks (DVDs) or other optical storage devices, hard disks, cassettes, magnetic tape, magnetic disk storage A device or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing desired information and accessible by a computer.

"Computer-readable signal medium" refers to a signal-bearing medium that is configured to transmit instructions to hardware of computing device 1010, such as via a network. Signal media typically embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave, data signal or other transport mechanism. Signaling media also includes any information delivery media. The term "modulated data signal" refers to a signal that encodes information in the signal in such a manner as to set or change one or more of its characteristics. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired as well as wireless media such as acoustic, RF, infrared, and other wireless media.

As previously mentioned, hardware elements 1014 and computer-readable media 1012 represent instructions, modules, programmable device logic, and/or fixed device logic implemented in hardware that, in some embodiments, may be used to implement the techniques described herein. At least some aspects. Hardware elements may include integrated circuits or systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, a hardware element may act as a processing device for performing program tasks defined by the instructions, modules and/or logic embodied by the hardware element, as well as a hardware device for storing instructions for execution, eg, previously described computer-readable storage medium.

Combinations of the foregoing may also be used to implement the various techniques and modules described herein. Accordingly, software, hardware or program modules and other program modules may be implemented as one or more instructions and/or logic embodied on some form of computer-readable storage medium and/or by one or more hardware elements 1014 . Computing device 1010 may be configured to implement specific instructions and/or functions corresponding to software and/or hardware modules. Thus, a module may be implemented at least in part in hardware as a module executable by computing device 1010 as software, eg, through the use of computer-readable storage media and/or hardware elements 1014 of the processing system. The instructions and/or functions may be executable/operable by one or more articles of manufacture (eg, one or more computing devices 1010 and/or processing system 1011 ) to implement the techniques, modules, and examples described herein.

In various implementations, computing device 1010 may take various different configurations. Each of these configurations includes devices that may have generally different configurations and capabilities, and thus computing device 1010 may be configured according to one or more of different device classes. For example, computing device 1010 may be implemented as a computer-type device including a personal computer, desktop computer, multi-screen computer, laptop computer, netbook, and the like. Computing device 1010 may also be implemented as a mobile device-type device including mobile devices such as mobile phones, portable music players, portable gaming devices, tablet computers, multi-screen computers, and the like. Computing device 1010 may also be implemented as a television-type device, including televisions, set-top boxes, game consoles, and the like.

The techniques described herein may be supported by these various configurations of computing device 1010 and are not limited to the specific examples of the techniques described herein. Computing device 1010 may interact with "cloud" 1020 as a platform for the system. In some embodiments, the functionality of computing device 1010 may also be implemented in whole or in part on "cloud" 1020 through the use of a distributed system, such as through platform 1030 as described below.

Cloud 1020 includes and/or represents a platform 1030 for resources 1032 . The platform 1030 abstracts the underlying functionality of the hardware (eg, servers) and software resources of the cloud 1020 . Resources 1032 may include applications and/or data that may be used when computer processing is performed on servers remote from computing device 1010 . Resources 1032 may also include services provided over the Internet and/or over subscriber networks, such as cellular or Wi-Fi networks.

Platform 1030 may abstract resources and functions to connect computing device 1010 with other computing devices. Platform 1030 may also be used to abstract the hierarchy of resources to provide a hierarchy of corresponding levels of requirements encountered for resources 1032 implemented via platform 1030 . Thus, in an interconnected device embodiment, implementation of the functions described herein may be distributed throughout the system 1000 . For example, functionality may be implemented in part on computing device 1010 and through platform 1030 that abstracts the functionality of cloud 1020 .

In the discussion herein, various different embodiments are described. It should be appreciated and understood that each embodiment described herein may be used alone or in association with one or more of the other embodiments described herein.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims

Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed subject matter, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, the indefinite articles "a" or "an" do not exclude a plurality, and "a plurality" means two or more. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims (15)

1. An image recognition method, comprising: Predict the lighting conditions in future periods based on multiple images captured by the camera at multiple consecutive moments; generating imaging parameters to be sent to the camera based at least on the predicted lighting conditions; initiating sending of the camera parameters to the camera so that the camera captures an image to be identified based on the camera parameters in the future time period; and An object of interest is identified from the to-be-identified image. 2. The method of claim 1, wherein said predicting lighting conditions for future periods comprises: Use the residual network to sequentially extract the respective illumination features of the multiple images; Recursively processing the respective lighting features using a temporal recurrent neural network, wherein the residual network and the temporal recurrent neural network have been trained to learn changes in lighting conditions over time; and the lighting conditions predicted by the temporal recurrent neural network output, wherein the plurality of moments includes N moments, N is a positive integer greater than 1, and Wherein, both the illumination feature of X _k+1 and the processing result of the temporal recurrent neural network on the illumination feature of X _k are input to the temporal recurrent neural network for recursive processing, where X _k represents the N time instants , and X _k+1 denotes an image captured at the k+1 th time among the N time instants, where k is an integer and 1≦k<N. 3. The method of claim 2, wherein the temporal recurrent neural network comprises a long short term memory network. 4. The method of claim 1, wherein said generating the camera parameters to be sent to the camera comprises using a residual structure-based neural network to map the predicted lighting conditions to the camera parameters of the first version, wherein The residual structure-based neural network has been trained to learn a mapping of lighting conditions to camera parameters, and wherein the first version of the camera parameters may be selected as the camera parameters to be sent to the camera. 5. The method of claim 4, wherein the residual structure-based neural network comprises a plurality of concatenated residual structures, and wherein the mapping of the predicted lighting conditions to the camera parameters of the first version comprises: The features characterizing the predicted lighting conditions are input into the residual structure-based neural network, so that the residual structure-based neural network performs operations including: reshape the features that characterize the predicted lighting conditions; Convolve the reshaped features; apply an activation function to the convolved features; Perform batch normalization on the features after applying the activation function to get a normalized feature map; processing the normalized feature map using the plurality of concatenated residual structures; and The processed feature maps are fed into a fully connected layer to obtain the first version of the camera parameters. 6. The method of claim 4, further comprising: controlling the camera to capture a sample image in real time before capturing the to-be-recognized image; based on the sample image, generating a second version of the camera parameters; and performing a weighted average of the camera parameters of the first version and the camera parameters of the second version to obtain weighted camera parameters, which may be selected as the camera parameters to be sent to the camera ,in: in a first mode of operation, the first version of the camera parameters is selected as the camera parameters to be sent to the camera, and In a second operating mode different from the first operating mode, the weighted imaging parameters are selected as the imaging parameters to be sent to the camera. 7. The method of claim 1, wherein the object of interest comprises a subject's face, and wherein the identifying the object of interest comprises: using a second deep neural network to detect face regions from the to-be-identified image; and Facial features are extracted from the detected facial regions in the to-be-identified image using a third deep neural network for identifying the subject's identity. 8. The method of claim 7, further comprising: Using the second deep neural network to detect facial regions from a registered image, wherein the registered image is an electronic photograph corresponding to the identity of the subject; extracting facial features from the detected facial regions in the registered image using the third deep neural network; and The facial features extracted from the image to be recognized are compared with the facial features extracted from the registered image to determine whether the subject matches the identity. 9. The method of claim 8, further comprising initiating reporting of an alert message to a system platform in response to an abnormal event in which the subject does not match the identity. 10. The method of claim 1, further comprising: controlling the camera to capture a sample image in real time before capturing the to-be-recognized image; Using a first deep neural network to estimate the depth of the sample image; Determine the foreground area and the background area, the brightness of the foreground area, and the brightness of the background area in the sample image according to the estimated depth; adjusting the illumination intensity of at least one adjustable light source associated with the camera in response to a difference between the brightness of the foreground area and the brightness of the background area being outside a predetermined range; and The steps of real-time shooting, estimating and determining are repeated until the brightness of the foreground region and the brightness of the background region in the current sample image are within the predetermined range. 11. The method of claim 10, wherein the estimating the depth of the sample image comprises: resizing the sample image to have a first size; and inputting the resized sample image into the first deep neural network, causing the first deep neural network to perform operations comprising: performing multiple convolutions on the sample image of the first size; and The convolved sample images are repeatedly processed to obtain a depth map of the first size, wherein the processing includes upsampling and convolution. 12. An image recognition device, comprising: an illumination prediction module, configured to predict illumination conditions in future periods based on a plurality of images captured by the camera at a plurality of consecutive moments; a camera parameter adjustment module configured to generate camera parameters to be sent to the camera based at least on the predicted lighting conditions, and initiate sending of the camera parameters to the camera so that the camera can use the camera parameters for the future time period based on the camera parameters shooting the image to be identified with the camera parameters; and An image recognition module configured to recognize an object of interest from the to-be-recognized image. 13. A computing device comprising a memory and a processor, the memory configured to store thereon computer program instructions that, when executed on the processor, cause the processor to perform claim 1 The method of any one of -11. 14. A computer-readable storage medium having stored thereon computer program instructions which, when executed on a processor, cause the processor to perform the method of any of claims 1-11. 15. An image recognition system, comprising: cameras; and The computing device of claim 13.