JP2018185623A - Object detection device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately detect an individual object from a photographed image in which a space where congestion may occur is photographed. An image analysis device uses a density estimator that learns the image characteristics of each density image obtained by photographing a space in which an object exists at a predetermined density at the density of the object. The density estimation means 50 for estimating the density distribution, the candidate positions where individual objects can exist in the captured image are set, and the candidate positions correspond to each of a plurality of parts constituting the object in the captured image. A partial area is set, a partial evaluation value indicating the degree to which the image feature of the part corresponding to the partial area appears is calculated for each partial area, and a partial evaluation value of a plurality of partial areas set based on the candidate position is obtained. The object position determination means 51 is provided, in which a candidate position whose integrated evaluation value satisfies a predetermined determination criterion is determined as an object position by emphasizing the lower the density of the partial region. [Selection diagram] Fig. 2

Description

  The present invention relates to an object detection device that detects individual objects from a captured image in which a space in which a predetermined object such as a person can exist is captured, and in particular, an individual object from a captured image in which a space in which congestion can occur is captured. The present invention relates to an object detection device for detecting the above.

  In an event venue or other space where congestion can occur, countermeasures such as placing a large number of guards in the crowded area are required to prevent accidents. Therefore, monitoring cameras can be arranged at various locations in the venue to estimate the distribution of people from the captured images and display the estimated distribution, thereby facilitating the understanding of the congestion situation by the monitoring staff.

  At that time, the position of each person is detected, and a model imitating the person's shape is displayed at each detected position, or / and the positional relationship of the person (for example, forming a matrix or surrounding) Further analysis efficiency can be expected by analyzing and notifying the analysis result.

  The method of detecting the position of each person from a photographed image taken by multiple people, combining multiple models that imitate people, and applying it to the photographed image, as well as feature values of images taken by a single person in advance There is a method of detecting a position where an image feature of a single person appears from a captured image using a previously prepared image feature of a single person, such as a method of scanning a captured image using a discriminator that has been learned.

  For example, in the moving object tracking device described in Patent Document 1, a moving object model imitating the shape of the moving object being tracked is extracted at the position where the changed pixel is extracted by comparing the monitoring image and the background image. The positions of individual moving objects are detected by combining and applying the number of moving objects. In this moving object tracking apparatus, the use of a moving object model that approximates the shape of a person's whole body is exemplified.

  Further, for example, the object detection device described in Patent Document 2 detects a person from an input image using a classifier previously learned using a large number of “human” image data and “non-human” image data. . It has been suggested that the classifier used by this object detection apparatus is learned using image data of the whole body of a person.

JP2012-159958 JP2011-186633

  However, in a captured image in which a space where congestion can occur is captured, the concealment state of a person changes according to the congestion state. Therefore, as an image feature of a single person, there is a problem that it is difficult to accurately detect an individual person if the image feature of each part of the object is always evaluated uniformly regardless of the congestion state. .

  That is, for a captured image in which there are many people who are not crowded and the whole body has been photographed, both in the method using a model imitating a person and in the method using a discriminator that learns a human image, The person using the image feature can be detected with higher accuracy than using a part of the image feature (for example, only near the head).

  On the other hand, for captured images that are congested and frequently concealed, both in the method using a model imitating a person and in the method using a discriminator that learns a human image, the image features of the whole body are used. However, it is possible to detect the person with higher accuracy by using only the image feature that is less likely to be concealed.

  Therefore, the person photographed on the left side of the crowded area in the photographed image is photographed on the left side, the person photographed on the right is the right part thereof, the person photographed on the upper part is photographed on the top, and below. The person who is deciding that the lower part is more important than the other parts can detect it with higher accuracy.

  In addition, for example, if image features only near the head are always used in order to increase detection accuracy during congestion, detection accuracy when congestion does not occur decreases, and detection accuracy when congestion does not occur increases. For this reason, if the image features of the whole body are always used, the detection accuracy at the time of congestion decreases.

  In this way, in the captured image in which a space where congestion can occur is captured, the individual concealment state of the object to be detected changes according to the congestion state, so that each part of the object is always evaluated uniformly. It has been difficult to accurately detect individual objects from the captured image.

  The present invention has been made in view of the above problems, and provides an object detection device that can accurately detect individual objects in a captured image even in a captured image of a space in which congestion can occur. The purpose is to do.

  In order to achieve such an object, the present invention is an object detection device for detecting individual objects from a captured image in which a space in which congestion due to a predetermined object may occur is captured, and the object is detected at each predetermined density. Density estimation means for estimating the density distribution of an object photographed in the photographed image using a density estimator that has learned the image characteristics of each density image photographed in the space where the object exists, and each object in the photographed image An image of a portion corresponding to each partial region is set for each partial region by setting a candidate region that can exist and setting a partial region corresponding to each of a plurality of portions constituting the object in the captured image with reference to the candidate position Integrated evaluation value that calculates the partial evaluation value that represents the degree of the appearance of features and integrates the partial evaluation values of multiple partial areas that are set based on the candidate position, focusing on the lower the density of the partial areas. To provide an object detecting apparatus comprising the, and object position determining means for determining the position of the object candidate positions satisfying a predetermined criterion.

  In addition, the object position determination unit sets a weighting factor that is higher as the density in the partial area is lower for each partial area and is lower as the density in the partial area is higher, and performs partial evaluation of a plurality of partial areas that are set based on the candidate position. It is preferable to calculate the integrated evaluation value by weighting and summing the values with the weighting coefficients of the partial areas.

  In addition, it is preferable that the object position determination unit sets a partial region corresponding to a smaller portion of the portions constituting the object as the density at the candidate position is higher.

  In addition, the object position determination means includes a position generation means for generating a plurality of different arrangements each including one or more candidate positions, and a plurality of portions based on each candidate position for each of the plurality of arrangements. The model image generation means for generating a model image by drawing a partial model imitating the part in the corresponding partial area, and the similarity of the captured image of the partial model for each partial area for each of the model images in a plurality of arrangements An evaluation value calculating means for calculating a partial evaluation value representing the degree of the evaluation, integrating partial evaluation values of a plurality of partial areas to calculate an integrated evaluation value, and a candidate position in an arrangement with the maximum integrated evaluation value as an object position And an optimum arrangement determining means for determining.

  In addition, the object position determination means includes candidate position setting means for setting a plurality of candidate positions in the photographed image, and image features of partial areas corresponding to each of the plurality of parts set on the basis of each candidate position. The partial evaluation value of the partial area is calculated by inputting the image feature into the learned classifier, and the partial evaluation values of the partial areas set based on the candidate position are integrated for each candidate position to obtain the integrated evaluation value. It is preferable to include evaluation value calculation means for calculating, and position determination means for determining a candidate position where an integrated evaluation value that satisfies the determination criterion is calculated as the position of the object.

  According to the present invention, it is possible to accurately detect individual objects from a captured image in which a space where congestion can occur is captured.

It is a block diagram which shows the schematic structure of the image monitoring apparatus which concerns on 1st and 2nd embodiment. It is a functional block diagram which shows the function of the image monitoring apparatus which concerns on 1st and 2nd embodiment. It is a functional block diagram which shows the function of the image monitoring apparatus which concerns on 1st embodiment. It is the figure which represented typically an example of the information of the object model which the object model memory | storage means which concerns on 1st embodiment has memorize | stored. It is the figure which represented typically an example of the information of the object model which the object model memory | storage means which concerns on 1st embodiment has memorize | stored. It is the figure which represented typically an example of the weight (ratio of weighting coefficient) which the weight memory | storage means which concerns on 1st embodiment has memorize | stored. It is the figure which showed typically the example of a process by the density estimation means, arrangement | positioning production | generation means, and model image generation means which concern on 1st embodiment. It is the image which showed typically the weighting coefficient which the model image production | generation means calculated corresponding to each partial area | region of the model image 743 which concerns on 1st embodiment. It is the flowchart which showed operation | movement of the image monitoring apparatus which concerns on 1st and 2nd embodiment. It is a flowchart of the object position determination process by the image monitoring apparatus which concerns on 1st embodiment. It is a flowchart of the object position determination process by the image monitoring apparatus which concerns on 1st embodiment. It is a functional block diagram which shows the function of the image monitoring apparatus which concerns on 2nd embodiment. It is the figure which represented typically the information of the partial identifier which the partial identifier memory | storage means based on 2nd Embodiment has memorize | stored, and the information of the weight which the weight memory | storage means has memorize | stored. It is the figure which showed typically a mode that the evaluation value calculation means which concerns on 2nd embodiment calculates an integrated score about a candidate position. It is a flowchart of the object position determination process by the image monitoring apparatus which concerns on 2nd embodiment.

[First embodiment]
Hereinafter, as an embodiment of the present invention, an example of an image monitoring device 1 that includes an example of an object detection device that detects an individual person from a captured image obtained by shooting an event venue, and that displays a detection result to a monitor will be described. To do. In the image monitoring apparatus 1 according to this embodiment, in particular, the object detection device detects an individual person using an object model that imitates a person, and the object detection apparatus at that time includes a plurality of partial models that constitute the object model. Includes examples of using to detect individual people.

<Configuration of Image Monitoring Device 1 according to First Embodiment>
FIG. 1 is a block diagram showing a schematic configuration of the image monitoring apparatus 1. The image monitoring apparatus 1 includes a photographing unit 2, a communication unit 3, a storage unit 4, an image processing unit 5, and a display unit 6.

  The photographing unit 2 is a surveillance camera and is connected to the image processing unit 5 via the communication unit 3. The photographing unit 2 shoots the monitoring space at a predetermined time interval to generate a photographed image, and sequentially captures the photographed image to the image processing unit 5. It is a photographing means to input. For example, the imaging unit 2 is installed on a pole installed at an event site with a view of the monitoring space. The visual field may be fixed, or may be changed according to a schedule in advance or an instruction from the outside via the communication unit 3. Further, for example, the imaging unit 2 captures the monitoring space with a frame period of 1 second and generates a color image. A monochrome image may be generated instead of the color image.

  The communication unit 3 is a communication circuit, one end of which is connected to the image processing unit 5 and the other end is connected to the photographing unit 2 and the display unit 6 via a communication network such as a coaxial cable, a LAN (Local Area Network), or the Internet. Connected. The communication unit 3 acquires a captured image from the imaging unit 2 and inputs the acquired image to the image processing unit 5, and outputs the detection result input from the image processing unit 5 to the display unit 6.

  The storage unit 4 is a memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), and stores various programs and various data. The storage unit 4 is connected to the image processing unit 5 and inputs / outputs such information to / from the image processing unit 5.

  The image processing unit 5 is configured by an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit). The image processing unit 5 is connected to the storage unit 4, operates as various processing units / control units by reading out and executing a program from the storage unit 4, and stores and reads various data in the storage unit 4. The image processing unit 5 is also connected to the imaging unit 2 and the display unit 6 via the communication unit 3 and detects individual persons by analyzing the captured image acquired from the imaging unit 2 via the communication unit 3. The detection result is displayed on the display unit 6 via the communication unit 3.

  The display unit 6 is a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display, and is a display unit that is connected to the image processing unit 5 via the communication unit 3 and displays a detection result by the image processing unit 5. . The monitor visually checks the displayed detection result to determine the occurrence of congestion, and takes measures such as changing the personnel arrangement as necessary.

  In the present embodiment, the image monitoring apparatus 1 in which the number of the photographing units 2 and the image processing units 5 is 1: 1 is illustrated, but in another embodiment, the number of the photographing units 2 and the image processing units 5 is illustrated. Can be many-to-one or many-to-many.

<Function of the image monitoring apparatus 1 according to the first embodiment>
2 and 3 are functional block diagrams showing functions of the image monitoring apparatus 1. The communication unit 3 functions as the image acquisition unit 30, the object position output unit 31, and the like, and the storage unit 4 functions as the density estimator storage unit 40, the single feature storage unit 41, and the like. The image processing unit 5 functions as a density estimation unit 50, an object position determination unit 51, and the like. The single feature storage unit 41 includes functions as an object model storage unit 410a and a weight storage unit 412a. The object position determination unit 51 includes an arrangement generation unit 510a, a model image generation unit 512a, an evaluation value calculation unit 514a, and an optimum arrangement determination. The function as the means 516a is included.

  The image acquisition unit 30 sequentially acquires captured images from the imaging unit 2 that is an imaging unit, and sequentially outputs the acquired captured images to the density estimation unit 50 and the object position determination unit 51.

  The density estimator storage unit 40 is an estimated density calculation function that learns the image features of each density image obtained by photographing a space where an object (person) exists at a predetermined density for each predetermined density. When input, an estimated value (estimated density) of the density of an object photographed in the image is calculated, and information on an estimator (density estimator) that outputs the calculated estimated density is stored in advance. That is, parameters such as the coefficient of the estimated density calculation function are stored in advance as information on the density estimator.

  The density estimation unit 50 extracts a density estimation feature amount (estimation feature amount) from each part of the captured image input from the image acquisition unit 30 and reads the density estimator from the density estimator storage unit 40 to extract it. Each estimated feature amount is input to a density estimator to estimate an estimated density distribution (density distribution), and the estimated density distribution is output to the object position determination unit 51.

  The density estimation process and the density estimator will be specifically described.

  The density estimation means 50 sets a window (estimation extraction window) at the position of each pixel of the captured image, and extracts an estimation feature amount from the captured image in each estimation extraction window. The estimation feature amount is a GLCM (Gray Level Co-occurrence Matrix) feature.

  It is desirable that the area in the monitoring space photographed by each estimation extraction window is the same size. That is, preferably, the density estimation means 50 reads out the camera parameters of the photographing unit 2 stored in advance from a camera parameter storage means (not shown), and is photographed at an arbitrary pixel of the photographed image by homography conversion using the camera parameters. The estimation feature amount is extracted after the captured image is deformed so that the areas in the monitoring space have the same size.

  The density estimator can be realized by a classifier that identifies multi-class images, and can be a discrimination function learned by a multi-class SVM (Support Vector Machine) method.

Density, for example, there is no human "Background" class is 0 people / m higher than ² is two / m ² or less "low density" class, higher than two / m ² 4 persons / m ² or less It can be defined as 4 classes of “medium density” class, “high density” class higher than 4 persons / m ² .

  The estimated density is a value given in advance to each class, and is a value output as a result of distribution estimation. In the present embodiment, values corresponding to each class are expressed as “background”, “low density”, “medium density”, and “high density”.

  That is, the density estimator applies the multi-class SVM method to the feature quantities of a large number of images (density images) belonging to the “background” class, “low density” class, “medium density” class, and “high density” class. This is an identification function for discriminating the images of each class from other classes. The parameters of the discriminant function derived by this learning are stored as a density estimator. The feature amount of the density image is the same type as the estimation feature amount and is a GLCM feature.

  The density estimation means 50 acquires the estimated density which is the output value by inputting each of the estimation feature quantities extracted corresponding to each pixel to the density estimator. In addition, when the estimated feature amount is extracted by deforming the captured image, the density estimating unit 50 deforms the density distribution into the original captured image shape by homography conversion using the camera parameter.

  A collection of estimated densities for each pixel of the captured image thus obtained is a density distribution.

From the density distribution output by the density estimation means 50, the density of people at various locations in the photographed image can be understood, but the position of each person cannot be determined from the density distribution.
On the other hand, the object position determination means 51 subsequent to the density estimation means 50 is a means for determining the position of each person appearing in the captured image.

  The object position determination means 51 detects an individual object by searching a place where an image feature as a single person (object) appears on the captured image, and determines the position of the individual object. That is, the object position determination unit 51 sets a candidate position where an individual object can exist in the photographed image, and represents the degree to which the image feature (single feature) of a single object appears in the photographed image at the candidate position. An evaluation value (integrated evaluation value) is calculated, and a candidate position whose integrated evaluation value is greater than or equal to a predetermined value is determined as the position of the object. For example, the single feature is a human shape, and the single feature storage unit 41 stores the single feature in advance. Further, for example, the integrated evaluation value is a similarity between a captured image edge and a model representing a human shape. The integrated evaluation value is an evaluation value obtained by integrating evaluation values (partial evaluation values) for a plurality of parts constituting the object.

  Here, in a captured image in which a space in which congestion can occur is captured, the higher the density, the easier the concealment occurs, and thus the reliability of the evaluation value is considered to be low. On the contrary, it is considered that the lower the density is, the less concealment occurs and the reliability of the evaluation value is high. In addition, in a captured image in which a space where congestion can occur is captured, there can be many objects in which a highly reliable portion and a low portion are mixed, such as an object existing at the boundary of density.

Therefore, the object position determination unit 51 improves the detection accuracy of the object by weighting according to the density for each part of the object with reference to the density distribution, instead of evaluating the entire object uniformly.
That is, the object position determination unit 51 sets candidate positions where individual objects can exist in the photographed image, and sets partial areas corresponding to each of a plurality of parts constituting the object in the photographed image with the candidate position as a reference. Set and calculate a partial evaluation value indicating the degree of appearance of the image feature of the part corresponding to the partial region for each partial region, and set the partial evaluation value of each partial region set based on the candidate position as the partial region The candidate position where the integrated evaluation value that is integrated with emphasis as the density of the image satisfies a predetermined determination criterion is determined as the position of the object. More specifically, the object position determination unit 51 sets a weighting factor that is higher as the density in the partial area is lower for each partial area and lower as the density in the partial area is higher. The integrated evaluation value is calculated by weighting and summing the partial evaluation values of the partial areas with the weighting coefficients of the partial areas.

  In addition, the higher the density, the higher the possibility that severe concealment will occur. If the hidden part is also evaluated, the chance of erroneous evaluation increases, leading to a decrease in object detection accuracy. For example, when the photographing unit 2 is installed in a bird's-eye view, concealment is more likely to occur as it is closer to the foot, and concealment is less likely to occur as it is closer to the head. Considering this, if the single feature is only the head of a person so as to adapt to the congestion, the detection loss at the time of congestion is reduced. However, since the single feature of the head only has a relatively high integrated evaluation value for the shoulder or the like, the number of false detections when there is no congestion increases.

  The object position determination unit 51 refers to the density distribution to eliminate the trade-off between the number of parts to be evaluated and the detection accuracy of each object. That is, the object position determination unit 51 sets a partial area corresponding to a smaller part of the parts constituting the object as the density at the candidate position is higher. That is, the object position determination unit 51 calculates the integrated evaluation value by evaluating the image features of a smaller part of the parts constituting the object as the density at the candidate position is higher. For example, if the estimated density of the candidate positions is low density, the object position determination unit 51 evaluates the image characteristics of the whole body and calculates an integrated evaluation value, and if it is medium density, evaluates the image characteristics of the upper body and performs integrated evaluation. A value is calculated, and if it is high density, an image feature near the head is evaluated to calculate an integrated evaluation value.

  Hereinafter, detection of individual objects and single unit characteristics will be described.

  The single feature storage unit 41 includes an object model storage unit 410a that stores in advance information on an object model that imitates the shape of a single person (object), and a weight storage unit 412a that stores information on weights used in calculation of evaluation values in advance. And stores object model information and weight information as single features.

  4 to 6 are diagrams schematically showing the single features stored in the single feature storage means 41. FIG. 4 and 5 are examples of object model information stored in the object model storage unit 410a, and FIG. 6 is an example of weights (weight coefficient ratios) stored in the weight storage unit 412a. is there.

  The object model stored in the object model storage unit 410a is specifically a three-dimensional model 700 including three spheroids corresponding to the head, torso, and legs of a standing person. Note that the center of gravity of the head is the representative position of the person. Further, the object model storage unit 410a stores an evaluation range 702 for each density together with the stereo model 700, and stores the camera parameters 701 of the imaging unit 2 in order to project the stereo model 700 onto the coordinate system of the captured image. ing. The camera parameters 701 include external parameters such as the installation position and imaging direction of the imaging unit 2 in the actual monitoring space, internal parameters such as the focal length, angle of view, lens distortion and other lens characteristics of the imaging unit 2 and the number of pixels of the imaging device. It is information to include.

The evaluation range 702 is set for each density divided into a plurality of parts, and the higher the density, the smaller the part that constitutes a single object.
Specifically, the object model storage means 410a associates the values representing the low density class with the six parts “upper 1/3, left 1/2”, “up 1/3, right 1/2”, Evaluation representing "middle 1/3, left 1/2", "middle 1/3, right 1/2", "lower 1/3, left 1/2" and "lower 1/3, right 1/2" Remember the range. When these six parts are combined, it becomes the “whole” of the person.
In addition, the object model storage unit 410a associates the value representing the medium density class with the four parts "upper 1/3, left 1/2", "up 1/3, right 1/2", "middle 1". / 3, left 1/2 "and evaluation ranges representing" middle 1/3, right 1/2 "are stored. When these four parts are combined, it becomes the “upper 2/3” of a person.
The object model storage unit 410a also associates the values representing the high-density class with the evaluation ranges representing the four parts “upper 1/3, left 1/2” and “upper 1/3, right 1/2”. Is remembered. When these two parts are combined, it becomes the “upper third” of the person.

  Hereinafter, the object model 710 represented by a combination of the evaluation range “upper ３, left ½”, the solid model 700 and the camera parameter 701 is the upper left model, and the evaluation range “upper ３, right ½”. The partial model 711 represented by the combination of the stereo model 700 and the camera parameter 701 is represented by the upper right model, the evaluation range “middle 1/3, left 1/2”, and the combination of the stereo model 700 and the camera parameter 701. The partial model 712 represented by the combination of the left middle model, the evaluation range “middle 1/3, right 1/2”, the solid model 700 and the camera parameter 701 is the right middle model, and the evaluation range “lower”. The partial model 714 represented by the combination of the 1/3, the left 1/2 ", the solid model 700, and the camera parameter 701 is displayed on the lower left. Model, evaluation range "lower third, right 1/2" and referred to as the lower right model partial model 715 represented by a combination of a three-dimensional model 700 and the camera parameter 701. An object model 720 represented by a combination of the evaluation range “whole”, the stereoscopic model 700 and the camera parameter 701 is represented by a whole body model, and an evaluation range “upper 2/3”, a combination of the stereoscopic model 700 and the camera parameter 701. The object model 721 represented by the combination of the upper body model, the evaluation range “upper 1/3”, the three-dimensional model 700, and the camera parameter 701 is referred to as a head vicinity model.

As described above, the object model storage unit 410a is associated with the low density class, and includes the upper left model 710, the upper right model 711, the left middle model 712, the right middle model 713, the lower left model 714, and the lower right model 715. The model 720 is associated with the middle density class, and the upper body model 721 including the upper left model 710, the upper right model 711, the left middle model 712, and the right middle model 713 is associated with the higher density class, and the upper left model 710 and the upper right model. A head vicinity model 722 comprising a part model 711 is stored as object model information.
Note that what is drawn as the partial model is a contour line (solid line portion in FIG. 5) representing the shape of the object.

  The weight is a degree of emphasizing the part according to the density of the partial region corresponding to each part of the object, and is represented by a relative ratio between the densities. Since the lower density part is more important and the higher density part is disregarded, the lower the density, the higher the density and the lower the weight. For example, the weights for low density, medium density and high density can be 10: 7: 5. In determining the density of the partial region, the background class can be regarded as a low density class.

  As described above, the weight storage unit 412a stores the weight for each density, which is higher as the density is lower and lower as the density is higher.

  The arrangement generation unit 510a generates a plurality of different arrangements each including one or more candidate positions, and outputs the generated arrangements to the model image generation unit 512a.

  For this purpose, the arrangement generation unit 510a selects one or more pixels (arrangement number) of pixels that have an estimated density of low density, medium density, or high density from among the pixels of the captured image based on random numbers. Arrangement is generated by randomly determining the position of each determined pixel as a candidate position. The arrangement generation unit 510a generates a plurality of different arrangements by repeating this generation by a predetermined number of times for each arrangement number while sequentially increasing the number of arrangements. Note that the upper limit of the number of arrangements can be the upper limit of the number of objects that can exist in the monitoring space. For example, it is possible to arrange a standing person's three-dimensional model in a virtual space imitating the monitoring space without overlapping. It can be calculated as a number.

  The model image generation unit 512a draws a partial model imitating the part in a partial region corresponding to each of a plurality of parts with respect to each candidate position for each of the plurality of arrangements input from the arrangement generation unit 510a. To generate a model image. At that time, the model image generation unit 512a draws an object model that imitates a small part of the parts constituting a single object at each candidate position as the density at the candidate position is higher. The generated model images are output to the evaluation value calculation means 514a.

  For this purpose, the model image generation unit 512a reads the camera parameters from the object model storage unit 410a, and uses the camera parameters for each arrangement to set each candidate position to the height of the center of gravity of the head of the stereo model (for example, 1.5 m). The representative position in the virtual space imitating the monitoring space of the three-dimensional model projected at the candidate position is calculated by back-projecting to the horizontal plane.

  Further, the model image generation unit 512a reads out the head vicinity model from the object model storage unit 410a, arranges the head vicinity model at a representative position in the virtual space corresponding to each candidate position, and uses the camera parameters to The neighborhood model is projected onto the coordinate system of the captured image. Then, the model image generation unit 512a refers to the density distribution input from the density estimation unit 50 and totals the estimated density in the projection region of the head vicinity model corresponding to each candidate position, and the largest number at each candidate position. The estimated density (excluding the background class) is determined as the density of the candidate position.

In addition, the model image generation unit 512a reads an object model corresponding to the density of the candidate positions from the object model storage unit 410a for each candidate position. Specifically, the model image generating unit 512a reads a whole body model including six partial models if the density of candidate positions is low, and reads an upper body model including four partial models if the density is high. If it is a density, a head vicinity model composed of two partial models is read out. Then, for each arrangement, the model image generating unit 512a arranges the object model read corresponding to each candidate position at a representative position in the virtual space corresponding to the candidate position, and uses the camera parameters to identify each object model. Each partial model to be performed is projected on the coordinate system of the captured image, and the shape (contour line) of the object is drawn to generate a model image for each arrangement.
Note that the model image generation unit 512a sequentially projects from the object model arranged at the representative position that is far from the photographing unit 2, and generates a model image expressing the concealment between the object models by overwriting the projection area. .

In addition, the model image generation unit 512a calculates a concealment degree that represents the degree of overlap between the object models in the model image for each arrangement according to the following equation.
Concealment degree = area of overlapping area between object models / area of sum area of projection area of object model (1)

  In addition, the model image generation unit 512a calculates a weighting coefficient corresponding to the estimated density of the partial area for each partial area in the model image, corresponding to each model image.

For this purpose, the model image generation unit 512a aggregates the estimated densities in each partial region with reference to the density distribution, and determines the highest estimated density in each partial region as the density of the partial region. However, the background class is counted as a low density class.
Next, the model image generation unit 512a reads the weight (weight coefficient ratio) corresponding to the density of each partial region from the weight storage unit 412a, and obtains the sum of the weights of all the partial regions for each arrangement.
Subsequently, for each arrangement, the model image generation unit 512a divides the weight of each partial region by the sum of the weights, and calculates a weight coefficient for the partial region. That is, the sum of the weighting coefficients in each arrangement is normalized so as to be 1.

  The model image generation unit 512a associates the layout, the model image, the concealment degree, and the weighting coefficient thus obtained and outputs them to the evaluation value calculation unit 514a.

FIG. 7 is a diagram schematically illustrating a processing example by the density estimation unit 50, the arrangement generation unit 510a, and the model image generation unit 512a according to the first embodiment.
The image 740 is an image of the density distribution estimated by the density estimation unit 50. In the density distribution, the white area is the area where the estimated density is the background, the horizontal line area is the area where the estimated density is low density, the shaded area is the area where the estimated density is medium density, and the lattice area is the high density estimated area Each region is shown.
The image 741 is obtained by plotting the eight candidate positions included in the arrangement generated by the arrangement generation unit 510a in the coordinate system of the photographed image with x marks.
The three-dimensional model 742 illustrates a state in which the model image generating unit 512a arranges the three-dimensional model at the representative positions in the virtual space corresponding to the eight candidate positions shown in the image 741.
The image 743 is created by the model image generation unit 512a specifying the density of each candidate position based on the density distribution shown in the image 740, and projecting a three-dimensional model in the evaluation range corresponding to the density to each candidate position. A model image is shown.

  In the arrangement represented by the model image 743, one partial area (the left middle part of the middle right person in the group) is completely hidden, and 33 partial areas are drawn. The breakdown of the 33 partial areas is 23 with low density or background, 6 with medium density, and 4 with high density. The sum of the weighting factor ratios for all partial regions in the arrangement represented by the model image 743 is 10 × 23 + 7 × 6 + 5 × 4 = 292. Of the 33 partial regions, the partial region 744 has a low density, the partial region 745 has a medium density, and the partial region 746 has a high density. The weighting factor of the partial region 744 is 10/292, the weighting factor of the partial region 745 is 7/292, and the weighting factor of the partial region 746 is 5/292. The weighting coefficient is calculated in the same manner for other partial areas.

  An image 750 in FIG. 8 is an image schematically showing the weighting coefficient calculated by the model image generating unit 512a corresponding to each partial region of the model image 743.

  The evaluation value calculation unit 514a calculates an integrated evaluation value representing the degree of similarity of the model image input from the model image generation unit 512a to the captured image for each of the plurality of arrangements, and sets the integrated evaluation value for each arrangement to the optimal arrangement. It outputs to the determination means 516a.

Specifically, the evaluation value calculation unit 514a calculates the weighted similarity between each model image and the captured image according to the following equation.
Weighted similarity = Weighted shape conformity-W _Ha x Concealment (2)
However, W _Ha is a weighting coefficient larger than 0, and is preset based on a prior experiment. The degree of concealment subtracted from the weighted shape matching degree is a penalty value for suppressing an overlap of excessive object models. In this way, by determining the optimum arrangement based on the similarity including the concealment degree, it is possible to prevent erroneous detection of the object position due to the application of an object model equal to or more than the original number of objects.

  The evaluation value calculation means 514a calculates the weighted shape suitability by summing the shape suitability for each partial area of the model image and the captured image with the weighting coefficient of the partial area. The shape suitability for each partial area is a partial evaluation value. The shape matching degree can be an edge similarity. The evaluation value calculation unit 514a extracts an edge from each model image and the photographed image, and for each model image, a pixel from which a valid edge is extracted from the model image and an edge of the pixel of the photographed image corresponding to the pixel. The absolute value of the difference is calculated and summed, and the sum is divided by the sum of the number of pixels from which valid edges are extracted from the captured image and the number of pixels from which valid edges are extracted from the model image, and the sign is inverted. The value is calculated as a weighted shape suitability.

  Alternatively, the evaluation value calculation unit 514a generates an edge image from each model image and the captured image, and for each model image, chamfer matching between the edge image generated from the captured image and the edge image generated from the model image is performed. The value obtained by inverting the sign of the distance obtained by performing (Chamfer Matching) is divided by the sum of the number of pixels from which the valid edges are extracted from the captured image and the number of pixels from which the valid edges are extracted from the model image, The weighted shape conformity of the model image can also be used.

The optimum arrangement determining unit 516a refers to the integrated evaluation value for each arrangement input from the evaluation value calculating unit 514a, determines the candidate position in the arrangement having the maximum integrated evaluation value as the object position, and information on the determined object position Is output to the object position output means 31. That is, the optimum arrangement determining unit 516a determines each candidate position included in the arrangement where the maximum similarity is calculated as the position of each person photographed in the photographed image.
For example, the optimum arrangement determining unit 516a generates and outputs information on the object position by drawing an object model in each object position in a color-coded manner according to the density of the object position so that the observer can easily recognize it. Alternatively, the object position information may be the coordinate value of the object position itself, and the object position information may be a region of each drawn object model that does not overlap with other object models. Alternatively, the object position information may be data including two or more of the above-described data.

  The object position output unit 31 sequentially outputs the object position information input from the object position determination unit 51 to the display unit 6, and the display unit 6 displays the object position information input from the object position output unit 31. For example, the information on the object position is transmitted / received via the Internet and displayed on the display unit 6. The monitoring person grasps a point where the monitoring space is congested by viewing the displayed information, and takes measures such as dispatching or increasing the number of guards at the point.

<Operation of the image monitoring apparatus 1 according to the first embodiment>
The operation of the image monitoring apparatus 1 will be described with reference to the flowcharts of FIGS.

  When the image monitoring apparatus 1 starts operation, the image capturing unit 2 installed in the event venue captures the monitoring space every predetermined time and sequentially transmits the captured images to the image analysis center in which the image processing unit 5 is installed. To do. The image processing unit 5 repeats the operation according to the flowchart of FIG. 9 every time a captured image is received.

  First, the communication unit 3 operates as the image acquisition unit 30 and waits to receive a captured image from the imaging unit 2. The image acquisition means 30 that acquired the captured image outputs the captured image to the image processing unit 5 (step S1).

  The image processing unit 5 to which the photographed image is input operates as the density estimating means 50, and estimates the density distribution from the photographed image (step S2). The density estimation unit 50 extracts the estimation feature quantity at the position of each pixel of the captured image, reads the density estimator from the density estimator storage unit 40 of the storage unit 4, and uses each estimation feature quantity as the density estimator. The density distribution is estimated by inputting and obtaining the estimated density at each pixel of the captured image.

  The image processing unit 5 that has estimated the density distribution also operates as the object position determination unit 51. The captured image is input from the image acquisition unit 30 and the density distribution is input from the density estimation unit 50 to the object position determination unit 51. . The object position determination means 51 that has received these confirms whether the density distribution includes an estimated density other than the background class (step S3).

  If the estimated density other than the background class is included (YES in step S3), the object position determination unit 51 determines the position of each object from the captured image, assuming that at least one person is captured. A determination process is performed (step S4). On the other hand, in the case of only the background class (NO in step S3), the processes in steps S4 and S5 are omitted assuming that no person is photographed.

  The object position determination process in step S4 will be described with reference to the flowcharts of FIGS. The single feature storage unit 41 operates as the object model storage unit 410a and the weight storage unit 412a, and the object position determination unit 51 operates as the arrangement generation unit 510a, the model image generation unit 512a, the evaluation value calculation unit 514a, and the optimum arrangement determination unit 516a. Then, the object position determination process is executed.

  The arrangement generation unit 510a sequentially sets the number of arrangements within a range from 1 to the upper limit number (step S100), and controls the loop processing of steps S100 to S117.

  The arrangement generation unit 510a prepares a variable T for counting the number of iterations, initializes T to 0 (step S101), and starts the iterative process of steps S102 to S116.

  Next, the arrangement generation unit 510a sets the same number of candidate positions as the number of arrangements set in step S100 in the low density, medium density, or high density region in the density distribution input from the density estimation unit 50. By setting at random, a T-th arrangement in the arrangement number is generated and output to the model image generation unit 512a (step S102).

  The model image generation unit 512a reads camera parameters from the object model storage unit 410a, and converts each candidate position included in the arrangement generated in step S102 into three-dimensional coordinates in the virtual space using the camera parameters (step S103). .

  Next, the model image generation unit 512a prepares and initializes a model image having the same size as the captured image, calculates the distance to the imaging unit 2 of the three-dimensional coordinates of each candidate position, and the candidate position with a long distance Are sequentially set as processing targets (step S104), and the loop processing of steps S104 to S110 is executed.

  Subsequently, the model image generating unit 512a specifies the density of the candidate position to be processed with reference to the density distribution (step S105). The model image generation unit 512a reads out the head vicinity model from the object model storage unit 410a, arranges it in the three-dimensional coordinates of the candidate position, and projects the head vicinity model onto the coordinate system of the captured image using the camera parameters. The highest estimated density (except for the background class) in the projection area is specified as the density of candidate positions.

Subsequently, the model image generation unit 512a reads the object model corresponding to the density specified in step S105 from the object model storage unit 410a (step S106), arranges it at the three-dimensional coordinates of the candidate position to be processed, and sets the camera parameter. Is used to over-project the placed object model onto the model image (step S107). At this time, the model image generation means 512a records the projected area of the object model.
The object model is composed of a plurality of partial models, and the contours of the objects in the plurality of partial models are overwritten and projected at each position based on the candidate position in the model image. A projection area obtained by projecting a partial model of each part is a partial area of the part. The model image generation unit 512a assigns an identification number to the combination of the candidate position and the part, and sets the identification number of the projected part together with the intensity value of the contour of the object in the partial model to the pixel value of the partial area corresponding to the part. To do.

  Subsequently, the model image generating unit 512a specifies the density of each partial region with reference to the density distribution (step S108). The model image generation means 512a specifies the highest estimated density (however, the background class is regarded as a low density class) in each partial area as the density of the partial area.

  Subsequently, the model image generation unit 512a refers to the weight storage unit 412a and sets the weight (weight coefficient ratio) corresponding to the density of each partial region (step S109). That is, the model image generating unit 512a records the weight of each partial area together with the identification number of the part corresponding to the partial area. At this time, the weight of the partial area completely hidden on the model image by the overwriting projection in step S107 is deleted from the recording.

  Then, the model image generating means 512a checks whether or not all candidate positions included in the Tth arrangement in the current arrangement number have been processed (step S110), and if there is an unprocessed candidate position ( NO in step S110), the process returns to step S104 to process the next candidate position.

  On the other hand, when all the candidate positions have been processed (YES in step S110), the model image for the Tth arrangement in the current arrangement number is completed. The model image generating means 512a that has completed the model image calculates the degree of concealment of the object model in the model image (step S111). That is, the model image generation unit 512a obtains the area of the projection area on the model image that is “the area of the sum area of the model projection areas” and sums up the projection areas for each object model recorded in step S107. Then, by subtracting the area of the sum area of the projected areas of the model from the total value, the “area of the overlapping area between the models” is obtained, and these are substituted into Equation (1) to calculate the degree of concealment.

  The model image generating means 512a calculates the weighting coefficient of each partial area by summing the weights of the partial areas recorded in step S109 and dividing the weight of each partial area by the total value (step S112).

  The model image generating unit 512a that has calculated the weighting coefficient outputs the model image, the degree of concealment, and the weighting factor for each partial area to the evaluation value calculating unit 514a.

  The evaluation value calculation means 514a, to which the model image, the concealment degree, and the weighting coefficient are input, calculates the shape matching degree for each partial area of the model image and the captured image as a partial evaluation value of the partial area (step S113), and further From the weighted shape matching degree and the concealment degree obtained by summing up the products of the shape matching degree for each partial area and the weight coefficient of the partial area, the weighted similarity between the model image and the photographed image is calculated for the Tth arrangement in the current arrangement number. The integrated evaluation value is calculated (step S114). That is, the evaluation value calculation unit 514a generates an edge image from each of the model image and the photographed image input from the model image generation unit 512a, and the degree of similarity for each partial region of these edge images and the weighting coefficient for each partial region. To calculate the weighted shape suitability. Then, the weighted similarity is calculated by substituting the weighted shape matching degree and the concealment degree into Expression (2).

When the weighted similarity for the Tth arrangement in the current arrangement number is calculated, the evaluation value calculation unit 514a records the arrangement and the weighted similarity in association with each other, and the arrangement generation unit 510a sets the number of iterations T to one. Increment (step S115) and compare with the specified number of times T _MAX (step S116). If T is less than T _MAX (NO in step S116), the process returns to step S102 to repeat the current arrangement number. Let it continue.

When the number of iterations T has reached the specified number of times T _MAX (YES in step S116), the arrangement generation unit 510a ends the iterative process for the current arrangement number, and confirms whether all the arrangement numbers have been set. (Step S117). If there is an unset number of arrangements (NO in step S117), the process returns to step S100 to perform processing for the next number of arrangements.

  On the other hand, when all the arrangement numbers have been set (YES in step S117), evaluation value calculation means 514a inputs the arrangement recorded in step S115 and the weighted similarity to optimum arrangement determination means 516a to determine the optimum arrangement. The means 516a identifies an arrangement having the maximum weighted similarity among them (step S118), and determines that the arrangement is information indicating the position of each person photographed in the photographed image.

  The description will be continued with reference to FIG. 9 again. The object position determination unit 51 outputs information on the position (object position) of each person determined in step S4 to the communication unit 3 (step S5). The communication unit 3 to which the object position information is input operates as the object position output unit 31 and transmits the object position information to the display unit 6.

  When the above process is completed, the process returns to step S1, and the process for the next captured image is performed.

[Second Embodiment]
Hereinafter, as a preferred embodiment of the present invention different from the first embodiment, an individual person is detected using a discriminator that has learned the image characteristics of a single person, and in particular, a plurality of persons constituting a single person are detected. An example of the image monitoring apparatus 1 including an example of an object detection apparatus that detects individual persons using a partial classifier that has learned the image characteristics of each part will be described.

  The image monitoring apparatus according to the second embodiment differs from the image monitoring apparatus according to the first embodiment in the details of the single feature stored in the single feature storage unit 41 and the details of the processing performed by the object position determination unit 51. A part of the general configuration, the general function, and the operation is common. Therefore, a part of the schematic configuration, the schematic function, and the operation will be described with reference again to the block diagram of FIG. 1, the functional block diagram of FIG. 2, and the flowchart of FIG. 9 respectively referred to in the first embodiment. .

<Configuration of Image Monitoring Device 1 according to Second Embodiment>
The schematic configuration of the image monitoring apparatus 1 according to the second embodiment will be described with reference to the block diagram of FIG.
As in the first embodiment, the image monitoring apparatus 1 includes a photographing unit 2 that photographs a monitoring space at predetermined time intervals and outputs a photographed image, and a display unit 6 that receives information on an object position and displays the information. An image processing unit 5 that acquires a captured image, detects an individual person (object) from the captured image, and generates and outputs information on the position (object position) of the detected object. In addition to being connected to the communication unit 3 through which input and output of position information and the like are connected, a storage unit 4 that stores programs and various data and inputs and outputs them is connected to the image processing unit 5.

<Function of the image monitoring apparatus 1 according to the second embodiment>
With reference to the functional block diagrams of FIGS. 2 and 12, functions of the image monitoring apparatus 1 according to the second embodiment will be described.

  Similar to the first embodiment, the communication unit 3 receives a captured image from the image capturing unit 2 and outputs the captured image to the density estimation unit 50 and the object position determination unit 51 and the object position determination unit 51. A function as an object position output means 31 or the like for outputting information on the object position to the display unit 6.

  In addition, as in the first embodiment, the storage unit 4 stores a density estimator that learns the image features of each density image obtained by photographing a space where an object exists at the predetermined density for each predetermined density. It includes a function as an estimator storage unit 40 and a unit feature storage unit 41 that stores an image feature (single unit feature) of a single object generated by pre-learning. The single feature is information on the image feature for each part constituting the object and information on the weight for the density.

  Further, as in the first embodiment, the image processing unit 5 estimates the density distribution of the object photographed in the photographed image by scanning the photographed image with the density estimator, and determines the estimated density distribution as the object position. The density estimation means 50 output to the means 51, and an evaluation value representing the degree to which the image feature of a single object appears in the photographed image at the candidate position by setting candidate positions where individual objects can exist in the photographed image (Integrated evaluation value) is calculated, and a function as an object position determination unit 51 or the like that determines the candidate position that satisfies the determination criterion as the position of the object and outputs information on the object position to the object position output unit 31. The object position determination means 51 sets a partial area corresponding to each of a plurality of parts constituting the object in the captured image with reference to the candidate position, and an image of a part corresponding to the partial area for each partial area Calculate the partial evaluation value that represents the degree that the sign appears, and calculate the integrated evaluation value by focusing on the partial evaluation value of each partial area set based on the candidate position as the density of the partial area is lower To do.

  However, as described above, the details of the processing performed by the object position determination unit 51 according to the second embodiment and the details of the single feature stored in the single feature storage unit 41 are the image monitoring apparatus according to the first embodiment. Different from 1. These points will be described with reference to the functional block diagram of FIG.

  The single feature storage unit 41 according to the second embodiment is a partial identification that stores in advance a classifier (partial classifier) for each part that has learned each image feature of a plurality of parts constituting a single person (object). It functions as a unit storage unit 411b and a weight storage unit 412b that stores in advance the weight information used in calculating the evaluation value, and stores the partial classifier information and the weight information as a single feature.

  FIG. 13 shows the single feature stored in the single feature storage unit 41 according to the second embodiment, that is, the partial classifier information stored in the partial classifier storage unit 411b and the weight storage unit 412b. It is the figure which represented the information of the weight which has been shown typically.

  Each of the partial classifiers for each part of the object calculates a partial score (partial evaluation value) indicating the likelihood that the image is an image of the part (partial image) when the feature amount of the image is input. It is expressed by the coefficient of the score calculation function to be output. The partial discriminator storage unit 411b stores parameters such as the correspondence between the relative position and density of each part from the reference position and the partial discriminator to be used, such as a coefficient representing each partial discriminator. The reference position is, for example, the center of gravity of the head.

The partial discriminator can be, for example, a discriminator that learns by applying the linear SVM method to the feature amount of a learning image including a partial image and a large number of unmanned images for a large number of people. When linear SVM is used as the learning algorithm, the coefficient of the score calculation function is a weight vector. This weight vector is a weight for each element of the feature quantity, and the value of the inner product of the feature quantity of the input image and the weight vector represents the partial score. In learning, when the inner product of the weight vector and the feature amount is 0 or more, adjustment is made so that it is identified as a partial image of a person, and when it is less than 0, it is not a partial image of a person. Therefore, the threshold value that is applied to the integrated score and identifies whether the input image is an image of a single person is 0 in principle, and the threshold value can usually be set to 0. However, the threshold value may be set to a value smaller than 0 in order to reduce the error of identifying a single person image as not being a single person image.
The feature amount of the learning image may be, for example, a HOG (Histograms of Oriented Gradients) feature amount.

Specifically, the partial discriminator storage unit 411b associates the values representing the low density class with the six parts "upper 1/3, left 1/2", "up 1/3, right 1/2". ”,“ Middle 1/3, Left 1/2 ”,“ Middle 1/3, Right 1/2 ”,“ Lower 1/3, Left 1/2 ”and“ Lower 1/3, Right 1/2 ” The partial discriminators 800 to 805 that have learned image features are stored. When these six parts are combined, it becomes the “whole” of the person.
Hereinafter, “Up 1/3, Left 1/2” is upper left, “Up 1/3, Right 1/2” is upper right, “Middle 1/3, Left 1/2” is left middle, “Middle 1 / 3, right 1/2 "is called the right middle part," lower 1/3, left 1/2 "is called the lower left part, and" lower 1/3, right 1/2 "is called the lower right part. Further, the upper left partial discriminator 800 is the upper left discriminator, the upper right partial discriminator 801 is the upper right discriminator, the left middle partial discriminator 802 is the left middle discriminator, and the right middle partial discriminator. The unit 803 is referred to as a right middle classifier, the lower left partial classifier 804 is referred to as a lower left classifier, and the lower right partial classifier 805 is referred to as a lower right classifier. In addition, a set 810 of six partial discriminators 800 to 805 whose evaluation range is “whole” is referred to as a total discriminator.
The partial discriminator storage unit 411b stores an upper left discriminator 800, an upper right discriminator 801, a left middle discriminator 802, and a right middle discriminator 803 in association with values representing the medium density class. When the evaluation ranges of these four partial classifiers are combined, it becomes the “upper 2/3” of the person. Together, a set 811 of four partial classifiers 800 to 803 having an evaluation range of “upper 2/3” is referred to as an upper body classifier.
The partial discriminator storage unit 411b stores an upper left discriminator 800 and an upper right discriminator 801 in association with values representing high density classes. When the evaluation ranges of these two partial classifiers are combined, it becomes “upper １ ／” of a person. Together, the set 812 of two partial discriminators 800 and 801 whose evaluation range is “upper 1/3” is referred to as a head vicinity discriminator.

As described above, the partial discriminator storage unit 411b is associated with the low density class, the upper left discriminator 800, the upper right discriminator 801, the left middle discriminator 802, the right middle discriminator 803, the lower left discriminator 804, and the right An overall classifier 810 comprising a lower classifier 805 is associated with a middle density class, and an upper body classifier 811 comprising an upper left classifier 800, an upper right classifier 801, a left middle classifier 802, and a right middle classifier 803, A head vicinity discriminator 812 including an upper left discriminator 800 and an upper right discriminator 801 is stored in association with the high density class together with a threshold value applied to the integrated evaluation value.
Here, an example in which partial identifiers of the same part are shared between the densities is shown, but the division of the parts may be different for each density.

  The weight 820 is a degree of emphasizing the part according to the density of the partial region corresponding to each part of the object, and is represented by a relative ratio between the densities. Since the lower density part is more important and the higher density part is disregarded, a lower weight 820 is set as the density is lower and the density is higher. For example, the low density, medium density, and high density weights 820 can have a ratio of 10: 7: 5. In determining the density of the partial region, the background class can be regarded as a low density class.

  As described above, the weight storage unit 412b stores the weight for each density, which is higher as the density is lower and lower as the density is higher.

  The candidate position setting unit 511b sets a plurality of candidate positions in the captured image at predetermined intervals, and outputs the set candidate positions to the evaluation value calculation unit 514b. Specifically, the predetermined interval is one pixel, and the candidate position setting unit 511b sequentially sets the position of each pixel of the captured image as the candidate position. The candidate position represents the human head center of gravity.

The evaluation value calculation unit 514b sets a partial region corresponding to each of a plurality of portions constituting a single object in the photographed image with each candidate position input from the candidate position setting unit 511b as a reference. A partial evaluation value of the partial area is calculated by inputting the image characteristic of the partial area corresponding to each of the partial areas to the partial classifier that has learned the image characteristic of the part, and a part set based on the candidate position for each candidate position The integrated evaluation value is calculated by emphasizing and integrating the partial evaluation value of the region as the density of the partial region is lower, and the calculated integrated evaluation value and information associated therewith are output to the position determining unit 517b.
At that time, the evaluation value calculation unit 514b sets a partial area corresponding to a smaller part of the parts constituting the single object as the density at the candidate position is higher at each candidate position.

  For this purpose, the evaluation value calculation unit 514b sets an upper 1/3 window at each candidate position, refers to the density distribution input from the density estimation unit 50, and totals the estimated density in the window. However, the background class is excluded. Then, the evaluation value calculation unit 514b determines the highest estimated density at each candidate position as the density of the candidate position.

  Further, the evaluation value calculation unit 514b reads out information on the partial classifiers corresponding to the density of each candidate position from the partial classifier storage unit 411b, and a partial classifier associated with each candidate position corresponding to the density of the candidate position. Corresponding windows (partial areas) are set, and feature quantities for identification (identification feature quantities) are extracted from captured images in each partial area. These partial areas have the shape of the partial image (solid-line rectangle shown in FIG. 13) used for learning of the partial classifier of each part, and are windows that are enlarged or reduced at a plurality of predetermined magnifications. It is. 6 partial areas that are "overall" for a low density candidate position, 4 partial areas that are "upper 2/3" for a medium density candidate position, high When combined with the density candidate position, two partial regions that are “upper 1/3” of the person are set. The feature amount for identification is the same type as the feature amount of the learning image, and is a HOG feature amount.

  Further, the evaluation value calculation means 514b inputs the identification feature quantity of each part to the partial classifier of the part, and acquires the partial score that is the output value as the partial evaluation value.

  Further, the evaluation value calculation unit 514b calculates a weighting coefficient corresponding to the estimated density of the partial area for each partial area.

For this purpose, the evaluation value calculation means 514b refers to the density distribution and totals the estimated density in each partial area, and determines the highest estimated density in each partial area as the density of the partial area. However, the background class is counted as a low density class.
Next, the evaluation value calculation unit 514b reads the weight (ratio of weighting factors) corresponding to the density of each partial region from the weight storage unit 412b, and sums the partial region weights set based on the candidate position for each candidate position. Ask for.
Subsequently, for each candidate position, the evaluation value calculation means 514b divides the weight of each partial area by the sum of the weights of all partial areas and calculates the weight coefficient of the partial area. That is, the sum of the weighting coefficients at each candidate position is normalized so as to be 1.
In this way, the evaluation value calculation unit 514b sets a weighting factor that is higher as the density in the partial area is lower for each partial area and lower as the density in the partial area is higher.

  Further, for each candidate position, the evaluation value calculation unit 514b calculates an integrated evaluation value by weighting and summing the partial evaluation values of each partial area set with reference to the candidate position with the weighting coefficient of the partial area.

That is, the partial score by the upper left classifier is S _UL , the partial score by the upper right classifier is S _UR , the partial score by the left middle classifier is S _ML , the partial score by the right middle classifier is S _MR , and the lower left classifier S _LL , the partial score by the lower right discriminator as S _LR , the upper left weighting factor is W _UL , the upper right weighting factor is W _UR , the left middle weighting factor is W _ML , and the right middle weight is Assuming that the coefficient is W _MR , the lower left weight coefficient is W _LL , and the lower right weight coefficient is W _LR , the evaluation value calculation means 514 b calculates the integrated score as follows.

The evaluation value calculation unit 514b calculates an integrated score of the candidate position according to the following equation if the density of the candidate position of interest is low.
Integrated score = W _UL S _UL + W _UR S _UR + W _ML S _ML + W _MR S _MR
+ W _LL S _LL + W _LR S _LR (3)
Moreover, if the density of the candidate position of interest is medium density, the evaluation value calculation unit 514b calculates the integrated score of the candidate position by the following formula.
Integrated score = W _UL S _UL + W _UR S _UR + W _ML S _ML + W _MR S _MR (4)
Further, if the density of the candidate position of interest is high, the evaluation value calculation unit 514b calculates an integrated score of the candidate position by the following formula.
Integrated score = W _UL S _UL + W _UR S _UR (5)

  FIG. 14 is a diagram schematically illustrating how the evaluation value calculation unit 514b calculates an integrated score for each candidate position illustrated in FIG. 7 when the density distribution illustrated in FIG. 7 is obtained. The image 830 shows the relationship between each part and the weighting coefficient for three candidate positions having a low density among these candidate positions. The image 831 shows the relationship between each part and the weighting coefficient for three candidate positions having a medium density. The image 832 shows the relationship between each part and the weighting coefficient for two candidate positions having a high density.

For example, since the candidate position 840 has a low density, six partial areas are set on the basis of the candidate position 840, and the density of each of the six partial areas is low. Is 10 and the sum is 60. Therefore, the weighting coefficient becomes W _UL = W _UR = W _ML = W _MR = W _LL = W _LR = _10/60 , and these weighting coefficients and the partial scores of the respective parts are substituted into the expression (3) to obtain the integrated score. Calculated.
Further, for example, the candidate position 841 has a medium density, and therefore, four partial areas are set based on the candidate position 841. The upper two of the four partial areas have a medium density, and thus the weighting factor. The ratio is 7, and the lower two are low in density, so the weight coefficient ratio is 10, and the sum is 34. Therefore, the weighting factors are W _UL = W _UR = _7/34 , W _ML = W _MR = 10/34, and the integrated score is calculated by substituting these weighting factors and the partial scores of each part into the equation (4). The
Further, for example, since the candidate position 842 has a high density, two partial areas are set based on the candidate position 842, and the density of each of the two partial areas is high because the density of the two partial areas is high. The ratio is 5 and the sum is 10. Therefore, the weighting coefficient becomes W _UL = W _UR = 5/10, and the integrated score is calculated by substituting these weighting coefficients and the partial scores of the respective parts into the equation (5).
The integrated score is similarly calculated for other candidate positions.

  Then, the evaluation value calculation unit 514b outputs, for each candidate position, information that associates the candidate position, the density of the candidate position, the integrated score, and the sum area (integrated window) of the used partial areas to the position determining unit 517b.

  The position determining unit 517b refers to the information input from the evaluation value calculating unit 514b, and determines a candidate position where an integrated evaluation value that satisfies a predetermined criterion is calculated as the position of the object.

  Specifically, the position determination unit 517b extracts candidate positions whose integrated score is equal to or greater than a predetermined threshold (for example, 0), and a plurality of adjacent candidate positions having the same density and being close to each other. Candidate positions (candidate positions where the overlap between integrated windows is larger than a predetermined ratio) are combined into one, and the combined candidate positions are determined as positions where a person is photographed.

  The process of grouping the candidate positions is performed in order to cope with the fact that a high integrated score is calculated for the same person in the vicinity in addition to the position where the person is actually photographed. Specifically, for example, for each density of candidate positions, the position determination unit 517b sequentially sets candidate positions for which an integrated score equal to or greater than the threshold is calculated as the attention position in descending order of the integrated score, and the integrated score is higher than the attention position. A low candidate position is set as a comparison position. Then, the position determination unit 517b deletes one of the plurality of candidate positions by deleting information on the comparison position in which the overlap between the integrated window of the comparison position and the integrated window of the target position is larger than a predetermined ratio. To summarize.

  Then, the position determination unit 517b outputs the position where the person is photographed and the determined candidate position to the object position output unit 31 as object position information.

<Operation of Image Monitoring Device 1 according to Second Embodiment>
Hereinafter, the operation of the image monitoring apparatus 1 according to the second embodiment will be described with reference to FIGS. 9 and 15.

  When the image monitoring apparatus 1 starts its operation, as in the first embodiment, the image capturing unit 2 sequentially transmits captured images, and the image processing unit 5 operates in accordance with the flowchart of FIG. 9 every time it receives a captured image. repeat.

  The communication unit 3 operates as the image acquisition unit 30, receives a captured image, and outputs it to the image processing unit 5 (step S1). The image processing unit 5 to which the photographed image is input operates as the density estimating unit 50, reads the density estimator from the density estimator storage unit 40 of the storage unit 4, and scans the photographed image with the density estimator, thereby density distribution. Is estimated (step S2).

  Next, the image processing unit 5 operates as the object position determination unit 51. The object position determination unit 51 receives the captured image from the image acquisition unit 30 and the density distribution from the density estimation unit 50, and the density distribution is not a background class. It is confirmed whether or not the estimated density is included (step S3).

  When the estimated density other than the background class is included (YES in step S3), the object position determination unit 51 performs a process of determining the position of each object from the captured image (step S4), and only the background class In the case of (NO in step S3), the processes in steps S4 and S5 are omitted.

  The object position determination process in step S4 will be described with reference to the flowchart in FIG. The single feature storage unit 41 operates as the partial classifier storage unit 411b and the weight storage unit 412b, and the object position determination unit 51 operates as the candidate position setting unit 511b, the evaluation value calculation unit 514b, and the position determination unit 517b. Judgment processing is executed.

  The candidate position setting unit 511b sequentially sets the position of each pixel in the captured image as a candidate position and inputs it to the evaluation value calculation unit 514b (step S200), and controls the loop processing of steps S200 to S206.

  The evaluation value calculation unit 514b that has received the candidate position specifies the density of the candidate position with reference to the density distribution (step S201). That is, the evaluation value calculating means 514b sets a window defined in the shape of the upper third of a single person at the candidate position, and uses the highest estimated density (except for the background class) in the window as the density of the candidate position. Identify.

  The evaluation value calculation unit 514b that identifies the density of the candidate positions reads out a plurality of partial classifiers corresponding to the density from the partial classifier storage unit 411b, sets a partial area corresponding to each partial classifier, and sets the partial area in the partial area Are extracted from the captured images (step S202), and the extracted identification features are input to the corresponding partial classifiers to calculate respective partial scores (partial evaluation values) (step S203). .

  The evaluation value calculation means 514b that has calculated the partial evaluation value refers to the density distribution and specifies the density of each partial region (step 204). That is, the evaluation value calculation means 514b specifies the highest estimated density in each partial area (however, the background class is regarded as the low density class) as the density of the partial area.

  The evaluation value calculation unit 514b that identifies the density of each partial region reads the weighting factor ratio according to the density of each partial region from the weight storage unit 412b, and calculates the weighting factor ratio of each partial region as the weighting factor of all partial regions. The weight coefficient of the partial area is calculated by dividing by the sum of the ratios, and the integrated evaluation value of the candidate position is calculated by multiplying the calculated weight coefficient by the partial evaluation value calculated in step S203 (step S205). . When the density of the candidate area is low, the product is summed according to Expression (3), when the density is medium, according to Expression (4), and when the density is high, according to Expression (5).

  The evaluation value calculating unit 514b records the candidate position, the sum area window (integrated window) of the partial areas, the density of the candidate position, and the integrated evaluation value in association with each other, and records all the pixels of the captured image. Is determined as a candidate position (step S206). If there is an unset pixel (NO in step S206), the process returns to step S200 to process the position of the next pixel. To do.

  On the other hand, when the positions of all the pixels have been set as the candidate positions (YES in step S206), position determination means 517b determines the candidate position, the integrated window, the density of candidate positions, and the integrated evaluation value recorded in step S206. The group whose integrated evaluation value is less than the threshold value is deleted from the group of () (step S207), and for each group that remains without being deleted, the integration window of each other is higher than a predetermined ratio for each density of candidate positions. Large overlapping sets are grouped into one set as those of the same person (step S208). Then, the position determination unit 517b determines each set of candidate positions after being collected as the position (object position) of each person photographed in the photographed image.

  The description will be continued with reference to FIG. 9 again. The object position determination unit 51 outputs the information on the object position determined in step S4 to the communication unit 3 (step S5), and the communication unit 3 operates as the object position output unit 31 to display the object position information on the display unit 6. Send.

  When the above process is completed, the process returns to step S1, and the process for the next captured image is performed.

<Modification>
(1) In each of the above-described embodiments and modifications thereof, an example in which the object to be detected is a person has been shown. However, the present invention is not limited thereto, and the object to be detected is a vehicle, an animal such as a cow or a sheep, or the like. You can also.

  (2) In each of the above-described embodiments and modifications thereof, an example is shown in which a partial region is set with a unit obtained by dividing the object into three parts in the height direction and two parts in the width direction. Absent. Depending on the difference in the characteristics of the detection target, the monitoring space to be photographed, the feature amount to be used, the type of the evaluation value, and the like, the partial areas can be divided at different ratios suitable for each. Moreover, you may overlap and set a partial area | region.

  (3) The weights shown in the above embodiments and the modifications thereof are examples, and are suitable for each according to the difference in the characteristics of the detection target, the monitoring space to be photographed, the feature amount to be used, the type of evaluation value, and the like. It can be another value.

(4) In each of the above-described embodiments and modifications thereof, the density estimator learned by the multi-class SVM method has been exemplified, but instead of the multi-class SVM method, a decision tree type random forest method, a multi-class Various density estimators such as a density estimator learned by the AdaBoost method or the multi-class logistic regression method can be used.
Alternatively, a density estimator using a discriminating CNN (Convolutional Neural Network) may be used.

(5) In each of the above-described embodiments and modifications thereof, the class of density other than the background estimated by the density estimator is set to three classes, but the class may be divided more finely.
In this case, instead of the three-stage weights, finer-stage weights corresponding to the classification can be used, and the classes and the weights can be associated with each other and stored in the single feature storage unit 41. Alternatively, classes and three-stage weights can be stored in the single feature storage unit 41 in a many-to-one correspondence.

(6) In each of the above-described embodiments and modifications thereof, a density estimator that classifies into multiple classes is illustrated, but instead of this, a regression type density estimation that regresses a density value (estimated density) from a feature quantity It can also be a container. That is, a density estimator that has learned the parameters of the regression function for obtaining the estimated density from the features by ridge regression method, support vector regression method, regression tree-type random forest method or Gaussian Process Regression, etc. can do.
Alternatively, a density estimator using a regression type CNN may be used.
In these cases, the estimated density range output as a continuous value instead of the density class value is stored in the single feature storage unit 41 in association with the partial model and weight, or the partial classifier and weight. In these cases, the density of the partial area may be the average value or the median value of the estimated densities in the partial area, in addition to the highest estimated density in the partial area.

  (7) In each of the above-described embodiments and modifications thereof, the GLCM feature is exemplified as the feature amount learned by the density estimator and the estimation feature amount. However, instead of the GLCM feature, the local binary pattern (Local Binary Pattern (LBP) feature value, Haar-like feature value, HOG feature value, luminance pattern, and other various feature values, or a combination of GLCM features and a plurality of them You can also

  (8) In each of the above-described embodiments and modifications thereof, an example is shown in which the density estimation unit 50 and the object position determination unit 51 scan and process at intervals of one pixel. It is also possible to carry out at intervals.

(9) In each of the above-described embodiments and the modifications thereof, an example is shown in which candidate positions are selected and set from an area with a low, medium, or high estimated density. Each of the setting means 511b can also set candidate positions limited to the change region. In that case, the storage unit 4 includes background image storage means (not shown) for storing the background image of the monitoring space, and the image processing unit 5 performs a difference process between the captured image and the background image, and the difference value is a predetermined difference. A change area extraction unit that extracts a group of pixels that are equal to or greater than a threshold value as a change area, or performs a correlation process between a captured image and a background image to extract a group of pixels that have a correlation value equal to or less than a predetermined correlation threshold as a change area (Not shown), each of the arrangement generation means 510a and the candidate position setting means 511b sets a candidate position with reference to the change area extracted by the change area extraction means.
Note that, when the area where the candidate position is set is limited, the arrangement generation unit 510a can change the upper limit of the arrangement number according to the size of the limited area.
By limiting the region where the candidate positions are set, accidental similarity between the captured image and the model image or accidental calculation of a high identification score for the background can be prevented, and erroneous detection of the object position can be reduced.

  (10) In the first embodiment and the modifications thereof, the example has been shown in which the arrangement generation unit 510a randomly generates an arrangement for each iteration. However, from the candidate position one time before the second iteration, The arrangement may be generated by updating to a slightly shifted candidate position, or the MCMC (Markov chain Monte Carlo) method is used to determine the similarity to the previous arrangement after the second iteration. The arrangement may be generated by sequentially improving the candidate positions by a method for searching for candidate positions or a hill-climbing method.

  (11) In each of the above-described embodiments and modifications thereof, a projection area of a model defined in the shape of the upper third of the person or a window defined in the shape is set at the candidate position of interest. The example in which the estimated density at the candidate position is determined by aggregating the estimated density in the area has been shown. However, in order to reduce the processing amount, the pixel at the candidate position and the 8 neighboring areas at the candidate position are replaced with the area. Alternatively, it may be a small area such as the 16 neighborhood area. Alternatively, in order to improve the accuracy, a projection area of a model defined in the shape of the upper 2/3 of a single person with the candidate position as a representative position instead of the area, a window defined in the shape, or a candidate position It can also be a projection area of a model defined in the shape of the whole body of a single person as a representative position or a large area such as a window defined in the shape.

  (12) In the second embodiment and its modification, the partial classifier learned by the linear SVM method is exemplified, but various types of conventionally known learning such as the AdaBoost method instead of the linear SVM method are exemplified. It can also be a partial classifier learned by using the method. In addition, a pattern matching device can be used instead of the discriminator, and the partial score in this case is the inner product of the average pattern of feature values extracted from the human learning image and the feature value of the input image, etc. The function can be a function having the score as an output value and a feature amount of the photographed image as an input value. Further, an identification type CNN may be used as the partial classifier.

  (13) In the second embodiment and the modifications thereof, the HOG feature amount is exemplified as the feature amount used by the partial classifier for learning and identification. However, these are local binary pattern features instead of the HOG feature amount. Various feature amounts such as an amount, a Haar-like feature amount, and a luminance pattern can be used, or a HOG feature amount and a feature amount obtained by combining a plurality of these can be used.

  (14) In the second embodiment and its modifications, the evaluation value calculation unit 514b normalizes the candidate so that the sum of the weights is 1 for each candidate position. ) May be normalized so that the sum of the weights is 1.

  (15) In the second embodiment and the modifications thereof, the example in which the degree of emphasis on the object portion is switched by the weight is shown, but the threshold value (partial threshold value) for the partial evaluation value can also be switched. In that case, for example, the single feature storage means 41 stores a predetermined partial threshold value that is higher as the density is lower and higher as the density is higher, in association with each density. The ratio of the number of partial areas is calculated as an integrated evaluation value. Then, the position determination unit 517b determines a candidate position whose integrated evaluation value is equal to or greater than a predetermined threshold (for example, 50%) as an object position.

  According to each of the above embodiments and the modifications thereof, the object detection device switches the feature of the part suitable for the concealment state that can occur in the object and the degree of emphasis according to the density for each candidate position and partial region. Therefore, it is possible to detect an object with high accuracy adapted to the change in the concealment state of the object accompanying the change in the congestion state.

  In addition, the object detection device according to the first embodiment and the modification thereof has a density that emphasizes a partial model representing an image feature of each part of the object and a partial evaluation value for a captured image of the partial model. By switching according to the above, it is possible to detect the object with high accuracy adapted to the change in the concealment state of the object accompanying the change in the congestion state.

In addition, the object detection device according to the second embodiment and the modified example of the second embodiment emphasizes the partial discriminator that learned the image feature of each part of the object and the sum of the partial evaluation values for each part by the partial discriminator. By switching the degree according to the density, it is possible to detect the object with high accuracy adapted to the change in the concealment state of the object accompanying the change in the congestion state.

DESCRIPTION OF SYMBOLS 1 ... Image monitoring apparatus 2 ... Imaging | photography part 3 ... Communication part 30 ... Image acquisition means 31 ... Object position output means 4 ... Memory | storage part 40 ... Density estimator memory | storage means 41 ... Single feature storage means 410a ... Object model storage means 411b ... Partial identifier storage means 412a, 412b ... Weight storage means 5 ... Image processing section 50 ... Density estimation means 51 ... Object position determination means 510a ... arrangement generation means 511b ... candidate position setting means 512a ... model image generation means 514a, 514b ... evaluation value calculation means 516a ... optimal arrangement determination means 517b ... Position determining means 6 ... display section

Claims (5)

An object detection device for detecting individual objects from a captured image in which a space in which congestion due to a predetermined object may occur is captured,
Estimate the distribution of the density of the object imaged in the captured image using a density estimator that learns the image characteristics of each density image captured in the space where the object exists at the density for each predetermined density Density estimation means to perform,
A candidate position where each of the objects can exist in the captured image is set, and a partial region corresponding to each of a plurality of parts constituting the object is set in the captured image based on the candidate position, For each partial area, a partial evaluation value representing the degree of appearance of the image feature of the part corresponding to the partial area is calculated, and the partial evaluation values of the partial areas set on the basis of the candidate positions are used as the partial area. An object position determination means for determining, as the position of the object, a candidate position that satisfies a predetermined determination criterion by integrating and evaluating the integrated evaluation value as the density of the region is lower;
An object detection apparatus comprising:   The object position determination unit sets a weighting factor that is higher as the density in the partial area is lower for each partial area and lower as the density in the partial area is higher, and the plurality of the parts that are set based on the candidate position The object detection apparatus according to claim 1, wherein the integrated evaluation value is calculated by weighting and summing the partial evaluation values of a region with the weighting coefficient of the partial region.   The object detection apparatus according to claim 1, wherein the object position determination unit sets the partial region corresponding to a smaller part of the parts constituting the object as the density at the candidate position is higher. The object position determination means includes
Arrangement generation means for generating a plurality of different arrangements each including one or more candidate positions;
For each of the plurality of arrangements, model image generation means for generating a model image by drawing a partial model imitating the part in the partial region corresponding to each of the plurality of parts with each candidate position as a reference When,
For each of the model images in the plurality of arrangements, the partial evaluation value representing the degree of similarity of the partial model to the captured image is calculated for each partial region, and the partial evaluation values of the partial regions are integrated. Evaluation value calculating means for calculating the integrated evaluation value;
An optimum arrangement determining means for determining the candidate position as the position of the object in the arrangement having the maximum integrated evaluation value;
The object detection device according to any one of claims 1 to 3. The object position determination means includes
Candidate position setting means for setting a plurality of candidate positions in the captured image;
The partial evaluation value of the partial region is calculated by inputting the image feature of the partial region corresponding to each of the plurality of portions set based on the candidate positions to the classifier that has learned the image feature of the partial region. An evaluation value calculation means for calculating the integrated evaluation value by integrating the partial evaluation values of the plurality of partial areas set with reference to the candidate position for each candidate position;
Position determining means for determining, as the position of the object, the candidate position where the integrated evaluation value that satisfies the determination criterion is calculated;
The object detection device according to any one of claims 1 to 3.

Images