JP2018169831A - Image comparison device - Google Patents

Abstract

Even if there is a calibration error between an actual image and a simulated image, and blurring or discontinuity of an edge in the actual image, different pixels of the actual image are accurately extracted by comparison with the simulated image. Edge extraction means 51 extracts real edge information and simulated edge information including at least the edge direction of each pixel from each of a real image and a simulated image. The different pixel extraction unit 52 obtains a simulated edge pattern formed by simulated edge information of a plurality of pixels in a region near the target pixel of the actual image, and a similar pattern similar to the simulated edge pattern with respect to the actual edge information is equal to or more than a predetermined reference. A pixel of interest that is not detected is set as a different pixel. [Selection] Figure 2

Description

  The present invention relates to an image comparison device that compares a real image obtained by photographing a predetermined space with a simulated image generated by rendering a three-dimensional model of the space and extracts a different pixel from the real image.

  For the purpose of crime prevention or the like, a technique is known in which an image such as a person or a suspicious object that appears in the monitoring space is detected by performing background subtraction processing on a captured image obtained by shooting the monitoring space. However, in crowded spaces where many people come and go, such as event venues, the appearance frequency of scenes in which no people or suspicious objects are captured is low, making it difficult to generate a background image. Therefore, conventionally, a background image is generated by rendering a three-dimensional model imitating a monitoring space.

  In view of the fact that minute errors in texture may be included between a captured image and a background image generated from a three-dimensional model, conventionally, background difference processing has been performed in units of small areas.

  For example, in the object detection devices described in Patent Documents 1 and 2, the captured image is obtained for each block obtained by dividing the captured image into a grid pattern or for each superpixel in which neighboring pixels having similar pixel values are grouped in the captured image. And a background image generated from a three-dimensional model, an average pixel value difference, a pixel value variance difference, and an edge difference are calculated.

JP, 2006-194778, A Japanese Patent Application Laid-Open No. 2006-194779

  By the way, if the angle of view of the camera is increased to widen the monitoring space, the size of an image of a person or the like shown in the image can be reduced, so that background difference processing in units of pixels is necessary. In addition, when performing background difference in units of pixels, it is desirable to extract different pixels due to objects appearing in the monitoring space based on differences (edge differences) based on edge information that is robust to illumination variations.

  However, if an edge difference is simply extracted between corresponding pixels of a captured image (actual image) and a background image (simulated image) generated from a three-dimensional model, it may be due to calibration error, edge blurring or interruption in the actual image, etc. There was a problem that erroneous extraction occurred.

  That is, when there is a calibration error, a corresponding pixel shift occurs between the actual image and the simulated image, and a part of the background object is erroneously detected as a change region.

  Also, in the actual image, the edge of the white line, fine irregularities, block dirt, fine irregularities, etc. cause blurring or discontinuity of the edge, but if the fading or discontinuity cannot be reproduced in the simulated image, the background object Is detected as a change area by mistake.

  The present invention has been made in view of the above problems, and even if there is a calibration error between the real image and the simulated image, and the edge of the actual image is blurred or interrupted, the difference between the actual image and the simulated image is compared. An object of the present invention is to provide an image comparison apparatus that can extract pixels with high accuracy.

  (1) The image comparison apparatus according to the present invention compares an actual image in which a predetermined space is photographed with a simulated image generated by rendering a three-dimensional model of the space, and the actual image An apparatus for extracting different pixels different from the simulated image, the simulated image storing means storing the simulated image, and an actual edge including at least an edge direction of each pixel from the actual image and the simulated image An edge extracting means for extracting information and simulated edge information; and a simulated edge pattern formed by the simulated edge information of a plurality of pixels in a region near the target pixel of the actual image, and the simulated edge pattern relating to the actual edge information And a different pixel extracting means that uses the target pixel for which a similar pattern similar to a predetermined reference or more is not detected as the different pixel. That.

  (2) In the image comparison apparatus according to (1), the different pixel extraction unit uses the simulated edge pattern formed from a plurality of pixels having similar edge directions in the simulated edge information. It can be set as the structure which detects.

  (3) In the image comparison apparatus described in (1) or (2) above, the simulated image storage unit identifies, for each pixel of the simulated image, a structural line of the space rendered on the pixel. An identifier is stored, and the different pixel extraction unit is configured to detect the similar pattern using the simulated edge pattern formed from the simulated edge information of a plurality of pixels having the same structural line identifier. it can.

  (4) In the image comparison apparatus described in (1) to (3) above, in the image comparison apparatus that sequentially compares a plurality of the actual images with the simulated image, the actual image and the Map storage means for storing the positional relationship between pixels that are the same in the simulated image, and the different pixel extraction means identifies the target pixel in which the similar pattern is detected as a pixel in the simulated image. The simulated image forming the positional relationship of the target pixel in the actual image to be newly compared with the positional relationship stored in the map storage unit by storing the positional relationship in the map storage unit The similar pattern can be detected using the simulated edge pattern including the pixels.

  (5) In the image comparison device described in (1) to (4) above, the different pixel extraction unit is an area set around the pixel of interest with a predetermined size narrower than the neighboring area. A pixel having the simulated edge information most similar to the actual edge information of the pixel of interest can be searched, and the similar pattern can be detected using the simulated edge pattern including the pixel.

  (6) In the image comparison device described in (1) to (5) above, the edge extraction unit may further extract edge strength as each piece of edge information.

  According to the present invention, even if there is a calibration error between an actual image and a simulated image, and blurring or interruption of an edge in the actual image, it is possible to accurately extract a different pixel between the actual image and the simulated image.

1 is a block diagram illustrating a schematic configuration of an image monitoring apparatus according to an embodiment of the present invention. 1 is a schematic functional block diagram of an image monitoring apparatus according to an embodiment of the present invention. It is a schematic diagram which shows the example of a real edge image and a simulation edge image. It is a schematic diagram which shows the example of the search area | region of the similar pixel set to the simulation edge image of FIG.3 (b). It is a schematic diagram for demonstrating the extraction process of the simulated edge pattern. It is a schematic diagram which shows the actual edge pattern set corresponding to the simulation edge pattern. It is a schematic diagram for demonstrating the other production | generation method of a reference edge pattern. It is a flowchart of the operation | movement of the outline of the image monitoring apparatus which concerns on embodiment of this invention. It is a general | schematic flowchart of a different pixel extraction process.

  Hereinafter, an image monitoring apparatus 1 according to an embodiment of the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings. The image monitoring apparatus 1 generates a background image generated from a three-dimensional model imitating a monitoring space when a foreground object such as a person or a suspicious object does not exist, by capturing an image (actual image) of a predetermined space (monitoring space). Including an example of an image comparison device according to the present invention that extracts a change pixel in a real image (a different pixel that is different from the simulation image in the real image) by comparing with a (simulation image). An object such as an object is detected and notified.

[Configuration of Image Monitoring Apparatus 1]
FIG. 1 is a block diagram showing a schematic configuration of the image monitoring apparatus 1. The image monitoring apparatus 1 includes a photographing unit 2, a communication unit 3, a storage unit 4, an image processing unit 5, and a notification unit 6.

  The photographing unit 2 is a monitoring camera and is connected to the image processing unit 5 via the communication unit 3. The photographing unit 2 captures the monitoring space at a predetermined time interval to generate a real image, and sequentially converts the real image to the image processing unit 5. It is a photographing means to input. For example, the imaging unit 2 is installed on a pole installed at one corner of an event venue that is a monitoring space, with a predetermined fixed field of view over the monitoring space, and images the monitoring space with a frame period of 1 second to make a color. Generate an image. Note that the photographing unit 2 may generate a monochrome image instead of a color image.

  The communication unit 3 is a communication circuit, one end of which is connected to the image processing unit 5 and the other end is connected to the photographing unit 2 and the notification unit 6. The communication unit 3 acquires a real image from the photographing unit 2 and inputs it to the image processing unit 5, and outputs detection information input from the image processing unit 5 to the notification unit 6. The detection information is information including the type of object detected by the image processing unit 5 and the detected area.

  For example, when the photographing unit 2 and the notification unit 6 are installed in a monitoring center in the event hall and the communication unit 3, the storage unit 4, and the image processing unit 5 are installed in a remote image analysis center, the photographing unit 3 and the photographing unit 6 are photographed. The unit 2, the communication unit 3, and the notification unit 6 may be connected via an Internet line, and the communication unit 3 and the image processing unit 5 may be connected via a bus. In addition, for example, when installing each part in the same building, the communication part 3 and the imaging | photography part 2 are connected by a coaxial cable or LAN (Local Area Network), the communication part 3 and the alerting | reporting part 6 are a display cable, the communication part 3, and The image processing unit 5 is appropriately connected in a form corresponding to the installation location of each unit, such as connecting by a bus.

  The storage unit 4 is a memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), and stores various programs and various data. The storage unit 4 is connected to the image processing unit 5 and inputs / outputs such information to / from the image processing unit 5.

  The image processing unit 5 is configured by an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit). The image processing unit 5 operates as various processing units / control units by reading out and executing a program from the storage unit 4, reads various data from the storage unit 4 as necessary, and stores the generated data in the storage unit 4. Remember. Further, the image processing unit 5 detects an object such as a person or a suspicious object from the actual image acquired from the imaging unit 2 via the communication unit 3, generates detection information regarding the detected object, and causes the communication unit 3 to output the detection information.

  The notification unit 6 is a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display, and displays information such as the type and area of the object included in the detection information input from the communication unit 3 to the monitor. Inform. The notification unit 6 can further include a buzzer, a lamp, and the like in order to emphasize the alerting. The monitor visually recognizes the displayed detection information to determine whether or not a countermeasure is necessary, and takes measures such as urgently handling the countermeasure if necessary.

  In the present embodiment, the image monitoring apparatus 1 having one photographing unit 2 with respect to the set of the communication unit 3 and the image processing unit 5 is illustrated, but in another embodiment, the communication unit 3 and the image are combined. A configuration in which two or more photographing units 2 are connected to the set of processing units 5 can also be adopted. In that case, the communication unit 3 receives real images from each photographing unit 2 in a time division manner, and the image processing unit 5 performs time division processing or parallel processing on the real images from the respective photographing units.

[Function of the image monitoring apparatus 1]
FIG. 2 is a schematic functional block diagram of the image monitoring apparatus 1. FIG. 2 exclusively shows the functions of the communication unit 3, the storage unit 4, and the image processing unit 5. Specifically, the communication unit 3 functions as an actual image acquisition unit 30, a detection information output unit 31, and the like. The storage unit 4 functions as a three-dimensional model storage unit 40, a simulated image storage unit (background image storage unit) 41, a map storage unit 42, a camera information storage unit 43, and the like, and the image processing unit 5 includes a simulated image generation unit ( It functions as background image generation means) 50, edge extraction means 51, different pixel extraction means (changed pixel extraction means) 52, object detection means 53, and the like.

  The real image acquisition unit 30 sequentially acquires real images from the photographing unit 2 that is a photographing unit, and sequentially outputs the acquired real images to the edge extraction unit 51.

  The three-dimensional model storage means 40 stores in advance a three-dimensional model of a plurality of components that constitute a monitoring space that is imaged by the imaging unit 2.

  The three-dimensional model can be acquired from building information according to the IFC (Industry Foundation Classes) standard, three-dimensional CAD data, or the like, which has been created at the time of architectural design, or from prior actual measurement data.

  The components are, for example, building structures such as walls, floors, and doors, and furniture such as desks and chairs, and are permanent objects that should be present in the monitoring space from the viewpoint of monitoring the monitoring space. An identifier for identifying the component is assigned in advance to each component. Moreover, when the component has a complicated shape, the identifier is assigned to each surface constituting the component. The identifier is used for edge identification, which will be described later, and is hereinafter referred to as a structural line identifier. The structural line identifier may be assigned to each line element describing a constituent.

  The three-dimensional model includes the position, orientation, and three-dimensional shape of each component in the XYZ coordinate system that imitates the surveillance space, and the surface color, texture, and reflection characteristics of each component. These data are stored in association with the corresponding structural line identifier.

  The more accurate the 3D model, the better. For example, the door plate and door knob for a door, and the top plate and leg for a desk are defined as components with different colors, textures, and reflection characteristics. It is preferable to store the three-dimensional model. On the other hand, since a large structure such as a wall has a large difference in distance from the light source of each surface, it is preferable to define each surface as a structure.

  Further, the three-dimensional model storage means 40 further stores an illumination model of the monitoring space in advance. The illumination model includes, for one or more light sources that illuminate the monitoring space, illumination characteristics represented by the position of the light source in the XYZ coordinate system simulating the monitoring space, the light distribution of the light source, the color temperature, and the like. The light source is artificial lighting or the sun. Each light source is preliminarily assigned a light source number for identifying the light source, and position and illumination characteristic data are stored in association with the corresponding light source number. The illumination model can simulate a plurality of illumination conditions for the monitoring space by setting the light-off state of each light source and calculating the output of the light source in which the light-on state is set to illumination characteristics.

  That is, the three-dimensional model storage means 40 stores the three-dimensional coordinate values of each of the plurality of constituents constituting the predetermined space that is the subject of photographing by the photographing unit 2, the color, texture, reflection characteristics, and the space of each constituent. An environmental model including the illumination characteristics of the light source to be illuminated is stored.

  The camera information storage unit 43 stores in advance camera parameters of the photographing unit 2 in an XYZ coordinate system simulating a monitoring space. Camera parameters consist of external parameters and internal parameters. The external parameter is the position and orientation of the photographing unit 2 in the XYZ coordinate system. The internal parameters are the focal length, center coordinates, distortion coefficient, etc. of the photographing unit 2. Camera parameters are measured by prior calibration and stored in the camera information storage unit 43. By applying this camera parameter to the pinhole camera model, the coordinates of the XYZ coordinate system can be converted to the xy coordinate system representing the imaging surface of the imaging unit 2.

  The simulated image generation means 50 reads the 3D model and the illumination model from the 3D model storage means 40 and reads the camera parameters from the camera information storage means 43. Then, a background image of the monitoring space is virtually generated as a simulated image by rendering a three-dimensional model corresponding to the illumination condition on the imaging surface of the imaging unit 2 obtained from the camera parameters, and the generated background image is simulated The image is stored in the image storage unit 41. In the present embodiment, the simulated image generation unit 50 generates a simulated image in synchronization with the timing at which the actual image acquisition unit 30 acquires the actual image.

  Specifically, first, the simulated image generation means 50 derives the imaging surface of the imaging unit 2 in the XYZ coordinate system from the camera parameters. Next, the simulated image generating means 50 generates light ray data according to the position and the illumination characteristics stored in the illumination model, and adjusts the presence / absence and output of the light ray data according to each illumination condition. Subsequently, the simulated image generating means 50 adjusts the color of the light beam data and the change in the output due to the reflection on the component according to the color, texture, and reflection characteristics of the component. The simulated image generating means 50 generates a background image by setting the light data of the direct light and reflected light reaching the imaging surface from the light source as the pixel values of the imaging surface.

  Further, the simulated image generating unit 50 associates the pixel value set on the imaging surface of the imaging unit 2 with the component that is the reflection source of the light beam or the structural line identifier of the surface, and stores it in the simulated image storage unit 41.

  The simulated image storage unit 41 stores the background image generated by the simulated image generation unit 50 and the structural line identifier associated with the pixel.

  The edge extraction means 51 extracts edge information including at least the edge direction of each pixel from each of the actual image and the simulated image. That is, the edge extraction unit 51 extracts edge information (real edge information) of each pixel by applying an edge operator to the actual image input from the actual image acquisition unit 30, and uses the extracted actual edge information as the different pixel extraction unit 52. Output to. Further, the edge extraction unit 51 reads a simulated image from the simulated image storage unit 41, applies an edge operator to the read simulated image, generates an edge image (simulated edge information) of each pixel, and extracts the extracted simulated edge information. It outputs to the different pixel extraction means 52.

  In the present embodiment, the edge extraction unit 51 calculates edge information including edge strength and edge direction for each pixel of the real image and the simulated image. Here, the edge direction is a gradient value of edge strength, and is information representing the shape of an actual edge pattern and a simulated edge pattern, which will be described later, in the normal direction. For example, the edge direction can be undirected information having a range of 0 degrees or more and less than 180 degrees. In addition, the edge direction may be directed information having a range of 0 degrees or more and less than 360 degrees. In this case, for example, the outline of the white line painted on the road surface may be used in any direction. It can be information that further includes the state of the neighborhood, such as whether there is black asphalt. The edge direction may be a value obtained by converting a gradient value representing the normal direction into a tangential direction. For example, a Sobel operator, a Laplacian filter, or the like can be used as the edge operator.

  Note that only the edge direction can be used as the edge information in the processing described later, and in this case, the edge extraction unit 51 only needs to output the edge direction as the edge information.

Edge information is obtained for each pixel as described above. That is, the edge information is calculated and defined for all pixels as a result of the edge operator calculation, not limited to the pixels corresponding to the contour line of the object. However, for example, in an image region in which whiteout or blackout occurs due to lighting conditions, the calculated edge information may be noise or information with low reliability. Thus, the edge information edge intensity is less than the threshold value T _E may be be excluded. Threshold T _E can be predetermined based on experimental.

  As described above, the edge extraction unit 51 extracts the actual edge information and the simulated edge information for each pixel from each of the actual image and the simulated image. The edge information extracted by the edge extraction unit 51 includes at least the edge direction, and preferably the edge information extracted by the edge extraction unit 51 includes the edge direction and the edge strength.

  Data obtained by arranging the edge information of each pixel in correspondence with the pixel arrangement in the image is referred to as an edge image. Therefore, data in which the real edge information of each pixel is arranged is referred to as a real edge image, and data in which the simulated edge information of each pixel is arranged is referred to as a simulated edge image.

  The different pixel extracting means 52 extracts, as a different pixel, a pixel in which a similar edge pattern is not detected between the real image and the simulated image in the vicinity of the pixels of the real image. That is, each pixel of a real image is used as a pixel of interest, and a simulated edge pattern formed by simulated edge information of a plurality of pixels is obtained in the vicinity of the pixel. A pixel of interest for which no is detected is set as a different pixel. By comparing the actual image and the simulated image not only with the pixel of the same coordinate but also the vicinity including the pixel as the comparison range, it is possible to accurately extract the different pixel even if there is a calibration error.

  Here, when similarity of edge information is determined in units of pixels, pixels with high similarity in edge information often appear in the neighborhood region, although they are pixels that should originally be different pixels. For this reason, if the difference is determined in units of pixels, the different pixels may not be sufficiently extracted and a foreground object such as a person or a suspicious object may not be detected. Therefore, paying attention to the fact that contours and textures appearing as edge information are similar in the neighboring area if the pixel has the same structural line in the real image and the simulated image, pixels that do not detect similar edge patterns in the neighboring area It is extracted as a different pixel.

  Specifically, each pixel in the actual image is sequentially set as the target pixel to determine whether the target pixel is a different pixel. The simulated edge pattern is a shape pattern formed from simulated edge information of a plurality of continuous pixels in the vicinity region centered on the target pixel. For example, the simulated edge pattern is obtained corresponding to a luminance boundary line in a simulated image such as an object outline. On the other hand, it is an actual edge pattern that is a target for determining whether or not it is a similar pattern. The actual edge pattern is a shape pattern formed from the actual edge information of continuous pixels in the vicinity region. For example, the actual edge pattern is obtained corresponding to the luminance boundary line in the actual image.

  In the present embodiment, the different pixel extraction unit 52 searches for a pixel (similar pixel) of a simulated image having edge information similar to the target pixel in a region near the target pixel, and simulates the pixel corresponding to the position of the similar pixel. Generate an edge pattern. Correspondingly, it can be expected that the actual edge pattern including the target pixel becomes a similar pattern. Therefore, it is assumed here that the similar pattern is an actual edge pattern including the target pixel.

For example, the different pixel extraction unit 52 sets a pixel having edge information similarity S _PX (pixel similarity) equal to or higher than a threshold value T _PX as a similar pixel. In the case where the pixel similarity S _PX in the region near the above pixel threshold T _PX contains multiple, and similar pixels with the highest pixel pixel similarity S _PX. The pixel similarity _SPX is defined by the following equation, for example.

Here, S _PX (•) is the pixel similarity, i represents a pixel of interest, and b represents a pixel in the neighborhood region. Also, m _i, m _b is the edge intensity of the pixel i, b respectively, d _i, d _b are each pixel i, the edge direction of b, lambda the pixels of the difference between the difference between the edge direction of the edge intensities An adjustment coefficient for adjusting the contribution to the similarity, M (•) is a difference in edge strength, and D (•) is a difference in edge direction.

  The neighborhood area is preliminarily set to a size corresponding to the estimated upper limit value by estimating the upper limit of the calibration error that can occur and the upper limit size of the edge pattern. The upper limit size of the edge pattern has the significance of being able to cope with a rotation error included in the calibration error. For example, when the upper limit of the calibration error is estimated to be ± 2 pixels (5 × 5 pixel region) and the upper limit size of the edge pattern is ± 4 pixels (9 × 9 pixel region), the neighboring region is a region of 13 × 13 pixels. Is set.

  Here, a neighborhood area of 13 × 13 pixels may be fixedly set based on the target pixel. However, in the present embodiment, an area of 5 × 5 pixels is set based on the target pixel, and the region within the area is set. The center of the edge pattern is determined, and a 9 × 9 pixel region is set based on the determined center of the edge pattern, so that the processing amount is reduced.

  In other words, the processing load can be reduced by searching for similar pixels in a range corresponding to the estimation of the calibration error and extracting the edge pattern in the range of the maximum size of the edge pattern with reference to the similar pixels.

  In this way, the different pixel extraction means 52 is a simulated edge that is most similar to the actual edge information of the target pixel in a region (search region) set around the target pixel with a predetermined size that is narrower than the neighboring region. A similar pixel having information can be searched and a similar pattern can be detected using a simulated edge pattern including the similar pixel.

  For each pixel of the real image, a wider neighborhood region may be set as the distance from the photographing unit 2 to the subject captured in the pixel is narrower and as the distance from the photographing unit 2 to the subject captured in the pixel is closer. .

  As described above, in the present embodiment, the similar pattern is an actual edge pattern including the target pixel. Further, here, the actual edge pattern that is a candidate for the similar pattern has the same pixel arrangement pattern as the simulated edge pattern, and if the similarity of the actual edge pattern to the simulated edge pattern is equal to or greater than a threshold value, the actual edge pattern Is a similar pattern.

Specifically, the detection of the similar pattern is performed by calculating the similarity (pattern similarity) S _PT between the simulated edge pattern and the actual edge pattern according to the following formula, and comparing the pattern similarity S _PT with the threshold value T _PT . A real edge pattern satisfying S _PT ≧ T _PT is determined as a similar pattern of the simulated edge pattern obtained corresponding to the target pixel.

Here, S _{PT (·)} is the pattern similarity, P ^B mock edge pattern, P ^I is a real edge pattern, T is the number of pixels forming the simulated edge patterns (or actual edge patterns), D (·) is It is a difference in edge direction, and p ^B _i and p ^I _i represent the edge direction of the i-th pixel among the pixels forming the simulated edge pattern and the actual edge pattern, respectively. Note that the simulated edge pattern and the actual edge pattern are the same pixel arrangement pattern as described above, and pixels having the same index i are in the same position in the pixel arrangement pattern.

  When the similar pattern is not detected as a result of the similar pattern detection process, the target pixel is determined to be a different pixel. On the other hand, if a similar pattern is detected, the target pixel is identified as the pixel of the background image, and the target pixel in the actual image and the corresponding pixel of the simulated image (from the positional relationship between the simulated edge pattern and the similar pattern ( The positional relationship with the corresponding pixel) is obtained. In the present embodiment, the similar pattern includes the pixel of interest and has the same pixel arrangement pattern as the simulated edge pattern. In this case, the similar pixel is a corresponding pixel.

When an actual edge pattern or a simulated edge pattern is defined corresponding to a luminance boundary line in an image, such as an object outline, the edge strength may be basically higher than the edge strength in a pixel that does not have a luminance boundary line. I can expect. However, even in such a case, the actual edge information is characterized in that the edge strength is likely to fluctuate due to various factors such as the dirt and minute irregularities already described, and blurring and interruptions are likely to occur. On the other hand, simulated edge information is characterized in that such factors are discarded and blurring and interruptions are unlikely to occur. Because of this difference, defining the pattern similarity _SPT including the edge strength component is an unstable measure. On the other hand, the feature of the shape can be expressed by the edge direction. Therefore, as shown in the above equation, the pattern similarity S _PT corresponding to the difference in edge direction is used.

  Also, because of this feature, by generating a reference edge pattern used for detecting a similar pattern from simulated edge information, it is possible to detect a similar pattern that is less susceptible to blurring and interruptions.

Also for the edge direction, it is highly likely that the luminance boundary line that greatly changes is a combination of the contour lines of different objects, and there is a slight change in the positional relationship of these objects and a slight change in the line-of-sight direction during rendering. The pattern similarity _SPT may be affected by a difference or the like. Therefore, it is preferable that the continuous pixels constituting the simulated edge pattern corresponding to the luminance boundary line have a range in which the edge directions in the simulated edge information are similar.

  That is, continuous pixels having similar edge directions in the simulated edge information are extracted as simulated edge patterns, and it is determined whether or not a similar pattern similar to the simulated edge pattern appears at the position of the target pixel in the actual edge image. It is good.

  As described above, the different pixel extraction unit 52 is arranged along the luminance boundary line in the simulated image (aligned along the luminance boundary line represented by the simulated edge information) and the edge direction in the simulated edge information is similar in the vicinity region. A simulated edge pattern formed from a plurality of pixels is extracted, and a similar pattern is detected using the simulated edge pattern.

  Further, the different pixel extraction unit 52 uses the structural line identifier included in the simulated image and generates a simulated edge pattern from simulated edge information of the pixels to which the same structural line identifier is assigned, thereby generating a plurality of structural lines. It is possible to prevent generation of a shape pattern mixed with, and to reduce the possibility of erroneous extraction as a different pixel. That is, the different pixel extraction unit 52 improves the extraction accuracy of different pixels by detecting a similar pattern using a simulated edge pattern formed from simulated edge information of a plurality of pixels having the same structural line identifier in the neighboring region. Can be made.

  In addition, when a foreground object such as a person or a suspicious object in the real image is set as the pixel of interest, the area where the foreground object appears is set as the neighborhood area, and the edge information of the background is hidden in the area. In many cases, the simulated edge information similar to the actual edge information of the pixel of interest is not detected in the first place, and it is often the case that the pixel unit comparison is sufficient without comparing the edge patterns.

Therefore, difference pixel extracting means 52, prior to the comparison of the edge pattern is compared with a threshold value T _PX calculates the pixel similarity S _PX, pixel similarity S _PX is less than the threshold T _PX pixel (dissimilar pixels ) As a different pixel, the processing load can be reduced. That is, the different pixel extraction unit 52 extracts the target pixel as a different pixel when a pixel similar to the actual edge information of the target pixel is not detected in the region (a neighboring region or a search region) for searching for a similar pixel. .

  In addition, as described above, the edge information of the background is hidden in the region where the foreground object appears in the real image, so that it is expected that the pixel of the real image will not be identified as the pixel of the simulated image. The edge information between a real image pixel and a simulated image pixel at an irrelevant position is coincidentally coincidentally, and a pixel that should not be a similar pixel or a corresponding pixel can be a similar pixel or a corresponding pixel. Sex goes up. Therefore, the different pixel extraction unit 52 stores information on past corresponding pixels in the map storage unit 42 and reuses the information to improve the specific accuracy of the corresponding pixels and improve the extraction accuracy of the different pixels. Plan.

  The map storage means 42 stores the positional relationship between the pixels of the actual image and the corresponding pixels corresponding to the pixel in the simulated image. For example, for each pixel of the real image, the coordinates of the corresponding pixel or the relative coordinates of the corresponding pixel with respect to the coordinates of the pixel of the real image are stored.

  Specifically, the positional relationship between the target pixel when a similar pattern to the simulated edge pattern is detected in the actual edge image and the pixel that forms the simulated edge pattern and has the highest pixel similarity with the target pixel Can be stored in the map storage means 42.

  The positional relationship stored in the map storage means 42 represents the amount of deviation in pixel units between the actual image and the simulated image. In other words, the positional relationship corresponds to a calibration error between the actual image and the simulated image.

  As described above, the different pixel extraction unit 52 can store the positional relationship in the map storage unit 42. On the other hand, the different pixel extraction means 52 reads and uses these positional relationships. That is, the different pixel extraction unit 52 identifies the target pixel in which the similar pattern is detected as the pixel of the simulated image in the process of sequentially comparing the time-series actual images acquired by the photographing unit 2 with the simulated image, The positional relationship between the identical pixels is stored in the map storage means 42, and among the target pixels in the new image to be compared, those that are stored in the map storage means 42 have the positional relationship. A similar pattern is detected using a simulated edge pattern including pixels of the simulated image.

  Hereinafter, an example of processing performed by the image monitoring apparatus 1 will be described in detail with reference to FIGS. FIG. 3 is a schematic diagram showing examples of real edge images and simulated edge images. The real edge image 100 shown in FIG. 3A shows real edges 101 and 102 as examples of real edge patterns. Specifically, for the real edges 101 and 102, effective real edge information is extracted from pixels on the solid line. On the other hand, effective real edge information is not extracted from the pixel indicated by the dotted line, and a part of the real edge 101 is blurred. FIG. 3A shows a neighborhood area of 13 × 13 pixels centered on the target pixel 110 as the real edge image 100.

  The simulated edge image 200 shown in FIG. 3B shows simulated edges 201 and 202 as examples of simulated edge patterns. Specifically, for the simulated edges 201 and 202, effective simulated edge information is extracted from the pixels on the solid line. Note that the simulated edges 201 and 202 are not blurred. FIG. 3B shows a simulated edge image 200 in which the coordinates on the imaging plane are the same as the real edge image 100 in FIG. 3A, that is, a 13 × 13 pixel area that is the same as the vicinity area of the target pixel 110. Has been.

  FIG. 4 is a schematic diagram showing an example of a similar pixel search region set in the simulated edge image 200 of FIG. A 5 × 5 pixel search area 211 is set on the simulated edge image 200 around the target pixel 210, and a similar pixel 212 is detected from the search area 211. In FIG. 4, the simulated edges 201 and 202 in FIG. 3B are indicated by solid lines, and the real edges 101 and 102 in FIG. 3A are indicated by alternate long and short dash lines.

  FIG. 5 is a schematic diagram for explaining the extraction process of the simulated edge pattern. In the simulated edge image 200 of FIG. 5A, a 9 × 9 pixel pattern extraction window 220 centered on a similar pixel 212 is shown. FIG. 5B is a schematic diagram of the structural line identifier stored in the simulated image storage unit 41, and shows the structural line identifier in association with the pixels of the simulated image 300. The simulated image 300 shows the same neighboring region as the actual edge image 100 and the simulated edge image 200, and the structural line identifier “2” is assigned to the pixel corresponding to the simulated edge 201 and the structural line identifier “2” is assigned to the pixel corresponding to the simulated edge 202. 1 "is defined.

  The different pixel extraction unit 52 obtains a simulated edge pattern from the pattern extraction window 220. Here, an example is shown in which the simulated edge pattern is generated from simulated edge information of pixels to which the same structural line identifier is assigned, and the different pixel extraction means 52 has the same structural line identifier “2” as the similar pixel 212. A pattern 310 consisting of pixels in the pattern extraction window 220 is extracted from among the pixels for which is set. A simulated edge pattern 230 is extracted from the simulated edge image 200 in correspondence with the pattern 310 based on the structural line identifier. The data of the simulated edge pattern 230 can be, for example, a vector expression 231 in which the information on the edge direction of each pixel constituting the pattern is arranged along the arrangement of pixels.

  FIG. 6 is a schematic diagram showing an actual edge pattern set corresponding to the simulated edge pattern 230. A region 130 shown in the actual edge image 100 is obtained by translating the pixel arrangement pattern in the simulated edge pattern 230 so that the position of the similar pixel 212 in the pattern 230 matches the target pixel 110. In the example shown here, the actual edge pattern 131 is generated from the area 130. The vector representation 132 of the data of the actual edge pattern 131 is defined in the same manner as the vector representation 231 of the simulated edge pattern.

The different pixel extraction unit 52 calculates the pattern similarity S _PT by applying the expressions (4) and (5) to the simulated edge pattern 230 of the vector expression 231 and the actual edge pattern 131 of the vector expression 132, If S _PT ≧ T _PT , the actual edge pattern 131 is determined to be a similar pattern of the simulated edge pattern 230 obtained corresponding to the target pixel.

  Here, another method for generating a reference edge pattern used for similar pattern extraction will be described with reference to a schematic diagram shown in FIG.

  For example, a configuration in which a simulated edge pattern including pixels having different structural line identifiers is used as the reference edge pattern. FIG. 7A shows a real edge pattern region 140 obtained by applying the configuration to the examples of the real edge image 100, the simulated edge image 200, and the simulated image 300 shown in FIGS. Yes. The pixel arrangement pattern of the region 140 is a pixel arrangement of a simulated edge pattern made up of the pixels of the structural line identifiers “2” and “1” in the 9 × 9 pixel pattern extraction window 220 centering on the similar pixel 212 in FIG. Same as pattern. Since the position of the actual edge 102 with respect to the actual edge 101 (or the target pixel 110) in the actual image 100 of FIG. 5 is shifted from the position of the simulated edge 202 with respect to the simulated edge 201 (or similar pixel 212) in the simulated edge image 200, The real edge 102 is out of the region 140.

As described above, the configuration using the simulated edge pattern as a reference has a feature that the simulated edge pattern is not easily affected by blurring or interruption, and this feature can be obtained even when a simulated edge pattern including a plurality of structural line identifiers is used as a reference. On the other hand, in the simulated edge pattern comprising a plurality of structural lines identifier, may like the change in the positional relationship of the edge between the structure line identifier is different occurs, due to it reduces the pattern similarity S _PT, difference pixel Misextraction and overextraction are likely to occur. In this respect, it is better that the number of the structural line identifiers included in the reference simulated edge pattern is smaller, and the simulated edge pattern including the pixels of the same structural line identifier described above is preferable.

  FIG. 7B is a schematic diagram for explaining the problem of the configuration for generating the reference edge pattern from the actual image 100. As described above, the real edge 101 is blurred. As a result of such blurring, the actual edge pattern region 140 set corresponding to the target pixel 110 becomes smaller as shown in FIG. 7B, and the possibility of coincidental coincidence with the foreground edge increases. . In this respect, as described above, a method for generating a reference based on a simulated edge pattern is preferable.

  The object detection unit 53 determines whether or not the different pixel is due to the monitoring target object, and generates detection information indicating that the monitoring target object is detected when it is determined that the different pixel is due to the monitoring target object. The detected information is output to the detection information output means 31.

  For example, the object detection unit 53 performs a labeling process for labeling different pixels extracted in the vicinity of each other (for example, in the vicinity of 8) as being based on the same object. Then, pattern matching is performed between the different edge image obtained by extracting the real edge information of the different pixels for each label and a human shape model that imitates the shape of a person, and the difference pixels of the label whose matching degree is equal to or greater than a predetermined value are determined by humans. Judge that it is. Alternatively, a machine-learned classifier may be used instead of the pattern matching. In this case, the object detection means 53 sets, for example, a circumscribed rectangle of each label, and sets an identification window that overlaps with the circumscribed rectangle. Then, the actual image in the identification window is input to a human classifier that has previously learned the human image characteristics, and the different pixels in the identification window that have been identified as a person are determined to be due to the person and identified as a person. It is determined that the label without the identification window is due to a suspicious object.

  The detection information output unit 31 outputs the detection information input from the object detection unit 53 to the notification unit 6.

[Operation of the image monitoring apparatus 1]
FIG. 8 is a schematic operation flowchart of the image monitoring apparatus 1.

  The communication unit 3 operates as the real image acquisition unit 30 and sequentially acquires real images from the photographing unit 2 (step S1).

  Each time the real image is acquired from the real image acquisition unit 30, the image processing unit 5 operates as the simulation image generation unit 50 and generates a simulation image (step S2). As a result, it is possible to generate a simulated image according to the sunshine state estimated from the shooting time of the actual image and the lighting state of the artificial illumination grasped from the lighting switch state at the shooting time. Further, when the field of view of the photographing unit 2 changes, a corresponding simulated image can be generated. The simulated image generation unit 50 stores the generated background image in the simulated image storage unit 41.

  Further, every time the real image is acquired from the real image acquisition unit 30, the image processing unit 5 operates as the edge extraction unit 51, generates a real edge image from the real image (step S3), and acquires the real image. The corresponding simulated image is read from the simulated image storage means 41 and a simulated edge image is generated from the simulated image (step S4).

  Next, the image processing unit 5 operates as the different pixel extracting unit 52 and extracts different pixels based on the actual edge image generated in step S3 and the simulated edge image generated in step S4 (step S4). S5).

  FIG. 9 is a schematic flowchart of the different pixel extraction process S5. The different pixel extraction means 52 sequentially sets each pixel of the actual image as a target pixel (step S10). Here, since a calibration error may exist between the actual image and the simulated image, the target pixel of the actual image may correspond to a pixel at a different position in the simulated image. In this regard, there is a possibility that the position of the corresponding pixel of the simulated image corresponding to the pixel of interest is stored in the map storage means 42 in the different pixel extraction process for the actual image acquired in the past. Therefore, the different pixel extraction unit 52 checks whether or not the corresponding pixel information exists in the map storage unit 42 (step S12). If the information is not recorded ("NO" in step S12). ), A similar pixel having edge information similar to the target pixel is searched for in the simulated image (step S14).

  The search for similar pixels is performed in the vicinity of the target pixel. In the present embodiment, a search area having a size corresponding to the upper limit of the calibration error is set around the target pixel, and a similar pixel is searched for in the search area. For example, the search area is a rectangular area of 5 × 5 pixels as described above.

  As a result of the search, if a similar pixel is not found in the simulated image (“NO” in step S16), the different pixel extraction unit 52 determines that the target pixel is not a background (simulated image) pixel as a different pixel. (Step S18). For example, as an example in which it is determined that the pixel is a different pixel in step S18, the target pixel is located on a luminance boundary line (such as an object contour) that appears in the actual image and forms a high-strength edge. There may be no corresponding edge.

  On the other hand, if a similar pixel is detected in the simulated image as a result of the search (“YES” in step S16), the different pixel extracting unit 52 acquires the structural line identifier information from the simulated image storage unit 41 (step S16). S20) A simulated edge pattern is generated from simulated edge information of a pixel to which the same structural line identifier as a similar pixel is assigned (step S22). For example, the different pixel extracting means 52 is a plurality of pixels including similar pixels as the simulated edge pattern, arranged along the luminance boundary line appearing in the simulated image, and adjacent pixels having a common structural line identifier Determine the placement and edge direction for the group. In addition to or in addition to the condition that the structural line identifiers are the same, the variation range in the edge direction in a plurality of adjacent pixels forming the simulated edge pattern is equal to or less than a predetermined threshold value, or between adjacent pixels The simulated edge pattern may be determined by imposing a condition that the difference in the edge direction is equal to or less than a predetermined threshold value. The threshold is preferably less than 90 degrees, for example, 45 degrees.

When the different pixel extraction unit 52 generates the simulated edge pattern corresponding to the similar pixel in step S22, next, the similar pattern which is similar to the simulated edge pattern is included in the actual edge pattern in the actual edge image. It is checked whether it exists (step S24). In the present embodiment, the actual edge pattern is the same pixel arrangement pattern as the simulated edge pattern, which is arranged so that the position of the similar pixel in the simulated edge pattern overlaps the target pixel. The pattern similarity S _PT defined by the above equations (4) and (5) is calculated. If S _PT is equal to or greater than the threshold value T _PT , the actual edge pattern is determined as a similar pattern, and S _PT is the threshold value T _PT. If it is less, it is determined that there is no similar pattern.

  If a similar pattern exists (in the case of “YES” in step S24), the different pixel extraction means 52 determines that the similar pixel obtained in step S14 is a corresponding pixel of the target pixel set in step S10 ( In step S26), the position of the corresponding pixel is recorded in the map storage unit 42 as the corresponding pixel information regarding the target pixel (step S28).

  On the other hand, when it is determined that there is no similar pattern (when “NO” in step S24), the different pixel extraction unit 52 determines that the target pixel is a different pixel (step S18).

  Thus, even if a similar pixel is found in the simulated image with respect to the target pixel of the real image in step S16, it is not determined immediately that the target pixel is not a different pixel by using only the edge information of this one pixel. By further comparing with an edge pattern composed of pixels and determining whether or not they are different pixels, it is possible to detect different pixels with high accuracy.

  Since the corresponding pixel information is recorded as described above (step S28), the corresponding pixel information may exist for the target pixel set in step S10 (in the case of “YES” in step S12). In this case, the different pixel extraction unit 52 acquires the corresponding pixel information from the map storage unit 42 (step S30), omits the processing (steps S14 and S16) related to the search for similar pixels, and uses the edge pattern. Proceeding to the comparison process (steps S20 to S24), it is determined whether the pixel is a different pixel or a corresponding pixel (steps S18 and S26). As a result, the processing load is reduced and the processing time is shortened.

  Incidentally, in this case, a simulated edge pattern is generated corresponding to the corresponding pixel. That is, the different pixel extraction unit 52 acquires the information of the structural line identifier from the simulated image storage unit 41 (step S20), and generates a simulated edge pattern from the simulated edge information of the pixel assigned with the same structural line identifier as the corresponding pixel. (Step S22).

  The above processing S10 to S30 is repeated for each pixel of the real image (in the case of “NO” in step S32), and when the processing is completed for all the pixels in the real image (in the case of “YES” in step S32), the different pixels The extraction means 52 advances the process to step S6 in FIG.

  When a different pixel is extracted in step S5 (in the case of “YES” in step S6), the image processing unit 5 operates as the object detection unit 53, and whether or not the different pixel is due to an object to be monitored. Object recognition processing for determining whether or not is performed (step S7). When it is determined that the object is a monitoring target object, detection information indicating that the monitoring target object is detected is generated, and the generated detection information is output to the detection information output unit 31.

  The communication unit 3 receives the detection information from the object detection unit 53, operates as the detection information output unit 31, and outputs the detection information to the notification unit 6 (step S8).

  The image monitoring apparatus 1 repeats the above-described steps S1 to S8 every time a real image is acquired. If no different pixel is extracted in step S5 (in the case of “NO” in step S6), steps S7 and S8 are omitted, and the process proceeds to the next actual image.

[Modification]
(1) In the above-described embodiment, the simulation image generated from the three-dimensional model imitating the monitoring space when the foreground object does not exist, that is, the simulation image of the background image is compared with the actual image. Is not limited to this. For example, the simulated image may be generated from a three-dimensional model including a model of a monitoring target such as a fire extinguisher. The image comparison apparatus according to the present invention can be used for the purpose of comparing the simulated image and the actual image and detecting the removal or movement of the monitoring target based on the different pixels.

  The simulated image may be generated from a three-dimensional model of an identification object such as a vehicle. The image comparison apparatus according to the present invention can be used for the purpose of comparing the simulated image and the actual image, and identifying or searching for the identification object based on the different pixels.

  The simulated image may be generated from a three-dimensional model of an inspection object such as a factory-produced product. The image comparison apparatus according to the present invention can be used for the purpose of comparing the simulated image and the actual image and inspecting the presence or absence of scratches on the inspection object based on the different pixels.

  (2) In the above-described embodiment, the actual edge pattern and the simulated edge pattern have been described using the shape pattern obtained corresponding to the luminance boundary line in the actual image and the simulated image, such as the outline of the object. The edge pattern is not limited to such a shape pattern. As described above, the edge information is not limited to the pixels corresponding to the contour line of the object, and is basically obtained for all the pixels as the calculation result of the edge operator. Therefore, in general, an edge pattern can be defined as a set of edge information in an area composed of a plurality of pixels.

  Therefore, the present invention can be applied even if the target pixel in the actual image does not constitute a contour line or the like.

  (3) In the above-described embodiment, a shape pattern that passes through similar pixels is generated as a simulated edge pattern. However, the simulated edge pattern may be a shape pattern such as a contour line that passes through the vicinity of similar pixels. For example, when a shape pattern having a larger edge strength than the similar pixel exists near the similar pixel in the simulated edge image, the shape pattern obtained near the similar pixel is set as the simulated edge pattern. In this case, for example, as in the above-described embodiment, in the configuration in which the same pixel arrangement pattern as the simulated edge pattern is assumed as the actual edge pattern, the actual edge pattern is the simulated edge pattern according to the positional relationship between the target pixel and the similar pixel. May be set by translating. That is, the actual edge pattern does not include the target pixel and can be set in the vicinity thereof.

  (4) In the above-described embodiment, the same pixel arrangement pattern as the shape pattern that is the simulated edge pattern is assumed as the actual edge pattern. However, the actual edge pattern may be extracted independently of the simulated edge pattern. That is, similar to extracting a simulated edge pattern by connecting pixels in the direction in which the simulated edge extends from a similar pixel, extract the actual edge pattern by connecting pixels in the direction in which the actual edge extends from the target pixel, The edge information in the pixel of the actual edge pattern and the edge information in the pixel of the simulated edge pattern may be compared to determine whether the actual edge pattern is a similar pattern of the simulated edge pattern.

  (5) In the above-described embodiment, the field of view of the photographing unit 2 is fixed, but may be variable. In other words, the photographing unit 2 may include a scanning mechanism to change the photographing direction, or may include a zoom mechanism to change the angle of view.

  DESCRIPTION OF SYMBOLS 1 Image monitoring apparatus, 2 imaging | photography part, 3 communication part, 4 memory | storage part, 5 image processing part, 6 alerting | reporting part, 30 real image acquisition means, 31 detection information output means, 40 3D model storage means, 41 simulated image storage means , 42 Map storage means, 43 Camera information storage means, 50 Simulated image generation means, 51 Edge extraction means, 52 Different pixel extraction means, 53 Object detection means, 100 Real edge image, 101,102 Real edge, 110 Target pixel, 200 Simulated edge image, 201, 202 Simulated edge.

Claims (6)

An image comparison that compares a real image in which a predetermined space is photographed with a simulated image generated by rendering a three-dimensional model of the space, and extracts a different pixel from the simulated image in the real image A device,
Simulated image storage means for storing the simulated image;
Edge extraction means for extracting actual edge information and simulated edge information including at least an edge direction in each pixel from each of the actual image and the simulated image;
A simulated edge pattern formed by the simulated edge information of a plurality of pixels is obtained in a region near the target pixel of the actual image, and a similar pattern similar to the simulated edge pattern is not detected with respect to the actual edge information. A different pixel extracting means for setting a target pixel as the different pixel;
An image comparison apparatus comprising:   The said different pixel extraction means detects the said similar pattern using the said simulated edge pattern formed from the several pixel with which the said edge direction in the said simulated edge information was similar, The said similar pattern is characterized by the above-mentioned. Image comparison device. The simulated image storage means stores, for each pixel of the simulated image, a structural line identifier that identifies a structural line of the space rendered on the pixel,
The different pixel extracting means detects the similar pattern using the simulated edge pattern formed from the simulated edge information of a plurality of pixels having the same structural line identifier;
The image comparison apparatus according to claim 1, wherein: In an image comparison device that sequentially compares a plurality of actual images with the simulated image,
Map storage means for storing a positional relationship between pixels that are identical in the actual image and the simulated image in the past comparison;
The different pixel extracting means includes
Identifying the target pixel in which the similar pattern is detected as a pixel of the simulated image and storing the positional relationship in the map storage means;
Among the target pixels in the actual image to be newly compared, for the pixel in which the positional relationship is stored in the map storage unit, the simulated edge pattern including the pixels of the simulated image that form the positional relationship is used. Detecting similar patterns,
The image comparison apparatus according to claim 1, wherein:   The different pixel extracting means is a pixel having the simulated edge information most similar to the actual edge information of the target pixel in an area set around the target pixel with a predetermined size narrower than the neighboring area. The image comparison apparatus according to claim 1, wherein the similarity pattern is detected using the simulated edge pattern including the pixel.   6. The image comparison apparatus according to claim 1, wherein the edge extraction unit further extracts an edge strength as each piece of edge information.

Images