JP2018004581A - Image processing device, image processing method, program and system - Google Patents

Abstract

An image processing apparatus and the like capable of accurately calculating the depth of pixels included in a captured image with a reduced amount of calculation. An image processing apparatus (A) includes a first imaging unit (2) that captures a visible light image, a second imaging unit (3) that captures an infrared image including a region overlapping the visible light image, and an infrared light irradiation unit (8). Prepare. A first depth determination unit 12 determines the depth of pixels included in the overlap region. A partial region generation unit 13 generates a plurality of partial regions from overlapping regions in the infrared image. A clustering unit 15 classifies a plurality of partial regions into a plurality of clusters. The function determining unit 16 determines a function representing the relationship between pixel brightness and depth for each of the plurality of clusters, based on the pixel depth determined by the first depth determining unit. The second depth determination unit 19 uses the function determined by the function determination unit to determine the depth of pixels included in the infrared image newly captured by the second imaging unit. [Selection drawing] Fig. 2

Description

  The present invention relates to an image processing apparatus, an image processing method, a program, and a system that determine the depth of a captured image.

  2. Description of the Related Art Conventionally, a technique for generating a parallax image by photographing a subject to be photographed using a stereo camera and displaying a stereoscopic image that gives a feeling of depth is known. Patent Document 1 discloses a technique for generating a depth map of a three-dimensional object by irradiating an object with light of different frequencies from three directions.

JP 2009-81853 A

  By the way, when calculating the depth of a captured image with high accuracy based on a captured parallax image, a large calculation capability is required. Therefore, in order to accurately calculate the depth of the captured moving image based on the parallax image, there is a problem that a high-performance processor is necessary and a great deal of cost is required.

  Therefore, the present invention has been made in view of these points, and an object of the present invention is to accurately calculate the depth of pixels included in a captured image while suppressing the amount of calculation.

  In the first aspect of the present invention, a first imaging unit that captures an image, a second imaging unit that captures an infrared image including an overlapping area that overlaps an area captured by the first imaging unit, and at least the An infrared light irradiating unit that irradiates an area including an overlapping region with infrared light, a first depth determining unit that determines a depth of a pixel included in the overlapping region, and a plurality of portions from the overlapping region in the infrared image A pixel for each of the plurality of clusters based on the depth of the pixels determined by the partial region generation unit that generates a region, a clustering unit that classifies the plurality of partial regions into a plurality of clusters, and the first depth determination unit A function determining unit that determines a function representing the relationship between the brightness and the depth of the image, and a second depth determining unit that determines the depth of a pixel included in the infrared image newly captured by the second imaging unit using the function; An image processing device comprising To provide.

  The image processing apparatus may further include a feature amount calculation unit that calculates feature amounts of the plurality of partial regions, and the clustering unit may further classify the plurality of partial regions into the plurality of clusters based on the feature amount.

  The function determining unit updates the function, and the second depth determining unit, when the function determining unit updates the function, an infrared image captured by the second imaging unit using the updated new function. You may determine the depth of the pixel which contains.

  The partial region generation unit may generate the plurality of partial regions so that one partial region and another partial region include the same pixel.

  In the second aspect of the present invention, the step of capturing an image, the step of capturing an infrared image including an overlapping region that overlaps the region to be captured, and irradiating at least the region including the overlapping region with infrared light Determining a depth of pixels included in the overlapping region; generating a plurality of partial regions from the overlapping region in the infrared image; and the pixels included in each of the plurality of partial regions Classifying the plurality of partial regions into a plurality of clusters based on at least the depth and the luminance of the plurality of clusters, determining a function representing a relationship between the luminance and the depth of a pixel for each of the plurality of clusters, Determining a depth of a pixel included in the infrared image using a function, and providing an image processing method.

  In the third aspect of the present invention, the step of capturing an image, the step of capturing an infrared image including an overlapping region that overlaps the region to be captured, and irradiating at least the region including the overlapping region with infrared light Determining a depth of pixels included in the overlapping region; generating a plurality of partial regions from the overlapping region in the infrared image; and the pixels included in each of the plurality of partial regions Classifying the plurality of partial regions into a plurality of clusters based on at least the depth and the luminance of the plurality of clusters, determining a function representing a relationship between the luminance and the depth of a pixel for each of the plurality of clusters, Determining a depth of a pixel included in the infrared image using a function, and providing a program executed by the computer.

  In a fourth aspect of the present invention, there is provided a system in which a plurality of the image processing devices are installed, and the shooting target is three-dimensionally constructed based on the shooting target depth data measured by each of the image processing devices.

  According to the present invention, there is an effect that the depth of pixels included in a captured image can be accurately calculated with a reduced calculation amount.

It is a front view of an imaging device provided with the image processing device concerning an embodiment. It is a figure which shows the structure of the image processing apparatus which concerns on embodiment. It is a figure for demonstrating the overlapping area | region of a visible light image and an infrared image. It is a figure for demonstrating classification | category of the cluster in feature-value space. It is the figure by which the point which shows the brightness | luminance and the depth of the pixel contained in a partial area was plotted in multiple numbers. It is a flowchart of a process of an image processing apparatus. It is a figure for demonstrating the system which installed multiple imaging devices.

[Overview]
Conventionally, there is a technique for capturing an image with a stereo camera and determining the depth of each pixel of the captured image based on a parallax image. When determining the depth of each pixel of the captured image based on the parallax image, pattern matching is performed on the objects that are duplicated in the two images in order to obtain the parallax. Therefore, in order to accurately calculate the depth of the captured moving image in real time, high calculation capability is required.

  The image processing apparatus according to the embodiment photographs an object to be photographed with a visible light camera and an infrared camera. The image processing apparatus determines the depth of each pixel of the infrared image based on the parallax image obtained by photographing. The image processing apparatus divides an infrared image in which the depth of each pixel is determined into a plurality of partial areas, and clusters the plurality of divided areas. The image processing apparatus determines a function representing the relationship between the luminance and depth of the pixels included in the infrared image for each of the plurality of clusters, so that the captured image captured thereafter is a pixel from only the infrared image. Determine the depth of the. For example, the image processing apparatus generates a depth map by using a visible light image for only the first frame out of 10 frames of an infrared image captured in one second, and has determined functions for the remaining nine frames. Generate a depth map using.

  As described above, the image processing apparatus according to the embodiment can significantly reduce the work of pattern matching between a visible light image and an infrared image, and therefore, for example, a moving image without requiring high calculation capability. An image depth map can be generated. Hereinafter, an image processing apparatus according to an embodiment will be described with reference to the drawings.

[Configuration of imaging device]
FIG. 1 is a front view of an imaging apparatus 1 including an image processing apparatus according to an embodiment. On the front surface of the imaging apparatus 1, a first photographing unit 2, a second photographing unit 3, a bandpass filter 4, a microphone 5, a speaker 6, a slit 7, and an infrared light irradiation unit 8 are arranged. The first imaging unit 2 is, for example, a visible light camera. The second photographing unit 3 is, for example, an infrared camera. For example, the infrared light irradiation unit 8 is provided in parallel in the slit 7, and is disposed immediately next to the second imaging unit 3 so that infrared light is irradiated onto the imaging target in parallel as much as possible. The band-pass filter 4 is provided so as to overlap with the lens of the second imaging unit 3 and, for example, allows only infrared light having a wavelength near 940 nm to pass therethrough. The second imaging unit 3 emits the infrared light irradiation unit 8 to irradiate the imaging target with infrared light when capturing an infrared image of the imaging target. Further, the number of pixels of the infrared image captured by the second imaging unit 3 is, for example, about 5 million pixels.

  Each of the modules arranged in front of the imaging apparatus 1 is controlled by a control unit included in the image processing apparatus. The imaging device 1 can record using the microphone 5. Further, the imaging device 1 can output sound using the speaker 6. In addition, the imaging apparatus 1 can capture an image of an imaging target by synchronizing the first imaging unit 2 and the second imaging unit 3. For example, the imaging device 1 has a plug shape on the opposite side of the illustrated front, and can be used by being inserted into a socket of a light bulb arranged on the ceiling. The imaging device 1 is realized using a computer such as Raspberry Pi (registered trademark). The configuration of the image processing apparatus will be described below with reference to FIG.

[Configuration of image processing apparatus]
FIG. 2 is a diagram illustrating a configuration of the image processing apparatus A according to the embodiment. The image processing apparatus A includes a storage unit 9, a control unit 10, a first imaging unit 2, a second imaging unit 3, and an infrared light irradiation unit 8. FIG. 2 shows a functional configuration for realizing the image processing apparatus A according to the embodiment, and other configurations are omitted. Each part of FIG. 2 can be configured by a CPU (Central Processing Unit), a main memory, and other LSI (Large Scale Integration) of the imaging apparatus 1 in hardware. The software is realized by a program loaded in the main memory of the imaging apparatus 1. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  The storage unit 9 includes, for example, a ROM (Read Only Memory) in which a program executed by the control unit 10 is recorded and a RAM (Random Access Memory) that temporarily stores data. The visible light image photographed by the first photographing unit 2 and the infrared image photographed by the second photographing unit 3 are recorded in the RAM. In addition, a depth map determined based on the visible light image and the infrared image is recorded in the RAM. The depth map refers to data in which the correspondence between each pixel and the depth of the overlapping region between the visible light image and the infrared image is recorded. In addition, the RAM stores a feature amount related to each partial area when the infrared image is divided into partial areas. In the RAM, clusters for classifying the partial areas are recorded. A function of brightness and depth determined for each cluster is recorded in the RAM.

  The control unit 10 executes the program recorded in the storage unit 9 to thereby perform an overlapping region determination unit 11, a first depth determination unit 12, a partial region generation unit 13, a feature amount calculation unit 14, a clustering unit 15, and a function determination. It also functions as the unit 16, the partial region determination unit 17, the cluster determination unit 18, and the second depth determination unit 19. Further, the control unit 10 controls the first photographing unit 2 and the second photographing unit 3 to photograph a visible light image and an infrared image to be photographed. For example, the control unit 10 shoots the first photographing unit 2 and the second photographing unit 3 in synchronization, and records a parallax image to be photographed in the storage unit 9. The overlapping area determination unit 11 determines an overlapping area between the visible light image and the infrared image. Hereinafter, the overlapping region R between the visible light image and the infrared image will be described with reference to FIG.

FIG. 3 is a diagram for explaining an overlapping region R between the visible light image and the infrared image. The first imaging unit 2 and the second imaging unit 3 are installed, for example, at a distance of about 2 cm from each other. In FIG. 3, an overlapping region R between the visible light image and the infrared image is within a frame surrounded by a thick line.
In FIG. 3, a region obtained by adding the rectangle 31 to the overlapping region R is a visible light image photographed by the first photographing unit 2, and a region obtained by adding the rectangle 32 to the overlapping region R is photographed by the second photographing unit 3. Infrared image. After determining the overlapping region R, the overlapping region determining unit 11 notifies the first depth determining unit 12 and the partial region generating unit 13. Hereinafter, a method in which the first depth determination unit 12 determines the depth of the pixels included in the overlapping region R will be described with reference to FIG.

  The first depth determination unit 12 determines the depth of the pixels included in the overlapping region R. In FIG. 3, the subject is shown near the center of the overlapping region R. In the visible light image, the subject is located to the right of the center, and in the infrared image, the subject is located to the left of the center. The first depth determination unit 12 determines the depth of the pixels included in the overlapping region R based on the distance between the first imaging unit 2 and the second imaging unit 3 and the displacement of the subject position in the visible light image and the infrared image. Can be determined. The 1st depth determination part 12 may determine the depth of the pixel contained in the duplication area | region R using a known technique. When the first depth determination unit 12 determines the depth of the overlapping region R, for example, the overlapping region R may be reduced to a resolution of 320 pixels × 240 pixels. Thereby, the first depth determination unit 12 can reduce the amount of calculation of the depth of the overlapping region R.

  The partial region generation unit 13 generates a plurality of partial regions from the overlapping region R in the infrared image. The partial area is a unit of processing when the image processing apparatus A determines the pixel depth from only the infrared image, and is, for example, a rectangular area having 9 × 9 pixels included in the overlapping area R. The partial region generation unit 13 may generate a plurality of partial regions so that one partial region and another partial region include the same pixel. That is, two different partial regions may overlap in the overlapping region R. This increases the number of partial areas that can be generated from the overlapping area R, and the image processing apparatus A can create a smoother depth map. For example, about 60000 to 70000 are generated as the partial areas, and 10000 is randomly extracted from the generated partial areas.

  The feature amount calculation unit 14 calculates feature amounts of the plurality of partial areas. The feature amount of the partial area is calculated by a known method based on, for example, luminance information and edge information.

  The clustering unit 15 classifies the plurality of partial areas into a plurality of clusters. Specifically, the clustering unit 15 classifies the plurality of partial areas into a plurality of clusters based on the feature amount calculated by the feature amount calculation unit 14, for example. Hereinafter, a method in which the clustering unit 15 classifies a plurality of partial areas into a plurality of clusters will be described with reference to FIG.

  FIG. 4 is a diagram for explaining cluster classification in the feature amount space. FIG. 4 shows a state in which a plurality of partial regions R1, R2, and R3 are classified into a plurality of clusters C1, C2, and C3 in the feature amount space, respectively. The feature amount space has an I axis that represents the luminance dimension, a p axis that represents the position of the partial region in the overlapping region R, and an f axis that represents the other feature amount dimension. The feature amount space is a multidimensional space in which luminance and center coordinates are also one of feature amounts. Moreover, the position of the partial area in the overlapping area R is, for example, the coordinates of the pixel located at the center of the partial area. For example, the clustering unit 15 classifies the plurality of partial regions into a plurality of N clusters (for example, N = 32) by the k-means method. The clustering unit 15 records the classified cluster information in the storage unit 9. For example, the clustering unit 15 records the centroid coordinates of the clusters in the feature amount space in the storage unit 9.

  Based on the pixel depth determined by the first depth determination unit 12, the function determination unit 16 determines a function representing the relationship between the luminance and depth of the pixel for each of the plurality of clusters. The function determined by the function determination unit 16 will be described below with reference to FIG.

  FIG. 5 is a diagram in which a plurality of points indicating the luminance and depth of pixels included in a partial region included in a certain cluster are plotted. As shown in FIG. 5, the depth of the pixel is approximately inversely proportional to the square root of the luminance. The function determination unit 16 determines a function by, for example, regression analysis based on statistical information on the luminance and depth of each pixel included in the partial region belonging to the cluster. The function determination unit 16 records the determined function of each cluster in the storage unit 9.

  Specifically, a function indicating the relationship between the pixel luminance I and the pixel depth d can be expressed as d = Ac / √I using an unknown constant Ac, for example. Here, Ac (c = 1,..., N) is a constant determined for each cluster. The function determination unit 16 obtains Ac for each of the plurality of clusters by using a known iterative solution related to the least square method. Thereby, the function determination part 16 can determine the function which shows the relationship between the brightness | luminance I of a pixel and the depth d of a pixel for every cluster. The form of the function indicating the relationship between the pixel luminance I and the pixel depth d is not limited to the above. In addition to this, the depth d may be expressed by a polynomial of luminance I, or the above function may be expressed by a combination of linear or nonlinear functions related to other luminance I.

  The partial area determination unit 17 determines a partial area corresponding to each pixel included in the infrared image captured by the second imaging unit 3. The partial region determination unit 17 determines, for example, a rectangular region having 9 × 9 pixels centered on the pixel as a partial region for each pixel of the infrared image. Note that a pixel in which a rectangular area having 9 × 9 pixels cannot be determined in an infrared image is not a pixel that determines a partial area.

  The cluster determination unit 18 determines to which cluster the partial region generated from the infrared image captured by the second imaging unit 3 belongs belongs. Specifically, the cluster determination unit 18 acquires the center-of-gravity coordinates of each cluster in the feature amount space recorded in the storage unit 9, calculates which cluster the partial region is closest to, and the partial region belongs to Determine the cluster.

  The second depth determination unit 19 determines the depth of the pixels included in the infrared image newly captured by the second imaging unit 3 using the function recorded in the storage unit 9. Specifically, the second depth determination unit 19 acquires a function corresponding to the cluster to which the partial region belongs, and determines the depth of the pixel from the luminance information of the pixel included in the partial region. Thereby, the image processing apparatus A can determine the depth of the pixels included in the captured image with a small amount of calculation.

[Processing flow of image processing apparatus A]
As described above, the image processing apparatus A classifies the plurality of partial regions generated from the infrared image into a plurality of clusters, and after determining the function, the pixels included in the infrared image captured thereafter Can be determined without pattern matching with a visible light image. For example, the image processing apparatus A captures an infrared image of 10 frames per second, and determines the depth of pixels included in the infrared image using the same function for one second. For example, the image processing apparatus A updates a cluster and a function every second using a newly captured visible light image and infrared image. The processing flow of the image processing apparatus A will be described below with reference to FIG.

  FIG. 6 is a flowchart of processing of the image processing apparatus A. First, the process starts from where the first photographing unit 2 and the second photographing unit 3 photograph a pair of parallax images (S601). Subsequently, the overlapping area determination unit 11 determines the overlapping area R of the parallax images (S602). Subsequently, the first depth determination unit 12 determines the depth of the pixels included in the overlapping region R (S603). Subsequently, the partial region generation unit 13 generates a partial region of the infrared image (S604). Subsequently, the feature amount calculation unit 14 calculates the feature amount of the partial region (S605). Subsequently, the clustering unit 15 classifies the partial regions into clusters based on the feature amount including at least the luminance and the depth (S606). Subsequently, the function determination unit 16 determines a function of brightness and depth for each cluster (S607).

  Subsequently, the second imaging unit 3 captures an infrared image again (S608). The partial region determination unit 17 determines a partial region for each pixel included in the infrared image, and the cluster determination unit 18 determines a cluster to which the partial region belongs. After determining the function corresponding to the cluster to which the pixel belongs, the cluster determining unit 18 determines the depth of the pixel from the luminance information of the pixel of the infrared image (S609). When the cluster and the function are learned again (YES in S610), that is, when the cluster and the function are determined from the visible light image and the infrared image, the process returns to S601.

  When the cluster and the function are not learned again (NO in S610), the image processing apparatus A determines whether to end the process (S611). If the image processing apparatus A does not terminate the process (NO in S611), the process returns to S608. When the image processing apparatus A ends the process (YES in S611), the image processing apparatus A ends the process.

[Effects of image processing apparatus A]
As described above, the image processing apparatus A according to the embodiment determines the depth of each pixel included in the infrared image based on the parallax image between the visible light image and the infrared image. The image processing apparatus A generates a plurality of partial regions of the infrared image and determines a plurality of clusters for classifying each partial region. The image processing apparatus A determines a function that associates brightness and depth with each cluster. The image processing apparatus A can determine the pixel depth from the luminance information of the pixel by associating the partial region with each pixel of the infrared image and determining the cluster to which the partial region belongs. By doing so, the image processing apparatus A can determine the depth of the pixels included in the infrared image using only the infrared image without using a pair of parallax images. Accordingly, since the image processing apparatus A can determine the depth of the pixels included in the captured image with a small amount of calculation, a depth map with high accuracy can be quickly generated at a low cost.

[System provided with a plurality of imaging devices 1]
As described above, the image processing apparatus A determines the depth of the captured image based on the luminance information of the infrared light reflected from the imaging target by irradiating the infrared light. The image processing apparatus A determines a function of brightness and depth on the assumption that infrared light projected by another imaging apparatus 1 overlaps the target. Therefore, unlike the method of projecting some pattern to the target and calculating the depth from the deformation of the pattern, such as the structured-light method, even if a plurality of image processing apparatuses A are used simultaneously, they interfere with each other and do not function. There is no.

  Due to the above advantageous effects, it is possible to design a system in which a plurality of image processing apparatuses A are installed and a shooting target is three-dimensionally constructed based on the depth data of the shooting targets measured by each of the plurality of image processing apparatuses A.

  FIG. 7 is a diagram for explaining a system in which a plurality of imaging apparatuses 1 are installed. The system includes a plurality of imaging devices 1 each including an image processing device A, and images the target H from three different directions. In the above system, for example, the dog that is the subject H is photographed from directly above, from the front, and from the back.

  When the system shown in FIG. 7 is used, the depth data can be calculated more accurately with respect to the region where the imaging ranges overlap in the plurality of imaging devices 1. Further, the three-dimensional data of the target H can be obtained by adding the depth data from the front direction of the target H and the depth data from the back direction.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

A ... Image processing device H ... Target R ... Overlapping region 1 ... Imaging device 2 ... First imaging unit 3 ... Second imaging unit 4 ... Band pass filter 5 ... -Microphone 6 ... Speaker 7 ... Slit 8 ... Infrared light irradiation unit 9 ... Storage unit 10 ... Control unit 11 ... Overlapping region determination unit 12 ... First depth determination unit 13 ... Partial region generation unit 14 ... Feature quantity calculation unit 15 ... Clustering unit 16 ... Function determination unit 17 ... Partial region determination unit 18 ... Cluster determination unit 19 ... Second Depth determination unit

Claims (7)

A first photographing unit for photographing an image;
A second imaging unit that captures an infrared image including an overlapping area that overlaps an area captured by the first imaging unit;
An infrared light irradiation unit that irradiates infrared light to a region including at least the overlapping region;
A first depth determination unit that determines the depth of pixels included in the overlapping region;
A partial region generation unit that generates a plurality of partial regions from the overlapping region in the infrared image;
A clustering unit for classifying the plurality of partial regions into a plurality of clusters;
A function determining unit that determines a function representing a relationship between a luminance and a depth of a pixel for each of the plurality of clusters based on the depth of the pixel determined by the first depth determining unit;
A second depth determination unit that determines a depth of a pixel included in an infrared image newly captured by the second imaging unit using the function;
An image processing apparatus comprising: A feature amount calculating unit for calculating the feature amounts of the plurality of partial areas;
The clustering unit further classifies the plurality of partial regions into the plurality of clusters based on the feature amount;
The image processing apparatus according to claim 1. The function determination unit updates the function,
The second depth determination unit determines a depth of a pixel included in an infrared image captured by the second imaging unit using the updated new function when the function determination unit updates the function.
The image processing apparatus according to claim 1. The partial region generation unit generates the plurality of partial regions so that one partial region and another partial region include the same pixel.
The image processing apparatus according to claim 1. Taking a picture, and
Photographing an infrared image including an overlapping region overlapping the region to be photographed;
Irradiating the region including at least the overlapping region with infrared light;
Determining the depth of pixels included in the overlap region;
Generating a plurality of partial regions from the overlapping region in the infrared image;
Classifying the plurality of partial regions into a plurality of clusters based at least on the depth and brightness of the pixels included in each of the plurality of partial regions;
Determining a function representing a relationship between pixel brightness and depth for each of the plurality of clusters;
Determining the depth of pixels that the infrared image comprises using the function;
An image processing method comprising: Taking a picture, and
Photographing an infrared image including an overlapping region overlapping the region to be photographed;
Irradiating the region including at least the overlapping region with infrared light;
Determining the depth of pixels included in the overlap region;
Generating a plurality of partial regions from the overlapping region in the infrared image;
Classifying the plurality of partial regions into a plurality of clusters based at least on the depth and brightness of the pixels included in each of the plurality of partial regions;
Determining a function representing a relationship between pixel brightness and depth for each of the plurality of clusters;
Determining the depth of pixels that the infrared image comprises using the function;
A program executed by a computer including A system in which a plurality of image processing apparatuses according to any one of claims 1 to 4 are installed, and the imaging target is three-dimensionally constructed based on depth data of the imaging target measured by each of the image processing apparatuses.

Images