WO2016114427A1 - Method and device for reducing computational complexity when depth information is estimated from image - Google Patents

Abstract

Provided is a method for reducing computational complexity when depth information is estimated from an image. The method according to one embodiment of the present invention may comprise the steps of: acquiring a first image by using a first aperture, and acquiring a second image by using a second aperture; sequentially computing the first image and a plurality of blur kernels so as to acquire a plurality of blurred images; comparing similarities between the pixels of each of the plurality of blurred images and pixels of the second image; and estimating depth information from the image on the basis of the similarities.

Description

Method and apparatus for reducing computational complexity when estimating depth information from images

The present invention relates to a method and apparatus for reducing the computational complexity when estimating depth information from an image, and more particularly, by sequentially performing the operation of the blur kernels in a blur-based depth estimation. A method and apparatus for reducing the complexity of the

Research on estimating depth information of objects from images has been done for decades. Conventional approaches for estimating depth information of objects in an image include depth estimation using stereo vision and time-of-flight (ToF) cameras. Recently, a depth estimation method using a single lens dual aperture (DA) camera has been used as a depth information estimation technique using blur of an object in an image. An image of an object that is farther away from the object closer to the camera than the focal plane may be blurred. Single Lens Dual Aperture (DA) cameras may include very small diameter IR (IR) apertures and regular size diameter color (RGB) apertures. Using a single lens dual aperture (DA) camera, depth information can be estimated by comparing the blurring of IR and RGB images.

There is a need to reduce computational complexity when estimating depth information from an image.

As an embodiment of the present invention, a method and apparatus for reducing computational complexity when estimating depth information from an image by sequentially applying a plurality of blur kernels to an image may be provided.

According to the method and apparatus provided as an embodiment of the present invention, the computational complexity can be greatly reduced by allowing the image to be sequentially calculated with a plurality of blur kernels. In addition, according to an embodiment of the present invention, it is possible to use a relatively small memory required for the calculation, the operation time is shortened, thereby reducing the power consumption required for the calculation.

1 is a schematic diagram of a conventional depth information estimation method.

2 is a schematic diagram of a method for estimating depth information according to an embodiment of the present invention.

3 is a flowchart illustrating a method for reducing computational complexity when estimating depth information from an image according to an embodiment of the present invention.

4 illustrates a method of estimating depth information through sequential operations according to an embodiment of the present invention.

5 illustrates a depth information estimation method through sequential operation according to another embodiment of the present invention.

6 illustrates a depth information estimation method through sequential operation according to another embodiment of the present invention.

7 illustrates a depth information estimation method through sequential operation according to another embodiment of the present invention.

8 is a block diagram illustrating an apparatus for reducing computational complexity when estimating depth information from an image according to an embodiment of the present invention.

In one embodiment of the invention, a method is provided for reducing computational complexity when estimating depth information from an image. According to an embodiment of the present invention, a method of acquiring a first image of an object using a first aperture and a second image of the object using a second aperture are performed. Sequentially calculating kernels, comparing similarities between pixels of the plurality of blur images obtained as the sequentially calculated result and pixels of the second image, and estimating depth information from the image based on the similarity. It may include.

Computing the first image and the plurality of blur kernels sequentially according to an embodiment of the present invention may include setting N block kernels having different sizes and first and second image among the N block kernels. Performing a cascade operation of obtaining a first operation result by operating a second operation result and obtaining a second operation result by operating the first operation result with a second block kernel, wherein N is a natural number and a cascade The id operation may be performed on all N block kernels.

The size of the Nth blocker kernel according to an embodiment of the present invention may be smaller than that of the existing Nth blocker kernel.

N block kernels according to an embodiment of the present invention may be a matrix having at least one of MxM, 1xM, and Mx1, and M may be a natural number.

As an embodiment of the present invention, an apparatus for reducing the computational complexity when estimating depth information from an image may be provided. An image acquisition unit for acquiring a first image of an object using a first aperture according to an embodiment of the present invention, and obtaining a second image of an object using a second aperture, a first image and a plurality of blurs A calculation unit for sequentially calculating kernels, a comparison unit comparing similarities between pixels of the plurality of blur images obtained as a result of the sequentially calculation, and pixels of the second image, and an estimation of estimating depth information from the image based on the similarity It may include wealth.

The calculation unit according to an embodiment of the present invention sets N block kernels of different sizes, calculates a first image and a first image among the N block kernels, and obtains a first calculation result. A cascade operation is performed in which the second operation result is obtained by operating the operation result with the second block kernel, N is a natural number, and the cascade operation may be performed for all N block kernels.

The size of the Nth blocker kernel according to an embodiment of the present invention may be smaller than that of the existing Nth blocker kernel.

N block kernels according to an embodiment of the present invention may be a matrix having at least one of MxM, 1xM, and Mx1, and M may be a natural number.

Meanwhile, according to an embodiment of the present invention, a computer-readable recording medium having recorded thereon a program for executing the above-described method on a computer can be provided.

Terms used herein will be briefly described and the present invention will be described in detail.

The terms used in the present invention have been selected as widely used general terms as possible in consideration of the functions in the present invention, but this may vary according to the intention or precedent of the person skilled in the art, the emergence of new technologies and the like. In addition, in certain cases, there is also a term arbitrarily selected by the applicant, in which case the meaning will be described in detail in the description of the invention. Therefore, the terms used in the present invention should be defined based on the meanings of the terms and the contents throughout the present invention, rather than the names of the simple terms.

When any part of the specification is to "include" any component, this means that it may further include other components, except to exclude other components unless otherwise stated. In addition, the terms "... unit", "module", etc. described in the specification mean a unit for processing at least one function or operation, which may be implemented in hardware or software or a combination of hardware and software. .

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a conventional depth information estimation method. According to the prior art, it is possible to obtain blur images through calculation with blur kernels having different variance values (eg, blur kernel 1 to blur kernel NO) for the infrared image 1. The size of the blur kernel may be determined according to the amount of blur. If the dispersion value is large, the blur amount may be large, and if the dispersion value is small, the blur amount may be small. Blur may refer to effects such as defocusing an image or spreading out the color of each point in the image. In addition, according to the related art, the depth information of a pixel on an image may be obtained from the image based on the similarity determination through comparison between the patch 4 and the blur patches surrounding the pixel on the color image 2. Can be.

Conventionally, since the infrared image 1 and each of the plurality of block kernels have been calculated, the amount of calculation and the size of the memory required for the calculation process are enormous. In other words, the conventional technique is poor in calculating the depth information from the image, so the use of the same method as the embodiment of the present invention for increasing the computational efficiency is urgent.

2 is a schematic diagram of a method for estimating depth information according to an embodiment of the present invention.

According to an embodiment of the present invention, the blur images (eg, BP_1) through sequential calculation with the blur kernels (eg, 10 to N0) having respective different variance values with respect to the infrared image 1. To BP_N) may be obtained. In other words, the first blur image BP_1 may be obtained through calculation of the first patch 3 and the first blur kernel 10, which are obtainable by cropping a predetermined area of the first image 1, The second blur image BP_2 may be obtained through the calculation result of the first image 1, the first blur kernel 10, and the calculation of the second blur kernel 20. In addition, the third blur image BP_3 may be obtained through the calculation result of the second blur kernel 20 and the calculation of the third blur kernel 30. In other words, each of the blur images (eg, BP_1 to BP_N) may be obtained by computing the first image 1 cumulatively with the plurality of blur kernels (eg, 10 to N0). The cumulative operation according to an embodiment of the present invention may be referred to as a cascade operation. The operations of the image and the blur kernel can include convolution operations, addition or multiplication operations, and the like. According to an embodiment of the present invention, depth information of a pixel included in an image is included in the image by comparing the similarity between the pixels of the obtained blur images BP_1 to BP_N and the pixels of the second image 2. Depth information of the included pixel may be estimated.

A Gaussian blocker kernel according to an embodiment of the present invention may be represented by the operation of two different Gaussian blocker kernels as shown in Equation 1 below.

Equation 1

σ _L represents the standard deviation of the larger blocker among the two different Gaussian blockers, and σ _S represents the standard deviation of the smaller blocker. G also represents a Gaussian blocker function.

The blurred image through the operation with each of the blur kernels may be represented by Equation 2 below. Part of the blurred image may be referred to as a blur patch. I may refer to the original image, I1 may be a blurred image obtained by calculation with a small size blur kernel, and I2 may refer to a blurred image obtained by calculation with a large size blur kernel.

Equation 2

According to an embodiment of the present invention, it may be expressed as Equation 3 following the blurred image.

Equation 3

일 때, 각각의 블러커널들과 연산되어 블러링된 이미지(예컨대, I1 내지 IN)는 다음의 수학식 4와 같이 나타낼 수 있다. 또한,

일 수 있다. According to an embodiment of the present invention, the standard deviation set of the black kernels

In this case, an image (eg, I1 to IN) calculated and blurred with each of the blur kernels may be represented by Equation 4 below. Also,

Can be.

Equation 4

In addition, Equation 4 may be represented by Equation 5 as follows.

Equation 5

의 표준편차를 갖는 블러커널의 크기는

의 표준편차를 갖는 블러커널의 크기보다 작을 수 있다. 따라서, 본 발명의 일 실시예에 따르면 연산에 사용될 메모리의 크기 및 곱셈 및 누적 연산 횟수가 획기적으로 감소될 수 있으며, 이에 따라 연산 효율이 대폭 증가될 수 있다.

The size of a block kernel with a standard deviation of

It may be smaller than the size of the block kernel with a standard deviation of. Therefore, according to an embodiment of the present invention, the size of the memory to be used for the calculation and the number of multiplication and accumulation operations can be significantly reduced, and thus the computational efficiency can be greatly increased.

3 is a flowchart illustrating a method for reducing computational complexity when estimating depth information from an image according to an embodiment of the present invention.

A method for reducing computational complexity when estimating depth information from an image according to an embodiment of the present invention may include obtaining a first image using a first aperture and using a second aperture to obtain a second image. Acquiring an image (S1000), sequentially calculating the first image and the plurality of blur kernels to acquire a plurality of blur images (S2000), using peripheral pixels or patches of a pixel for which depth information is to be estimated. The method may include comparing the similarity between the pixels of the plurality of blur images with the pixels of the second image (S3000) and estimating depth information of the image based on the similarity (S4000).

According to an embodiment of the present invention, the step of sequentially calculating the first image and the plurality of block kernels (S2000) may include setting N block kernels having different sizes and a first block kernel among the N block kernels. And calculating a first image to obtain a first operation result, and performing a cascade operation to obtain a second operation result by calculating the first operation result with the second block kernel. In this case, N is a natural number, and the cascade operation may be performed on all N blocks.

The first stop according to an embodiment of the present invention may be an infrared (IR) filter, the first image may be an infrared image. In addition, the second aperture may be an aperture for adjusting the amount of light that may be used in a camera system, and the second image may be a color image in which an object is photographed. The first patch of the predetermined area on the first image may be an image area having an N × N pixel area (N is a natural number). Patches may also be referred to as windows.

According to an embodiment of the present invention, the first image and the plurality of block kernels may be sequentially calculated. In other words, when N block kernels of different sizes are set, the first block kernel 10 and the first image 1 of the N block kernels (N is a natural number) are calculated to obtain a first calculation result. Thereafter, the second calculation result may be obtained by calculating the first calculation result with the second block kernel 20. This cumulative operation on the image may be referred to as a cascade operation. This cascade operation may also be performed on all N block kernels.

The size of the N-th blocker kernel according to an embodiment of the present invention may be a relatively small block kernel. In other words, the size of the N-th blocker kernel may be smaller than that of the conventional N-th blocker kernel.

According to an embodiment of the present invention, depth information of an image is compared by comparing similarities between the aforementioned pixels and each of the pixels of the second image using blur patches surrounding the plurality of pixels obtained through the cascade operation. Can be estimated. Using a cascaded operation according to an embodiment of the present invention can greatly reduce the complexity of the operation and the image can be more blurred than before.

4 illustrates a method of estimating depth information through sequential operations according to an embodiment of the present invention.

For example, suppose there are 46 block kernels with sizes between 3x3 and 25x25, for a first image 1 of size WxH (W and H are natural numbers) (6σ ₁ -1) as shown in FIG. A first block kernel 10 of size) x (6σ ₁ -1) may be applied. In other words, the first image 1 and the first blurring kernel 10 may be calculated. An operation in which the pixel values of the pixels included in the first image 1 and the matrix value of the first block kernel 10 are multiplied and accumulated may be performed. As an example of such an operation, a convolution operation may be included. Then, the calculation result of the first image 1 and the first blurr kernel 10 may be provided as an input for calculation with the second blurr kernel. The size of the second blocker kernel may be smaller than the size of the first blocker kernel. This sequential operation may be applied from the first block kernel to the Nth block kernel, as shown in FIG. 4.

5 illustrates a depth information estimation method through sequential operation according to another embodiment of the present invention.

N block kernels according to an embodiment of the present invention may be a matrix of at least one of MxM, 1xM, and Mx1. Here, M may be a natural number. Gaussian block kernels according to an embodiment of the present invention may be applied to an image in the form of a matrix of 1xM and Mx1. In other words, separate Gaussian black kernels can be computed with the image. For example, a Gaussian blocker kernel with a standard deviation of sigma _i can be computed with an infrared image as a block kernel 11 in matrix form of 1xM and a block kernel 12 in matrix form of Mx1. As shown in FIG. 5, the 1xM matrix-like block kernel 11 is calculated with the first image 1, and the calculation result may be sequentially calculated with the matrix-type block kernel 12.

6 illustrates a depth information estimation method through sequential operation according to another embodiment of the present invention.

By applying a running-average filter (RAF) to the image multiple times, the gaussian blurring effect can be approximated. For example, applying a moving-average filter four times on an image can achieve an effect that is sufficiently approximated with a Gaussian blurring effect. According to an embodiment of the present invention, the first RAF may be applied to the image to perform an operation with the bluish kernel 13 having an mxm size, and the calculation result may be provided as an input for applying the second RAF. For example, the RAF operation may include a sum of a matrix value of a block kernel and a pixel value included in a patch of an image, a multiplication operation of 1 / m ² , and the like.

7 illustrates a depth information estimation method through sequential operation according to another embodiment of the present invention.

As described above with reference to FIG. 6, a Gaussian blurring effect and an approximation effect may be achieved by applying a moving-average filter (RAF) to the image at least four times. According to an exemplary embodiment of the present invention, a 1xm matrix blocker kernel 14 and an mx1 matrix blocker kernel 15 may be applied to an image as the first RAF application. In other words, the application of a moving-average filter (RAF) to an infrared image or the like according to the present invention performs an operation with the 1xm matrix-type blur kernel 14 and blurs the result of performing such an operation with the matrix-type blur of mx1. It may be provided as an input of the kernel 15 so that a sequential operation may be performed on the plurality of block kernels. For example, an operation with a 1xm matrix blocker 14 or an mx1 matrix blocker 15 is performed by adding the matrix values of each blocker and the pixel values included in the image and And multiplication operations.

In addition, the method for reducing the computational complexity when estimating depth information from an image according to an embodiment of the present invention may be equally applied to non-Gaussian black kernels as well as Gaussian black kernels. have. As a blur-based depth estimation method using Point Spread Functions (PSF), the method as shown in FIG. 2 may be used according to an embodiment of the present invention, thereby greatly reducing the amount of computation.

According to the conventional technique as shown in FIG. 1, the blurred image may be represented by Equation 6 below. B1 to BN represent the block kernels, I represents the original image, and I1 to IN represent the blurred image obtained by applying respective block kernels.

Equation 6

According to an embodiment of the present invention, the blurred image may be represented by Equation 7 below.

Equation 7

블러커널의 크기는 제

의 크기보다 작을 수 있다. Where ⓧ represents a deconvolution operator. Also, my

The size of the black kernel

It may be less than the size of.

8 is a block diagram illustrating an apparatus for reducing computational complexity when estimating depth information from an image according to an embodiment of the present invention.

The apparatus 1000 for reducing a computational complexity when estimating depth information from an image according to an embodiment of the present invention acquires a first image using a first aperture and uses a second image to use a second aperture. An image acquisition unit 1100 for acquiring a symbol, an operation unit 1200 for sequentially calculating a first image and a plurality of blur kernels to obtain a plurality of blur images, pixels of the plurality of blur images, and pixels of the second image It may include a comparison unit 1300 for comparing the similarity between and the estimator 1400 for estimating depth information of the image based on the similarity.

The calculating unit 1200 sets N block kernels of different sizes, calculates a first block kernel and a first patch among the set N block kernels, obtains a first calculation result, and then sets the first calculation result as a second. A cascade operation is performed in which the second operation result is obtained by operating with the block kernel, N is a natural number, and the cascade operation may be performed for all N block kernels.

The size of the Nth blocker kernel according to an embodiment of the present invention may be smaller than that of the existing Nth blocker kernel.

N block kernels according to an embodiment of the present invention may be a matrix having at least one of MxM, 1xM, and Mx1, and M may be a natural number.

With regard to the apparatus according to an embodiment of the present invention, the above-described method may be applied. Therefore, with respect to the apparatus, the description of the same contents as those of the above-described method is omitted.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

Claims (9)

In the method for reducing the computational complexity when estimating depth information from an image, Acquiring a first image using a first aperture and acquiring a second image using the second aperture; Sequentially calculating the first image and the plurality of black kernels; Comparing the similarity between the pixels of the second image and each of the pixels of the plurality of blur images obtained as a result of the sequentially calculating; And Estimating depth information of the image based on the similarity. The method of claim 1, Computing the first image and the plurality of blur kernels sequentially, Setting N block kernels of different sizes; And A cascade that obtains a first operation result by calculating a first blur kernel and the first image among the N block kernels, and obtains a second operation result by calculating the first operation result with a second block kernel. Performing an id operation; Wherein N is a natural number and the cascade operation is performed on all of the N black kernels. The method of claim 2, And wherein the size of the Nth black kernel is a relatively small Nth black kernel. The method of claim 2, Wherein the N black kernels are matrices in the form of at least one of MxM, 1xM, and Mx1, and M is a natural number. An apparatus for reducing computational complexity when estimating depth information from an image, An image acquisition unit for acquiring a first image using a first aperture and acquiring a second image using a second aperture; An operation unit that sequentially computes the first image and the plurality of blur kernels to obtain a plurality of blur images; A comparison unit comparing the similarity between each of the pixels of the plurality of blur images with pixels of the second image; And And an estimator for estimating depth information of the image based on the similarity. The method of claim 5, wherein The calculation unit, N block kernels of different sizes are set, the first block kernel among the set N block kernels and the first image are calculated to obtain a first calculation result, and the first calculation result is compared with the second block kernel. Perform a cascade operation that causes the second operation result to be obtained by the operation, Wherein N is a natural number and the cascade operation is performed on all of the N black kernels. The method of claim 6, Wherein the size of the Nth black kernel is a relatively small Nth black kernel. The method of claim 6, Wherein the N black kernels are matrices in the form of at least one of MxM, 1xM, and Mx1, and M is a natural number. A computer-readable recording medium having recorded thereon a program for implementing the method of any one of claims 1 to 4.

Images