WO2019012660A1 - Image processing device and light field imaging device - Google Patents

Abstract

The image processing apparatus (2) according to the present invention receives a plurality of frames of light field images acquired over time, in order to detect events without missing events from light field images acquired over a plurality of frames, An event evaluation value which is a representative value of numerical values indicating deviation of a pixel value of a pixel included in a light field image from at least a partial region of a light field image of at least one frame among a plurality of frames. Is calculated and output.

Description

Image processing apparatus and light field imaging apparatus

The present invention relates to an image processing apparatus and a light field imaging apparatus.

Conventionally, a subject is provided with an imaging device in which a plurality of pixels are two-dimensionally arranged, and a microlens array having a microlens corresponding to each of a plurality of pixels of the imaging device on the subject side with respect to the imaging device There is known a light field imaging apparatus for imaging a three-dimensional distribution of (see, for example, Patent Document 1).

In general, an image acquired by a light field imaging apparatus (hereinafter referred to as a light field image) itself is different from an image acquired by a normal imaging apparatus, in which images of a large number of three-dimensionally distributed points are overlapped Therefore, basic information such as the planar position and distance of the subject can not be intuitively understood unless image processing is performed. In addition, since the light field image has an enormous amount of information, it is difficult to process the image in real time to reconstruct a three-dimensional image composed of a plurality of slice images.

In addition, by performing moving image shooting and time-lapse shooting with the light field imaging apparatus, it is performed to obtain light field images of a plurality of frames arranged in time series. In this case, photographing is performed for a long time in anticipation of occurrence of change (event) in the observation target at any timing.

JP, 2010-102230, A

However, as described above, it is not easy to detect the occurrence of an event by visually observing a light field image in which the images of a large number of three-dimensionally distributed points are overlapped unlike a normal image. Even if there is a discernible change, visual observation of a large number of images requires labor and time, and there is a disadvantage that an event may be overlooked.

The present invention has been made in view of the above-described circumstances, and it is an object of the present invention to provide an image processing apparatus and a light field imaging apparatus capable of detecting without missing events from light field images acquired over a plurality of frames. And

In one aspect of the present invention, light field images of a plurality of frames acquired over time are input, and the light field of at least a part of the light field image of at least one of the plurality of frames is received. The image processing apparatus is an image processing apparatus that calculates and outputs an event evaluation value that is a representative value of a numerical value indicating deviation of a pixel value of a pixel included in an image from a predetermined reference value.
According to this aspect, when light field images of a plurality of frames acquired over time are input, predetermined reference values of pixels included in the light field image for at least a partial region of at least one light field image The event evaluation value which is a representative value of the numerical value indicating the deviation from is calculated and output.

That is, the occurrence of a significant change (event) in any position or frame in the input light field image of a plurality of frames from the predetermined reference value of the pixel value It can be easily confirmed by an event evaluation value which is a representative value of at least a part of the numerical value indicating the deviation. Thereby, as compared with the case where the occurrence of an event is confirmed by visual observation for all the light field images, the event can be detected without missing the event.

In the above aspect, the reference value may be an average value of pixel values of corresponding pixels of the light field image.
Further, in the above aspect, the reference value may be an average value of pixel values of a plurality of pixels of the light field image of a plurality of the frames.
Further, in the above aspect, the reference value may be an average value of pixel values of all pixels of all the light field images.

Further, in the above aspect, the reference value may be a pixel value of a corresponding pixel in the light field image of the frame different in the time axis direction.
In the above aspect, the reference value may be a pixel value of a corresponding pixel in the light field image of the frame adjacent in the time axis direction.

In another aspect of the present invention, an imaging optical system for condensing light from an object and forming an image of the object, a position of a primary image formed by the imaging optical system, or the primary image A microlens array having a plurality of microlenses arranged two-dimensionally at positions conjugate to an image and focusing the light from the imaging optical system, and a plurality of pixels for receiving the light focused by the microlenses And an image sensor for generating the light field image by photoelectrically converting light received in the pixel, and any of the above image processing devices for processing the light field image generated by the image sensor. It is a light field imaging device.

In the above aspect, the pixels are arranged for each of the areas corresponding to the microlenses, and the image processing apparatus is arranged for each of the pixels corresponding to each of the microlenses according to the incident angle of light on each of the microlenses. The event evaluation value may be calculated by weighting.
Further, in the above aspect, the image processing device divides each light field image into a plurality of regions corresponding to incident angles of light to the microlenses, and the event evaluation is performed for each of the divided regions. The value may be calculated.

According to the present invention, it is possible to detect a light field image acquired over a plurality of frames without missing events.

It is a schematic diagram which shows the light field imaging device which concerns on one Embodiment of this invention. It is a figure which shows an example of the change for every flame | frame of the event evaluation value output by the image processing apparatus with which the light field imaging device of FIG. 1 was equipped. It is a figure explaining the case where an event evaluation value is calculated for every field which divided the light field picture by the image processing device of FIG. It is an enlarged view which shows the relationship between the boundary of the area | region in the range A of FIG. 3, and a micro lens. It is a figure which shows an example of the time change of the event evaluation value output by the image processing apparatus of FIG. It is a figure which shows an example of the light field image obtained by image | photographing a uniform white plane with the light field imaging device of FIG. It is a figure which shows the case where a to-be-photographed object is arrange | positioned in the focus position of the imaging optical system of the light field imaging device of FIG. FIG. 6 is a diagram illustrating a case where the subject is disposed behind the focus position of the imaging optical system. FIG. 6 is a diagram illustrating a case where the subject is disposed on the front side of the focus position of the imaging optical system. It is a figure which shows an example of the light field image acquired in the case of FIG. 7A. It is a figure which shows an example of the light field image acquired in the case of FIG. 7B. It is a figure which shows an example of the light field image acquired in the case of FIG. 7C. It is a figure explaining the case where area | region division of light field image information is carried out to the optical axis direction, and an event evaluation value is calculated about the area | region arrange | positioned at the focus position of an imaging optical system. It is a figure explaining the case where area | region division of light field image information is carried out to the optical axis direction, and an event evaluation value is calculated about the area | region arrange | positioned from the focus position of an imaging optical system.

An image processing apparatus 2 and a light field imaging apparatus 1 according to an embodiment of the present invention will be described below with reference to the drawings.
As shown in FIG. 1, the light field imaging apparatus 1 according to the present embodiment includes an imaging optical system 3 that condenses light from a subject S (object point) to form an image of the subject S, and imaging optical system A micro lens array 5 having a plurality of micro lenses 5a for collecting light from the system 3; and an imaging element 9 comprising a plurality of pixels 9a for receiving and photoelectrically converting the light collected by the plurality of micro lenses 5a An image processing apparatus 2 according to the present embodiment for processing a light field image acquired by the imaging device 9 and a display unit 4 are provided.

As shown in FIG. 1, the microlens array 5 is configured such that the plurality of microlenses 5 a having positive power are positioned at the focal point of the imaging optical system 3 (the position of the primary image or the position conjugate with the primary image). It is arranged in a two-dimensional array along a plane orthogonal to the axis L. The plurality of microlenses 5 a are arranged at a sufficiently large pitch (for example, a pitch eight times the pixel pitch of the imaging device 9) compared to the pixel pitch of the imaging device 9.

The imaging element 9 is also configured by two-dimensionally arranging the pixels 9 a in a direction orthogonal to the optical axis L of the imaging optical system 3. A plurality of (for example, 8 × 8 in the above example) are arranged in each region corresponding to the plurality of microlenses 5 a of the microlens array 5. The plurality of pixels 9 a photoelectrically convert the detected light and output a light intensity signal (pixel value) as light field image information of the subject S.

The imaging device 9 is configured to sequentially output light field image information of a plurality of frames acquired at different times in the time axis direction. For example, movie shooting or time-lapse shooting.

The image processing apparatus 2 is configured by a processor, and the event evaluation value A is calculated according to Equation 1 for the first k-th light field image of a sequence of light field images of a plurality of frames acquired by the imaging device 9 at predetermined time intervals It is designed to calculate _k .
The event evaluation value _Ak is a representative value for each light field image of a numerical value indicating a deviation from the average value (predetermined reference value) of the entire sequence of pixel values of the pixels included in the light field image.

 
 
　ここで、
　ｎはライトフィールド画像における画素位置、
　ｎｔｏｔａｌは各ライトフィールド画像の画素数、
　ｋｔｏｔａｌはフレームの総数、
　Ｉ_ｋｎはｋ番目のライトフィールド画像におけるｎ番目の画素の画素値、
　Ｉａｖｇ_ｎはシーケンス全体の平均画像のｎ番目の画素値（シーケンス全体の各ライトフィールド画像のｎ番目の画素の画素値の平均値）である。

here,
n is the pixel position in the light field image,
ntotal is the number of pixels of each light field image,
ktotal is the total number of frames,
I _kn is the pixel value of the n-th pixel in the k-th light field image,
Iavg _n is the n-th pixel value of the average image of the entire sequence (average value of the pixel values of n-th pixels in each light field image of the entire sequence).

The display unit 4 is a monitor that displays an event evaluation value _Ak calculated by being processed by the image processing device 2.

The operation of the image processing apparatus 2 and the light field imaging apparatus 1 according to the present embodiment configured as described above will be described.
According to the light field imaging apparatus 1 according to the present embodiment, light from the subject S is collected by the imaging optical system 3 and arranged at the focal point position by performing imaging at predetermined time intervals. The light is collected by the respective microlenses 5a of the microlens array 5 and received by the pixels of the image pickup device 9, whereby light field image information is acquired.

Light field image information of the acquired plurality of frames, by being sent to the image processing apparatus 2 is calculated event evaluation value A _k is a graph showing changes in each frame of the event evaluation value A _k is generated, the display unit 4 Is displayed on.

An example of the graph which shows the change for every flame | frame of event evaluation value _Ak is shown in FIG.
According to FIG. 2, at a timing when large frame and event evaluation values A _k has taken rapidly changing frames of event evaluation values A _k, the event that changes the light intensity obtained by the imaging device 9 has occurred It is understood that there is a possibility.

Therefore, by deriving the event evaluation values A _k from the sequence of light field images of a plurality of frames, can be estimated event information and event occurrence time and duration based on the event evaluation values A _k, than with visual observation Also, there is an advantage that the occurrence of an event can be confirmed easily and quickly.

In this embodiment, the event evaluation value _Ak is calculated using the pixel value of the average image of the whole sequence, but the sequence is divided into a plurality of sections along the time axis, and the pixel value of the average image in each section May be used. For example, in the case of time-lapse photography to repeat the imaging for 10 seconds every hour, either one of frames of event evaluation value light field image of the photographing section when calculating the A _k of photographing section It is sufficient to use the pixel value of the average image calculated only from.

This has the advantage that it is possible to check whether or not an event has occurred before the end of the sequence, even for long sequences.

When it is desired to detect the occurrence of an event each time a light field image of each frame is acquired, the pixel value (moving average) of the average image of the light field images of the plurality of frames immediately before may be used.
This also makes it possible to confirm the occurrence of an event without ending the entire sequence. In addition, even when the entire pixel value changes gradually regardless of the event, it is possible to accurately detect the occurrence of the event without being affected by the influence.

 
 
　ここで、
　Ｉａｖｇは、シーケンス全体の画素値の平均値である。 Further, in the above embodiment, the average value is obtained for each pixel position, but instead, as shown in Equation 2, an average value of pixel values of the entire sequence may be used.

here,
Iavg is an average value of pixel values in the entire sequence.

According to this method, since the event evaluation value _Ak does not change even if the subject S moves, it is effective in the case where only a change in light intensity is desired.

Also, using the pixel value of the corresponding pixel of the light field image adjacent (immediately) in the time axis direction instead of the average value as the predetermined reference value, as shown in equation 3, the absolute value of the difference is the entire light field image The event evaluation value B _k may be used as the sum of the above.

 
　このようにすることで、ほぼリアルタイムにイベントの発生の有無をモニタすることができる。

By doing this, it is possible to monitor the occurrence of an event substantially in real time.

In the above embodiment, one event evaluation value _Ak is calculated for each light field image of each frame, but instead, as shown in FIG. 3, the light field image of each frame is May be divided into a plurality of (for example, 3 × 3 = 9) areas, and the event evaluation value _Ak may be calculated for each area.

In this case, as shown in FIG. 4, it is preferable to make the boundary of the region coincide with the boundary of the pixel range corresponding to the microlens 5a. Thereby, as shown in FIG. 5, an event evaluation value equal to the number of regions can be obtained.

By doing this, there is an advantage that the sensitivity of the change of the event evaluation value _Ak can be improved. That is, when an event occurs in a very small area of the light field image, calculating the event evaluation value _Ak for the entire light field image results in a small value and it is difficult to distinguish it from noise. The event evaluation value _Ak can be made a large value by

In addition, when the event occurrence position changes with time, calculating the event evaluation value _Ak for the entire light field image only continues the constant event evaluation value _Ak indicating the occurrence of the event, but for each area in by calculating an event evaluation value a _k, there is an advantage that it is possible to detect the occurrence location and time of the event in the light field images simultaneously.

Further, in the case of calculating the one event evaluation values A _k in the entire light field image also it may be possible to calculate an event evaluation value A _k by weighting for each region is divided into a plurality of areas. This can reduce the influence of unimportant regions and make it easier to estimate event information more clearly.

For example, when it is known in advance that a specific part of the subject S should be focused, the weight of an unimportant area can be reduced to reduce the influence, or the weight can be made zero and the calculation time It can be shortened.

Further, in the light field image, as shown in FIG. 6, the pixel value tends to be greatly reduced in the pixels corresponding to the vicinity of the boundary of the microlens 5a. These pixels correspond to the end of the pupil of the imaging optical system 3, and the pixels corresponding to the vicinity of the center of the microlens 5 a correspond to the center of the pupil of the imaging optical system 3.

Therefore, when calculating the event evaluation values A _k, may be to perform a weighting for each pixel position corresponding to the micro lenses 5a. For example, the weight of the pixels arranged in the vicinity of the boundary of the micro lens 5a can be reduced to make it less susceptible to noise. When calculating the event evaluation value _Ak , when summing the absolute value of the difference from the reference value of the pixel value of each pixel, the weight is low for the pixels near the boundary of the microlens 5a, and near the center of the microlens 5a The weights may be increased for the pixels corresponding to.
As a method of weighting in this case, it is possible to acquire an image as shown in FIG. 6 in which a uniform plane (for example, a plane of white only) or a space is photographed in advance, and adopt a weight corresponding to the intensity distribution. . If the amount of light decreases or the image is distorted at a high image height, the image height may also be taken into consideration for weighting. Further, the area may be divided for each pixel position corresponding to the microlens 5a, and the evaluation value may be calculated for each area.

Further, the light-field image has the three-dimensional information, but also contains event information generated within the object S, clearly the event evaluation values A _k calculated from the entire light field image Sometimes it is difficult for symptoms to appear. Therefore, the light field image may be divided into areas in the direction of the optical axis L, and the event evaluation value _Ak may be calculated for each area.

In the light field imaging device 1 according to the present embodiment, only the light beam of a specific angle (the incident angle of light to the microlens 5a covering one pixel 9a is a specific range) in each pixel 9a on the imaging device 9 Since the light is not incident, the angle of light incident on the microlens 5a is determined when the one pixel 9a is determined.

In this case, the image at the position away from the in-focus position does not simply blur on the imaging device 9 as taken by a normal camera, but in the light field image as shown in FIGS. 7A to 7C. The image of one point of the subject S is light as shown in FIG. 8A to FIG. 8C according to the position z1, z2, z3 (the incident angle of light to the micro lens 5a) in the optical axis L direction of that point. It is distributed to a plurality of two-dimensional coordinates in a plane orthogonal to the axis L direction.

This is because the positions of the subject S in the optical axis L direction z1, z2, z3 are different, so that the incident angle of the light incident on the microlens 5a is changed, and the light is incident on different pixels. The light field imaging device 1 can calculate the three-dimensional coordinates of the original position based on the intersections of a plurality of light rays corresponding to a plurality of two-dimensional coordinates. It is supposed to shoot including information. A method for simply refocusing using this is known, and the event evaluation value _Ak may be calculated by dividing the area in the direction of the optical axis L using this method. As the above method, for example, a method may be adopted in which the influence of a point to be emphasized by refocusing is increased, and the influence of a point that is spread over a wide range and weakened by refocusing is reduced. Further, the refocused image does not necessarily have to be created, and the pixels to be used may be changed according to the L direction positions z1, z2 and z3.

That is, as shown in FIGS. 9A and 9B, the pixels used when calculating the event evaluation value are changed according to the position in the optical axis L direction. Assuming that the number of pixels of the image pickup device 9 corresponding to the microlens 5 a is m × n and the number of pixels of the light field image is p × q, the event evaluation value Az _k is as shown in Expression 4.

 
　ここで、
　ｚは光軸Ｌ方向位置、
　ｃは撮像光学系３の構成により定まる係数、
　Ｉｚ_ｋ（ｘ，ｙ）は光軸Ｌ方向位置ｚ、ｋ番目のフレームの座標（ｘ，ｙ）のマイクロレンズに対応する画素の画素値の合計、
　Ｉａｖｅｚ（ｘ，ｙ）は光軸Ｌ方向位置ｚの座標（ｘ，ｙ）のマイクロレンズ５ａに対応する画素の画素値の合計値のシーケンス全体の平均値である。
　また、ｃ×ｚは整数になるように丸められる。

here,
z is the position in the optical axis L direction,
c is a coefficient determined by the configuration of the imaging optical system 3,
Iz _k (x, y) is the sum of the pixel values of the pixels corresponding to the microlens at the coordinate (x, y) of the kth frame, with the position z in the optical axis L direction,
Iavez (x, y) is the average value of the entire sequence of the sum of the pixel values of the pixels corresponding to the microlens 5 a at the coordinates (x, y) of the position z in the optical axis L direction.
Also, c × z is rounded to be an integer.

  For example, FIGS. 9A and 9B schematically show a part of a light field image of m = n = 5.
For example, in the case of c × z = 0, that is, as shown in FIGS. 7A and 8A, in the case where the subject S is disposed at the position z in the optical axis L direction focusing on the microlens 5a, FIG. As shown in Equations (5) and (5), pixel values of different pixels corresponding to the same microlens 5a are added.

  Also, in the case of c × z = 1, that is, as shown in FIGS. 7B, 7C, 8B and 8C, when the in-focus position is shifted, it is shown in FIGS. Thus, the luminances of pixels shifted by 5 × i in the x direction and 5 × j in the y direction are added.

  In the case of dividing into a plurality of areas according to the optical axis L direction position z of the subject S, the event evaluation value Az _k calculated for the optical axis L direction position z in the range of each area may be averaged.
In Equation 4, the difference between Iz _k (x, y) and the sequence average value (reference value) is squared so that the effect of the point emphasized in refocusing becomes large. This part is weakened by refocusing by not using it for deriving the evaluation value when abs (Iz _k (x, y)-Iavgz (x, y)) is less than or equal to the threshold value TH, as shown in Eq. You may make it the influence of the point to be made small.

The distance between the optical axis L direction positions z in each region may be, for example, an optical axis L direction position z such that c × z is an integer, or the distance between the optical axis L direction positions z is theoretical It may be set to have about the resolution. The range of the interval or area of the optical axis L direction position z calculated according to the optical axis L direction position z may be changed. For example, it may be fine at places where the optical axis L direction position z is small and rough at places where it is large.

Thus, by dividing the area in the direction of the optical axis L to calculate an event evaluation value Az _k, the change in the event evaluation values Az _k due to the occurrence of the event becomes more prominent, it is possible to improve the sensitivity of detection of events it can. Furthermore, it is possible to confirm at which optical axis L-direction position z the event has occurred, and it is possible to simultaneously grasp the optical axis L-direction position z of event occurrence and the time.

In addition, in the case where the position in the direction of the optical axis L in which an event occurs is previously included, the event evaluation value Az _k may be calculated only in the range of the position z in the narrow optical axis L direction. Has the advantage of being able to
In addition, although the case where rounding is performed so that c × z is an integer has been described, the decimal value can also be handled by a method of interpolating pixel values from a plurality of pixels. Further, although the pixel values of all the pixels corresponding to all the microlenses 5a are added when calculating the sum of the pixel values, the calculation may be performed using only some pixels.

  Further, when it is desired to monitor the event evaluation value Az _k in real time, the sum of squares of differences with the pixel value of the corresponding pixel of the light field image of the immediately preceding frame is calculated as shown in Eq. It may be an event evaluation value Az _k .

In addition, machine learning may be used to calculate the event evaluation value Az _k , determine the event, classify the event, and the like from the light field image and the feature of the sequence.
By learning about what an event is by using light field images of a large number of sequences in advance, an event evaluation value Az _k representing an event likeness based on a learning result is obtained from the acquired light field image sequence. It can be acquired. In addition, classification of the generated event may be performed by learning including the type of event or noise component (for example, calcium emission, movement of subject S, incidence of external light, etc.).

Further, in the present embodiment, for each pixel of the light field image of each frame, a value obtained by adding the absolute value of the difference value indicating the deviation from the reference value for the entire sequence is used as the event evaluation value. Any other representative value, for example, any statistical value such as an average value, a maximum value, a minimum value, and a median value may be adopted as the event information.

Further, in the present embodiment, the case where the pixel 9a of the imaging device 9 matches the pixel on the light field image used for event detection is illustrated, but instead, the pixel 9a of the imaging device 9 and the light field image The upper pixel may not match.
In an actual optical system, the pitch of the micro lens 5a may not be an integral multiple of the pixel pitch, or setting errors may occur such as being slightly rotated and disposed, and the pixel is not A calibration process may be performed to interpolate and rearrange. In this case, strictly speaking, the pixel 9a of the image sensor 9 and the pixel on the light field image used for event detection do not match.

DESCRIPTION OF SYMBOLS 1 light field imaging device 2 image processing device 3 imaging optical system 5 micro lens array 5a micro lens 9 imaging element 9a pixel S object

Claims (9)

Multiple frames of light field images acquired over time are input,
The representative value of the numerical values indicating the deviation from the predetermined reference value of the pixel values of the pixels included in the light field image for at least a partial region of the light field image of at least one of the plurality of frames An image processing apparatus that calculates and outputs a certain event evaluation value. The image processing apparatus according to claim 1, wherein the reference value is an average value of pixel values of corresponding pixels of the light field image. The image processing apparatus according to claim 1, wherein the reference value is an average value of pixel values of a plurality of pixels of the light field image of the plurality of frames. The image processing apparatus according to claim 1, wherein the reference value is an average value of pixel values of all pixels of all the light field images. The image processing apparatus according to claim 1, wherein the reference value is a pixel value of a corresponding pixel in the light field image of the frame different in the time axis direction. The image processing apparatus according to claim 5, wherein the reference value is a pixel value of a corresponding pixel in the light field image of the frame adjacent in the time axis direction. An imaging optical system which condenses light from a subject and forms an image of the subject;
A microlens array having a plurality of microlenses arranged two-dimensionally at the position of a primary image formed by the imaging optical system or at a position conjugate to the primary image and focusing the light from the imaging optical system When,
An imaging device having a plurality of pixels for receiving light collected by the microlens, and photoelectrically converting the light received in the pixels to generate the light field image;
A light field imaging apparatus comprising: the image processing apparatus according to any one of claims 1 to 6, which processes the light field image generated by the imaging element. The pixel is disposed in each area corresponding to the microlens.
The light field imaging according to claim 7, wherein the image processing device calculates the event evaluation value by weighting each of the pixels corresponding to each of the microlenses in accordance with an incident angle of light to each of the microlenses. apparatus. The image processing apparatus divides each of the light field images into a plurality of regions corresponding to the incident angle of light to each of the microlenses, and calculates the event evaluation value for each of the divided regions. The light field imaging device according to claim 1.

Images