JP2011101158A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problems such as the deterioration in the SN of images, the degradation in a resolution due to the decline of the number of pixels and the increase in power consumption when distortion is corrected by a rolling shutter system. <P>SOLUTION: The imaging apparatus includes: an image sensor in which a plurality of pixels are arranged in a matrix shape, and which captures an object and outputs image signals; a read means for reading the image signals in a row unit by prescribed cycle by the rolling shutter system; a generation means for generating images; a first detection means for detecting the motion vector of a moving object included in the object in the image; a second detection means for detecting a distortion angle indicating the angle of a vertical direction component edge which is the edge of the component in a direction vertical to a row direction of the moving object in the image; a first calculation means for calculating a correlation value indicating the correlation of both on the basis of the motion vector and the distortion angle; a decision means for determining whether distortion is generated in the moving object on the basis of whether the correlation value exceeds a prescribed value; and a change means for changing a cycle on the basis of the decision results of the decision means. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging apparatus.

  2. Description of the Related Art Conventionally, a camera that reduces the influence of distortion generated on a moving image acquired by a CMOS solid-state imaging device is known (for example, Patent Document 1).

JP 2004-140479 A

  However, since the image is read at a high rate regardless of the magnitude of the distortion, there are problems such as deterioration in the SN of the image, deterioration in resolution due to a decrease in the number of pixels, and increase in power consumption.

  An image pickup apparatus according to a first aspect of the present invention includes a plurality of pixels arranged in a matrix, an image pickup device that picks up an image of a subject and outputs an image signal, and a predetermined image signal in units of rows from the image pickup device by a rolling shutter system. A reading means that reads out every period of time, a generating means that generates an image based on the read image signal, a first detection means that detects a motion vector of a moving subject included in the subject, and an image Based on the second detection means for detecting the distortion angle indicating the angle of the vertical component edge that is the edge of the component in the direction perpendicular to the row direction of the moving object, the detected motion vector, and the detected distortion angle Then, based on whether or not the calculated correlation value exceeds a predetermined value, whether or not distortion has occurred in the moving subject is calculated. A constant determining means, based on the determination result of the determining means, characterized in that it comprises a changing means for changing the cycle.

  According to the present invention, it is possible to determine whether distortion has occurred based on the motion vector of the moving subject and the distortion angle of the vertical component, and to change the period for acquiring the image signal.

The figure which shows the principal part structure of the digital camera by embodiment of this invention. The block diagram which shows the structure of the control system of the digital camera by embodiment Diagram explaining rolling distortion The figure explaining the histogram of the diagonal component Timing chart explaining frame rate change Flowchart for explaining operation of digital camera according to embodiment Flowchart for explaining operation of digital camera according to embodiment Flowchart for explaining operation of digital camera according to embodiment Flowchart for explaining operation of digital camera according to embodiment Flowchart for explaining operation of digital camera according to embodiment

  A camera according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing a main configuration of the digital camera 1. An interchangeable lens 2 including a photographing lens L1 and a diaphragm 20 is detachably mounted on the body of the digital camera 1. On the body side of the digital camera 1, a quick return mirror 10, a focusing screen 11, a pentaprism 12, an eyepiece lens 13, and an image sensor 14 are provided.

  FIG. 2 is a block diagram schematically showing the control system of the digital camera 1. In FIG. 2, the components shown in FIG. The control system of the digital camera 1 includes an image sensor 14, an A / D conversion circuit 16, a timing generator 17, a signal processing / control circuit 18, an LCD drive circuit 19, a liquid crystal display 191, a memory 20, an operation unit 30, and a memory card. An interface 31 is provided.

  Referring to FIG. 1, subject light that has entered the digital camera 1 after passing through the photographing lens L1 is guided upward by a quick return mirror 10 positioned as shown by a solid line in FIG. 1 before the shutter release. An image is formed on the focusing screen 11. The subject image formed on the focusing screen 11 is guided to the eyepiece 13 by the pentaprism 12. As a result, the subject image is observed by the user. Part of the subject light passes through the semi-transmissive region of the quick return mirror 10, is reflected downward by the sub mirror 10a, and enters a focus detection sensor (not shown). After the release, the quick return mirror 10 is rotated to the position indicated by the broken line in FIG. 1, the subject light is guided to the imaging device 14, and the subject image is formed on the imaging surface.

The control system will be described in detail with reference to FIG.
The imaging element 14 is an XY address type photoelectric conversion element having a large number of pixels arranged in a matrix. The image sensor 14 is driven under the control of a signal processing / control circuit 18 to be described later to capture a subject image input through the photographing lens L1, and outputs an image signal obtained by the imaging. Further, the image sensor 14 is driven by a method (so-called rolling shutter method) in which shutters are sequentially released for each scanning line (pixel row) every predetermined horizontal scanning period.

  R (red), G (green), and B (blue) color filters are provided on the imaging surface of the imaging device 14 so as to correspond to the pixel positions, respectively. Since the image sensor 14 captures a subject image through the color filter, the image signal output from the image sensor 14 has RGB color system color information. The image signal output from the image sensor 14 is subjected to analog processing (such as gain control) by an AFE circuit (not shown) and the like, and is input to the A / D conversion circuit 16. The A / D conversion circuit 16 is a circuit that performs analog processing on an image signal output from the image sensor 14 and converts the image signal into a digital image signal. The timing generator 17 outputs a timing signal to the image sensor 14 and the A / D conversion circuit 16 according to a command from the signal processing / control circuit 18, and sets the drive timing of the image sensor 14 and the A / D conversion circuit 16. It is a circuit to control.

  The signal processing / control circuit 18 includes an unillustrated CPU, ROM, RAM, and the like, and is an arithmetic circuit that controls each component of the digital camera 1 and executes various data processing. The signal processing / control circuit 18 controls the timing generator 17 described above. The signal processing / control circuit 18 performs image processing such as Bayer processing, white balance processing, color processing, and gradation correction processing on the input image signal to generate image data. Then, the signal processing / control circuit 18 executes JPEG compression processing on the image data that has been subjected to image processing.

  The signal processing / control circuit 18 functionally includes a distortion determination unit 181, a frame rate change unit 182, a motion detection unit 183, an oblique line detection unit 184, and a rate calculation unit 185. The distortion determination unit 181 determines the degree of distortion (hereinafter referred to as rolling distortion) that occurs in the subject due to the rolling shutter method described later. Rolling distortion is caused by the difference between the charge accumulation start time of the pixel arranged at the position corresponding to the upper part of the image and the charge accumulation start time of the pixel arranged at the position corresponding to the lower part of the image in the image sensor 14. . That is, since positions where exposure start times are different occur in the same subject, the subject is distorted on the acquired image.

  FIG. 3A shows a case where the moving subject Q moves in the arrow direction P. FIG. 3B shows a state in which rolling distortion has occurred in the moving subject Q on the captured image Z. Such rolling distortion is particularly noticeable when a subject having an edge in a direction (vertical direction) perpendicular to the row direction of the captured image Z is a moving subject that moves at high speed in the row direction (horizontal direction). Based on the determination result of the distortion determination unit 181, the frame rate changing unit 182 sets the frame rate Sr of the live view image to a normal cycle (for example, Sr = 1/30 [s]) and a plurality of high-speed cycles (for example, as described later). Sr = 1/60 [s], 1/120 [s], etc.). Note that the distortion determination unit 181, the frame rate change unit 182, the motion detection unit 183, the oblique line detection unit 184, and the rate calculation unit 185 will be described in detail later.

  The memory 20 is a volatile storage medium used for temporarily storing data during and after image processing and image compression processing. The memory card interface 31 is an interface to which the memory card 32 can be attached and detached. The memory card interface 31 writes image data to the memory card 32 or reads image data recorded on the memory card 32 based on the control of the control circuit 18. The memory card 32 is a semiconductor memory card such as a compact flash (registered trademark) or an SD card.

  The LCD drive circuit 19 is a circuit that drives the liquid crystal display 191 based on a command from the signal processing / control circuit 18. The liquid crystal display 191 displays the display image data created by the signal processing / control circuit 18 based on the image data recorded on the memory card 32 in the reproduction mode. The liquid crystal display 191 is configured to display a so-called live view image. The live view is a display mode in which the quick return mirror 10 is flipped upward before the release and the image captured by the image sensor 14 is displayed on the liquid crystal display 191 in real time. is there. In this case, the signal processing / control circuit 18 creates display image data of VGA size (640 pixels × 480 pixels) based on the image data and outputs the display image data to the LCD drive circuit 19.

  The operation unit 30 is a switch that receives a user operation. The operation unit 30 includes a power switch, a release switch, other setting menu display changeover switches, a setting menu determination button, and the like. The moving image shooting mode and the setting of the live view mode for displaying the above-described live view image are also performed by the user operating the operation unit 30.

  When the live view mode or the moving image shooting mode is set by operating the operation unit 30, the signal processing / control circuit 18 rotates the quick return mirror 10 to the position indicated by the broken line in FIG. 1 and passes through the shooting lens L1. The subject light is guided to the image sensor 14. At this time, the frame rate changing unit 182 sets the frame rate Sr (= 1/30 [s]) of the normal cycle. The frame rate changing unit 182 outputs (thinned out) an image signal from pixels included in a pixel row (for example, every three rows) in the vertical direction among all the pixels constituting the image sensor 14. The timing generator 17 is instructed as follows.

  Next, a process for determining the amount of rolling distortion and a process for changing the readout cycle of the image signal based on the rolling distortion, that is, the frame rate Sr will be described. This processing is executed by the distortion determination unit 181, the frame rate change unit 182, the motion detection unit 183, the oblique line detection unit 184, and the rate calculation unit 185 of the signal processing / control circuit 18.

The rolling distortion amount determination processing includes motion detection processing, oblique line detection processing, distortion amount determination processing, and rate calculation processing. Details of each process will be described below.
-Motion detection process-
The motion detection unit 183 calculates the moving speed of the moving subject using the image data. In this case, the motion detection unit 183 performs a known block matching operation or the like based on at least the (n−1) th frame image data F (n−1) and the nth frame image data F (n). To calculate the motion vector of each pixel. Then, the motion detection unit 183 divides the entire area of the image data F (n) into, for example, a 16 × 16 pixel area, that is, 256 divided areas, and a motion vector amount for each divided area.
Is calculated. Here, (n, m) indicates the coordinates of the pixels included in the divided area. The motion detection unit 183 calculates the sum VP (I, J) of the motion vector amounts of each divided region for each RGB component in each divided region using the following equation (1).
(1)

  The motion detection unit 183 creates a motion vector map indicating the distribution of motion vector amounts based on the sum VP (I, J) of motion vector amounts for each divided region. That is, the motion detection unit 183 detects the maximum value of the sum VP (I, J) of the motion vector amounts for each divided region, and uses the maximum value as a reference for the motion vector amount of each divided region with respect to the maximum value. The ratio of the sum VP (I, J) is calculated.

-Diagonal line detection processing-
The oblique line detection unit 184 detects an oblique component included in the image data F (n). In the present embodiment, a vertical sobel operator f (x) (n × n: n ≧ 3) (hereinafter referred to as a vertical sobel) shown in Expression (2) is used as the oblique line detection operator. Note that a differential filter (ie, Laplacian) may be used as the oblique line detection operator.
(2)

The oblique line detection unit 184 performs oblique line detection by scanning (filtering) the image data F (n) using the vertical sobel f (x) for each of the 256 blocks of RGB components, and calculates an oblique component. The oblique line detection unit 184 calculates the sum E (I, J) of the oblique components for each divided region using the following equation (3), and stores it in the memory 20. Here, (n, m) indicates the coordinates of the pixels included in the divided area. Then, the oblique line detection unit 184 creates an intensity map indicating the intensity distribution of the oblique component based on the sum E (I, J) of the oblique component for each divided region. That is, the oblique line detection unit 184 detects the maximum value of the sum E (I, J) of the oblique components for each divided region, and uses the maximum value as a reference, the sum E of the oblique components for each divided region with respect to the maximum value. Calculate the ratio of (I, J).
(3)

-Distortion amount judgment process-
The distortion determination unit 181 calculates the correlation value Ref (I, J) between the motion vector map and the intensity map using the following equation (4). Then, when the correlation value Ref (I, J) satisfies the following relational expression (5), that is, when the correlation value Ref (I, J) exceeds a predetermined threshold C, the distortion determination unit 181 performs the rolling distortion. Is determined to have occurred. That is, the distortion determination unit 181 determines that noticeable distortion has occurred on the image when the moving amount of the moving subject is large and the moving subject has a large amount of oblique components.
Ref (I, J) = E (I, J) * VP (I, J) (4)
Ref (I, J)> C (5)

-Rate calculation processing-
When the distortion determination unit 182 determines that the rolling distortion has occurred in the moving subject through the distortion amount determination process, the rate calculation unit 185 sets a new frame rate Sr to change the frame rate in the live view mode. calculate. That is, when the influence of the rolling distortion is significant, the rate calculation unit 185 causes the frame rate Sr so that the distortion of the edge component in the vertical line portion of the moving subject, that is, the direction perpendicular to the row direction of the captured image is not noticeable. To decide.

The rate calculation unit 185 detects the vertical edge included in the image data for each RGB component using the vertical sobel f (x). For the edges detected in each RGB component, the rate calculation unit 185 performs filtering to detect the angle of the vertical edge. The rate calculation unit 185 includes a vertical sobel f (x) expressed by the above-described equation (2) and a horizontal sobel filter f (y) expressed by the following equation (6) (hereinafter referred to as a horizontal sobel). Based on the above, angle detection is performed by detecting the tangent angle θ between the vertical sobel f (x) and the horizontal sobel f (y) as shown in the following equation (7).
(6)
θ = tan ⁻¹ (f (y) / f (x)) (7)

  When the angle calculation is performed as described above, the rate calculation unit 185 creates a histogram indicating the angle distribution. FIG. 4 shows an example of a histogram. In FIG. 4, the horizontal axis is an angle [rad], and 0 represents perpendicular to the row direction of the captured image. The rate calculation unit 185 calculates a new frame rate Sr when the peak of the distribution of angles included in the range X excluding 0 and its vicinity in FIG. 4 exceeds a predetermined threshold D. In the histogram shown in FIG. 4, a distribution (intensity) peak exceeding the threshold D occurs at the angle θ1. That is, it is indicated that there are strong vertical edges in the direction of the angle θ1 or a large number of vertical edges in the image.

The rate calculation unit 185 calculates the reference angle θ0. The rate calculation unit 185 performs the angle detection using the image data F (n0) acquired at the fastest frame rate Sr0, and sets the detection angle at this time as the reference angle θ0. And the rate calculation part 185 calculates angle (phi) using the following formula | equation (8).
φ = θ1-θ0 (8)

  The distortion determination unit 181 determines that no rolling distortion has occurred when the angle φ calculated by the above equation (8) is less than the predetermined threshold E. That is, the distortion determination unit 181 determines that the oblique component of the image data F (n) is not due to rolling distortion, but is due to an edge component having high strength in the oblique direction inherent to the moving subject. In such a case, the distortion determination unit 181 determines that no rolling distortion has occurred in the moving subject.

When the angle φ is greater than or equal to the threshold E, the rate calculation unit 185 calculates a new frame rate Sr using the following equation (9) according to the angle θ1 exceeding the threshold D as described above.
Sr = 30 * INT (θ1 / π) (9)
When there are a plurality of angles θ having intensities exceeding the threshold D, the rate calculation unit 185 calculates the frame rate Sr based on the angle corresponding to the maximum intensity. It is assumed that a predetermined upper limit value is set in advance for the frame rate Sr based on the S / N ratio of the image.

-Frame rate change processing-
The frame rate changing unit 182 instructs the image sensor 14 via the timing generator 17 so that the frame rate Sr is calculated by the rate calculating unit 185 using Expression (9). Assuming that the calculated frame rate Sr is 1/60 [s], the frame rate changing unit 182 has a frame rate Sr of 1/60 [s] when acquiring the image data F (n + 1) of the (n + 1) th frame. ] Is instructed to the timing generator 17. That is, the frame rate changing unit 182 sends the image signal to the timing generator 17 so that image signals are thinned out from pixels included in the pixel rows at intervals of six rows, for example, among all the pixels constituting the image sensor 14. Instruct. As a result, the number of pixels that output an image signal is reduced, so that the frame rate Sr can be set at a high speed.

  It should be noted that the number of pixels that output image signals by thinning is not less than the number of pixels required to create image data of VGA size (640 pixels × 480 pixels). In other words, when the number of pixels corresponding to the high-speed frame rate Sr calculated by the rate calculation unit 185 is less than the number of pixels of the VGA size, the frame rate changing unit 182 does not depend on the calculated frame rate Sr. Assume that the interval between pixel rows to be thinned out is determined so as to be the number of pixels of size. The same processing as the processing described above is applied to image data acquired after the image data F (n + 1) of the (n + 1) th frame every time image data in each frame is acquired.

  FIG. 5 shows a timing chart for changing the frame rate Sr for each frame. FIG. 5A shows a state in which image data is acquired at a frame rate Sr (= 1/30 [s]) of a normal cycle. On the other hand, FIG. 5B shows a case where image data is acquired while performing the above-described change processing of the frame rate Sr. The frame rate Sr of the image data F (n + 1) is set to 1/60 [s] using the image data F (n). Similarly, the frame rate Sr of the image data F (n + 2) is set to 1/60 [s] using the image data F (n + 1). Since it is determined that no rolling distortion has occurred using the image data F (n + 3), the frame rate Sr of the image data F (n + 4) is the frame rate Sr (= 1/1 /) of the image data F (n + 3). 120 [s]) is set at a lower speed (1/60 [s]). For image data F (n + 5) and later, since it is determined that no rolling distortion has occurred, the frame rate Sr is sequentially changed to become lower, and the frame rate Sr (= 1/30 [s]) of the normal cycle is set. Is set.

  The operation of the digital camera 1 according to the embodiment will be described with reference to the flowcharts of FIGS. The processing of FIGS. 6 to 10 is performed by executing a program in the signal processing / control circuit 18. This program is stored in a memory (not shown), and is activated and executed when a signal instructing setting of a live view mode or a moving image shooting mode is input by the operation unit 30.

  In step S101, the frame rate changing unit 182 sets the frame rate Sr of the image sensor 14 to the normal cycle (Sr = 1/30 [s]), and proceeds to step S102. In step S102, the frame rate changing unit 182 sets the frame in the blank period during the acquisition of the image data F (n) of the nth frame and the image data F (n + 1) of the (n + 1) th frame. The image sensor 14 is instructed via the timing generator 17 so that the rate Sr is reached, and the process proceeds to step S103.

  In step S103, the signal processing / control circuit 18 takes in the image signal output from the image sensor 14 and proceeds to step S104. In step S104, the frame rate changing unit 182 sets the frame rate Sr of the image sensor 14 to the fastest cycle (Sr0). Then, the frame rate changing unit 182 instructs the image sensor 14 via the timing generator 17 so that the set frame rate Sr0 is obtained, and proceeds to step S105.

  In step S105, the signal processing / control circuit 18 takes in the image signal output from the image sensor 14, acquires the image data F (n0), and proceeds to step S106. In step S106, the signal processing / control circuit 18 performs various image processing (for example, white balance processing, color processing, gradation correction processing, etc.) on the image signal of the nth frame acquired in step S103. Image data F (n) is generated, and the process proceeds to step S107. In step S107, the signal processing / control circuit 18 generates display image data using the image data F (n) generated in step S106. Then, the signal processing / control circuit 18 outputs the generated display image data to the LCD drive circuit 19 to display it on the liquid crystal display 191 as an image of the nth frame, and proceeds to step S108. In step S108, the signal processing / control circuit 18 performs a rolling distortion amount determination process and proceeds to step S109 in FIG. Details of the process in step S108 will be described later.

  In step S109 in FIG. 7, based on the processing in step S108, the distortion determination unit 181 determines whether rolling distortion has occurred in the moving subject on the image corresponding to the image data F (n) of the nth frame. Determine. If rolling distortion has occurred, an affirmative determination is made in step S109 and the process proceeds to step S110. If no rolling distortion has occurred, a negative determination is made in step S109 and the process proceeds to step S111 described later.

  In step S110, the rate calculation unit 185 detects the angle by detecting the tangent angle θ between the vertical sobel f (x) and the horizontal sobel f (y) as shown in Expression (7). Then, the rate calculation unit 185 creates a histogram of the detected angles, detects the angle θ1 having the maximum intensity, and proceeds to step S111. In step S111, the rate calculation unit 185 determines whether the angle θ1 is greater than or equal to the threshold value D. If the angle θ1 is equal to or greater than the threshold value D, step S111 is affirmed by the rate calculation unit 185, and the process proceeds to step S112. When the angle θ1 is less than the threshold value D, the rate calculation unit 185 makes a negative determination in step S111 and returns to step S102 in FIG.

  In step S112, the rate calculation unit 185 detects the angle θ0 using the vertical sobel f (x) and the horizontal sobel f (y) for the image data F (n0) acquired in step S105, and then proceeds to step S113. move on. In step S113, rate calculation unit 185 calculates angle φ using equation (8), and proceeds to step S114.

  In step S114, the distortion determination unit 181 determines whether the angle φ calculated in step S113 is greater than or equal to the threshold value E. When the angle φ is greater than or equal to the threshold value E, the distortion determination unit 181 makes a positive determination in step S114 and proceeds to step S115. When the angle φ is less than the threshold E, the distortion determination unit 181 makes a negative determination in step S114 and proceeds to step S116. In step S115, the distortion determination unit 181 calculates a new frame rate Sr using the above-described equation (9), and proceeds to step S117. In step S116, the frame rate changing unit 182 reduces the frame rate Sr of the image sensor 14 to 2/3 of the current frame rate, and returns to step S117.

  In step S117, it is determined whether the end of the live view mode or the moving image mode is instructed. When a signal for instructing the end of the live view mode or the moving image mode is input along with an operation such as switching to the playback mode, an affirmative determination is made in step S117 and the process ends. If a signal for instructing termination is not input, a negative determination is made in step S117, and the process returns to step S102 in FIG.

Next, details of the process in step S108 will be described using the flowcharts of FIGS.
In step S201, the motion detection unit 183 performs a motion detection process and proceeds to step S202. The details of the motion detection process will be described later with reference to the flowchart of FIG. In step S202, the oblique line detection unit 184 performs an oblique line detection process and proceeds to step S203. Details of the oblique line detection processing will be described later with reference to the flowchart of FIG.

  In step S203, the distortion determination unit 181 calculates a correlation value Ref (I, J) using equation (4) based on the motion vector map created in step S201 and the intensity map created in step S202. Then, the process proceeds to step S204. In step S204, the distortion determination unit 181 determines whether or not the correlation value Ref (I, J) exceeds the threshold value C, that is, whether or not the relational expression (5) is satisfied. If the relational expression (5) is satisfied, the distortion determination unit 181 makes a positive determination in step S204 and proceeds to step S205. In step S205, the distortion determination unit 181 determines that the image data F (n) has distortion, ends the rolling distortion amount determination processing, and proceeds to step S109 in FIG. If the relational expression (5) is not satisfied, the distortion determination unit 181 makes a negative determination in step S204 and proceeds to step S206. In step S206, the distortion determination unit 181 determines that there is no distortion with respect to the image data F (n), ends the rolling distortion amount determination processing, and proceeds to step S109 in FIG.

Details of the motion detection process in step S201 will be described with reference to the flowchart of FIG.
In step S301, the motion detection unit 183 uses a method such as block matching by using the image data F (n-1) of the (n-1) th frame and the image data F (n) of the nth frame. Then, the motion vector of each pixel is calculated and the process proceeds to step S302. In step S302, the motion detection unit 185 calculates the sum VP (I, J) of the motion vectors for each block using the equation (1), and proceeds to step S303.

  In step S303, the motion detection unit 183 calculates the ratio of the sum VP (I, J) of the motion vectors of each block to the maximum value of the sum VP (I, J) of the motion vectors, and calculates the motion vector map. The motion detection process is completed, and the process proceeds to step S202 in FIG.

Next, details of the oblique line detection processing in step S202 will be described using the flowchart of FIG.
In step S401, the oblique line detection unit 184 performs oblique line detection on the RGB components of each block using the vertical sobel f (x), and proceeds to step S402. In step S402, the oblique line detection unit 184 calculates the sum E (I, J) of the oblique components for each block using the equation (3), and proceeds to step S403. In step S403, the oblique line detection unit 184 stores the calculated total sum E (I, J) of each oblique component in the memory 20, and proceeds to step S404. In step S404, the oblique line detection unit 184 calculates the ratio of the sum E (I, J) of the oblique components for each block to the maximum value of the sum E (I, J) of the oblique components for each block, and the intensity. A map is created to end the oblique line detection process, and the process proceeds to step S203 in FIG.

According to the digital camera 1 according to the embodiment described above, the following operational effects can be obtained.
(1) The signal processing / control circuit 18 reads an image signal from the image sensor 14 in a row unit by a rolling shutter method at a predetermined frame rate Sr. Based on the read image signal, the image data F (n) is read out. Generate. Then, the motion detection unit 183 detects the motion vector of the moving subject included in the subject in the image data F (n), and the oblique line detection unit 184 uses the vertical sobel f (x) in the image data F (n). The distortion angle indicating the angle of the vertical component edge, which is the edge of the component in the pixel row direction and the vertical direction, is detected using the moving subject. The distortion determination unit 181 calculates the correlation value Ref (I, J) based on the motion vector map based on the motion vector and the intensity map based on the distortion angle, and the correlation value Ref (I, J) exceeds the predetermined value C. In such a case, it is determined that rolling distortion has occurred in the moving subject. The frame rate changing unit 182 changes the frame rate Sr based on the determination result of the distortion determining unit 181. Therefore, since the frame rate Sr can be changed according to the degree of distortion of the moving subject, it is accompanied by a decrease in the number of pixels that accumulate charge compared to the case where the moving subject is always changed to a higher frame rate when there is movement. It is possible to acquire a high-quality image that prevents S / N degradation and resolution degradation of the image and suppresses rolling distortion. Furthermore, power consumption can be reduced compared to the case where the frame rate Sr is always increased.

(2) The oblique line detection unit 184 detects the distortion angle of the horizontal component of the moving subject using the horizontal sobel f (y), and based on the detection results of the vertical sobel f (x) and the horizontal sobel f (y). Thus, the distortion angle in the oblique direction of the moving subject is detected. Then, the rate calculation unit 185 calculates a new frame rate Sr based on the oblique distortion angle, and the frame rate change unit 182 changes the current frame rate Sr to the new frame rate Sr. . As shown in FIG. 3B, the rolling distortion occurring when the frame rate Sr is 1/30 [s] is the case where the frame rate Sr shown in FIG. 3C is 1/60 [s]. Has been suppressed. However, as shown in FIG. 3D, when the frame rate Sr is set to 1/120 [s], the influence of rolling distortion is hardly noticeable on the image. That is, when shooting the moving subject Q in FIG. 3A, the rate calculation unit 185 calculates 1/120 [s] as the new frame rate Sr. Therefore, depending on the amount of rolling distortion, it is possible to change the frame rate Sr suitable for making the rolling distortion inconspicuous, that is, the frame rate Sr that can suppress the influence of the occurrence of rolling distortion, so that live view images and videos The image quality of the image can be improved.

(3) The rate calculation unit 185 uses the equation (9) when the distortion determination unit 182 determines that the correlation value Ref (I, J) exceeds the predetermined value C based on the relational equation (5). The new frame rate Sr is calculated so that the cycle is short, that is, the frame rate Sr is high. Further, the rate calculation unit 185 uses the equation (9) when the distortion determination unit 182 determines that the correlation value Ref (I, J) does not exceed the predetermined value C based on the relational equation (5). Thus, the new frame rate Sr is calculated so that the cycle becomes longer, that is, the frame rate Sr becomes lower. Therefore, when the influence of rolling distortion on the moving subject becomes inconspicuous, the frame rate Sr can be changed to a low speed, so that power consumption can be suppressed.

(4) The rate calculation unit 185 detects the angle of the diagonal component inherent to the moving subject as the reference angle θ0 based on the vertical sobel f (x) and the horizontal sobel f (y), and the distortion determination unit No. 181, when the angle φ based on the reference angle θ0 is less than a predetermined threshold E, it is determined that the moving subject is not distorted. Therefore, for example, when it is difficult to detect a motion vector such as shooting while enlarging characters on a single color background, the moving subject can be excluded from the rolling distortion determination target, and the distortion amount determination accuracy can be improved. .

The digital camera 1 of the embodiment described above can be modified as follows.
(1) The threshold value D may be set to be variable. In this case, the rate calculation unit 185 may change the threshold value D so that the threshold value D becomes smaller as the value of the motion vector increases, based on the motion vector of the moving subject detected by the motion detection unit 183. It is assumed that a table or the like in which the magnitude of the motion vector value is associated with the threshold value D is recorded in advance in a predetermined memory (not shown).

  The threshold value D may be set to be variable based on the luminance value. In this case, for example, when the luminance value calculated based on the RGB components of the image data is high, the rate calculation unit 185 may change the threshold value D to be a small value because the contrast of the image is high. . Also in this case, a table in which the brightness value and the threshold value D are associated is recorded in advance in a predetermined memory (not shown).

  Furthermore, the threshold value D may be set to be variable based on any one of the R, G, and B color components of the image data. For example, when the image data includes a large amount of B components among the color components, there is a high possibility that a subject with few vertical edges, such as the sea or the sky, has been shot. Therefore, in such a case, the rate calculation unit 185 may change so that the threshold value D becomes a small value.

(2) The frame rate changing unit 182 is not limited to changing the frame rate Sr based on the distortion amount determination result every time the image data F (n) of each frame is acquired. Sr may be changed. In this case, the distortion determination unit 181 generates rolling distortion based on the motion vector map obtained by integrating the motion vectors for five frames and the intensity map obtained by integrating the sum E (I, J) of the diagonal components for five frames. What is necessary is just to determine the presence or absence.

(3) The distortion determination unit 181 calculates the angle φ (n−1) calculated when the image data F (n−1) is acquired and the angle φ (n calculated when the image data F (n) is acquired. If the absolute value of the difference from () is less than or equal to a predetermined threshold G, it may be determined that no rolling distortion has occurred in the moving subject. That is, the distortion determination unit 181 determines that the angle φ (n) does not vary much from the angle φ (n−1). When the distortion determination unit 181 determines that the angle φ (n) does not vary, the rate calculation unit 185 does not newly calculate a frame rate for acquiring the image data F (n + 1). As a result, since the frame rate changing unit 182 does not change the frame rate Sr, the image data F (n + 1) is acquired at the same frame rate Sr as when the image data F (n) was acquired.

(4) The motion detection unit 183 may detect the motion vector of the moving object subject to whether or not the user is performing panning photographing. In this case, the digital camera 1 further includes an acceleration sensor and the like. Then, the motion detection unit 183 calculates the motion vector of the digital camera 1 body detected by the acceleration sensor, and the motion vector amount
Subtract from

(5) When the frame rate Sr is changed, a clock for the frame rate changing unit 182 to output a timing signal to the timing generator 17 instead of changing the interval of the pixel rows from which the image signal is read, that is, changing the thinned rows. May be changed. Further, the frame rate changing unit 182 may perform both the thinning-out line and the clock change.

  In addition, the present invention is not limited to the above-described embodiment as long as the characteristics of the present invention are not impaired, and other forms conceivable within the scope of the technical idea of the present invention are also within the scope of the present invention. included. The embodiments and modifications used in the description may be configured by appropriately combining them.

14 image sensor, 18 signal processing / control circuit,
181 distortion determination unit, 182 frame rate change unit,
183 motion detector, 184 diagonal detector,
185 Rate calculator

Claims (7)

An image sensor in which a plurality of pixels are arranged in a matrix, and images a subject and outputs an image signal;
Reading means for reading out the image signal in units of rows by the rolling shutter method from the image sensor;
Generating means for generating an image based on the read image signal;
First detection means for detecting a motion vector of a moving subject included in the subject in the image;
Second detection means for detecting a distortion angle indicating an angle of a vertical component edge that is an edge of a component in a direction perpendicular to the row direction of the moving subject in the image;
First calculation means for calculating a correlation value indicating a correlation between the detected motion vector and the detected distortion angle based on the detected motion vector;
Determining means for determining whether or not the moving subject is distorted based on whether or not the calculated correlation value exceeds a predetermined value;
An imaging apparatus comprising: changing means for changing the cycle based on a determination result of the determination means. The imaging device according to claim 1,
Third detection means for detecting a distortion angle of a lateral component edge that is an edge of a component in a direction parallel to the row direction of the moving subject;
Fourth detection means for detecting an oblique distortion angle of the moving subject based on the distortion angles detected by the second and third detection means;
Second calculation means for calculating a new period based on the oblique strain angle,
The first calculation means calculates a correlation value indicating a correlation between the oblique distortion angle and the motion vector,
The determination means determines whether or not the distortion occurs based on whether or not the correlation value exceeds the predetermined value,
The changing device changes the cycle to the new cycle calculated by the second calculating unit when the determining unit determines that the distortion is generated. The imaging device according to claim 2,
The second calculating unit calculates the new cycle so that the cycle is shortened when the determining unit determines that the correlation value exceeds the predetermined value, and the determining unit calculates the correlation value as the correlation value. An image pickup apparatus, wherein when it is determined that the predetermined value is not exceeded, the new cycle is calculated so that the cycle becomes longer. In the imaging device according to claim 2 or 3,
A fifth detecting means for detecting an angle of an oblique component of the moving object as a reference angle;
The determination unit determines that no distortion has occurred in the moving subject when the detected reference angle is less than a predetermined threshold. The imaging apparatus according to claim 4,
The change unit changes the cycle to the highest speed when acquiring an image for detecting the reference angle. In the imaging device according to any one of claims 2 to 5,
The first detection unit divides the image into a plurality of regions, calculates a motion vector for each of the divided regions,
The fourth detection means detects the oblique distortion angle for each divided area based on the distortion angle detected for each divided area by the second and third detection means,
The first calculation means calculates the correlation value for each of the divided regions,
The determination unit determines whether or not the moving subject is distorted based on whether or not a maximum value of the correlation values calculated for each of the divided regions exceeds the predetermined value. An imaging apparatus characterized by that. In the imaging device according to any one of claims 2 to 5,
The imaging apparatus according to claim 4, wherein the fourth detection unit considers that there is no variation if the variation in the detected distortion angle in the oblique direction is within a predetermined range with respect to the previously detected distortion angle.