CN108235815B - Imaging control device, imaging system, moving object, imaging control method, and medium - Google Patents

Abstract

A camera control device includes an acquisition unit that acquires a first image contained in a first camera image captured when the camera surface and lens are in a first positional relationship, and a second image contained in a second camera image captured when the camera surface and lens are in a second positional relationship. The camera control device includes a calculation unit that calculates the blur amount of each of the first and second images. The camera control device also includes a control unit that controls the positional relationship between the camera surface and the lens based on the blur amounts of the first and second images when the difference between the blur amount of the first image and the blur amount of the second image is greater than or equal to the first threshold.

Description

Camera control device, camera device, camera system, moving body, camera control method, and camera medium

technical field

The present invention relates to an imaging control device, an imaging device, an imaging system, a moving body, an imaging control method, and a medium.

Background technique

Patent Document 1 discloses an image processing device that calculates distance information of a subject in the images using a plurality of images with different blurs captured with different imaging parameters.

Patent Document 1: JP Patent No. 5932476

SUMMARY OF THE INVENTION

When calculating the distance to the subject based on the blur amounts of the plurality of images, if the difference between the blur amounts of the plurality of images is small, the distance to the subject may not be accurately calculated.

The imaging control device of the present invention may include an acquisition unit that acquires a first image and a second image, the first image being included in the first captured image captured in a state where the imaging plane and the lens are in the first positional relationship, and the second image The image is included in the second captured image captured in a state where the imaging plane and the lens are in the second positional relationship. The imaging control device may include a calculation unit that calculates the blurring amount of each of the first image and the second image. The imaging control device may include a control unit that controls the blurring amount of the first image and the second image based on the blurring amount of each of the first image and the second image when the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than a first threshold value. The positional relationship between the camera surface and the lens.

When the difference between the blurring amount of the first image and the blurring amount of the second image is smaller than the first threshold value, the obtaining unit may further obtain a third captured image captured in a state where the imaging plane and the lens are in a third positional relationship Included third image. The calculation section may further calculate the blurring amount of the third image. The control unit may control the positional relationship between the imaging surface and the lens based on the respective blur amounts of the first image and the third image when the difference between the blur amount of the first image and the blur amount of the third image is equal to or greater than the first threshold value .

The acquisition unit may further acquire a fourth image included in the first captured image and a fifth image included in the second captured image. The calculation unit may further calculate the blurring amount of each of the fourth image and the fifth image. When the difference between the blurring amount of the first image and the blurring amount of the second image is smaller than the first threshold value, and the difference between the blurring amount of the fourth image and the blurring amount of the fifth image is equal to or greater than the first threshold value, the control unit may The blurring amount of each of the fourth image and the fifth image controls the positional relationship between the imaging plane and the lens.

The acquiring unit may further acquire a fourth image included in the first captured image that is adjacent to the first image and a fifth image included in the second captured image that is adjacent to the second image. The calculation unit may further calculate the blurring amount of each of the fourth image and the fifth image. The control unit may, when the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than the first threshold value and the difference between the blurring amount of the fourth image and the blurring amount of the fifth image is equal to or greater than the first threshold value The positional relationship between the imaging plane and the lens is controlled based on the blurring amount of each of the first image, the second image, the fourth image, and the fifth image.

The imaging control device may include a derivation unit that derives the first distance to the first target object included in the first image and the second image based on the blurring amount of the first image and the blurring amount of the second image, and based on the fourth image The blurring amount of the image and the blurring amount of the fifth image are derived to the fourth image and the second distance of the second target object contained in the fifth image. The control unit may control the positional relationship between the imaging plane and the lens based on the first distance and the second distance.

When the difference between the blurring amount of the fourth image and the blurring amount of the fifth image is smaller than the first threshold value, the obtaining unit may further obtain a third captured image captured in a state where the imaging plane and the lens are in a third positional relationship Included sixth image. The calculation section may further calculate the blurring amount of the sixth image. The control unit may, when the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than the first threshold value and the difference between the blurring amount of the fourth image and the blurring amount of the sixth image is equal to or greater than the first threshold value The positional relationship between the imaging plane and the lens is controlled based on the blurring amount of each of the first image, the second image, the fourth image, and the sixth image.

The acquisition unit may further acquire a fourth image included in the first captured image and a fifth image included in the second captured image. The calculation unit may further calculate the blurring amount of each of the fourth image and the fifth image. The imaging control device may include a derivation unit that derives the first distance to the first target object included in the first image and the second image based on the blurring amounts of the first image and the second image, and derives the first distance based on the fourth image and the second image. The blur amount of each of the five images is derived to the fourth image and the second distance to the second target object contained in the fifth image. When the first distance satisfies the predetermined imaging condition, the second distance does not satisfy the imaging condition, and the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than the first threshold value, the control unit may be based on the first Distance to control the positional relationship between the camera surface and the lens.

When the first distance satisfies the imaging conditions, the second distance does not satisfy the imaging conditions, and the difference between the blurring amount of the first image and the blurring amount of the second image is smaller than the first threshold, the acquisition unit may further acquire the image capturing surface and the lens. The sixth image included in the third captured image captured in the third positional relationship.

The calculation section may further calculate the blurring amount of the sixth image. The control unit may control the imaging surface and the lens based on the blur amount of the first image and the blur amount of the sixth image when the difference between the blur amount of the first image and the blur amount of the sixth image is equal to or greater than the first threshold value positional relationship.

A determination unit may further include a determination unit that determines a region of the second captured image corresponding to the first image by comparing the feature points included in the first image with the feature points of the second captured image. When the difference between the position of the first image in the first captured image and the position of the region in the second captured image is equal to or smaller than the second threshold value, the obtaining unit may obtain, for the second captured image, the difference between the first captured image and the first captured image. An image of an area of the second captured image having the same positional relationship as the first image is used as the second image.

The acquisition unit may acquire the region as the second image when the difference between the position of the first image in the first captured image and the position of the region in the second captured image is larger than the second threshold value.

When the difference between the position of the first image in the first captured image and the position of the region in the second captured image is larger than the second threshold and equal to or less than the third threshold, the obtaining unit may obtain the image of the region as the second image.

The determination unit may determine the respective feature points based on the brightness of the first image and the brightness of the region of the second captured image.

The determination unit may determine the center of gravity of the brightness of the first image as the feature point included in the first image, and the center of gravity of the brightness of the region of the second captured image as the feature point of the second captured image.

By changing the position of the focus lens included in the lens, it is possible to change from the state in which the imaging surface and the lens are in the first positional relationship to the state in which they are in the second positional relationship.

By changing the position of the imaging plane, it is possible to change from the state in which the imaging plane and the lens are in the first positional relationship to the state in which they are in the second positional relationship.

An imaging device according to an aspect of the present invention includes the imaging control device described above, an image sensor having an imaging surface, and a lens.

An imaging system according to an aspect of the present invention includes the imaging device described above and a support mechanism for supporting the imaging device.

A moving body according to an aspect of the present invention mounts the above-described imaging system and moves.

The imaging control method according to one aspect of the present invention may include a stage of acquiring a first image and a second image, the first image being included in a first captured image captured in a state where the imaging plane and the lens are in a first positional relationship , and the second image is included in the second captured image captured in a state where the imaging plane and the lens are in the second positional relationship. The imaging control method may include a stage of calculating the blurring amount of each of the first image and the second image. The imaging control method may include controlling the imaging plane and the lens based on the respective blurring amounts of the first image and the second image when the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than a first threshold value stage of the positional relationship.

The program according to one aspect of the present invention can cause a computer to execute the step of acquiring the first image and the second image, the first image being included in the first captured image captured in a state where the imaging plane and the lens are in the first positional relationship , and the second image is included in the second captured image captured in a state where the imaging plane and the lens are in the second positional relationship. The program may cause the computer to execute the stage of calculating the blur amount of each of the first image and the second image. The program can cause the computer to execute: when the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than a first threshold, control the imaging surface and the lens based on the blurring amounts of the first image and the second image respectively stage of the positional relationship.

According to one aspect of the present invention, the positional relationship between the imaging plane and the lens can be adjusted more accurately based on the blurring amounts of the plurality of images.

The above summary of the invention does not enumerate all the features of the present invention. Combinations of these feature groups can also be inventions.

Description of drawings

FIG. 1 is a diagram showing an example of the appearance of an unmanned aerial vehicle and a remote control device.

FIG. 2 is a diagram showing an example of a functional block of an unmanned aerial vehicle.

FIG. 3 is a diagram showing an example of a curve representing the relationship between the blurring amount and the lens position.

FIG. 4 is a diagram illustrating an example of a procedure for calculating a distance to a target object based on the blurring amount.

FIG. 5 is a diagram for explaining the relationship between the position of the target object, the position of the lens, and the focal distance.

FIG. 6A is a diagram for explaining the relationship between the difference in blur amount and the specific accuracy of the lens position.

FIG. 6B is a diagram for explaining the relationship between the difference in blur amount and the specific accuracy of the lens position.

FIG. 6C is a diagram for explaining the relationship between the difference in the blurring amount and the specific accuracy of the lens position.

FIG. 7 is a diagram for explaining an image for calculating a blur amount.

FIG. 8 is a diagram showing an example of a curve showing the relationship between the blurring amount and the lens position.

FIG. 9 is a flowchart showing an example of the procedure of AF processing in the BDAF method.

FIG. 10 is a flowchart showing another example of the procedure of the AF process in the BDAF method.

FIG. 11 is a flowchart showing another example of the procedure of the AF process in the BDAF method.

FIG. 12 is a flowchart showing another example of the procedure of the AF process in the BDAF method.

FIG. 13 is a diagram showing another example of functional modules of the unmanned aerial vehicle.

FIG. 14 is a diagram for explaining the movement amount of the target object.

FIG. 15A is a diagram for explaining an example in which the movement amount of the target object is determined based on the center of gravity of luminance.

FIG. 15B is a diagram for explaining an example in which the movement amount of the target object is determined based on the center of gravity of luminance.

FIG. 16 is a flowchart showing an example of a procedure of moving the AF processing frame in accordance with the movement amount of the target object.

FIG. 17 is a diagram showing an example of a hardware configuration.

Detailed ways

Hereinafter, the present invention will be described based on the embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all combinations of the features described in the embodiments are necessary for the solution means of the invention. It will be apparent to those skilled in the art that various changes or improvements can be made to the following embodiments. As is apparent from the description of the claims, such modifications or improvements can be included in the technical scope of the present invention.

The claims, description, drawings, and abstracts include matters subject to copyright protection. The copyright owner has no objection to the reproduction by anyone of these documents as long as it appears in the Patent Office files or records. However, in other cases, all copyrights are reserved.

Various embodiments of the invention may be described with reference to flowcharts and block diagrams, where blocks may represent (1) stages of a process for performing operations or (2) "portions" of means that function to perform operations. Certain stages and "sections" may be implemented by programmable circuits and/or processors. Dedicated circuits may include digital and/or analog hardware circuits. Integrated circuits (ICs) and/or discrete circuits may be included. Programmable circuits may include hardware circuits that can be reconfigured. Reconfigurable hardware circuits may include logical AND, logical OR, logical XOR, logical NAND, logical NOR and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. memory elements, etc.

Computer readable media may include any tangible device capable of storing commands for execution by an appropriate device. As a result, a computer-readable medium having commands stored therein will have an article of manufacture containing commands executable to generate means for performing the operations specified in the flowchart or block diagram. As examples of the computer-readable medium, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like may be included. As more specific examples of the computer-readable medium, a floppy disk (registered trademark), a magnetic tape, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM), or a flash memory may be included ), Electrically Erasable Programmable Read Only Memory (EEPROM), Static Random Access Memory (SRAM), Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD), Blu-ray (RTM) Disc, Memory Stick , integrated circuit card, etc.

Computer readable commands may include either source code or object code written in any combination of one or more programming languages. Source code or object code contains legacy procedural programming languages. Conventional procedural programming languages can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or oriented programming such as Smalltalk, JAVA (registered trademark), and C++. Object programming languages, and the "C" programming language or similar programming languages. Computer readable commands may be provided against a processor or programmable circuitry of a general purpose computer, special purpose computer, or other programmable data processing apparatus, locally or through a wide area network (WAN) such as a local area network (LAN), the Internet, or the like. A processor or programmable circuit may execute computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

FIG. 1 shows an example of the appearance of an unmanned aerial vehicle (UAV) 10 and a remote control device 300 . The UAV 10 includes a UAV main body 20 , a gimbal 50 , a plurality of imaging devices 60 , and an imaging device 100 . The gimbal 50 and the imaging device 100 are examples of imaging systems. The UAV 10 is an example of a moving body propelled by the propelling unit. A moving object is a concept that includes, in addition to UAVs, other flying objects such as aircraft moving in the air, vehicles moving on the ground, ships moving on water, and the like.

The UAV main body 20 includes a plurality of rotors. The plurality of rotors is an example of a propulsion unit. The UAV main body 20 makes the UAV 10 fly by controlling the rotation of the plurality of rotors. The UAV body 20 flies the UAV 10 using, for example, four rotors. The number of rotors is not limited to four. In addition, UAV10 can also be a fixed-wing aircraft without rotary wings.

The imaging device 100 is an imaging camera for imaging a subject included in a desired imaging range. The gimbal 50 supports the camera device 100 in a rotatable manner. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 supports the camera device 100 rotatably with a pitch axis using an actuator. The gimbal 50 further supports the camera device 100 rotatably around the roll axis and the yaw axis using an actuator. The gimbal 50 can change the posture of the imaging device 100 by rotating the imaging device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of imaging devices 60 are cameras for sensing that image the surroundings of the UAV 10 in order to control the flight of the UAV 10 . The two camera devices 60 can be arranged on the nose of the UAV 10 , that is, the front. Furthermore, the other two camera devices 60 can be arranged on the bottom surface of the UAV 10 . The two imaging devices 60 on the front side are paired and can function as a so-called stereo camera. The two imaging devices 60 on the bottom side are also paired, and can function as a stereo camera. Three-dimensional space data around the UAV 10 can be generated based on images captured by the plurality of cameras 60 . The number of imaging devices 60 included in the UAV 10 is not limited to four. The UAV 10 may include at least one camera 60 . The UAV 10 may also be provided with at least one camera device 60 on each of the nose, the tail, the side, the bottom, and the top of the UAV 10 . The angle of view that can be set in the imaging device 60 may be wider than the angle of view that can be set in the imaging device 100 . The camera device 60 may also have a single focus lens or a fisheye lens.

The remote operation device 300 communicates with the UAV 10 to remotely operate the UAV 10 . The remote operation device 300 can wirelessly communicate with the UAV 10 . The remote control device 300 transmits instruction information indicating various commands related to movement of the UAV 10, such as ascending, descending, acceleration, deceleration, forward, backward, and rotation, to the UAV 10. The instruction information includes, for example, instruction information to raise the altitude of the UAV 10 . The indication information may show the altitude at which the UAV 10 should be. The UAV 10 moves so as to be at the height indicated by the instruction information received from the remote operation device 300 . The instruction information may contain an ascend command to ascend the UAV 10 . When the UAV 10 receives an ascending command, it ascends. When the height of the UAV 10 reaches the upper limit height, the UAV 10 can restrict the ascent even if the ascent command is received.

FIG. 2 shows an example of functional modules of the UAV 10 . The UAV 10 includes a UAV control unit 30 , a memory 32 , a communication interface 34 , a propulsion unit 40 , a GPS receiver 41 , an inertial measurement device 42 , a magnetic compass 43 , a barometric altimeter 44 , a temperature sensor 45 , a gimbal 50 , and a camera device 100 .

The communication interface 34 communicates with other devices such as the remote operation device 300 . The communication interface 34 can receive instruction information including various commands to the UAV control unit 30 from the remote operation device 300 . The memory 32 stores the information performed by the UAV control unit 30 on the propulsion unit 40 , the GPS receiver 41 , the inertial measurement unit (IMU) 42 , the magnetic compass 43 , the barometric altimeter 44 , the temperature sensor 45 , the gimbal 50 , the camera 60 , and the camera 100 . Programs required for control, etc. The memory 32 may be a computer-readable recording medium, and may include at least one of SRAM, DRAM, EPROM, EEPROM, and flash memory such as USB memory. The memory 32 may be provided inside the UAV body 20 . It can be provided so as to be detachable from the UAV main body 20 .

The UAV control unit 30 controls the flight and imaging of the UAV 10 according to the program stored in the memory 32 . The UAV control unit 30 may be constituted by a microprocessor such as a CPU or an MPU, a microcontroller such as an MCU, or the like. The UAV control unit 30 controls the flight and imaging of the UAV 10 in accordance with commands received from the remote control device 300 via the communication interface 34 . The propulsion unit 40 propels the UAV 10 . The propulsion unit 40 includes a plurality of rotor blades and a plurality of drive motors for rotating the plurality of rotor blades. The propulsion unit 40 rotates the plurality of rotors via the plurality of drive motors in accordance with a command from the UAV control unit 30 to fly the UAV 10 .

The GPS receiver 41 receives a plurality of signals representing time transmitted from a plurality of GPS satellites. The GPS receiver 41 calculates the position of the GPS receiver 41 , that is, the position of the UAV 10 based on the received plurality of signals. IMU42 detects the pose of UAV10. The IMU 42 detects, as the posture of the UAV 10 , the acceleration in the three-axis directions of the UAV 10 , the front-rear, left-right, and vertical directions, and the angular velocity in the three-axis directions of the pitch, roll, and yaw. The magnetic compass 43 detects the orientation of the nose of the UAV 10 . The barometric altimeter 44 detects the altitude at which the UAV 10 is flying. The barometric altimeter 44 detects the air pressure around the UAV 10, converts the detected air pressure into an altitude, and detects the altitude. The temperature sensor 45 detects the temperature around the UAV 10 .

The imaging device 100 includes an imaging unit 102 and a lens unit 200 . The lens unit 200 is an example of a lens device. The imaging unit 102 includes an image sensor 120 , an imaging control unit 110 , and a memory 130 . The image sensor 120 may be composed of CCD or CMOS. The image sensor 120 outputs image data of the optical image formed through the plurality of lenses 210 to the imaging control unit 110 . The imaging control unit 110 may be constituted by a microprocessor such as a CPU or an MPU, a microcontroller such as an MCU, or the like. The imaging control unit 110 can control the imaging device 100 according to an operation command of the imaging device 100 from the UAV control unit 30 . The memory 130 may be a computer-readable recording medium, and may include at least one of SRAM, DRAM, EPROM, EEPROM, and flash memory such as USB memory. The memory 130 stores programs and the like necessary for the imaging control unit 110 to control the image sensor 120 and the like. The memory 130 may be provided inside the casing of the camera 100 . The memory 130 may be provided to be detachable from the housing of the camera device 100 .

The lens unit 200 includes a plurality of lenses 210 , a lens moving mechanism 212 , and a lens control unit 220 . The plurality of lenses 210 can function as a zoom lens, a variable focal length lens, and a focus lens. At least a part or all of the plurality of lenses 210 are configured to be movable along the optical axis. The lens unit 200 may be an interchangeable lens provided to be detachable from the imaging unit 102 . The lens moving mechanism 212 moves at least a part or all of the plurality of lenses 210 along the optical axis. The lens control unit 220 drives the lens moving mechanism 212 in accordance with the lens control command from the imaging unit 102 to move one or more lenses 210 along the optical axis direction. The lens control commands are, for example, zoom control commands and focus control commands.

The imaging apparatus 100 configured in this way executes auto focus processing (AF processing) to capture an image of a desired subject.

The imaging apparatus 100 determines the distance from the lens to the subject (subject distance) in order to execute the AF process. As a method for determining the subject distance, there is a method based on the blurring amount of a plurality of images captured in a state where the positional relationship between the lens and the imaging plane is different. Here, this method is referred to as a Bokeh Detection Auto Focus (BDAF) method.

For example, the blurring amount (Cost) of the image can be represented by the following equation (1) using a Gaussian function. In Equation (1), x represents the pixel position in the horizontal direction. σ represents the standard deviation value.

[Formula 1]

FIG. 3 shows an example of the curve represented by the formula (1). By aligning the focusing lens to the lens position corresponding to the minimum point 502 of the curve 500, the target object contained in the image I can be brought into focus.

FIG. 4 is a flowchart showing an example of a distance calculation procedure in the BDAF method. First, the imaging device 100 captures the first image I ₁ in a state in which the lens and the imaging plane are in the first positional relationship, and stores the image in the memory 130 . Next, by moving the focus lens or the imaging surface of the image sensor 120 in the optical axis direction, the lens and the imaging surface are in a second positional relationship, and the imaging device 100 captures and stores the second image _I2 in the memory 130 (S101). For example, as in so-called hill-climbing AF, the focus lens or the imaging surface of the image sensor 120 is moved in the optical axis direction so as not to exceed the focus point. The movement amount of the focus lens or the imaging surface of the image sensor 120 may be, for example, 10 μm.

Next, the imaging apparatus 100 divides the image _I1 into a plurality of regions ( S102 ). The feature amount can be calculated for each pixel in the image I2, and the image _I1 can be divided into a plurality of regions using a group of pixels having similar feature amounts as one region. The pixel group in the range set by the AF processing frame in the image _I1 may be divided into a plurality of regions. The imaging apparatus 100 divides the image I ₂ into a plurality of regions corresponding to the plurality of regions of the image I ₁ . The imaging apparatus 100 calculates the target object included in each of the plurality of regions for each of the plurality of regions, based on the respective blurring amounts of the plurality of regions of the image _I1 and the respective blurring amounts of the plurality of regions of the image _I2 the distance to the end (S103).

The calculation process of the distance will be further described with reference to FIG. 5 . Let A be the distance from the lens L (principal point) to the target object 510 (object plane), and B be the distance from the lens L (principal point) to the position (image plane) where the target object 510 is imaged on the imaging plane, Set the focus distance to F. In this case, the relationship between the distance A, the distance B, and the focal distance F can be expressed by the following equation (2) based on the formula of the lens.

[Formula 2]

The focal distance F is determined by the lens position. Therefore, as long as the distance B at which the target object 510 is imaged on the imaging plane can be determined, the distance A from the lens L to the target object 510 can be determined using the formula (2).

As shown in FIG. 5 , the distance B and then the distance A can be determined by calculating the imaging position of the target object 510 according to the blurred size of the target object 510 projected on the imaging surface (circles of confusion 512 and 514 ). That is, the imaging position can be determined by taking into consideration that the magnitude of the blur (blur amount) is proportional to the imaging plane and the imaging position.

Here, the distance from the image I ₁ that is closer to the imaging plane to the lens L is D ₁ . The distance from the image I ₂ farther from the imaging plane to the lens L is set to D ₂ . Individual images are blurred. Let the point spread function at this time be PSF, and let the images at D ₁ and D ₂ be I _d1 and I _d2 , respectively. In this case, for example, the image I ₁ can be represented by the following equation (3) by the convolution operation.

[Formula 3]

I ₁ =PSF*I _d1 …(3)

Furthermore, let the Fourier transform functions of the image data I _d1 and I _d2 be f, and the optical transfer functions (Optical Transfer Functions) obtained by Fourier transforming the point image spread functions PSF ₁ and PSF ₂ of the images I _d1 and I _d2 Transfer Function) are set to OTF ₁ and OTF ₂ , and take the ratio as shown in the following formula (4).

[Formula 4]

The value C shown in Equation (4) corresponds to the amount of change in the blurring amount of each of the images I _d1 and I _d2 , that is, the value C corresponds to the difference between the blurring amount of the image I _d1 and the blurring amount of the image I _d2n .

However, when calculating the distance to the target object included in the two images from two images with different blur amounts, if the difference between the blur amounts of the two images is small, it may not be possible to accurately determine as shown in FIG. 3 . The curve 500 results in a situation where the distance to the target object cannot be accurately calculated. For example, as shown in FIG. 6A , if the difference between the blur amount C(t ₀ ) obtained from the image I ₀ and the blur amount C(t ₁ ) obtained from the image I ₁ is smaller than the threshold Th, the blur amount C(t _{0 )} ) and the curve 522 of the Gaussian function determined by the blur amount C(t ₁ ) may not be an ideal curve. The lens position 524 determined from the curve 522 deviates from the ideal lens position 520 corresponding to the imaging position. Similarly in FIG. 6B , the difference between the blur amount C(t ₀ ) obtained from the image I ₀ and the blur amount C(t ₂ ) obtained from the image I ₂ is smaller than the threshold value Th. In this case, the lens position 528 determined by the curve 526 of the Gaussian function determined from the blur amount C(t ₀ ) and the blur amount C(t ₂ ) deviates from the ideal lens position 520 . On the other hand, for example, as shown in FIG. 6C , when the difference between the blur amount C(t ₀ ) obtained from the image I ₀ and the blur amount C(t ₃ ) obtained from the image I ₃ is equal to or larger than the threshold Th, The lens position 532 determined by the curve 530 of the Gaussian function determined from the blur amount C(t ₀ ) and the blur amount C(t ₃ ) coincides with the ideal lens position 520 . x or X indicates the position of the focusing lens.

As described above, when the distance to the target object is determined by the BDAF method, it is desirable that the difference in the amount of blur between images to be compared is equal to or greater than a predetermined threshold value.

Therefore, the imaging control unit 110 included in the imaging apparatus 100 according to the present embodiment includes an acquisition unit 112 , a calculation unit 114 , a derivation unit 116 , and a focus control unit 140 as shown in FIG. 2 .

The acquiring unit 112 acquires a first image included in a first captured image captured in a state where the imaging plane and the lens are in a first positional relationship, and a second image captured in a state in which the imaging surface and the lens are in a second positional relationship. A second image included in the captured image. For example, as shown in FIG. 7 , the acquisition unit 112 acquires an image 601 from among the images 601 to 605 in the AF processing frame 610 of the captured image 600 captured with the imaging plane and the lens in the first positional relationship. Furthermore, the acquisition unit 112 acquires images 621 to 625 in the AF processing frame 630 of the captured image 620 captured after the positional relationship between the imaging surface and the lens has been changed by moving the focus lens or the imaging surface of the image sensor 120 . image 621. The acquisition unit 112 acquires an image of an area having a feature amount that satisfies a predetermined condition from among a plurality of areas within the AF processing frame. The acquisition unit 112 can acquire an image for each area when the feature amount of each of the plurality of areas within the AF processing frame satisfies a predetermined condition.

The calculation unit 114 calculates the blurring amount of each of the first image and the second image. The calculation unit 114 calculates, for example, the blurring amount C(t) of each of the image 601 and the image 621 . The focus control unit 140 controls the positions of the imaging plane and the lens based on the respective blur amounts of the first image and the second image when the difference between the blur amount of the first image and the blur amount of the second image is equal to or greater than the first threshold Th1 relation. The focus control unit 140 can control the positional relationship between the imaging plane and the lens by controlling the position of at least one of the imaging plane and the lens. The first threshold Th1 can be determined according to the specifications of the imaging device 100 . The first threshold Th1 may be determined based on the lens characteristics of the imaging device 100 . The first threshold Th1 may be determined based on the pixel pitch of the image sensor 120 . The focus control unit 140 is an example of a control unit that controls the positional relationship between the imaging plane and the lens.

The deriving unit 116 derives the first distance to the first target object included in the first image and the second image based on the blurring amount of the first image and the blurring amount of the second image. The deriving unit 116 can derive the distance to the target object 650 included in the image 601 and the image 621 based on the above-described equation (2) and the geometric relationship shown in FIG. 5 , for example. The focus control unit 140 can execute the AF process by moving the focus lens or the imaging surface of the image sensor 120 based on the distance to the target object 650 derived by the deriving unit 116 .

When the difference between the blurring amount of the first image and the blurring amount of the second image is smaller than the first threshold Th1, the acquiring unit 112 may further acquire a third image captured in a state where the imaging plane and the lens are in the third positional relationship. The third image included in the image. The acquisition unit 112 may acquire, for example, the image 661 included in the captured image 660 captured in the third positional relationship. The calculation section 114 may further calculate the blurring amount of the third image. The focus control unit 140 may control the imaging surface and the lens based on the respective blur amounts of the first image and the third image when the difference between the blur amount of the first image and the blur amount of the third image is equal to or greater than the first threshold Th1 positional relationship. When the difference between the blurring amount of the image 601 and the blurring amount of the image 661 is equal to or larger than the first threshold Th1, the focus control unit 140 may focus the imaging surface of the image 601 or the image sensor 120 on the basis of the blurring amounts of the image 601 and the image 661. Move to bring focus to target object 650 .

The acquiring unit 112 may further acquire the fourth image included in the first captured image and the fifth image included in the second captured image. The acquisition unit 112 can acquire, for example, the image 602 included in the captured image 600 and the image 622 included in the captured image 620 . The calculation unit 114 can calculate the blurring amount of each of the fourth image and the fifth image. The calculation unit 114 can calculate the blurring amount of each of the image 602 and the image 622, for example. The focus control unit 140 may be such that the difference between the blurring amount of the first image and the blurring amount of the second image is smaller than the first threshold value Th1, and the difference between the blurring amount of the fourth image and the blurring amount of the fifth image is equal to or greater than the first threshold value Th1. In this case, the positional relationship between the imaging plane and the lens is controlled based on the blurring amount of each of the fourth image and the fifth image. The deriving unit 116 may derive the second distance to the second target object included in the fourth image and the fifth image based on the blurring amount of the fourth image and the blurring amount of the fifth image. The deriving unit 116 can derive the distances to the image 602 and the target object 652 included in the image 622 based on the above-described equation (2) and the geometric relationship shown in FIG. 5 , for example. The focus control unit 140 may be, for example, when the difference between the blurring amount of the image 601 and the blurring amount of the image 621 is smaller than the first threshold value Th1, and the difference between the blurring amount of the image 602 and the blurring amount of the image 622 is the first threshold value Th1 or more. Next, the positional relationship between the imaging plane and the lens is controlled based on the blurring amounts of the image 602 and the image 622, respectively. The focus control unit 140 may move the focus lens or the imaging surface of the image sensor 120 to focus on the target object 652 based on the distance to the image 602 and the target object 652 included in the image 622 derived by the deriving unit 116 .

The acquisition unit 112 can acquire a fourth image included in the first captured image and adjacent to the first image and a fifth image included in the second captured image and adjacent to the second image. The acquisition unit 112 can acquire, for example, an image 602 included in the captured image 600 adjacent to the image 601 and an image 622 included in the captured image 620 adjacent to the image 621 . The calculation unit 114 can calculate the blurring amount of each of the fourth image and the fifth image. The calculation unit 114 can calculate the blurring amount of each of the image 602 and the image 622, for example.

The focus control unit 140 may be configured such that the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than the first threshold value Th1, and the difference between the blurring amount of the fourth image and the blurring amount of the fifth image is equal to or greater than the first threshold value Th1 In the case of , the positional relationship between the imaging plane and the lens is controlled based on the blurring amount of each of the first image, the second image, the fourth image, and the fifth image. For example, the focus control unit 140 may be configured such that the difference between the blurring amount of the image 601 and the blurring amount of the image 621 is equal to or greater than the first threshold value Th1, and the difference between the blurring amount of the image 602 and the blurring amount of the image 622 is equal to or greater than the first threshold value Th1. In this case, the positional relationship between the imaging plane and the lens is controlled based on the blurring amount of each of the image 601 , the image 621 , the image 602 , and the image 622 . The focus control unit 140 can control the distance between the image plane and the target object 650 according to the distance to the target object 650 determined based on the blurring amount of the image 601 and the image 621 , and the distance to the target object 652 determined based on the blurring amount of the image 602 and the image 622 . The positional relationship of the lens. The focus control unit 140 may determine the distance of the subject in the AF processing frame based on the weighted respective distances of the respective areas in accordance with the weights set in advance for the respective areas within the AF processing frame.

When the difference between the blurring amount of the fourth image and the blurring amount of the fifth image is smaller than the first threshold value Th1, the acquisition unit 112 may further acquire a third image captured in a state where the imaging plane and the lens are in the third positional relationship. The sixth image included in the image. The acquisition unit 112 may further acquire the image 662 included in the captured image 660 after further changing the positional relationship between the imaging plane and the lens from the second positional relationship to the third positional relationship, for example. The calculation section 114 may calculate the blurring amount of the sixth image. The calculation section 114 can calculate the blurring amount of the image 662 . The focus control unit 140 may be configured such that the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than the first threshold value Th1, and the difference between the blurring amount of the fourth image and the blurring amount of the sixth image is equal to or greater than the first threshold value Th1 In the case of , the positional relationship between the imaging plane and the lens is controlled based on the respective blurring amounts of the first image, the second image, the fourth image, and the sixth image. The focus control unit 140 may, when the difference between the blurring amount of the image 601 and the blurring amount of the image 621 is equal to or greater than the first threshold value Th1 and the difference between the blurring amount of the image 602 and the blurring amount of the image 662 is equal to or greater than the first threshold value Th1 The positional relationship between the imaging plane and the lens is controlled based on the blurring amount of each of the image 601 , the image 621 , the image 602 , and the image 662 .

The focus control unit 140 may satisfy the predetermined imaging condition when the first distance derived by the deriving unit 116 satisfies the predetermined imaging condition, and the second distance derived by the deriving unit 116 does not satisfy the predetermined imaging condition, and the amount of blur of the first image and the second image may be different. When the difference between the blur amounts is equal to or greater than the first threshold, the positional relationship between the imaging plane and the lens is controlled based on the first distance. The imaging conditions may be conditions determined in accordance with imaging modes such as portrait mode and landscape mode. The imaging conditions can be conditions such as extreme near-end priority, infinite far-end priority, and the like. If the photographing condition is extremely close-up priority, the focus control unit 140 may determine the image of the area with the shortest distance to the target object from among the distances to the target object in each area within the AF processing frame derived by the deriving unit 116 as the image For images that satisfy the imaging conditions, images in other AF processing frames are determined to be images that do not satisfy the imaging conditions. Here, when the first distance satisfies the imaging conditions, the second distance does not satisfy the imaging conditions, and the difference between the blurring amount of the first image and the blurring amount of the second image is smaller than the first threshold Th1, the acquiring unit 112 may further acquire A sixth image included in a third captured image captured in a state where the imaging plane and the lens are in the third positional relationship. The acquisition unit 112 can acquire the image 661 from the captured image 660 , for example. The focus control unit 140 may control the imaging plane based on the blur amount of the first image and the blur amount of the sixth image when the difference between the blur amount of the first image and the blur amount of the sixth image is equal to or greater than the first threshold value. The positional relationship with the lens. The deriving unit 116 can derive the distance to the image 601 and the target object 650 included in the image 661 based on the blurring amount of the image 601 and the blurring amount of the image 661 . The focus control section 140 may move the focus lens or the image plane of the image sensor 120 based on the distance to the target object 650 derived by the deriving section 116 so that the focus is on the target object 650 .

For example, as shown in FIG. 8 , the calculation unit 114 calculates the blur amount C(t ₀ ) of the image 601 , the blur amount C(t ₁ ) of the image 621 , the blur amount C′(t ₀ ) of the image 602 , and the blur of the image 622 . Quantity C'(t ₁ ). The difference Ta between the blur amount C(t ₀ ) of the image 601 and the blur amount C(t ₁ ) of the image 621 at this time is smaller than the first threshold value Th1. On the other hand, the difference Tb between the blur amount C′(t ₀ ) of the image 602 and the blur amount C′(t ₁ ) of the image 621 is equal to or larger than the first threshold Th1 . If the imaging condition in this case is infinity priority, the focus control unit 140 can determine to focus on the image 602 and the target object 652 included in the image 622 . On the other hand, if the imaging condition is the extreme near-end priority, the focus control unit 140 may determine that the image 601 and the target object 650 included in the image 621 are brought into focus. Here, since the difference Tb is equal to or greater than the first threshold Th1, the curve 700 determined based on the blur amount C'(t ₀ ) of the image 602 and the blur amount C'(t ₁ ) of the image 622 has high accuracy. Therefore, the focus control unit 140 can execute the AF process with high accuracy by moving the focus lens or the imaging surface of the image sensor 120 based on the lens position 712 determined from the curve 700 . On the other hand, since the difference Ta is smaller than the first threshold Th1, the accuracy of the curve 702 determined based on the blur amount C(t ₀ ) of the image 601 and the blur amount C(t ₁ ) of the image 621 is low. Therefore, if the focus control unit 140 moves the focus lens or the imaging surface of the image sensor 120 according to the lens position 714 determined based on the curve 702 in the case of the extreme near end priority, the AF process cannot be executed with high accuracy. Therefore, the imaging apparatus 100 further moves the imaging surface of the focus lens or the image sensor 120 to capture the captured image 660 . Then, the calculation unit 114 calculates the blur amount C(t ₂ ) of the image 661 of the captured image 660 . The difference Tc between the blur amount C(t _{0 ) of the image 601 and the blur amount C(t 2 )} of the image 661 is equal to or greater than the first threshold Th1. Therefore, the curve 704 determined based on the blur amount C(t ₀ ) of the image 601 and the blur amount C(t ₂ ) of the image 661 is more accurate than the curve 702 . Thereby, the focus control unit 140 can execute the AF process with high accuracy even in the case of extremely near-end priority by moving the focus lens or the imaging surface of the image sensor 120 based on the lens position 716 determined from the curve 704 .

FIG. 9 is a flowchart showing an example of the procedure of AF processing in the BDAF method.

The imaging device 100 moves the focus lens to the X(t ₀ ) position (S201). In the imaging device 100, for example, the focus lens is moved by 10 μm in the optical axis direction. The acquiring unit 112 acquires the image I(t _{0 ) from the captured image captured at the position X(t 0 )} ( S202 ). The imaging control unit 110 increments the counter ( S203 ). Next, the imaging control unit 110 moves the focus lens to the X(t _n ) position via the lens control unit 220 ( S204 ). The acquiring unit 112 acquires the image I(t _n ) at the position corresponding to the image I(t ₀ ) from the captured images captured at the position X(t _n ) ( S205 ). The calculation unit 114 calculates the blur amount C(t _{0 ) of the image I(t 0 )} and the blur amount C(t _n ) of the image I(t _n ) ( S206 ). If the difference |C(t _n )−C(t ₀ )| between the blur amount C(t ₀ ) and the blur amount C(t _n ) is smaller than the first threshold value Th1, the imaging control unit 110 repeats the operation in order to further move the focus lens Processes after step S203.

If the difference |C(t _n )−C(t ₀ )| is equal to or larger than the first threshold Th1, the deriving unit 116 derives the image I(t ₀ based on the blur amount C(t ₀ ) and the blur amount C(t _n ) ) and the distance of the target object contained in the image I(t _n ). The focus control unit 140 determines the distance to the subject based on the distance ( S208 ). The focus control unit 140 moves the focus lens to the predicted focus position based on the determined distance ( S209 ).

As described above, when the difference in blur amount between images is small, the imaging apparatus 100 moves the focus lens until the difference in blur amount between images becomes equal to or greater than the first threshold Th1. Therefore, the imaging apparatus 100 can execute the AF process based on the BDAF method with higher accuracy and high speed.

FIG. 10 is a flowchart showing another example of the procedure of the AF process in the BDAF method. The procedure shown in FIG. 10 can be applied to the case of the shooting mode in which the speed of the AF processing is prioritized.

The imaging apparatus 100 moves the focus lens to the X(t ₀ ) position (S301). In the imaging device 100, for example, the focus lens is moved by 10 μm in the optical axis direction. The acquisition unit 112 acquires a plurality of images within the AF processing frame set for the captured image captured at the position X(t ₀ ). The acquisition unit 112 divides the AF processing frame into a plurality of areas, and acquires an image for each area. The acquisition unit 112 calculates the feature amount of each of the plurality of images ( S302 ). The acquisition unit 112 can calculate the feature amount based on pixel values, luminance values, edge detection, and the like of each of the plurality of images. If there is no image having a feature amount equal to or greater than the threshold value among the plurality of images, the AF process by the BDAF method is not executed, and the process ends.

On the other hand, if there are images having a feature amount equal to or greater than the threshold value ( S303 ), the acquisition unit 112 acquires these images I(t ₀ ) ( S304 ). Next, the imaging control unit 110 increments the counter ( S305 ). The imaging control unit 110 moves the focus lens to the X(t _n ) position via the lens control unit 220 ( S306 ). The acquiring unit 112 acquires each image I(t _n ) at a position corresponding to each image I(t ₀ ) from the captured images captured at the position X(t _n ) ( S307 ). The calculation unit 114 calculates the blur amount C(t ₀ ) of each image I(t ₀ ) and the blur amount C(t _n ) of each image I(t _n ) ( S308 ). If there is no image where the difference |C(t _n )−C(t ₀ )| between the blur amount C(t ₀ ) and the blur amount C(t _n ) is equal to or larger than the first threshold Th1, the imaging control unit 110 repeats step S305 later processing.

If there is an image whose difference |C(t _n )−C(t ₀ )| is equal to or greater than the first threshold Th1, the deriving unit 116 calculates the blurring amount C(t _{0 ) of the corresponding image I(t 0 )} and the image I( The blur amount C(t _n ) of t _n ), derived from the distance to the image I(t ₀ ) and the target object contained in the image I(t _n ). The focus control unit 140 determines the distance to the subject based on the distance derived by the deriving unit 116, and moves the focus lens to the predicted focus position (S311).

According to the above processing, the focus control unit 140 can immediately move the focus lens to the predicted focus position at the stage where the difference in blur amount is equal to or greater than the first threshold value from among a plurality of images in the AF processing frame.

FIG. 11 is a flowchart showing another example of the procedure of the AF process in the BDAF method. The procedure shown in FIG. 11 can be applied to a multi-point AF mode in which weights are set for each area within the AF processing frame.

The imaging device 100 moves the focus lens to the X(t ₀ ) position (S401). In the imaging device 100, for example, the focus lens is moved by 10 μm in the optical axis direction. The acquisition unit 112 acquires a plurality of images within the AF processing frame set for the captured image captured at the position X(t ₀ ). The acquisition unit 112 divides the AF processing frame into a plurality of areas, and acquires an image for each area. The acquisition unit 112 calculates the feature amount of each of the plurality of images ( S402 ). If there is no image having a feature amount equal to or greater than the threshold value among the plurality of images, the AF process by the BDAF method is not executed, and the process ends.

On the other hand, if there are images having a feature amount equal to or greater than the threshold value ( S403 ), the acquisition unit 112 acquires these images I(t ₀ ) ( S404 ). Next, the imaging control unit 110 increments the counter ( S405 ). The imaging control unit 110 moves the focus lens to the X(t _n ) position via the lens control unit 220 ( S406 ). The acquiring unit 112 acquires each image I(t _n ) at a position corresponding to each image I(t ₀ ) from the captured images captured at the position X(t _n ) ( S407 ). The calculation unit 114 calculates the blur amount C(t ₀ ) of each image I(t ₀ ) and the blur amount C(t _n ) of each image I(t _n ) ( S408 ).

If there is no image where the difference |C(t _n )−C(t ₀ )| is greater than or equal to the first threshold Th1 , the imaging control unit 110 obtains the position of the lens and the imaging surface in order to move the focus lens via the lens control unit 220 The processing from step S405 onward is repeated for the captured images with different relationships.

If there is an image whose difference |C(t _n )−C(t ₀ )| is greater than or equal to the first threshold Th1, the blur amount C(t _{0 ) based on the corresponding image I(t 0 )} and the image I(t _n ) The blur amount C(t _n ) of , is derived to the image I(t ₀ ) and the distance of the target object contained in the image I(t _n ) ( S410 ).

If the distance to the target object is not derived for all the images having the feature value greater than or equal to the threshold value, the imaging control unit 110 moves the focus lens via the lens control unit 220 to acquire captured images in which the positional relationship between the lens and the imaging surface is different. On the other hand, the processing after step S405 is repeated.

When the distances to the target object are derived for all the images having the feature amount equal to or greater than the threshold value, the focus control unit 140 determines the distances to the object based on these distances ( S410 ). For example, in the case of the AF processing block 610 shown in FIG. 7 , the areas of the respective images within the AF processing block 610 may be individually weighted. A weight of "100" may be set for the area of the image 601 in the center of the AF processing frame 610 , a weight of "70" may be set for the image 602 and the image 603 adjacent to the left and right of the image 601 , and the upper side of the image 601 is adjacent to the weight. A weight of "70" is set for the image 604 of the image 601, and a weight of "50" is set for the image 605 adjacent to the image 601 below. Furthermore, the focus control unit 140 may weight the distances to the target object for the respective areas within the AF processing frame 610 by weighting each area, and determine the distance to the subject based on the weighted distances. The focus control unit 140 moves the focus lens to the predicted focus position based on the determined distance to the subject ( S413 ).

As described above, when calculating distances with respect to a plurality of regions, an image in which the difference between the blurring amounts is equal to or greater than the first threshold value is obtained for each of the plurality of regions. As a result, it is possible to prevent variations in the accuracy of the distances derived between the plurality of regions.

FIG. 12 is a flowchart showing another example of the procedure of the AF process in the BDAF method. The process shown in FIG. 12 can be applied to camera modes such as infinity-end first or very near-end first.

The imaging apparatus 100 moves the focus lens to the X(t ₀ ) position (S501). In the imaging device 100, for example, the focus lens is moved by 10 μm in the optical axis direction. The acquisition unit 112 acquires a plurality of images within the AF processing frame set for the captured image captured at the position X(t ₀ ). The acquisition unit 112 divides the AF processing frame into a plurality of areas, and acquires an image for each area. The acquisition unit 112 calculates the feature amount of each of the plurality of images ( S502 ). If there is no image having a feature amount equal to or greater than the threshold value among the plurality of images, the AF process by the BDAF method is not executed, and the process ends.

On the other hand, if there are images having a feature amount equal to or greater than the threshold value ( S503 ), the acquisition unit 112 acquires these images I(t ₀ ) ( S504 ). Next, the imaging control unit 110 increments the counter ( S505 ). The imaging control unit 110 moves the focus lens to the X(t _n ) position via the lens control unit 220 ( S506 ). The acquiring unit 112 acquires each image I(t _n ) at a position corresponding to each image I(t ₀ ) from the captured images captured at the position X(t _n ) ( S507 ). The calculation unit 114 calculates the blur amount C(t ₀ ) of each image I(t ₀ ) and the blur amount C(t _n ) of each image I(t _n ) ( S508 ). The deriving unit 116 _derives each image I _{(t 0 ) and each} image I( The distance of each target object contained in t _n ) (S509).

The focus control unit 140 selects the image I(t ₀ ) and the image (t _n ) that satisfy the imaging conditions from the calculated distances ( S510 ). For example, if the extreme near end is prioritized, the focus control unit 140 selects the image I with the shortest distance from the calculated distances. If infinity is given priority, the focus control unit 140 selects the image I with the longest distance from the calculated distances.

The focus control unit 140 determines the difference between the blur amount C(t ₀ ) of the selected image I(t ₀ ) and the blur amount C(t _n ) of the image (t _n ) |C(t _n )−C(t ₀ ) |Whether it is equal to or greater than the first threshold Th1 (S511). If the difference |C(t _n )−C(t ₀ )| is equal to or greater than the first threshold Th1, the focus control unit 140 includes the image I(t ₀ ) and the image I(t _n ) based on the value derived by the deriving unit 116 The distance to the subject is determined based on the distance of the target object, and the focus lens is moved to the predicted focus position (S516).

Regarding the focus control unit 140, if the difference between the blur amount C(t ₀ ) of the selected image I(t ₀ ) and the blur amount C(t _n ) of the image (t _n ) |C(t _n )−C(t ₀ )| is less than the first threshold Th1, the imaging control unit 110 increments the counter (S512). Next, the imaging control unit 110 moves the focus lens to the X(t _n ) position via the lens control unit 220 ( S513 ). The acquiring unit 112 acquires the image I(t _n ) at the position corresponding to the image I(t ₀ ) selected in step S510 from the captured images captured at the position X(t _n ) ( S514 ). The calculation unit 114 calculates the blur amount C(t _{n ) of the acquired image I(t n )} ( S515 ).

The imaging control unit 110 repeats the process of step S511 until an image (t _n ) in which the difference between the blur amount and the blur amount C(t ₀ ) of the selected image I(t ₀ ) is equal to or greater than the first threshold Th1 is obtained. Then, when the difference |C(t _n )−C(t ₀ )| is obtained as an image (t _n ) in which the difference |C(t n )−C(t 0 )| is equal to or greater than the first threshold value Th1 , the focus control unit 140 obtains an image I(t 0 ) based on the image I(t ₀ ) derived by the deriving unit 116 and the distance to the target object included in the image I(t _n ) to determine the distance to the subject, and move the focus lens to the predicted focus position (S516).

As described above, even when the AF processing is performed in the imaging mode such as infinity priority or extreme near priority, the AF processing of the BDAF method can be performed with high precision.

However, for example, when imaging is performed by the imaging device 100 mounted on a moving object such as the UAV 10, the target object in the captured image may move in the captured image. If the movement amount of the target object is large, the imaging control unit 110 may not be able to execute the AF process with high accuracy in the BDAF method.

Therefore, the imaging control unit 110 can execute the AF process of the BDAF method in consideration of the movement amount of the target object in the image.

FIG. 13 shows an example of functional modules of the UAV 10 according to another embodiment. The UAV 10 shown in FIG. 13 is different from the UAV 10 shown in FIG. 2 in that the imaging control unit 110 has a determination unit 113 .

The determination unit 113 compares the feature points included in the first image in the first captured image captured in the state where the lens and the imaging plane are in the first positional relationship with the image in the state where the lens and the imaging plane are in the second positional relationship. The obtained feature points of the second captured image are compared, so as to determine an area of the second captured image corresponding to the first image. When the difference between the position of the first image in the first captured image and the position of the region in the second captured image is equal to or less than the second threshold Th2, the obtaining unit 112 obtains the second captured image at a distance from the first captured image. An image of the region of the second captured image having the same positional relationship as the first image is used as the second image.

When the difference between the position of the first image in the first captured image and the position of the region in the second captured image is larger than the second threshold value, the obtaining unit 112 may obtain the region as the second image. When the difference between the position of the first image in the first captured image and the position of the region in the second captured image is larger than the second threshold Th2 and equal to or less than the third threshold Th3, the acquiring unit 112 may acquire the image of the region as the difference. second image. The second threshold Th2 and the third threshold Th3 can be determined based on the pixel pitch of the image sensor 120 or the like. The determination unit 113 may divide the image into a plurality of blocks, and may search for feature points on a block-by-block basis. Therefore, the second threshold Th2 and the third threshold Th3 can be determined based on the size of the block in which the feature points are searched.

The determination unit 113 determines feature points 820 from the first image 812 of the AF processing frame 810 in the first captured image 800 , for example, as shown in FIG. 14 . The determination section 113 may determine the feature points 820 based on pixel values, brightness, edge detection, and the like. Next, a feature point 821 corresponding to the feature point 820 is specified from the second captured image 802 captured in a state where the positional relationship between the lens and the imaging plane has been changed. The determination unit 113 determines a region 814 of the second captured image 802 corresponding to the first image 812 by comparing the feature points 820 of the first captured image 800 with the feature points 821 of the second captured image 802 . When the difference between the position of the first image 812 in the first captured image 800 and the position of the region 814 in the second captured image 802 is equal to or less than the second threshold Th2, the obtaining unit 112 obtains the second captured image 802 in the An image of the region 816 of the second captured image 802 in the same positional relationship as the first captured image 800 and the first image 812 is used as the second image. That is, the focus control unit 140 does not move the AF processing frame 811 in the second captured image 802 relative to the AF processing frame 810 in the first captured image 800 , but executes the AF processing of the BDAF method.

On the other hand, it is assumed that the specifying unit 113 specifies the feature point 822 corresponding to the feature point 820 from the third captured image 804 captured with the positional relationship between the lens and the imaging plane changed. In this case, the determination unit 113 determines the region 816 of the third captured image 804 corresponding to the first image 812 by comparing the feature points 820 of the first captured image 800 and the feature points 822 of the third captured image 804 . The acquisition unit 112 determines that the difference between the position of the first image 812 in the first captured image 800 and the position of the region 816 in the third captured image 804 is larger than the second threshold Th2, and acquires the region 816 as the second image. That is, the focus control unit 140 moves the AF processing frame 810 in the third captured image 804 relative to the AF processing frame 810 of the first captured image 800 by an amount corresponding to the amount of movement of the feature points corresponding to the area 816, and uses the moved The AF processing block 810 executes the AF processing of the BDAF method.

In this way, when the target object to be compared has moved within the image within the allowable range, the position of the image to be compared is changed, and the calculation unit 114 calculates the blurring amount from the changed image. Therefore, even if the target object moves within the image, the AF process in the BDAF method can be executed with high accuracy.

The feature points identified by the identifying unit 113 may be identified based on the center of gravity of the brightness in the image. For example, as shown in FIG. 15A , the target object 902 is included in the first captured image 900 . The determination unit 113 divides the first captured image 900 into a plurality of blocks (eg, 8×8 pixels), calculates the luminance for each block, and generates a monochromatic image 901 representing the luminance on a block-by-block basis. The determination unit 113 determines the position of the center of gravity 903 of luminance from the monochrome image 901 . Similarly, the determination unit 113 divides the second captured image 910 into a plurality of blocks, calculates the luminance for each block, and generates a monochrome image 911 . Furthermore, the specifying unit 113 specifies the position of the center of gravity 913 of the luminance from the monochrome image 911 . The determination unit 113 determines that the center of gravity of the luminance has moved from the position of the center of gravity 903 to the position of the center of gravity 913 . If the shift amount of the center of gravity of luminance is within the range of one block, that is, the shift amount of the center of gravity of luminance is within 8×8 pixels, the determination unit 113 may determine that the shift amount of the feature point is within the second threshold. On the other hand, if the movement amount of the center of gravity is within the range of two blocks, the determination unit 113 may determine that the movement amount of the feature point is within the third threshold value. Then, as shown in FIG. 15B , the acquiring unit 112 acquires not the image of the region 931 but the image of the region 932 as the second image acquired from the second captured image 910 with respect to the first image 930 of the first captured image 900 . image. Thereby, the influence of the movement of the target objects 902 and 912 between the images can be avoided.

FIG. 16 is a flowchart showing an example of a procedure of moving the AF processing frame in accordance with the movement amount of the target object.

The acquisition unit 112 acquires the image I(t ₀ ) from the first captured image captured in a state where the lens and the imaging plane are in the first positional relationship ( S601 ). Next, the acquiring unit 112 acquires the image I(t ₁ ) at the position corresponding to the image I(t ₀ ) from the second captured image captured in the state where the lens and the imaging plane are in the second positional relationship ( S602 ) . The determination unit 113 calculates the movement amount X of the target object S in the image I(t ₁ ) with respect to the target object S in the image I(t ₀ ) ( S603 ). As described above, the determination unit 113 can calculate the movement amount X by specifying the feature points based on pixel values, brightness, and the like, and comparing the positions of the feature points with each other.

Next, the focus control unit 140 determines whether or not the movement amount X is equal to or less than the second threshold Th2 ( S604 ). The focus control unit 140 can determine whether or not the feature point of the second captured image exists in the block corresponding to the position of the feature point of the first captured image. For example, the focus control unit 140 can determine whether or not the movement amount X is within 8×8 pixels. If the movement amount X is equal to or smaller than the second threshold Th2, the focus control unit 140 does not move the AF processing frame for the second captured image, and executes the distance based on the blurring amount using the image I(t ₁ ) acquired in step S602 Calculation processing (S605). That is, the focus control unit 140 determines the distance to the target object S based on the blurring amount of the image I(t ₀ ) and the blurring amount of the image I(t ₁ ). Then, the focus control unit 140 moves the focus lens based on the distance to the target object S. FIG.

On the other hand, when the movement amount X is larger than the second threshold value Th2, the focus control unit 140 determines whether or not the movement amount X is equal to or less than the third threshold value Th3 (S606). The focus control unit 140 may determine whether or not the feature point of the second captured image exists in a block adjacent to the block corresponding to the position of the feature point of the first captured image. For example, the focus control unit 140 can determine whether or not the movement amount is within 24×24 pixels. When the movement amount X is equal to or less than the third threshold Th3, the focus control unit 140 moves the AF processing frame of the comparison target image I(t ₁ ) by the movement amount X ( S607 ). Next, the focus control unit 140 determines the distance to the target object S based on the blur amount of the image I(t ₁ ) and the blur amount of the image I(t ₀ ) acquired from the moved AF processing frame ( S605 ). Then, the focus control unit 140 moves the focus lens based on the distance to the target object S. FIG.

As described above, by adjusting the position of the image of the comparison object in consideration of the movement of the target object in the image, the AF process of the BDAF method can be executed with higher accuracy.

FIG. 17 shows an example of a computer 1200 that can embody various aspects of the present invention in whole or in part. A program installed in the computer 1200 can cause the computer 1200 to function as an operation associated with the apparatus according to the embodiment of the present invention or as one or more "sections" of the apparatus. Alternatively, the program can cause computer 1200 to perform the operation or the one or more "sections." The program enables the computer 1200 to execute the process or stages of the process involved in the embodiments of the present invention. Such a program can be executed by the CPU 1212 in order to cause the computer 1200 to perform specific operations associated with some or all of the flowcharts and blocks in the block diagrams described in this specification.

The computer 1200 according to the present embodiment includes a CPU 1212 and a RAM 1214 , which are connected to each other via the host controller 1210 . The computer 1200 also includes a communication interface 1222, an input/output unit, which is connected to the host controller 1210 via the input/output controller 1220. Computer 1200 also includes ROM 1230 . The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

The communication interface 1222 communicates with other electronic devices via a network. The hard disk drive can store programs and data used by the CPU 1212 in the computer 1200 . The ROM 1230 stores therein a boot program and the like executed by the computer 1200 at the time of startup, and/or programs depending on the hardware of the computer 1200 . The program is provided via a computer-readable recording medium such as a CR-ROM, a USB memory, or an IC card, or a network. The program is installed in the RAM 1214 or the ROM 1230, which are also examples of a computer-readable recording medium, and is executed by the CPU 1212. The information processing described in these programs is read by the computer 1200, and cooperation between the programs and the various types of hardware resources described above is realized. The apparatus or method may be constituted by implementing operations or processing of information in accordance with the use of the computer 1200 .

For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. The communication interface 1222 reads the transmission data stored in the transmission buffer provided in the recording medium such as the RAM 1214 or the USB memory under the control of the CPU 1212, and transmits the read transmission data to the network, or receives the transmission data from the network. The received received data is written to a receive buffer or the like provided on the recording medium.

In addition, the CPU 1212 can read all or necessary parts of a file or database stored in an external recording medium such as a USB memory into the RAM 1214 and execute various types of processing on the data on the RAM 1214 . The CPU 1212 may then write the processed data back to the external recording medium.

Various types of programs, data, tables, and various types of information such as databases are stored in the recording medium and can be processed by information. The CPU 1212 can execute various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information retrieval, and information processing described in various places in the present invention, including those specified by the instruction sequence commands of the program, with respect to the data read from the RAM 1214 . Various types of processing, including permutation, etc., are performed, and the results are written back to RAM 1214. In addition, the CPU 1212 can retrieve information in files, databases, and the like within the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 may retrieve the attribute specifying the first attribute from the plurality of entries For an entry whose value conditions match, the attribute value of the second attribute stored in the entry is read, thereby acquiring the attribute value of the second attribute associated with the first attribute satisfying the predetermined condition.

The programs or software modules described above may be stored in computer-readable storage media on or near the computer 1200 . Also, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby supplying the program to the computer 1200 through the network.

It should be noted that the execution order of each process, such as actions, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is not expressly stated as "preceding". , "before", etc., and the output of the preceding processing is not used in the subsequent processing, and can be realized in any order. Even if the operation flow in the claims, the description, and the drawings is described using "first", "next", etc. for convenience, it does not necessarily mean that it must be carried out in this order.

Description of reference numerals

10 UAVs

20 UAV bodies

30 UAV Control Section

32 memory

34 Communication interface

40 Propulsion Department

41 GPS receivers

42 Inertial measurement devices

43 Magnetic Compass

44 Barometric altimeter

45 Temperature sensor

500 gimbal bracket

60 Cameras

100 Cameras

102 Camera Department

110 Camera Control Section

112 Acquisition Department

113 Determination Department

114 Computing Department

116 Export section

120 image sensors

130 memory

140 Focus Control Section

200 lenses

210 Lens

212 Lens moving mechanism

220 Lens Control Section

300 Remote Control Unit

1200 computers

1210 Host Controller

1212 CPUs

1214 RAM

1220 Input/Output Controller

1222 communication interface

1230 ROMs.

Claims (19)

1. A camera control device, comprising: an acquisition unit that acquires a first image included in a first captured image captured in a state where the imaging plane and the lens are in a first positional relationship, and a second image included in the second image a second captured image captured when the imaging surface and the lens are in a second positional relationship; a calculation section that calculates the blur amount of each of the first image and the second image; and a control unit configured to, when the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than a first threshold value, based on the blurring amounts of the first image and the second image, respectively, to control the positional relationship between the camera surface and the lens; The acquisition unit further acquires a fourth image included in the first captured image and a fifth image included in the second captured image, The calculation unit further calculates the blurring amount of the fourth image and the fifth image, respectively, The camera control device further includes: a derivation unit that derives a first distance to a first target object included in the first image and the second image based on the blurring amounts of the first image and the second image, and based on the first image The respective blur amounts of the fourth image and the fifth image are derived from the second distance to the fourth image and the second target object contained in the fifth image, The control unit satisfies a predetermined imaging condition at the first distance, does not satisfy the imaging condition at the second distance, and the difference between the blurring amount of the first image and the blurring amount of the second image is: When the value is equal to or greater than the first threshold value, the positional relationship between the imaging plane and the lens is controlled based on the first distance. 2. The camera control device according to claim 1, wherein: The acquisition unit further acquires a third image included in the a third captured image captured when the imaging surface and the lens are in a third positional relationship, The calculating part further calculates the blurring amount of the third image, The control unit, when the difference between the blurring amount of the first image and the blurring amount of the third image is equal to or greater than the first threshold value, based on the blurring of each of the first image and the third image to control the positional relationship between the imaging surface and the lens. 3. The camera control device according to claim 1, wherein: The acquisition unit further acquires a fourth image included in the first captured image and a fifth image included in the second captured image, The calculation unit further calculates the blurring amount of the fourth image and the fifth image, respectively, The control unit is configured when the difference between the blurring amount of the first image and the blurring amount of the second image is smaller than the first threshold value and the difference between the blurring amount of the fourth image and the blurring amount of the fifth image is When the difference is equal to or larger than the first threshold value, the positional relationship between the imaging plane and the lens is controlled based on the blurring amount of each of the fourth image and the fifth image. 4. The camera control device according to claim 1, wherein: The acquiring unit further acquires a fourth image included in the first captured image that is adjacent to the first image and a fifth image included in the second captured image that is adjacent to the second image, The calculation unit further calculates the blurring amount of the fourth image and the fifth image, respectively, the control unit, when the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than the first threshold value, and the blurring amount of the fourth image and the blurring amount of the fifth image When the difference is equal to or greater than the first threshold value, the image pickup plane and the Describe the positional relationship of the lens. 5. The imaging control device according to claim 4, further comprising: a deriving section that derives a first distance to the first image and a first target object contained in the second image based on the blurring amount of the first image and the blurring amount of the second image, and based on the blurring amount of the fourth image and the blurring amount of the fifth image, the second distance derived to the fourth image and the second target object contained in the fifth image, The control unit controls the positional relationship between the imaging plane and the lens based on the first distance and the second distance. 6. The camera control device according to claim 4, wherein: The acquisition unit further acquires a sixth image, which is included in the a third captured image obtained when the imaging surface and the lens are in a third positional relationship, The calculating part further calculates the blurring amount of the sixth image, The control unit is configured to, when the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than the first threshold value, and the blurring amount of the fourth image and the blurring amount of the sixth image When the difference is equal to or greater than the first threshold value, the image pickup plane and the Describe the positional relationship of the lens. 7. The camera control device according to claim 1, wherein: The acquisition unit satisfies the imaging condition at the first distance, does not satisfy the imaging condition at the second distance, and the difference between the blurring amount of the first image and the blurring amount of the second image is smaller than the predetermined distance. In the case of the first threshold value, a sixth image included in a third captured image captured in a state where the imaging plane and the lens are in a third positional relationship is further acquired, The calculating part further calculates the blurring amount of the sixth image, When the difference between the blurring amount of the first image and the blurring amount of the sixth image is equal to or greater than the first threshold value, the control unit is based on the blurring amount of the first image and the sixth image The respective blur amounts are used to control the positional relationship between the imaging plane and the lens. 8. The camera control device according to claim 1, further comprising: a determination unit that determines an area of the second captured image corresponding to the first image by comparing the feature points included in the first image with the feature points of the second captured image, When the difference between the position of the first image in the first captured image and the position of the region in the second captured image is equal to or less than a second threshold For the captured image, an image of the region of the second captured image in the same positional relationship as the first captured image and the first image is obtained as the second image. 9. The camera control device according to claim 8, wherein: The acquisition unit acquires the region when the difference between the position of the first image in the first captured image and the position of the region in the second captured image is larger than the second threshold value as the second image. 10. The camera control device according to claim 8, wherein: When the difference between the position of the first image in the first captured image and the position of the region in the second captured image is larger than the second threshold and equal to or less than a third threshold by the acquisition unit Next, an image of the area is acquired as the second image. 11. The camera control device according to claim 8, wherein: The specifying unit specifies the respective feature points based on the luminance of the first image and the luminance of the region of the second captured image. 12. The camera control device according to claim 11, wherein: The determining unit determines the center of gravity of the brightness of the first image as the feature point included in the first image, and determines the center of gravity of the brightness of the region of the second captured image as the second the feature points of the captured image. 13. The camera control device according to claim 1, wherein: By changing the position of the focus lens included in the lens, a state in which the imaging surface and the lens are in the first positional relationship is changed to a state in which they are in the second positional relationship. 14. The camera control device according to claim 1, wherein, By changing the position of the imaging plane, the imaging plane and the lens are changed from a state in which the imaging plane and the lens are in the first positional relationship to a state in which they are in the second positional relationship. 15. A camera device, comprising: The camera control device according to any one of claims 1 to 14; an image sensor having the imaging surface; and the lens. 16. A camera system, comprising: The camera device of claim 15; and A support mechanism supports the imaging device. 17 . A moving body which is mounted and moved with the imaging system according to claim 16 . 18 . 18. A camera control method, comprising: In the stage of acquiring a first image and a second image, the first image is included in the first captured image captured in a state where the imaging plane and the lens are in the first positional relationship, and the second image is included in the captured image. a second captured image captured when the surface and the lens are in a second positional relationship; a stage of calculating the respective blur amounts of the first image and the second image; When the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than a first threshold value, controlling the blurring amount of the first image and the second image based on the respective blurring amounts the stage of the positional relationship between the imaging plane and the lens; obtaining a fourth image included in the first captured image and a fifth image included in the second captured image; calculating the respective blurring amounts of the fourth image and the fifth image; A first distance to a first target object contained in the first image and the second image is derived based on the respective blur amounts of the first image and the second image, and based on the fourth image and the the respective blurring amounts of the fifth image, derived from the fourth image and the second distance of the second target object contained in the fifth image; When the first distance satisfies a predetermined imaging condition, the second distance does not satisfy the imaging condition, and the difference between the blurring amount of the first image and the blurring amount of the second image is the first When the threshold value is greater than or equal to the threshold value, the positional relationship between the imaging plane and the lens is controlled based on the first distance. 19. A computer-readable storage medium storing a computer program for implementing the following stages when the computer program is executed by a processor: In the stage of acquiring a first image and a second image, the first image is included in the first captured image captured in a state where the imaging plane and the lens are in the first positional relationship, and the second image is included in the captured image. a second captured image captured when the surface and the lens are in a second positional relationship; a stage of calculating the respective blur amounts of the first image and the second image; When the difference between the blurring amount of the first image and the blurring amount of the second image is equal to or greater than a first threshold value, controlling the blurring amount of the first image and the second image based on the respective blurring amounts the stage of the positional relationship between the imaging plane and the lens; a stage of acquiring a fourth image included in the first captured image and a fifth image included in the second captured image; a stage of calculating the blur amount of each of the fourth image and the fifth image; A first distance to a first target object contained in the first image and the second image is derived based on the respective blur amounts of the first image and the second image, and based on the fourth image and the The respective blur amounts of the fifth image are derived to the stage of the second distance of the fourth image and the second target object contained in the fifth image; When the first distance satisfies a predetermined imaging condition, the second distance does not satisfy the imaging condition, and the difference between the blurring amount of the first image and the blurring amount of the second image is the first When the threshold value is greater than or equal to the threshold value, the stage of the positional relationship between the imaging plane and the lens is controlled based on the first distance.

Images