JP2012015642A - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To facilitate accurate and easy setting of a subject-of-interest to a tracking target.SOLUTION: An imaging device has a function of selecting and indicating a tracking target by using a touch panel. Under the state that motion pictures of subjects SUBto SUBare displayed, the touch panel accepts a user's specified position input operation for specifying a tracking target. A tracking target setting unit corrects a specified position (321) as a touch position to a position (322) shorter from a subject by using a distance image 330 representing subject distance information as an image, and sets the subject at the corrected specified position to a tracking target. Thereafter, the tracking processing of the tracking target based on an image feature amount is executed.

Description

  The present invention relates to an imaging apparatus such as a digital camera, and more particularly to an imaging apparatus having a function of tracking a subject.

  Various methods for tracking a tracking target on a moving image using image analysis have been proposed. As a method for realizing the tracking process, there is a pattern matching method (for example, refer to Patent Document 1 below) for searching for the position of a pattern in an image after setting a tracking pattern, or a method for tracking an image feature such as a color (for example, the following Patent Document). 2). In addition, after calculating a template image as a tracking pattern, a method of calculating similarity by using a subject distance measurement result when calculating similarity by template matching has been proposed (for example, see Patent Document 1 below). ).

  Incidentally, the shooting range of the digital camera often includes various subjects (unspecified persons, animals, plants, etc.) in addition to the subject of interest of the user (for example, the user's child). Therefore, in order to track a user's target subject as a tracking target on a moving image, first, an operation for designating the target subject to be the tracking target is required. A method for realizing this operation using a touch panel has also been proposed (see, for example, Patent Document 3 below). In this method, when the user touches the face area of the attention person on the monitor, the attention person is set as a tracking target.

  FIG. 18A shows the state of the monitor when this method is realized. On the display screen 901 of the monitor, a shooting moving image of the digital camera is displayed, and the persons 911 to 913 are moving every moment on the displayed moving image. When the person (photographer) 's attention person is the person 911, as shown in FIG. 18B, the person 911 is set as a tracking target by performing an operation of touching the person 911. In FIG. 18A and the like, a region filled with dots represents a monitor casing.

JP 2009-141475 A JP-A-4-282899 JP 2006-101186 A

  However, since the display screen 901 of a small digital camera that is popular for general consumers is not so large, each subject on the display screen 901 is not so large in comparison with the size of a human finger. . In particular, when the distance in the real space between the person of interest and the digital camera is long, or when photographing with a wide angle of view, the display portion of the person of interest on the display screen 901 may be considerably small. For this reason, it may be difficult to select a person of interest accurately on the display screen 901 by a touch panel operation. In addition, along with the movement of the person of interest in the real space, the person of interest also moves on the display screen 51, which makes the selection more difficult. For example, in spite of desiring to select the person 911 as the tracking target, the display portion of the person 912 is accidentally touched as shown in FIG. It can also be set as a target.

  SUMMARY An advantage of some aspects of the invention is that it provides an imaging apparatus having a function of supporting a user to specify a tracking target.

  An imaging apparatus according to the present invention includes an imaging element that outputs a signal representing an image sequence obtained by sequentially photographing a subject, a distance information acquisition unit that acquires distance information between the subject and the imaging element, and the imaging A display screen that displays a display image based on the output of the element, an operation receiving unit that receives an input operation of a specified position on the display screen, and a specific subject is set as a tracking target based on the distance information and the specified position And a tracking processing unit that tracks the tracking target on the image sequence based on image data of the image sequence.

  In general, a subject that is noticed by the user has a relatively short subject distance compared to other subjects. Therefore, for example, if the distance information is referred to together with the designated position by the user, even if the designated position does not correctly indicate the subject of interest, the subject of interest of the user is estimated and the subject of interest is set as a tracking target. Is possible. Alternatively, for example, it is possible to display a subject that is likely to be selected as a tracking target by using the distance information so as to support the designation of the subject. As described above, if the tracking target is set based on the distance information and the specified position as described above, the user can support the tracking target specifying operation, and the desired subject can be accurately and easily set as the tracking target. It becomes possible to set.

  Specifically, for example, the tracking target setting unit corrects the specified position using the distance information, and sets the subject at the corrected specified position as the tracking target.

  As a result, even if the designated position does not correctly indicate the subject of interest, the subject of interest can be set as a tracking target.

  Alternatively, for example, when the input operation is accepted when a plurality of subjects appear on the display image, the display screen displays a plurality of indices corresponding to the plurality of subjects together with the display image, The tracking target setting unit determines the size of the plurality of indices based on the distance information, and sets the tracking target based on a positional relationship between the designated position and each index on the display screen.

  This makes it possible to assign an indicator with a larger display size to a subject with a shorter subject distance, which is estimated to be highly likely to be selected as a tracking target. As a result, it becomes easier for the user to select and specify the target subject as the tracking target.

  More specifically, for example, the plurality of subjects include first to nth subjects, and the plurality of indicators include first to nth indicators corresponding to the first to nth subjects (n is 2), the tracking target setting unit determines the i th subject corresponding to the i th index when the specified position is a position corresponding to the i th index (i is a natural number equal to or less than n). Set the tracking target.

  Further, for example, the image sequence includes a reference image and a plurality of non-reference images obtained by photographing after the reference image, and the tracking target setting unit uses the distance information to determine the tracking target. An initial tracking area including image data is set in the reference image, and the tracking processing unit uses each image data in the initial tracking area in the reference image and image data of each non-reference image to each non-reference image. By detecting the position of the tracking target at, the tracking target is tracked on the image sequence.

  By setting the initial tracking area using the distance information, improvement in tracking accuracy is expected.

  Specifically, for example, the tracking target setting unit sets the initial tracking area using the distance information so that image data of subjects other than the tracking target is excluded from the initial tracking area.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the imaging device which has a function which assists designation | designated operation of the tracking target by a user.

1 is a schematic overall block diagram of an imaging apparatus according to a first embodiment of the present invention. It is an internal block diagram of the imaging part shown by FIG. It is a schematic exploded view of the display part shown by FIG. It is the figure (a) which shows the relationship between a display screen and an XY coordinate plane, and the figure (b) which shows the relationship between a two-dimensional image and an XY coordinate plane. It is the external appearance perspective view of an imaging device seen from the photographer side. FIG. 2 is a block diagram of a part particularly involved in a subject tracking function included in the imaging apparatus of FIG. 1. 6 is an operation flowchart of the imaging apparatus in a tracking mode according to the first embodiment of the present invention. It is a figure showing distance relation between an imaging device and a plurality of subjects. It is a figure which concerns on 1st Embodiment of this invention and shows the frame image obtained by image | photographing several subjects, and the distance image corresponding to it. It is a figure which concerns on 1st Embodiment of this invention and shows the frame image obtained by image | photographing several subjects, and the distance image corresponding to it. It is a flowchart of the initial registration process performed in tracking mode. 6 is a flowchart illustrating a deformation operation of the imaging apparatus in the tracking mode according to the first embodiment of the present invention. It is a figure which shows the example of a display content of the display screen at the time of reception of tracking object designation | designated operation (designated position input operation) concerning 1st Embodiment of this invention. FIGS. 6A and 6B show an initial frame image and a distance image corresponding to the initial frame image, and FIGS. 3C and 3D show examples of an initial tracking area set in the initial frame image. FIGS. 9 is an operation flowchart of the imaging apparatus in the tracking mode according to the second embodiment of the present invention. It is a figure which concerns on 2nd Embodiment of this invention and shows the frame image obtained by image | photographing several subjects, and the distance image corresponding to it. It is a figure which shows the example of a display content of the display screen at the time of reception of tracking object selection operation (designated position input operation) concerning 2nd Embodiment of this invention. It is a figure for demonstrating the conventional tracking object selection operation using a touch panel.

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle.

<< First Embodiment >>
A first embodiment of the present invention will be described. FIG. 1 is a schematic overall block diagram of an imaging apparatus 1 according to the first embodiment. The imaging device 1 is a digital still camera capable of capturing and recording still images, or a digital video camera capable of capturing and recording still images and moving images. The imaging device 1 may be mounted on a mobile terminal such as a mobile phone.

  The imaging device 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a main control unit 13, an internal memory 14, a display unit 15, a recording medium 16, and an operation unit 17.

  FIG. 2 shows an internal configuration diagram of the imaging unit 11. The imaging unit 11 drives and controls the optical system 35, the diaphragm 32, the imaging element 33 such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the optical system 35 or the diaphragm 32. And a driver 34. The optical system 35 is formed from a plurality of lenses including the zoom lens 30 and the focus lens 31. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction. The driver 34 drives and controls the positions of the zoom lens 30 and the focus lens 31 and the opening degree of the diaphragm 32 based on the control signal from the main control unit 13, so that the focal length (view angle) and focus of the imaging unit 11 are controlled. The position and the amount of light incident on the image sensor 33 (in other words, the aperture value) are controlled.

  The image sensor 33 photoelectrically converts an optical image representing a subject incident through the optical system 35 and the diaphragm 32 and outputs an electrical signal obtained by the photoelectric conversion to the AFE 12. More specifically, the image sensor 33 includes a plurality of light receiving pixels arranged two-dimensionally in a matrix, and in each photographing, each light receiving pixel stores a signal charge having a charge amount corresponding to the exposure time. An analog signal from each light receiving pixel having a magnitude proportional to the amount of stored signal charge is sequentially output to the AFE 12 in accordance with a drive pulse generated in the imaging device 1.

  The AFE 12 amplifies the analog signal output from the imaging unit 11 (image sensor 33), and converts the amplified analog signal into a digital signal. The AFE 12 outputs this digital signal to the main control unit 13 as RAW data. The amplification degree of signal amplification in the AFE 12 is controlled by the main control unit 13.

  The main control unit 13 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. Based on the RAW data from the AFE 12, the main control unit 13 generates image data representing an image captured by the imaging unit 11 (hereinafter also referred to as a captured image). The image data generated here includes, for example, a luminance signal and a color difference signal. However, the RAW data itself is a kind of image data, and an analog signal output from the imaging unit 11 is also a kind of image data. The main control unit 13 also has a function as display control means for controlling the display content of the display unit 15, and performs control necessary for display on the display unit 15.

  The internal memory 14 is formed by SDRAM (Synchronous Dynamic Random Access Memory) or the like, and temporarily stores various data generated in the imaging device 1. The display unit 15 is a display device having a display screen such as a liquid crystal display panel, and displays captured images, images recorded on the recording medium 16, and the like under the control of the main control unit 13.

  The recording medium 16 is a non-volatile memory such as a card-like semiconductor memory or a magnetic disk, and stores a photographed image or the like under the control of the main control unit 13. The operation unit 17 includes a shutter button 20 that receives a still image shooting instruction, and receives various operations from the outside. The operation on the operation unit 17 is also referred to as a button operation in order to distinguish it from the touch panel operation. The content of the operation on the operation unit 17 is transmitted to the main control unit 13.

  The display unit 15 is provided with a touch panel. FIG. 3 is a schematic exploded view of the display unit 15 (in other words, a schematic exploded view of the touch panel). The display unit 15 is provided with a display screen 51 formed of a liquid crystal display or the like, and a touch detection unit 52 that detects a position where the operating body touches the display screen 51 (a position where pressure is applied). . The user can give a specific instruction to the imaging apparatus 1 by touching the display screen 51 of the display unit 15 with an operating tool. An operation by touching the display screen 51 with an operating body is referred to as a touch panel operation. A contact position between the operating tool and the display screen 51 is referred to as a touch position. When the operating body is touching the display screen 51, the touch detection unit 52 outputs touch position information indicating the touched position (ie, touch position) to the main control unit 13 in real time. The operation body is a finger, a pen, or the like. In the following, it is mainly assumed that the operation body is a finger. Further, in this specification, when the term “display” is simply used, it means the display on the display screen 51.

  As shown in FIG. 4A, the position on the display screen 51 is defined as a position on a two-dimensional XY coordinate plane. Furthermore, as shown in FIG. 4B, in the imaging device 1, an arbitrary two-dimensional image 300 is also handled as an image on the XY coordinate plane. The XY coordinate plane has an X axis extending in the horizontal direction of the display screen 51 and the two-dimensional image 300 and a Y axis extending in the vertical direction of the display screen 51 and the two-dimensional image 300 as coordinate axes. All images described herein are two-dimensional images unless otherwise specified. The position of a certain point of interest on the display screen 51 and the two-dimensional image 300 is represented by (x, y). x represents the X-axis coordinate value of the attention point, and also represents the horizontal position of the attention point on the display screen 51 and the two-dimensional image 300. y represents the Y-axis coordinate value of the target point, and also represents the vertical position of the target point on the display screen 51 and the two-dimensional image 300. When the 2D image 300 is displayed on the display screen 51 (when the 2D image 300 is displayed using the entire display screen 51), the image at the position (x, y) on the 2D image 300 is displayed on the display screen. 51 is displayed at position (x, y).

  The operation modes of the imaging apparatus 1 include a shooting mode in which a still image or a moving image can be shot and a playback mode in which the still image or the moving image recorded on the recording medium 16 can be played on the display unit 15. In the shooting mode, a subject is periodically shot at a predetermined frame period, and RAW data representing a shot image sequence of the subject is output from the imaging unit 11 (more specifically, from the AFE 12). An image sequence typified by a captured image sequence refers to a collection of images arranged in time series. One image is represented by image data for one frame period. One captured image expressed by image data for one frame period from the AFE 12 is also referred to as a frame image. You may interpret that the image obtained by performing predetermined image processing (Democycling processing, noise removal processing, color correction processing, etc.) with respect to the picked-up image by RAW data is a frame image.

  FIG. 5 shows an external perspective view of the imaging apparatus 1 as seen from the photographer side. FIG. 5 also shows a person as a subject. A frame image sequence obtained by sequential shooting is displayed on the display screen 51 as a moving image. A user as a photographer confirms the photographing range of the imaging device 1 (in other words, the photographing range of the imaging unit 11 or the imaging element 33) by confirming the state of the subject displayed on the display screen 51. Can do.

  The imaging apparatus 1 has a subject tracking function using image processing, and performs a characteristic operation when the subject tracking function is realized. The subject tracking function is realized in the shooting mode. An operation mode that realizes the subject tracking function, which is one form of the shooting mode, is referred to as a tracking mode. The operation in the tracking mode is executed by performing a predetermined touch panel operation or button operation. The operations described below are operations of the imaging apparatus 1 in the tracking mode unless otherwise specified.

  FIG. 6 is a block diagram of a part of the imaging apparatus 1 that is particularly related to the subject tracking function. A subject distance information generation unit (distance information acquisition unit) 61, a tracking processing unit 62, and a display control unit 64 can be provided in the main control unit 13 of FIG. The tracking processing unit 62 includes a tracking target setting unit 63. The image data of each frame image forming the frame image sequence is sequentially given to the tracking processing unit 62 and the display control unit 64.

  A subject distance information generation unit (hereinafter sometimes abbreviated as a distance information generation unit) 61 executes subject distance detection processing for detecting the subject distance of each subject that is within the shooting range of the imaging apparatus 1. Thus, subject distance information representing the subject distance of the subject at each position on the frame image is generated. The subject distance for a certain subject refers to the distance in real space between the subject and the imaging device 1 (more specifically, the imaging device 33). In the present specification, the expression “the distance is large” and the expression “the distance is long” are synonymous, and the expression “the distance is small” and the expression “the distance is short” are synonymous. The subject distance detection process can be executed periodically or at a desired timing. The subject distance information is a distance image in which each pixel value forming the subject has a subject distance measurement value. Any method including a known method (for example, a method described in Japanese Patent Application Laid-Open No. 2009-272799) can be used as a method for detecting a subject distance and a method for generating subject distance information.

  The subject distance information may be generated from the image data of the frame image, or the subject distance information may be generated from information other than the image data of the frame image. For example, subject distance information can be generated using a distance measuring sensor (not shown) that measures the subject distance of each subject. As the distance measuring sensor, any known distance measuring sensor such as a distance measuring sensor based on the triangulation method can be used. Alternatively, for example, the distance image may be generated using a compound eye camera.

  Alternatively, for example, the imaging unit 11 may be formed so that the RAW data includes information representing the subject distance, and the subject distance information may be generated from the RAW data. In order to realize this, for example, a method called “Light Field Photography” (for example, a method described in International Publication No. 06/039486 pamphlet or Japanese Patent Application Laid-Open No. 2009-224982; hereinafter referred to as “Light Field method”) is used. May be. In the light field method, by using an imaging lens having an aperture stop and a microlens array, the image signal obtained from the imaging device has information on the light traveling direction in addition to the light intensity distribution on the light receiving surface of the imaging device. It comes to include. Therefore, although not shown in FIG. 2, when the light field method is used, an optical member necessary for realizing the light field method is provided in the imaging unit 11. The optical member includes a microlens array or the like, and incident light from the subject enters the light receiving surface (in other words, the imaging surface) of the image sensor 33 via the microlens array or the like. The microlens array is composed of a plurality of microlenses, and one microlens is assigned to one or a plurality of light receiving pixels on the image sensor 33. As a result, the output signal of the image sensor 33 includes information on the traveling direction of the incident light to the image sensor 33 in addition to the light intensity distribution on the light receiving surface of the image sensor 33.

  In addition to the image data of the frame image sequence, subject distance information and touch position information are input to the tracking processing unit 62. The tracking processing unit 62 tracks the position of the specific subject in the frame image sequence by sequentially detecting the position of the specific subject in each frame image based on the image data of the frame image sequence. A process for executing the tracking is called a tracking process. A specific subject to be tracked is called a tracking target.

  The tracking target setting unit 63 sets a tracking target from among subjects that are within the shooting range of the imaging apparatus 1 based on the content of the operation by the user. In this setting, subject distance information is used, and an example of how to use it will be described later. The user's operation that is the basis for setting the tracking target is, for example, a predetermined touch panel operation. In this case, the tracking target is set based on the touch position information.

  The display control unit 64 controls the display content of the display screen 51, generates image data of a display image from the image data of the frame image, and sends this to the display unit 15, thereby displaying the display image on the display screen 51. Is displayed. Since the frame images are sequentially acquired sequentially, the display image is also periodically generated and updated and displayed on the display screen. This display image is generated by reflecting the tracking result by the tracking processing unit 62 in the frame image.

The operation in the tracking mode will be described in more detail with reference to FIG. FIG. 7 is a flowchart showing an operation flow of the imaging apparatus 1 in the tracking mode. In order to explain this operation more specifically, it is assumed that the _four subjects SUB _{1 to} SUB 4 shown in FIG. 8 are included in the subjects that fall within the imaging range of the imaging apparatus 1. Subjects SUB _{1 to} SUB ₄ are subjects in a soccer game. Each of the subjects SUB _{1 to} SUB ₃ is a person who can move in the real space, and the subject SUB ₄ is a soccer goal that is stationary in the real space. The subject distances of the subjects SUB _{1 to} SUB ₄ are represented by d _{1 to} d _4, respectively. Here, it is assumed that the inequality “d ₁ <d ₂ <d ₃ <d ₄ ” always holds.

  When the operation mode of the imaging device 1 is set to the tracking mode, first, in step S11, the imaging device 1 accepts an input of a tracking target specifying operation by the user, and when the input is made, step S12 and subsequent steps are performed. Execute the process. The tracking target specifying operation in step S11 is realized by touching a part on the display screen 51 where a subject to be tracked is displayed with a finger. In step S11, the position where the user touches the display screen 51 in order to specify the tracking target is referred to as a specified position. Then, it can be said that the tracking target specifying operation in step S11 is a specified position input operation by the user, and the touch detection unit 52 in FIG. 3 functions as an operation receiving unit that receives the specified position input operation.

  In step S12, the distance information generating unit 61 generates subject distance information by executing subject distance detection processing. The subject distance detection process in step S12 can be performed at the time when the tracking target specifying operation is performed or immediately after the tracking target specifying operation is performed. Further, the subject distance information in step S12 may be generated from the result of the subject distance detection process executed immediately before the tracking target designation operation is performed.

  In subsequent step S13, the tracking target setting unit 63 corrects the designated position determined in step S11 using the subject distance information generated in step S12. This correction method is illustrated.

For example, consider the case where the position 321 on the display screen 51 is set as the designated position before correction by the user's designated position input operation when the frame image 320 of FIG. 9A is displayed as the display image. . In position 321, there are image data of the ground portion of the soccer field, the object distance of the ground portion is made larger than that of the subject SUB _1. An image 330 in FIG. 9B is a distance image as subject distance information corresponding to the frame image 320. The arbitrary distance image is a gray scale image. In the distance image, each pixel has a pixel value closer to zero as the corresponding subject distance is smaller, and has a larger pixel value as the corresponding subject distance is larger. Further, in the drawing showing the distance image, a portion having a relatively small pixel value (that is, a portion having a relatively small subject distance) is shown in white, and a portion having a relatively large pixel value (that is, a portion having a relatively large subject distance). ) Is shown relatively black. For simplification of illustration, in the distance image 330 of FIG. 9B, all portions where the subject distance is larger than a certain extent are shown in black (the same applies to other distance images described later).

When the position 321 is set as the designated position before correction, the tracking target setting unit 63 has a subject having a subject distance smaller than the subject distance of the ground portion at a position spatially close to the position 321. Is determined based on the subject distance information (that is, the distance image 330) in step S12, and if the presence is recognized, the designated position is corrected to the position of the subject having a smaller subject distance. In the example of FIG. 9A, the subject SUB ₁ having a subject distance smaller than the subject distance of the ground portion exists at a position spatially close as viewed from the position 321. Therefore, in this example, the tracking target setting unit 63 can correct the designated position from the position 321 to the position 322 where the image data of the subject SUB ₁ exists. A position spatially close to the target position (position 321 in this example) is, for example, a position within a region having a certain size with the target position as the center.

  Further, for example, when the frame image 340 of FIG. 10A is displayed as a display image, the position 341 on the display screen 51 is set as the designated position before correction by the user's designated position input operation. Think. An image 350 in FIG. 10B is a distance image as subject distance information corresponding to the frame image 340.

In position 341, the image data of the subject SUB ₃ is present. That is, the subject at the position 341 is the subject SUB ₃ . Also in this case, the tracking target setting unit 63 determines whether or not a subject having a subject distance smaller than the subject distance of the subject SUB ₃ exists in a spatially close position as viewed from the position 341. When the determination is made based on the information (that is, the distance image 350) and the presence is recognized, the designated position is corrected to the position of the subject having a smaller subject distance. In the example of FIG. 10A, the subject SUB ₁ having a subject distance smaller than the subject distance of the subject SUB ₃ exists at a position spatially close to the position 341. Therefore, in this example, the tracking target setting unit 63 can correct the designated position from the position 341 to the position 342 where the image data of the subject SUB ₁ exists.

Further, for example, the designated position may be corrected to the position on the subject in the focus area. The in-focus area is an image area where image data of a subject in focus exists. A subject located within the depth of field containing the in-focus distance, that is, a subject having a subject distance within the depth of field is focused. Therefore, the image area where the image data of the subject located within the depth of field is equivalent to the in-focus area. For example, consider the case where the position 321 on the display screen 51 is set as the designated position before correction by the user's designated position input operation when the frame image 320 of FIG. 9A is displayed as the display image. . At this time, the tracking target setting unit 63 obtains the in-focus distance and the depth of field from the position of the focus lens 31 and the aperture value at the time of shooting the frame image 320, and the obtained in-focus distance and the object field. The focus area in the frame image 320 is specified from the depth and the subject distance information (that is, the distance image 330) in step S12. Assuming that the subject SUB ₁ is located within the depth of field in real space, the image data of the subject SUB ₁ exists in the specified in-focus area. Further, it is assumed that the position 321 does not exist in the focus area. In this case, the tracking target setting unit 63 can correct the designated position from the position 321 to the position 322 where the image data of the subject SUB ₁ exists.

  Information on the specified position after correction is transmitted from the tracking processing unit 62 to the display control unit 64. After the correction in step S13, in step S14, the display control unit 64 superimposes the specified position after correction on the latest frame image. And displayed on the display screen 51. A frame surrounding the entire subject at the specified position after correction may be displayed superimposed on the latest frame image. Thereafter, in step S15, the main control unit 13 determines whether or not the user has performed a tracking start instruction operation. The tracking start instruction operation is an operation for instructing the start of the tracking process after approving the correction in step S13, and is realized by a predetermined touch panel operation or the like. When the tracking start instruction operation is performed, the process proceeds from step S15 to step S16, and the processes after step S16 and step S16 are executed. On the other hand, when the tracking start instruction operation is not performed and an operation for requesting resetting of the designated position is performed, the process returns from step S15 to step S11, and the process from step S11 is executed again. Basically, the display image sequence based on the frame image sequence is displayed as a moving image on the display screen 51, but during the period when the tracking start instruction operation is accepted, it corresponds to the specified position after correction. In order to make it easier for the user to understand the correspondence with the subject, the reproduction of the moving image on the display screen 51 may be temporarily stopped.

In step S16, the tracking target setting unit 63 sets the subject at the specified position after correction as the tracking target, and executes tracking target initial registration processing in subsequent step S17. For example, when the position 322 or 342 is a corrected designated position as in the example of FIG. 9A or FIG. 10A, the subject SUB ₁ is set as a tracking target.

  The initial registration process is a process for registering initial information to be tracked, and includes the processes of steps S31 and S32 in FIG. In step S31, the latest frame image is set as an initial frame image (reference image), and a tracking area is set in the initial frame image. Each frame image obtained after the initial frame image is called a tracking target frame image (non-reference image). The tracking area refers to an image area in which image data to be tracked exists. The tracking area set in the initial frame image is particularly called an initial tracking area. The initial tracking area is set based on the specified position after correction. Simply, for example, a rectangular area having a predetermined size and centered on the corrected designated position can be set as the initial tracking area. However, the shape of the tracking area to be set in the initial frame image and each tracking target frame image is not limited to a rectangle.

  In step S32, the tracking processing unit 62 or the tracking target setting unit 63 extracts the image feature amount from the image data in the initial tracking area, and stores the initial registration information including the extracted image feature amount in a memory (for example, the internal memory 14). save. The image feature amount extracted here is an image feature amount useful for tracking the tracking target, and includes, for example, color information and edge information. The initial registration information can include various information necessary or useful for executing the tracking process in addition to the image feature amount. For example, the initial registration information may include the subject distance in the initial tracking area (that is, the subject distance to be tracked in the initial frame image) and the position, size, and shape of the initial tracking area on the initial frame image. it can.

  When the initial registration process in step S17 of FIG. 7 is completed by storing the initial registration information, the tracking process is started in step S18. That is, the tracking processing unit 62 executes the tracking process on the tracking target frame image sequence obtained after the process of step S17. In the tracking process, the position of the tracking target in each tracking target frame image (in other words, the position of the tracking area) is detected based on the image data in the initial tracking area of the initial frame image and the image data of each tracking target frame image. As a result, the tracking target is tracked on the frame image sequence including the initial frame image and each tracking target frame image. Since the image feature amount of the initial tracking area is stored in the initial registration information, in practice, the tracking process may be performed based on the initial registration information and the image data of each tracking target frame image.

  The tracking processing unit 62 can execute the tracking process using any known arbitrary tracking processing method. For example, tracking processing can be performed based on color information, and tracking methods based on color information are disclosed in Japanese Patent Application Laid-Open Nos. 5-284411, 2000-48211, 2001-169169, and the like. Can be used. Further, for example, the tracking process may be executed by a method using pattern matching as described in Japanese Patent Application Laid-Open No. 2009-141475.

  Specifically, for example, after the tracking area is set in the first frame image that is the initial frame image or an arbitrary tracking target frame image and the image feature amount of the tracking area in the first frame image is obtained, What is necessary is just to perform the following processes with respect to the 2nd frame image obtained after a frame image. An evaluation block having the same size as the size of the tracking area is set in the second frame image, and the image feature amount of the image in the evaluation block and the image feature amount of the tracking area in the first frame image The similarity evaluation is performed while sequentially changing the position of the evaluation block within the search range, and it is determined that the tracking target on the second frame image exists at the position of the evaluation block where the maximum similarity is obtained (that is, The image area in the evaluation block where the maximum similarity is obtained is determined to be the tracking area of the second frame image). The search range for the second frame image is set with reference to the position of the tracking area of the first frame image. By such tracking processing, the position of the tracking target is successively detected between temporally consecutive frame images, and the position of the tracking target in each tracking target frame image is detected.

  The tracking process may be performed by further using subject distance information obtained periodically. At this time, the continuity in the time direction of the subject distance to be tracked may be considered. In other words, the principle that the subject distance of the tracking target does not differ greatly between temporally adjacent frame images (however, the frame period is about 1/60 seconds and the tracking target is a person or the like). The tracking target in each tracking target frame image may be detected. For example, the similarity may be obtained in consideration of subject distance information, or the search range may be limited in consideration of subject distance information.

  The tracking process started in step S18 is continuously executed until there is a tracking process stop instruction or a tracking target redesignation operation by the user (steps S19 and S20). If there is a tracking target redesignation operation, the process returns to step S11 and the processing from step S11 is executed again.

  Note that, in an arbitrary frame image, when the first to nth subject regions having different subject distances are distinguished, an area division process based on subject distance information can be used (n is an integer of 2 or more). The i-th subject area refers to an image area in which image data of the i-th subject exists (i is an integer). When i and j are different integers, it is assumed that the i-th and j-th subjects are different subjects. By this region division processing, the entire image region of the frame image is divided into a plurality of image regions based on the subject distance information, and the first to nth subject regions are included in the plurality of image regions. In the region division processing, the entire image region of the frame image is divided so that a set of pixels having similar feature amounts and spatially close together forms one unit. A known region unification method or the like can be used for forming the set and dividing the image area. In the region division processing based on the subject distance information, the subject distance is used as a feature amount.

  In the above description, it is assumed that the designated position is corrected in step S13, but as a result, correction may not be performed. For example, when the position 322 in FIG. 9A is input as the designated position in step S11, the designated position is not corrected. In this case as well, for the sake of convenience, the position 322 may be regarded as the specified position after correction, and step S14 and subsequent processing may be executed.

  When an operation (designated position input operation) for designating the subject of interest as a tracking target is performed using the touch panel, the designated position is set because the subject of interest has a small display size or the subject of interest moves on the display screen 51. It may be difficult to accurately match the subject of interest. On the other hand, the user's attention subject is generally often located on the near view side and often has a relatively short subject distance compared to other subjects. Considering this, as described above, the function of correcting the user-specified position using the subject distance information is provided. As a result, even if the designated position does not correctly indicate the subject of interest, it is possible to guess the subject of interest of the user and set the subject of interest as a tracking target. As a result, it is possible to assist the user in specifying the tracking target, and to accurately and easily set a desired subject as the tracking target.

  Hereinafter, modification techniques applicable to the above-described operation will be described as first to seventh application examples. As long as there is no contradiction, a plurality of application examples can be combined.

[First application example]
A first application example will be described. In step S14, both the specified position before correction and the specified position after correction may be displayed. For example, when the positions 341 and 342 in FIG. 10A are designated positions before and after correction, the positions 341 and 342 may be superimposed on the latest frame image and displayed together with the latest frame image. In a state where such a display is made, the user can select one of the positions 341 and 342. If the position 341 is selected, the subject SUB ₃ corresponding to the designated position before correction may be set as the tracking target, and the processing in step S17 and subsequent steps may be executed. If the position 342 is selected, the correction is performed. After the subject SUB ₁ corresponding to the designated position is set as the tracking target, step S17 and subsequent processing may be executed.

Although similar to this, in step S14, both the subject area of the subject at the designated position before correction and the subject area of the subject at the designated position after correction may be displayed. For example, when the positions 341 and 342 in FIG. 10A are designated positions before and after correction, the contour of the subject SUB _{3 and} the contour of the subject SUB ₁ are superimposed on the latest frame image together with the latest frame image. May be displayed. In a state in which form such a display, the user to select one of the subject SUB ₃ and SUB _1, step S17 and it in terms of setting the subject SUB ₃ to tracking target if the subject SUB ₃ is selected may be executed subsequent processing may be executed steps S17 and subsequent processing in terms of setting the subject SUB ₁ if the subject SUB ₁ has been selected in the tracking target.

  In general, there is a high possibility that the subject of interest of the user (photographer) is present on the near view side, but in some cases, the subject of interest of the user is present on the far view side. Considering this, if the above selection operation is made possible, the validity of the specified position correction can be confirmed (confirmed by the user), and the tracking target can be set as intended by the user.

[Second application example]
A second application example will be described. During the execution of the tracking process, a frame indicating the tracking area may be displayed while displaying the frame image sequence as a moving image. A display such as a frame indicating the tracking area is referred to as a tracking area display. With the tracking area display, the user can confirm the tracking target and the tracking area on the display screen 51 on which the frame image sequence is displayed. It should be noted that the period from when the tracking process starts until a certain time elapses may be regarded as the tracking target redesignation period, and the tracking area display may be displayed only during the tracking target redesignation period. At this time, the above-described tracking target redesignation operation (see FIG. 7) may be accepted only during the tracking target redesignation period. When it is desired to respecify the tracking target, the user can perform a tracking target redesignation operation by a touch panel operation or the like.

[Third application example]
A third application example will be described. The flowchart of FIG. 7 may be changed to the flowchart of FIG. That is, after completion of the correction process in step S13, the process may proceed immediately from step S13 to step S16 without executing the processes in steps S14 and S15, and the processes in step S16 and subsequent steps may be executed. In this case, since the user is not given an opportunity to confirm whether or not the subject corresponding to the corrected designated position may be set as the tracking target, the second application example also enables the re-designation of the tracking target. It is good to carry out.

[Fourth application example]
A fourth application example will be described. When the tracking process is started after confirming the presence / absence of the tracking start instruction operation as in step S15, the target subject to be tracked may greatly move before the tracking start instruction operation is performed. is there. When such a large movement occurs, it may occur that the subject of interest does not exist at the specified position after correction when the processes of steps S16 and S17 are executed. Considering this, the following processing may be performed.

  After the specified position is corrected in step S13, each of the first subject at the specified position before correction and the second subject at the specified position after correction is regarded as a provisional tracking target (in other words, a tracking target candidate). Thus, the tracking process for the first subject and the tracking process for the second subject are immediately started separately regardless of whether or not the tracking start instruction operation is performed. Then, the tracking area display is performed for each of the first and second subjects, and in this state, a user's selection operation for selecting one of the first and second subjects is accepted. When the first subject is selected by this selection operation, the first subject is set as a true tracking target, and then step S17 and subsequent processing are executed, and the second subject is selected by this selection operation. In this case, the second subject may be set as a true tracking target, and then step S17 and subsequent processing may be executed.

[Fifth application example]
An example of the designated position input operation will be described as a fifth application example.
The designated position input operation in step S11 is, for example, an operation of touching one point on the display screen 51 with a finger, and in this case, the position of the one point itself becomes the designated position before correction.
Alternatively, for example, the designated position input operation in step S11 may be an operation of surrounding a specific area on the display screen 51 with the finger while maintaining a contact state between the finger and the display screen 51. In this case, the position in the specific area (for example, the center position or the gravity center position of the specific area) becomes the designated position before correction.
Alternatively, for example, the entire display area of the display screen 51 displaying the frame image is divided in the horizontal and vertical directions to set a plurality of divided display areas, and the user can select any one of the divided display areas by operating the touch panel. Thus, the designated position in step S11 may be determined. At this time, as shown in FIG. 13, it is preferable to display the boundary line between adjacent divided display areas so as to be superimposed on the frame image. In this case, if a point in a certain divided display area is touched with a finger, the position in the divided display area including the touch position (for example, the center position or the center of gravity position of the divided display area) is not corrected. It becomes the specified position. In FIG. 13, a region filled with dots represents a housing portion of the display unit 15 (the same applies to FIG. 17 described later).

[Sixth application example]
A sixth application example will be described. An initial tracking area from which an initial image feature amount should be extracted may be determined using subject distance information. For example, consider a case in which the image 400 in FIG. 14A is an initial frame image and the subject SUB ₁ is set as a tracking target (ie, the corrected specified position is a position on the subject SUB ₁ ). Think about the case). The initial frame image 400 is an image obtained by photographing the subject SUB ₁ as a person with the forest in the background. An image 410 in FIG. 14B is a distance image corresponding to the initial frame image 400 (for example, a distance image based on the result of subject distance detection processing immediately before, during, or immediately after the initial frame image 400).

  In such a case, for example, if the initial tracking area is a rectangular area 401 (see FIG. 14C) having a certain size, a subject other than the tracking target in the initial tracking area (in this example, a tree) Or ground) image data may be included. If image data of a subject other than the tracking target is included in the initial tracking area, the image feature amount of the subject other than the tracking target is included in the image feature amount of the initial registration information, and the tracking accuracy deteriorates. This deterioration is particularly noticeable when the tracking target and the background are similar in color or the like. In order to avoid this, the distance image 410 as the subject distance information is used so that the image data of the subject other than the tracking target is excluded from the initial tracking region (in other words, the subject other than the tracking target is excluded from the initial tracking region). It is better to set the position, size and shape of the initial tracking area. As a result, the tracking accuracy can be improved.

Specifically, for example, by the region dividing processing using the distance image 410, cut the entire image area of the initial frame image 400 in the image region other than the subject region in the subject area and the subject SUB ₁ subject SUB _1, the subject SUB ₁ An area including all or a part of the subject area and not including the image area other than the subject area of the subject SUB ₁ may be set as the initial tracking area. An area surrounded by a broken line 402 in FIG. 14D is an example of such an initial tracking area.

[Seventh application example]
A seventh application example will be described. The tracking result can be used in various forms. For example, the position of the focus lens 31 is adjusted so that the subject in the tracking area continues to be focused, the ISO sensitivity is adjusted so that the exposure to the subject in the tracking area is optimized, or the subject in the tracking area is adjusted. The white balance can be adjusted so that the white balance is optimized.

  In addition, a cutout area including the tracking area is set in each frame image, an image in the cutout area (hereinafter referred to as a cutout image) is extracted from each frame image, and a moving image including the cutout image sequence is stored in the recording medium 16. You can also save it. The cutout area may be the tracking area itself. The moving image composed of the cut-out image sequence may be stored in the recording medium 16 after the resolution is increased. As a method for increasing the resolution, for example, the method described in JP-A-2009-147727 can be used. Of course, by storing the entire image data of each frame image in the recording medium 16, the frame image sequence itself can be recorded as a moving image.

<< Second Embodiment >>
A second embodiment of the present invention will be described. The second embodiment is an embodiment based on the first embodiment. Regarding matters not specifically described in the second embodiment, the description of the first embodiment is applied to the second embodiment as long as there is no contradiction. .

  The operation in the tracking mode of the second embodiment will be described with reference to FIG. FIG. 15 is a flowchart showing the flow of the operation of the imaging apparatus 1 in the tracking mode. When the operation mode of the imaging apparatus 1 is set to the tracking mode, first, in step S51, the distance information generation unit 61 generates subject distance information by executing subject distance detection processing. In subsequent step S52, the tracking target setting unit 63 sets a tracking target candidate based on the subject distance information in step S51. The tracking target candidate is a tracking target candidate to be set in step S55 described later.

Consider a case where the frame image at the time of setting the tracking target candidate is the image 500 in FIG. The frame image 500, the image data of the object SUB ₂ having subject SUB ₁ and a relatively large object distance _{d 2} has a relatively small object distance _{d 1} is present. For the sake of simplification, it is assumed that there is a background subject whose subject distance is sufficiently larger than d ₁ and d ₂ other than the subject areas of the subjects SUB ₁ and SUB ₂ . An image 510 in FIG. 16B is a distance image corresponding to the frame image 500 based on the subject distance information in step S51. In step S52, for example, each subject whose subject distance is smaller than a predetermined distance is set as a tracking target candidate. Consider a case where the subject distances d ₁ and d ₂ are both smaller than a predetermined distance. Then, each of the subjects SUB ₁ and SUB ₂ is set as a tracking target candidate. When only one tracking target candidate is set, the processing execution of steps S53 and S54 described later is omitted, and the one tracking target candidate is set as the tracking target, and step S17 and the subsequent steps are set. What is necessary is just to perform a process.

In step S53 following step S52, the tracking target setting unit 63 generates a tag for each tracking target candidate based on the subject distance information obtained in step S51, and cooperates with the display control unit 64 to frame each tag. The image is displayed on the display screen 51 together with the image. An example of display contents on the display screen 51 in step S53 is shown in FIG. In the display screen 51 of FIG. 17, a tag 521 corresponding to the subject SUB ₁ and a tag 522 corresponding to the subject SUB ₂ are displayed together with the frame image 500. The tag 521 is an index for informing the user that the subject SUB ₁ is _one of the tracking target candidates, and the tag 522 is for informing the user that the subject SUB ₂ is one of the tracking target candidates. It is an indicator.

Here, the size of each tag on the display screen 51 is determined so as to increase as the subject distance of the corresponding tracking target candidate decreases based on the subject distance information obtained in step S51. Since the subject distance d ₁ of SUB ₁ corresponding to the tag 521 is smaller than the subject distance d _{2 of} SUB ₂ corresponding to the tag 522, the size of the tag 521 is larger than the size of the tag 522 on the display screen 51. . In the example of FIG. 17, the shapes of the tags 521 and 522 are similar, but the shapes may be different from each other. Further, each tag is arranged in the vicinity of the corresponding tracking target candidate on the display screen 51 so that the correspondence between the tag and the tracking target candidate can be understood. At this time, in order to prevent the main subject from being difficult to confirm on the display screen 51 due to the display of the tag, each tag may be displayed in the display region of the background subject or the subject display region where the subject distance is a predetermined distance or more. .

  During the display of each tag, in step S54, the input of the tracking target selection operation by the user is accepted by the touch panel (touch detection body 52), and when the input is made, step S55 and subsequent processing are executed. The tracking target selection operation in step S54 is realized by touching any tag on the display screen 51 with a finger. If the touch position for designating any tag is called the designated position, it can be said that the tracking target selection operation in step S54 is a designated position input operation by the user, and the touch detection unit 52 in FIG. It functions as an operation reception unit that receives a specified position input operation. Basically, the display image sequence based on the frame image sequence is displayed as a moving image on the display screen 51. During the period when the tracking target selection operation is accepted, the operation is facilitated. You may make it stop reproduction | regeneration of the moving image on the display screen 51 temporarily.

In step S55, the tracking target setting unit 63 selects a tracking target from a plurality of tracking target candidates according to the content of the tracking target selection operation. That is, the tracking target is set based on the positional relationship between the designated position by the tracking target selection operation and the display position of each tag. More specifically, if the specified position by the tracking target selection operation is the position where the tag 521 is displayed, the tracking target candidate corresponding to the tag 521 (that is, the subject SUB ₁ ) is set as the tracking target, and the tracking target is set. If the designated position by the selection operation is the position where the tag 522 is displayed, the tracking target candidate (that is, the subject SUB ₂ ) corresponding to the tag 522 is set as the tracking target. After setting the tracking target, the tracking process is executed through the initial registration process of the tracking target (steps S17 to S20). The processing executed after setting the tracking target is the same as that described in the first embodiment.

  In the above example, the operation when the number of tracking target candidates is 2 has been described, but the same applies when the number of tracking target candidates is 3. Further, the type of subject that is a tracking target candidate may be limited to a specific type (for example, a person).

  According to the present embodiment, a tag with a large display size is generated and displayed for a subject with a short subject distance, which is likely to be set as a tracking target, and a corresponding tag can be touched by touching it. The subject is set as a tracking target. For this reason, the user can easily and accurately designate a desired subject as a tracking target.

  As long as there is no contradiction, each application example described in the first embodiment can be applied to this embodiment. In particular, for example, the application of the fourth application example to the second embodiment is beneficial. That is, the following operation may be performed. Immediately after the processing of steps S51 to S53, regardless of whether or not a tracking target selection operation is performed, each of the set plurality of tracking target candidates is regarded as a provisional tracking target, and tracking processing for each tracking target candidate is executed individually. Start. If the position of each tracking target candidate is changed on the display screen 51 during the execution of this tracking process, the display position of the tag corresponding to each tracking target candidate is also changed so as to follow it. In this state, the tracking target selection operation in step S54 may be accepted.

<< Deformation, etc. >>
The embodiments of the present invention can be appropriately modified in various ways within the scope of the technical idea shown in the claims. The above embodiment is merely an example of the embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the above embodiment. The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values.

  The imaging apparatus 1 in FIG. 1 can be configured by hardware or a combination of hardware and software. When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. A function realized using software may be described as a program, and the function may be realized by executing the program on a program execution device (for example, a computer).

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 15 Display part 30 Zoom lens 31 Focus lens 32 Aperture 33 Imaging element 51 Display screen 52 Touch detection part 61 Subject distance information generation part 62 Tracking process part 63 Tracking object setting part

Claims (6)

An image sensor that outputs a signal representing an image sequence obtained by sequentially photographing a subject;
A distance information acquisition unit that acquires distance information between the subject and the image sensor;
A display screen for displaying an image for display based on the output of the image sensor;
An operation accepting unit for accepting an input operation at a designated position on the display screen;
A tracking target setting unit that sets a specific subject as a tracking target based on the distance information and the specified position, and a tracking processing unit that tracks the tracking target on the image sequence based on image data of the image sequence; An imaging apparatus comprising: The imaging apparatus according to claim 1, wherein the tracking target setting unit corrects the specified position using the distance information, and sets a subject at the corrected specified position as the tracking target. The display screen displays a plurality of indices corresponding to the plurality of subjects together with the display image when receiving the input operation when a plurality of subjects appear on the display image,
The tracking target setting unit determines the size of the plurality of indices based on the distance information, and sets the tracking target based on a positional relationship between the designated position and each index on the display screen. The imaging apparatus according to claim 1. The plurality of subjects include first to nth subjects, and the plurality of indicators include first to nth indicators corresponding to the first to nth subjects (n is an integer of 2 or more),
The tracking target setting unit sets the i-th subject corresponding to the i-th index as the tracking target when the designated position is a position corresponding to the i-th index (i is a natural number equal to or less than n). The imaging apparatus according to claim 3. The image sequence includes a reference image and a plurality of non-reference images obtained by photographing after the reference image,
The tracking target setting unit uses the distance information to set an initial tracking area containing the tracking target image data in the reference image,
The tracking processing unit detects the position of the tracking target in each non-reference image using the image data in the initial tracking area in the reference image and the image data of each non-reference image. The imaging apparatus according to claim 1, wherein the tracking target is tracked. The tracking target setting unit sets the initial tracking area using the distance information so that image data of a subject other than the tracking target is excluded from the initial tracking area. The imaging device described.

Images