JP2012039210A - Object specification program and camera - Google Patents

Abstract

To identify a subject in an image.
A program for identifying a subject selects a target image from continuously input frames, and classifies one image into a plurality of segmented images based on color information or luminance information of the target image. A binarization procedure for binarizing each of the plurality of segmented images using color information or luminance information to generate a plurality of binarized segmented images, and for each of the plurality of binarized segmented images, An evaluation value calculation procedure for calculating an evaluation value used for specifying a subject in the target image, a subject specification procedure for specifying a subject in the target image based on the evaluation value, and a target image specified in the subject specification procedure The computer is caused to execute a storage procedure for storing the position of the subject in, and a tracking procedure for tracking the subject specified in the subject specifying procedure between frames that are continuously input.
[Selection] Figure 1

Description

  The present invention relates to a subject specifying program and a camera.

  The following imaging devices are known. This imaging apparatus specifies a subject position based on the AF area selected by the user, and performs focus adjustment processing on the specified subject (for example, Patent Document 1).

Japanese Patent Laid-Open No. 2004-20585

  However, in the conventional imaging device, it is difficult to specify the subject when the subject is small or moving at high speed.

A program for identifying a subject according to the present invention selects a target image from continuously input frames, and classifies one image into a plurality of segmented images based on color information or luminance information of the target image; A binarization procedure for binarizing each of a plurality of segmented images using color information or luminance information to generate a plurality of binarized segmented images, and a target for each of the plurality of binarized segmented images An evaluation value calculation procedure for calculating an evaluation value used for specifying a subject in the image, a subject specifying procedure for specifying a subject in the target image based on the evaluation value, and a target image specified in the subject specifying procedure It is characterized in that the computer executes a storing procedure for storing the position of the subject and a tracking procedure for tracking the subject specified in the subject specifying procedure between frames input continuously. To.
A camera according to the present invention is characterized by comprising execution means for executing the subject specifying program.

  According to the present invention, even when the subject is small or moving at high speed, the subject can be identified and tracked with high accuracy.

It is a block diagram which shows the structure of one Embodiment of a camera. It is a flowchart figure which shows the flow of a subject specific process. It is a flowchart figure which shows the flow of a subject extraction process. It is a figure which shows the specific example of a target image. It is a figure which shows the specific example of the binarized Y plane image, the binarized Cb plane image, the binarized Cr plane image, and the binarized Y complement image. It is the figure which showed the example of extraction of a labeling area | region typically. It is a figure which shows the specific example at the time of adding an identifier with respect to a labeling area | region. It is a figure which shows the specific example of the 1st evaluation value calculated about the binarization division image.

  FIG. 1 is a block diagram illustrating a configuration of an embodiment of a camera according to the present embodiment. The camera 100 includes an operation member 101, a lens 102, an image sensor 103, a control device 104, a memory card slot 105, and a monitor 106. The operation member 101 includes various input members operated by the user, such as a power button, a release button, a zoom button, a cross key, an enter button, a play button, and a delete button.

  The lens 102 is composed of a plurality of optical lenses, but is representatively represented by one lens in FIG. The image sensor 103 is an image sensor such as a CCD or a CMOS, for example, and captures a subject image formed by the lens 102. Then, an image signal obtained by imaging is output to the control device 104.

  The control device 104 generates image data in a predetermined image format, for example, JPEG format (hereinafter referred to as “main image data”) based on the image signal input from the image sensor 103. Further, the control device 104 generates display image data, for example, thumbnail image data, based on the generated image data. The control device 104 generates an image file that includes the generated main image data and thumbnail image data, and further includes header information, and outputs the image file to the memory card slot 105. In the present embodiment, it is assumed that both the main image data and the thumbnail image data are image data expressed in the RGB color system.

  The memory card slot 105 is a slot for inserting a memory card as a storage medium, and the image file output from the control device 104 is written and recorded on the memory card. The memory card slot 105 reads an image file stored in the memory card based on an instruction from the control device 104.

  The monitor 106 is a liquid crystal monitor (rear monitor) mounted on the back surface of the camera 100, and the monitor 106 displays an image stored in a memory card, a setting menu for setting the camera 100, and the like. . Further, when the user sets the mode of the camera 100 to the shooting mode, the control device 104 outputs image data for display of images acquired from the image sensor 103 in time series to the monitor 106. As a result, a through image is displayed on the monitor 106.

  The control device 104 includes a CPU, a memory, and other peripheral circuits, and controls the camera 100. Note that the memory constituting the control device 104 includes SDRAM and flash memory. The SDRAM is a volatile memory, and is used as a work memory for the CPU to develop a program when the program is executed or as a buffer memory for temporarily recording data. The flash memory is a non-volatile memory in which data of a program executed by the control device 104, various parameters read during program execution, and the like are recorded.

  In the present embodiment, the control device 104 specifies the position of the subject in the image and the position, size, and shape of the subject by performing the following processing, and the time series from the image sensor 103. The subject is tracked between a plurality of frames that are input continuously. FIG. 2 is a flowchart showing the flow of subject identification processing in the present embodiment. Hereinafter, the subject specifying process in the present embodiment will be described with reference to FIG.

  In step S10, the control device 104 sets the first frame of images continuously input from the image sensor 103 as a target image, and performs subject extraction processing shown in FIG. 3 on the target image. Here, the subject extraction processing in the present embodiment will be described with reference to FIG. In step S510, the control device 104 reads the image data of the first frame, and proceeds to step S520. In step S520, the control device 104 reduces the image size of the read target image so that the vertical size becomes, for example, about 30 to 60 pixels. As a result, the subsequent processing can be speeded up.

After that, the process proceeds to step S530, and the control device 104 converts the target image into an image in YCbCr format, and Y component image (Y plane image), Cr component image (Cr plane image), and Cb component image ( Cb plane images) are generated respectively. Further, the control device 104 generates a Y complement plane image in which white pixels and black pixels of the Y plane image are inverted. Specifically, a target image represented by the RGB color system is represented by a luminance image composed of luminance components (Y components) in a YCbCr color space and color difference components (Cb components, To a color difference image composed of (Cr component). That is, for the target image, the control device 104 generates a luminance image composed of the Y component as a Y plane image using the following equation (1), and a color difference composed of the Cb component using the following equations (2) and (3). An image and a color difference image composed of Cr components are generated as a Cb plane image and a Cr plane image, respectively.
Y = 0.299R + 0.587G + 0.114B (1)
Cb = −0.169R−0.332G + 0.500B (2)
Cr = 0.500R−0.419G−0.081B (3)

  Then, for each of the generated Y plane image, Cb plane image, Cr plane image, and Y complement plane image, the control device 104 examines the density values of all the pixels in the image, and calculates the average value M of each density value and each of the density values. The density standard deviation σ is calculated, and the process proceeds to step S540.

  In step S540, the control device 104 sets each pixel of the Y plane image, Cb plane image, Cr plane image, and Y complement plane image to an average value M + standard deviation σ, an average value M, an average value M−standard deviation σ. And binarize each to create 16 binarized segment images. For example, for the target image shown in FIG. 4, as shown in FIG. 5, the average value M + standard deviation σ is the first threshold value, and pixels whose density is larger than the first threshold value are white pixels and small. A binarized segmented image 5a is generated by binarizing the Y plane image with pixels as black pixels. Further, a binary value obtained by binarizing the Y plane image using the average value M as the second threshold value, the density being larger than the second threshold value, the pixels below the first threshold value being white pixels, and the others being black pixels. The segmentation image 5b is created. Further, the average value M−standard deviation σ is set as the third threshold value, and the Y plane image is binary with the density larger than the third threshold value and the pixels equal to or lower than the second threshold value as white pixels and the other as black pixels. A binarized segmented image 5c is created. Also, a binarized segmented image 5d is generated by binarizing the Y plane image, with pixels having a density equal to or lower than the third threshold as white pixels and pixels larger than the threshold as black pixels.

  Similarly, the control device 104 binarizes the Cb plane image using the first threshold value, the second threshold value, and the third threshold value, and binarizes the segmented images 5e, 5f, 5g, and 5h. Create Similarly, the control device 104 binarizes the Cr plane image using the first threshold value, the second threshold value, and the third threshold value, thereby binarizing the divided images 5i, 5j, 5k, and 5l. Create Similarly, the control device 104 binarizes the Y-complement (Yc) plane image using the first threshold value, the second threshold value, and the third threshold value, thereby binarizing the segmented images 5m and 5n. 5o and 5p are created. Through the above processing, 16-segment binarized segment images 5a to 5p are generated.

  Thereafter, the process proceeds to step S550, and the control device 104 sets a subject position estimation point where the subject is estimated to be located in the target image. For example, the control device 104 prompts the user to specify the subject position, and sets the subject position designated by the user as the subject position estimation point accordingly. Alternatively, the control device 104 prompts the user to specify an AF position for performing focus adjustment, and sets the AF instruction position specified by the user as a subject position estimation point in response to this. Alternatively, the control device 104 sets the center position of the target image as the subject position estimation point. Thereby, the approximate position of the subject in the target image can be estimated, and the extraction accuracy of the subject from the target image can be improved. Thereafter, the process proceeds to step S560.

  In step S560, the control device 104 prompts the user to input the approximate size of the subject in the target image in order to further improve the accuracy of extracting the subject from the target image. For example, the control device 104 presents the subject size candidates to the user in about three stages, and asks the user to select a candidate closest to the subject size in the target image from the candidates. To do. Thereafter, the process proceeds to step S600. Note that the processing from step S600 to step S700 is executed for each of the 16 binarized segmented images 5a to 5p obtained by binarization in step S540. That is, the processes in steps S600 to S700 are processed in a loop of 16 times for each binarized segmented image. In the following description, the case where the processing of step S600 to step S700 is executed on the binarized section image 5a as a representative will be described, but the same processing is performed on the other binarized section images 5b to 5p. Is done.

  In step S600, the control device 104 performs a labeling process on the binarized segmented image 5a. Specifically, the control device 104 extracts an area composed of a group of white pixels from the binarized segmented image as a labeling area, and uses an identifier, for example, a number for identifying each labeling area. Append. For example, when three labeling regions 6a, 6b, and 6c are extracted as shown in FIG. 6, 1 to 3 for each labeling region as shown in FIG. Add a number. In the present embodiment, a labeling region including white pixels extracted by the labeling process is referred to as “island”.

  Thereafter, the process proceeds to step S610, where the control device 104 calculates the area of each island extracted from the binarized segment image 5a, and proceeds to step S620. In step S620, the control device 104 uses the area (white pixel area) of each island calculated in step S610 as an evaluation value, and sets the white pixel area and the approximate size of the subject input by the user in step S560. Based on the islands in the binarized divided image 5a, the islands that are unlikely to constitute the subject are identified and excluded from the subsequent processing targets. Specifically, the control device 104 calculates the area of the island calculated in step S610 out of the islands extracted from the binarized segmented image 5a and the approximate size of the subject input by the user in step S560. Exclude islands with a difference greater than or equal to a predetermined value from the target of subsequent processing. In the processing after step S630, the control device 104 executes processing for islands other than the islands excluded here.

Thereafter, the process proceeds to step S630, and the control device 104 sets a rectangle that encloses each island for each island in the binarized segmented image 5a. Then, when the island in the envelope rectangle is faint or uneven, the control device 104 is unlikely to form a subject, and therefore, according to the following equation (4), A second evaluation value for identifying an island that is unlikely to constitute is calculated.
Second evaluation value = island area / envelope rectangle area (4)

  Thereafter, the process proceeds to step S640, and when the shape of the island is elongated, the control device 104 calculates the aspect ratio for each of the envelope rectangles set in step S630 because the island is unlikely to constitute a subject. The process proceeds to step S650 as a third evaluation value for identifying an island having a low possibility of constituting the subject. In step S650, the control device 104 has a low possibility that the islands at both ends of the binarized segmented image 5a constitute a subject. Therefore, in order to identify such an island, the island is included in the binarized segmented image 5a. A fourth evaluation value indicating the proportion of the end point is calculated, and the process proceeds to step S660. For example, in the binarized section image 5m in FIG. 5, since the islands are on both ends of the upper part of the image, such islands are specified by the fourth evaluation value.

  In step S660, the control device 104 configures a subject from each island in the binarized divided image 5a using the above-described second evaluation value, third evaluation value, and fourth evaluation value. Exclude less likely islands from further processing. Specifically, among the islands in the binarized segmented image 5a, the control device 104 calculates the area of the island / envelope rectangle, which is the second evaluation value, smaller than a predetermined value. An island in which the aspect ratio of the envelope rectangle that is the evaluation value is equal to or greater than a predetermined ratio and the ratio of the island that is the fourth evaluation value to the end point of the binarized segmented image 5a is equal to or greater than the predetermined ratio. Excluded from subsequent processing. In the processing after step S670, the control device 104 executes processing for islands other than the islands excluded here. Thereafter, the process proceeds to step S670.

In step S670, the control device 104 targets the islands other than those excluded in the above processing, and the inertia moment (the inertia moment around the subject position estimation point) around the subject position estimation point in the binarized segmented image 5a. Is calculated. The calculation method of the moment of inertia in the binarized segmented image 5a is well known and will not be described in detail. For example, it can be calculated by the following equation (5). Thereafter, the process proceeds to step S680.

In step S680, the control device 104 calculates the first pixel area by the following equation (6) based on the white pixel area calculated in step S610, that is, the island area, and the moment of inertia around the subject position estimation point calculated in step S670. An evaluation value of 1 is calculated. The first evaluation value calculated here is a value representing a grouping state of white pixels in the island (white pixel region).
First evaluation value = white pixel area / moment of inertia around subject position estimation point (6)

  Thereafter, the process proceeds to step S690, and the control device 104 secondly selects the island having the largest first evaluation value (first island) calculated in step S680 from among the islands in the binarized segment image 5a. A large island (second island) is specified, and these are set as island candidates constituting the subject in the target image.

  Thereafter, the process proceeds to step S700, and the control device 104 determines whether or not the processing of steps S600 to S690 has been completed for all the binarized segment images (masks). If a negative determination is made in step S700, the process returns to step S600. In contrast, if an affirmative determination is made in step S700, the process proceeds to step S710. As a result, the first island and the second island are specified for all the 16 binarized segment images shown in FIG.

  In step S710, the control device 104 ranks the first island and the second island specified for the 16-segment binarized segment image, that is, a total of 32 islands based on the first evaluation value, A plurality of islands with higher ranks, for example, first to third islands are identified as islands constituting the subject in the target image, and the process proceeds to step S720. For example, for each binarized segment image 5a to 5p shown in FIG. 8, when a first evaluation value as shown below each binarized segment image is calculated, the first evaluation value is The top three, that is, the first binarized segment image 5h, the second binarized segment image 5i, and the third binarized segment image 5f are specified.

  In step S720, when the plurality of islands specified in step S710 are adjacent to each other and are combined to form one subject, the control device 104 combines these islands to specify the shape of the subject. To do. On the other hand, if the plurality of islands identified in step S710 are not adjacent, the island having the largest first evaluation value among the islands identified in step S710 is identified as the island constituting the subject, and the identification is performed. Based on the result, the position, size, and shape of the subject in the target image are specified. That is, in the example shown in FIG. 8, the island 8a existing in the first binarized segmented image 5h is specified as the island constituting the subject.

  Thereafter, the process proceeds to step S730, and the control device 104 specifies the position of the center of gravity of the subject in the target image based on the shape of the subject specified in step S720, and information indicating the specified position of the center of gravity, for example, coordinates of the position of the center of gravity The value is recorded in a memory (not shown). In addition, the control device 104 records information for specifying the island used for specifying the subject shape, for example, a number (mask No.) added to each island by the labeling process in step S600 in a memory (not shown). Thereafter, the process returns to the process shown in FIG.

  With the above processing, the first frame is set as the target image, and the position, size, and shape of the subject in the target image name can be specified. Thereafter, the processing for the second and subsequent images is executed by the processing after step S20 in FIG. In step S20, the control apparatus 104 reads the next frame of the image continuously input from the image sensor 103 as a target image, and proceeds to step S30. The processing from step S30 to step S50 is the same as the processing from step S520 to step S540 described above with reference to FIG.

  Thereafter, the process proceeds to step S60, and the control device 104 extracts an image in a predetermined range including the subject position in the previous frame from the target image. In the present embodiment, the processing speed is increased by limiting the execution target of subsequent processing to the image in the range extracted in step S60. However, the process may be performed on the entire target image without performing the process of step S60. Thereafter, the process proceeds to step S70. The processing from step S70 to step S130 is the same as the processing from step S600 to step S660 described above with reference to FIG.

  In step S140, the control device 104 calculates an inertial moment (inertia moment around the subject gravity point) centered on the center of gravity of the island used to identify the subject position, for example, the subject shape, recorded in the memory in the previous frame. ) And the process proceeds to step S150. The process in step S150 is the same as the process in step S670 described above with reference to FIG.

  Thereafter, the process proceeds to step S160, and the control device 104 stores the island having the maximum first evaluation value calculated in step S150 in the memory as a composite area. Thereafter, the process proceeds to step S170, and the control device 104 determines whether or not the processing of steps S60 to S160 has been completed for all the binarized segment images (masks). If a negative determination is made in step S170, the process returns to step S60. On the other hand, if a positive determination is made in step S170, the process proceeds to step S180.

  In step S180, the control device 104 displays the composite area recorded in the memory in step S160 on the monitor 106, and proceeds to step S190. In step S190, information on the center of gravity position of the island recorded as the composite area in step S160, for example, the coordinate value of the center of gravity is stored in the memory. Thereafter, the process proceeds to step S200, and the control device 104 determines whether or not the current frame is the last frame of the frames input from the image sensor 103 in time series. If a negative determination is made in step S200, the process returns to step S20 and is repeated. On the other hand, when an affirmative determination is made in step S200, the process ends.

According to the present embodiment described above, the following operational effects can be obtained.
(1) The control device 104 sets the first frame of frames continuously input from the image sensor 103 as a target image, and is divided into 16 sections based on color information (Cb, Cr) or luminance information (Y) of the target image. A binarized segmented image is created, and an evaluation value used to identify a subject in the target image is calculated for each of these binarized segmented images. Based on the calculated evaluation value, The subject in was specified. Thereby, the position, size, and shape of the subject in the target image can be specified with high accuracy.

(2) In the second and subsequent frames, the control device 104 specifies the subject in the frame based on the subject position of the previous frame and the first to fourth evaluation values. Thus, the processing speed can be improved because the subject position of the current frame is specified taking the subject position of the previous frame into account.

(3) The evaluation value is calculated based on a white pixel area composed of white pixels in the binarized segmented image, that is, an area of an island and a value representing a grouping state of white pixels in the white pixel area. The second evaluation value calculated based on the first evaluation value and the area of the envelope rectangle enclosing the group of white pixels in the binarized segmented image, the third calculated based on the aspect ratio of the envelope rectangle And a fourth evaluation value indicating the ratio of the white pixel area covering the end points of the binarized segmented image. In this way, after excluding islands with shapes and sizes that are not possible for normal subjects from the processing target in advance, white pixels within the islands are taken into account and the subject's position, size, and shape are specified with high accuracy. can do.

-Modification-
The camera according to the above-described embodiment can be modified as follows.
(1) In the above-described embodiment, an example has been described in which the control device 104 receives designation of a subject position in a frame by a user and sets a subject position estimation point in a target image. However, the control device 104 accepts the specification of the subject position in the frame by the user, sets the subject position estimation point, and based on the set subject position estimation point, from among the 16 segmented binarized segment images, A binarized segment image in which the subject position estimation point is included in the white pixel area is specified, and a binarized segment image used for specifying the subject is selected from the specified binarized segment image. Also good. As a result, it is possible to narrow down the binary segmented images used for specifying the subject based on the subject position estimation point, so that the processing speed can be improved.

(2) In the above-described embodiment, the control device 104 has described the example of specifying the position, size, and shape of the subject in the target image. However, the control device 104 may specify at least one of the position, size, and shape of the subject in the target image.

(3) In the embodiment described above, the control device 104 uses the island area and the four evaluation values of the evaluation values 2 to 4 to exclude an island that is unlikely to be a subject from the processing target. explained. However, the control device 104 may exclude an island that is unlikely to be a subject from the processing target by using at least one of the four evaluation values.

(4) In the embodiment described above, the control device 104 prompts the user to input the approximate size of the subject in the target image in step S560 in FIG. 3, and in step S620, Based on the area and the approximate size of the subject input by the user, from among the islands in the binarized segmented image, an island that is unlikely to constitute the subject is identified, and the subsequent processing target An example in which the accuracy of extracting a subject from the target image is further improved by excluding it has been described. However, the control device 104 estimates the size of the subject in the frame by the following method without allowing the user to input the approximate size of the subject in the target image, and determines the subject from the target image. You may make it improve extraction accuracy. That is, the control device 104 generates a difference image by taking a difference between frames continuously input from the image sensor 103, and determines the size of the subject in the frame based on the size and shape of the generated difference image. The shape may be estimated.

(5) In the above-described embodiment, the case where the present invention is applied to a camera has been described. However, the present invention can also be applied to other devices that can read and process images, such as personal computers and portable terminals.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired. Moreover, it is good also as a structure which combined the above-mentioned embodiment and a some modification.

100 Camera, 101 Operation member, 102 Lens, 103 Image sensor, 104 Control device, 105 Memory card slot, 106 Monitor

Claims (6)

A classification procedure for selecting a target image from continuously input frames and dividing one image into a plurality of divided images based on color information or luminance information of the target image;
A binarization procedure for binarizing each of the plurality of segmented images using the color information or the luminance information to generate a plurality of binarized segment images;
An evaluation value calculation procedure for calculating an evaluation value used for specifying a subject in the target image for each of the plurality of binarized segment images;
A subject specifying procedure for specifying a subject in the target image based on the evaluation value;
A storage procedure for storing the position of the subject in the target image identified by the subject identification procedure;
A program for specifying a subject for causing a computer to execute a tracking procedure for tracking the subject specified by the subject specifying procedure between the continuously input frames. In the subject specifying program according to claim 1,
The subject specifying procedure is based on the position of the subject in the target image stored in the storing procedure and the evaluation value calculated in the evaluation value calculating procedure in frames continuously input after the target image. An object specifying program for specifying an object in the frame. In the subject specifying program according to claim 1 or 2,
The evaluation value is a first evaluation value calculated based on an area of a white pixel region formed of white pixels in the binarized segmented image and a value representing a grouping state of white pixels in the white pixel region. A second evaluation value calculated based on an area of an envelope rectangle enclosing a group of white pixels in the binarized segmented image, a third evaluation value calculated based on an aspect ratio of the envelope rectangle, And at least one fourth evaluation value indicating a ratio of the white pixel region covering the end points of the binarized segmented image. The subject specifying program according to claim 3,
A reception procedure for receiving designation of a subject position in the frame by a user;
The subject specifying procedure specifies a binarized segment image in which the designated subject position received in the acceptance procedure is included in the white pixel region from the plurality of binarized segment images, and the identified binary A program for identifying a subject, wherein a binarized segmented image used for identifying a subject in the frame is selected from the segmented segmented images. A difference image generation procedure for generating a difference image by taking a difference between consecutively input frames;
An estimation procedure for estimating the size and shape of a subject in a frame based on the size and shape of the difference image generated in the difference image generation procedure;
A subject specifying program for causing a computer to execute a subject specifying procedure for specifying a subject in a frame based on an estimation result obtained by the estimation procedure.   6. A camera comprising execution means for executing the subject specifying program according to claim 1.

Images