JP2013037522A - Object tracking program and object tracking device - Google Patents

Abstract

An object of the present invention is to appropriately track a main subject in a frame image input in time series.
An object tracking program binarizes a plurality of element images and element image generation processing for generating a plurality of element images based on color information and luminance information of each frame image input in time series. Binarization element image generation processing for generating a plurality of binarization element images, AND operation processing for ANDing a plurality of binarization element images, and labeling processing for a binary AND image after the AND operation Specific processing for specifying the position of the main subject in each frame image, reduction processing for reducing the range specified based on the logical product of the range specified by the specification processing and the predetermined range, and the previous frame The computer executes a morphological process for expanding a range reduced by the reduction process in an image to obtain a predetermined range.
[Selection] Figure 13

Description

  The present invention relates to a subject tracking program and a subject tracking device.

  A subject tracking technique is known in which the position of a main subject in each frame is specified by searching for the position of an image having a high similarity to the template image in frame images input in time series (see Patent Document 1). .

JP 2008-299834 A

  In the prior art, the background position other than the main subject is included in the specified position, so that the subject position is unclear.

An object tracking program according to the present invention includes an element image generation process for generating a plurality of element images based on color information and luminance information of each frame image input in time series to a computer, and each of the plurality of element images is binarized. A binarized element image generation process for generating a plurality of binarized element images, a logical product operation process for performing a logical AND operation on the plurality of binarized element images, and a binary logical product image after the AND operation A specifying process for specifying the position of the main subject in each frame image based on the labeling process, and a reducing process for reducing the specified range based on a logical product operation of the range specified by the specifying process and the predetermined range; Morphological processing for expanding a range reduced by the reduction processing in the previous frame image to obtain a predetermined range is performed.
A subject tracking program according to the present invention is a search region in which a computer determines, as a search region, a morphological process for expanding a binary image of a main subject in a template and a range enclosing the binary image after expansion by the morphological process. A determination process, and a similarity calculation process for calculating the similarity between the image in the search frame and the template at each search frame position while moving the search frame in the search area among the frame images input in time series And a specifying process for specifying the position of the search frame with the highest similarity as the position of the main subject.

  According to the present invention, it is possible to appropriately track a main subject in a frame image input in time series.

It is a block diagram which illustrates the composition of the camera in a first embodiment. It is a flowchart which shows the process of the control apparatus in 1st embodiment. It is a figure which illustrates the binarized Y plane image. It is a figure which illustrates the binarized Cb plane image. It is a figure which illustrates the binarized Cr plane image. It is a figure which illustrates the binarized image after a 1st AND operation. It is a figure which illustrates the binarized image by a prior art. It is a figure which illustrates the binarized image after an expansion process. It is a figure which illustrates the binarized image after a 2nd AND operation. It is a figure which illustrates the target frame displayed on the frame image. It is a figure which illustrates the range containing the main subject. It is a figure which illustrates a template. It is a figure explaining template matching. It is a figure which illustrates background noise. 4 is a diagram illustrating an expansion mask D. FIG. It is a figure explaining background noise cut. It is a figure explaining the search area | region about the frame which envelopes the expansion mask D. FIG. It is a figure which shows the position of the first template in a search area | region. It is a figure which shows the position of the last template in a search area | region. It is a figure which shows the movement amount of the template at the time of one matching calculation. It is a flowchart which shows the process of the control apparatus in 2nd embodiment.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a block diagram illustrating the configuration of an embodiment when a camera is used as the subject tracking device in the first embodiment. In the first embodiment, a subject is tracked using a labeling method. The camera 100 includes an operation member 101, a lens 102, an image sensor 103, a control device 104, a memory card slot 105, and a monitor 106. The operation member 101 includes various input members operated by the user, such as a power button, a release switch, a zoom button, a cross key, a determination button, a playback button, and a delete button.

  The lens 102 is composed of a plurality of optical lens groups, but is represented by a single lens in FIG. The image sensor 103 is configured by a CCD or a CMOS image sensor, for example, and acquires an image by capturing a subject image formed by the lens 102. The acquired image data (image data) is output to the control device 104. The control device 104 compresses the image data acquired by the image sensor 103 into a predetermined image format, for example, JPEG format, and generates an image file in a predetermined format such as Exif (Exchangeable Image File Format Digital Still Camera). Output to the memory card slot 105.

  The memory card slot 105 is a slot for inserting a memory card as a storage medium, and the image file output from the control device 104 is written and recorded on the memory card. Further, based on an instruction from the control device 104, an image file stored in the memory card is read.

  The monitor 106 is a liquid crystal monitor (rear monitor) mounted on the back surface of the camera 100, and the monitor 106 displays an image stored in a memory card, a setting menu for setting the camera 100, and the like. . Further, the control device 104 acquires images in time series from the image sensor 103 and outputs them to the monitor 106. As a result, the image of each frame is sequentially displayed on the monitor 106 at predetermined time intervals. That is, an image is displayed on the monitor 106.

  The control device 104 includes a CPU, a memory, and other peripheral circuits, and functionally includes a calculation unit 104a described later. Note that the memory constituting the control device 104 includes a memory for storing a program, a work memory, and a memory used as a buffer memory. For example, flash memory, RAM, SDRAM and the like.

  FIG. 2 is a flowchart showing the processing of the camera 100 in the first embodiment. The processing shown in FIG. 2 is executed by the arithmetic unit 104a in the control device 104 as a program that starts when image input from the image sensor 103 is started.

  In step S51 of FIG. 2, the calculation unit 104a of the control device 104 reads the frame image and proceeds to step S52. In step S52, the calculation unit 104a extracts an image within a predetermined range from the read frame image based on an instruction from the user, and proceeds to step S53.

  In step S53, the calculation unit 104a calculates an image in the YCbCr format based on the extracted image, and outputs a Y component image (Y plane image), a Cr component image (Cr plane image), and a Cb component image (Cb Each plane image) is generated. Specifically, a target image represented by the RGB color system is represented by a luminance image composed of luminance components (Y components) in a YCbCr color space and color difference components (Cb components, To a color difference image composed of (Cr component).

That is, the calculation unit 104a generates a luminance image composed of a Y component as a Y plane image from the extracted image using the following expression (1). The computing unit 104a further generates a color difference image composed of a Cb component and a color difference image composed of a Cr component as a Cb plane image and a Cr plane image from the extracted image using the following equations (2) and (3), respectively. Proceed to S54.
Y = 0.299R + 0.587G + 0.114B (1)
Cb = −0.169R−0.332G + 0.500B (2)
Cr = 0.500R−0.419G−0.081B (3)

  In step S54, the calculation unit 104a performs binarization calculation. For example, in the Y plane image, the Cb plane image, and the Cr plane image, the calculation unit 104a calculates a binarized image obtained by binarizing each average value as a threshold value, and the process proceeds to step S55. FIG. 3 is a diagram illustrating a binarized Y plane image. FIG. 4 is a diagram illustrating a binarized Cb plane image. FIG. 5 is a diagram illustrating a binarized Cr plane image. The threshold used for binarization may be a value obtained by adding and subtracting each average value obtained by multiplying the standard deviation σ by a coefficient.

  In step S55, the operation unit 104a performs a first AND operation. Specifically, each of the binarized plane images is subjected to a logical product operation for each pixel, and the process proceeds to step S56. FIG. 6 is a diagram illustrating a binarized image obtained by performing a first AND operation on the binarized Y plane image, Cb plane image, and Cr plane image.

  In step S56, the calculation unit 104a performs a known labeling process on the binarized image after the first AND operation, and recognizes a labeling region including white pixels as an “island”. The selection process selects an island closer to the subject by a predetermined evaluation process from the extracted labeling areas.

  In step S57, the calculation unit 104a determines whether it is the first frame. If not the first frame (that is, the second and subsequent frames), the arithmetic unit 104a makes a negative determination in step S57 and proceeds to step S58. In the case of the first frame, the arithmetic unit 104a makes a positive determination in step S57 and proceeds to step S59.

  In step S58, the operation unit 104a performs a second AND operation. Specifically, a logical product operation is performed on the binarized image after the labeling and selection process and the binarized image (FIG. 8) that has been subjected to the dilation process in the previous frame, and the process proceeds to step S59. FIG. 9 is a diagram illustrating a binarized image after the second AND operation. In FIG. 9, a frame surrounding a group of white pixels is a target frame for clearly indicating a range where the main subject exists.

  In step S <b> 59, the arithmetic unit 104 a performs expansion processing based on morphology on the binarized image after the second AND. FIG. 8 is a diagram illustrating a binarized image after the expansion process. Note that the arithmetic unit 104a when an affirmative determination is made in step S57 performs an expansion process by morphology on the binarized image of the first frame.

  In step S60, the calculation unit 104a displays the target frame on the frame image, and proceeds to step S61. FIG. 10 is a diagram illustrating the target frame 151 on the frame image. The target frame 151 corresponds to an envelope frame surrounding a white region that is an “island” indicating a main subject in the binarized image. In the first embodiment, a range surrounded by the target frame 151 is a tracking result (a range where the main subject exists).

  In step S61, the calculation unit 104a determines whether or not it is the end of the frame. When the image input is completed, the calculation unit 104a makes a positive determination in step S61 and ends the process of FIG. If the input of the image has not ended, the arithmetic unit 104a makes a negative determination in step S61 and returns to step S51. When returning to step S51, the next frame is read and the above-described processing is repeated.

According to the first embodiment described above, the following operational effects can be obtained.
(1) The camera 100 performs element image generation processing for generating a plurality of plane images Y, Cb, Cr based on color information and luminance information of each frame image input in time series, and a plurality of plane images Y, Cb. , Cr is binarized to generate a plurality of binarized plane images, a first AND operation to AND a plurality of binarized plane images, and a first AND operation Based on the labeling processing for the binary image, and the labeling and selection processing for specifying the position of the main subject in each frame image, and the logical product operation between the range specified by the labeling and selection processing and the predetermined range. A second AND operation process for reducing the range specified in the above and an expansion process for expanding the range reduced by the reduction process in the previous frame image to obtain a predetermined range. Because mounting the subject tracking program to be executed by 04a, it can track properly main subject in the frame image input to the time series.

  A comparative example between the above-described binarized image after the second AND operation (FIG. 9) and the case where the second AND operation is not performed will be described. FIG. 7 is a diagram illustrating a binarized image according to the prior art in which labeling and selection processing are performed on the binarized image (FIG. 6) after the first AND operation. Comparing FIG. 7 and FIG. 9, the binarized image after the second AND operation (FIG. 9) has a smaller range in which the main subject exists (that is, a group of white pixels), and the main subject is accurately detected. You can trace. Since the range in which the main subject exists (that is, a group of white pixels) can be reduced, the possibility of including background noise other than the main subject is suppressed to a low level.

(2) In the subject tracking program of (1), the element image generation process generates the luminance image Y and the color difference images Cb and Cr, respectively, so that an element image suitable for main subject tracking can be obtained.

(Second embodiment)
In the second embodiment, the control unit 104a performs template matching processing on each frame image input from the image sensor 103, and specifies an area in which the main subject is shown in the frame image. Then, the specified area is traced between frames. Specifically, the process is as follows.

  FIG. 11 is a diagram illustrating a frame image displayed on the monitor 106. For example, when the through image of the first frame image is displayed on the monitor 106, the user operates the operation member 101 to track a subject (referred to as main subject A) to be tracked between frames within the first frame. A range E including The computing unit 104a extracts an image within the range E designated by the user and stores it in the memory as a template T. FIG. 12 is a diagram illustrating a template T.

  As illustrated in FIG. 13, the calculation unit 104 a performs a matching calculation between the template T and image data of each frame (referred to as a target image I) input in time series from the image sensor 103. Specifically, the main subject position in the target image I is specified using the template T.

Although the template matching method is well known and will not be described in detail, the similarity can be calculated by, for example, the residual sum shown in the following equation (4) or the normalized correlation shown in the following equation (5). When the similarity is calculated by the residual sum shown in the following equation (4), the smaller the calculated r is, the higher the similarity between the template T and the target image I is. On the other hand, when the similarity is calculated by the normalized correlation shown in the following equation (5), the larger the calculated r is, the higher the similarity between the template T and the target image I is.

  When the input of the image from the image sensor 103 is started, the calculation unit 104a sets the image of each frame (frame) as the target image I, and searches for the same size as the template T at a predetermined position on the target image I. Set frame B. The calculation unit 104a performs matching calculation between the image in the search frame B at each position and the template T while moving the set search frame B in the target image I. As a result of the matching calculation, the position of the search frame B having the highest similarity with the template T is specified as the subject position.

  Note that the calculation unit 104a of the second embodiment performs template matching not on the entire target image I but on a predetermined range (referred to as a search region S2) including a range E in which the template T is extracted from the frame image. . The reason why the size of the search area S2 is made smaller than the target image I in this way is to avoid the processing time required for the entire screen and the fact that pseudo matching with the background tends to occur. How to determine the size of the search area S2 will be described later.

  By the way, in the template T including many parts other than the main subject A as shown in FIG. 12, background noise is likely to be mixed as shown in FIG. 14 depending on the state of the target image I used during the matching calculation, and matching accuracy may not be obtained. Many. Therefore, as shown in FIG. 15, an expansion mask D is set in which the main subject A is subjected to expansion processing of a known morphology (mathematical morphology). That is, the template T is binarized and the binarized image is expanded to obtain the expansion mask D. In the expansion mask D, as shown in FIG. 15, the inner pixels are all 1 and the outer pixels are all 0. By performing an AND operation on the expansion mask D and the target image I, the background noise included in the target image I is cut. FIG. 16 is a diagram for explaining background noise cut. According to FIG. 16, only the image inside the expansion mask D (pixel 1) remains in the target image I, and the image including background noise outside (pixel 0) is deleted.

  As illustrated in FIG. 17, the calculation unit 104a in the case of employing the expansion mask D sets a frame that envelops the expansion mask D around the main subject A as a search region S2. FIG. 18 shows the position of the first template T in the search area S2, and FIG. 19 shows the position of the last template T in the search area S2. The portion corresponding to the outside of the expansion mask D in the template T, that is, the lower right and upper left portions of the template T can be excluded from the target of the matching calculation.

  FIG. 20 shows the movement amounts X2 and Y2 of the template T during one matching calculation. In the second embodiment employing the expansion mask D, the size of the search area S2 is uniquely determined. Experiments have shown that the size of the search area S2 by the expansion mask D is often smaller than the search area used in the conventional matching calculation. By introducing the expansion mask D in this way, not only the background noise can be reduced but also the matching operation can be shortened.

  The computing unit 104a performs the above-described processing on each input frame image. The calculation unit 104a sets a position having the highest similarity in matching among the main subject positions specified by the matching calculation as the subject position. A target frame can be displayed with the subject position as a tracking target, and subject tracking can be performed between the frames.

  FIG. 21 is a flowchart showing the processing of the camera 100 in the second embodiment. The processing shown in FIG. 21 is executed by the arithmetic unit 104a of the control device 104 as a program that starts when image input from the image sensor 103 is started.

  In step S11 of FIG. 21, the calculation unit 104a reads a frame image and proceeds to step S12. In step S12, the calculation unit 104a extracts an image within a predetermined range from the frame image based on an instruction from the user, and proceeds to step S13.

  In step S13, the calculation unit 104a determines whether or not it is the first frame. If not the first frame (that is, the second and subsequent frames), the arithmetic unit 104a makes a negative determination in step S13 and proceeds to step S14. In the case of the first frame, the arithmetic unit 104a makes a positive determination in step S13 and proceeds to step S18.

  In step S14, the calculation unit 104a performs binarization calculation. The computing unit 104a calculates a binarized image obtained by binarizing the image within the predetermined range, and proceeds to step S15. In step S15, the calculation unit 104a performs a morphological expansion process on the binarized image, sets an expansion mask D, and proceeds to step S16. In step S16, the calculation unit 104a sets the frame enveloping the expansion mask D as the search area S2 and proceeds to step S17. In step S17, the calculation unit 104a performs a matching calculation and proceeds to step S18.

  In step S18, the calculation unit 104a displays a target frame on the frame image for clearly indicating the range in which the main subject exists, and proceeds to step S19. In step S19, the calculation unit 104a determines whether or not it is the end of the frame. The arithmetic unit 104a makes a positive determination in step S19 when the input of the image is completed, and ends the process of FIG. If the image input has not ended, the arithmetic unit 104a makes a negative determination in step S19 and returns to step S11. When returning to step S11, the next frame is read and the above-described processing is repeated.

According to the second embodiment described above, the following operational effects can be obtained.
(1) The camera 100 determines a search area in which a morphological process for expanding the binarized image of the main subject A in the template T and a range enveloping the binarized image after the expansion by the morphological process are set as the search area S2. Processing and similarity for calculating similarity between the image in the search frame B and the template T at each search frame position while moving the search frame B in the search region S2 among the frame images input in time series Since the subject tracking program that causes the control device 104 to execute the calculation process and the specifying process for specifying the position of the search frame B with the highest similarity as the position of the main subject is installed, in the frame image input in time series The main subject can be tracked appropriately.

(2) The subject tracking program of (1) further includes an expansion mask process for extracting an image from a frame image based on a binarized image after expansion, and the similarity calculation process is performed in time series. Since the search frame B is moved in the search area S2 in the frame image, the similarity between the image after the expansion mask processing and the template T in the search frame B at each search frame position is calculated. Even if background noise other than the main subject A is included, the background noise can be cut and the main subject A can be tracked appropriately.

(Modification)
In the above-described embodiment, an example in which the control device 104 of the camera 100 performs the subject tracking process has been described. However, the subject tracking process may be configured to be performed by a computer. The subject tracking device is configured by causing a computer to execute a program that performs processing based on the flowcharts illustrated in FIGS. 2 and 21. When a program is used by being taken into a computer, the program is executed after being loaded into a data storage device of the computer.

  The program may be loaded into the computer by setting a storage medium such as a CD-ROM storing the program in the computer, or may be loaded into the computer by a method via a communication line such as a network. When passing through a communication line, the program is stored in a storage device of a server (computer) connected to the communication line. The program can be supplied as various forms of computer program products such as provision via a storage medium or a communication line.

  The above description is merely an example, and is not limited to the configuration of the above embodiment.

DESCRIPTION OF SYMBOLS 100 ... Camera 101 ... Operation member 104 ... Control apparatus 104a ... Calculation part 106 ... Monitor

Claims (5)

On the computer,
Element image generation processing for generating a plurality of element images based on color information and luminance information of each frame image input in time series;
Binarization element image generation processing for binarizing each of the plurality of element images to generate a plurality of binarization element images;
AND operation processing that performs an AND operation on the plurality of binarized element images;
A specifying process for specifying the position of the main subject in each frame image based on a labeling process for the binary AND image after the AND operation;
A reduction process for reducing the specified range based on a logical product operation of the range specified in the specifying process and a predetermined range;
Morphology processing for expanding the range reduced by the reduction processing in the previous frame image to obtain the predetermined range;
A subject tracking program characterized by causing The subject tracking program according to claim 1,
The subject image generation process generates a luminance image and a color difference image, respectively. On the computer,
Morphological processing for expanding the binarized image of the main subject in the template;
A search region determination process for determining a search region as a range for enveloping the binary image after expansion by the morphology processing;
A similarity calculation process for calculating the similarity between the image in the search frame at each search frame position and the template while moving the search frame in the search region among the frame images input in time series,
A specifying process for specifying the position of the search frame having the highest similarity as the position of the main subject;
A subject tracking program characterized by causing In the subject tracking program according to claim 3,
Further comprising expansion mask processing for extracting an image from the frame image based on the binarized image after expansion,
The similarity calculation processing is performed by moving the search frame in the search area among the frame images input in time series, and the image after the expansion mask processing in the search frame at each search frame position and the template A subject tracking program characterized by calculating a similarity to   An object tracking apparatus comprising execution means for executing the object tracking program according to claim 1.

Images