JP2011210238A - Advertisement effect measuring device and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an advertisement effect measuring device capable of accurately measuring an audience rate for the number of persons viewing a display by measuring not only the number of persons browsing the display but also the number of persons in front of the display.SOLUTION: The advertisement effect measuring device 2 outputs at least an audience rate of a digital signage 3 as an index of the advertisement effect of the digital signage 3 by analyzing a frame contained in an image transmitted from a video camera 4 at predetermined time intervals, detecting a human body area from the frame, and further detecting a face area from the human body area.

Description

  The present invention relates to a technique for measuring the advertising effectiveness of a display. The display includes a digital signage (Digital Signage), which is an advertising medium for displaying an advertisement using a liquid crystal display or a projector, as well as a show window, an advertisement signboard, a specific product display shelf in a store, and the like.

  Digital signage, which is an advertising medium that displays advertisements using a display such as a liquid crystal display or a projector, has begun to be installed in various places. By using digital signage, not only will it be possible to provide rich content using video and audio, but it will also be possible to efficiently deliver advertisements according to the location of digital signage. Expansion is expected.

  When an advertiser introduces digital signage, it is necessary to present an objective advertising effect to the advertiser, such as the TV audience rating, the number of copies of newspapers and magazines, etc. As inventions, for example, Patent Literature 1 and Patent Literature 2 are disclosed.

  In the invention disclosed in Patent Document 1, a human face rectangular area existing in an image of an area where digital signage is installed is recognized, and a person's line-of-sight direction is extracted from the recognized face rectangular area. As the advertisement effect of signage, the number of viewers browsing the advertisement display device and the attributes (for example, sex, age) of the viewer obtained from the recognized face rectangular area are measured.

  The invention disclosed in Patent Document 2 tracks an individual who has been in an area where digital signage is installed without recognizing the rectangular area of the viewer's face, and displays an advertisement display device based on the stay time in the area. It is an invention for measuring the number of people who pay attention to.

JP 2008-112401 A Japanese Patent No. 3800257

  One of the important indicators of the advertising effect is a view rate indicating the percentage of people who are viewing the display among the people in front of the display. In the invention disclosed in Patent Document 1, the face of a person is considered. It is impossible to measure the visibility rate because only the detection is detected.

  In the invention disclosed in Patent Document 2, a person is photographed from above and the direction in which the person is facing is estimated from the brightness of hair and face and the characteristics of eyes and mouth. If this is the case, there is a problem that it is impossible to specify the exact number of viewers and the credibility of indicators such as the viewing rate measured as the advertising effect is lost.

  Therefore, the present invention provides an advertisement effect measuring device and a computer that can accurately measure the number of people who are watching the display by measuring not only the number of people viewing the display but also the number of people in front of the display. The purpose is to provide a program.

  According to a first aspect of the present invention for solving the above-described problem, a human body detecting means for detecting a position / size of a human body rectangular area from a frame constituting an image captured by a camera that captures a person in front of a display, and the human body rectangle The human body rectangular area by associating the human body rectangular area in the preceding and succeeding frames with the face detection means for detecting the position and size of the face rectangular area from the image in the area, and using the feature amount of the image in the human body rectangular area Tracking means for storing the tracking log including the position / size of the human rectangular area, the date and time when the frame was captured, and the detection result of the facial rectangular area for each human rectangular area to be tracked, By counting the tracking logs, the total number of the human body rectangular areas to be tracked and the face rectangular area An advertisement effect measuring device comprising an advertisement effect measuring means for obtaining the number of the human body rectangular regions in which an area is detected and outputting a visual recognition rate as an index related to the advertisement effect.

  According to the first invention, the advertisement effect measuring device includes the human body detecting means and the face detecting means, so that the advertisement effect measuring device can display the person for each person included in the frame. The tracking log indicating whether or not the user has browsed can be stored. If the tracking logs are aggregated, not only the number of people who have browsed the display but also the number of people in front of the display can be measured. You can measure the visibility of how many people were watching.

  Further, by detecting the front face rectangular area from the image in the human body rectangular area, it is possible to prevent a part of the background image from being erroneously detected as the front face rectangular area, and accuracy of the visibility rate is reduced. As well as the processing time required for face detection.

  Further, the second invention is the advertising effect measuring device according to the first invention, wherein the tracking means uses a first tracking method using a color feature and a second tracking using a local feature. Corresponding to a technique, and after tracking the human rectangular area by the first tracking technique, the human rectangular area that could not be tracked by the first tracking technique is tracked by the second tracking technique. To do.

  According to the second invention, since the shape of the object can be recognized by using the second tracking method, even if the occlusion occurs and the human body rectangular area cannot be tracked by the first tracking method, the tracking is not performed. By tracking the method by the second tracking method, the human body rectangular region can be tracked.

  Furthermore, a third invention is the advertising effect measurement device according to the first invention or the second invention, wherein a foreground indicating a foreground region other than the background image is obtained by obtaining a difference between the frame and the background image. Foreground mask generation means for generating a mask, and the human body detection means obtains a ratio of the foreground area in the foreground mask area corresponding to the human body rectangular area for each human body rectangular area detected from the frame. The human body rectangular area whose ratio is less than a threshold is treated as a false detection, and the tracking means calculates a feature amount of an image in the human body rectangular area from the frame after mask processing using the foreground mask. It is characterized by.

  The position and angle of the camera that shoots the person in front of the display is fixed, and the video captured by the camera includes a background image that does not move in time. As in the third invention, the foreground If the background image can be removed from the frame using a mask, the human body rectangular area and the face rectangular area can be accurately detected.

  Further, a fourth invention is the advertising effect measurement device according to the third invention, wherein the human body detecting means divides the human rectangular area not treated as a false detection into clusters by a designated clustering method, An integration process for integrating the human rectangular regions belonging to the same cluster into one is executed.

  According to the fourth invention described above, a plurality of human body rectangular areas that can be determined as the same person are integrated into a single human body rectangular area even when a plurality of human body rectangular areas are detected for one person. it can. In the third invention, since the human rectangular area is divided into clusters using a foreground mask, even if the human rectangular areas belonging to the same cluster are integrated into one, it is similar to the shape of a person. A region in which an object (for example, a stepladder, a pillar, or the like) is erroneously detected as a person is not integrated.

  Furthermore, a fifth invention is the advertising effectiveness measuring device according to any one of the first to fourth inventions, wherein the face detecting means converts the image in the human body rectangular area to a predetermined size. The face rectangular area is detected after normalizing to.

  According to the fifth aspect described above, even when the size of the detected face rectangular area is small, the face detection target area is small by normalizing the image in the human body rectangular area to a predetermined size. In this case, since the enlargement process is performed, the face rectangular area can be easily detected.

  Further, a sixth invention is the advertising effect measurement device according to the fifth invention, wherein the face detection means normalizes the image in the human body rectangular area after normalizing to a predetermined size. The search size at the start of face detection is determined according to the size of the human body rectangular area.

  Furthermore, according to the sixth aspect of the invention, since the size of the human rectangular area before normalization is known, if the search size at the start of face detection is determined according to the size, the processing speed of face detection is improved. In addition, an effect of preventing erroneous detection of the face can be obtained.

  Furthermore, a seventh invention is the advertising effect measuring device according to any one of the first to sixth inventions, wherein the advertising effect measuring means stores the tracking log for each human body rectangular region. The total time is obtained by tracking the human rectangular area, and the residence time is output for each human rectangular area as an index related to the advertising effect.

  Further, an eighth invention is the advertising effectiveness measuring device according to any one of the first invention to the seventh invention, wherein the advertising effectiveness measuring means records the tracking log for each human body rectangular area. By counting, the time during which the face rectangular area can be detected from the human rectangular area is obtained, and browsing time is output for each human rectangular area as an index related to the advertising effect.

  Furthermore, a ninth invention is the advertising effect measuring device according to any one of the first to eighth inventions, wherein the advertising effect measuring means stores the tracking log for each human body rectangular region. By counting, the position of the human body rectangular area is output for each human body rectangular area as an index related to the advertising effect.

  Furthermore, the tenth invention is the advertising effect measurement device according to any one of the first invention to the ninth invention, wherein the person attribute is specified from the face rectangular area detected by the face detecting means. Human attribute specifying means, wherein the tracking means includes the person attribute in the tracking log of the human body rectangular area where the face rectangular area has been detected, and the advertising effectiveness measuring means includes the tracking log as the human body rectangle. By summing up for each area, a person attribute is output for each of the human body rectangular areas as an index related to the advertising effect.

  In addition, as an index related to the advertising effect, not only the viewing rate but also the dwell time, the viewing time, the position of the human body rectangular area, and the person of the person viewing the display as in the eighth to tenth inventions It is desirable to be able to output attributes (age, gender, etc.).

  Furthermore, an eleventh aspect of the invention is a computer program for causing a computer to function as the advertisement effect measuring apparatus according to any one of the first to tenth aspects of the invention.

  The eleventh invention relates to a computer program.

  According to the above-described invention, it is possible to provide an advertisement effect measuring apparatus and a computer program that can accurately detect a human body and a face included in an image captured by a video camera.

The figure explaining the structure of an advertising effectiveness measurement system. The hardware block diagram of an advertisement effect measuring apparatus. The functional block diagram of an advertisement effect measuring apparatus. The flowchart explaining the process which an advertising effectiveness measurement apparatus analyzes a flame | frame. The figure which illustrates a foreground mask typically. The figure explaining the false detection of a human body rectangular area. The figure explaining the detail of a tracking process. The figure explaining a mask process. The figure explaining the 2nd tracking method. The figure which supplements description of a 2nd tracking method. The flowchart explaining the human body detection process in the modification 1. The figure explaining the effect of the modification 1. FIG. The figure explaining the flow of the face detection process in the modification 2. The figure explaining the effect of the modification 2. FIG.

  From this, the embodiments of the present invention will be described to the extent that those skilled in the art of the present invention can understand the contents of the present invention and implement the present invention.

  In the present embodiment, the present invention is applied to the measurement of the advertising effect of the digital signage 3 installed on the street or the like. FIG. 1 illustrates the configuration of the advertising effect measuring system 1 in which the advertising effect measuring device 2 is installed. It is a figure to do.

  As shown in FIG. 1, the advertisement effect measurement system 1 includes a digital signage 3 that is installed in a street or a store and displays an advertisement video using a display 3a (display in FIG. 1). . The digital signage 3 includes a server for controlling the advertising video reproduced by the digital signage 3 in addition to the display 3a for reproducing the advertising video, but this server is not shown in FIG.

  On the display 3a of the digital signage 3, a video camera 4 having an angle set so that a face of a person who is watching an advertisement image reproduced on the display 3a is photographed is installed.

  The video shot by the video camera 4 is input to the advertisement effect measuring device 2 using a USB port or the like, and the advertising effect measuring device 2 uses the frames included in the video transmitted from the video camera 4 at predetermined intervals. And detecting a human body rectangular area from the frame, and further detecting a face rectangular area from the human body rectangular area, thereby outputting an advertisement effect index of the digital signage 3 (for example, the visibility of the digital signage 3).

  In the apparatus constituting the advertisement effect measurement system 1 illustrated in FIG. 1, commercially available apparatuses can be used for the digital signage 3 and the video camera 4, but the advertisement effect measurement apparatus 2 has features that are not found in the prior art. From here, the advertisement effect measuring apparatus 2 will be described in detail.

  FIG. 2 is a hardware block diagram of the advertisement effect measuring device 2 constituting the advertisement effect measuring system 1, and FIG. 3 is a functional block diagram of the advertisement effect measuring device 2.

  The advertisement effect measuring device 2 can be realized by using a general-purpose computer. The advertisement effect measuring device 2 includes hardware similar to that of a general-purpose computer. In FIG. 2, as the hardware, a CPU 2a (CPU: Central: Processing Unit), ROM 2b (ROM: Read-Only Memory) in which BIOS is mounted, RAM 2c (RAM: Random Access Memory) which is a main memory of a computer, and a large-capacity data storage device 2d (for example, an external storage device) , Hard disk), input / output interface 2e for data communication with external devices, network interface 2f for network communication, display device 2g (for example, liquid crystal display), and character input device 2h (for example, keyboard) , Pointing device 2i (eg mouse) To have.

  A computer program for operating the CPU 2a is installed in the data storage device 2d of the advertisement effect measuring device 2, and the advertisement effect measuring device 2 is provided with the means shown in FIG. 3 by this computer program.

  As shown in FIG. 3, the input of the advertising effectiveness measuring device 2 is an image taken by the video camera 4, and the output of the advertising effectiveness measuring device 2 is stored in a log file in which an index of the advertising effectiveness of digital signage is stored. Become.

  The advertisement effect measuring device 2 obtains a difference between the frame to be processed and the background image by removing the background image from the frame to be processed as means for analyzing the frame of the video captured by the video camera 4. Included in the foreground mask generation means 20 for generating a foreground mask indicating a foreground area that is an area other than the background image, a human body detection means 21 for detecting a human body rectangular area from the frame, and a human body rectangular area detected by the human body detection means 21 Face detection means 22 for detecting a front face rectangular area, and tracking means 23 for tracking a human body rectangular area using a feature amount obtained from an image in the human body rectangular area and storing a tracking log indicating a tracking result. ing.

  Furthermore, in this embodiment, the advertising effectiveness measuring device 2 estimates the person attributes (age and gender) from the image included in the face rectangular area in order to obtain the attributes (age and gender) of the person who viewed the digital signage 3. A person attribute calculating means 24 is provided.

  Further, the advertisement effect measuring device 2 includes an advertisement effect measuring means 25 for outputting a log file obtained by counting the results of the tracking log stored by the tracking means 23 in order to measure the advertisement effect of the digital signage 3. In the present embodiment, the advertisement effect measuring means 25 of the advertisement effect measuring device 2 includes a visibility log file in which the visibility of the digital signage 3 is recorded and a browsing of the digital signage 3 as log files obtained by summing up the results of the tracking log. A browsing time log file in which time is recorded, a position log file in which the locus of the position of the human rectangular area is recorded, and a person attribute log in which the personal attributes (for example, age and gender) of the person who viewed the digital signage are recorded Output a file.

  First, the advertisement effect measuring apparatus 2 is provided to analyze the frame of the video captured by the video camera 4 while explaining the process of the advertising effect measuring apparatus 2 analyzing the frame of the video transmitted from the video camera 4. Each means will be described.

  FIG. 4 is a flowchart for explaining processing in which the advertisement effect measuring apparatus 2 analyzes a frame. The advertisement effect measuring device 2 executes processing for analyzing the video of the video camera 4 at regular intervals, and when the advertising effect measuring device 2 analyzes the video of the video camera 4, first, in the N frame to be processed. Foreground mask generation processing S1 for generating a foreground mask is executed.

  When the foreground mask generation process S1 is executed, the advertisement effect measuring apparatus 2 uses the foreground mask to execute the human body detection process S2 for detecting the human body rectangular area from the N frames, and then detects the human body detected in the human body detection process S2. A face detection process S3 for detecting a face rectangular area from the rectangular area is executed.

  Then, after executing the face detection process S3, the advertisement effect measuring apparatus 2 executes the person attribute calculation process S4 for obtaining the person attribute of the face rectangular area detected in the face detection process S3, and then uses the foreground mask to A tracking process S5 for tracking the human body rectangular area by updating the position / size of the human body rectangular area based on the human body detection result of the frame and storing a tracking log indicating the tracking result is executed, and this procedure is terminated.

  First, the foreground mask generation process S1 will be described. The means responsible for the foreground mask generation processing S1 is the foreground mask generation means 20 of the advertisement effect measuring apparatus 2. The advertisement effect measuring device 2 executes the foreground mask generation processing S1 as shown in FIG. 1 in which the position and angle of the video camera 4 provided at the upper portion of the display 3a of the digital signage 3 are fixed. Therefore, the video captured by the video camera 4 includes a background image that does not move with time, and if the background image can be removed from the frame to be processed, the human body rectangular area and the face rectangular area can be detected with high accuracy. Because it becomes like this.

  In the present embodiment, the background removal unit 20 of the advertisement effect measuring apparatus 2 uses the dynamic background subtraction method that is an existing technology to obtain the difference between the N frame and the background image, and is not a background image in the N frame. A foreground mask indicating the foreground region is generated.

  FIG. 5 is a diagram schematically illustrating the foreground mask. FIG. 5A is an example of a background image, FIG. 5B is an example of an N frame, and FIG. 5C is an example of a foreground mask. is there. When the foreground mask 5c illustrated in FIG. 5C is generated, an area other than the background image 5a illustrated in FIG. 5A is obtained as the foreground area 5d in the N frame 5b illustrated in FIG. 5B. , The foreground mask 5c illustrated in FIG. 5C is generated.

  Next, the human body detection process S2 executed by the human body detection means 21 of the advertisement effect measuring device 2 will be described. Various methods have been devised as a human body detection method for detecting a human body rectangular area from N frames, and the human body detection method can be arbitrarily selected. However, in this embodiment, the human body detection means 21 is implemented as an example. As a human body detection method, a human body detection method based on an Adaboost algorithm using HOG (Histogram of Oriented Gradients) features is adopted.

  In the human body detection method using the Adaboost algorithm using the HOG feature amount, the position and rectangular size of the rectangular human body detection frame are changed within the N frame, and the edge gradient direction histogram is displayed for each cell that divides the human body detection frame into a plurality of cells. A certain HOG feature value is obtained, and this HOG feature value is assigned to a classifier configured by combining weak classifiers, and the position / rectangular size of the human body detection frame where the score of the classifier is equal to or greater than a threshold is obtained as a human body rectangular area It is done. For the human body detection method using the Adaboost algorithm using the HOG (Histogram of Oriented Gradients) feature as a weak classifier, see “N. Dalal and B. Triggs,” Histograms of Oriented Gradientstional Conference on Computer Vision, pp. 734-741. , 2003. "

  In the human body detection process S2, the foreground mask obtained by the foreground mask generation process S1 confirms whether or not the human body detection means 21 of the advertisement effect measuring apparatus 2 has erroneously detected a part of the background as a human body rectangular area. Used for.

  The human body detecting means 21 of the advertising effect measuring apparatus 2 refers to the foreground mask obtained in the foreground mask processing S1 when checking whether a partial image of the background is erroneously detected as a human body rectangular area, and from the N frame. For each detected human rectangular area, the ratio of the foreground area, which is an area where the foreground mask and the human rectangular area overlap, is specified, and it is determined that a human rectangular area whose ratio of the foreground area is less than the threshold is a false detection. The human body rectangular area whose ratio is less than the threshold is not used in the subsequent processing.

  FIG. 6 is a diagram for explaining erroneous detection of the human body rectangular area. FIG. 6A is an example of an N frame. The N frame 6a illustrated in FIG. 6A includes two pillars 6b, and the shape of the pillar 6b is similar to the shape of a human body. In the N frame 6a shown in (a), the human body detecting means 21 of the advertising effect measuring apparatus 2 erroneously detects this pillar 6b as the human body rectangular area 6c.

  FIG. 6B is an example of the foreground mask. In the foreground mask 6d of FIG. 6B, the region 6e corresponding to the human body rectangular region 6c of the column 6b of the N frame 6a does not include the foreground region. The human body detecting means 21 of the advertising effect measuring apparatus 2 treats the human body rectangular area column 6c corresponding to the column 6b as a false detection.

  Next, the face detection process S3 executed by the face detection unit 22 of the advertisement effect measuring apparatus 2 will be described. The face detection means 22 of the advertising effect measurement device 2 detects the front face rectangular area from the image in the human body rectangular area for each human body rectangular area obtained from the N frame by the human body detection means 21 of the advertising effect measurement apparatus 2. , And the position / rectangular size of the face rectangular area is obtained for each detected face rectangular area.

  Thus, if the front face rectangular area is detected from the image in the human body rectangular area, it is possible not only to prevent erroneous detection of a part of the background as the front face rectangular area, but also the processing time required for the face detection process S3. Can also be shortened.

  In the present embodiment, the face detection method performed by the face detection means 22 of the advertisement effect measuring device 2 employs a face detection method based on the Adaboost algorithm using a black and white Haar-Like feature as a weak discriminator. When detecting the face rectangular area from the image in the human body rectangular area, the face detecting means 2 of 2 performs processing for normalizing the size of the image of the human body rectangular area to a predetermined size (size determined by the number of pixels). After that, the search size, which is the size of the area for detecting the face using the classifier, is started from a predetermined size, and the process for detecting the face from within the area of the search size using the classifier is performed after normalization. By moving the position of the search size area within the human body rectangular area, the face rectangular area corresponding to the search size is detected from the normalized image, and the search size is gradually increased to search By repeating this until the noise becomes the predetermined maximum size, all face rectangle areas of various sizes are detected from the normalized human body rectangle area. The black and white Haar-Like feature was used as a weak classifier. For face detection using the Adaboost algorithm, see “Paul Viola and Michael J. Jones,“ Rapid Object Detection using a Boosted Cascade of Simple Features ”, IEEE CVPR, 2001.”, “Rainer Lienhart and Jochen Maydt,“ An Extended Set of Haar-like Features for Rapid Object Detection ", IEEE ICIP 2002, Vol. 1, pp. 900-903, Sep. 2002."

  The reason for normalizing the image of the human rectangular area to a predetermined size is to reduce the number of undetected faces. Since the size of the face rectangle area of the person far from the video camera 4 is smaller than the size of the face rectangle area of the person near the video camera 4, it is difficult to detect the face rectangle area. By normalizing the image to a predetermined size, the size of the face rectangular area of the person far away from the video camera 4 becomes large, and it becomes easy to detect the face rectangular area. Since the small size can be excluded from the search target of the face, the face rectangular area can be detected stably.

  The face detection means 22 of the advertisement effect measuring apparatus 2 can detect that the person corresponding to the human body rectangular area is browsing the digital signage 3 when the face rectangular area can be detected from within the human body rectangular area. A viewing flag is associated with the human body rectangular area.

  Next, the person attribute calculation process S4 executed by the person attribute calculation means 24 provided in the advertisement effect measuring apparatus 2 will be described. In the person attribute calculation process S4, the person attribute calculation means 24 provided in the advertising effect measurement device 2 analyzes the image in the face rectangle area according to a predetermined algorithm for each face rectangle area obtained in the face detection process S3. Thus, the person attribute (age / gender) is calculated, and the calculated person attribute is associated with the human body rectangular area where the face image is detected.

  A method for estimating a person attribute (age / gender) from a face rectangular area is widely used in cigarette vending machines and the like, and for example, a technique disclosed in Japanese Patent Application Laid-Open No. 2007-080057 can be used.

  Next, the tracking process S5 executed by the tracking unit 23 of the advertisement effect measuring apparatus 2 will be described.

  Various methods have been devised as tracking methods for a human body rectangular area. In this embodiment, the tracking means 23 of the advertisement effect measuring apparatus 2 uses a color feature quantity as a feature quantity calculated from an image in the human body rectangular area. The human body rectangular region is tracked by combining the first tracking method using the second tracking method and the second tracking method using the local feature amount.

  The tracking of the human body rectangular area is performed by combining the first tracking method that uses the color feature amount and the second tracking method that uses the local feature amount. When the occlusion where the human bodies overlap is generated, This is because detection cannot be performed, and only the first tracking method using the color feature amount loses sight of the human body rectangular area when occlusion occurs.

  In the present embodiment, in order to associate the human body rectangular areas in the previous and subsequent frames, the tracking means 23 of the advertising effect measuring apparatus 2 generates a Human object for each human body rectangular area to be tracked, and attributes of this Human object The value includes at least a Human ID given to identify a Human object, the position / size of a human body rectangular area corresponding to the Human object, and a human body color feature amount.

  FIG. 7 is a diagram for explaining the details of the tracking process S4. When processing the N frame to be processed, the tracking unit 23 of the advertising effect measuring apparatus 2 first performs a mask process for masking the N frame using the N frame foreground mask obtained in the foreground mask generation process S1 ( S10) is executed.

  FIG. 8 is a diagram for explaining the mask processing (S10). 8A illustrates an N frame, FIG. 8B schematically illustrates an N frame foreground mask, and FIG. 8C illustrates an N frame after mask processing.

  By applying a mask process to the N frame 7a shown in FIG. 8A using the foreground mask 7b shown in FIG. 8B, the background image of the N frame 7a is removed and only the human body image is extracted. , The N frame 7c after the mask processing shown in FIG. 8 is obtained. In addition, the rectangle illustrated by the dotted line in FIG. 8 indicates a human body rectangular area, and is not a picture taken by the video camera 4.

  When executing the mask process (S10), the tracking unit 23 of the advertisement effect measuring apparatus 2 executes the first tracking process by the first tracking method using the color feature amount using the N frame after the mask process ( S11).

  In the first tracking process by the first tracking method using the color feature amount, the color feature amount calculated from the image in the human body rectangular area detected in the N frame is already generated at the time of the N frame. A process of associating the human body rectangular area detected in the N frame with the Human object is performed.

  In the first tracking processing, the position and size of the Human object are updated to the position and size of the human body rectangular area associated with the Human object for the Human object associated with the human body rectangular area detected in the N frame. Further, a process of updating the color feature amount of the Human object to the color feature amount obtained from the cut-out image of the human rectangular area associated with the Human object is executed.

When the human body rectangular areas in the preceding and following frames are associated with each other using the color feature amount, the human body rectangular areas in the preceding and following frames may be associated by obtaining a color histogram of the human body rectangular area and performing histogram matching. As indices for histogram matching, correlation, chi-square, intersection, and the like can be mentioned. If the Bhattacharyya distance expressed by Equation 1 is adopted as this index, matching accuracy is increased.

  When using the Bhattacharyya distance, a color histogram is used as a color feature, and for each Human object already generated at the time of N frames, a color histogram calculated from a human body rectangular area detected from the N frame and a Human object The Bhattacharyya distance representing the similarity between the color histograms is obtained, and the human rectangular area is associated with the Human object having the smallest Bhattacharyya distance.

  When the first tracking process (S11) by the first tracking method using the color feature amount is performed, the tracking unit 23 of the advertisement effect measuring apparatus 2 stores a tracking log indicating the result of the first tracking process ( S12).

  The tracking means 23 of the advertisement effect measuring device 2 generates a tracking log indicating the result of the first tracking process for each Human object whose position and size are updated in the first tracking process. The Human ID of this Human object, the date and time when the N frame was photographed, and the position and rectangle size of the updated Human object are included.

  In the first tracking process, when a viewing flag is associated with the human body rectangular area associated with the Human object, the viewing flag and the person attribute are included in the tracking log of the Human object.

  When the tracking log indicating the result of the first tracking process is stored, the tracking means 23 of the advertisement effect measuring apparatus 2 checks whether there is a Human object whose position / size has not been updated in the first tracking process (S13). If there is a Human object whose position / size has not been updated in the first tracking process, the tracking means 23 of the advertisement effect measuring apparatus 2 switches the tracking method to the second tracking method using the local feature quantity, The second tracking process by the second tracking method is executed on the Human object whose position / size has not been updated in the tracking process (S14).

  If there is no Human object whose position / size has not been updated in the first tracking process, the process proceeds to S16 in FIG. 7 without executing the second tracking process.

  FIG. 9 is a diagram for explaining the second tracking method. FIG. 9A illustrates an example of the N-1 frame, and FIG. 9B illustrates an example of the N frame. In the N-1 frame 8a illustrated in FIG. 9A, five human bodies are detected, but in the N frame 8b illustrated in FIG. 9B, the middle person is bowing. The middle person is not detected because of the appearance.

  As the second tracking method using the local feature amount, it is desirable to use the tracking method using the HOG feature amount described above. When the tracking method using the HOG feature amount is used for the second tracking method, the human body rectangular region corresponding to the Human object whose position / size has not been updated by the first tracking process in N frames is N frames. Is set in the vicinity of the position of the Human object, a search window 8e equal to the size of the Human object is set, and the position of the Human object (position 8c in FIGS. 9A and 9B). Is searched for a human body rectangular area corresponding to the Human object while changing the position of the search window 8e within a predetermined search area 8d obtained from the start point of the search window 8e.

  FIG. 10 is a diagram supplementing the description of the second tracking method. FIG. 10A shows a luminance gradient 9a, a HOG feature amount 9b, and a gradient histogram 9c obtained from the human body rectangular area of the N-1 frame corresponding to the Human object whose position and size have not been updated in the first tracking process. FIG. 10B shows the luminance gradient 9d, the HOG feature amount 9e, and the gradient histogram 9f obtained from the search window 8e in the vicinity of the Human object.

  As can be seen by comparing the luminance gradients 9a and 9d in FIGS. 10 (a) and 10 (b), there is a difference in the human head in FIGS. 10 (a) and 10 (b), but there is a difference in the torso of the human body. There are not a few differences between the HOG feature values 9b and 9e and the gradient histograms 9c and 9f of the human torso.

  Therefore, when searching for a human body rectangular area corresponding to a Human object whose position / size has not been updated in the first tracking process while changing the position of the search window 8e in the search area 8d, the advertisement effect measuring apparatus 2 The tracking means 23 obtains the similarity of the gradient histogram obtained from the HOG feature value for each cell included in the search window 8e, using correlation, chi-square, intersection, Bhattacharyya distance, etc. as an index, and calculates the similarity in the search window 8e. A search is made for a position where the sum of the values exceeds the threshold. If there is one position where the sum of the similarity values in the search window 8e exceeds the threshold, that position is set as the position of the human body rectangular area in the N frame 8b, and exceeds the threshold in the search range. If there are a plurality of locations, the position where the highest similarity is obtained is set as the position of the human body rectangular area in the N frame 8b.

  When the position of the human body rectangular area in the N frame can be searched using the second tracking method, the tracking means 23 of the advertisement effect measuring device 2 has not been updated in position and size in the first tracking process. Only the position of the Human object is updated to the position of this search window, and the size and color feature of the Human object are not updated.

  When the second tracking process by the second tracking method using the local feature amount is performed, the tracking unit 23 of the advertisement effect measuring apparatus 2 stores a tracking log indicating the result of the second tracking process (S15).

  The tracking means 23 of the advertisement effect measuring apparatus 2 generates a tracking log indicating the result of the second tracking process for each Human object whose position and size have been updated in the second tracking process. The Human ID of this Human object, the date and time when the N frame was photographed, and the position and rectangle size of the updated Human object are included.

  In the second tracking process, when a viewing flag is associated with the human body rectangular area associated with the Human object, the viewing flag and the person attribute are included in the tracking log of the Human object.

  Even if the first tracking process and the second tracking process are executed, if there is a Human object that is not associated with the human body rectangular area of N frames, the human body rectangular area corresponding to this Human object is displayed on the video camera. 4 is determined to have moved out of the shooting area, and is excluded from tracking.

  When the tracking log indicating the result of the second tracking process is stored, it is checked whether there is a human body rectangular area that is not associated with the Human object in the N frame (S16). A human object corresponding to is newly generated, and this human body rectangular area is newly added to the tracking target (S17).

  Note that the position / rectangular size of the human rectangular area associated with the newly generated Human object is used as the position / rectangular size of the newly generated Human object. Further, the tracking means 23 of the advertisement effect measuring apparatus 2 obtains the color feature amount from the image in the human body rectangular area that is not associated with the Human object, and gives it to the newly generated Human object.

  When the tracking unit 23 of the advertisement effect measuring apparatus 2 newly generates a Human object, after storing the tracking log of the newly generated Human object (S18), the procedure of FIG. 7 ends.

  The tracking means 23 of the advertisement effect measuring apparatus 2 generates a tracking log for each newly generated Human object. The tracking log includes the HumanID of the Human object, the date and time when the N frame was shot, the position of the Human object, Include rectangle size.

  When a viewing flag is associated with a human body rectangular area associated with a newly generated Human object, the viewing flag and the person attribute are included in the tracking log of the Human object.

  If there is no Human object whose position / size has not been updated in the first tracking process in S16 of FIG. 7, the procedure of FIG. 7 ends without generating a new Human object.

  Finally, the advertisement effect measuring means 25 of the advertisement effect measuring device 2 will be described. The advertisement effect measuring means 25 of the advertisement effect measuring device 2 is a means for summing up the tracking logs stored by the tracking means 23 and outputting a log file describing the result of the summing process.

  In this embodiment, the advertisement effect measuring means 25 of the advertisement effect measuring device 2 includes a totaling process 1 for outputting the visual recognition rate of the digital signage 3, a totaling process 2 for outputting the staying time before the digital signage 3, and a digital Aggregation process 3 for outputting the viewing time of signage 3, aggregation process 4 for outputting the locus of the position of the human body rectangular area, and aggregation process 5 for outputting the personal attributes (for example, age and sex) of the person who has viewed the digital signage It corresponds to.

  The aggregation process 1 for outputting the visual recognition rate of the digital signage 3 is executed every time a certain period (for example, 1 hour) elapses, and the advertisement effect measuring means 25 of the advertisement effect measuring device 2 includes the date and time in the certain period. For the tracking log, the total number of people who were in front of the digital signage 3, the number of viewers of the digital signage 3, and the visibility log file including the visibility that is obtained by dividing the number of viewers by the total number of people is output.

  The total number of people in front of the digital signage 3 can be obtained by counting the number of HumanIDs recorded in the target tracking log without overlapping HumanIDs. The number of viewers of the digital signage 3 can be obtained by counting only the tracking log including the browsing flag and counting the number of Human IDs recorded in the tracking log without duplication of Human IDs.

  The aggregation process 2 for outputting the staying time in front of the digital signage 3 is executed every time a human object that is not tracked is generated, and the advertisement effect measuring means 25 of the advertisement effect measuring device 2 A residence time log file including the HumanID, tracking start date and time and tracking end date and time of the resulting Human object is output.

  In the counting process 2, the date and time of the tracking log including the HumanID of the Human object that is not tracked is referred to, the oldest date and time is used as the tracking start date and time, and the latest date and time is used as the tracking end date and time.

  The aggregation process 3 for outputting the browsing time of the digital signage 3 is executed every time a Human object that is not tracked is generated, and the advertisement effect measuring means 25 of the advertisement effect measuring device 2 is used for the Human that is not tracked. A residence time log file including the HumanID, browsing start date and browsing end date and time of the object is output.

  In this aggregation process 3, the tracking log including the viewing flag is selected from the tracking logs including the HumanID of the Human object that is not tracked, and the oldest date and time is included among the dates and times included in the tracking log. Is used as the browsing start date and time, and the latest date and time is used as the browsing end date and time.

  The aggregation process 4 for outputting the locus of the position of the human body rectangular area is executed every time a human object that is not tracked is generated, and the advertisement effect measuring means 25 of the advertisement effect measuring device 2 is not tracked. A location log file in which tracking logs including the Human object's HumanID are collected together is output.

  The tabulation process 5 for outputting the person attributes (for example, age and sex) of the person who has viewed the digital signage is executed every time a certain period (for example, one day) elapses, and the advertisement effect measuring unit of the advertisement effect measuring device 2 25 outputs a person attribute log file including a pair of HumanID and person attribute included in the tracking log including a certain date and time.

  In this counting process 5, a tracking log including a browsing flag is targeted from tracking logs including a certain date and time, and a pair of HumanID and person attribute included in this tracking log is a duplicate of HumanID. It is extracted except.

(Modification 1)
From here, the modification of embodiment mentioned above is demonstrated. In the human body detection process S2 of FIG. 4, there are cases where a plurality of human body rectangular areas are detected for a single person. Therefore, in the first modification, a plurality of human body rectangular areas that can be determined as the same person are represented by one human body rectangle. This is an embodiment that is integrated into a region.

  In the first modification in which a plurality of human body rectangular areas that can be determined as the same person are integrated into one human body rectangular area, the contents of the human body detection process S2 in FIG. 1 will be described.

 First, the human body detection process in the first modification will be described. FIG. 11 is a flowchart for explaining the human body detection process in the first modification. In the human body detection process according to the first modification, the human body detection unit 21 of the advertising effect measurement device 2 according to the first modification first uses the HOG feature amount as in the above-described embodiment when executing the human body detection process. All the human body rectangular areas in the N frame are detected by the human body detection method using the Adaboost algorithm (S20).

  Next, the human body detecting means 21 of the advertising effect measuring apparatus 2 according to the modification 1 is detected from the N frame with reference to the foreground mask obtained in the foreground mask processing S1 of FIG. 4 as in the above-described embodiment. A process of specifying a human body rectangular area that is erroneously detected and handled from among the human body rectangular areas is performed (S21), and the human body rectangular area that is erroneously detected and handled is not used in the subsequent processes.

  Next, the human body detecting means 21 of the advertising effectiveness measuring device 2 according to the first modification cuts a plurality of human body rectangular areas that are not detected and handled into clusters by a designated clustering method, and the human bodies belonging to the same cluster An integration process for integrating the rectangular areas into one is executed (S22), and the flow of FIG. 11 ends.

In the integration process, the human body detection means 21 of the advertising effect measurement device 2 according to the modification uses the MeanShift clustering expressed by Equation 2 as a method for clustering a plurality of human body rectangular areas that can be determined as the same person, and performs false detection. For each human rectangular area that has not been specified, a MeanShift Vector is calculated from the center coordinates of the human rectangular area, whereby the human rectangular area is subjected to MeanShift clustering, and the center of gravity of each human rectangular area after clustering is obtained.

 Then, the human body detecting means 21 of the advertising effect measuring device 2 according to the modified example 1 obtains the center of gravity after clustering for each of the human body rectangular areas that are not identified as false detections, by the Nearest Neighbor method (nearest neighbor determination rule). By integrating the centers of gravity, a plurality of human body rectangular areas that can be determined as the same person are integrated into one human body rectangular area. The position of the human rectangular area after integration is the position where the center of gravity is integrated by the Nearest Neighbor method, and the rectangular size of the human body rectangular area is a rectangle obtained by integrating the rectangular size of the human rectangular area where the center of gravity is integrated. It becomes size.

  In the flow illustrated in FIG. 11, after performing a process of specifying a human body rectangular area to be treated as a false detection, an integration process for integrating a plurality of human body rectangular areas that can be determined as the same person into one human body rectangular area is performed. Thus, when an object similar to the shape of a person (for example, a ladder, a pillar, etc.) is erroneously detected as a human body rectangular area, it is possible to prevent the areas related to the object from being integrated.

 FIG. 12 is a diagram for explaining the effect of the first modification. FIG. 12A illustrates a human body rectangular area detected in the human body detection process of FIG. In FIG. 12A, a stepladder 10d is illustrated as an object similar to the shape of a person. In FIG. The detected human body rectangular area 11d is included.

 12B is a diagram when the integration process is performed in the state of FIG. 12A, and when the integration process is performed from the state of FIG. 12A, the person 10c in FIG. Since the distance of the stepladder 10d is short, the human body rectangular area is integrated so that the human body rectangular area 11c related to the person 10c and the human body rectangular area 11d related to the stepladder 10d are included.

  FIG. 12C illustrates a human body rectangular region that is not treated as a false detection. In the human rectangular area 11d related to the stepladder 10d, the ratio of the foreground area, which is an area where the foreground mask and the human rectangular area overlap, is reduced, so that the human rectangular area 11d related to the stepladder 10d is treated as a false detection. Since the human body rectangular area that is not treated as detection becomes the human body rectangular areas 11a to 11c in which the persons 10a to 10c are detected, even if the integration process is performed, the human body rectangular area is not affected by an object similar to the shape of the person. It will be over.

(Modification 2)
Next, Modification 2 will be described. Modification 2 is an embodiment in which the size of the human rectangular area after normalization is known, and therefore the search size at the start of face detection is determined according to the size of the human rectangular area after normalization. .

 In the embodiment described above, face detection is performed after normalizing the image of the human body rectangular area to a predetermined size, and it is not necessary to significantly change the search size at the start of face detection. Since the size of the human rectangular area after normalization is adjusted according to the installation status of the human body, the search size at the start of face detection can be automatically determined according to the size of the human rectangular area after normalization Then, an effect of improving the processing speed of face detection and preventing erroneous detection of face detection can be expected.

 FIG. 13 is a diagram illustrating a flow of face detection processing in the second modification. Also in the second modification, as in the above-described embodiment, the face detection unit 22 of the advertising effect measurement apparatus 2 according to the second modification performs a process of normalizing a human body rectangular area to be face-detected to a predetermined size. After that (S30), the search size at the start of face detection is calculated according to the normalized size of the human body rectangular area (S31). For example, the ratio between the normalized human body rectangular area size and the search size at the start of face detection is set in advance, and the normalized human body rectangular area size is multiplied by the ratio to thereby detect the face. It is possible to calculate the minimum search size when starting. Then, when the face detection means 22 of the advertising effect measurement device 2 according to the modified example 2 calculates the search size at the start of face detection, the face detection unit 22 starts from the search size of the calculated size, and starts from the normalized person rectangle region. A region is detected (S32).

  FIG. 14 is a diagram for explaining the effect of the second modification, and shows the processing speed according to the presence / absence of processing for calculating the size of the discriminator at the start of face detection. FIG. 14 shows the result of processing a video approaching gradually from a position 6 m away from the camera by arranging five subjects at equal intervals in a 1280 × 280 image. As shown in FIG. 14, the number of frames that can be processed per second is approximately doubled by calculating the search size at the start of face detection according to the size of the human body rectangular area before normalization. Obtained.

DESCRIPTION OF SYMBOLS 1 Advertisement effect measuring system 2 Advertising effect measuring apparatus 20 Background removal means 21 Human body detection means 22 Face detection means 23 Tracking means 24 Person attribute calculation means 25 Advertising effect measurement means 3 Digital signage 3a Display 4 Video camera

Claims (11)

  Human body detection means for detecting the position and size of the human rectangular area from the frame constituting the video taken by the camera that shoots the person in front of the display, and the position and size of the face rectangular area from the image in the human rectangular area The human body rectangular area is tracked by associating the human body rectangular area in the preceding and succeeding frames using the face detection means for detecting the image and the feature amount of the image in the human body rectangular area, and the position / size of the human body rectangular area Tracking means for storing a tracking log including the date and time when the frame was captured and the detection result of the face rectangular area for each of the human body rectangular areas to be tracked; The total number of the human body rectangular areas and the number of the human body rectangular areas in which the face rectangular area is detected. Because, characterized by comprising the advertisement effect measurement means for outputting a visual index as an index related to the advertisement effect advertisement effect measurement device.   The tracking means corresponds to a first tracking method that uses a color feature amount and a second tracking method that uses a local feature amount, and after tracking the human body rectangular region by the first tracking method, The advertisement effect measuring apparatus according to claim 1, wherein the human body rectangular area that cannot be tracked by the first tracking method is tracked by the second tracking method.   Foreground mask generation means for generating a foreground mask indicating a foreground area other than the background image by obtaining a difference between the frame and the background image, the human body detection means for each human body rectangular area detected from the frame , Determining a ratio of the foreground area in the foreground mask area corresponding to the human body rectangular area, treating the human body rectangular area with the ratio less than a threshold as a false detection, and the tracking means uses the foreground mask. 3. The advertisement effect measuring apparatus according to claim 1, wherein a feature amount of the image in the human body rectangular area is calculated from the frame after the mask processing.   The human body detection means is configured to perform an integration process in which the human body rectangular area that is not treated as a false detection is divided into clusters by a designated clustering method, and the human body rectangular areas belonging to the same cluster are integrated into one. The advertisement effect measuring device according to claim 3.   5. The face detection unit according to claim 1, wherein the face detection unit detects the face rectangular area after normalizing an image in the human rectangular area to a predetermined size. 6. Advertising effect measurement device.   The face detecting means normalizes an image in the human body rectangular area to a predetermined size, and then determines a search size at the start of face detection according to the size of the human body rectangular area after normalization. The advertisement effect measuring apparatus according to claim 5, wherein   The advertisement effect measuring means obtains the time during which the human rectangular area was tracked by aggregating the tracking log for each human rectangular area, and the residence time as an index related to the advertising effect is obtained for each human rectangular area. The advertisement effect measuring apparatus according to claim 1, wherein the advertisement effect measuring apparatus outputs the advertisement effect.   The advertisement effect measuring means obtains a time during which the face rectangular area can be detected from the human body rectangular area by aggregating the tracking log for each human body rectangular area, and uses the viewing time as an index related to the advertisement effect. The advertisement effect measuring apparatus according to claim 1, wherein the advertisement effect measuring apparatus outputs the information for each rectangular area.   The advertisement effect measuring means outputs the position of the human body rectangular area for each of the human body rectangular areas as an index related to the advertisement effect by counting the tracking log for each of the human body rectangular areas. The advertisement effect measuring device according to any one of claims 1 to 8.   Human attribute specifying means for specifying a person attribute from the face rectangular area detected by the face detecting means is provided, and the tracking means adds the person attribute to the tracking log of the human rectangular area from which the face rectangular area has been detected. The advertisement effect measuring means includes the tracking log for each human rectangular area, and outputs a person attribute for each human rectangular area as an index related to the advertising effect. The advertising effectiveness measuring device according to any one of claims 1 to 9. A computer program for causing a computer to function as the advertisement effect measuring device according to any one of claims 1 to 10.

Images