JP2011070629A - Advertising effect measurement system and advertising effect measurement device - Google Patents

Abstract

A digital signage advertising effect measurement system capable of speeding up the process of tracking the face of a person browsing digital signage and tracking the face of a person who has lost sight.
An advertisement effect measuring device (2) of a digital signage advertisement effect measuring system (1) includes a face detection means (21) for detecting a face image from a frame of a video photographed by a video camera (4), and a face output from the face detection means (22). When there is tracking means 23 for associating the detection frame data with N frames and N-1 frames, and there is N-1 frame face detection frame data that is not associated with the tracking means 23, the moving image analysis method is used to detect the N-1 frames. Moving image analysis means 24 for detecting a face image corresponding to the face detection frame data from N frames, and managing state transitions of the face object generated for each face detection frame data, and at least a time during which the face object is in the browsing state State transition management means 25 for storing is provided.
[Selection] Figure 3

Description

  The present invention relates to a technique for measuring the advertising effectiveness of a display. The display includes a digital signage (Digital Signage), which is an advertising medium for displaying an advertisement using a liquid crystal display or a projector, as well as a show window, an advertisement signboard, a specific product display shelf in a store, and the like.

Digital signage, which is an advertising medium that displays advertisements using a display such as a liquid crystal display or a projector, has begun to be installed in various places.
By using digital signage, not only will it be possible to provide rich content using video and audio, but it will also be possible to efficiently deliver advertisements according to the location of digital signage. Expansion is expected.

  When an advertiser introduces digital signage, it is necessary to present an objective advertising effect to the advertiser, such as the TV audience rating, the number of copies of newspapers and magazines, etc. As an invention, for example, Patent Document 1 is disclosed.

  The display device disclosed in Patent Document 1 includes a video camera that takes a picture of a person looking at a digital signage (digital signage), and is installed at a face detection result and other places (accounting corner). By associating camera face detection results with each other, it is a device that obtains the relationship between the advertisement and the consumption behavior as the advertisement effect of the advertisement displayed on the electronic signboard.

  In addition to the advertising effect described above, the display device disclosed in Patent Document 1 includes a statistical analysis module for calculating the advertising effect obtained by the display device alone, and this statistical analysis module is captured by a video camera. The video is analyzed, and the number of individual faces included in the video, gender data, age category data, the staying time that is the time facing the electronic signboard, and the like are calculated.

JP 2005-251170 A

  In addition to digital signage, which is an advertising medium that displays advertisements using liquid crystal displays and projectors, such as the staying time, which is the time when you are facing the digital signage, it is also possible to display specific information in the show window, advertising signage, and in the store. In order to measure the time of browsing a display such as a product display shelf for each person, it is necessary to track the face of the person included in the video. Although the matching method and Kalman filtering method are described, these methods have the advantage of being able to continue tracking even if the face is lost, but the processing speed of these methods is slow and the overall processing is It becomes rate-limiting.

  Therefore, the present invention can speed up the process of tracking the face of a person browsing a display such as digital signage, and can generate a log that can be used to measure the advertising effectiveness of the display. It is an object of the present invention to provide an advertisement effect measurement system that can track the face of a person.

  A first invention that solves the above-described problem is an advertising effect that measures the advertising effect of the display by analyzing a camera that captures a person in front of the display and a frame included in a video captured by the camera. An advertising effectiveness measuring system comprising at least a measuring device, wherein the advertising effectiveness measuring device detects a face image photographed in the frame transmitted from the camera, and for each detected face image, For each face detection frame data acquired from the face detection means that outputs the position / rectangular size of the face detection frame as face detection frame data, and for each face detection frame data acquired from the face detection means, the frame is acquired from the previous frame. An evaluation value indicating a position / rectangular size difference between the detected face detection frame data and the face detection frame data acquired from the frame processed by the face detection unit this time , Tracking means for associating the previous face detection frame data with the smallest evaluation value with each face detection frame data, and generating a face object for each face image newly detected by the face detection means, Referencing the previous and current face detection frame data correspondence results obtained from the tracking means, transitioning the state of the face object to the viewing state according to a predetermined state transition table, and advertising effect of the display An advertisement effect measurement system comprising: a state transition management unit that stores, for each face object, at least a time during which the state of the face object is in the browsing state as a log serving as an index of.

  According to the first aspect described above, the position / rectangular size of the face detection frame is used, and the face detection frame data of this time is associated with the face detection frame data of the previous position / rectangular size close to each other. Therefore, complicated image processing is not performed, and this processing can be performed at high speed. In addition, by providing the advertisement effect measuring device with the state transition management means, it is possible to acquire the browsing time of the person browsing the display as a log as the advertising effect of the display.

  Further, according to a second aspect of the present invention, the tracking means of the advertising effect measuring device includes two tracking values within a range obtained by adding a predetermined set value to the minimum value of the evaluation value for each face detection frame data. It is confirmed whether or not the evaluation value is included, and when there is the face detection frame data including two or more evaluation values, the face image included in the area of the face detection frame data and the previous one A value indicating the feature difference of the face image included in the face detection frame data image area is calculated and added to the evaluation value of the current face detection frame data, and then the previous face detection is performed. The advertising effect measurement system according to the first aspect is characterized in that a process of associating frame data with the current face detection frame data is executed.

  According to the second aspect described above, when the face detection frame is very close in the current frame, the previous face detection frame data corresponding to the face detection frame with a similar face is the current face. It can be associated with the detection frame data.

  Furthermore, in the third aspect of the present invention, when the face detection frame data associated with the previous face detection frame data is not detected from the current frame, the advertisement effect measuring device uses a moving image analysis method. The advertisement effect measurement according to claim 1 or 2, further comprising moving image analysis means for detecting the face image corresponding to the previous face detection frame data from the current frame. System.

  According to the third invention, when the advertisement effect measuring device includes the moving image analyzing unit, the face detection frame associated with the previous face detection frame data is lost when the face detection frame is lost. Even if data is not detected from the current frame, a response can be taken. As the method executed by the moving image analysis means, any or all of the time-series particle filter, the LK method, and CamShift can be used.

  Further, according to a fourth aspect of the present invention, when there is no previous face detection frame data associated with the current face detection frame data, the state transition management unit of the advertisement effect measurement device is configured to detect the current face detection frame data. The state of the face object including the previous face detection frame data associated with the current face detection frame data is created by newly creating the face object including Is in the browsing state, the face object state is not changed, the face object position / size is updated to the face detection frame data position / size, and the current face detection frame data is not associated When the state of the face object including the previous face detection frame data is in the browsing state, the moving image analyzing unit is used to detect the face image corresponding to the face object, When the face image is detected from the frame, the face object is updated to the detected position / size without changing the state of the face object, and the face image is updated. When it cannot be detected from the current frame, the operation of setting the face object in a non-browsing state is performed, and the time during which the face object is in the browsing state is stored as a log of the browsing time of the face object. The advertisement effect measurement system according to any one of the first to third inventions.

  Furthermore, in a fifth aspect of the present invention, the state transition management unit of the advertisement effect measuring device stores the position of the face detection frame data included in the face object as a log while the face object is in the browsing state. The advertisement effect measuring system according to any one of the first to fourth inventions, characterized in that

  Further, the sixth invention is characterized in that the advertising effect measuring device comprises a person attribute estimating means for estimating the attribute of a person corresponding to the face image by analyzing the face image, The state transition management means causes the person attribute estimation means to analyze the face image corresponding to the newly generated face object, and stores it as a log indicating the attribute of the face image. To the advertisement effect measuring system according to any one of the fifth to fifth aspects of the invention.

  Further, according to a seventh aspect of the present invention, the advertising effect measuring device includes a human body detecting unit that detects a human body photographed in the frame, and the advertising effect measuring device displays the number of human bodies detected in front of the display. The advertisement effect measurement system according to any one of the first to sixth inventions, wherein at least a log of the number of persons in the store is stored.

  If the state of the face object is changed as in the fourth aspect, the browsing time for each person who has viewed the display can be stored as a log. According to the fifth aspect, the person who has viewed the display The position can be stored as a log every time.

  Further, as in the sixth invention, if the advertisement effect measuring device is provided with the person attribute estimating means, the person attribute (age / gender) of the person who has viewed the display can be stored as a log, As in the seventh invention, if the advertising effect measuring apparatus is provided with the human body detecting means, the total number of people in front of the display can be stored as a log.

  Further, according to an eighth aspect of the present invention, the advertising effect measurement device includes background removal means for removing a background image of the frame, and the face detection means and the human body detection means are configured to remove a background image by the background removal means. The advertising effect measurement system according to any one of the first to seventh inventions, wherein the frame is analyzed.

  According to the eighth invention, by providing the advertising effect measuring apparatus with the background removing means, the accuracy of face detection / human body detection can be improved and the processing speed can be improved.

  Further, according to a ninth aspect of the present invention, when the state transition management unit of the advertisement effect measuring apparatus newly creates the face object, the state of the face object is set to a candidate state before being set to a browsing state, When the state of the face object including the previous face detection frame data associated with the face detection frame data is a candidate state, the position / size of the face object is the position of the current face detection frame data. -A face object in a candidate state that is updated to size and increments a counter provided for the face object, and the counter value does not reach a predetermined number of times even if a predetermined set time elapses Cease to use the face object in the candidate state whose counter value has reached the predetermined number of times before the preset set time elapses. It characterized the door, an advertisement effect measurement system according to any one of the eighth aspect of the third invention.

  Furthermore, in the tenth aspect of the invention, the state transition management means of the advertisement effect measuring device causes the state of the face object to transition to a standby state before making the state of the face object non-browsing. If the face detection frame data that matches the face detection frame data included in the face object in the standby state is not detected even after a predetermined time has elapsed, the state of the face object is changed to a non-viewing state. If the face detection frame data that matches the face detection frame data included in the face object in the standby state is detected before the predetermined time has elapsed, the state of the face object is set to the browsing state. The advertisement effect measuring system according to any one of the third to ninth inventions, characterized in that state transition is performed.

  If the state transition is performed as in the ninth aspect, the face image detected by the face detection means can be handled as noise when the face image is randomly generated, or as in the tenth aspect. If the state transition is made to, tracking can be resumed even if it is temporarily completely lost.

  Furthermore, an eleventh aspect of the invention is an advertising effectiveness measuring device for measuring advertising effectiveness of a display, wherein a facial image captured in a frame included in a video captured by a camera that captures a person in front of the display is displayed. For each detected face image, face detection means for outputting the position / rectangular size of the face detection frame as face detection frame data, and for each face detection frame data acquired from the face detection means, the face detection An evaluation value indicating a position / rectangular size difference between the face detection frame data acquired from the frame processed by the means before and the face detection frame data acquired from the frame processed by the face detection means this time is calculated. , Tracking means for associating the previous face detection frame data having the smallest evaluation value with each face detection frame data, and for each face image newly detected by the face detection means. Generating an object, referring to the previous and current face detection frame data correspondence results obtained from the tracking means, and transitioning the state of the face object to a browsing state according to a predetermined state transition table; An advertisement effect measuring apparatus comprising: a state transition management unit that stores at least a time for which each face object is in a browsing state as a log serving as an index of the advertisement effect of the display. It is.

  Further, according to a twelfth aspect of the present invention, the tracking means of the advertisement effect measuring apparatus includes two tracking values within a range obtained by adding a predetermined set value to the minimum value of the evaluation value for each face detection frame data. It is confirmed whether or not the evaluation value is included, and when there is the face detection frame data including two or more evaluation values, the face image included in the area of the face detection frame data and the previous one A value indicating the feature difference of the face image included in the face detection frame data image area is calculated and added to the evaluation value of the current face detection frame data, and then the previous face detection is performed. The advertising effect measuring apparatus according to the eleventh aspect is characterized in that a process of associating frame data with the current face detection frame data is executed.

  Furthermore, in a thirteenth aspect of the present invention, when the face detection frame data associated with the previous face detection frame data is not detected from the current frame, the advertisement effect measuring device uses a moving image analysis method. The advertisement according to the eleventh invention or the twelfth invention, further comprising a moving image analysis means for detecting the face image corresponding to the previous face detection frame data from the current frame. It is an effect measuring device.

  Further, according to a fourteenth aspect of the present invention, when the state transition management means of the advertisement effect measuring device has no previous face detection frame data associated with the current face detection frame data, the current face detection frame data The state of the face object including the previous face detection frame data associated with the current face detection frame data is created by newly creating the face object including Is in the browsing state, the face object state is not changed, the face object position / size is updated to the face detection frame data position / size, and the current face detection frame data is not associated When the state of the face object including the previous face detection frame data is the browsing state, the moving image analyzing unit is used to detect the face image corresponding to the face object, When the face image is detected from the frame, the position and size of the face object are updated to the position and size of the detected face image without changing the state of the face object. When the face object cannot be detected from the current frame, the operation of setting the face object to the non-browsing state is performed, and the time during which the face object is in the browsing state is stored as a log of the browsing time of the face object. The advertising effect measuring apparatus according to the thirteenth aspect of the invention.

  Furthermore, in the fifteenth aspect of the invention, the state transition management means of the advertisement effect measuring device stores the position of the face detection frame data included in the face object as a log while the face object is in the browsing state. The advertisement effect measuring device according to any one of the eleventh to fourteenth inventions.

  Furthermore, the sixteenth aspect of the invention is characterized in that the advertisement effect measuring device comprises person attribute estimating means for estimating the attribute of a person corresponding to the face image by analyzing the face image. From the eleventh invention, the state transition management means causes the person attribute estimation means to analyze the face image corresponding to the newly generated face object, and stores it as a log indicating the attribute of the face image. The advertising effect measurement device according to any one of the fifteenth inventions.

  Further, according to a seventeenth aspect of the invention, the advertising effect measuring device includes a human body detecting unit that detects a human body photographed in the frame, and the advertising effect measuring device indicates the number of human bodies detected in front of the display. The advertisement effect measuring device according to any one of the eleventh to sixteenth inventions, wherein at least a log of the number of persons in the store is stored.

  Further, according to an eighteenth aspect of the present invention, the advertisement effect measuring apparatus includes background removal means for removing a background image of the frame, and the face detection means and the human body detection means are configured to remove a background image by the background removal means. The advertising effect measuring apparatus according to any one of the eleventh to seventeenth aspects, wherein the frame is analyzed.

  Furthermore, in the nineteenth aspect of the invention, when the state transition management unit of the advertising effect measuring apparatus newly creates the face object, the state of the face object is set to a candidate state before the state is set to a browsing state. When the state of the face object including the previous face detection frame data associated with the face detection frame data is a candidate state, the position / size of the face object is the position of the current face detection frame data. -A face object in a candidate state that is updated to size and increments a counter provided for the face object, and the counter value does not reach a predetermined number of times even if a predetermined set time elapses Is stopped, and the face object in the candidate state whose counter value has reached the predetermined number of times before the set time elapses changes to the browsing state It is an advertisement effect measurement apparatus according to any one of the eighteenth aspect of the eleventh invention, wherein.

  Furthermore, in the twentieth invention, the state transition management means of the advertising effect measuring device causes the state of the face object to transition to a standby state before making the state of the face object non-browsing. If the face detection frame data that matches the face detection frame data included in the face object in the standby state is not detected even after a predetermined time has elapsed, the state of the face object is changed to a non-viewing state. If the face detection frame data that matches the face detection frame data included in the face object in the standby state is detected before the predetermined time has elapsed, the state of the face object is set to the browsing state. The advertisement effect measuring apparatus according to any one of the eleventh to nineteenth aspects, wherein the state is changed.

  The eleventh to twentieth inventions are inventions of an advertisement effect measuring device required when constructing the advertisement effect measuring system according to the first to tenth inventions.

  According to the above-described invention, in addition to digital signage which is an advertising medium for displaying an advertisement using a liquid crystal display or a projector, a display such as a show window, an advertisement signboard, or a specific product display shelf in a store is provided. Advertising effect that can speed up the process of tracking the face of the person being viewed, can generate a log that can be used to measure the advertising effectiveness of the display, and can track the face of a person who has lost sight A measurement system can be provided.

The figure explaining the structure of an advertising effectiveness measurement system. The hardware block diagram of an advertisement effect measuring apparatus. The functional block diagram of an advertisement effect measuring apparatus. The flowchart explaining the process which an advertising effectiveness measurement apparatus analyzes a flame | frame. The flowchart for demonstrating a tracking process. The flowchart for demonstrating face detection frame data matching processing. The figure explaining the state transition table in this embodiment. The figure for demonstrating a human body and a face detection result.

  From this, the embodiments of the present invention will be described to the extent that those skilled in the art of the present invention can understand the contents of the present invention and implement the present invention.

  FIG. 1 is a diagram for explaining the configuration of an advertisement effect measurement system 1 according to the present embodiment, FIG. 2 is a hardware block diagram of an advertisement effect measurement device 2 constituting the advertisement effect measurement system 1, and FIG. 3 is an advertisement effect measurement. 3 is a functional block diagram realized by a computer program installed in the apparatus 2. FIG.

  As shown in FIG. 1, the advertising effect measurement system 1 includes a digital signage 3 that is installed in a street or a store and displays an advertisement using a display 3a (display in FIG. 1).

  The digital signage 3 includes a server for controlling the advertising video reproduced by the digital signage 3 in addition to the display 3a for reproducing the advertising video, but this server is not shown in FIG.

  An angle is set on the display 3a of the digital signage 3 so that the face of a person watching the advertisement video reproduced on the display 3a is photographed, and the advertisement video reproduced on the display 3a is viewed. A video camera 4 for photographing a person is installed.

  The video imaged by the video camera 4 is input to the advertising effect measurement device 2 using a USB port or the like, and the advertising effect measurement device 2 analyzes the frame included in the video image transmitted from the video camera 4, A log that can be used for measuring the advertising effect of the digital signage 3 by detecting the face of the person in front of the display 3a or the face of the person who has viewed the advertisement video reproduced on the display 3a (for example, the viewing time of the digital signage 3) ) Is memorized.

  In the apparatus constituting the advertisement effect measurement system 1 illustrated in FIG. 1, commercially available apparatuses can be used for the digital signage 3 and the video camera 4, but the advertisement effect measurement apparatus 2 has features that are not found in the prior art. From here, the advertisement effect measuring apparatus 2 will be described in detail.

  The advertisement effect measuring device 2 can be realized by using a general-purpose computer. The advertisement effect measuring device 2 includes hardware similar to that of a general-purpose computer. In FIG. 2, as the hardware, a CPU 2a (CPU: Central: Processing Unit), ROM 2b (ROM: Read-Only Memory) in which BIOS is mounted, RAM 2c (RAM: Random Access Memory) which is a main memory of a computer, and a large-capacity data storage device 2d (for example, an external storage device) , Hard disk), input / output interface 2e for data communication with external devices, network interface 2f for network communication, display device 2g (for example, liquid crystal display), and character input device 2h (for example, keyboard) , Pointing device 2i (eg mouse) To have.

  A computer program for operating the CPU 2a is installed in the data storage device 2d of the advertisement effect measuring device 2, and the advertisement effect measuring device 2 is provided with the means shown in FIG. 3 by this computer program.

  As shown in FIG. 3, the input of the advertisement effect measuring device 2 is an image taken by the video camera 4, and the output of the advertisement effect measuring device 2 stores data that can be used for measuring the advertising effect of digital signage. Log file.

  The advertisement effect measuring apparatus 2 includes a background removing unit 20 that removes a background image of a frame of the video shot by the video camera 4 and a background removing unit 20 as means for analyzing the frame of the video shot by the video camera 4. Face detection means 21 for detecting a human face from the frame from which the background image has been removed, human body detection means 22 for detecting a human body from the frame from which the background image has been removed by the background removal means 20, and the face detection means 21 have detected. For each face image newly detected by the face detection means 21, the tracking means 23 for associating the face with the previous and next frames, the moving picture analysis means 24 for detecting the specified face image from the frame using a moving picture analysis technique such as a particle filter, etc. A face object is generated and the previous and current face detection frame data obtained from the tracking means 23 are A state transition management means 25 that refers to the result of attachment, changes the state of the face object in accordance with a predetermined state transition table, and stores a log corresponding to the state transition of the face object. In order to include in the log data the attributes (age and gender) of the person who has viewed the digital signage 3, the person attribute estimation means 26 for estimating the person attributes (age and gender) of the person from the face image detected by the face detection means 21 It has.

  As the advertisement effect measuring device 2 analyzes the frames of the video taken by the video camera 4 in time series, the data storage device 2d of the advertisement effect measuring device 2 has a log file that can be used for measuring the advertisement effect. , The browsing time log file that stores the browsing time of digital signage, the location log file that stores the position of the person who viewed the digital signage, and the person attributes (for example, age and gender) of the person who viewed the digital signage A person attribute log file that is stored, a total number of persons who are in front of digital signage, a number of persons who are not browsing digital signage, and a number log file that stores the number of persons who have viewed digital signage are stored, A log file output means 27 for outputting these log files is provided in the advertisement effect measuring apparatus 2. It has been.

  First, each means provided for analyzing the frame of the video photographed by the video camera 4 will be described while explaining the processing of the advertisement effect measuring apparatus 2 analyzing the frame of the video transmitted from the video camera 4. .

  FIG. 4 is a flowchart for explaining processing in which the advertisement effect measuring device 2 analyzes the frame of the video transmitted from the video camera 4. Although details of each process will be described later, when one frame of video is input to the advertisement effect measuring apparatus 2, the advertisement effect measuring apparatus 2 performs background removal processing S1 on the frame, and after the background removal processing S1 is performed. Face detection processing S2 and human body detection processing S3 are performed on the frame.

  The advertisement effect measuring apparatus 2 performs the face detection process S2 and the human body detection process S3 on the frame after the background removal process S1, and then uses the result of the face detection process S2 to determine the frame to be processed this time. A tracking process S4 that associates a face detected from a certain N frame with a face detected from the N-1 frame that is the previous frame, and changes the state of the face object based on the result of the tracking process S4 The transition management process S5 is executed.

  First, the background removal process S1 will be described. The means responsible for the background removal processing S1 is the background removal means 20 of the advertisement effect measuring apparatus 2. The advertisement effect measuring device 2 executes the background removal processing S1 because the position and angle of the video camera 4 provided on the upper part of the display 3a of the digital signage 3 are fixed as shown in FIG. This is because the video captured by the video camera 4 includes a background image that does not change, and by removing this background image, the human body / face can be detected with high accuracy.

  As the background removal processing executed by the background removal means 20 of the advertisement effect measuring device 2, the existing technology can be used, and the video taken by the video camera 4 may change in the morning, noon, and night, for example. It is preferable to use a dynamic background update method that takes into account temporal changes in the background.

  Examples of dynamic background update methods that take into account temporal changes in the background include "Shinji Morita, Kazumasa Yamazawa, Norihiko Terasawa, Naokazu Yokoya:" Network-enabled remote monitoring system using omnidirectional image sensors ", electronic The method described in the Journal of Information and Communication Engineers (D-II), Vol. J88-D-II, No. 5, pp. 864-875, (2005.5) can be used.

  Next, the face detection process S2 executed by the face detection means 21 of the advertisement effect measuring apparatus 2 will be described. Various face detection methods including the face detection method described in Patent Document 1 are disclosed as face detection methods performed in the face detection process S2, but in this embodiment, weak face discriminators are used. The face detection method by Adaboost algorithm using black and white Haar-Like feature is adopted. For the face detection method by Adaboost algorithm using black and white Haar-Like features as weak classifiers, see “Paul Viola and Michael J. Jones,“ Rapid Object Detection using a Boosted Cascade of Simple Features ”, IEEE CVPR, 2001. ", Rainer Lienhart and Jochen Maydt," An Extended Set of Haar-like Features for Rapid Object Detection ", IEEE ICIP 2002, Vol. 1, pp. 900-903, Sep. 2002.

  Face detection frame data is obtained for each face image included in the frame by executing the face detection method by Adaboost algorithm using black and white Haar-Like features as weak classifiers. The position of the face detection frame used when detecting the image (for example, the coordinates of the upper left corner) and the rectangular size (width and height) are included.

  Next, the human body detection process S3 executed by the human body detection unit 22 of the advertisement effect measuring apparatus 2 will be described. As a technique for detecting a human body, an infrared sensor is used and a technique for detecting a human body using the body temperature of a person is well known. However, in this embodiment, the human body detection method implemented in the human body detection process S3 is weak. A human body detection method based on the Adaboost algorithm using HOG (Histogram of Oriented Gradients) features is adopted as a discriminator. For the human body detection method using the Adaboost algorithm using the HOG (Histogram of Oriented Gradients) feature as a weak classifier, see “N. Dalal and B. Triggs,” Histograms of Oriented Gradientstional Conference on Computer Vision, pp. 734-741. , 2003. "

  Human body detection frame data is obtained for each human body included in the frame by executing the human body detection method based on the Adaboost algorithm using the HOG feature as a weak discriminator. The position (for example, the coordinates of the upper left corner of the human body detection frame) and the rectangular size (width and height) of the human body detection frame used in the above are obtained.

  FIG. 8 is a diagram for explaining the human body and face detection results. The person photographed in the frame 7 of FIG. 8 includes a total of six persons 7a to 7f, and the human body detection means 22 of the advertisement effect measuring apparatus 2 detects each person 7a to 7f, and each person 7a Human body detection frame data 70a to 70f corresponding to ˜7f are output. The face detection means 21 of the advertisement effect measuring apparatus 2 detects the faces of the persons 7a to 7c whose both eyes are photographed, and outputs face detection frame data 71a to 71c corresponding to each face.

  Next, the tracking process S4 executed by the tracking unit 23 of the advertisement effect measuring apparatus 2 will be described. In the tracking process S4, the face detection frame data detected by the face detection unit 21 from the N-1 frame and the face detection frame data detected by the face detection unit 21 from the N frame are associated with each other by the tracking unit 23 of the advertisement effect measuring apparatus 2. Processing is executed.

  From here, the tracking process performed by the tracking means 23 of the advertisement effect measuring apparatus 2 will be described in detail. FIG. 5 is a flowchart for explaining the tracking process S4 executed by the tracking unit 23 of the advertisement effect measuring apparatus 2.

  The tracking means 23 of the advertisement effect measuring apparatus 2 performs the tracking process S4 on the N frame, first, the face detection frame data and the human body detection frame data obtained from the N frame are respectively received from the face detection means 21 and the human body detection means 22. Obtain (S10).

  In the next tracking process S4, since the face detection frame data obtained from the N frame is used as the face detection frame data of the N-1 frame, the tracking means 23 of the advertisement effect measuring apparatus 2 starts from the N frame. The obtained face detection frame data is stored in the RAM 2c or the data storage device 2d.

  When the tracking means 23 of the advertisement effect measuring apparatus 2 acquires the face detection frame data and the human body detection frame data of N frames, it performs a digital signage browsing determination for each human body detection frame data of N frames (S11).

  As described above, the human body detection frame data includes the position and rectangular size of the human body detection frame, and the face detection frame data includes the face detection frame because the position and rectangular size of the face detection frame are included. The human body detection frame data can be determined as the human body detection frame data of the person who is browsing the digital signage 3, and the human body detection frame data not including the face detection frame is the human body of the person who is not browsing the digital signage 3. It can be determined as detection frame data.

  When the tracking means 23 of the advertisement effect measuring apparatus 2 performs the digital signage browsing determination for each human body detection frame data of N frames in this way, the digital signage 3 is used as a person log file when the N frames are photographed. The total number of persons in front of the person, that is, the number of human body detection frame data detected by the human body detection means 22, and the number of persons who are not browsing the digital signage 3, that is, the human body not including the face detection frame Generate a number log file that describes the number of detection frame data and the number of people browsing the digital signage 3, that is, the number of human body detection frame data including the face detection frame, To be stored in the data storage device 2d.

  When the tracking means 23 of the advertisement effect measuring apparatus 2 performs browsing judgment of digital signage for each human body detection frame data of N frames, the face detection frame data detected by the face detection means 21 from the N-1 frame and face detection. The face detection frame data association process S12 for associating the face detection frame data detected from the N frames by the means 21 is executed.

  FIG. 6 is a flowchart for explaining the face detection frame data association processing S12. In this embodiment, the evaluation value obtained by using the evaluation function of Equation 1 described below in the flow shown in FIG. Is used to associate the face detection frame data.

Note that the evaluation function f1 () of Equation 1 is an evaluation function using the nearest neighbor method, and the evaluation value obtained by the evaluation function f1 () is an evaluation indicating the difference between the position / rectangular size of the face detection frame data. Value. Further, the evaluation value obtained by the evaluation function f2 () of Expression 1 is obtained from the face image included in the face detection frame specified by the face detection frame data to the evaluation value obtained from the evaluation function f1 (). The difference between the SURF feature amounts indicating the features of the image becomes an evaluation value added by weighting.

  In order to associate the face detection frame data detected from the N-1 frame with the face detection frame data detected from the N frame, the tracking means 23 of the advertisement effect measuring apparatus 2 first calculates the face detection frame data obtained from the N frame. The loop processing L1 is executed by the number.

  In this loop processing L1, the tracking means 23 of the advertisement effect measuring apparatus 2 first executes the loop processing L2 by the number of face detection frame data detected from the N−1 frame. In this loop processing L2, the loop processing L1 The position / rectangular size of the face detection frame data to be processed and the position / rectangular size of the face detection frame data to be processed by the loop processing L2 are substituted into the evaluation function f1 () of Equation 1 to obtain the evaluation value. An evaluation value is calculated (S120), and an evaluation value indicating a difference in position and size from the face detection frame data to be subjected to the loop processing L1 is calculated for each face detection frame data detected from the N-1 frame.

  The tracking means 23 of the advertisement effect measuring apparatus 2 calculates an evaluation value indicating a position / size difference from the face detection frame data to be processed by the loop processing L1 for each face detection frame data detected from the N-1 frame. When calculated, the minimum value of the evaluation value is searched (S121), and after calculating the difference between the minimum value of the evaluation value and another evaluation value (S122), it is determined whether there is the difference value equal to or less than the threshold value (S122). S123).

  Then, the tracking means 23 of the advertisement effect measuring apparatus 2 includes the difference between the minimum value of the evaluation value indicating the position / size difference from the face detection frame data to be processed by the loop processing L1 and the other evaluation values. When there is a difference equal to or smaller than the threshold, the tracking unit 23 of the advertisement effect measuring apparatus 2 executes the loop process L3 for the face detection frame data whose evaluation value is within the threshold.

  In this loop process L3, the face image in the face detection frame specified by the face detection frame data to be processed by the loop process L1 and the N-1 frame face detection frame data to be processed by the loop process L3 are specified. The difference of the SURF feature quantity from the face image in the face detection frame to be obtained is obtained, the difference of the SURF feature quantity is substituted into the evaluation function f2 () of Formula 1, and the evaluation value obtained by adding the difference of the SURF feature quantity is calculated (S124).

  The evaluation value obtained by using the evaluation function f1 () using only the nearest neighbor method is used to calculate the evaluation value obtained by adding the difference of the SURF feature amount using the evaluation function f2 () expressed by the mathematical formula 1. If the difference value between the minimum value and other evaluation values is equal to or smaller than the threshold value, it is considered that face detection frames having similar sizes are close to each other (for example, person 7a, b in FIG. 8), and the nearest neighbor method This is because the N-1 frame face detection frame data associated with the N frame face detection frame data cannot be determined from the above evaluation values.

  By using the evaluation function f2 () shown in Equation 1 and calculating an evaluation value obtained by adding the difference between the SURF feature values, an evaluation value in consideration of facial features is calculated. When face detection frames having similar sizes are close to each other, the N-1 frame face detection frame data having a similar face is associated with the N frame face detection frame data.

Then, the tracking means 23 of the advertisement effect measuring apparatus 2 uses the N-1 frame face detection frame data whose evaluation value obtained from the evaluation function of Equation 1 is the minimum value as the N frame of the loop processing L1. A process of associating with the face detection frame data is executed (S125).
If the evaluation value using the evaluation function f2 () shown in Equation 1 is not calculated, the evaluation value used in this process is the value obtained from the evaluation function f1 () shown in Equation 1. Thus, when the evaluation value using the evaluation function f2 () shown in Equation 1 is calculated, the evaluation value used in this process is the value obtained from the evaluation function f2 () shown in Equation 1. Become.

  When the loop processing L1 is completed and the tracking means 23 of the advertising effect measuring apparatus 2 associates the face detection frame data of the N frame with the face detection frame data of the N-1 frame, the face detection frame data of the N-1 frame is obtained. It is confirmed whether or not it is duplicated and associated with N frames of face detection frame data (S126).

  When the face detection frame data of the N-1 frame overlaps and is associated with the face detection frame data of the N frame, the tracking unit 23 of the advertisement effect measuring device 2 performs the N- By referring to the evaluation value of the face detection frame data of one frame and recursively executing the process of associating the smaller evaluation value with the face detection frame data of the N frame, the face detection frame of the N frame is finally obtained. N-1 frame face detection frame data to be associated with the data is determined (S127).

From here, it returns to description of the flow illustrated in FIG. When the tracking process S4 is completed, it is obtained from the tracking process S4 by the state transition management means 25 of the advertisement effect measuring device 2,
State transition management processing that refers to the result of matching the previous and current face detection frame data, transitions the state of the face object according to a predetermined state transition table, and stores a log according to the state transition of the face object When S5 is executed and there is a predetermined state transition in the state transition management process S5, a predetermined log file corresponding to the state transition is stored in the data storage device 2d.

  In order to manage the state transition of the face object, the state transition management unit 25 of the advertisement effect measuring device 2 has a state transition table in which the state of the face object and rules for making the state transition are defined in advance. The tracking means 23 of the advertisement effect measuring apparatus 2 refers to this state transition table and changes the state of the face object based on the result of the face detection frame association processing.

  From here, an example of the state transition table will be exemplified, and the state transition management process S5 executed by the state transition management unit 25 of the advertisement effect measuring device 2 will be described while explaining the state transition table.

  FIG. 7 is a diagram for explaining the state transition table 6 in the present embodiment. The state transition table 6 shown in FIG. 7 defines the face object state and the transition from the N-1 frame state to the N frame state. The vertical axis of the state transition table 6 indicates the N-1 frame state. , The horizontal axis is in the state of N frames, and conditions for state transition are described where the vertical axis and the horizontal axis intersect. In the state transition table, “-” indicates an illegal state transition.

  In the state transition table 6 illustrated in FIG. 7, None, candidate face, current face, standby face, noise face, and end face are defined as face object states, and the state transitions defined in the state transition table are described. However, each state will be described.

  None, which is one of the face object states, means a state in which no face object exists, and there is N frame face detection frame data that is not associated with N-1 frame face detection frame data (see FIG. 7), the state transition management means 25 of the advertising effect measurement device 2 uses the ID value for identifying the face object, the face detection data of the N frame, the data related to the state assigned to the face object, etc. Is newly generated, and the state of the face object is set as a candidate Face.

  The candidate face that is one of the face object states means a state in which the newly detected face image may be a noise, and the candidate face is provided as one of the face object states. This is because in the case of a complex background, erroneous detection of a face image is likely to occur even if background removal processing is performed, and the newly detected face image may have noise.

  The face object in the candidate face state is given, as data related to the candidate face state, a state ID indicating the candidate face state, a date and time when the state transition is made to the candidate face, and a counter.

  The states that can be changed from the candidate face are the candidate face, the current face, and the noise face, and a predetermined number of face images corresponding to the face objects in the candidate face state are consecutive within a predetermined set time. If the tracking is successful (condition 2-2 in FIG. 7), the state of the face object changes from the candidate face to the current face.

  A counter is provided for the attribute of the face object in the candidate face state in order to count the number of times that the face detection frame corresponding to the face object in the candidate face state can be continuously tracked within the set time. Then, the state transition management unit 25 of the advertisement effect measuring device 2 determines that the state of the face object including the N-1 frame face detection data associated with the N frame face detection frame data is the candidate face. The face detection frame data attached to the face object is updated to N frame face detection frame data, and the counter of the face object is incremented.

  Then, when executing the state transition management process S5, the state transition management unit 25 of the advertisement effect measuring device 2 refers to the date and time when the state transition to the candidate face is performed for each face object that is the candidate face, and within the set time If the value of the counter has reached a preset value, the state of the face object is changed to the current face. Further, the state transition management unit 25 of the advertisement effect measuring apparatus 2 causes the state of the face object that has not reached the set value at this time to transition to the noise Face (see FIG. 7), the state of the face object for which the set time has not elapsed is not changed (condition 2-1 in FIG. 7).

  The noise face that is one of the states of the face object is a state in which the face image detected by the face detection means 21 of the advertisement effect measuring apparatus 2 is determined to be noise, and the face object that has transitioned to the noise face has disappeared. It is regarded and is not used for the subsequent state transition management processing S5.

  The current face, which is one of the face object states, is a state in which a person corresponding to the face object can determine that the digital signage 3 is in the browsing state. It is time to browse the signage 3.

  When the state transition management unit 25 of the advertisement effect measuring apparatus 2 changes the state of the face object from the candidate face to the current face, the face transition frame data of the face object is updated to N frame face detection frame data, As data related to the face, a state ID indicating the current face state and a date and time when the state is changed to the current face are given to the face object.

  In addition, when the state of the face object is changed to “Face” in order to store the person attribute (for example, age / gender) of the person who is viewing the digital signage as a log, the state transition management unit of the advertisement effect measuring apparatus 2 25 activates the person attribute estimating means 26 to acquire the person attribute obtained by analyzing the image in the face detection frame specified by the face detection frame data of the face object whose state has been changed to “Face”. An attribute log file in which the object ID and person attribute of the object are described is stored in the data storage device 2d.

  Although the detailed description of the person attribute estimating means 26 provided in the advertisement effect measuring device 2 is not made, it is not possible to automatically identify the person attribute (age / gender) of the person from the face image of the person. For example, the technology disclosed in Japanese Patent Application Laid-Open No. 2007-080057 can be used.

  Furthermore, when the state transition management unit 25 of the advertisement effect measuring apparatus 2 changes the state of the face object to “Face”, a position log file for storing the position of the person browsing the digital signage 3 in time series is generated. It is newly generated in the data storage device 2d. The position log file at the time of generation is given the object ID of the face object that has made a transition to the current Face and the face detection frame data included in the face object that has made a transition to the current Face.

  The states that can be changed from the current face state are the current face and the standby face. The state transition management unit 25 of the advertisement effect measuring apparatus 2 is configured to execute the case where the state of the face object including the face detection data of the N-1 frame associated with the face detection frame data of the N frame is currently Face (condition 3-1) The face detection frame data attached to the face object is updated to face detection frame data in N frames, and the face detection frame data is added to the position log file specified by the object ID of the face object.

  Further, when the state transition management unit 25 of the advertisement effect measuring apparatus 2 performs the state transition management process S5, N-1 frame face detection frame data that is not associated with the N frame face detection frame data is added. When the state of the face object is currently “Face”, the moving image analyzing unit 24 is activated, and the moving image analyzing method detects the face image corresponding to the N−1 frame face detection frame data from the N frame. Perform the process.

  In the present embodiment, the moving image analysis unit 24 of the advertisement effect measuring apparatus 2 first has N frames that are already associated with N-1 frame face detection frame data that has not been associated with N frame face detection frame data. The occlusion state is determined between the face detection frame data of the two, and it is confirmed whether the target person's face is completely occluded.

The moving image analysis means 24 of the advertisement effect measuring device 2 performs processing for determining the occlusion state of the face object according to Equation 2 for all face objects that exist at this time and are currently in the face, candidate face, and standby face states. Execute.

  When the moving image analysis unit 24 of the advertisement effect measuring apparatus 2 executes the process of determining the occlusion state of the face object according to Equation 2, the process branches based on the determination result.

  When it is determined that there is a high possibility that the person to be tracked is an occlusion in a completely hidden state (when the criterion 1 of Expression 2 is satisfied), tracking by the particle filter is performed, and the position of the target face object・ Detect the rectangle size. The particle filter is described in many literatures such as “Takekazu Kato:“ Particle Filter and its Implementation ”, IPSJ Research Report, CVIM-157, pp.161-168 (2007).” .

  In addition, when it is determined that the possibility is high due to occlusion in which the person to be tracked is half-hidden (when it corresponds to the criterion 2 of Formula 2), tracking is performed by the LK method (Lucus-Kanade algorithm) The position / rectangular size of the face object is detected. The LK method is described in “Lucas, BD and Kanade, T .:“ An Iterative Image Registration Technique with an Application to Stereo Vision ”, Proc. DARPA Image Understanding Workshop, pp. 121-130, 1981.” ing.

  When it is determined that there is a high possibility that the person to be tracked does not have occlusion (when the criterion 3 of Expression 3 is satisfied), the video analysis means 24 of the advertisement effect measuring apparatus 2 uses the CamShift method. Tracking is performed to detect the position and rectangular size of the target face object. The CamShift method is described in “G. R. Bradski:“ Computer vision face tracking for use in a perceptual user interface, ”Intel Technology Journal, Q2, 1998.”.

  When the target face image can be detected from N frames by any of these methods, the state transition management unit 25 of the advertisement effect measuring apparatus 2 uses the face detection data of the face object currently in the face state as When the position / rectangular size detected by the method is updated and the target face image cannot be tracked by any of these methods, the state of the face object that is currently in the face state is changed to the standby face ( Condition 3-2 in FIG.

  The standby face, which is one of the states of the face object, is a state in which a face image corresponding to the face object can no longer be detected even using the moving image analysis means 24 provided in the advertisement effect measuring apparatus 2.

  Further, when the state transition management unit 25 of the advertisement effect measuring apparatus 2 changes the state of the face object to the standby face, the face detection frame data of the face object is not updated, and the data of the standby face is used as data related to the standby face. A state ID indicating a state, a date and time when the face object is changed to the current face, and a date and time when the face object is changed to the standby face are assigned to the face object.

  The state in which state transition is possible from the standby face is the current face or the end face. The state transition management means 25 of the advertisement effect measuring device 2 searches for a face object including face detection frame data of N frames before the predetermined time has elapsed since the state transition to the standby face, and When the state is the standby face, the state of the face object is changed from the standby face to the current face (condition 4-1 in FIG. 7).

  When the state of the face object is changed from the standby face to the current face, the state transition management unit 25 of the advertisement effect measuring apparatus 2 indicates the date and time when the face object has changed to the current face in the state of the standby face. The date and time assigned to the face object at the time of is used.

  Further, when the tracking means 23 of the advertising effect measuring apparatus 2 executes the process for managing the state transition of the face object, the state of the face object after a predetermined time has elapsed from the transition to the standby face to the end face. The state is changed (condition 4-3 in FIG. 7), and the state of the face object for which the set time has not passed is not changed (condition 4-2 in FIG. 7).

  The end face, which is one of the face object states, is a state corresponding to a person who can no longer be detected by the advertisement effect measuring apparatus 2, and the face object whose state is the end face is considered to have disappeared. It is not used in the state transition management process S5.

  Note that the state transition management unit 25 of the advertisement effect measuring device 2 uses the object ID of the face object and the date and time when the face object is currently transitioned to Face before the state transition of the face object to end Face. A browsing time log file that describes a browsing start time and a browsing end time that is the date and time when the face object changes to the standby face is generated and stored in the data storage device 2d.

  As described above in detail, the advertisement effect measuring apparatus 2 has five states, None, Candidate Face, Current Face, Standby Face, Noise Face, and End Face, as the state of the face object generated for each face detected by the face detection means 21. The state transition table 6 is defined, and the state of the face object is transitioned according to the tracking result of the face corresponding to the face object, and the browsing time of the digital signage 3 is stored as a log according to the state transition of the face object. Is possible.

  According to the above description, while the face object state is currently Face, the face corresponding to the face object can be continuously detected. Therefore, the time in the Face state is the digital signage 3 It will be browsing time.

  Also, by defining the candidate Face as the face object state, even if a face is erroneously detected due to noise, the viewing time of the digital signage 3 is not affected. Also, by defining the standby face as the state of the face object, even if the same face is detected after losing sight of the face, it can be handled as the same face.

DESCRIPTION OF SYMBOLS 1 Advertisement effect measuring system 2 Advertising effect measuring apparatus 20 Background removal means 21 Face detection means 22 Human body detection means 23 Tracking means 24 Moving image analysis means 25 State transition management means 26 Person attribute estimation means 27 Log file output means 3 Digital signage 3a Display 4 Video camera 5 Network 6 State transition table

Claims (20)

An advertisement effect measurement system comprising at least a camera for photographing a person in front of a display and an advertisement effect measurement device for measuring the advertisement effect of the display by analyzing a frame included in a video photographed by the camera Because
The advertising effectiveness measuring device includes:
A face detection means for detecting a face image photographed in the frame transmitted from the camera and outputting the position / rectangular size of the face detection frame as face detection frame data for each detected face image;
For each face detection frame data acquired from the face detection means, the face detection frame data acquired from the frame processed by the face detection means one time ago and the face detection means acquired from the frame processed this time A tracking means for calculating an evaluation value indicating a position / rectangular size difference from the face detection frame data, and associating the previous face detection frame data with the smallest evaluation value with each face detection frame data;
A face object is generated for each face image newly detected by the face detection means, and the state transition determined in advance is referred to by referring to the result of association between the previous and current face detection frame data obtained from the tracking means. State transition management that changes the state of the face object to the browsing state according to the table, and stores at least the time during which the state of the face object was in the browsing state as a log serving as an index of the advertising effect of the display Equipped with means,
An advertising effect measurement system characterized by that.   The tracking means of the advertisement effect measuring device includes two or more evaluation values within a range in which a predetermined set value is added to the minimum value of the evaluation value for each face detection frame data of this time. If there is the face detection frame data including two or more evaluation values, the face image included in the face detection frame data region and the previous face detection frame data image region After calculating a value indicating the feature difference of the included face image and adding it to the evaluation value of the current face detection frame data, the previous face detection frame data is converted to the current face detection frame data. The advertisement effect measuring system according to claim 1, wherein processing associated with frame data is executed.   When the face detection frame data associated with the previous face detection frame data is not detected from the current frame, the advertisement effect measuring device uses the moving image analysis method to detect the previous face detection. The advertisement effect measurement system according to claim 1 or 2, further comprising a moving image analysis unit that detects the face image corresponding to frame data from the current frame.   When there is no previous face detection frame data to be associated with the current face detection frame data, the state transition management unit of the advertisement effect measuring apparatus newly adds the face object including the current face detection frame data. When the state of the face object including the previous face detection frame data associated with the current face detection frame data is in the browsing state, the operation is performed and the state of the face object is changed to the browsing state. Update the position / size of the face object to the position / size of the face detection frame data without changing the state of the object, and the previous face detection frame not associated with the current face detection frame data When the state of the face object including data is a viewing state, the moving image analyzing unit is used to detect the face image corresponding to the face object, and from the current frame When the image is detected, the face object is updated to the detected position / size without changing the state of the face object, and the face image is updated from the current frame. An operation for setting the state of the face object to a non-browsing state when it cannot be detected, and storing the time during which the face object is in the browsing state as a log of the browsing time of the face object. 3. The advertising effectiveness measurement system according to 3.   2. The state transition management unit of the advertisement effect measuring apparatus stores the position of the face detection frame data included in the face object as a log while the face object is in a browsing state. The advertising effectiveness measurement system according to any one of claims 1 to 4.   The advertisement effect measuring device includes person attribute estimating means for estimating the attribute of a person corresponding to the face image by analyzing the face image, and the state transition managing means of the advertisement effect measuring device is newly 6. The face image corresponding to the generated face object is analyzed by the person attribute estimating means and stored as a log indicating the attribute of the face image. The advertising effectiveness measurement system described in 1.   The advertising effect measuring device includes a human body detecting unit that detects a human body photographed in the frame, and the advertising effect measuring device displays the number of human bodies detected as a log of the number of persons in front of the display. The advertisement effect measuring system according to any one of claims 1 to 6, wherein at least the information is stored.   The advertisement effect measuring apparatus includes background removal means for removing a background image of the frame, and the face detection means and the human body detection means analyze the frame from which the background image has been removed by the background removal means. The advertisement effect measurement system according to any one of claims 1 to 7, wherein the advertisement effect measurement system according to any one of claims 1 to 7 is characterized.   When the state transition management unit of the advertisement effect measuring apparatus newly creates the face object, the state transition management unit sets the state of the face object to a candidate state before entering the viewing state, and associates the current face detection frame data with the current state. When the state of the face object including the previous face detection frame data is a candidate state, the position / size of the face object is updated to the position / size of the current face detection frame data, and Executes the process of incrementing the counter provided for the face object, stops the use of the face object in the candidate state where the counter value does not reach the predetermined number of times even if the predetermined set time elapses. The candidate face object whose counter value has reached a predetermined number of times before the set time has elapsed is transitioned to a browsing state. Advertisement effect measurement system according to any one of claims 8 to claim 1.   The state transition management means of the advertisement effect measuring device causes the face object to transition to a standby state before the face object is set to a non-viewing state, and a predetermined time has elapsed. If the face detection frame data matching the face detection frame data included in the face object in the standby state is not detected, the state of the face object is changed to the non-viewing state, and a predetermined time is When the face detection frame data that matches the face detection frame data included in the face object in the standby state is detected before the passage, the state of the face object is changed to the browsing state. The advertisement effect measurement system according to any one of claims 1 to 9. An advertising effectiveness measuring device for measuring advertising effectiveness of a display,
A face image captured in a frame included in a video captured by a camera that captures a person in front of the display is detected, and the position / rectangular size of the face detection frame is determined for each detected face image. Face detection means for outputting as data,
For each face detection frame data acquired from the face detection means, the face detection frame data acquired from the frame processed by the face detection means one time ago and the face detection means acquired from the frame processed this time A tracking means for calculating an evaluation value indicating a position / rectangular size difference from the face detection frame data, and associating the previous face detection frame data with the smallest evaluation value with each face detection frame data;
A face object is generated for each face image newly detected by the face detection means, and the state transition determined in advance is referred to by referring to the result of association between the previous and current face detection frame data obtained from the tracking means. State transition management that changes the state of the face object to the browsing state according to the table, and stores at least the time during which the state of the face object was in the browsing state as a log serving as an index of the advertising effect of the display Equipped with means,
An advertising effect measuring device characterized by that.   The tracking means of the advertisement effect measuring device includes two or more evaluation values within a range in which a predetermined set value is added to the minimum value of the evaluation value for each face detection frame data of this time. If there is the face detection frame data including two or more evaluation values, the face image included in the face detection frame data region and the previous face detection frame data image region After calculating a value indicating the feature difference of the included face image and adding it to the evaluation value of the current face detection frame data, the previous face detection frame data is converted to the current face detection frame data. The advertisement effect measuring apparatus according to claim 11, wherein a process associated with frame data is executed.   When the face detection frame data associated with the previous face detection frame data is not detected from the current frame, the advertisement effect measuring device uses the moving image analysis method to detect the previous face detection. The advertisement effect measuring device according to claim 11 or 12, further comprising a moving image analyzing means for detecting the face image corresponding to frame data from the current frame.   When there is no previous face detection frame data to be associated with the current face detection frame data, the state transition management unit of the advertisement effect measuring apparatus newly adds the face object including the current face detection frame data. When the state of the face object including the previous face detection frame data associated with the current face detection frame data is in the browsing state, the operation is performed and the state of the face object is changed to the browsing state. Update the position / size of the face object to the position / size of the face detection frame data without changing the state of the object, and the previous face detection frame not associated with the current face detection frame data When the state of the face object including data is a viewing state, the moving image analyzing unit is used to detect the face image corresponding to the face object, and from the current frame When the image is detected, the face object is updated to the detected position / size without changing the state of the face object, and the face image is updated from the current frame. An operation for setting the state of the face object to a non-browsing state is performed when the face object cannot be detected, and the time during which the face object is in the browsing state is stored as a log of the browsing time of the face object. 13. The advertising effectiveness measuring device according to 13.   12. The state transition management means of the advertisement effect measuring apparatus stores the position of the face detection frame data included in the face object as a log while the face object is in a browsing state. The advertising effectiveness measuring device according to any one of claims 14 to 14.   The advertisement effect measuring device includes person attribute estimating means for estimating the attribute of a person corresponding to the face image by analyzing the face image, and the state transition managing means of the advertisement effect measuring device is newly 16. The face image corresponding to the generated face object is analyzed by the person attribute estimating means and stored as a log indicating the attribute of the face image. The advertising effectiveness measuring device described in 1.   The advertising effect measuring device includes a human body detecting unit that detects a human body photographed in the frame, and the advertising effect measuring device uses the number of human bodies detected as a log of the number of persons in front of the display. The advertisement effect measuring device according to any one of claims 11 to 16, wherein at least the information is stored.   The advertisement effect measuring device includes background removal means for removing a background image of the frame, and the face detection means and the human body detection means analyze the frame from which the background image has been removed by the background removal means. The advertisement effect measuring device according to any one of claims 11 to 17, characterized in that   When the state transition management unit of the advertisement effect measuring apparatus newly creates the face object, the state transition management unit sets the state of the face object to a candidate state before entering the viewing state, and associates the current face detection frame data with the current state. When the state of the face object including the previous face detection frame data is a candidate state, the position / size of the face object is updated to the position / size of the current face detection frame data, and Executes the process of incrementing the counter provided for the face object, stops the use of the face object in the candidate state where the counter value does not reach the predetermined number of times even if the predetermined set time elapses. The candidate face object whose counter value has reached a predetermined number of times before the set time has elapsed is transitioned to a browsing state. Advertisement effect measurement apparatus according to any one of claims 18 to claim 11. The state transition management means of the advertisement effect measuring device causes the face object to transition to a standby state before the face object is set to a non-viewing state, and a predetermined time has elapsed. If the face detection frame data matching the face detection frame data included in the face object in the standby state is not detected, the state of the face object is changed to the non-viewing state, and a predetermined time is When the face detection frame data that matches the face detection frame data included in the face object in the standby state is detected before the passage, the state of the face object is changed to the browsing state. The advertisement effect measuring device according to any one of claims 11 to 19.

Images