JP2010211485A - Gaze degree measurement device, gaze degree measurement method, gaze degree measurement program and recording medium with the same program recorded - Google Patents

Abstract

A technique for accurately calculating a gaze degree by eliminating double counting of the same person.
A face area detection unit 12 detects a face area from each input frame, and acquires detected face data such as a detection position of a detected face. The face posture estimation unit 13 estimates a face posture parameter. The state determination unit 14 associates a face area with a parameter between the previous frame and the current frame, and determines the state. The detection ID etc. assigning unit assigns a detection ID and a gaze time to the face area of the current frame, and stores the face feature amount in the storage unit. When the feature amount matches between the current frame and the past frame, the matching processing unit 16 replaces the detection ID of the current frame, adds the gaze time, and updates the storage data of the storage unit. The number-of-interests counting unit 20 obtains the number of attentions from the log of detected face data / stored data, and calculates the average gaze time of each gaze person.
[Selection] Figure 1

Description

  The present invention relates to a technique for measuring a gaze degree of a gaze target from a captured image by applying a human face detection / posture estimation technique.

  As is well known, in the fields of advertisements, images, paintings, crafts (sculptures, pottery, etc.), people (entertainers, politicians, etc.), landscapes, etc., measure the degree of gaze of people and investigate the interests and interests of the public. There are many things to do.

  In particular, in recent years, advertising media called “digital signage”, which transmits images using electric display devices outdoors or in stores, has rapidly spread, and various techniques for measuring the degree of attention have been proposed.

  This new advertising form is considered to be more cost-effective than conventional advertising media because advertisers can send and update the latest information in real time by cooperating with the network.

  As a method for objectively measuring the advertisement effect, the advertisement effect confirmation system of Patent Document 1 is known. This measurement method is to install a camera on the media side, extract the person reflected in the camera based on the contour information, etc., evaluate the extracted number of people as the degree of attention to the media, and total the evaluation values. is there. By combining this measurement method with the face detection technique of Non-Patent Document 1, a more accurate gaze degree can be measured. As the degree of gazing at that time, “the number of people who gazed at the advertisement within a certain period of time” or “the average value of the time when each person gazed at the advertisement (hereinafter referred to as gazing time)” can be considered.

JP-A-11-153777

Yuji Mita, Toshimitsu Kaneko, Osamu Hori, “Probabilistic Incremental Code Correlation Suitable for Image Matching of Individual Differences” IEICE Transactions D-II Vol. J88-D-II No. 8 pp. 1614-1623, The Institute of Electronics, Information and Communication Engineers 2005 Shingo Ando, Yoshinori Kusachi, Akira Suzuki, Kenichi Arakawa, “Pose Estimation Method for 3D Objects Using Support Vector Regression”, IEICE Transactions D Vol. J89-D No. 8 pp. 1840-1847, The Institute of Electronics, Information and Communication Engineers 2006 Kazuhiro Hotta, Kengo Mishima, Takio Kurita, Shinji Umeyama “Face Recognition Using Weighted Matching Based on Information Amount of Gaber Features”, IEICE Technical Report HIP2000-5 (2000-5), The Institute of Electronics, Information and Communication Engineers

  However, when the face facing the front of the camera temporarily turns back and turns the front again, there is a possibility that it is determined as a different person regardless of the face of the same person. In this case, since the same person is counted twice, it is difficult to accurately measure the gaze degree.

  In addition, the same person may temporarily leave the place after seeing the advertisement and then see the advertisement again. Even in this case, the same person is counted twice.

  The present invention has been made to solve the above-described problems, and it is an object of the present invention to provide a technique for eliminating the double count of the same person and accurately calculating the degree of gaze.

  Therefore, in order to solve the above-described problem, the present invention collates the face feature amount of the detected human face with the face feature amount of the past frame when a tracking human face is newly detected after being lost. As a result of this collation, if the facial feature values match, it is recognized as the same person and double counting is prevented.

  One aspect of the present invention is an apparatus for measuring the degree of gaze of a gaze object from an image obtained by imaging the periphery of the gaze object, and a face is present in a frame input in time series from the imaging apparatus. Face area detecting means for sequentially detecting areas and acquiring detection position / size / similarity of face-likeness / detection time as detected face data, and a parameter representing the face posture of the face area detected by the face area detecting means A face posture estimation means for estimating a face area between a continuous previous frame and a current frame, and a state determination means for determining the state of the face area of the current frame by associating the face area with the detected face data and the parameter The detection ID and the gaze time of the face area determined by the state determination means are given based on the detection ID and the gaze time in the associated previous frame, and the face feature amount of the face area is stored in the storage means The If the face feature quantity of the current frame matches the face feature quantity of the discontinuous past frame with the current frame, the detection ID of the face area having the face feature quantity is detected as the past frame detection ID. The matching processing means for adding the gaze time of the replaced detection ID to the gaze time of the face area of the current frame and updating the storage data of the storage means, and the attention from the log of the detected face data and the storage data A number-of-interests counting means for calculating a number and calculating an average gaze time of each gaze person.

  Another aspect of the present invention is a method of measuring a gaze degree of a gaze object from an image obtained by imaging the periphery of the gaze object, wherein the face area detection means is input in time series from the imaging device. A first step of sequentially detecting a region where a face is present in the frame and acquiring a detection position / size / similarity of face-likeness / detection time as detected face data, and a face posture estimation unit includes the face region detection A second step of estimating a parameter representing the face posture of the face area detected by the means, and the state determination unit associates the face area between the continuous previous frame and the current frame with the detected face data and the parameter Then, the third step for determining the state of the face area of the current frame, and the detection ID etc. assigning means determine the detection ID and the gaze time of the face area determined in the third step in the associated previous frame. of A fourth step which is given based on the outgoing ID and the gaze time and stores the facial feature quantity of the face area in the storage means; and the matching processing means is the face feature quantity of the current frame and the face of the past frame discontinuous to the current frame If the feature quantity matches, the detection ID of the face area having the face feature quantity is replaced with the detection ID of the past frame, the gaze time of the replaced detection ID is added to the gaze time of the face area of the current frame, and the memory A fifth step of updating the storage data of the means, and a sixth step in which the number-of-interests counting unit calculates the number of attentions from the detected face data and the log of the storage data, and calculates an average gazing time of each gazing person And having.

  The present invention may be in the form of a gaze degree measurement program that causes a computer to function as the apparatus, or may be provided in the form of a recording medium that records the program.

  According to the present invention, since the double count of the same person is eliminated, the gaze degree is accurately calculated.

1 is a schematic configuration diagram of an apparatus according to an embodiment of the present invention. FIG. 5 is a detailed view of the detection ID and the like / face feature amount storage unit. (A) is a process flow diagram which records an attention log, (b) is a process flow figure which totals and outputs an attention log. Explanatory drawing of the same state discrimination | determination process. (A) is a figure which shows the example of a detection result of a previous frame, (b) is a figure which shows the example of a detection result of the present frame before state discrimination, (c) is a figure which shows the example of the detection result of the present frame after state discrimination.

  According to the present invention, when a gaze degree of a gaze object is measured, a feature amount of the face is stored in the storage device for a certain time after the face of the person is lost. Here, gaze objects include various objects that require measurement of the degree of gaze, such as advertisements, images, paintings, crafts (sculptures, ceramics, etc.), buildings, people (celebrities, politicians, etc.), and landscapes.

  Then, the feature amount extracted from the detected face when a new human face is detected matches the feature amount of the detected face currently stored in the storage device (extracted from the past detected face). Then, they are recognized as the same person (hereinafter, this process is referred to as a collation process). This matching process prevents double counting of the same person.

  However, if all the feature values of the detected face are saved, the capacity of the storage device may be compressed and there is a risk of failure, so the feature value is limited to one for one detected face being tracked. Embodiments are desirable.

  In addition, since the face feature amount generally fluctuates greatly with respect to the face direction, there is a mode in which the feature amount is extracted from a face image closest to the front face of the face being tracked and stored in the storage device. desirable. When the face feature quantity is stored in the storage device, the face orientation angle (yaw angle, pitch angle) information is also stored together.

  The collation process is preferably executed only when the difference between the face orientation angles of the two faces to be collated is within the specified range. As a result, the overall processing becomes efficient, and the collation process can be performed stably.

  Here, when a new face is detected, a flag (initial value OFF) indicating whether or not the detected face matches any of the past detected faces is assigned, and the matching process is performed by appropriately checking the flag. It may be decided whether or not to execute. In other words, once matching with a past detected face is performed, the flag is turned on, and the detected face is not subjected to subsequent collation processing, thereby further improving the overall processing efficiency.

  In addition, if the orientation angle of the detected face never falls within the specified range, the entire process can be made more efficient by setting the storage device so that the feature amount is not stored. This can be realized by performing face detection processing, face posture estimation processing, and collation processing from image processing. Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings.

(1) Configuration Example of Embodiment FIG. 1 shows a configuration example of the embodiment of the present invention. Here, the gaze degree measuring device of the present invention is applied to the advertisement effect measuring device 10 that measures the gaze degree of advertising media. This advertising effect measuring apparatus 10 performs post hoc aggregation of the degree of gaze as “the number of people who have seen the advertising media within a certain time” and “the average value of the gaze time”.

  It is assumed that the advertisement effect measuring apparatus 10 is connected to an imaging apparatus (not shown) (for example, a digital camera or a video camera) that captures the surroundings of the advertising media via a network. This imaging device is assumed to be installed in the vicinity of the advertisement media posting position (for example, the upper part of the display). In addition to the digital signage, the advertising media includes ordinary poster advertisements and general home TV broadcast commercials (commercials).

  Specifically, the advertisement effect measuring device 10 is configured by a computer, and is a normal computer hardware resource, that is, a storage device such as a CPU (Central Processor Unit), a memory (RAM) or a hard disk drive device, a communication interface, Display means such as a display and input means such as a keyboard and a mouse are provided.

  As a result of the cooperation between the hardware resource and the software, the advertisement effect measuring apparatus 10 is provided with the image input unit 11, the face detection unit 12, the face posture estimation unit 13, the state determination unit 14, the detection ID, etc. It functions as a storage unit 15, face feature amount matching unit 16, face feature amount deletion unit 17, previous detection state storage unit 18, attention log storage unit 19, attention number totaling unit 20, and result output unit 21. Here, the image input unit 11 is realized by the communication interface.

  That is, digital image images captured by the imaging device are input to the image input unit 11 in time series via a network. The face detection unit 12 detects a human face area from the input image, and the face posture estimation unit 13 estimates a face posture parameter representing the face posture based on the detected face region.

  The state determination unit 14 associates the face area between the continuous previous frame and the current frame using the face posture parameter or the like, and according to the type prepared in advance for the state of the face area of the current frame. Determine.

  As shown in FIG. 2, the detection ID etc. assigning / face feature quantity storage unit 15 includes a detection ID etc. adding unit 15a for giving a detection ID to the face area, and a face area of the current frame to which the detection ID is given. A feature amount extraction unit 15b that extracts a face feature amount from the image, and a face feature amount storage unit 15c that stores the detection ID and the face feature amount. In the face feature amount storage unit 15c, the face feature amounts of the face area are sequentially stored for each frame to which the detection ID is assigned, and the face feature amounts of each past frame are accumulated.

  The face feature amount matching unit 16 compares the face feature amount of the face area of the current frame with the face feature amount of the face area of the past frame stored in the face feature amount storage unit 15c. The past frame to be collated here corresponds to a discontinuous frame with the current frame. As a result of the collation, if the facial feature amounts match, the detection ID of the current frame is recorded in the attention log recording unit 19 as a missing ID log.

  After this recording, the detection ID of the face area of the current frame is replaced with the detection ID of the past frame, and the gaze time of the replaced detection ID is added to the gaze time of the face area of the current frame. Thereby, the recording data of the face feature amount storage unit 15c is updated. Here, the face feature amount of the current frame that matches the face feature amount of the past frame is not subject to collation in the collation processing in the subsequent frames.

  The face feature amount deletion unit 17 deletes the face feature amount stored in the feature amount storage unit 15c after a predetermined time has elapsed. Here, the related information (detection ID, gaze time, posture parameter, disappearance time, etc.) of the face feature is also deleted.

  In the pre-detection state storage unit 18, the processing results of the respective units 12.14.15a are recorded for use in the processing after the next frame. The attention log storage unit 19 stores detection data of the face detection unit 12 and storage data of the face feature amount storage unit 15c as attention logs.

  The number-of-interests counting unit 20 counts the number of viewers within the target period from the recorded data of the viewer's log recording unit 19, and calculates the average watching time of each viewer. Here, with reference to the missing ID log, the average gaze time is calculated by excluding the missing ID. The result output unit 21 outputs the measurement result of the number-of-interests counting unit 20. Each storage unit 15c. 18.19 is realized with a storage device such as a memory (RAM) or a hard disk drive.

  Hereinafter, specific processing contents of the functional blocks 11 to 21 will be described based on the processing flow of FIG. The contents of this process are the steps (S11 to S19) of acquiring the attention log shown in the imaging device of FIG. 3 (a), and the steps of counting and outputting the attention logs of FIG. 3 (b) (S20.S21). It is roughly divided into

(2) Step of Acquiring Interest Log In FIG. 3A, the processing steps of S11 to S20 are repeated as long as an image frame is input from the imaging device.

  S11: The image input unit 11 receives image frames continuously input from the imaging device (S11). The input image frame is transmitted to the face detection unit 12.

  S12: The face detection unit 12 detects a region with a face from each image frame transmitted from the image input unit 11. Here, for example, it is possible to detect at high speed using a statistical method such as stochastic incremental code correlation described in Non-Patent Document 1.

  At this time, the collation evaluation value used for detection, that is, the similarity indicating the face-likeness is used for the processing of the state determination unit 14. Further, the face detection unit 12 measures and acquires the detection position / size / detection time of the face area as detected face information together with the similarity.

  S13: The face posture estimation unit 13 estimates the face posture with high accuracy by using the brightness pattern of the face region (hereinafter referred to as a detected face) detected in S12. Here, for example, a parameter estimation method combining principal component analysis and support vector regression described in Non-Patent Document 2 is applicable.

  Two types of parameters representing the posture of the face are used: a yaw angle (horizontal angle) and a pitch angle (vertical angle). The front direction when viewed from the imaging device is set to yaw angle = 0 degrees and pitch = 0 degrees.

  S14: The state determination unit 14 reads the detection position / size and the state of the detected face in the previous frame with reference to the previous detection state storage unit 16, and determines the state of each detection face in the current frame. Here, the current frame is a frame (n: n = 1.2.3..k), and the previous frame is a frame (n-1). Assume that the detected face states determined in S14 are defined in the following three types and programs.

State 1: Temporary detection result State 2: Positive detection result State 3: Positive detection result and gaze state Specifically, first, each detected face of the current frame is sorted by the degree of similarity representing the likelihood of a face. Next, in order from the one with the highest similarity, the detection position and the size are used to associate with the detection face of the previous frame. This association is performed on a one-to-one basis, and is associated with a condition that satisfies both of the following two conditions.

Condition 1: When the size of the detected face in the current frame is “s”, the size “s _pre ” of the previous frame detected face is applied to the following equation (1).

Condition 2: When the center position of the detected face in the current frame is (x, y), the center position (x _pre , y _pre ) of the detected face in the previous frame is applied to the following equation (2).

In Expressions (1) and (2), “k _min ”, “k _max ”, “l _min ”, “l _max ”, “m _min ”, and “m _max ” are constants given in advance. When there are two or more corresponding items, the one having the smallest distance between the center positions is adopted. Then, after all the detected faces in the current frame are associated with the detected faces in the previous frame, their states are determined based on the processing shown in FIG. Expressions (1) and (2) are defined in the program.

  In FIG. 4, when the detection result of the previous frame (detected face discrimination result) is “state 1”, the detection result associated with the current frame, that is, the previous frame that satisfies the conditions of equations (1) and (2) It is determined whether or not a detection result exists (S14a). If there is no detection result, the detection result of the current frame is set to “no detection”.

  Similarly, when the detection result of the previous frame is “state 2” or “state 3”, the detection result associated with the current frame, that is, the detection face of the previous frame that satisfies the conditions of equations (1) and (2) exists. It is determined whether or not to perform (S14b). If it does not exist, the detection result of the current frame is set to “state 1”. In this case, the detection position and size of the previous frame are inherited and reflected in the detection result of the current frame.

  Step S14a. If there is an associated detection result in S14b, it is determined whether or not the posture is within the specified range (S14c). If the determination result is not within the specified range, the detection result of the current frame is set to “state 2”, whereas if it is within the specified range, “state 3” is set. Here, “within the designated range” refers to a case where the posture parameters (yaw angle, pitch angle) of the detected face in the current frame satisfy the following expression (3).

In Expression (3), “y _min ”, “y _max ”, “p _min ”, and “p _max ” are constants given in advance. By appropriately setting these constant values, a certain degree of freedom can be given to the relative positional relationship between the imaging device and the advertising media. For example, it is possible to cope with a state where an image is picked up from the side of the slant. Note that equation (3) is also defined in the program.

  When the detection result of the previous frame is “no detection”, it is determined whether or not the current frame that is not associated with any result of the previous frame remains (step S14d). The detection result is “state 1”. The detection result of the current frame detected here is recorded in the previous detection state storage unit 18. An example of the result of state determination is shown in FIG. 5 (previous frame-current frame: 1 sec).

  S15: In the detection ID etc. adding section 15a, information such as the detection ID in the current frame is given based on the detection ID information and gaze time information of the detection face of the previous frame associated with each detected face state. To do.

  At this time, the face feature quantity extraction unit 15b extracts the face feature quantity of the detected face according to a predetermined condition and stores it in the face feature quantity storage unit 15c. The face feature amount storage unit 15c sequentially stores face feature amounts of face areas for each frame to which a detection ID is assigned. In this sense, the face feature value storage unit 15c stores the face feature values of the detected face of each past frame. A specific processing method is shown below.

  When the detected face of the current frame is “state 1” and there is no detected face corresponding to the previous frame, a new detection ID is issued and assigned, and the gaze time is 0 second. For example, the detected face (ID: 14) in FIG.

Here, the designated numerical values are substituted for each of y ′ _min , y ′ _max , p ′ _min , and p ′ _max (these determine whether or not to store the face feature amount in the face feature amount storage unit 15c. It is used in the event that it may be updated later, so it must be unique for each detection ID). For example, using the constants described above,
y ′ _min ← y _min , y ′ _max ← y _max , p ′ _min ← p _min , p ′ _max ← p _max
May be specified, or a completely different value may be entered. Further, the flag Flg1 indicating whether or not it has been determined as “state 2” or “state 3” and the flag Flg2 indicating whether or not the face feature amount of the past frame is matched are turned off (these are also detected for each detection ID). Inherently). Here, the past frame is assumed to be a frame (n-2.3.4... K).

  When the detected face of the current frame is “state 1” and there is a detected face corresponding to the previous frame, the detection ID and gaze time information are inherited from the corresponding previous frame detected face as they are. For example, the detected face (ID: 10) in FIG.

  When Flg1 of the detected face of the previous frame is ON, the disappearance time is recorded as the attribute information of the face feature amount with the same ID stored in the feature amount storage unit 15c (if it is OFF, it is stored once. No face feature amount with the same ID exists). At this time, Flg′1 indicating whether the face has disappeared from the frame is turned ON. This point is also the face feature value attribute information stored in the feature value storage unit 15c, similar to the disappearance time.

  In the case of “state 2”, the detection ID and the gaze time information are directly inherited from the detection face of the corresponding previous frame. For example, the detected face (ID: 11) in FIG. Here, when the posture parameters (yaw angle, pitch angle) of the detected face of the current frame satisfy the condition of Expression (4), processing for storing the face feature amount in the face feature amount storage unit 15c is executed.

Here, it is assumed that the values of y ′ _min , y ′ _max , p ′ _min , and p ′ _max are sequentially updated as follows.

y ′ _min ← − | yaw |, y ′ _max ← | yaw |
p ′ _min ← − | pitch |, p ′ _max ← | pitch |
A face feature amount suitable for collation is selected, and a typical example is the gabor feature of Non-Patent Document 3. After extraction of the facial feature amount of the detected face, it is stored in the facial feature amount storage unit 15c together with the detection ID, gaze time, and posture parameters (yaw angle, pitch angle), and the flag Flg1 is turned on. Also, Flg′1 indicating whether or not the detected face has disappeared from the frame is turned OFF. Here, if the same detection ID exists in the face feature amount storage unit 15c, the previous one is overwritten. By performing the above update processing, for the same detection ID, the face feature amount closer to the front direction is stored in the storage unit 15, and the face direction variation is flexibly dealt with.

  In the case of “state 3”, the detection ID is inherited from the previous frame as it is, and the gaze time is expressed as (gaze time information of the detection face of the corresponding previous frame) + (elapsed time between the previous frame and the current frame). To do. For example, the detected face (ID: 13) in FIG. The gaze time of this detected face (ID: 13) is calculated as 3 sec.

  Here, as in the case of “state 2”, when a predetermined condition is satisfied, the face feature quantity, detection ID, gaze time, and posture parameters (yaw angle, pitch angle) are stored in the face feature quantity storage unit 15c. At the same time, Flg1 is turned on and Flg'1 is turned off.

  S16: When a face area that has not been detected in several frames is newly detected in the current frame, the face feature amount matching unit 16 determines whether the face feature amount comparison unit 16c compares the face feature amount of the past frame stored in the face feature amount storage unit 15c. Perform verification processing.

  For example, if the detected face of “state 2” or “state 3” in the past frame (n−4) is not detected in the past frame (n−3), S14b. As shown in S14c, it is determined as “state 1”. If this detected face is not detected in the past frame (n−2), it is determined as “no detection” as shown in S14a, and the detected face is lost.

  However, when the same face area is newly detected in the previous frame (n-1), a new ID is assigned and double counting occurs as shown in S14d. This detected face is further displayed in S14b. According to S14c, there is a risk of being determined as “state 2” or “state 3”.

  Therefore, the face feature amount matching unit 16 refers to Flg2 indicating whether or not the face feature amount of the past frame matches, and when it is OFF, the face feature of the past frame stored in the face feature amount storage unit 15c. Check the amount. Specifically, first, a feature amount is extracted from a face image cut out from the detection position of the current frame (if there is a face feature amount already extracted by the face feature amount storage unit 15c, it may be reused). Good.)

  Next, it is extracted from the face feature value stored in the face feature value storage unit 15c (only if the flag Flg′1 indicating whether or not the detected face has disappeared from the frame is ON) and the detected face of the current frame. Matched feature values are checked. Here, collation is performed only when the difference between the posture parameter of the face feature amount stored in the face feature amount storage unit 15c and the posture parameter of the current frame detected face is within the specified range.

  If the feature values match as a result of the collation, the detection ID of the current frame detection face is recorded as the missing ID log in the attention log storage unit 19 (the missing ID log is recorded by the appending method). When the face feature amount corresponding to the detection ID recorded here exists in the face feature amount storage unit 15, information related to the feature amount (detection ID, gaze time, posture parameter, disappearance time, Flg'1) Are excluded from the storage unit 15.

  Then, the detection ID of the detected face of the current frame is replaced with the detection ID of the face feature quantity in the face feature quantity storage unit 15c that has been matched. As a result, the same detected face is limited to one face feature amount and the like, and the compression of the storage capacity of the face feature amount storage unit 15c is suppressed. At this time, the gaze time of the face feature amount in the face feature amount storage unit 15c that matches the previous time is added to the gaze time of the detected face of the current frame. That is, the gaze time of the detected face in the current frame is updated to a value obtained by adding the gaze time of the face feature amount in the face feature amount storage unit 15c.

  Further, the flag Flg′1 indicating whether or not the detected face has disappeared from the frame is switched OFF, and the flag Flg2 indicating whether or not the detected face matches the face feature amount of the past frame is switched ON. Therefore, when the face feature of the detected face in the past frame matches even once, the face feature matching process is not performed again. These processes are applied to the detected faces in “state 2” or “state 3” detected in the current frame.

  S17: In the face feature value deletion unit 17, the flag Flg′1 indicating whether or not the detected face has disappeared from the frame is ON in the stored data of the face feature value storage unit 15c, and the disappearance time is the current time. Pick up facial features before a certain time (set in the program in advance).

  The information (face feature value, detection ID, gaze time, posture parameter, disappearance time, Flg′1) regarding the face feature value picked up here is excluded from the face feature value storage unit 15c.

S18: The pre-detection state recording unit 18 includes S12. S14. Processing result of S15, that is, face detection time, position, size, state, detection ID, gaze time, y ′ _min , y ′ _max , p ′ _min , p ′ _max , Flg1 for all detected faces in the current frame , Flg2 is recorded. At that time, the previously recorded data is deleted.

  S19: The attention log storage unit 19 includes S12. S15. Processing result of S16, that is, detection times of all detected faces of the current frame detected by the face detection unit 12, detection IDs and gaze times stored in the face feature amount storage unit 15c (after processing of S15 to S17) Are recorded in chronological order as the attention log. At that time, the previously recorded amount is not erased but is accumulated in the form of additional writing.

  If an image frame has been input even after the process of S19, the process returns to S11 to start the process of the next frame. In this case, the recorded information of the previous detection state recording unit 18 is detected as state detection unit 14. This is transmitted to the detection ID etc. assigning unit / face feature amount storage unit 15. On the other hand, if there is no input of an image frame, the attention log acquisition process is terminated.

(3) Output step S20: In the attention index totaling unit 20, the attention log recorded in the attention log storage unit 19, the target period specified by the user, and the gaze time threshold specified by the user are inputted, and “fixed time” The number of people who watched the advertising media within.

  Specifically, first, an attention log within a target period is read from the attention log storage unit 20. At this time, the target period is specified by the user with an input means or the like, but the target advertisement may be limited to a part of the time zone in digital signage, television commercials, and the like. In that case, the time schedule information of the target advertising media is input, and further narrowed down by the time zone in which the advertisement is displayed.

  Next, detection face information with different detection IDs (here, detection time, detection ID, gaze time) is picked up one by one while referring to the detection time from the latest detection time. As a result of this operation, only the detected face information with the same detection ID is left with the largest gaze time information. However, the log of the face recognized as the same person from the middle is also mixed. Therefore, with reference to the aforementioned missing ID log, those with missing IDs are excluded.

  Furthermore, information whose gaze time information is smaller than the gaze time threshold is excluded (this process may be omitted). The total number of detected faces remaining in the end is totaled, and the “number of people who watched the advertisement within a certain time” is obtained. Further, the “average value of the gaze time” is calculated by dividing the total of the gaze time of the remaining detected faces by the number of detected faces.

  S21: The result output unit 21 outputs the counting result obtained in S20. This output may be displayed on display means such as a display, or may be output by a printer or the like.

  As described above, according to the advertisement effect measuring apparatus 10, when an undetected face area is detected in the current frame for several frames, it is compared with the face feature amount of the past frame stored in the face feature amount storage unit 15. Processing is performed and double counting of the same person is prevented.

  At this time, the collation process is limited to the case where the detected face posture parameter difference is within the specified range between both frames, so the collation target is limited, the entire process becomes efficient, and the process is stabilized.

  In addition, when the face feature amount matches between both frames, the flag Flg'1 is switched to OFF and the flag Flg2 is switched to ON. Therefore, the face feature amount of the frame that matches the detected face feature of the past frame even once. Are not used in the subsequent collation processing, the processing target of the collation processing is further limited, and the overall processing efficiency is improved. Further, since the feature amount of each frame is not stored in the face feature amount storage unit 15c when the face posture parameter does not satisfy the condition of the expression (4), the collation target is also limited in this respect.

  The present invention can also be constructed as a program that causes a computer to function as the advertisement effect measuring device. This program may cause the computer to execute all steps S11 to S21, or may execute only some steps such as S11 to S19.

  This program can be provided through a network such as a website or e-mail. The program is recorded in a recording medium 10 such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, Blu-ray Disk (registered trademark). It can also be stored and distributed. This recording medium is read using a recording medium driving device (such as an optical drive), and the program code itself realizes the processing of the above-described embodiment, so that the recording medium also constitutes the present invention.

10 ... Advertising effect measuring device (gazing degree measuring device)
DESCRIPTION OF SYMBOLS 11 ... Image input part 12 ... Face detection part (face detection means)
13. Face orientation estimation unit (face orientation estimation means)
14: State determination unit (state determination means)
15 ... Detection ID etc./Facial feature amount storage unit 15a ... Detection ID etc. granting unit
15b ... face feature amount extraction unit 15c ... face feature amount storage unit (storage means)
16: Face feature amount matching unit (matching processing means)
Reference numeral 17: Facial feature amount deletion unit 18: Previous detection state storage unit 19 ... Attention log storage unit 20 ... Attention number totaling unit
21 ... Result output section

Claims (10)

A device that measures the degree of gaze of the gaze object from an image obtained by imaging the periphery of the gaze object,
A face area detecting unit that sequentially detects a region where a face exists in a frame input in time series from the imaging apparatus, and obtains a detection position, a size, a similarity of face-likeness, and a detection time as detected face data; ,
Face posture estimation means for estimating a parameter representing the face posture of the face area detected by the face area detection means;
A state determination means for determining a state of the face area of the current frame by associating a face area between the continuous previous frame and the current frame with the detected face data and the parameter;
The detection ID / gaze time of the face area determined by the state determination unit is assigned based on the detection ID / gaze time in the associated previous frame, and the facial feature amount of the face area is stored in the storage unit A granting means such as detection ID;
If the face feature amount of the current frame matches the face feature amount of the past frame that is discontinuous with the current frame, the detection ID of the face area having the face feature amount is replaced with the detection ID of the past frame, and the detection ID of the replaced detection ID Collation processing means for adding the gaze time to the gaze time of the face area of the current frame and updating the storage data of the storage means;
Number-of-interests counting means for obtaining the number of attentions from the log of the detected face data and the stored data, and calculating an average attention time for each of the attentions;
A gaze degree measuring device comprising: The matching processing unit determines that the face feature amount matches when the difference in the parameter is within a specified range in the face area between both frames,
The gaze degree measurement apparatus according to claim 1, wherein the determination is not performed after a match with a facial feature amount of a facial region in any past frame. The said detection ID etc. provision means stores a face feature-value in the said memory | storage means, only when the said parameter of a face area is in the designated range, The said storage means is characterized by the above-mentioned. Gaze degree measuring device. The said detection ID etc. provision means memorize | stores the face feature-value of the face area | region where the face attitude | position which the said parameter represents closest to the front in the said memory | storage memory means. Gaze degree measuring device. A method for measuring the degree of gaze of the gaze object from an image obtained by imaging the periphery of the gaze object,
The face area detecting means sequentially detects the area where the face exists in the frames input in time series from the imaging device, and obtains the detected position / size / similarity / detection time as detected face data. A first step to:
A second step in which the face posture estimation means estimates a parameter representing the face posture of the face area detected by the face area detection means;
A third step of determining a state of the face area of the current frame by associating a face area between the continuous previous frame and the current frame with the detected face data and the parameter;
A detection ID and the like assigning unit assigns the detection ID and the gaze time of the face area determined in the third step based on the detection ID and the gaze time in the associated previous frame, and the facial features of the face area A fourth step of storing the amount in the storage means;
When the face processing amount of the current frame matches the face feature amount of the discontinuous past frame in the current frame, the matching processing means replaces the detection ID of the face area having the face feature amount with the detection ID of the past frame, A fifth step of adding the gaze time of the replaced detection ID to the gaze time of the face area of the current frame and updating the storage data of the storage means;
A sixth step in which a number-of-interests counting unit obtains the number of attentions from the log of the detected face data and the stored data, and calculates an average gaze time of each of the viewers;
A gaze degree measuring method characterized by comprising: In the fifth step, when the difference between the parameters is within a specified range in the face area between both frames, it is determined that the face feature amount matches,
The gaze degree measurement method according to claim 5, wherein the determination is not performed thereafter once the facial feature amount of the face area in any past frame is matched. 7. The gaze degree according to claim 5, wherein, in the fourth step, the facial feature amount is stored in the storage unit only when the parameter of the face region is within a specified range. Measuring method. The gaze according to any one of claims 5 to 7, wherein, in the fourth step, a face feature amount of a face region whose face posture represented by the parameter is closest to the front is stored in the storage means. Degree measurement method.   A gaze degree measurement program for causing a computer to function as each means of the gaze degree measurement apparatus according to any one of claims 1 to 4.   The recording medium which recorded the gaze degree measurement program described in Claim 9.

Images