JP2016035759A - Individual discrimination device and individual discrimination method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an individual identification apparatus capable of precisely identifying individuals based on an image including plural users.SOLUTION: A frame storage section 30 stores images which include at least an area of bodies of users. A life sign signal detection section 40, using a predetermined number of images stored in the frame storage section 30, detects a signal string of life signs which periodically changes from plural imaging areas of the bodies of the users. A correlation calculation section 50 calculates correlation among the life signs in the signal string detected from imaging areas of the bodies. An identity determination section 60 determines whether or not the imaging area of the bodies belong to an identical user based on the correlation among the life signs in the signal string detected from the imaging area of the bodies.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an apparatus and method for performing individual discrimination from a captured image.

  Some game machines in the home are equipped with a camera, which captures a user playing a game and displays the user's image on the game screen, or detects a user's action from the captured image of the user to play a game. It can be reflected. It is also possible to detect the user's facial expression from the user's image captured by the camera, grasp the user's mental state during game play, and reflect it in the development of the game. In addition, a plurality of players each have a controller and can operate a game machine, and a game is often played by a plurality of people including family members and friends.

  In addition, in order to detect the degree of tension of the user during game play, a sensor for detecting a pulse is attached to the user, and the measurement result of the pulse output from the sensor is input to the game machine and reflected in the game. ing.

  Patent Document 1 describes a method of measuring vital signs from temporal changes in the density of a captured image of a subject.

JP-A-2005-218507

  When a plurality of players operate a game machine with a controller, or when there are family members or friends in the vicinity in addition to the player having the controller, a plurality of users are shown in the image captured by the camera. It is necessary to make a decision. It is also possible to recognize individual areas by recognizing areas such as faces and hands from the image, but if the hands of the player who operates the controller cross during game play, it becomes difficult to associate the hands and faces. There is.

  The present invention has been made in view of these problems, and an object of the present invention is to provide an individual determination apparatus and an individual determination method capable of accurately performing individual determination in a captured image.

  In order to solve the above problems, an individual identification device according to an aspect of the present invention includes a storage unit that stores an image of at least a part of a user's body, and a predetermined number of frames stored in the storage unit. A vital signs signal detector that detects a periodically changing vital signs signal sequence from a plurality of imaging areas of the user's body using the captured images of the user, and a vital signs signal sequence detected from each imaging area of the body Identity that determines whether each imaging region of the body belongs to the same user, based on the correlation between the correlation calculation unit that calculates the correlation between the image and the vital signs signal sequence detected from each imaging region of the body And a determination unit.

  Another aspect of the present invention is an individual discrimination method. This method periodically changes from a plurality of imaging regions of the user's body by referring to a memory that stores an image in which at least a part of the user's body is captured using a predetermined number of frames. A vital sign signal detection step for detecting a vital sign signal sequence, a correlation calculation step for obtaining a correlation of the vital sign signal sequence detected from each imaging region of the body, and a vital sign signal detected from each imaging region of the body An identity determination step of determining whether or not each imaging region of the body belongs to the same user based on the correlation of the signal sequence.

  It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a data structure, a recording medium, and the like are also effective as an aspect of the present invention.

  According to the present invention, it is possible to accurately identify an individual from a captured image.

It is a lineblock diagram of a game system concerning an embodiment. It is a lineblock diagram of the individual discriminating device concerning an embodiment. It is a figure explaining the image of the user image | photographed with the camera. It is a figure explaining the determination result of the imaging region of the body by the identity determination part of FIG. It is a flowchart explaining the individual identification procedure by the individual identification apparatus of FIG. It is a flowchart explaining another individual identification procedure by the individual identification apparatus of FIG.

  FIG. 1 is a configuration diagram of a game system according to the embodiment. The game system includes a game machine 200, a display 20, a controller 300, and cameras 10a and 10b.

  The controller 300 is connected to the game machine 200 wirelessly or by wire. The display 20 has a built-in camera 10a for photographing a user who is playing a game with the controller 300, a friend or family who is with the user. A game screen is displayed on the display 20. The video camera 10b can focus and shoot a part of the user's body, for example, a hand holding a controller or a specific user's face, and the captured image is input to the game machine 200. The display 20 can also display a user image taken by the cameras 10a and 10b.

  Note that the embodiment will be described here by taking a game play as an example, but the embodiment is not limited to a game but can be applied to a case where a video or the like is viewed on a television screen.

  FIG. 2 is a configuration diagram of the individual identification device 100 according to the embodiment. A plurality of cameras 10a and 10b, a display 20, a controller 300, and a mounting sensor 310 are connected to the individual identification device 100. The individual identification device 100 has a functional configuration mounted on the game machine 200 of FIG.

  The cameras 10a and 10b capture a part or the entire region of the user's body. One or more cameras may be used. For example, a plurality of people including game players may be photographed with one camera. Alternatively, a camera that shoots with the player at the center and another camera that shoots the whole of a plurality of persons including or not including the player may be provided. Alternatively, one camera may capture the user's hand and another camera may capture the user's face. The captured image is stored in the frame storage unit 30.

  The vital sign signal detection unit 40 uses a predetermined number of frames of captured images stored in the frame storage unit 30 to detect a vital signs signal sequence that periodically varies from the imaging region of the user's body. The vital sign signal detection unit 40 can detect a periodically changing sign series of vital signs by detecting changes in skin vibration or light absorption characteristics in a body imaging region in a predetermined number of frames. it can. Periodically changing vital signs are vital signs such as heartbeat and respiration. For example, a waveform data string of vital signs such as a heartbeat can be acquired by detecting changes in skin vibration and light absorption characteristics of a body part where the skin is exposed in a captured image of about 5 frames. . The change in the light absorption characteristic is detected as a change in the density or color of the skin image.

  The vital sign signal detection unit 40 may divide the imaging area into small areas and detect a vital signs signal sequence for each of the divided small areas. In order to reduce the amount of calculation and noise, it is more preferable to detect a vital signs signal sequence for each region of a predetermined area.

A method for measuring a pulse by detecting a change in skin vibration or light absorption characteristics in an imaging region of the body is described in detail in, for example, the following documents.
"Heart rate measurement based on a time-lapse image", Medical Engineering and Physics, 2007
"Advancements in non-contact, multiparameter physiological measurements using a webcam", IEEE Trans Biomed Eng, 2010.
"Non-contact, automated cardiac pulse measurements using video imaging and blind source separation", Optics Express, 2010.

  The correlation calculation unit 50 obtains the correlation of the vital signs signal sequence detected from each imaging region of the body. When the vital sign signal sequence detected in the first body imaging region is H1 (t) and the vital sign signal sequence detected in the second body imaging region is H2 (t), two signal sequences The correlation R between H1 (t) and H2 (t) can be obtained by the following formula, but a calculation formula different from this can also be used.

  The identity determination unit 60 determines whether or not each imaging region of the body belongs to the same user based on the correlation of the vital signs signal sequence detected from each imaging region of the body. If the correlation R between the vital signs signal sequence H1 (t) of the first body imaging region and the vital signs signal sequence H2 (t) of the second body imaging region is greater than a predetermined threshold T, the first It is determined that the imaging region of the body and the imaging region of the second body belong to the same person, and if the correlation R between the two signal sequences H1 (t) and H2 (t) is equal to or less than a predetermined threshold T, the first It is determined that the imaging area of the body and the imaging area of the second body belong to different persons.

  The first body imaging area and the second body imaging area may be captured by different cameras 10a and 10b, respectively. For example, when the user's hand is imaged by the first camera 10a and the user's face is imaged by the second camera 10b, the vital signs signal sequence H1 (t) detected from the captured image of the user's hand And the vital signs signal sequence H2 (t) detected from the captured image of the user's face, and if the correlation R is high, the captured user's hand and face belong to the same person If the correlation R is low, it is determined that the captured user's hand and face belong to different persons. In this way, even if the user's hand and face are not necessarily captured by the same camera, individual discrimination can be performed by obtaining the correlation between the vital signs signal sequences detected from the hand and the face.

  The display unit 70 performs the same highlighting on a plurality of imaging regions of the body determined to belong to the same person and displays the same on the display 20. For example, a plurality of imaging regions of the body determined to belong to the same person are displayed in the same color. Thereby, it is possible to confirm in the screen how the individual determination apparatus 100 performed the individual determination.

  The sensor input unit 80 receives a signal sequence of vital signs that periodically varies from a sensor mounted on the controller 300 or a mounting sensor 310. For example, the controller 300 is equipped with a pulse sensor and the like, and the pulse of the operator of the controller 300 can be measured and input to the sensor input unit 80. Further, the user can wear a wearing sensor 310 such as a pulse sensor on the wrist or ankle, measure the pulse, and input it to the sensor input unit 80.

  The correlation calculation unit 50 may obtain a correlation between the vital signs signal sequence obtained from the sensor input unit 80 and the vital signs signal sequence detected from the imaging region of the body. Based on the correlation between the vital signs signal sequence obtained from the sensor and the vital signs signal sequence detected from the body imaging area, the identity determination unit 60 determines whether the body imaging area is an operator of the controller or It can be determined whether or not the sensor belongs to the sensor wearer. For example, when the correlation between the vital signs signal sequence detected from the imaging region of the hand of the user making a specific gesture and the vital signs signal sequence obtained from the sensor is high, the gesture hand is It can be determined that it belongs to the controller operator. Accordingly, it is possible to grasp that the gesture by the hand and the operation by the controller are performed by the same person, and the gesture by the hand and the operation by the controller can be handled as input from the same user for the game.

  The individual information management unit 90 stores and manages the imaging region of the body determined to belong to the same person by the identity determination unit 60 in the individual information storage unit 92 as user individual information. The individual information management unit 90 manages a specific body imaging area determined to belong to the controller operator or the sensor wearer in association with a flag indicating the controller operator or the sensor wearer. As a result, the body area of the controller operator or the sensor wearer is identified from the plurality of users shown in the captured image, and for example, the satisfaction level of the user during game play from the facial expression of the controller operator's face Can be determined.

  The vital sign signal detection unit 40 obtains a characteristic amount indicating the periodicity of the vital sign of an individual by performing discrete Fourier transform on the vital sign signal string and detecting the frequency. Alternatively, the feature quantity indicating the periodicity of the individual vital signs is obtained by obtaining the autocorrelation of the vital signs signal sequence and detecting the period of the signal. For example, if the vital sign is a heart rate, the heart rate is obtained as a feature amount indicating the periodicity of the individual's vital sign. The individual information management unit 90 classifies the feature amount indicating the periodicity of the vital signs of the individual in units of users and stores it as individual information in the individual information storage unit 92. Thereby, for example, it is possible to detect the degree of excitement of the user during the game play, such as an increase in the heart rate of a specific user.

  Convenience and entertainment can be improved by feeding back the user satisfaction and excitement obtained in this way to the game development. For example, when it is determined that the user is relaxing based on the degree of excitement of the user, short-term control that gives a sudden change to the game development may be performed. Alternatively, based on the user's satisfaction, it is possible to grasp the user's proficiency with respect to controller operations, game rules, etc., and perform long-term control to adjust the speed of game development according to the user's proficiency Good. Further, when the heart rate of the player is too high or when the heart rate is slow to return to the original level, the player may be suggested to take a break. Also, the user's satisfaction level and excitement level may be recorded and fed back to the game developer via a network or the like for use in game updates or future game development.

  Further, a face of a user other than the player having the controller is detected using the face detection function, and the heart rate fluctuation between the user and the player and the correlation value of the breathing timing are regarded as an index indicating empathy with the player. The excitement of the place including the observer may be evaluated. For example, in a party game or the like, when the index indicating empathy is low and the place is not exciting even when only the player is excited, the game development can be adjusted based on such an index.

  Alternatively, the player's posture may be estimated based on physical information of the same individual such as the face, hand, and foot, and advice may be given to correct the posture when an unfavorable posture lasts for a long time. In the case of an application that moves the player's body, such as yoga, it can be used as a material for determining whether the user's posture is good or bad.

  FIG. 3A is a diagram for explaining a user's image taken by the cameras 10a and 10b. It is assumed that four users 400, 402, 404, and 406 are looking at the game screen in front of the game machine 200. These users are family members, and the second user 404 (mother) from the right holds the controller 300. The second user 402 (father) from the left wears the wearing sensor 310 on the wrist (not shown).

  FIG. 3B is a diagram for explaining the determination result of the imaging region of the body by the identity determination unit 60. The two body imaging regions 410a and 410b are determined to belong to the same person 400 because the correlation of the vital signs signal sequences detected from these imaging regions is higher than a predetermined threshold. The correlation threshold value used for the identity determination can be experimentally determined so that the success or failure of the identity determination is within a certain error range.

  Similarly, the two body imaging regions 412a and 412b are determined to belong to the same person 402 because the correlation of the vital signs signal sequence is higher than a predetermined threshold, and the four body imaging regions 414a, 414b and 414c are determined. 414d is determined to belong to the same person 404 because the correlation of the vital signs signal sequence is higher than a predetermined threshold, and the three body imaging regions 416a, 416b, and 416c are correlated with the vital signs signal sequence. Since it is higher than the predetermined threshold value, it is determined that they belong to the same person 406.

  Here, since the user 402 (father) is wearing the wearing sensor 310, the user 402 (father) can acquire the signal sequence of the father's vital signs from the wearing sensor 310. Since the vital signs signal sequence from the wearing sensor 310 has a correlation higher than a predetermined threshold value with the vital signs signal sequence in the imaging regions 412a and 412b of the two bodies, The imaging regions 412a and 412b are determined to belong to the wearer (here, the father) of the wear sensor 310. The individual information management unit 90 stores the body imaging regions 412 a and 412 b in the individual information storage unit 92 in association with the identification information of the wearer of the wearing sensor 310.

  In addition, the user 404 (mother) holds the controller 300 in his / her hand, and can obtain a signal sequence of the mother's vital signs from a sensor equipped in the controller 300. Since the vital signs signal sequence from the sensor of the controller 300 has a correlation with the vital signs signal sequences of the four body imaging regions 414a, 414b, 414c, and 414d higher than a predetermined threshold, the identity determination unit 60 Determines that these body imaging areas 414a, 414b, 414c, 414d belong to the operator (here, mother) of the controller 300. The individual information management unit 90 stores the body imaging regions 414a, 414b, 414c, and 414d in the individual information storage unit 92 in association with the identification information of the wearer of the controller 300.

  FIG. 4 is a flowchart illustrating an individual determination procedure performed by the individual determination apparatus 100. Here, the vital sign is assumed to be a heartbeat.

  The vital sign signal detection unit 40 detects the heartbeat signal sequence H1 (t) from the imaging region of the first body (S10). The vital sign signal detection unit 40 detects a heartbeat signal string H2 (t) from the imaging region of the second body (S12).

  The correlation calculation unit 50 calculates a correlation R between the first heartbeat signal sequence H1 (t) and the second heartbeat signal sequence H2 (t) (S14).

  If the correlation R is greater than the threshold T (Y in S16), the identity determination unit 60 determines that the first body imaging region and the second body imaging region belong to the same person, and the individual information management unit 90 Classifies the imaging region of the first body and the imaging region of the second body as individual information of the person and registers them in the individual information storage unit 92 (S18). Thereby, for example, a user's hand and face are associated with each other as belonging to the same person.

  If the correlation R is equal to or less than the threshold T (N in S16), the identity determination unit 60 determines that the first body imaging region and the second body imaging region belong to different persons, and the individual information management unit 90 classifies the imaging region of the first body and the imaging region of the second body as individual information of different persons, and registers them in the individual information storage unit 92 (S20). Thereby, a plurality of users can be individually identified and classified.

  FIG. 5 is a flowchart for explaining another individual determination procedure by the individual determination apparatus 100.

  The sensor input unit 80 detects a heartbeat signal string H1 (t) from a sensor mounted on the controller 300 or a mounting sensor 310 mounted on the user's body (S30). The vital sign signal detection unit 40 detects a heartbeat signal sequence H2 (t) from the imaging region of the body imaged by the cameras 10a and 10b (S32).

  The correlation calculating unit 50 calculates the correlation R between the first heartbeat signal sequence H1 (t) and the second heartbeat signal sequence H2 (t) (S34).

  If the correlation R is greater than the threshold T (Y in S36), the identity determination unit 60 determines that the imaging region of the body belongs to the controller operator or the sensor wearer, and the individual information management unit 90 The imaging area is classified as individual information of the controller operator or sensor wearer and registered in the individual information storage unit 92 (S38). Thus, the body region of the controller operator or the sensor wearer, for example, the face can be specified, and control such as changing the development of the game based on the facial expression of the controller operator or the sensor wearer becomes possible. .

  If the correlation R is equal to or less than the threshold T (N in S36), the identity determination unit 60 determines that the imaging region of the body does not belong to the controller operator or the sensor wearer (S40). Thereby, a controller operator or a sensor wearer can be distinguished from other users.

  In the above description, the method for determining the identity of the body imaging region based on vital signs has been described, but once the identity determination of the body imaging region is finished, using techniques such as template matching, By tracking the imaging region of the body such as the face, the identity determination of the imaging region of the body may be continued without detecting vital signs. When the user moves his body greatly, or when the user's position is changed, tracking fails, so the identity determination based on vital signs may be resumed.

  In the embodiment, a heartbeat is taken as an example of a vital sign, but breathing may be detected as a vital sign from vibrations of the body of the user who has been imaged. In addition, since a person has a characteristic that the heart rate increases during an inhalation operation and the heart rate decreases during an exhalation operation, respiration can also be detected from a signal train of heartbeats. Hereinafter, a method for detecting the respiration cycle or frequency from the heartbeat will be described.

  (1) First, a heart rate is obtained by detecting a frequency from a waveform data string of heartbeats by Fourier transform. You may obtain | require a heart rate from the period obtained using the autocorrelation of the waveform data sequence of a heartbeat.

  (2) Next, the obtained heart rate is continuously obtained at a time interval equal to or less than the heartbeat period, for example, at intervals of 0.5 seconds to obtain a heart rate sequence.

  When no exercise that affects the heart rate is performed, a heart rate fluctuation corresponding to the breathing motion appears in the heart rate sequence. The heart rate increases during the inhalation operation, and the heart rate decreases during the exhalation operation. Therefore, if the frequency and cycle of heart rate fluctuation are obtained from the heart rate sequence by the same method as in (1) above, it can be regarded as the breathing frequency and cycle. Advantageously, breathing can be detected even when dressed. It is also possible to detect whether or not the plurality of players breathe during the game play from the synchronization of the breathing waveforms of the players.

  As described above, according to the individual discriminating apparatus 100 of the present embodiment, it is possible to identify the region of the same pulse source, and perform individual discrimination and classification by correlating the time variation of the pulse-detected region. it can. Even if there are a plurality of users in the captured image, it is possible to individually determine areas such as the face, hand, and foot to determine whether they belong to the same person. Further, based on the correlation with pulse information from a contact sensor such as a controller or an appliance, the captured hand or face image can be associated with a controller operator or a sensor wearer.

  The face, hands, and feet are far away from the body and away from the heart, but the pulse waveforms and frequencies are almost the same in each part of the body. The pulse waveform data sequence of each part of the body has a waveform that resembles the difference in phase due to a shift according to the distance from the heart. In order to deal with such a case, when obtaining the correlation between the pulse waveform data sequences of each part of the body, a plurality of correlations whose phases are shifted back and forth are obtained, and the maximum correlation value among them is adopted. This makes it possible to determine whether or not the pulse waveform data is obtained from the body of the same person even if the body parts are different. A high correlation may be obtained instantaneously with another person's pulse waveform data. However, when observing for a long time, the waveforms do not match or the phase shift fluctuates with time, so a high correlation cannot be obtained.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. . Such a modification will be described.

  10 camera, 20 display, 30 frame storage unit, 40 vital sign signal detection unit, 50 correlation calculation unit, 60 identity determination unit, 70 display unit, 80 sensor input unit, 90 individual information management unit, 92 individual information storage unit, 100 individual discriminating device, 200 game machine, 300 controller, 310 wearing sensor.

Claims (7)

A storage unit for storing an image obtained by imaging at least a part of the user's body;
Using the captured images of a predetermined number of frames stored in the storage unit, the captured image is divided into a plurality of imaging regions, and a signal sequence of vital signs periodically changing from each of the plurality of imaging regions of the user's body A vital sign signal detection unit to detect;
A correlation calculation unit for obtaining a correlation of a signal sequence of vital signs detected from each imaging region of the body;
And an identity determining unit that determines whether or not each imaging region of the body belongs to the same user based on the correlation of the vital signs signal sequence detected from each imaging region of the body, An individual discrimination device.   The vital sign signal detection unit detects a periodically changing vital sign signal sequence by detecting a change in skin vibration or light absorption characteristics in an imaging region of the body in a predetermined number of frames. The individual discrimination device according to claim 1, wherein The vital sign is a heartbeat;
The identity determination unit determines whether each imaging region of the body belongs to the same user based on the correlation of the detected heartbeat signal sequence. The individual identification apparatus described. A sensor signal input unit that receives an input of a periodically changing vital signs signal sequence from a sensor worn by a user on a body or a sensor equipped on a controller operated by the user;
The correlation calculation unit obtains the correlation between the vital signs signal sequence obtained from the sensor and the vital signs signal sequence detected from the imaging region of the body,
Based on the correlation between the vital signs signal sequence obtained from the sensor and the vital signs signal sequence detected from the body imaging area, the identity determination unit is a user wearing the sensor in the body imaging area. It is determined whether it belongs to the user who operates a controller, The individual discriminating device in any one of Claim 1 to 3 characterized by the above-mentioned. The life sign signal detection unit periodically changes from the first body imaging region imaged by the first imaging unit and the second body imaging region imaged by the second imaging unit. Detect the signal train of signs,
The correlation calculation unit obtains a correlation between the signal sequences of the respective vital signs detected from the first body imaging region and the second body imaging region,
The identity determination unit is configured to detect the first body imaging region and the second body based on the correlation between the signal sequences of the vital signs detected from the first body imaging region and the second body imaging region. 4. The individual discriminating apparatus according to claim 1, wherein it is determined whether or not a body imaging region belongs to the same user. Using a captured image of a predetermined number of frames from a memory that stores an image in which at least a part of the user's body is captured, the captured image is divided into a plurality of imaging regions, and a plurality of images of the user's body are captured. A vital signs signal detection step for detecting a cyclic signal string of vital signs from each of the regions;
A correlation calculation step for obtaining a correlation of a signal sequence of vital signs detected from each imaging region of the body;
An identity determination step for determining whether or not each imaging region of the body belongs to the same user based on the correlation of the vital signs signal sequence detected from each imaging region of the body, Individual identification method. Using a captured image of a predetermined number of frames from a memory that stores an image in which at least a part of the user's body is captured, the captured image is divided into a plurality of imaging regions, and a plurality of images of the user's body are captured. A vital signs signal detection step for detecting a cyclic signal string of vital signs from each of the regions;
A correlation calculation step for obtaining a correlation of a signal sequence of vital signs detected from each imaging region of the body;
Causing the computer to execute an identity determination step for determining whether or not each imaging region of the body belongs to the same user based on the correlation of the vital signs signal sequence detected from each imaging region of the body A program characterized by

Images