JP2005199078A - Condition monitoring device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a state monitoring device capable of detecting a human yawn and recognizing its state. <P>SOLUTION: The condition monitoring device comprises a motion detecting part 1 to observe a human and to detect its motion from observed images, a determining part 2 to convert the motion to a quantitative motion value and to determine if the human yawned or not by the converted motion value, and a controlling part 3 to control an outer device based on a determined yawn state. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a state monitoring device that monitors the behavior of a human or animal subject in a living space or behavior space.

  Conventional user interfaces have been designed for the purpose of recognizing actions that humans intentionally cause and performing control reflecting their intentions.

  For example, pointing to a target object and detecting the direction of the finger by image recognition is used for pointing (for example, see Non-Patent Document 1).

  In this case, it is necessary to always pay attention to the device. For example, when a person is not consciously acting, such as at bedtime, the conventional method cannot convey a human intention. If you go to sleep with the lights on at this time, you cannot tell your intention to turn off the lights.

  Even if a human is not conscious, it is necessary to think about how to control the device by recognizing the human state. For example, in order to detect actions that humans are not aware of, such as turning over at bedtime, yawning, opening and closing eyelids, and controlling devices based on the information, it is necessary to accurately grasp the state of the person.

  In order to judge the human condition, various measurement is performed by attaching a special measurement device (eg, electroencephalogram measurement, blood pressure measurement device, data glove, data suit, etc.) to a human, and based on the result, an electric device, etc. There are also things like manipulating. However, wearing such a device on the body is troublesome and troublesome for humans, and there is a problem that it is difficult to grasp the state at bedtime. To that end, it is considered appropriate to recognize the state using image information.

Also, various types of monitoring devices are known to perform the same function, and information is acquired using image information and used for security in conjunction with an alarm device or the like. However, the current situation is that they do not manage the detailed movements of human beings but manage the entry and exit of people near the door.
Masaaki Fukumoto, Kenji Mase, Yasuhito Suenaga: “Experimental System for Detection of Instructional Motion Using Image Processing” 1991 IEICE Spring National Conference Proceedings A-251 (1991))

  As described above, a method for controlling a device by recognizing the state of the person without being conscious of the person has not been realized.

  Therefore, the present invention provides a state monitoring device that detects unconscious behavior of a human or animal and recognizes the state.

  The state monitoring apparatus of the present invention includes an image input means for observing a subject, a motion detection means for detecting a motion of the subject from an image observed by the image input means, and a motion of the subject detected by the motion detection means. Motion value converting means for converting the motion value into a quantitative motion value, and determination means for determining the state of the subject from the motion value converted by the motion value conversion means, the motion value conversion means comprising the motion detection The value of the evaluation function composed of the product of the amount of movement of the subject detected by the means and the weighting coefficient is set as a quantitative operation value of the movement of the subject.

  The state monitoring apparatus of the present invention will be described.

  The image input means observes the subject and inputs the movement as an image.

  The motion detection means detects the motion of the subject from the image observed by the image input means. That is, posture, face, eyes, mouth are detected, and states such as yawning and eyelid opening / closing are detected.

  The motion value conversion means converts the motion of the subject detected by the motion detection means into a quantitative motion value. That is, the motion value conversion means uses the value of the evaluation function formed by the product of the amount of motion of the subject detected by the motion detection means and the weighting coefficient as a quantitative motion value of the motion of the subject.

  The determination means determines the state of the subject from the motion value converted by the motion value conversion means. That is, the behavioral state of the subject is classified into several types of states in the active state and the sleeping state, respectively, and the determination is made by paying attention to the behavior that the subject unconsciously causes.

  The state monitoring apparatus of the present invention classifies the behavioral state of a subject such as a human being or an animal into several types of states, and can determine the behavior that the subject causes unconsciously.

  An embodiment of the present invention will be described with reference to the drawings. In this embodiment, the setting is made such that the subject is a human and there is one human in the environment monitored by the system. “Subject” refers to humans and animals such as lions, elephants, and dogs.

  FIG. 1 shows the configuration of the entire system.

  The system apparatus is composed of a detection unit 1, a determination unit 2, and a control unit 3.

  The detection unit 1 is for detecting a room or a human condition, and sends the information to the determination unit 2.

  The determination unit 2 determines the human state based on the information received from the detection unit 1.

  The control unit 3 has an interface for controlling the device and controls the device.

[Detection unit 1]
First, the detection unit 1 will be described with reference to FIG.

  The detection unit 1 has a plurality of cameras, microphones, temperature, and humidity sensors as input devices. The input control unit 11 processes these inputs, the detection function units 14, 1516, and 17, and various model databases. 12 includes a result collection unit 13 that integrates the results and summarizes the output to the determination unit 2.

  The functions of the detection unit 1 include a posture detection function, a face detection function, an eye part observation function, and a mouth part observation function.

[Attitude detection function unit 14]
The posture detection function unit 14 detects a human posture existing in images taken by a plurality of cameras. This will be described with reference to FIG.

  The moving object is narrowed down by obtaining a differential image of the moving images (time-series images).

  First, the difference image generation unit 22 generates a difference image from the images stored in the image memory 21.

  The labeling unit 23 labels the area of the moving object.

  For each labeled region, an edge extraction unit 24 newly extracts an edge from the original image.

  The contour information is obtained from the obtained edge image by the contour information generation unit 25, and the model matching unit 27 is used by using a three-dimensional model as shown in FIGS. 4 (a) and 4 (b) stored in the model database 26. Matching is performed to determine the posture.

  If it is possible to quantify what kind of motion a human has done, it is possible to determine the motion using the value of the evaluation function.

  In this embodiment, the following evaluation function is defined and calculated by the evaluation function calculator 28. This result is an operation value obtained by quantifying the movement of the posture.

The bending angle of each joint, such as the movement of the hand, foot, neck, etc., rotation in the body axis direction, and bending of the waist, does not represent an angle in a three-dimensional space, but on a two-dimensional plane (image). The angle at the time of projection is taken, the difference between FIGS. 4B and 4C is taken, and θx (where x is the number of joints to be watched 1 to n) as shown in (d). Also, when each weighting factor is wx, the evaluation function f is

  Can be defined.

  It is assumed that this weight constant is obtained in advance by experiments or the like. The evaluation functions listed above are evaluation functions for the movement of the whole body.

For example, as a function to evaluate the movement of only the right arm,

The movement of the limbs is determined using the quantified movement value of the evaluation function instead of monitoring the movements of individual joints.

[Face detection function unit 15]
The face detection function unit 15 will be described. The face detection function is a function for detecting a human face and estimating the direction in which the person is facing, as shown in FIG.

  By the posture detection described above, it is possible to assume where the face is present in the acquired image.

  From the image stored in the image memory 31, the head is estimated from the posture detected by the posture detection function unit 14, and a partial image is generated by the face portion estimation cutout unit 32.

  For the partial image, the hue / saturation image generation unit 33 creates a hue / saturation image and detects a skin color region.

  In order to determine whether or not the detected region is a face, matching is performed by the face portion extraction unit 35 based on the face model stored in the model database 34.

  Next, the facial part extraction unit 36 searches for facial parts from the result.

  After extracting characteristic parts such as eyes and mouth, the face orientation determination unit 37 determines how many times it faces the human opposite axis from the relative position of the face area where each part exists. The direction in which the face is facing is estimated, such as whether the person is facing and in which direction the room is facing.

For example, assuming that the vertical rotation of the face is θx, the horizontal rotation of the face is θy, and the weighting coefficients are wx and wy, respectively, the evaluation function f is

f = wx · θx + wy · θy

It becomes.

  Then, the movement determination of the face is performed with the movement value quantified by the evaluation function.

  Here, the case of a dark room will be described.

  In this embodiment, a method using an infrared camera is used. That is, color information cannot be used. Therefore, detection using shape information and luminance information is performed. These processes are estimated from all the observation cameras, that is, images projected from the respective camera plates, and the results of the estimation are combined to calculate one result.

[Eye observation function 16]
The eye part observing function unit 16 is a function for observing information that a human is opening and closing eyes and eye movements in a sleeping state, and is shown in FIG.

  The face detection function unit 15 knows the position of the eyes, and the eye part detection unit 42 creates a rectangular partial image trimmed so as to include the eye part from the image in the image memory 41.

  In the partial image, the eye portion is not always parallel to the rectangle, so the image is rotated by the rotated image generation unit 43 for post-processing.

  The pupil detection unit 44 detects the image based on the pupil (iris and pupil) and white eye models.

  The cases where this fails are (1) failure when generating a partial image, that is, when the eye portion does not exist in the image, and (2) when the eye is closed.

(1) There are cases where it is hidden behind something and cannot be detected. If re-extraction is attempted and acquisition is impossible, extraction is performed from another viewpoint, and if acquisition is impossible from anywhere, processing is stopped. At this time, the detection module determines that it is “hidden” by another object.

(2) In the case where the eyes are closed, it is determined whether or not the eyes are closed by extracting the eyelid line formed when the eyes are closed when the eyes are closed. The eyelid line 45 is detected by the eyelid detection unit 45. When the eyes are closed, the eyelid line is observed to have a lower luminance than the face surface color, and is detected as an edge on the image relatively stably.

  This will be described with reference to FIG.

  In step a1, edge detection is performed from the image.

  In step a2, binarization is performed.

  In step a3, arc extraction using the Hough transform (HKYuen, J. Illingworth, J. Kittler, “Ellipse detection using the Hough Transform”, Alvey Vision Conference, Manchester, (1988)) and the like is performed. Do.

  In step a4, the eyelid is detected in consideration of the position of the arc, the combination of arcs extracted as a plurality of candidates from the combination detection, and the relative positional relationship.

  Next, consider the following about the sleep state.

  A sleep state that causes REM (Rapid Eye Movement), which is a phenomenon in which the eyeball moves rapidly during sleep, is called REM sleep. It is known that this state is a light sleep state. In order to detect these sleep states, a method for acquiring eyelid spasm detection using image information will be described with reference to FIGS. 8 (a), (b), and (c).

  The eye movement detection unit 46 performs processing as shown in FIG. 8A from a time-series image obtained by photographing a person.

  In step b1 of FIG. 8A, a difference image is generated for the partial image around the eye (FIG. 8B).

  When the difference image of the eye portion is finely changed, it is determined as eye movement.

In step b2 of FIG. 8 (a), the evaluation function m is determined from the respective pixel values of the difference image as shown in FIG. 8 (c) by the eyelid part (skin color area H) and the eyelid line (low luminance area L). Because there is

(Ii is a certain pixel value in the partial image).

  In steps b3 and b4 of FIG. 8 (a), if a change equal to or greater than the threshold is recognized for several tens of seconds, it is determined that there is eye movement. Each weight constant (w2, w2) and threshold value are obtained from many samples by experiments. Each detection result is sent to the result integration unit 47, and the results are collected.

[Mouth observation unit 17]
The mouth portion observation function unit 17 is in a state where a human is talking or not,
It is a function to observe whether yawning.

  Here, since it is necessary to determine whether or not the mouth is open, description will be made based on FIGS. 9 (a), 9 (b), and 9 (c).

  In step c1 of FIG. 9 (a), first, in order to search for a mouth, the mouth part is searched by using position information and color information, and the cut partial image is cut into a rectangle like the eye part observation function unit 16. Generate. In the present embodiment, collation with a lip shape model as shown in FIG.

  In step c2 of FIG. 9A, the lip portion is extracted from the brightness information and binarized.

  In step c3 of FIG. 9A, the models of the upper lip and lower lip are matched with the size of the binarized image to match the models.

  In step c4 of FIG. 9 (a), as shown in FIG. 9 (c), the vertical and horizontal dimensions of the entire mouth and the size at the time of opening are measured.

  Next, detection of conversation and yawn will be described based on FIGS. 10 (a), (b), and (c).

  In the case of conversation, the change in opening and closing of the mouth is large when considering changes in the time axis.

  Also, yawns open their mouths for a long, long breath first, then open for a while and then close.

  FIG. 10B shows a method of preparing such a “conversation model” and “yawn model” as shown in FIG. 10A and recognizing a temporal change sequence as a pattern.

  In step d1 of FIG. 10 (b), a graph representing the mouth shape change as shown in FIG. 10 (a) is generated.

  In step d2 of FIG. 10 (b), as shown in FIG. 10 (c), the curve is differentiated, the time is divided at the inflection points, and projected onto a one-dimensional pattern.

In step d3 of FIG. 10 (b), a series of changes within a certain section is identified as follows. The number of segments in the divided time series (t _sep ) (example 16 above, example 3 below) is determined by performing pattern matching such as the length of time of each segment.

[Judgment unit 2]
Next, the determination unit 2 in FIG. 1 will be described with reference to FIG.

  The determination unit 2 determines how to control the device based on the information received from the detection unit 1 and what kind of instruction is given to the person. , Have conversation function.

[State integration determination function unit 50]
The state integration determination function unit 50 performs identification, history, and integration of various pieces of information obtained from the detection position, and manages a central function for issuing various instructions.

  First, the following several states are set for human states.

・ Activity (Waking up)
Still state (watching TV, reading books, etc.)
Conversation state (speaking)
Exercise state (including walking)
-Sleeping state (sleeping state)
Sleeping state (the state that seems to have fallen asleep)
Sleeping state (normally sleeping with your eyes closed)
Deep sleep (state that doesn't happen a little)
REM sleep state (sleep state where eye movement is observed)
Awake state (up to several minutes after waking up)
The observation information receiving unit 51 receives the information obtained from each detection function and converts each information into several levels. Since there are many cases where fine movements are made with respect to the posture, there is no provision for no movement. Therefore, it is necessary to change the gaze operation in each state.

The determination / determination unit 52 refers to the table shown in Table 1 from the state information database management unit 53 to determine the state.

Here, a case where the state using the table cannot be identified in any state due to a failure of the detection device will be described. For example, when the face cannot be detected, it is difficult to determine whether or not the user is sleeping. As an aid to this, it is determined that the person is “sleeping” when there is little movement from the degree of change in posture.

  When yawning, humans usually cover their mouths with their hands. In this case, as shown in FIG. 12, only the mouth shape recognition fails. Therefore, it is necessary to make an integrated judgment in combination with the arm movement evaluation function described above.

  The determination / determination unit 52 determines what instruction is issued to the control device, and sends an instruction to the control device based on the data registered in the control information database management unit 54. At this time, the state of the device to be controlled is also managed, and useless control (for example, the lighting is turned off but it is not tried to turn it off) or the necessity of temperature adjustment of the air conditioner is performed.

  The following information is registered in the control information database management unit 54.

  Summarize the processing for when active.

  If the status is determined, the following environment is set. When entering or leaving the room, that is, after confirming the presence or absence of a person in the room, turn on / off room lighting and turn on / off the power of other appliances such as a television. Depending on the direction of movement, the lighting of the moved place (such as a unit bath in a hotel room) is turned on and off. The determination device counts the number of yawns and saves the history. Observe the frequency over time, and if it increases, send a message prompting you to go to bed.

  Summarize the process for sleeping.

  When you go to sleep, turn off the TV, adjust the audio volume, turn off the power, and turn on and off the lights. When sleeping, the air temperature of the air conditioner is mainly adjusted. When getting up, turn on the TV, adjust the volume of the audio, turn on the power, and turn on / off the lights. In addition, the user sets the wake-up time, but the device does not necessarily wake up on time. When it can be determined that sleep is shallow like REM sleep at a time earlier than the set time, it is determined that the user is “awake” and prompts to wake up.

[Conversation Function Unit 55]
The conversation function unit 55 includes a conversation recognition module 56 and a speech synthesis alarm module 57, which functions as an interface with the user, accepts function selection from the user,
It is also a function for giving instructions to the user. For example, it receives an instruction to set a wake-up time and control a device from a human. The system gives instructions to humans (such as “You should sleep when you have a lot of yawns”), greetings, etc.

[Control unit 3]
The control unit 3 in FIG. 1 has a function of performing device control and is shown in FIG.

  The instruction selection unit 61 selects control device information from the determination unit 2 and sends it to the instruction conversion unit 62. The instruction conversion unit 62 converts the control instruction into a specific operation (such as an electrical signal for switching on). It has a transmission function 63 for performing infrared remote control of various electric products and a switching function by attaching an auxiliary device 64 for directly turning on / off the power supply like a lighting fixture. These are characterized by being easily adaptable to the current indoor facilities.

  In addition, this invention is not limited to the Example mentioned above.

(1) Modification of Attitude Detection Function Unit 14 In order to handle a complicated operation by the attitude detection function unit 14, the following can be considered.

  In recent CG research, a language processing system for describing a human motion is constructed to move a human model, and a motion sequence is generated using the language. Using this description system as an expression format, we consider a method of converting a feature quantity obtained from an image into such a language, and take a syntactic matching method using that language.

(2) Modification of Mouth Portion Observation Function Unit 17 In the mouth portion observation function unit 17, a snake (M. Kass, A. Witkin, D. Terzopouls, which is a dynamic contour extraction method based on the energy minimization principle) “Snakes: Active Contour Models”, In Proc. 1st Int. Conf. On Computer Vision, pp. 259-268 (1987)). Shape information is obtained by extracting and tracking the contour shape of the lips by snake.

(3) Modified example of the face detection function unit 15 etc. As for the module using the color information such as the face detection function unit 15 etc., the method of using the infrared camera is described when it becomes dark. There is also a modification in which more reliable information is obtained by lighting and photographing for a few seconds.

(4) Modified Example of Judgment Unit 2 A modified example of the judgment unit 2 is shown in FIG.

  Incorporating not only human beings but also the object recognizing unit 71, recognizing futons, clothes, etc., and expanding to obtain useful information on temperature control and the like.

  By adding the voice recognition unit 72, the voice recognition unit 72 is used to monitor the state of conversation, the state of snoring at bedtime, and the like, and is used to assist information obtained from image information. For example, if it is uncertain whether or not a conversation is being performed as image information, it can be determined by combining audio information.

  If each recognition module fails to acquire information, new information may be obtained due to the failure.

  For example, as an example in which posture detection fails, as shown in FIG. 14B, when a futon is put on during sleep, the entire body cannot be seen, so the posture cannot be acquired from the image well. However, if time passes and the futon is removed due to the heat, it will be possible to acquire postures that have failed so far. Here, the determination device can determine “hot, hard to sleep” or the like, and the state can be determined by taking into account changes in observation results over time.

  A module 73 incorporating such inference based on failure is adopted. The configuration of this module 73 is illustrated in FIG. That is, it includes a failure example database 82, a matching unit 83, and a history storage unit 71.

  In addition, since it is possible to acquire information indicating that a certain direction is being viewed all the time for control information, it is possible to expand operations such as turning on the power by simply looking at the direction of the television by expanding the database.

(5) In this example, it is assumed that humans are sleeping, but the detection of unconscious behavior such as `` yawning '' and `` blinks '' and the control of the associated devices are also effective in other situations. is there.

  For example, a modification in which unconscious behavior that occurs during normal behavior such as the following is detected is also conceivable.

  For each audience in a theater or venue, the number of yawns, blinks, etc., and the frequency of conversations are examined to determine the state of the entire audience, and lighting, air conditioning, etc. are controlled.

  By recording the number of yawns, blinks, etc. as the status of attendees at the meeting, the concentration of attendees is grasped.

  Record the number of blinks of the VDT worker, judge the degree of fatigue, and call attention.

  At this time, as a modified example of the posture detection function, by detecting the orientation of the face, the movement of the neck, the movement of the limbs, etc. It is conceivable to detect unconscious body movements such as “unraveling”. These operations are characterized by repeatedly performing the same operations. By recording the size and the number of times of each operation, it becomes possible to grasp the human state.

(6) In the above embodiment, the subject is a human, but instead, the system may be used with animals such as lions and elephants as the subject. In this case, for example, it can be used to monitor an animal in a cage of a zoo, and it can be judged whether this animal is sleeping or in the wake-up state, thereby reducing the burden on the keeper. . Also at Safari Park. The system can be used to monitor animals.

As described above, the present invention can be implemented with various modifications without departing from the spirit of the present invention.

1 is a block diagram of a system. It is a block diagram of a detection part. It is a block diagram of an attitude | position detection function part. It is explanatory drawing of a posture model. It is a block diagram of a face detection function part. It is a block diagram of an eye part observation function part. It is a flowchart of eyelid detection. It is a flowchart and explanatory drawing of REM sleep detection. It is a flowchart and explanatory drawing of a mouth detection. It is a time transition model of a mouth shape, a flowchart and an explanatory diagram of detection. It is a block diagram of a judgment part. It is a figure which shows the example of the integrated judgment of a yawn. It is a block diagram of a control part. It is a block diagram of the modification of a judgment part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Detection part 2 Judgment part 3 Control part

Claims (4)

An image input means for observing the subject;
Movement detecting means for detecting the movement of the subject from the image observed by the image input means;
Motion value conversion means for converting the motion of the subject detected by the motion detection means into a quantitative motion value;
A judgment means for judging the state of the subject from the motion value converted by the motion value conversion means;
The operation value conversion means includes:
A state monitoring apparatus characterized in that a value of an evaluation function composed of a product of the amount of motion of the subject detected by the motion detection means and a weight coefficient is used as a quantitative motion value of motion of the subject. The state monitoring apparatus according to claim 1, further comprising a control unit that controls an external device based on the state of the subject determined by the determination unit. The determination means includes
Storage means for storing an item indicating the state of the subject corresponding to the motion value;
The state monitoring apparatus according to claim 1, further comprising a determination / determination unit that calls an item corresponding to an operation value from the operation value conversion unit from the storage unit and sets the called item to a state of a subject. The movement detection means has mouth detection means for detecting the mouth state of the subject from the image observed by the image input means,
The mouth detecting means is
The state monitoring apparatus according to claim 1, wherein it is determined from the mouth image of the subject observed by the image input means whether the mouth is in a closed state, a yawning state, or a conversational state.

Images