JP6896925B2 - Stress coping style judgment system, stress coping style judgment method, learning device, learning method, program that makes the computer function as a means to judge the stress coping style of the subject, learns the spatial features of the subject's facial image Programs and trained models that act as means - Google Patents

Description

The present invention relates to a technique for determining a subject's stress coping mode in a non-contact state.

Techniques for grasping the stress state of a subject are known. For example, Patent Document 1 describes a technique for estimating a psychological change of a subject by measuring the amount of heat radiated from the face.
Further, Patent Document 2 describes a technique for measuring the level of the psychological state of a subject based on facial image information. According to these techniques, it is possible to detect that a subject has had some psychological change in a non-contact state, or to quantitatively analyze the psychological state of a subject in a non-contact state.

However, in the techniques of Patent Documents 1 and 2, since the type of stress felt by the subject cannot be grasped, it is not possible to perform a qualitative analysis on the stress of the subject.

Japanese Unexamined Patent Publication No. 6-54836 JP-A-2007-68620

By the way, it is known that human beings exhibit a characteristic reaction pattern in order to achieve the purpose of the cardiovascular system satisfying the metabolic demand from each body tissue when faced with a stress stimulus. That is, there are a pattern showing active coping, a pattern showing passive coping, and a pattern not taking specific coping with stress. These are called "stress coping modalities" and when active coping patterns appear, the subject can be presumed to be in good stress. Conversely, when a pattern of passive coping appears, the subject can be presumed to be in poor stress.
That is, by determining the stress coping mode of the subject, it becomes possible to grasp the type of stress felt by the subject.

The present invention presents a stress coping style determination system capable of determining a subject's stress coping style in a non-contact state, a stress coping style determination method, a learning device, a learning method, and a means for determining a subject's stress coping style using a computer. Provided are a program that functions as a computer, a program that causes a computer to function as a means for learning a spatial feature of a subject's facial image, and a trained model.

In order to solve the above problem, the stress coping style determination system according to claim 1 comprises a biometric information acquisition unit that acquires the biometric information of the subject in a non-contact state, and the biometric information and a response pattern specified in advance. It has a determination unit that determines the stress coping mode of the subject based on the subject, and the response pattern is a stress coping mode determination system specified by a blood circulation dynamics parameter, and the blood circulation dynamics parameters are average blood pressure and heart rate. The biometric information is a facial image, and the response pattern includes "active coping" and "passive coping". The judgment unit includes three types of patterns, "active coping" and "no coping", and the determination unit includes a spatial feature corresponding to "active coping", a spatial feature corresponding to "passive coping", and "no coping". It has a determination feature amount storage unit that stores the spatial feature amount corresponding to the above, and the stress is based on the biological information and each spatial feature amount stored in the determination feature amount storage unit. It is determined which of the patterns of "active coping", "passive coping", and "no coping" is shown in the coping mode, and the feature amount stored in the determination feature amount storage unit is determined. It is a feature quantity extracted by the machine learning unit, and the machine learning unit has a plurality of learning facial images labeled corresponding to each of "active coping", "passive coping", and "no coping". A learning data storage unit that stores the above, a feature amount extraction unit that extracts the spatial feature amount of the face image from the learning face image using a learned model, and an extraction result by the feature amount extraction unit. Based on the relationship with the label attached to the learning facial image as the extraction target, the network parameters of the trained model so that the extraction accuracy of the spatial feature amount by the feature amount extraction unit is high. It is characterized by having a feature amount learning unit for changing the above.

This stress coping style determination system acquires the subject's biometric information in a non-contact state, and based on the biometric information and the response pattern specified by the hemodynamic parameters, the stress coping style of the subject is not determined. Judgment is made based on the contact state.

Here, "hemodynamics" is a subdivision of circulatory physiology that targets blood circulation, and is a research division that applies the theories of mechanics, elastic body mechanics, and fluid mechanics to biological systems.
Specifically, it studies the internal pressure of the heart, pulsation, workload, stroke, elasticity of blood vessels and myocardium, pulse, blood flow velocity, and viscosity of blood. Therefore, in the present invention, the "hemodynamic parameter" means a numerical parameter such as internal pressure of the heart, pulsation, workload, pumping volume, elasticity of blood vessels and myocardium, pulse, blood flow velocity, and blood viscosity. Is.
This stress coping mode determination system acquires the biological information of the subject in a non-contact state, and the biological information and a plurality of the average blood pressure, heart rate, cardiac output, stroke volume, and total peripheral resistance. Based on the response pattern identified by the parameters of, the stress coping mode of the subject is determined. Hemodynamic parameters can also be generally identified by a continuous sphygmomanometer.

This stress coping style determination system acquires the biometric information of the subject in a non-contact state, and based on the biometric information, the stress coping style of the subject is "active coping" or "passive coping". And, it is determined which of the response patterns of "no action" is shown.

This stress coping style determination system stores spatial features corresponding to "active coping", spatial features corresponding to "passive coping", and spatial features corresponding to "no coping", and is subject to stress. Based on the examiner's biological information and each spatial feature, the subject's stress coping mode shows any response pattern of "active coping", "passive coping", and "no coping". Determine if it is a style.

This stress coping style determination system determines the spatial features corresponding to "active coping", the spatial features corresponding to "passive coping", and the spatial features corresponding to "no coping" in the machine learning unit. Extract by.
The machine learning unit stores a plurality of learning facial images labeled corresponding to "active coping", "passive coping", and "no coping", and from the learning facial image to the subject's facial image. The spatial feature amount of the subject's facial image is extracted using the trained model, and the spatial feature amount of the subject's facial image is based on the relationship between the extraction result and the label attached to the learning facial image to be extracted. Change the network parameters of the trained model so that the extraction accuracy of is high.

Further, the stress coping style determination system according to claim 2 is the stress coping style determination system according to claim 1, wherein the facial image is a facial thermal image or a facial visible image.

This stress coping style determination system acquires a facial thermal image or facial visible image of a subject in a non-contact state, and based on the facial thermal image or facial visible image and the response pattern, the stress of the subject. Determine the coping mode.
In this case, the "face visible image" is an image obtained by photographing the face of the subject with a generally widely used camera, that is, a device having an optical system for forming an image and taking an image. is there. In this case, a color image is preferable. The "facial thermal image" is an image in which infrared rays radiated from the face of a subject are analyzed and the heat distribution is represented as a diagram, and is obtained by taking an image by infrared thermography.

Further, the stress coping style determination system according to claim 3 is the stress coping style determination system according to claim 1, wherein the determination unit observes the stress response of a specific part of the face included in the facial image. It is characterized in that the stress coping mode of the subject is determined by.

This stress coping style determination system determines the stress coping style of the subject based on the stress response of a specific part of the face of the subject and the response pattern.

Further, the stress coping style determination system of claim 4 is the stress coping style determination system according to any one of claims 1 to 3, and the spatial feature amount is calculated based on the facial image of the subject. It is characterized by being a fractal dimension.

Further, the program of claim 5 is a determination to store the spatial feature amount corresponding to "active coping", the spatial feature amount corresponding to "passive coping", and the spatial feature amount corresponding to "no coping". Based on the feature amount storage step, the facial image of the subject, and each spatial feature amount memorized by the determination feature amount storage step, the stress coping style of the subject is "active coping" and "passive". It has a determination step of determining which of the response patterns of "target coping" and "no coping" is exhibited, and the response pattern is specified by a hemodynamic parameter, and is identified by "active coping" and "active coping". A learning data storage step for storing a plurality of learning facial images labeled corresponding to "passive coping" and "no coping", and a trained model for the spatial features of the learning facial image. Based on the relationship between the feature amount extraction step to be extracted using the feature amount, the extraction result by the feature amount extraction step, and the label attached to the learning facial image as the extraction target, the feature amount extraction step is performed. It has a learning step of changing the network parameters of the trained model so that the extraction accuracy of the spatial feature quantity is high, and the determination feature quantity storage step is the spatial feature quantity extracted by the feature quantity extraction step. It is characterized in that it is a step of memorizing a feature amount.

This program is installed and executed on multiple computers that work in cooperation with one or more computers, so that the system consisting of the one or more computers is treated with a spatial feature "passive" corresponding to "active coping". Based on the spatial features corresponding to "targeted coping", the spatial features corresponding to "no coping", and the facial image of the subject, the stress coping style of the subject is "active coping" and "active coping". passive coping "and" Ru to function as a means for determining whether the manner showing any response pattern of the Action None. "

This program is installed and executed on multiple computers that work in cooperation with one or more computers, so that the system consisting of the one or more computers is "actively dealt with", "passively dealt with", and "no action taken". A plurality of learning facial images labeled corresponding to each of the above were stored, and the spatial features of the learning facial images were extracted using the trained model, and the extraction results and their extraction targets were used. A means for changing the network parameters of the trained model and storing the extracted spatial features so that the extraction accuracy of the spatial features is high based on the relationship with the label attached to the learning facial image. To function as.

Further, the program of claim 6 is characterized in that, in the program of claim 5, the spatial feature amount is a fractal dimension calculated based on a facial image of a subject .

Further, the stress coping style determination method according to claim 7 is a stress coping style determination method by a system having a biometric information acquisition unit and a determination unit, and the biometric information acquisition unit does not remove the biometric information of the subject. The response pattern includes a biological information acquisition step acquired in a contact state and a determination step of determining a stress coping mode of a subject based on the biological information and a response pattern previously specified by the determination unit. Specified by hemodynamic parameters, the hemodynamic parameters include a plurality of parameters of mean blood pressure, heart rate, heart rate output, single stroke amount and total peripheral resistance, and the biometric information is a facial image. Yes, the response pattern includes three types of patterns, "active coping", "passive coping", and "no coping", and the determination step includes spatial features corresponding to "active coping". A quantity, a spatial feature quantity corresponding to "passive coping" and a spatial feature quantity corresponding to "no coping" are stored by a determination feature quantity storage step, and the biological information and the determination feature quantity storage step. A response pattern determination step for determining which of the "active coping", "passive coping", and "no coping" patterns the stress coping style is based on each of the spatial features. And, and the feature amount memorized by the determination feature amount storage step is a feature amount extracted by the machine learning unit, and the machine learning unit has "active coping" and "passive coping". A learning data storage step for storing a plurality of learning facial images labeled corresponding to each of "no action" and a model in which the spatial features of the facial image are learned from the learning facial image. Based on the relationship between the feature amount extraction step to be extracted using the feature amount extraction step, the extraction result by the feature amount extraction step, and the label attached to the learning facial image as the extraction target, the feature amount extraction step is performed. It is characterized by executing a feature amount learning step of changing the network parameters of the trained model so that the extraction accuracy of the spatial feature amount is high .

This stress coping style determination method acquires the subject's biological information in a non-contact state, and determines the stress coping style of the subject based on the biological information and the response pattern specified by the hemodynamic parameters. To do.

Further, the learning device according to claim 8 includes a learning data storage unit that stores a plurality of learning facial images labeled corresponding to response patterns specified by blood circulation dynamics parameters, and the learning facial image. It is attached to the feature amount extraction unit that extracts the spatial feature amount of the subject's facial image from the subject using the trained model, the extraction result by the feature amount extraction unit, and the learning facial image that is the extraction target. It is characterized by having a feature quantity learning unit that changes the network parameters of the trained model so that the extraction accuracy of the spatial feature quantity by the feature quantity extraction unit is high based on the relationship with the label. ..

This learning device stores a plurality of learning facial images labeled according to the response pattern specified by the hemodynamic parameter, and learns the spatial features of the subject's facial image from the learning facial image as a trained model. Based on the relationship between the extraction result and the label attached to the learning facial image that was the target of the extraction, the accuracy of extracting the spatial features of the subject's facial image should be high. Change the network parameters of the trained model .

Further, the learning device according to claim 9 is characterized in that, in the learning device according to claim 8, the spatial feature amount is a fractal dimension calculated based on a facial image of a subject.

Further, the learning method of claim 10 includes a learning data storage step for storing a plurality of learning facial images labeled according to a response pattern specified by a blood circulation dynamics parameter, and a subject from the learning facial image. A feature amount extraction step for extracting the spatial feature amount of the facial image of the above using a trained model, an extraction result by the feature amount extraction step, and a label attached to the learning facial image to be extracted. It is characterized by having a feature amount learning step in which the network parameters of the trained model are changed so that the extraction accuracy of the spatial feature amount by the feature amount extraction step is high based on the relationship of.

This learning method stores a plurality of learning facial images labeled according to the response patterns specified by the hemodynamic parameters, and learns the spatial features of the subject's facial image from the learning facial images. Extraction is performed using the completed model, and the accuracy of extracting the spatial feature amount of the subject's facial image is improved based on the relationship between the extraction result and the label attached to the learning facial image to be extracted. Change the network parameters of the trained model so that .

Further, the learning method of claim 11 is characterized in that, in the learning method of claim 10, the spatial feature amount is a fractal dimension calculated based on a facial image of a subject.

Further, the program of claim 12 includes a learning data storage step for storing a plurality of learning facial images labeled according to a response pattern specified by a blood circulation dynamics parameter, and a learning data storage step of the subject from the learning facial image. The feature amount extraction step for extracting the spatial feature amount of the facial image using the trained model, the extraction result by the feature amount extraction step, and the label attached to the learning facial image to be extracted. A computer characterized by having a feature amount learning step that changes the network parameters of the trained model so that the extraction accuracy of the spatial feature amount by the feature amount extraction step is high based on the relationship. It is characterized in that it functions as a means for learning the amount of spatial features of the facial image of the subject.

This program is installed and executed on multiple computers that work in tandem with one or more of them, thereby transforming the system of one or more computers into a response pattern specified by hemodynamic parameters. A plurality of learning facial images labeled correspondingly are stored, the spatial features of the subject's facial image are extracted from the learning facial images using a trained model, and the extraction result and its extraction target are extracted. It functions as a learning device that changes the network parameters of the trained model so that the extraction accuracy of the spatial features of the subject's facial image is high based on the relationship with the label attached to the learning facial image. Let me.

The program of claim 13 is characterized in that, in the program of claim 12, the spatial feature amount is a fractal dimension calculated based on a facial image of a subject.

The trained model of claim 14 uses a plurality of trained facial images labeled corresponding to the response patterns specified by the hemodynamic parameters as the teacher data, and uses the spatial features of the subject's facial image as a machine. It is a learned model generated by learning, and the learning data storage step for storing the plurality of learning facial images and the spatial feature amount of the subject's facial image from the learning facial image have been learned. The feature amount extraction step is based on the relationship between the feature amount extraction step extracted using the model, the extraction result by the feature amount extraction step, and the label attached to the learning facial image as the extraction target. It is characterized in that it is generated by repeating the feature amount learning step of changing the network parameters of the trained model so that the extraction accuracy of the spatial feature amount is high.

Further, the trained model of claim 15 is characterized in that, in the trained model of claim 14, the spatial feature amount is a fractal dimension calculated based on a facial image of a subject.

According to the stress coping mode determination system of claim 1, the biometric information of the subject is acquired in a non-contact state, and the subject's biometric information and the response pattern specified by the hemodynamic parameters are used as the basis for the subject's biometric information. Since the stress coping mode can be determined in a non-contact state, it is possible to grasp the type of stress felt by the subject without imposing behavioral restrictions on the subject.
In general, the word "stress" is widely used, but psychologically, "stress" is "a general term for non-specific reactions that occur to external stimuli (stressors) that act on living organisms." (Psychologist Dr. Hans Selye).
There are various stressors based on human survival, and in recent years it is known that there are many stressors especially in the social environment, which have various effects on living organisms, and in some cases, they are also the cause of diseases. It is medically known to be.
However, it is known that stress is not all bad for human existence, and even if stress is applied, the living body may be activated and have a positive effect when trying to cope with stress. Has been done. From this point of view, Dr. Serie has clarified that good stress (enstress) and bad stress (distress) can occur depending on the difference in biological conditions on the receiving side and the degree of stress.
Therefore, from such a psychological point of view, when considering stress, it is necessary to classify and consider the types of stress for the living body as described above, rather than grasping all stress in general as bad. Yes, and from this perspective, think socially, industrially, and positively about how to manage stress in various modern social activities, for example, to improve production efficiency and work efficiency. It becomes possible.
However, as described above, although the technique and the patented invention for grasping the stress state of the subject are known, the stress felt by the subject is generally grasped by the conventional method, and the stress felt by the subject is grasped as a whole. It is not possible to perform a qualitative analysis based on the type of.
According to the present invention, by classifying and grasping the types of stress according to the subject, it is possible to grasp the relationship between the stressor and the stress response by the subject in more detail. As a result, the relationship between humans and the social environment, which is a stressor, can be analyzed and examined more accurately, and it can be applied to various social environment fields to solve problems and various problems. It can be applied to various industrial fields to improve production efficiency and promote industrial activities.

According to this stress coping mode determination system, the biological information of the subject and the response pattern specified by any one of the mean blood pressure, heart rate, cardiac output, stroke volume and total peripheral resistance are obtained. Based on this, the stress coping mode of the subject can be determined in detail and accurately.

According to this stress coping style determination system, a face image of a subject is acquired in a non-contact state, and the stress coping style of the subject is based on the face image and the response pattern specified by the blood circulation dynamics parameter. Since the continuous blood pressure monitor is not used as in the conventional case, the subject can be quickly determined without restraining the subject and without imposing a physical burden.

According to this stress coping style determination system, the subject's biological information is acquired in a non-contact state, and the stress coping style of the subject is "active coping" or "passive" based on the bioinformation. It is possible to determine which of the response patterns, "Coping" and "No coping", is shown.
Therefore, the stress coping style can be classified into the three types of response patterns of "active coping", "passive coping", and "no coping". Therefore, analysis is performed based on this response pattern, and the analysis results are various. It can be applied to personnel management work, quality control work, etc. in the industrial field and contribute to quality improvement of various work.

According to this stress coping style determination system, the spatial features corresponding to "active coping", the spatial features corresponding to "passive coping", and the spatial features corresponding to "no coping" are memorized. , Based on the subject's biological information and each spatial feature, the subject's stress coping mode is any response pattern of "active coping", "passive coping", and "no coping". It can be determined whether the format indicates.

According to this stress coping style determination system, the network parameters of the trained model can be changed so that the extraction accuracy of the spatial features of the facial image of the subject is high.

According to the stress coping mode determination system of claim 2, the facial thermal image or facial visible image of the subject is acquired in a non-contact state, and the response pattern specified by the facial thermal image or facial visible image and the hemodynamic parameter. Based on the above, the stress coping mode of the subject can be determined based on psychophysiology.

According to the stress coping style determination system of claim 3, the stress coping style of the subject is easily and accurately determined based on the state of a specific part of the face of the subject and the response pattern specified by the hemodynamic parameters. can do.

According to the stress coping style determination system of claim 4, since the spatial feature amount can be quantified with high accuracy by the fractal dimension, the stress coping style of the subject can be determined with high accuracy in a non-contact state.

According to the program of claim 5, the spatial feature amount corresponding to "active coping", the spatial feature amount corresponding to "passive coping", the spatial feature amount corresponding to "no coping", and the subject. Based on the facial image of the subject, a system for determining whether the stress coping style of the subject shows a response pattern of "active coping", "passive coping", and "no coping". It can be realized by using one or a plurality of computers that work in cooperation with each other.

According to this program, a plurality of learning facial images labeled corresponding to "active coping", "passive coping", and "no coping" are stored, and spatial features of the learning facial image are stored. The quantity is extracted using the trained model, and the extraction accuracy of the spatial feature quantity is increased based on the relationship between the extraction result and the label attached to the learning facial image to be extracted. A system for changing the network parameters of the trained model and storing the extracted spatial features can be realized by using one computer or a plurality of computers working in cooperation with each other.

According to the program of claim 6, since the spatial features can be quantified with high accuracy by the fractal dimension, a system capable of determining the stress coping mode of the subject with higher accuracy in a non-contact state can be described. Alternatively, it can be realized by using a plurality of computers that work in cooperation with each other.

According to the stress coping mode determination method of claim 7, the biological information of the subject is acquired in a non-contact state, and the subject's biological information is based on the biological information and the response pattern specified by the hemodynamic parameter. Since the stress coping mode can be determined in a non-contact state, it is possible to grasp the type of stress felt by the subject without imposing behavioral restrictions on the subject.

According to the learning device of claim 8, the network parameters of the trained model can be changed so that the extraction accuracy of the spatial features of the facial image of the subject is high.

According to the learning device of claim 9, since the spatial features can be quantified with high accuracy by the fractal dimension, the network parameters of the trained model so that the extraction accuracy of the spatial features of the facial image of the subject becomes higher. Can be changed.

According to the learning method of claim 10, the network parameters of the trained model can be changed so that the extraction accuracy of the spatial features of the facial image of the subject is high.

According to the learning method of claim 11, since the spatial features can be quantified with high accuracy by the fractal dimension, the network parameters of the trained model so that the extraction accuracy of the spatial features of the facial image of the subject becomes higher. Can be changed.

According to the program of claim 12, by installing and executing this on one computer or a plurality of computers working in cooperation with each other, the trained model is designed so that the extraction accuracy of the spatial feature amount of the facial image of the subject is improved. A learning device that changes network parameters is realized.

According to the program of claim 13, the spatial feature amount can be quantified with high accuracy by the fractal dimension. Therefore, by installing and executing this program on one computer or a plurality of computers working in cooperation with each other, a facial image of a subject is obtained. A learning device that changes the network parameters of the trained model so that the extraction accuracy of the spatial feature quantity of the above is higher is realized.

According to the trained model of claim 14, by inputting the facial image of the subject into the model, the spatial feature amount of the facial image of the subject can be extracted.

According to the trained model of claim 15, the spatial features can be quantified with high accuracy by the fractal dimension. Therefore, by inputting the facial image of the subject into the trained model, the spatial image of the subject is spatially. Features can be extracted with high accuracy.

It is a block diagram of one Embodiment of the stress coping style determination system which concerns on this invention. It is a flowchart which shows the processing content of the determination apparatus which comprises the stress coping style determination system of FIG. (A) is an explanatory diagram conceptually exemplifying a group of facial images for learning labeled as "active coping". (B) is an explanatory diagram conceptually exemplifying a group of facial images for learning labeled as "passive coping". (C) is an explanatory diagram conceptually exemplifying a group of facial images for learning labeled as "no action". It is a flowchart which shows the processing content of the learning apparatus which comprises the stress coping style determination system of FIG. It is a table which shows the hemodynamic pattern reaction of the previous study of Experimental Example 1. It is a conceptual diagram which shows the measurement system. It is a conceptual diagram which shows a mirror drawing task. It is a graph which shows the time series change of MBP (mean blood pressure). It is a schematic diagram which shows the structure of CNN (convolutional neural network). It is a table which shows the filter size, stride, and the number of filters of a convolution layer. It is a table which shows the filter size and stride of a pooling layer. It is a figure which shows the facial feature in each case of active coping, passive coping, and no coping of each subject, and contrasts a thermal image with a facial feature map. It is a conceptual diagram which shows the experimental protocol of Experimental Example 2. It is a graph which shows the relationship between the degree of preference and the degree of concentration. It is a figure which shows the electroencephalogram electrode arrangement state to the subject, and the measurement system. A table showing changes in the multifaceted emotional scale for "positive" content. A table showing changes in the multifaceted emotional scale for "negative" content. It is a table showing changes in the multifaceted emotional scale for "horror (concentrated)" content. It is a table showing the fluctuation of the multifaceted emotional scale for "horror (non-concentrated)" content. It is a table showing the fluctuation of the subjective psychological index for "positive" content. It is a graph which shows the fluctuation of the subjective psychological index for "negative" content. It is a table showing the fluctuation of the subjective psychological index for "horror (concentration)" content. It is a graph which shows the fluctuation of the subjective psychological index for "horror (non-concentrated)" content. It is a table showing the time-series fluctuation of the physiological index for "positive" and "negative" content viewing. It is a table which shows the evaluation of the physiological index for each content. It is a graph which shows the time-series variation of the physiological index with respect to "horror (concentration)" and "horror (non-concentration)" content viewing. It is a graph which shows the relationship between the preference for TV content and the viewing mode. It is a conceptual diagram which shows the experimental protocol of Experimental Example 3. It is a conceptual diagram which shows the experimental protocol. It is a table which shows the structure of the neural network used for the estimation of "excitement-sedation", "stress coping mode", and "preference". It is a graph which shows the extraction method of a feature vector. It is a table which shows the evaluation of the subjective psychological index for each content. It is a table which shows the evaluation of the physiological index for each content. It is a table which shows the preference for TV contents and the correct discrimination rate of a viewing mode. It is a graph which shows the positive discrimination rate of a preference for a TV content and a viewing mode. It is a graph which shows the time variation of the measured value and the estimated value of the "excitement-sedation" state. It is a graph which shows the estimation error of the "excitement-sedation" state. It is a graph which shows the relationship between the measured value and the estimated value of the "excitement-sedation" state. It is a flowchart which illustrates the method of finding the fractal dimension as a spatial feature quantity. It is explanatory drawing which shows the example of the clustering process in FIG. 39. It is explanatory drawing which shows the Example of the image extraction process and the edge extraction process in FIG. 39. It is explanatory drawing which shows the example of the fractal analysis processing in FIG. 39.

Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.
[Constitution]
The stress coping style determination system 100 of one embodiment shown in FIG. 1 includes a biological information acquisition device (biological information acquisition unit) 110, a determination device (determination unit) 120, and a learning device (machine learning unit) 130. ..

The biological information acquisition device 110 is a device for acquiring the biological information of the subject P in a non-contact state.
Facial image IF can be mentioned as the most preferable example of biological information. In the following description, the case where the facial image IF is used as the biological information will be described.

The facial image IF may be a facial thermal image or a facial visible image. When the facial image IF is a facial thermal image, infrared thermography is used as the biological information acquisition device 110. When the face image IF is a face visible image, a so-called camera as a visible image imaging device is used as the biological information acquisition device 110.
As described above, the "face visible image" is obtained by photographing the face of the subject with a generally widely used camera, that is, a device having an optical system for forming an image and taking an image. It is a color image. The "facial thermal image" is an image in which infrared rays radiated from the face of a subject are analyzed and the heat distribution is represented as a diagram, and is obtained by taking an image by infrared thermography.
In this case, in the case of the image taken by the camera, it is due to visible light (wavelength of 380 nm to 800 nm), while the heat distribution image by infrared thermography is due to infrared rays (wavelength of 800 nm or more). In either case, since there is only a difference in wavelength, both infrared thermography and a general camera can be used as the biometric information acquisition device 110.

The determination device 120 is realized by installing and executing the program according to the present invention on a general-purpose computer.

The determination device 120 is a device having a function of determining the stress coping mode of the subject based on the facial image IF acquired by the biological information acquisition device 110 and the response pattern specified in advance. The response pattern includes three types of patterns consisting of "active coping" (pattern I), "passive coping" (pattern II) and "no coping" (pattern III). The response pattern is specified by hemodynamic parameters. Hemodynamic parameters include multiple parameters of mean blood pressure (MBP), heart rate (HR), cardiac output (CO), stroke volume (SV) and total peripheral resistance (TPR).

The determination device 120 includes a determination feature amount storage unit 121, a specific site reaction detection unit 122, and a response pattern determination unit 123.

The determination feature amount storage unit 121 stores the spatial feature amount corresponding to "active coping", the spatial feature amount corresponding to "passive coping", and the spatial feature amount corresponding to "no coping". It is a functional block. The spatial feature amount stored in the determination feature amount storage unit 121 is the spatial feature amount extracted by the learning device 130. When the facial image IF is a facial thermal image, the facial skin temperature distribution can be mentioned as an example of the spatial feature amount.

The specific site reaction detection unit 122 is a functional block that detects the stress response of the anatomical specific site of the face of the subject P included in the facial image IF. The anatomical specific site is one or more sites identified from anatomical findings as a site that is relatively small depending on individual differences. An example of an anatomically specific site is the crown of the nose.

In the response pattern determination unit 123, the stress coping mode of the subject P is set based on the stress response detected by the specific site reaction detection unit 122 and each spatial feature amount stored in the determination feature amount storage unit 121. It is a functional block that determines which of the response patterns of "active coping" (pattern I), "passive coping" (pattern II), and "no coping" (pattern III) is exhibited.

The learning device 130 includes a learning data storage unit 131, a feature amount extraction unit 132, and a feature amount learning unit 133. The learning device 130 is realized by installing and executing the program according to the present invention on a general-purpose computer.

The learning data storage unit 131 is a functional block that stores a plurality of learning facial image LGs labeled corresponding to each of "active coping", "passive coping", and "no coping". .. FIG. 2 conceptually illustrates the learning facial image LG. LGA1, LGA2, LGA3, ... Shown in FIG. 2 (A) are a group of facial images for learning labeled as "active coping". LGB1, LGB2, LGB3, ... Shown in FIG. 2B are a group of facial images for learning labeled as "passive coping". Further, LGC1, LGC2, LGC3, ... Shown in FIG. 2C are a group of facial images for learning labeled as "no action".

The feature amount extraction unit 132 is a functional block that extracts the spatial feature amount of the face image from the learning face image LG by using the trained model 134. The trained model 134 uses a plurality of learning face image LGs labeled corresponding to the response patterns specified by the hemodynamic parameters as the teacher data, and includes the face image of the subject P included in the learning face image LG. It is generated by machine learning the spatial features of.

The feature amount learning unit 133 is based on the relationship between the extraction result by the feature amount extraction unit 132 and the label attached to the learning facial image LG as the extraction target, and the spatial feature amount by the feature amount extraction unit 132. This is a functional block that changes the network parameters of the trained model 134 so that the extraction accuracy of the model 134 is high.

[motion]
Next, the flow of processing in the determination device 120 and the learning device 130 of the stress coping style determination system 100 configured as described above will be described with reference to the flowcharts of FIGS. 3 and 4.

As shown in FIG. 2, the determination device 120 executes the determination feature amount storage process S1, the specific site reaction detection process S2, and the response pattern determination process S3.

The determination feature amount storage process S1 is a spatial feature amount extracted by the learning device 130, that is, a spatial feature amount corresponding to "active coping", a spatial feature amount corresponding to "passive coping", and "coping". This is a process of storing the spatial feature amount corresponding to “none” in the determination feature amount storage unit 121.

The specific site reaction detection process S2 is a process for detecting the stress response of the anatomical specific site of the face of the subject P included in the facial image IF captured by the biological information acquisition device 110.

In the response pattern determination process S3, the stress coping mode of the subject P is set based on the stress response detected by the specific site reaction detection process S2 and each spatial feature amount stored in the determination feature amount storage unit 121. It is a process of determining which of the response patterns of "active coping", "passive coping", and "no coping" is shown.

As shown in FIG. 4, the learning device 130 executes the learning data storage process S11, the feature amount extraction process S12, and the feature amount learning process S13.

The learning data storage process S11 stores a plurality of learning facial image LGs labeled corresponding to each of "active coping", "passive coping", and "no coping" in the learning data storage unit 131. It is a process to do.

The feature amount extraction process S12 is a process of extracting the spatial feature amount of the subject's face image included in the learning face image LG from the learning face image LG by using the trained model 134.

The feature amount learning process S13 extracts the feature amount by the feature amount extraction process S12 based on the relationship between the extraction result by the feature amount extraction process S12 and the label attached to the learning facial image LG as the extraction target. This is a process of changing the network parameters of the trained model 134 so as to increase the accuracy.

[Spatial features]
In this embodiment, the fractal dimension calculated based on the facial image IF can be used as the spatial feature amount. By using the fractal dimension, the spatial features can be easily and highly accurately quantified.

The method of finding the fractal dimension as a spatial feature is arbitrary. In the processing flow illustrated in FIG. 39, the fractal dimension is obtained by executing a series of processes including the clustering process S21, the image extraction process S22, the edge extraction process S23, and the fractal analysis process S24.

The clustering process S21 is a process of clustering the facial image IF with respect to the temperature distribution. Although the clustering method is arbitrary, the Fuzzy-c-means method can be mentioned as a method suitable for this embodiment. The Fuzzy-c-means method does not consider the alternative question of whether or not a certain data group belongs to a cluster, but other than the situation where the data completely belongs to a single cluster (k-means method). In addition, it is a method that vaguely indicates the degree of belonging (attribution) of data to a cluster, assuming that it belongs to a plurality of clusters to some extent. FIG. 40 shows an example in which the number of clusters is set to 12 and the input image (face image IF) is clustered. In the example of FIG. 40, cluster1 belongs to the lowest temperature distribution and cluster12 belongs to the highest temperature distribution.

The image extraction process S22 is a process of extracting an image of a cluster having a temperature distribution in a temperature region equal to or higher than a predetermined temperature from an image of a plurality of clusters obtained by the clustering process S21. In the example of FIG. 41, from the images of clusters 1 to 12 illustrated in FIG. 40, the images of clusters 8 to 12, which are attribution images including the temperature of the face region, are extracted.

The edge extraction process S23 is a process of detecting an edge portion of the image extracted by the image extraction process S22 and generating an edge figure composed of a line segment represented by the edge portion. Although the edge detection method is arbitrary, the Canny method can be mentioned as a method suitable for this embodiment. The canny method uses a Gaussian filter for noise removal processing, a Sobel filter for brightness gradient (edge) extraction processing, non-maximum suppression processing for removing parts other than the part where the edge strength is maximized, and whether or not the edge is correct. This is a method of detecting an edge through a hysteresis threshold value determination based on a threshold value using hysteresis. In the example of FIG. 41, the edge portions of the images of clusters 8 to 12 are detected by the canny method, and an edge figure composed of a line segment represented by the edge portions is generated.

The fractal analysis process S24 is a process for obtaining the fractal dimension of the edge figure generated by the edge extraction process S23. The fractal dimension is an index that quantitatively expresses the self-similarity and complexity of figures and phenomena, and is generally a non-integer value. The method of fractal analysis is arbitrary, but a box counting method can be mentioned as a method suitable for this embodiment. In the box counting method, when the figure to be analyzed is divided into square boxes (lattice), the slope of the straight line when the relationship between the size of the box and the total number of boxes including the figure is linearly approximated on a log-log graph. This is a method of finding the fractal dimension from the absolute value of.

The formula for calculating the fractal dimension (D) is expressed by the following formula. r is the size of the box and N (r) is the number of boxes.

FIG. 42 shows an example of calculating the fractal dimension of the edge figure of cluster 12 in FIG. 41. In this example, for the edge figure of cluster12, r is varied in the range of 2 to 128 and r and N (r) are plotted on a log-log graph to obtain a value of 1.349.

[Action / Effect]
In the stress coping style determination system 100 configured as described above, the facial image IF of the subject P is imaged by the biological information acquisition device 110. The captured facial image IF is input to the determination device 120. The determination device 120 receives the input facial image IF and a response pattern specified by any of hemodynamic parameters, that is, mean blood pressure, heart rate, cardiac output, stroke volume, and total peripheral resistance. Based on patterns I, II and III), it is determined which of the patterns I, II and III the stress coping mode of the subject P is.

Therefore, according to this stress coping style determination system 100, the stress coping style of the subject P can be determined in a non-contact state based on the facial image IF of the subject P, so that the subject P is given a behavioral constraint. It is possible to grasp the type of stress felt by the subject P without having to do so.

Further, the stress coping mode determination system 100 is based on the reaction and response pattern (patterns I, II and III) of the anatomically specific part of the face included in the facial image IF of the subject P, and the subject P is the subject P. Judge the stress coping style of. The anatomical specific site is one or more sites identified from anatomical findings as a site that is relatively small depending on individual differences.

Therefore, according to the stress coping style determination system 100, a general-purpose system for determining the stress coping style can be constructed.

Further, the stress coping style determination system 100 extracts the feature amount corresponding to "active coping", the feature amount corresponding to "passive coping", and the feature amount corresponding to "no coping" by the learning device 130. .. The learning device 130 stores a plurality of learning facial image LGs labeled corresponding to "active coping", "passive coping", and "no coping", and from the learning facial image LGs. The spatial feature amount of the facial image is extracted using the trained model 134, and the spatial feature of the facial image is spatially based on the relationship between the extraction result and the label attached to the learning facial image LG as the extraction target. The network parameters of the trained model LG are changed so that the extraction accuracy of the feature amount becomes high. As the learning of the trained model LG progresses, the accuracy of extracting the spatial features of the facial image is improved, and the accuracy of the spatial features stored in the determination feature storage unit 121 is also improved.

Therefore, according to the stress coping style determination system 100, the determination accuracy of the stress coping mode can be improved as the learning of the learned model LG in the learning device 130 progresses.

Further, according to this stress coping style determination system, the determination accuracy of the stress coping style can be further improved by quantifying the spatial feature amount by the fractal dimension.

The present invention is not limited to the above embodiment. For example, in the above embodiment, the stress coping style determination system 100 includes the learning device 130, but the learning device 130 can be omitted. When the learning device 130 is omitted, the spatial feature amount extracted or generated by means other than the learning device 130 is stored in the determination feature amount storage unit 121 of the determination device 120.

<Experimental example 1>
An example of an experiment for collecting feature amount data stored in the determination feature amount storage unit 121 according to the present embodiment is shown below.

1. 1. Experimental method The experiment was conducted in a shielded room at 25.0 ± 1.5 ° C., and the subjects were 8 healthy adult males aged 18 to 21 years. The measurement system is shown in FIG. The subject was seated and was fitted with a measuring cuff of a continuous blood pressure / hemodynamic dynamics measuring device (Finometer model 2, Finapres Medical Systems B. V) on the second joint of the middle finger of the left hand.

Hemodynamic parameters measured mean blood pressure (MBP), heart rate (HR), cardiac output (CO), and total peripheral resistance (TPR). Infrared thermography (TVS-200EX, AVIONICS) was installed at an angle of view that can measure the entire face to measure facial thermal images at a position 1.2 m in front, and the shooting interval was set to 1 fps. When measuring the facial skin temperature, I sat on a backrest chair and projected the computer screen at a distance of 1.55 m toward the wall with a projector.

The experiment was conducted in the time divisions of resting eyes closed 60s (Rest1) before Task, active task 60s (Task1), no action task 60s (Task2), passive task 60s (Task3), and resting eyes closed 60s (Rest2) after Task. It is composed. Mental arithmetic tasks were performed for active tasks, mirror depiction tasks for passive tasks, and resting eyes closed for tasks without stress coping. In the mental arithmetic task, double-digit addition was performed every 4 seconds on the computer screen. The correctness of mental arithmetic was not announced each time. In the mirror drawing task, he was taught to reproduce the star-shaped figure (see Fig. 7) displayed on the screen on a computer and use a mouse with his right hand to pass between the lines. The movement of the mouse and the movement of the cursor on the screen were reversed vertically and horizontally.
The cursor and the trajectory of the cursor were displayed on the star-shaped figure, and when it protruded, the cursor was returned to the initial position and all the tracks were erased. The orbit start position was the top of the star shape. Resting eyes were used as a task without coping. Normalization was performed by subtracting the average value of Rest1 in the hemodynamic parameters as the baseline.

2. Analysis method 2-1 Discrimination of hemodynamic parameters Figure 8 shows the time-series changes in the normalized MBP of subject A. For each hemodynamic parameter, the range with a value greater than or equal to baseline + 2σ (σ: standard deviation of each hemodynamic parameter at Rest1) is defined as “+”, and the range with a value below baseline −2σ is defined as “-”. Defined. Based on the pattern reaction in Table 1, each hemodynamic parameter was discriminated as "Pattern I" (active coping) or "Pattern II" (passive coping), and when neither was applicable, it was judged as "no coping". In addition, the thermal images taken between Tasks 1 to 3 were labeled based on the discrimination of hemodynamic parameters.

2-2 Creation of input data In order to use the labeled thermal image for machine learning, the facial thermal image was created by trimming the facial part with 151 × 171 pixels and grayscale it. In addition, in order to match the number of facial thermal images (input data) labeled for each stress coping, random cropping, white Gaussian noise addition, and contrast adjustment were performed to expand the data.

2-3 Machine learning using CNN In this experiment, a convolutional neural network (CNN) was used to construct an individual discrimination model of stress coping style based on facial skin temperature distribution, and features related to stress coping style. Extraction was performed.

The CNN is composed of three convolutional layers for extracting features, three pooling layers, and one fully connected layer for discrimination. The structure of the CNN is shown in FIG. 9, and the filter parameters of the convolution layer and the pooling layer are shown in FIGS. 10 and 11. The feature analysis related to the stress coping mode was performed based on the feature map, which is the weight of the convolutional layer of CNN.

3. 3. Results and Discussion FIG. 12 shows a feature map of the second convolutional layer when facial thermal images of each stress coping mode were input to CNN for each subject.
As a result of observing the feature map for each coping style in the subject, in particular, in subject B, features were found on the left cheek when there was no active coping and stress coping, and on the right cheek when passive coping, and each stress coping style. It was shown that the characteristic sites expressed between them differed. As a result of looking at the feature map among the subjects, most of the subjects showed features in the nose, but each subject showed individual differences in the feature expression site.
Differences in the structure of blood vessels and fat are considered to be the reason why individual differences were observed in the site of characteristic expression among the subjects. However, by identifying anatomically meaningful feature sites, it is possible to pave the way for the construction of a general model for stress coping discrimination.

4. Summary In this experiment, we attempted to discriminate stress coping styles from facial skin temperature distribution and extract features related to stress coping styles by using CNN. As a result, the characteristic distribution expressed in the facial skin temperature distribution changed depending on the stress coping style, and individual differences were observed in the characteristic distribution related to the stress coping style.

References
[1] Yuki Sawada Exhibition: New Physiological Psychology (supervised by Hiroshi Miyata), Chapter 10, Kitaooji Shobo, pp. 172-195, (1998)
[2] Yuichiro Nagano: "Cardiovascular Reactions in Competitive Reflection Tasks", Physiological Psychology and Psychophysiology, Vol.22, No3, pp.237-246, (2004)
[3] Yasushi Bando, Hirotoshi Asano, Akio Nozawa: "Analysis of Physiological Effects of Reading Display Media", IEEJ Transactions on Electronics, Information and Systems, Vol.135, No.5, pp. 520-525, (2015)
[4] Takuya Watanuki, Akio Nozawa: "Analysis of TV viewing patterns based on stress coping styles", IEEJ Transactions on Electronics, Information and Systems, Vol.134, No.2, pp.205-211 ， (2014)
[5] Satoshi Hioki, Akio Nozawa, Tota Mizuno, Hideto Ide: "Physiological Psychological Evaluation of Mental Workloads under Temporal Pressure", IEEJ Transactions on Electronics, Information and Systems, Vol. 127, No.7, pp.1000-1006, (2007)
[6] Hiroki Ito, ShizukaBando, Kosuke Oiwa, AkioNozawa: "Evaluation of Variations in Autonomic NervousSystem's Activity During the Day Based on Facial ThermalImages Using Independent Component Analysis", IEEJTransactions onElectronics, Information and Systems, Vol.138, No.7, pp. 1-10, (2018)
[7] Kenta Matsumura, Yukihiro Sawada: "Cardiovascular reactions when performing two types of mental arithmetic tasks", Psychology Research, Vo79, No.6, pp. 473-480, (2008)

In addition, two experimental research examples by the inventor of the present application, which are basic research of the stress coping mode determination system according to the present invention, are shown below as one application example (example) of the present invention.
These experimental examples were performed by attaching a cuff to the subject using a continuous sphygmomanometer, instead of "acquiring the biological information of the subject in a non-contact state" as in the present invention. Similarly, based on the blood circulation dynamics parameters, the physiological and psychological states of the viewer when viewing the TV video content were analyzed, classified, and evaluated. Therefore, the invention of the present application can be applied to, for example, the determination of the stress mode of the subject in such a situation.

<Experimental example 2>
Purpose of the experiment Today, people around us are full of information devices. Among them, since its invention, television has been installed in the living room of homes and has been recognized as the center of news media and entertainment media. However, with the spread of miniaturized and high-performance information and communication equipment due to the progress of IT technology in recent years and the development of the information network environment, the way of information media has changed drastically, and today the way of watching TV has also changed drastically. I'm doing it.

Nishigaki states that he is far from a "concentrated viewer", such as always turning on the TV switch while doing chores and operating a mobile phone at the same time (1). However, if we can grasp the changing viewing mode, it will be possible to add new added value to TV, such as adjusting mood, feelings, and behavior in the desired direction by TV content design.
Fujiwara and Saito conducted an opinion poll, and Ohno conducted a questionnaire survey. The reasons for watching TV were "killing time", "enjoying interesting programs", "enhancing education and gaining knowledge", "somehow by habit", Reasons such as "to escape from the troublesome reality", "to change the mood", "BGM", and "to help the family" are given (2) (3).

In addition, Takahashi et al. Classify the time to watch TV into "breakfast", "commuting", "doing housework", "relaxing at home on weekdays", etc. (4). In addition, Tomomune et al. Changed the TV viewing mode to "dedicated viewing" in which "the head and body are concentrated on the TV" and "while watching this program while doing other things". It is categorized into "dedicated switching viewing" and "time-comfortable viewing" in which "the body concentrates on the TV, although there is nothing I want to see" (5).

In this way, it has been clarified that the viewing mode is various and changes depending on factors such as time, place, and mood. The taste for TV images may also be one of the factors that change the viewing style. However, there are no studies that classify viewing modes according to taste. Furthermore, most of the studies that discriminate these viewing modes use psychological responses such as opinion polls and introspection evaluations after the fact, and there are no studies that classify viewing patterns using physiological responses.

Since television video and audio are audiovisual stimuli for living organisms, they are considered stressors in their own right. The stressor encourages the living body to cope with stress and brings about physiological and psychological changes as a stress response. If various contents of television are regarded as stressors, it is expected that the physiological stress response will differ depending on the coping with the contents, that is, the viewing mode. Nomura et al. Classify how to deal with e-learning content by cardiovascular index (6).

Therefore, in this study, for the purpose of analyzing the viewing mode for TV contents by the cardiovascular index, cardiovascular measurement by continuous sphygmomanometer, non-contact facial skin temperature measurement by infrared thermography device, digital biological amplifier We evaluated the activity of the central nervous system and the activity of the autonomic nervous system by measuring the electrocardiogram and brain waves. In particular, we analyzed the viewing mode when watching TV images based on the stress coping style using hemodynamic parameters and the preference for TV images.

2. Factor extraction experiment We considered that the viewing mode was influenced by the video content and personal tastes for it, and classified the factors by a preliminary experiment.

<2.1> Experimental conditions The subjects were 20 healthy students (age: 19-22, average age: 21.1, male: 11, female: 9) enrolled in the university. The subjects were fully informed in advance of the content, purpose, and subject of the experiment, both verbally and in writing, and their consent to the cooperation of the experiment was confirmed by signature. The measurement was performed in a shielded room with no convection at room temperature of 26.0 ± 1.6 ° C, and it was assumed that there was no temperature input from the outside.

The subject was sitting and using an LCD TV (55 inches, HX850, BRAVIA, made by SONY) installed 2 m in front of him, 10 kinds of video contents (news, sports, documentary, music, horror, etc.) for 5 minutes. Watch comedy, drama, anime, cooking, shopping). The video content was played on a DVD player (SCPH-3900, made by SONY). The video content to be viewed will not be published until the subject views it. In addition, in order to exclude the order effect, the order of the video contents to be viewed is random.

<2.2> Experimental procedure As shown in Fig. 13, this experiment consists of 5 minutes of video viewing and 1 minute of rest and eye closure R1 and R2 before and after that. In addition, the preference and immersive feeling of each video content were evaluated by the Visual Analogue Scale (hereinafter abbreviated as VAS) and the immersive feeling survey.

<2 ・ 3> Evaluation method At the end of the experiment, the subjective sensory amount of "preference" was measured using VAS. In VAS, it is possible to measure the psychophysical quantity of a subject by arranging paired adjectives at both ends of the line segment and the subject checking at an arbitrary position on the line segment. I said "I hate it very much"-"I like it very much" for the words placed at both ends of the scale. In addition, the immersive feeling in the video was measured on a 5-point scale (1: I couldn't immerse at all-5: I was quite immersive).

<2.4> Results and discussion of factor extraction experiments The left side of Fig. 14 shows the average of all subjects in terms of preference and immersiveness of 10 types of content. Error bars in the figure represent standard errors in taste and immersiveness. On the right side of FIG. 14, the tastes and immersive feelings of 20 subjects in news and horror, which had a large individual difference among the 10 types of contents, are shown. Looking at the taste and immersive feeling of each content from the left of FIG. 14, the average taste and immersive feeling are in a direct proportional relationship. In FIG. 14, N = 20.

Regarding horror, it showed a peculiar result that the taste was low but the immersive feeling was high. This is thought to be due to the so-called "seen scary things" that I hate but are curious about and want to watch. From the right side of FIG. 14, since the taste and immersive feeling are different for each subject, it can be seen that individual differences are large. Therefore, in the physiological measurement experiment, based on the above-mentioned factor classification experiment, each subject has high immersive / positive preference (high preference) video content, low immersive / negative preference (low preference) video content, and negative preference. However. It was decided to divide and present the horror video showing the peculiar result of being divided into high immersive or low immersive.

3. Physiological measurement experiment Based on the factors extracted by the factor extraction experiment, the same subject was again subjected to physiological measurement using different images.
<3.1> Experimental conditions The subjects were 14 healthy students (age: 19-22, average age: 21.4, male: 7, female: 7) enrolled in a Japanese university. The experiment was conducted in the same shield room as in 2.1, and one experimenter was present in the same room to switch the video content and check the physiological measurement status. In order to adjust the body surface temperature to room temperature, the experiment was conducted 20 minutes or more after the subject entered the room. The video contents of positive preference and negative preference were set for each subject based on the results of the factor extraction experiment. Video content with an immersive feeling of 4 or more and a preference of 0.4 or more is the most preferred video content (hereinafter abbreviated as "Positive"), and an immersive feeling of 2 or less and a preference of 0.4 or less. The video content with the lowest preference was defined as negative preference (hereinafter abbreviated as "Negative"). Therefore, there are three types of video content to watch: "Positive,""Negative," and "horror."

<3.2> Experimental procedure Similar to the factor extraction experiment, it is composed of FIG. In addition, sensitivity fluctuations, preferences, and immersive feelings were evaluated for each video content by VAS, Multiple mood scale (hereinafter abbreviated as MMS), and immersive feeling survey.

<3.3> Measurement system The arrangement of the measurement system and EEG measurement electrodes is shown in FIG. The subject was in a sitting position and was equipped with a continuous sphygmomanometer, an EEG derivation electrode, and an ECG derivation electrode. A continuous sphygmomanometer (Finometer model2, manufactured by Finapres Medical Systems B.V.) was equipped with a cuff on the second joint of the middle finger of the left arm and recorded on a PC at a sampling frequency of 200 Hz. An infrared thermography device (TVS-200X) was installed at a position 0.7 m in front of the subject at an angle at which the face could be measured. The skin emissivity was ε = 0.98, the temperature resolution was 0.1 ° C or less, and the data was recorded on a PC at a sampling frequency of 1 Hz. EEG was measured by the reference electrode method with the left ear (A1) as a reference. The EEG derivation electrode position was set to 1 point (Pz) based on the international 10-20 law, and the electrode contact resistance was set to 10 kΩ or less.

Generally, α waves are recorded as O1 and O2, but in this study, we focus on α wave power as a constant index of cerebral activity, and since the purpose is exclusively to calculate its attenuation ratio, it is based on laterality and localization. Rather, the Pz in the vicinity was measured with an emphasis on the ease of mounting the electrodes and the reduction of measurement stress on the subject. Electrodes for ECG measurement were attached to the supternal green part (+) and apex (-) according to the NASA induction method in order to minimize the mixing of myoelectric potentials. The ground electrode is the crown (Cz) common to EEG and ECG. The EEG signal and ECG signal were amplified by a digital biological amplifier (5102, manufactured by NF ELECTRONIC INSTRUMENTS) and recorded on a PC via a processor box (5101, manufactured by NF ELECTRONIC INSTRUMENTS) at a sampling frequency of 200 Hz.

<3.4> Evaluation method In this study, we analyzed the correlation between physiological indicators and psychological indicators. Physiological indicators are mean blood pressure (hereinafter abbreviated as MP), heart rate (hereinafter abbreviated as HR), stroke volume (hereinafter abbreviated as SV), and cardiac output (hereinafter abbreviated as SV). , Total peripheral resistance (hereinafter abbreviated as TPR), Nasal skin temperature (hereinafter abbreviated as NST), ECG (Electrocardiograms, hereinafter abbreviated as ECG), and brain waves (hereinafter abbreviated as ECG) Electroencephalograms (hereinafter abbreviated as EEG). The frequency component of EEG from 8 Hz to 13 Hz is called an α wave, and it is known that it is conspicuously expressed at rest, eyes closed, and awake, and is attenuated when any of the conditions is impaired (8). In this study, Fourier transform (FFT) was performed on 1024 sample points every 10 seconds for the EEG time series sampled at 200 Hz, and the α-wave power spectrum was obtained every 10 seconds. Furthermore, the average α-wave power in each of the R1 and R2 sections in FIG. 13 is calculated, and the average α-wave power ratio in the R2 section based on the average α-wave power in the R1 section is used as an index of arousal fluctuation before and after watching TV. did.

NST is known as an index of sympathetic nervous system activity that controls peripheral blood flow. Since the enhancement / suppression of the sympathetic nervous system and the temporal fluctuation of NST correlate well, in this study, the fluctuation of NST every 10 seconds was used as a quantitative index of the sympathetic nervous system activity related to video viewing. A positive value means suppression of sympathetic nervous system activity, and a negative value means increased sympathetic nervous system activity. In each addition of the measured nasal thermal image time series, the spatial average temperature at 10 × 10 pixels of the nasal part was calculated and used as the NST time series. HF is a high-frequency component of the heart rate variability spectrum of 0.15 Hz to 0.4 Hz and is known as a respiratory arrhythmia component (7).

HF is used as an index of parasympathetic nervous system activity, and increases or decreases as parasympathetic nerves increase or decrease. The time series of R wave peak intervals is obtained from the ECG sample by threshold processing, and after third-order spline interpolation, resampling is performed at 20 Hz. FFT processing is performed on the resampled data every 1 s to obtain the time series of the heart rate variability power spectrum. The number of data for FFT processing was 512 points. In the time-series heart rate variability power spectrum, the integral value in the 0.15 Hz to 0.4 Hz band was obtained and used as the HF time series. Hemodynamic parameters (MP, HR, SV, CO, TPR) are known to exhibit characteristic response patterns (Pattern I, Pattern II) in response to external stress, and cardiovascular physiology to stress. It is an important concept in understanding the reaction.

Specifically, pattern I is characterized by increased contractile activity of the myocardium and increased blood volume to skeletal muscle due to vasodilation, which can be said to be an energy-consuming reaction (active coping). On the other hand, pattern II is characterized by the contraction of peripheral blood vessels, and the heart rate is also generally reduced, which can be said to be an energy-saving reaction (passive coping) (7).

The psychological indicators were the multifaceted emotional state scale MMS and VAS. MMS describes depression / anxiety (hereinafter abbreviated as DA), hostility (hereinafter abbreviated as H), malaise (hereinafter abbreviated as F), and temporary mood and emotional states that change depending on the conditions placed on the subject. From active pleasure (hereinafter abbreviated as A), inactive pleasure (hereinafter abbreviated as I), affinity (hereinafter abbreviated as AF), concentration (hereinafter abbreviated as D), startle (hereinafter abbreviated as S) It is indexed by eight types of emotional state scales (9). Before the start of the experiment and at the end of the experiment, four types of subjective sensory quantities, "awakening," "pleasure / discomfort," "fatigue," and "preference," were measured using VAS.

"Pleasure / discomfort" and "awakening" were selected as the basic components in Russell's emotional dualism as items of psychological evaluation in this study (10). In VAS, it is possible to measure the psychophysical quantity of a subject by arranging paired adjectives at both ends of the line segment and the subject checking at an arbitrary position on the line segment. Regarding the words placed at both ends of the scale, the awakening feeling is "very sleepy"-"clearly awake", the pleasure / discomfort is "very unpleasant"-"very comfortable", and the energy is "very tired". "-" Very energetic ", tastes" very dislike "-" very like ". We prepared each VAS on a separate page and taught them to record by hand without recursive reference. Furthermore, after the experiment was completed, the immersive feeling in the image was measured on a 5-point scale (1: I could not immerse at all-5: I was quite immersive). As a statistical analysis method, a paired t-test was used to test the difference between each psychological index before and after video viewing, and Wilcoxon signed rank test was used for the amount of change in the entire TEST section for each physiological index.

<3.5> Results and discussion Psychological indicators will be analyzed statistically to discuss psychological responses during and before and after watching TV video. The preference of all subjects for horror video stimulation was 0.4 or less, so we focus only on the immersive feeling. In a 5-step immersive evaluation after watching horror video, 4 or 5 is Horror (Concentration) (hereinafter abbreviated as "Horror (C)"), and 1 or 2 is Horror (Not concentration) (hereinafter "Horror (N)"). Classified as (abbreviation). Hereafter, the results will be described by the classification of "Positive", "Negative", "Horror (C)", and "Horror (N)". The number of "Horror (C)" is 10, and the number of "Horror (N)" is 4. The average of all subjects on each mood scale of MMS is shown in Figures 16-19.
N = 14 in FIG. 16, N = 14 in FIG. 17, N = 10 in FIG. 18, and N = 4 in FIG. The averages of all subjects in "pleasure / discomfort", "awakening", "vitality", "preference" and "immersion" of immersive feeling are shown in FIGS. 20 to 23. N = 14 in FIG. 20, N = 14 in FIG. 21, N = 10 in FIG. 22, and N = 4 in FIG. 23.

From Fig. 16, "Positive" increased significantly in A and AF, and decreased significantly in F. On the other hand, "Negative" increased significantly in H and F from Fig. 17. This is consistent with the act of viewing video content with positive and negative tastes, respectively. “Horror (C)” shows a significant decrease in A and I in FIG. 18, and it can be inferred that unpleasant emotions were aroused as a whole. In addition, there are more scales with a significant difference in FIG. 18 than in FIG.

From this, it can be inferred that the change in emotions of horror is greater when it is immersive than when it is not immersive. From FIG. 20, “Positive” was significantly increased in pleasure / discomfort, arousal, and vitality. From this, it is considered that the positive preference video content provided the subject with positive comfort accompanied by "exhilaration" and "comfort".

On the contrary, from FIG. 21, since "Negative" is significantly reduced in pleasure / discomfort, arousal, and vitality, the negative preference video content is reluctant to the subject with "depression" or "discomfort". It is thought that it caused physical discomfort. In addition, as shown in FIG. 22, “Horror (C)” was significantly increased in arousal and significantly decreased in pleasure / discomfort and vitality, and thus positive discomfort accompanied by'awakening'and'discomfort'. Is thought to have brought about.

Next, we discuss the physiological response when watching TV images and before and after watching TV images. The time variation of the average between subjects of each physiological index is shown in FIG. In FIG. 24, N = 14. From the top, NST, alpha wave, HF, MP, HR, SV, CO, TPR. The beginning of the TEST interval is 0. The error bars in the figure represent the standard error every 10s. In addition, NST, MP, HR, SV, CO, TPR and immersive feeling are standardized with the baseline of the R1 section set to 0, and α wave and HF are standardized with the baseline of the R1 section set to 1, and the displacement of the entire TEST section from the baseline. The significance probability p of Wilcoxon signed rank test for is shown in FIG. 25 (+: p <0.11, *: p <0.05, **: p <0.01). In the table, P means a positive response to each index, and N means a negative response. In FIG. 25, N = 14.

Looking at the time-series changes for each index from FIG. 24, it is inferred that the sympathetic nerves were enhanced because both “Positive” and “Negative” decreased with the start of the TEST section for NST. No significant change was observed in any of the α waves. From Fig. 25, it can be seen that HF did not show a significant change in "Positive", but decreased significantly in "Negative". Since HF is used as an index of parasympathetic nervous system activity and increases or decreases as the parasympathetic nerve increases or decreases, it is considered that the parasympathetic nerve is suppressed in "Negative". This is consistent with the NST interpretation above.

FIG. 26 shows the temporal variation of the average between subjects of each physiological index of the horror image. In FIG. 26, N = 14. From FIG. 26, it is inferred that the sympathetic nerve was enhanced in "Horror (N)" because the NST was lowered. No significant results were obtained for either alpha wave. It can be seen that HF decreased significantly in "Horror (C)" and increased significantly in "Horror (N)". HF did not agree with the above-mentioned interpretation in NST in "Horror (N)". This is thought to be due to the difference in mechanism. As shown below, "Horror (N)" indicates a passive response characterized by an increase in TPR. As a result, it is considered that the blood flow in the peripheral part decreased and the NST decreased.

Next, "Positive", "Negative", "Horror (C)" and "Horror (N)" are categorized according to the pattern of immersiveness and preference, and the hemodynamic response is compared and examined. As a result, as shown in FIG. 27, it can be summarized by the stress coping style and the axis of preference. In FIG. 27, N = 14.
Both "Positive" and "Horror (C)" are highly immersive, but "Positive" is a positive preference and "Horror (C)" is a negative preference. However, both have increased MP, HR, SV and CO, and decreased TPR. This is a typical pattern I response (active coping) with increased myocardial contractile activity.

In other words, it became clear that they were immersed in TV video content regardless of their tastes when actively coping with them. In addition, by analogy with the results of the factor extraction experiment, it is expected that the subjects who have a high preference for the horror image of the physiological measurement experiment also have a high immersive feeling and are located in the upper right region of FIG. 27. There was no such person. On the other hand, both "Negative" and "Horror (N)" have low immersiveness and taste. However, their physiological responses are different. In other words, in "Negative", TPR and HR decreased, but MP did not change. This is neither active coping nor passive coping, and it is considered that stress coping is not done. In FIG. 27, N = 14.

On the other hand, "Horror (N)" is characterized by a significant increase in TPR and MP, although there is no significant change in HR. This is a typical pattern II response (passive coping) with an increase in MP due to increased peripheral vasoconstriction. From this, it was clarified that when stress coping is not shown, the preference is usually low and the person is not absorbed in the TV video content, but passive coping is shown for horror video. In other words, it was clarified that the viewing mode cannot be determined only by the preference for TV images, and the viewing mode can be classified according to the stress coping style.

Four. Summary In this study, we measured the physiology and psychology of viewing video content with different tastes, and attempted to classify the viewing mode according to the stress coping style that is physiologically valid. Hemodynamic parameters (MP, HR, SV, CO, TPR) are measured as cardiovascular indicators, EEG α-wave power spectrum is measured as central nervous system indicators, and nasal skin temperature and heart rate variability HF components are measured as autonomic nervous system indicators. , Preference for simultaneously measured images, and a statistical analysis of the physiological and psychological state, including a psychological questionnaire, were performed to quantitatively evaluate the physiological and psychological effects when viewing the video content.

As a result, since the classification by psychological index regarding preference in this study was not evaluated at all in the previous study, a new classification for watching TV was obtained by combining stress coping and preference in this study. In conclusion, during active coping, the TV image is "Positive" and "Horror (C)", during passive coping, the TV image is "Horror (N)", when stress coping is not performed, the TV image is It was "Negative". In other words, it was clarified that the viewing mode cannot be discriminated only by the preference for TV images, but the viewing mode can be discriminated by the stress response by the hemodynamic parameters.

<References>
(1) Akihiro Hirata, Emi Moroto, Hiroshi Aramaki: "Current TV Viewing and Media Use (2) -From the" Japanese and TV 2010 "Survey-", Broadcasting Research and Survey, pp.2-27 (2010)
(2) Hiroshi Aramaki, Tomoko Masuda, Sachiko Nakano: "How TV faces people in their twenties", Broadcasting Research and Survey, pp.2-21 (2008)
(3) Y. Siki, Y. Murakami, and Y. Huzita: “Television viewing and simultaneous use of new media: anethnographic study of young people in Japan”, Institute for Media and Communication Research Keio University, Vol.59, No.1 , pp.131-140 (2009) (inJapanese) Yuko Shiki, Akira Murayama, Yuko Fujita: "Young people watching TV and media parallel use behavior &#8211; From Audience Ethnography Survey of University Students", Keio University Media Bulletin of Communication Research Institute, Vol.59, No.1, pp.131-140 (2009)
(4) S. Nomura, Y. Kurosawa, N. Ogawa, CM Althaff Irfan, K. Yajima, S. Handri, T. Yamagishi, KTNakahira, and Y. Fukumura: “Psysiological Evaluation of a student in E-learning Sessions by Hemodynamic Response ”, IEEJ Trans. EIS, Vol.131, No.1, pp.146-151 (2011) (in Japanese) Nomura Shusaku, Irfan CM Althaff, Takao Yamagishi, Yoshimasa Kurosawa, Kuniaki Yajima, Katsuko Nakahira・ Nobuyuki Ogawa ・ HANDRI Santoso ・ Yoshimi Fukumura: "Physiological evaluation study of e-learning students by hemodynamic parameters", Denkiron C, Vol.131, No.1, pp.146-151 (2011)
(5) T. Watanuki and A. Nozawa: “Visualization of a feeling of concentration for TV contents”, Bulletin of Institute of Electrical and Electronic Engineers of measurement, Vol.IM-12, No.63, pp.19-25 ( 2012) (in Japanese) Takuya Watanuki and Akio Nozawa: "Visualization of Immersiveness in TV Content", Institute of Electrical and Electronic Engineers, Vol.IM-12, No.63, pp.19-25 (2012)
(6) K. Mori, C. Ono, Y. Motomura, H. Asoh, and A. Sakurai: “Empirical Analysis of User Preference Models for Movie Recommendation”, IEIC Technical Report.NC.neurocomputing, Vol.104, No.759 , pp.77-82 (2005) (in Japanese) Shigeru Kurokawa, Tomohiro Ono, Yoichi Motomura, Hideki Asoh, Akito Sakurai: "Experimental Evaluation of User Preference Model for Recommendation of Movie Content", IEICE Technical Report .. NC. Neurocomputing, Vol.104, No.759, pp.77-82 (2005)
(7) Yuki Sawada Exhibition: "Physiological Psychology: New Physiological Psychology Volume I (Kiyoshi Fujisawa, Shoji Kakinoki, Katsuo Yamazaki)", Kitaooji Shobo, Chapter 10, p.187 (1998)
(8) JA Russell: “A circumplex model of affect”, J. Personality and Social Psychology, Vol.39, pp.1161-1178 (1980)

<Experimental example 3>
1. 1. Purpose of the experiment In recent years, digital and multi-channel television broadcasting has been increasing. However, with the spread of miniaturized and high-performance information and communication equipment due to the progress of IT technology and the development of the information network environment, the departure from television as entertainment is accelerating, and the reasons for watching television are diversifying. Hirata et al. Cited reasons for watching TV, such as "to know the events and movements of the world,""to rest and relax," and "to deepen and spread relationships with people." (1). Moreover, not only the reason for watching TV, but also the viewing mode has changed to a less involved view, such as vague viewing attitudes and weakening of viewing habits, as people move away from TV (2). Shiki et al. Watched "dedicated viewing" to watch TV intensively, "watching while watching" in parallel with other activities of daily living such as housework, eating and drinking, studying, and "vaguely watching various programs while zapping". It lists viewing modes such as "viewing" (3). In this way, it has been clarified that the viewing mode is various and changes depending on factors such as time, place, and mood.

However, most of the studies that classify these viewing modes use psychological responses such as opinion polls and introspection evaluations after the fact, and there are no studies that classify viewing patterns using physiological responses. Since television video and audio are audiovisual stimuli for living organisms, they are considered stressors in their own right. The stressor encourages the living body to cope with stress and brings about physiological and psychological changes as a stress response.
If various contents of television are regarded as stressors, it is expected that the physiological stress response will differ depending on the coping with the contents, that is, the viewing mode. Nomura et al. Classify how to deal with e-learning content by cardiovascular index (4).

Therefore, we have tried to classify viewing modes and preferences for TV video content using cardiovascular indicators (5). In addition to classification, research is also being conducted to build an estimation model.
Kurokawa et al. Constructed a user preference model for video content using naive Bayes, decision trees, Bayesian networks, and neural networks, and extracted prediction accuracy and important variables for content evaluation (6). However, this study is also a model that uses only psychological responses, and there is no study that constructs an estimation model of preference and viewing mode using physiological responses.

The results of previous studies suggest that stress coping patterns based on hemodynamic parameters and heart rate may be able to classify the physiological and psychological states of TV viewers, such as preference and immersion in content. Therefore, the purpose of this study is to experimentally examine the estimation of the physiological and psychological state of the viewer regarding the above-mentioned television viewing by the cardiovascular index. Feature vectors were extracted from the cardiovascular index, and estimation models for TV video content preference, viewing mode, and excitement-sedation were created and evaluated.

2. Experiment
Cardiovascular measurement was performed with a continuous sphygmomanometer when watching TV video content. After that, by performing pattern recognition using a hierarchical neural network based on the hemodynamic parameters, we estimated the preference, viewing mode, and excitement-sedation when watching TV video content.

<2.1> Experimental procedure
As shown in FIG. 28, this experiment consists of 5 minutes of video viewing and 1 minute of rest / eye closure R1 and R2 before and after that. From the Visual Analogue Scale (hereinafter abbreviated as VAS) before and after watching the video and the immersive feeling survey after watching the video, the preference and immersive feeling for each video content were evaluated. In addition, the subjective excitement-sedation fluctuations felt by the subject were recorded in real time during video viewing.
<2.2> Experimental conditions The subjects were 14 healthy students (age: 19-22, average age: 21.4, male: 7, female: 7) enrolled in a Japanese university. The subjects were fully informed in advance of the content, purpose, and subject of the experiment in oral and written form, and their consent to the cooperation of the experiment was confirmed by signature. The measurement was performed in a convection-free shielded room at room temperature of 26.0 ± 1.6 ° C, and one experimenter was present in the same room to switch video content and check the physiological measurement status. In order to adjust the body surface temperature to room temperature, the experiment was conducted 20 minutes or more after the subject entered the room.

The subject was sitting, using an LCD TV (55 inches, HX850, BRAVIA, made by SONY) installed 2 m in front of him, and had positive taste and highly immersive video content (hereinafter abbreviated as "Positive"), negative. Watch three types of video content: low-preference / low-immersive video content (hereinafter abbreviated as "Negative"), and horror video that shows unique results that there are large individual differences in low-preference but immersive feeling. The video content was played on a DVD player (SCPH-3900, made by SONY).

<2.3> Measurement system The measurement system is shown in FIG. The subject was sitting and wearing a continuous sphygmomanometer. A continuous sphygmomanometer (Finometer model2, manufactured by Finapres Medical Systems B.V.) was equipped with a cuff on the second joint of the middle finger of the left arm and recorded on a PC at a sampling frequency of 200 Hz. In addition, a keyboard (K270, made by Logitech) is installed in front of you, and software is used to input the subjective excitement-sedation fluctuations that the subject feels in real time from the up and down keys of the keyboard, and it is sequential and relative. Excitement-sedation was recorded.

<2.4> Evaluation method
Physiological indicators are mean blood pressure Mean pressure (hereinafter abbreviated as MP), heart rate (hereinafter abbreviated as HR), stroke volume (hereinafter abbreviated as SV), cardiac output (hereinafter abbreviated as Cardiac output). CO) and total peripheral resistance (hereinafter abbreviated as TPR). Hemodynamic parameters (MP, HR, SV, CO, TPR) are known to exhibit characteristic response patterns (Pattern I, Pattern II) in response to external stress, and cardiovascular physiology to stress. It is an important concept in understanding the reaction. Specifically, pattern I is characterized by increased contractile activity of the myocardium and increased blood volume to skeletal muscle due to vasodilation, which can be said to be an energy-consuming reaction (active coping). On the other hand, pattern II is characterized by the contraction of peripheral blood vessels, and the heart rate is also generally reduced, which can be said to be an energy-saving reaction (passive coping) (7). MP, HR, SV, CO, and TPR were standardized with the baseline of the R1 section set to 0.

In addition, as psychological indicators, four types of subjective sensory quantities, "awakening," "pleasure / discomfort," "vitality," and "preference," were measured using VAS before and at the end of the experiment. "Pleasure / discomfort" and "awakening" were selected as the basic components in Russell's emotional dualism as items of psychological evaluation in this study (8). In VAS, it is possible to measure the psychophysical quantity of a subject by arranging paired adjectives at both ends of the line segment and the subject checking at an arbitrary position on the line segment. Regarding the words placed at both ends of the scale, the awakening feeling is "very sleepy"-"clearly awake", the pleasure / discomfort is "very unpleasant"-"very comfortable", and the vitality is "very tired". -"Very energetic", taste "Very disliked"-"Very liked".

We prepared each VAS on a separate page and taught them to record by hand without recursive reference. Furthermore, after the experiment was completed, the immersive feeling in the image was measured on a 5-point scale (1: I could not immerse at all-5: I was quite immersive). A paired t-test was used to test the difference between each psychological index before and after viewing the video, and Wilcoxon signed rank test was used to test the amount of change in the entire TEST section for each physiological index. Excitement-sedation was normalized based on the maximum value of integration.

Pattern recognition by a hierarchical neural network was used to create a preference and viewing mode classification model and derive the discrimination rate, and to create an excitement-sedation estimation model and derive estimated values for each subject.
The structure of the hierarchical neural network is shown in FIG. The learning rule is the error back propagation method, and the output function is the sigmoid function. There are three layers: input layer, intermediate layer, and output layer. Cross-validation was used to estimate the accuracy of the model. Further, as shown in FIG. 31, the discrimination time td is assumed for the time-dependent fluctuation of each index. td is 300s while watching the video. In each model, we focus on the slope per Δti in each feature time series in the td interval, and let the feature vector S be a series of slopes at 10s intervals. The number of elements of S is 30. In line with this, in the neural network of this paper, the number of elements in the input layer is 30 and the number of elements in the intermediate layer is 16.

<2 ・ 4 ・ 1> Preference classification model Heart rate was used as a feature, and 10s, 20s, 30s, 40s, 50s, and 60s were examined as Δti. The total learning data had 28 patterns including the preferences (Positive, Negative) of 14 subjects, of which 27 patterns were used as learning data and the remaining 1 pattern was used as unknown data to evaluate the accuracy of the preference classification model. After training the feature vector S at the time of "Positive" and "Negative", the accuracy of the preference classification model was evaluated by inputting unknown data. The output layer consists of two elements, "Positive" and "Negative".
<2 ・ 4 ・ 2> The hemodynamic parameters were used as the features of the viewing mode classification model, and 10s, 20s, 30s, 40s, 50s, and 60s were examined as Δti. There are 42 patterns of all learning data, including each stress coping style (active coping, passive coping, no stress coping) of 14 subjects, of which 41 patterns are learning data and the remaining 1 pattern is unknown data, which is a preference classification model. The accuracy of was evaluated. The output layer consists of three elements: active coping, passive coping, and no stress coping.

<2 ・ 4 ・ 3> Excitement-sedation The hemodynamic parameters were used as the estimated model features, and Δti was set to 30 s. All learning data had 28 patterns including excitement-sedation values for each preference (Positive, Negative) of 14 subjects, of which 27 patterns were used as learning data and the remaining 1 pattern was used as unknown data to evaluate the accuracy of the preference classification model. .. The output layer is one element of excitement-sedation value.
<2.5> Results and discussion In VAS, the difference between the average of all subjects before and after video viewing on each mood scale is shown in Fig. 32 (+: p <0.1, *: p <0.05, **: p <0.01). ). In FIG. 32, N = 14. From FIG. 32, all the subjects had a low preference for horror video stimulation. Therefore, in the five-level immersive evaluation after watching the horror video, if it is 4 or 5, it is Horror (Concentration) (hereinafter abbreviated as "Horror (C)"), and if it is 1 or 2, it is Horror (No-concentration) (hereinafter "Horror (" It was classified as "N)" and abbreviated as). Subsequent results are classified into "Positive", "Negative", "Horror (C)", and "Horror (N)". The number of "Horror (C)" is 10, and the number of "Horror (N)" is 4.

The significance probability p of Wilcoxon signed rank test for the displacement of MP, HR, SV, CO, and TPR from the baseline in the entire TEST interval is shown in FIG. 33 (+: p <0.11, *: p <0. 05, **: p <0.01). In the table, P means a positive response to each index, and N means a negative response. In FIG. 33, N = 14.
From FIG. 33, the viewing mode when viewing TV images is classified. Both "Positive" and "Horror (C)" are highly immersive, but "Positive" is a positive preference and "Horror (C)" is a negative preference.
However, both have increased MP, HR, SV and CO, and decreased TPR. This is a typical pattern I response (active coping) with an increase in myocardial contractile activity. In other words, it became clear that they were immersed in TV video content regardless of their tastes when actively coping with them.

On the other hand, both "Negative" and "Horror (N)" have low immersiveness and taste. However, their physiological responses are different. In other words, in "Negative", TPR and HR decreased, but MP did not change. This is neither active coping nor passive coping, and it is considered that stress coping is not done. On the other hand, "Horror (N)" is characterized by a significant increase in TPR and MP, although there is no significant change in HR. This is a typical pattern II response (passive coping) with an increase in MP due to increased peripheral vasoconstriction.

From this, it was clarified that when stress coping is not shown, the preference is usually low and the person is not absorbed in the TV video content, but passive coping is shown for horror video. In other words, it was clarified that the viewing mode cannot be determined only by the preference for TV images, and the viewing mode can be classified according to the stress coping style. In addition, in FIG. 33, HR increased significantly in "Positive" and decreased significantly in "Negative".

Next, FIGS. 34 and 35 show the results of the discrimination rate when the preference and the viewing mode are discriminated from the estimation model constructed by using the neural network. In FIGS. 34 and 35, when Δti50s, the positive discrimination rate of preference was 83.3%, and the positive discrimination rate of the viewing mode was 75%, which were higher than those of other Δti.

In addition, FIG. 36 shows measured excitement-sedation and predicted excitement-sedation when excitement-sedation was estimated.
FIG. 36 shows the results of subject A with the smallest error between measured excitement-sedation and predicted excitement-sedation. In "Positive" on FIG. 36, the predicted excitement-sedation increased along with the measured excitement-sedation.

Moreover, in "Negative" at the bottom of FIG. 36, the predicted excitement-sedation decreased as well as the measured excitement-sedation. Next, FIG. 37 shows the average error of each subject when the absolute value of the difference between the measured excitement-sedation and the predicted excitement sedation is taken as the error. In FIG. 37, the average error of “, Positive” was 0.10 to 0.37, the average error of “Negative” was 0.11 to 0.29, and the average of the average errors among all subjects was 0.17. In other words, predicted excitement-sedation was estimated with an error of about 17% on average. In addition, FIG. 38 shows a series of actually measured excitement-sedation and predicted excitement-sedation relationships taken at intervals of 10 s for all subjects. In FIG. 38, the correlation coefficient was 0.89, which was a high correlation. Further, from FIG. 38, it can be seen that the distributions of actually measured excitement-sedation and predicted excitement-sedation are almost uniform regardless of the magnitude of the values.

3. Summary In this study, for the purpose of estimating the viewing mode, preference, excitement-sedation for TV video content using a neural network, a feature vector was extracted from the cardiovascular index, and the preference when watching TV video content, We created and evaluated an estimated model of viewing mode and excitement-sedation.

As a result, positive preference / high immersive "Positive" and negative preference / high immersive "Horror (C)" are active coping, and unfavorable / highly immersive "Horror (N)" are passive coping, negative. It was possible to classify that stress was not dealt with in "Negative", which has a taste and low immersiveness. Furthermore, the heart rates of ", Positive" and "Negative" were significantly different, and the results were obtained according to the general characteristics of the heart rate response. In addition, when the preference and viewing mode were discriminated using the estimation model, the positive discrimination rate was the highest, with the preference being 83.3% and the viewing mode being 75%.

As a result of estimating excitement-sedation for each subject, the average error of measured excitement-sedation and predicted excitement-sedation was 10 to 37% for "Positive", 11 to 29% for "Negative", and the average among all subjects was 17 It became%. Furthermore, measured excitement-sedation and predicted excitement-sedation showed a strong positive correlation with a correlation coefficient of 0.89. From the above, it was suggested that the hemodynamic parameters and heart rate could be used to estimate the preference, viewing mode, and excitement-sedation when watching TV video content. In the future, we plan to further improve accuracy by comparing and examining other physiological indicators and other classifiers.

<References>
(1) Toru Nishigaki: "Will TV Revive on the Internet?", Monthly Commercial Broadcasting (2001)
(2) Isao Fujiwara and Yumiko Saito: "Opinion Poll Report Evaluation and Expectations for Television-From the Results of the" Role of Television "Survey-", NHK Broadcasting Research and Survey June Issue, pp.2-15 (1989)
(3) Hiroaki Ohnogi: "Chapter 9 TV viewing attitudes of female university students and a few factors: From a preliminary survey on viewers' personality characteristics, interest in program content, and the purpose of watching TV (program analysis and viewing) Study of Learning Behavior: Aiming to Develop Taxonomy for Broadcast Education Programs) ”, Research Report, Vol.18, pp.153-172 (1990)
(4) Y. Takahashi and S. John: “Recommendation Models of TelevisionProgram Genre, Based on Survey and Analysis of Behavior in WatchingTelevision: Toward Human Content Interface Design (2)”, Bulletin of Japanese Society for Science of Design, Vol.46, No .133, pp.71-80 (1999) (in japanese)
Yasushi Takahashi and Shakulton John: "Program genre recommendation model based on research and analysis of TV viewing behavior: Toward Human Content Interface Design (2)", Design Studies, Vol.46, No.133, pp.71- 80 (1999)
(5) Yumiko Tomomune and Yumiko Hara: "TV as a" Time Comforter "-Relationship between Viewing Attitude and Total Variety of Programs-", Broadcasting Research and Survey, Vol.51, pp.2-17 (2001)
(6) S. Nomura, Y. Kurosawa, N. Ogawa, CM Althaff Irfan, K. Yajima, S. Handri, T. Yamagishi, KT Nakahira, and Y. Fukumura: “Psysiological Evaluation of a student in E-learning Sessions by Hemodynamic Response ”, IEEJ Trans. EIS, Vol.131, No.1, pp.146-151 (2011)
Osamu Nomura, Irfan CM Althaff, Takao Yamagishi, Yoshimasa Kurosawa, Kuniaki Yajima, Katsuko Nakahira, Nobuyuki Ogawa, HANDRI Santoso, Yoshimi Fukumura: "Physiological evaluation study of e-learning participants by hemodynamic parameters" C, Vol.131, No.1, pp.146-151 (2011)
(7) Yuki Sawada Exhibition: "Hematologic Mechanical Response; New Physiological Psychology Volume I (Kiyoshi Fujisawa, Shoji Kakinoki, Katsuo Yamazaki)", Kitaooji Shobo, Chapter 10, pp.187 (1998)
(8) A. Michimori: “Relationship between the alpha attenuation test, subjective sleepiness and performance test”, 10th Symposium on HumanInterface, Vol.10, No.1413, pp.233-236 (1992)
(9) M. Terasaki, Y. Kishimoto, and A. Koga: “Construction of a multiple mood scale”, The japanese journal of psychology, Vol.62, No.6, pp.350-356 (1992) Masaharu Terasaki and Kishimoto Yoichi and Misato Koga: "Creation of a Multifaceted Emotional State Scale", Journal of Psychology Research, Vol.62, No.6, pp.350-356 (1992)
(10) JA Russell: “A circumplex model of affect”, J. Personality and SocialPsychology, Vol.39, pp.1161-1178 (1980)

Since the stress coping style determination system of the present invention can qualitatively analyze the stress of a subject in a non-contact state, it is a means for grasping the stress state of a worker working in a factory or the like, while driving a car. It can be used in a wide range of technical fields, such as a means for grasping the stress state of a driver in a car, a means for grasping the stress state of a student in class, and the like.

100 Stress coping style judgment system 110 Biometric information acquisition device (Biological information acquisition unit)
120 Judgment device (judgment unit)
121 Judgment feature amount storage unit 122 Specific site reaction detection unit 123 Response pattern judgment unit 130 Learning device (machine learning unit)
131 Learning data storage unit 132 Feature amount extraction unit 133 Feature amount learning unit 134 Learned model P Subject IF Facial image S1 Judgment feature amount storage process (judgment feature amount storage step)
S2 Specific site reaction detection process (Specific site reaction detection step)
S3 Response pattern determination process (response pattern determination step)
S11 Learning data storage process (learning data storage step)
S12 Feature extraction process (feature extraction step)
S13 Feature learning process (feature learning step)
S21 Clustering process (clustering step)
S22 Image extraction process (image extraction step)
S23 Edge extraction process (edge extraction step)
S24 Fractal analysis process (fractal analysis step)

Claims (15)

The biological information acquisition unit that acquires the biological information of the subject in a non-contact state,
It has a determination unit that determines a stress coping mode of a subject based on the biological information and a response pattern specified in advance.
The response pattern is a stress coping mode determination system specified by hemodynamic parameters .
The hemodynamic parameters include a plurality of parameters of mean blood pressure, heart rate, cardiac output, stroke volume and total peripheral resistance.
The biological information is a facial image and is
The response patterns include three types of patterns, "active coping", "passive coping", and "no coping".
The determination unit stores a spatial feature amount corresponding to "active coping", a spatial feature amount corresponding to "passive coping", and a spatial feature amount corresponding to "no coping". It has a quantity storage unit and
Based on the biological information and each spatial feature amount stored in the determination feature amount storage unit, the stress coping mode is one of "active coping", "passive coping" and "no coping". Determine which pattern the style shows,
The feature amount stored in the determination feature amount storage unit is a feature amount extracted by the machine learning unit.
The machine learning unit
A learning data storage unit that stores a plurality of learning facial images labeled corresponding to "active coping", "passive coping", and "no coping".
A feature amount extraction unit that extracts the spatial feature amount of the face image from the learning face image using a trained model, and a feature amount extraction unit.
Based on the relationship between the extraction result by the feature amount extraction unit and the label attached to the learning facial image as the extraction target, the extraction accuracy of the spatial feature amount by the feature amount extraction unit is increased. A stress coping style determination system characterized by having a feature amount learning unit that changes the network parameters of the trained model as described above. The stress coping style determination system according to claim 1, wherein the facial image is a facial thermal image or a facial visible image. The stress coping style determination system according to claim 1 or 2, wherein the determination unit determines the stress coping mode of the subject by observing the stress response of a specific portion of the face included in the facial image. The stress coping style determination system according to any one of claims 1 to 3, wherein the spatial feature amount is a fractal dimension calculated based on a facial image of a subject. A judgment feature storage step that stores spatial features corresponding to "active coping", spatial features corresponding to "passive coping", and spatial features corresponding to "no coping", and
Based on the facial image of the subject and each spatial feature memorized by the determination feature storage step, the stress coping styles of the subject are "active coping", "passive coping" and "no coping". It has a determination step for determining which of the following response patterns is exhibited.
The response pattern is identified by hemodynamic parameters and
A learning data storage step that stores multiple learning facial images labeled for each of "active coping", "passive coping", and "no coping".
A feature amount extraction step for extracting the spatial feature amount of the learning facial image using the trained model, and a feature amount extraction step.
Based on the relationship between the extraction result by the feature amount extraction step and the label attached to the learning facial image as the extraction target, the extraction accuracy of the spatial feature amount by the feature amount extraction step is improved. With a learning step, such as changing the network parameters of the trained model,
The determination feature amount storage step is a step for storing the spatial feature amount extracted by the feature amount extraction step, and causes a computer to function as a means for determining a stress coping mode of a subject. Program . The program according to claim 5, wherein the spatial feature amount is a fractal dimension calculated based on a facial image of a subject . It is a stress coping mode determination method by a system having a biological information acquisition unit and a determination unit.
The biological information acquisition step of acquiring the biological information of the subject in a non-contact state by the biological information acquisition unit, and
It has a determination step of determining a stress coping mode of a subject based on the biological information and a response pattern specified in advance by the determination unit.
The response pattern is identified by hemodynamic parameters and
The hemodynamic parameters include a plurality of parameters of mean blood pressure, heart rate, cardiac output, stroke volume and total peripheral resistance.
The biological information is a facial image and is
The response patterns include three types of patterns, "active coping", "passive coping", and "no coping".
In the determination step,
A judgment feature storage step that stores spatial features corresponding to "active coping", spatial features corresponding to "passive coping", and spatial features corresponding to "no coping", and
Based on the biological information and each spatial feature amount memorized by the determination feature amount storage step, the stress coping mode is any of "active coping", "passive coping", and "no coping". A response pattern determination step, which determines whether the format indicates a pattern, is included.
The feature amount memorized by the determination feature amount storage step is a feature amount extracted by the machine learning unit.
The machine learning unit
A learning data storage step that stores multiple learning facial images labeled for each of "active coping", "passive coping", and "no coping".
A feature amount extraction step for extracting the spatial feature amount of the face image from the learning face image using a trained model, and a feature amount extraction step.
Based on the relationship between the extraction result by the feature amount extraction step and the label attached to the learning facial image as the extraction target, the extraction accuracy of the spatial feature amount by the feature amount extraction step is improved. A feature amount learning step for changing the network parameters of the trained model, and a stress coping style determination method, which comprises executing the feature amount learning step . A learning data storage unit that stores a plurality of learning facial images labeled according to a response pattern specified by a hemodynamic parameter, and a learning data storage unit.
A feature amount extraction unit that extracts the spatial feature amount of the subject's face image from the learning face image using the trained model, and a feature amount extraction unit.
Based on the relationship between the extraction result by the feature amount extraction unit and the label attached to the learning facial image as the extraction target, the extraction accuracy of the spatial feature amount by the feature amount extraction unit is improved. A learning device having a feature amount learning unit that changes the network parameters of the trained model as described above . The learning device according to claim 8, wherein the spatial feature amount is a fractal dimension calculated based on a facial image of a subject. A learning data storage step that stores multiple learning facial images labeled according to the response pattern identified by the hemodynamic parameters.
A feature amount extraction step for extracting the spatial feature amount of the subject's face image from the learning face image using the trained model, and a feature amount extraction step.
Based on the relationship between the extraction result by the feature amount extraction step and the label attached to the learning facial image as the extraction target, the extraction accuracy of the spatial feature amount by the feature amount extraction step is improved. A learning method characterized by having a feature amount learning step for changing the network parameters of the trained model as described above . The learning method according to claim 10, wherein the spatial feature amount is a fractal dimension calculated based on a facial image of a subject . A learning data storage step that stores multiple learning facial images labeled according to the response pattern identified by the hemodynamic parameters.
A feature amount extraction step for extracting the spatial feature amount of the subject's face image from the learning face image using the trained model, and a feature amount extraction step.
Based on the relationship between the extraction result by the feature amount extraction step and the label attached to the learning facial image as the extraction target, the extraction accuracy of the spatial feature amount by the feature amount extraction step is improved. A program that causes a computer to function as a means for learning the spatial features of a facial image of a subject, which comprises a feature learning step for changing the network parameters of the trained model . The program according to claim 12, wherein the spatial feature amount is a fractal dimension calculated based on a facial image of a subject . Learned generated by machine learning the spatial features of a subject's facial image using multiple learning facial images labeled for response patterns identified by hemodynamic parameters as teacher data. A model
A learning data storage step for storing the plurality of learning facial images, and
A feature amount extraction step for extracting the spatial feature amount of the subject's face image from the learning face image using the learned model, and a feature amount extraction step.
Based on the relationship between the extraction result by the feature amount extraction step and the label attached to the learning facial image as the extraction target, the extraction accuracy of the spatial feature amount by the feature amount extraction step is improved. A trained model characterized in that it is generated by repeating the feature quantity learning step of changing the network parameters of the trained model as described above. The trained model according to claim 14, wherein the spatial feature amount is a fractal dimension calculated based on a facial image of a subject. "

Images