JP7573206B1 - Information processing method, program, and communication device - Google Patents

Abstract

The information processing method performed by the communication device includes acquiring a first biosignal from a first earpiece equipped with a biosensor, acquiring a second biosignal from a second earpiece equipped with a biosensor, dividing the first biosignal and the second biosignal at predetermined time intervals, determining a similarity between first feature data based on the first biosignal for a predetermined divided section and second feature data based on the second biosignal for a predetermined time shifted in time with respect to the predetermined section, and performing a synchronization process to synchronize the first biosignal and the second biosignal based on the similarity.

Description

The present invention relates to an information processing method, a program, and a communication device.

Conventionally, wireless earphones equipped with biosensors are known. For example, Patent Literature 1 discloses a technology in which wireless earphones worn on the left and right ears transmit biosignals acquired by the sensors to a cradle.

JP 2019-195179 A

Here, in the technology described in Patent Document 1, when the wireless earphones transmit biosignals to the cradle, the cradle processes and displays the left and right biosignals separately. However, in the conventional technology, this is not a problem because the left and right biosignals are processed separately. However, when considering wirelessly transmitting biosignals from the left and right wireless earphones, there is a risk that an error will occur in the timing at which the receiving communication device receives the left and right biosignals due to communication delays, etc. In this case, there is a problem in that when processing data based on the left and right biosignals, it is not possible to process them appropriately.

The disclosed technology has been developed in consideration of these circumstances, and aims to provide technology that enables a communication terminal to properly synchronize left and right biological signals received from wireless earphones.

An information processing method that is one aspect of the disclosed technology is an information processing method executed by a communication device, and includes acquiring a first biosignal from a first earpiece equipped with a biosensor, acquiring a second biosignal from a second earpiece equipped with a biosensor, dividing the first biosignal and the second biosignal at predetermined time intervals, determining a similarity between first feature data based on the first biosignal of the divided predetermined section and second feature data based on the second biosignal of a predetermined time shifted in time with respect to the first feature data of the predetermined section, and performing a synchronization process to synchronize the first biosignal and the second biosignal based on the similarity.

A program according to another aspect of the present disclosure includes a first acquisition step of acquiring a first biosignal from a first earpiece having a biosensor, a second acquisition step of acquiring a second biosignal from a second earpiece having a biosensor, a division step of dividing the first biosignal and the second biosignal at predetermined time intervals, a determination step of determining a similarity between first feature data based on the first biosignal of the divided predetermined interval and second feature data based on the second biosignal of a predetermined interval shifted in time with reference to the first feature data of the predetermined interval, and a synchronization step of synchronizing the first biosignal and the second biosignal based on the similarity.

A communication device according to another aspect of the present disclosure includes a first acquisition unit that acquires a first biosignal from a first earpiece having a biosensor, a second acquisition unit that acquires a second biosignal from a second earpiece having a biosensor, a division unit that divides the first biosignal and the second biosignal at predetermined time intervals, a determination unit that determines a similarity between first feature data based on the first biosignal of a predetermined divided section and second feature data based on the second biosignal of a predetermined time shifted in time based on the first feature data of the predetermined section, and a synchronization unit that synchronizes the first biosignal and the second biosignal based on the similarity.

According to the present invention, the communication terminal can properly synchronize the left and right biological signals received from the wireless earphones.

FIG. 1 is a diagram for explaining an overview of the present embodiment. FIG. 2 is a block diagram showing the configuration of an earpiece according to the present embodiment. FIG. 2 is a diagram showing the external shape of an earpiece according to the present embodiment. 4 is a cross-sectional view showing the IV-IV cross section of the earpiece according to the present embodiment. FIG. 2 is a block diagram showing a configuration of a communication terminal according to the present embodiment. FIG. FIG. 13 is a diagram showing fluctuations in α waves of subject A. FIG. 2 shows the power spectral density for a given 4 second period when the subject had both eyes closed. FIG. 7 is a diagram showing a matrix of correlation coefficients between EP-L, EP-R, and EP-Diff shown in FIG. 6 and PM-T7, PM-Cz, and PM-T8. 11A and 11B are diagrams illustrating examples of a first biological signal and a second biological signal before synchronization processing by a synchronization unit. 13 is a diagram showing an example in which a determination unit determines a similarity between a first envelope and a second envelope; FIG. 11 is a diagram illustrating an example of a synchronization process performed by a synchronization unit. 11 is a flowchart showing an example of processing of a communication terminal M according to the embodiment. 5 is a flowchart relating to a process of a feature data extraction unit according to the present embodiment. FIG. 2 is a diagram illustrating a hardware configuration according to the present embodiment.

Hereinafter, the present embodiment will be described with reference to the attached drawings. In order to facilitate understanding of the description, the same components in each drawing are denoted by the same reference numerals as much as possible, and duplicate descriptions will be omitted.

An overview of this embodiment will be described with reference to FIG. 1. The earphone 2 according to this embodiment comprises a first earpiece 2R and a second earpiece 2L. The first earpiece 2R is worn on one ear of the user H. The second earpiece 2L is worn on the other ear of the user H. The first earpiece 2R and the second earpiece 2L are configured to be capable of communicating with a smartphone. The smartphone is an example of a communication terminal M.

<Earpiece configuration>
Next, components of the earphone 2 according to this embodiment will be described with reference to Fig. 2. The earphone 2 includes a first earpiece 2R and a second earpiece 2L. The first earpiece 2R includes, as components, a main sensor 271 (first sensor), a reference sensor 272 (third sensor), a ground sensor 273 (fifth sensor), a first A/D conversion unit 274, and a first transmission unit 275.

The main sensor 271 is provided at a position where the first biometric information of the user H can be acquired as an electrical signal. The position of the main sensor 271 will be described later. The main sensor 271 outputs the sensed first biometric information to the first A/D conversion unit 274.

The reference sensor 272 is provided at a position where the third biometric information of the user H can be acquired as an electrical signal. The position of the reference sensor 272 will be described later. The reference sensor 272 outputs the sensed third biometric information to the first A/D conversion unit 274.

The first biometric information and the third biometric information are, for example, information relating to the brain waves of user H. Generally, brain waves include a number of types of frequency bands, such as alpha waves (8-13 Hz), beta waves (14-25 Hz), theta waves (4-7 Hz), delta waves (1-3 Hz), and gamma waves (30 Hz or higher). Therefore, the first biometric information and the third biometric information may include brain waves of a number of frequency bands.

The ground sensor 273 is a sensor that acquires ground potential information as an electrical signal. The position of the ground sensor 273 will be described later. The ground sensor 273 outputs the sensed ground potential information to the first A/D conversion unit 274.

The first A/D conversion unit 274 converts each piece of information from analog information to digital information. The first A/D conversion unit 274 also outputs each piece of converted information to the first transmission unit 275.

The first transmission unit 275 transmits the first bioinformation output from the main sensor 271 and the third bioinformation output from the reference sensor 272, acquired from the first A/D conversion unit 274, to the communication terminal M. The first transmission unit 275 may also transmit the ground potential information output from the ground sensor 273, acquired from the first A/D conversion unit 274, to the communication terminal M. The first transmission unit 275 may generate first difference information, which is the difference between the first bioinformation output from the main sensor 271 and the ground potential information output from the ground sensor 273, acquired from the first A/D conversion unit 274, and transmit it to the communication terminal M. Similarly, the first transmission unit 275 may generate third difference information, which is the difference between the third bioinformation output from the reference sensor 272 and the ground potential information output from the ground sensor 273, acquired from the first A/D conversion unit 274, and transmit it to the communication terminal M.

In the above example, the first transmission unit 275 transmits the first biometric information or the first differential information to the communication terminal M over the first channel. The first transmission unit 275 also transmits the third biometric information or the third differential information to the communication terminal M over the third channel. The third channel may be the same channel as the first channel or a different channel. Each of the above-mentioned biometric information or ground potential information includes information sampled by the first A/D conversion unit 274.

The second earpiece 2L has as its components a main sensor 281 (second sensor), a reference sensor 282 (fourth sensor), a ground sensor 283 (sixth sensor), a second A/D conversion unit 284 and a second transmission unit 285.

The main sensor 281 is provided at a position where the second biometric information of the user H can be acquired as an electrical signal. The position of the main sensor 281 will be described later. The main sensor 281 outputs the sensed second biometric information to the second A/D conversion unit 284.

The reference sensor 282 is provided at a position where the fourth biometric information of the user H can be acquired as an electrical signal. The position of the reference sensor 282 will be described later. The reference sensor 282 outputs the sensed fourth biometric information to the second A/D conversion unit 284.

The second biometric information and the fourth biometric information are, for example, information regarding the brain waves of user H. Like the first biometric information and the third biometric information, the second biometric information and the fourth biometric information may include brain waves in multiple frequency bands.

The ground sensor 283 is a sensor that acquires ground potential information as an electrical signal. The position of the ground sensor 283 will be described later. The ground sensor 283 outputs the sensed ground potential information to the second A/D conversion unit 284.

The second A/D conversion unit 284 converts each piece of information from analog information to digital information. The second A/D conversion unit 284 also outputs each piece of converted information to the second transmission unit 285.

The second transmission unit 285 transmits the second bioinformation output from the main sensor 281 and the fourth bioinformation output from the reference sensor 282, acquired from the second A/D conversion unit 284, to the communication terminal M. The second transmission unit 285 may also transmit the ground potential information output from the ground sensor 283, acquired from the second A/D conversion unit 284, to the communication terminal M. The second transmission unit 285 may generate second difference information, which is the difference between the second bioinformation output from the main sensor 281 and the ground potential information output from the ground sensor 283, acquired from the second A/D conversion unit 284, and transmit it to the communication terminal M. Similarly, the second transmission unit 285 may generate fourth difference information, which is the difference between the fourth bioinformation output from the reference sensor 282 and the ground potential information output from the ground sensor 283, acquired from the second A/D conversion unit 284, and transmit it to the communication terminal M.

In the above example, the second transmission unit 285 transmits the second biometric information or the second differential information to the communication terminal M on the second channel. The second transmission unit 285 also transmits the fourth biometric information or the fourth differential information to the communication terminal M on the fourth channel. The fourth channel may be the same channel as the second channel or a different channel. Each of the above-mentioned biometric information or ground potential information includes information sampled by the second A/D conversion unit 284.

Next, an example of the external shape of the earpiece and the arrangement of each sensor will be described with reference to Figures 3 and 4. Figure 3 is a diagram showing the external shape of the earpiece, and Figure 4 is a cross-sectional view showing the IV-IV section of the earpiece. The first earpiece 2R and the second earpiece 2L have the same basic structure and differ only in the shape suitable for insertion into the left and right ears, so the first earpiece 2R will be used as an example for the description.

The first earpiece 2R includes a housing 21, an ear tip 22, and a wing 23. The housing 21 is a member having a cavity therein, and this cavity contains a speaker 24 and a battery 25. The ear tip 22 is attached to a nozzle 26 protruding from the housing 21.

The nozzle 26 has a shape such that a flange is formed at one end of a cylindrical member. Specifically, the nozzle 26 has a cylindrical portion 31 that is positioned in the wearer's ear canal when the first earpiece 2R is attached, a flange portion 32 that is fixed to the housing 21, and a sound guide portion 33 that connects the interiors of the cylindrical portion 31 and the flange portion 32 in the first direction X.

The cylindrical portion 31 extends in the first direction X so as to protrude from the housing 21, and a locking protrusion 35 for detachably locking the ear tip 22 is formed on the tip side of the cylindrical portion 31. The flange portion 32 is formed on the base end side of the cylindrical portion 31. The sound guide portion 33 functions as a passage through which sound from the speaker 24 passes. Such a nozzle 26 is formed as a rigid body. As long as it has this characteristic, the material for forming the nozzle 26 is not limited, but one example is hard ABS resin.

The housing 21 includes a nozzle fixing portion 40 to which the flange portion 32 of the nozzle 26 is fixed, and an expansion portion 42 that extends from the nozzle fixing portion 40 to the opposite side of the nozzle 26 and is more expanded than the nozzle fixing portion 40. The nozzle fixing portion 40 has an opening 50 through which the cylindrical portion 31 of the nozzle 26 is inserted, a restricting portion 52 that abuts against the flange portion 32 of the nozzle 26 to restrict the nozzle 26 from falling out of the opening 50, and an enclosing portion 54 that connects the restricting portion 52 and the expansion portion 42 and surrounds the flange portion 32. The nozzle 26 can be fixed in the nozzle fixing portion 40 by, for example, fitting the flange portion 32 into the enclosing portion 54, or by bonding the flange portion 32 to the restricting portion 52 or the enclosing portion 54.

The extension section 42 is formed so as to gradually expand as it moves away from the nozzle fixing section 40. Inside the extension section 42, the speaker 24 is housed on the nozzle 26 side, and the battery 25 is housed in an area that is larger than the speaker 24 housing. The extension section 42 also has an opening 60 for wiring at the end 42a where the expansion ends. The speaker 24 and battery 25 are wired and connected to the board 70 through this opening 60. The opening 60 is also used to place the speaker 24 and battery 25 into the housing 21 when assembling the first earpiece 2R.

The end surface 42b of the extension section 42 where the extension ends is formed as a flat surface. A plate 72 to which a substrate 70 is fixed is placed on the end surface 42b, and a cover 74 that covers the substrate 70 and the plate 72 is attached. An antenna for wireless communication is provided on the substrate 70. The antenna is compatible with wireless communication standards such as Bluetooth (registered trademark). Therefore, the first earpiece 2R is configured as a wireless earphone, and is wirelessly connected to devices such as mobile terminals and laptops, and communicates data such as sound with these devices. A first A/D conversion unit 274 and a first transmission unit 275 are provided on the substrate 70.

The wing 23 comprises an annular attachment portion 80 that is fitted into the peripheral surface of the cover 74, and an earmuff portion 82 that protrudes from the attachment portion 80. The earmuff portion 82 protrudes from the attachment portion 80 in a generally U-shape toward the ear tip 22. The earmuff portion 82 mainly functions to hook onto the wearer's outer ear when the first earpiece 2R is worn, and supports the first earpiece 2R so that it does not fall off the wearer's concha. The wing 23, like the housing 21, can be made of a material that has elasticity and flexibility.

The ear tip 22 is formed of a first member 22A and a second member 22B, both of which are electrically conductive. For example, the first member 22A and the second member 22B are formed of different materials, and each is detachable. The shape of the first member 22A is not limited to the examples shown in Figures 3 and 4, and it is sufficient that the first member 22A has a portion that contacts the inner wall of the wearer's ear canal and that this contact portion is configured to contact the ear canal appropriately. In addition, it is preferable that the surface area of this contact portion is large.

The ear tip 22 includes a first member 22A located on the eardrum side and a second member 22B located on the housing 21 side. The first member 22A is made of, for example, conductive rubber, and this conductive rubber contains silver or silver chloride. Preferably, to ensure appropriate conductivity, the conductive rubber contains silver or silver chloride in a predetermined mass % or more of the conductive material contained in the conductive rubber.

The first member 22A may be formed of a silicon material containing a metal-based filler. For example, the first member 22A can be made into a highly conductive material by appropriately blending metal-based fillers such as silver, copper, gold, aluminum, zinc, and nickel into the silicon material. In addition, it is not necessary for all of the filler contained to be silver or silver chloride, as long as only a portion of the filler is silver or silver chloride. This reduces the content of silver or silver chloride, thereby lowering the hardness of the rubber and making it possible to create a conductive rubber with an appropriate hardness.

It is preferable that the second member 22B be formed from an inexpensive non-conductive elastic material (such as silicone rubber).

The housing 21 is made of, for example, conductive rubber, which contains silver or silver chloride. Preferably, to ensure proper conductivity, the conductive rubber contains silver or silver chloride in a predetermined mass percent or more of the conductive material contained in the conductive rubber.

The housing 21 may be formed of a silicon material containing a metal-based filler. For example, the first member 22A can be made into a highly conductive material by appropriately mixing metal-based fillers such as silver, copper, gold, aluminum, zinc, and nickel into the silicon material. In addition, it is not necessary for all of the filler contained to be silver or silver chloride, as long as only a portion of the filler is silver or silver chloride. This reduces the content of silver or silver chloride, thereby lowering the hardness of the rubber and making it possible to create a conductive rubber with an appropriate hardness.

The earmuffs 82 are made of, for example, conductive rubber, which contains silver or silver chloride. Preferably, to ensure proper conductivity, the conductive rubber contains silver or silver chloride in a predetermined mass percent or more of the conductive material contained therein.

The earmuffs 82 may be made of a silicon material containing a metal-based filler. For example, the first member 22A can be made into a highly conductive material by appropriately mixing metal-based fillers such as silver, copper, gold, aluminum, zinc, and nickel into the silicon material. In addition, it is not necessary for all of the filler contained to be silver or silver chloride, as long as only a portion of the filler is silver or silver chloride. This reduces the content of silver or silver chloride, thereby lowering the hardness of the rubber and making it possible to create a conductive rubber with an appropriate hardness.

By configuring as described above, the first member 22A, the housing 21, and the earmuff 82 of the ear tip 22 are conductive. In the example shown in this embodiment, the first member 22A is used as the main sensor 271, the housing 21 is used as the ground sensor 273, and the earmuff 82 is used as the reference sensor 272, but this example is not limited to this. In addition, the conductive material of the housing 21, the conductive material of the first member 22A, and the conductive material of the earmuff 82 may be the same or different. In addition, the conductivity of the first member 22A may be higher than the conductivity of the housing 21 and the earmuff 82. Note that the above-mentioned earpiece is an example of the disclosed technology, and the disclosed technology can be applied to at least two biological information measuring devices that can acquire biological information and communicate independently.

Figure 5 is a block diagram showing the configuration of a communication terminal M according to this embodiment. The communication terminal M includes an information processing unit 251 and a memory unit 261.

The information processing unit 251 is, for example, configured to include a first acquisition unit 252, a second acquisition unit 253, a division unit 254, a feature data extraction unit 255, a judgment unit 259, and a synchronization unit 260.

The memory unit 261 stores, for example, a first biosignal 262, a second biosignal 263, a first feature data 264, a second feature data 265, and a correction value 266. The memory unit 261 corresponds to the memory 330 shown in FIG. 13.

The first biosignal 262 is a biosignal of the user H acquired from the first earpiece 2R. For example, the first biosignal 262 includes at least one of the first bioinformation, the third bioinformation, the ground potential information, the first difference information, and the third difference information.

The second biosignal 263 is a biosignal of the user H acquired from the second earpiece 2L. For example, the second biosignal 263 includes at least one of the second bioinformation, the fourth bioinformation, the ground potential information, the second difference information, and the fourth difference information.

The first feature data 264 is feature data related to the first biosignal 262. The first feature data 264 includes, for example, feature data of the time component or feature data of the frequency component of the first biosignal 262. As described below, the first feature data 264 may include at least one of a peak value of the first biosignal 262, a first power spectrum after frequency conversion of the first biosignal 262, a first power spectrum of a predetermined frequency band, and a first envelope based on the first power spectrum.

The second feature data 265 is feature data related to the second biosignal 263. The second feature data 265 includes, for example, feature data of the time component or feature data of the frequency component of the second biosignal 263. As described below, the second feature data 265 may include at least one of a peak value, a second power spectrum after frequency conversion of the second biosignal 263, a second power spectrum of a predetermined frequency band, and a second envelope based on the second power spectrum.

The correction value 266 is a correction value used when synchronizing the first biosignal 262 and the second biosignal 263. The correction value 266 includes a shift value used to shift at least one of the biosignals, and the details of which will be described later.

The first acquisition unit 252 acquires the first biosignal 262 from the first earpiece 2R. For example, the first acquisition unit 252 acquires the first biosignal 262 from the communication terminal M by communication via Bluetooth (registered trademark). Communication via Bluetooth (registered trademark) is realized by the network communication interface 320 shown in FIG. 14.

The second acquisition unit 253 acquires the second biosignal 263 from the second earpiece 2L. For example, the second acquisition unit 253 acquires the second biosignal 263 from the communication terminal M by communication via Bluetooth (registered trademark).

The division unit 254 divides the biosignal acquired in sequence by the first acquisition unit 252 or the second acquisition unit 253 into a predetermined time interval. For example, if the predetermined time is 10 seconds, the division unit 254 divides the biosignal acquired in sequence by the first acquisition unit 252 or the second acquisition unit 253 into 10-second intervals. The interval of the predetermined time interval for division can be set arbitrarily.

The determination unit 259 determines the similarity between the first feature data 264 based on the first biosignal 262 of a divided predetermined time (also called a predetermined section) and the second feature data 265 based on the second biosignal 263 of a predetermined time shifted in time with reference to the first feature data of the predetermined section. An example of the processing of the determination unit 259 will be described later using Figures 9 and 10. For the shift, for example, the predetermined section is shifted by a set time from -2 seconds to +2 seconds.

The synchronization unit 260 synchronizes the first biosignal 262 and the second biosignal 263 based on the similarity between the first feature data 264 and the second feature data 265 determined by the determination unit 259. For example, the synchronization unit 260 uses a time shift amount (correction value) between the first biosignal 262 and the second biosignal 263 obtained from the point in time at which the correlation value, which is one of the indices of similarity determined by the determination unit 259, is highest, to shift one of the biosignals in time by the correction value, thereby synchronizing the first biosignal 262 and the second biosignal 263. As a result, the left and right biosignals received by the communication terminal M from the earpieces can be appropriately synchronized.

The communication terminal M may include a feature data extraction unit 255. The feature data extraction unit 255 extracts first feature data 264 and second feature data 265 from the first biosignal 262 and the second biosignal 263, respectively.

The feature data extraction unit 255 may include, for example, a frequency conversion unit 256. The frequency conversion unit 256 performs frequency conversion on the first biosignal 262 and the second biosignal 263. The frequency conversion unit 256 applies FFT (Fast Fourier Transform) processing, wavelet transformation, or the like to the first biosignal 262 and the second biosignal 263, which include brain waves of multiple frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves, to calculate frequency components (e.g., power spectrum).

When the feature data extraction unit 255 includes a frequency conversion unit 256, the determination unit 259 may determine the similarity between a first power spectrum after frequency conversion of the first biological signal 262 included in the first feature data 264 and a second power spectrum after frequency conversion of the second biological signal 263 included in the second feature data 265. The determination unit 259 may determine, for example, the similarity between the first power spectrum of a predetermined interval of frequency components and the second power spectrum of a predetermined interval of frequency components shifted by a set time. As described above, by using frequency components in which features are likely to appear as feature data, it is possible to easily determine the similarity.

The feature data extraction unit 255 may further include, for example, a frequency band extraction unit 257. The frequency band extraction unit 257 extracts a predetermined frequency band after frequency conversion. As a preferred example, the frequency band extraction unit 257 applies a band pass filter to the frequency components calculated by the frequency conversion unit 256 to extract the power spectrum of alpha waves.

When the feature data extraction unit 255 includes a frequency band extraction unit 257, the determination unit 259 may determine the similarity using a first power spectrum of a predetermined frequency band among the frequency components of a predetermined section, and a second power spectrum of a predetermined frequency band among the frequency components of a predetermined section shifted by a set time. The determination unit 259 may determine the similarity between the first power spectrum of alpha waves and the second power spectrum of alpha waves, for example. As a result, it is possible to set a predetermined frequency band in which a wavy shape that can be used as a feature is likely to be formed, and determine the similarity based on the power spectrum of this predetermined frequency band.

The feature data extraction unit 255 may further include, for example, an envelope calculation unit 258. The envelope calculation unit 258 calculates a first envelope based on the first power spectrum and a second envelope based on the second power spectrum. The envelope calculation unit 258 may, for example, apply a Hilbert transform to the first power spectrum and the second power spectrum of a plurality of frequency bands to calculate the first envelope and the second envelope. The envelope calculation unit 258 may also apply a Hilbert transform to the first power spectrum and the second power spectrum of a predetermined frequency band to calculate the first envelope and the second envelope. As a suitable example, the envelope calculation unit 258 applies a Hilbert transform to the first power spectrum of alpha waves and the second power spectrum of alpha waves to calculate the first envelope and the second envelope. By performing the Hilbert transform, the first envelope and the second envelope can be calculated regardless of the difference in power value between the first power spectrum and the second power spectrum.

When the feature data extraction unit 255 includes the envelope calculation unit 258, the determination unit 259 may determine the similarity between the first envelope included in the first feature data 264 for a predetermined section and the second envelope included in the second feature data 265 for a predetermined section shifted by a set time. For example, the determination unit 259 may determine, for each predetermined section, the time at which the correlation value between the first envelope and the second envelope shifted by the set time is highest. As described above, by using an envelope that is not based on the difference in power values between the left and right as feature data, a more appropriate determination of similarity can be made.

The above-mentioned determination unit 259 may determine the similarity between the first feature data based on the first biosignal of the divided predetermined section and the second feature data based on the second biosignal of a predetermined time shifted in sequence with respect to the predetermined section of the first feature data. For example, when the division unit 254 divides the first biosignal 262 and the second biosignal 263 at 10-second intervals, the determination unit 259 shifts the second biosignal 263 of 10 seconds corresponding to the first biosignal 262 of the predetermined section in time with respect to the first biosignal 262 of the predetermined section. The determination unit 259 calculates a correlation value between the first feature data 264 of the first biosignal 262 of the predetermined section and the second feature data 265 of the section in which the second biosignal 263 is shifted in time. As a result, since an objective correlation value is used as the similarity between the first feature data 264 and the second feature data 265, the accuracy of similarity determination can be improved, and the synchronization process between the first biosignal and the second biosignal can be more appropriately performed.

The above-mentioned determination unit 259 may calculate the similarity between the amplitude value indicating the first feature data 264 and the amplitude value indicating the second feature data 265. The determination unit 259 may calculate the similarity between the amplitude value of the first power spectrum in a predetermined section and the amplitude value of the second power spectrum in a predetermined section shifted by a set time. This makes it possible to calculate the similarity using the amplitude value, making it easier to determine the similarity and reducing the processing load, thereby enabling synchronization processing between the first biosignal 262 and the second biosignal 263.

Here, we will describe the details of an experiment conducted by the inventors to confirm whether the synchronization processing of the disclosed technology works effectively on the biosignals obtained from each earpiece equipped with a biosensor, using alpha waves contained in brain waves.

<Experiment Overview>
(subject)
This experiment was conducted on three subjects, A to C. First, this experiment will be explained.

(Acquisition items)
The subject's electroencephalogram signal, which serves as a reference value, is acquired using an electroencephalogram meter, Polymate Mini (registered trademark), manufactured by Miyuki Giken Co., Ltd. In addition, the subject's electroencephalogram signal is acquired from both ears using earpieces similar to the first earpiece 2R and the second earpiece 2L.

(How to obtain)
For the same subject, electrodes were attached to the subject's head at T7 (left middle temporal region), Cz (median center), and T8 (right middle temporal region) according to the International 10/20 method using the above-mentioned electroencephalograph, and the above-mentioned earpieces were attached to the subject's left and right ears. The subject repeatedly opened and closed both eyes every 15 seconds, and electroencephalogram signals were obtained from the above-mentioned electroencephalograph and earpieces for at least 120 seconds (four cycles, assuming that opening and closing is one cycle).

(process)
Each electroencephalogram signal acquired by the electroencephalogram meter and the electroencephalogram signal acquired by the left earpiece are transmitted to a first personal computer. The electroencephalogram signal acquired by the right earpiece is transmitted to a second personal computer different from the first personal computer. The second personal computer transmits the electroencephalogram signal received from the right earpiece to the first personal computer. The first personal computer executes the above-mentioned synchronization process for each electroencephalogram signal. The first personal computer generates a difference signal between the electroencephalogram signal acquired by the left earpiece and the electroencephalogram signal acquired by the right earpiece after the synchronization process. In addition, frequency conversion and a bandpass filter are applied to each electroencephalogram signal including the difference signal, the left and right electroencephalogram signals, and the electroencephalogram signal acquired by the electroencephalogram meter, and the power spectrum of alpha waves is extracted.

(Evaluation index)
In this experiment, the indices used to evaluate the similarity of each extracted EEG signal are the alpha wave power obtained from the left ear EEG signal, the alpha wave power obtained from the right ear EEG signal, and the correlation coefficients (correlation values) between the alpha wave power of the differential signal between the left ear EEG signal and the right ear EEG signal, and the alpha wave power obtained from the EEG meter.

(Example of subject A)
An example of data of subject A in this experiment will be described. FIG. 6 is a diagram showing the fluctuation of the power of alpha waves of subject A. FIG. 6 is a diagram showing the 4-second time window for each of the electroencephalogram signals acquired by the electroencephalograph and the earpiece after the above-mentioned synchronization process, in which the 4-second time window is slid by 0.5 seconds, and the power of the alpha waves is calculated and plotted in chronological order by applying frequency conversion and bandpass filtering to the electroencephalogram signals of each time window. For example, the power shown in FIG. 6 indicates the peak value of the power of the alpha waves. EP-L, EP-R, and EP-Diff respectively represent the power of the alpha waves obtained from the electroencephalogram signal of the left ear, the power of the alpha waves obtained from the electroencephalogram signal of the right ear, and the power of the alpha waves of the difference signal between the electroencephalogram signal of the left ear and the electroencephalogram signal of the right ear. It should be noted that EP-Diff is the difference signal after the synchronization process between the electroencephalogram signal of the left ear and the electroencephalogram signal of the right ear. PM-T7, PM-Cz, and PM-T8 represent the alpha wave power obtained from the electroencephalographs attached to T7, Cz, and T8, respectively.

In the example shown in Figure 6, it can be seen that the power of alpha waves in each EEG signal also tends to decrease when the subject has both eyes open, and tends to increase when both eyes are closed. Alpha waves are said to occur in a relaxed state, and it is believed that by relaxing with the eyes closed, more alpha waves are generated than when the eyes are open. This means that each EEG signal can be said to be appropriately extracted from each device.

Figure 7 shows the power spectral density for a given 4-second period when the subject had both eyes closed. EP-L, EP-R, and EP-Diff in Figure 7 represent the power spectral density of the EEG signal at the left ear, the power spectral density of the EEG signal at the right ear, and the power spectral density of the differential signal between the EEG signals at the left and right ears, respectively. PM-T7, PM-Cz, and PM-T8 represent the power spectral density of the EEG signal obtained from the EEG monitors attached to T7, Cz, and T8, respectively.

In the example shown in Figure 7, the power spectral density of alpha waves, which is in the 8 to 13 Hz frequency band, peaks around 10 Hz for both EEG signals. This means that both the EEG and the earpieces can adequately extract the characteristics of the subject's brain state (e.g., alpha waves). It can also be said that the earpieces can extract the peak value of alpha waves in the same way as the EEG.

Figure 8 shows the matrix of correlation coefficients between EP-L, EP-R, and EP-Diff shown in Figure 6 and PM-T7, PM-Cz, and PM-T8. First, the EEG acquires EEG signals from the scalp at a specified position on the head, whereas the earpiece acquires EEG signals from the ear. In other words, the acquisition position of the EEG signal is different for each device. This experiment was conducted under the assumption that the EEG signal acquired by the EEG and the EEG signal acquired from the earpiece will not have exactly the same waveform due to differences in the acquisition position of the EEG signal and differences in the acquisition device. Therefore, in this experiment, if the correlation coefficient is 0.3 or more, the similarity between the power value of the alpha wave acquired by the EEG and the power value of the alpha wave acquired by the earpiece can be evaluated as being high.

In the example shown in Figure 8, all correlation coefficients are greater than or equal to 0.3. This indicates that there is a high degree of similarity between the alpha wave power values obtained from the electroencephalograph and the alpha wave power values obtained from the earpieces.

The above experimental results show that earpieces equipped with biosensors can properly acquire alpha waves contained in brain waves. In addition, it can be seen that the synchronization process of the disclosed technology works effectively with the biosignals acquired from each earpiece equipped with a biosensor using alpha waves contained in brain waves.

<Example of Determination and Synchronization Processing>
Next, an example of the processing of the determination unit 259 and the synchronization unit 260 will be described with reference to Figs. 9 to 11. Fig. 9 is a diagram showing an example of the first biological signal 262 and the second biological signal 263 before synchronization processing by the synchronization unit 260. In the example shown in Fig. 9, the solid line R represents the electroencephalogram signal itself for a given 1 second acquired from the first earpiece 2R, and the dashed line L represents the electroencephalogram signal itself for 1 second acquired from the second earpiece 2L in the same time period. In the example shown in Fig. 9, sampling is performed at 600 Hz, and there are sampling values of 1 second÷(1/600)=600 data points.

Figure 10 is a diagram showing an example in which the determination unit 259 determines the similarity between the first envelope of the first biosignal 262 shown in Figure 9 and the second envelope of the second biosignal 263. In the example shown in Figure 10, the determination unit 259 calculates the correlation value between the first envelope and each of the second envelopes of the second biosignal 263 for 1 second, which are shifted by 1 data point (e.g., 1/600 second) from -2.5 seconds to +2.5 seconds based on the first envelope, and displays a graph showing the fluctuation of the correlation value. In the example shown in Figure 10, the correlation value becomes highest by shifting the second biosignal 263 forward in time by 254 data points.

Figure 11 is a diagram showing an example of synchronization processing by the synchronization unit 260. In the example shown in Figure 11, the synchronization unit 260 shifts the one-second EEG signal acquired from the second earpiece 2L forward in time by 254 data points in accordance with the highest correlation value among the correlation values calculated by the determination unit 259. This makes it possible to synchronize the one-second EEG signal acquired from the first earpiece 2R with the one-second EEG signal acquired from the second earpiece 2L in the same time period.

<Operation Processing>
Next, the operation according to the embodiment will be described with reference to Fig. 12 and Fig. 13. Fig. 12 is a flowchart showing an example of the process of the communication terminal M according to the embodiment.

In step S11, the first acquisition unit 252 acquires a first biosignal 262 from the first earpiece 2R having a biosensor.

In step S12, the second acquisition unit 253 acquires a second bio-signal 263 from the second earpiece 2L having a bio-sensor.

In step S13, when a predetermined time has elapsed since acquiring the first biological signal 262 or the second biological signal 263, the division unit 254 divides the first biological signal 262 and the second biological signal 263 every predetermined time.

In step S14, the feature data extraction unit 255 extracts first feature data 264 based on the first biological signal 262 of the divided specified section and second feature data 265 based on the second biological signal 263 of the divided specified section.

In step S15, the judgment unit 259 judges the similarity between the first feature data 264 based on the first biological signal 262 of the divided specified section and the second feature data 265 based on the second biological signal 263 of a specified time shifted in time with respect to the specified section.

In step S16, the synchronization unit 260 synchronizes the first biological signal 262 and the second biological signal 263 based on the similarity.

A detailed example of the processing of the feature data extraction unit 255 will be described with reference to Fig. 13. Fig. 13 is a flowchart relating to the processing of the feature data extraction unit 255 according to this embodiment. In the example shown in the figure, in step S21, the frequency conversion unit 256 performs frequency conversion into a first biosignal 262 and a second biosignal 263.

In step S22, the frequency band extraction unit 257 extracts a specified frequency band after frequency conversion.

In step S23, the envelope calculation unit 258 applies a Hilbert transform to the first power spectrum to calculate a first envelope, and applies a Hilbert transform to the second power spectrum to calculate a second envelope. After that, in step S15 shown in FIG. 12, similarity is determined using each envelope as feature data.

<Modification>
The above-mentioned embodiment or each example is for facilitating understanding of the present invention, and is not to be construed as limiting the present invention. The present invention may be modified/improved without departing from the spirit thereof, and the present invention also includes equivalents thereof. In addition, the present invention can form various disclosures by appropriate combinations of multiple components disclosed in the above-mentioned embodiment or each example. For example, some components may be deleted from all components shown in the embodiment. Furthermore, components may be appropriately combined in different embodiments.

The feature data extraction unit 255 may include only the frequency conversion unit 256 and the envelope calculation unit 258, and may not include the frequency band extraction unit 257. In this case, the envelope calculation unit 258 calculates the first envelope and the second envelope based on the first power spectrum and the second power spectrum of the frequency components calculated by the frequency conversion unit 256. The envelope calculation unit 258 applies a Hilbert transform to the first power spectrum and the second power spectrum of the frequency components including multiple frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves calculated by the frequency conversion unit 256, to calculate the first envelope and the second envelope.

In the above embodiment, the peak value of the biosignal, the power spectrum after frequency conversion of the biosignal, the power spectrum of a predetermined frequency band, and the envelope based on the power spectrum are given as examples of the characteristic data of the biosignal, but the present invention is not limited to these. Characteristic noise contained in the biosignal may be used as the characteristic data of the biosignal. By aligning the position where this noise occurs, it is possible to synchronize at least two biosignals.

In the above embodiment, the frequency band extraction unit 257 extracts the power spectrum of alpha waves, but the present invention is not limited to this. The frequency band extraction unit 257 may extract the power spectrum of a frequency band other than alpha waves, such as beta waves, theta waves, delta waves, and gamma waves.

The synchronization unit 260 may also apply the correction value 266 used in the synchronization process for a specific section to another section to perform the synchronization process. For example, the synchronization unit 260 may apply the correction value 266 used in the synchronization process for a certain section to the synchronization process for a subsequent predetermined section. This makes it possible to omit processes such as extraction of feature data and determination of similarity for the predetermined section, thereby reducing the processing load of the synchronization process. When the predetermined section has elapsed, extraction of feature data, etc. is performed again.

The communication terminal M may include a post-processing unit that performs a predetermined process on the first biosignal 262 and the second biosignal 263 after the synchronization process. The post-processing unit may, for example, calculate a differential signal between the first biosignal 262 and the second biosignal 263. This makes it possible to cancel common noise contained in the first biosignal 262 and the second biosignal 263, and obtain a biosignal of good quality. The post-processing unit may also perform cross-reference processing to increase the signal components or increase the potential difference by performing cross-processing such as processing the first bioinformation and the fourth bioinformation as the main bioinformation and the reference bioinformation, respectively, and processing the second bioinformation and the third bioinformation as the main bioinformation and the reference bioinformation, respectively.

The post-processing unit may also extract brain waves of a predetermined frequency band from the biological signal after the synchronization process, for example. Specifically, the post-processing unit can check, for example, whether the user H is in a relaxed state by extracting alpha waves. The post-processing unit can check, for example, whether the user H is in an irritated state by extracting beta waves. The post-processing unit can check, for example, whether the user H is in a light sleep state by extracting theta waves. The post-processing unit can check, for example, whether the user H is in a deep sleep state by extracting delta waves. The post-processing unit can check, for example, whether the user H is in a focused state by extracting gamma waves. The post-processing unit may extract a frequency band different from the predetermined frequency band used when extracting the feature data, depending on the user's settings, etc.

<Hardware Configuration>
14 is a block diagram showing a hardware configuration according to the present embodiment. The communication terminal M is configured by an information processing device, and is, for example, a mobile terminal (such as a smartphone), a computer, a tablet terminal, etc. The communication terminal M is also referred to as a communication terminal 300.

The communication terminal 300 includes one or more processors (e.g., CPUs) 310, one or more network communication interfaces 320, memory 330, a user interface 350 and one or more communication buses 370 for interconnecting these components.

The user interface 350 includes a display 351 and an input device (such as a keyboard and/or a mouse or some other pointing device) 352. The user interface 350 may also be a touch panel.

Memory 330 may be, for example, a high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid-state storage device, or may be a non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

Another example of memory 330 may be one or more storage devices located remotely from processor 310. In one embodiment, memory 330 stores the following programs, modules and data structures, or a subset thereof. Memory 330 may also be a computer-readable non-transitory storage medium.

The one or more processors 310 read and execute a program from the memory 330 as necessary. For example, the one or more processors 310 execute a program stored in the memory 330 to perform the processing of the information processing unit 251.

2 Earphone 2R First earpiece 2L Second earpiece 22 Ear tip 251 Information processing unit 252 First acquisition unit 253 Second acquisition unit 254 Dividing unit 255 Feature data extraction unit 256 Frequency conversion unit 257 Frequency band extraction unit 258 Envelope calculation unit 259 Determination unit 260 Synchronization unit 261 Storage unit 271, 281 Main sensor 272, 282 Reference sensor 273, 283 Ground sensor 274 First A/D conversion unit 275 First transmission unit 284 Second A/D conversion unit 285 Second transmission unit 300 Communication terminal 310 Processor 320 Network communication interface 330 Memory
H User
M Communication Terminal

Claims (8)

An information processing method executed by a communication device, comprising:
Obtaining a first biosignal from a first earpiece comprising a biosensor;
acquiring a second biosignal from a second earpiece comprising a biosensor;
Dividing the first biological signal and the second biological signal at predetermined time intervals;
performing frequency conversion on the first biological signal and the second biological signal for each predetermined time;
determining a similarity between first feature data including a first power spectrum after frequency conversion of a first biological signal in a divided predetermined section and second feature data including a second power spectrum after frequency conversion of a second biological signal in the predetermined time shifted in time with respect to the first feature data in the predetermined section;
An information processing method that performs a synchronization process to synchronize the first biological signal and the second biological signal based on the similarity. The method further comprises extracting a predetermined frequency band after the frequency conversion;
The determining step comprises:
The information processing method according to claim 1 , further comprising determining the similarity using a first power spectrum and a second power spectrum of the predetermined frequency band. calculating a first envelope based on the first power spectrum and a second envelope based on the second power spectrum;
The determining step comprises:
The information processing method according to claim 2 , further comprising determining a similarity between the first envelope included in the first feature data and the second envelope included in the second feature data. The determining step comprises:
The information processing method according to claim 1 , further comprising: calculating a correlation value between the first feature data and the second feature data. The synchronizing step includes:
The information processing method according to claim 1 , further comprising: applying a correction value used in the synchronization process for the predetermined section to another section to perform the synchronization process. The information processing method according to claim 1 , further comprising: generating a differential signal using the first biological signal and the second biological signal after the synchronization process. A first acquisition step of acquiring a first biosignal from a first earpiece having a biosensor in a communication device;
a second acquisition step of acquiring a second biosignal from a second earpiece including a biosensor;
A dividing step of dividing the first biological signal and the second biological signal into predetermined time intervals;
performing frequency conversion on the first biological signal and the second biological signal at each predetermined time;
a determination step of determining a similarity between first feature data including a first power spectrum after frequency conversion of a first biological signal in a divided predetermined section and second feature data including a second power spectrum after frequency conversion of a second biological signal in the predetermined time shifted in time with respect to the first feature data in the predetermined section;
A synchronization step of synchronizing the first biological signal and the second biological signal based on the similarity. a first acquisition unit that acquires a first biosignal from a first earpiece that includes a biosensor;
a second acquisition unit that acquires a second biological signal from a second earpiece that includes a biological sensor;
A division unit that divides the first biological signal and the second biological signal at predetermined time intervals;
a frequency conversion unit that performs frequency conversion on the first biological signal and the second biological signal at each predetermined time;
a determination unit that determines a similarity between first feature data including a first power spectrum after frequency conversion of a first biological signal in a divided predetermined section and second feature data including a second power spectrum after frequency conversion of a second biological signal in the predetermined time shifted in time with respect to the first feature data in the predetermined section;
A communication device comprising: a synchronization unit that synchronizes the first biological signal and the second biological signal based on the similarity.

Images