TW202000124A - Algorithmic method for extracting human pulse rate from compressed video data of a human face - Google Patents

Abstract

This invention includes the following steps: a first overlap cutting step, a first processing step, a first overlap reconstructing step, a second overlap cutting step, a second processing step, and a second overlap reconstructing step. Based on these steps, the single channel data separation is adopted for obtaining the biological information. Only the green channel is considered because it has the lowest influence, followed by the singular spectrum analysis (SSA) process, two times relationship selection process, and the spectral masking process. At last, the final heart rate can be obtained. This invention has the following advantages: 1) its algorithm for extracting human pulse rate of a human face is unique; 2) its application scope is wide; and 3) the single channel separation method can minimize the transmission volume of video data.

Description

Heart rate extraction algorithm for face compressed image

The present invention relates to a heart rate extraction algorithm for face compressed images, in particular to a face with a face compression image that is quite novel, has a wide range of applications, and a single channel signal separation method greatly reduces the amount of video image transmission. Calculation method of heart rate extraction of compressed images.

In 2008, Wim Verkruysse and others pointed out that through natural light and consumer-grade cameras, heart rate waveforms can be detected, and physiological information can be analyzed remotely. In 2010 Ming-Zher Poh and others used blind source separation technology to analyze the color signal in the video and extract the heart rate signal from the human body. In 2012, Hao-Yu Wu and others proposed the Yura image magnification algorithm to amplify small changes in skin color in the video stream, so that changes in skin color can be observed. In 2013, Guha Balakrishnan and others pointed out that the rhythmic beating of the heart can cause tiny head shaking, which can be detected by detecting this kind of head shaking in the video. In 2015, the City University of Hong Kong and the Eindhoven University of Technology in the Netherlands each developed measurement techniques that limit the subject’s limbs in a non-moving state, for example, the subject can turn or shake on the head In the case of, a reliable heart rhythm value is detected. In 2016, Tulyakov (Tulyakov) and others proposed an adaptive matrix filling method. While detecting heart rate signals, it automatically detects which areas contain heart rate signals and uses these areas for heart rate detection. The above methods take uncompressed video data as the processing object, and adopt a three-channel signal separation technology to separate the heart rate signal from the original signal. When processing compressed video, because the compression algorithm has a great influence on the heart rate signal, none of the above methods can effectively obtain accurate heart rate information. In 2016, Hanfland and others compressed the original video and compared the compressed video with the original video. The research results show that the heart rate signal still exists in the compressed video, but the overall quality has been greatly reduced. In 2017, McDuff and others used two compression methods (x264 and x265) to compress the original video to different bit rates. The results show that video compression significantly reduces the signal-to-noise ratio of the heart rate signal. In addition, the current remote optical volume change tracing method (Rppg) uses uncompressed images as processing objects, that is, extracts the heart rate information hidden in the face from the uncompressed image data; one of the negative effects of this method It will cause storage difficulties. As we all know, the amount of data in uncompressed images is huge. For example, an image with a resolution of 640*480 and one minute at 30fps requires about 1.7GB of storage space. Such huge storage requirements will inevitably result in a waste of resources; another problem is that uncompressed image data cannot be transmitted over long distances at all, which puts great restrictions on the application of rPPG technology. The current network transmission capability cannot realize the real-time transmission of uncompressed images; the existing rPPG technology cannot be applied to occasions that require real-time transmission of long-distance images. In short, today's images are transmitted over long distances, and almost all images are compressed to reduce the amount of data transmitted. There are four common compression methods: x264, x265, vp8 and vp9. However, the compressed image can hardly effectively obtain accurate heart rate information. In view of this, it is necessary to develop technology that can solve the above-mentioned conventional shortcomings.

The purpose of the present invention is to provide a heart rate extraction algorithm for face compressed images, which has the advantages of a relatively new heart rate extraction algorithm for face compressed images, a wide range of applications, and a single channel signal separation method to greatly reduce the amount of video image transmission. . In particular, the problem to be solved by the present invention is that uncompressed image data cannot be transmitted over a long distance. The technical means to solve the above problem is to provide a heart rate extraction calculation method for face compressed images, which includes the following steps: 1. The first overlapping cutting step; two. The first processing step: [a] pre-processing step; [b] band-pass filter processing step; [c] first singular spectrum analysis step; [d] double relationship screening step; [e] reconstruction step; three. The first step of overlapping addition; 4. The second overlapping cutting step; 5. The second processing step: [f] frequency mask construction step; [g] second singular spectrum analysis step; [h] frequency mask screening step; VI. The second overlap and add step. The following examples and drawings are used to explain the present invention in detail:

Referring to Figures 1 and 2, the present invention is a heart rate extraction algorithm for a face compressed image, which includes the following steps: 1. The first overlapping cutting step S1: refer to FIG. 3 to obtain a face compressed video M after video compression processing, the time length of which is T1, the face compressed video M is overlapped and cut into a plurality of small video M1, the The time length of each small video M1 is T2; the method of overlapping cutting is defined as the beginning of the face compressed video M, the first of the small video M1 is captured, and then every other step for a long time T3 , And then capture another small video M1, repeat the aforementioned actions until the face compression video M ends, and then obtain a plurality of small video M1. For example: if the face compression video M is 600 seconds (that is, the time length T1), each small video M1 is 3 seconds (that is, the time length T2), and the step time T3 is 1.5 seconds, then the last It was cut into 399 short films M1. two. The first processing step S2 is to perform the following steps one by one for each small video segment M1: [a] Pre-processing step S21: each small video segment M1 includes the original image K (see Figure 4); for each A tracing of the original image K for the face area (see FIGS. 5A and 5B, which is a well-known technique), a face area P can be obtained, the face area P has X by Y pixels, for the X by Y The green values of the complex pixels are averaged to obtain a scalar quantity, which is defined as the average green value of a single frame; and so on, the average green value of N frame can be obtained, and a raw signal G0 is further obtained from the average green value of N frame (t) (see Figure 6), where t=1 to N. [b] Bandpass filter processing step S22: perform bandpass filter processing on the aforementioned original signal G0(t), select a signal with a frequency between 0.8 Hz and 2.0 Hz (see FIGS. 7 and 8), and obtain a first A signal G1(t) (see Figure 9), where t=1 to N. [c] The first singular spectrum analysis (SSA) step S23: the first signal G1(t) is decomposed into a plurality of first subsequences G1m (see FIG. 10), and all the A sub-sequence G1m is subjected to fast Fourier transform to obtain a plurality of spectrums of the first sub-sequence G1m, and then the frequency with the largest amplitude in the spectrum of each first sub-sequence G1m is taken as its first main frequency G1f. Among them, the aforementioned singular spectrum analysis is a known technology. In practice, it can be implemented based on general market application software (such as MATLAB), so the detailed process of the singular spectrum analysis will not be repeated here. [d] Double relationship screening step S24: compare the aforementioned first subsequences G1m two by two, if the two first main frequencies G1f have a double relationship after comparison, the two first main frequencies G1f are retained, otherwise The two first main frequencies G1f are discarded. If there is no one satisfying this double relationship after pairwise comparison, all the first subsequences G1m are retained; and at least two first reserved subsequences can be obtained. [e] Reconstruction step S25: Reconstruct all the first reserved subsequences of the previous step to obtain a second signal G2(t) (see FIG. 11), where t=1 to N, the second signal The horizontal axis of G2(t) is time, and the vertical axis is green value intensity. The second signal G2(t) has the time length T2. three. The first overlapping addition step S3: referring to FIG. 12, the plurality of second signals G2(t) obtained after the first processing step S2 are processed by using the existing overlapping cosine window method of overlapping addition technology to obtain a After the overlapping addition, the second signal G22 has the time length T1; the overlapping addition technique is defined as between the two adjacent second signals G2(t), which is added by the step length T3. That is, the above processing is performed on each small signal (such as the second signal G2(t)), the processing result is multiplied by a raised cosine window (Hanning window), and then the processing results are added to obtain a long time after processing Signal (for example, the second signal G22 after the overlapping addition). In practice, it can be implemented based on general market application software (such as MATLAB), so the detailed process will not be repeated here. four. The second overlapping cutting step S4: refer to FIG. 13 to obtain the aforementioned second signal G22 after overlap addition, which has the time length T1, and the second signal G22 is overlapped and cut into a plurality of small signals after the overlap addition, the Each small segment signal has the length of time T2; the way of overlapping cutting is defined as the beginning of the second signal G22 after the overlap is added, to capture a small segment signal, and then every other step for a long time T3, Retrieve another small segment signal and repeat the above-mentioned actions until the second signal G22 ends after the overlap and addition, and then a plurality of small segment signals can be obtained, and each small segment signal is defined as the third signal G3(t) , Where t=1 to N. Fives. Second processing step S5: Perform the following steps one by one for each third signal G3(t): [f] Frequency mask construction step S51: Use the frequency with the largest amplitude of the third signal G3(t) as its The center frequency G3j (see FIG. 14), and with the center frequency G3j as the center, set a passing upper limit frequency G3ju and a passing lower limit frequency G3jd to obtain a frequency mask range. [g] Step S52 of the second singular spectrum analysis (SSA): the third signal G3(t) is decomposed into a plurality of second subsequences G3m (see FIG. 15), and all the second subsequences G3m are quickly processed Fourier transform to obtain a plurality of second sub-sequences G3m spectrum, and then use the frequency of the largest amplitude in the spectrum of each second sub-sequence G3m as its second main frequency G3f. [h] Frequency mask screening step S53: only the second sub-sequence G3m whose second main frequency G3f is within the frequency mask range is reserved, then at least one second reserved sub-sequence can be obtained by screening, which becomes a The fourth signal G4(t), where t=1 to N, the horizontal axis of the fourth signal G4(t) is time, and the vertical axis is the intensity of the green value. If any second reserved subsequence is not obtained by screening, the fourth signal G4(t) is directly equal to the third signal G3(t). six. The second overlapping addition step S6: referring to FIG. 16, the plurality of fourth signals G4(t) obtained after the second processing step S5 are processed by the existing overlapping addition technique of the raised cosine window method, and A fourth signal G44 after overlapping addition is obtained, which has the time length T1, and is the final heart rate signal. In practice, in the step S1 of obtaining the original image, two sets of video devices 10 and a network connection device 20 are provided. The two sets of video devices 10 are connected to the device 20 through the network to achieve a video contact. The compression method of the face compressed image is selected from one of x264, x265, vp8, and vp9. Regarding the aforementioned singular spectrum analysis (SSA), which is a well-known technique, it is explained as follows: Input: a vector of length N . The first step: Hankel matrix embedding (English is Hankel matrix embedding), that is, converting y into matrix X as follows. ; Where K = N − L +1, L and K are the rows and columns of the matrix, set by the user. Step 2: Singular value decomposition (SVD) for the aforementioned matrix X: . among them Is the singular value of X, with Is the corresponding singular vector, for Of rank. make For a submatrix with rank 1, then: . The third step: reconstruction. The method is Diagonal averaging. This step is more complicated than that. Please refer to the attachment for details. The function is to convert the matrix addition form of the above formula into a vector addition form: . Every Called a reconstructed component (RC). Step 4: As needed, in Choose the RC that meets your needs to get the final time series. For example: In our algorithm, The time sequence obtained by a time window of the green channel, for example, if the time window (that is, the time length T2) is 3s and the frame rate is 30fps, then the length of x is N=90. K is generally set to half of N, which is 45. Calculated K=46. After SVD, r is generally the smaller of K and N, ie r=45. In the fourth step, there are r=45 , We can only consider the top 20 to speed up the calculation. The key point of the present invention is to fully consider the impact of compressed images on human physiological signals, and use a single-channel signal separation method to extract physiological signals. Then select the channel (G channel) that is least affected by the compression algorithm for processing. That is, only the green values of the complex pixels of the X times Y of the face area are captured and averaged. The advantage of this is to fully avoid the impact of the compression algorithm on the heart rate signal, so that the heart rate signal can be accurately and stably extracted from the compressed video. In addition, in conjunction with singular spectrum analysis (singular spectrum analysis, SSA), the frequency structure characteristics of the heart rate signal can be used to effectively identify the heart rate signal in the mixed signal including noise. The frequency mask is then used to further filter out noise and obtain a more accurate heart rate signal. The invention extracts the heart rate signal from the mixed signal through three methods, which can greatly ensure the accuracy of the heart rate signal, and can effectively avoid the influence of the compression algorithm on the calculation of the heart rate signal. For the experimental results of this case, please refer to Figures 17A to 18D. In a static film (the compression method is vp8, the bit rate is 100kb/3), when only the third step of band-pass filtering (filter for short) is performed, the resulting waveform in the frequency domain is shown in Figure 17A; If you continue to the first singular spectrum analysis and double relationship filtering (referred to as filter+SSA), the waveform in the frequency domain is shown in Figure 17B; if you continue to the second singular spectrum analysis and The frequency mask filtering step (referred to as filter+SSA+refine; in this case), the resulting waveform in the frequency domain is shown in Figure 17C. The comparison between these three and the actual heart rate signal in the time domain is shown in the 17D graph (the first curve L1, the second curve L2, the third curve L3, and the fourth curve L4 respectively represent the actual heart rate, bandpass filter, and band Pass filter + SSA, band pass filter + SSA + frequency mask), it can be seen that the result of this case is closest to the actual heart rate signal. In addition, in a dynamic film (the compression method is x264, the bit rate is 584kb/3), when only the third step of band-pass filtering (filter for short) is performed, the resulting waveform in the frequency domain is as shown in Figure 18A If you continue to the first singular spectrum analysis and double relationship filtering (referred to as filter+SSA), the waveform in the frequency domain is shown in Figure 18B; if you continue to the second singular spectrum The steps of analysis and frequency mask filtering (referred to as filter+SSA+refine; in this case), the waveform of the result in the frequency domain is shown in Figure 18C. The comparison between these three and the actual heart rate signal in the time domain is shown in the 18D diagram (the first curve LA, the second curve LB, the third curve LC, and the fourth curve LD respectively represent the actual heart rate, band pass filter, and band Pass filter + SSA, band pass filter + SSA + frequency mask), which can be seen that the result of this case is closest to the actual heart rate signal. The advantages and effects of the present invention are as follows: [1] The algorithm of heart rate extraction for face compressed images is quite novel. The invention adopts unique singular spectrum analysis, double relationship screening, frequency mask screening and other processing steps, and can extract the human heart rate signal from the compressed video data, which is an unprecedented technical means. Therefore, the heart rate extraction algorithm of the face compressed image in this case is quite novel. [2] Wide range of applications. The invention can be applied to all occasions requiring image compression. For example, in telemedicine, the patient's video data is compressed and transmitted to the hospital for further analysis. In the mobile phone application, the video taken by the user is transmitted to the cloud via a wireless network for heart rate measurement and analysis. With the help of the present invention, the technology can be applied to the fields of telemedicine, home care or fitness training, especially the technology of home care in the field of telemedicine has been improved. Therefore, the application range is wide. [3] The single-channel signal separation method greatly reduces the amount of video image transmission. The invention adopts a single-channel signal separation method to extract physiological signals, and then selects the channel (G channel) that is least affected by the compression algorithm for processing. That is, only the green values of the complex pixels of X times Y in the face area are captured and averaged, which can greatly reduce the amount of video image transmission. Therefore, the single-channel signal separation method greatly reduces the amount of video image transmission. The above is only a detailed description of the present invention through the preferred embodiment. Any simple modifications and changes made to this embodiment will not deviate from the spirit and scope of the present invention.

10‧‧‧Video installation 20‧‧‧Network connection device S1‧‧‧First overlapping cutting step S2‧‧‧ First processing steps S21‧‧‧Pretreatment steps S22‧‧‧band pass filter processing steps S23‧‧‧The first step of singular spectrum analysis S24‧‧‧Double screening steps S25‧‧‧Reconstruction steps S3‧‧‧The first step of overlapping addition S4‧‧‧Second overlapping cutting step S5‧‧‧second processing steps S51‧‧‧Frequency mask construction steps S52‧‧‧ Second singular spectrum analysis step S53‧‧‧Frequency mask screening steps S6‧‧‧The second overlapping and adding step M‧‧‧ face compression video M1‧‧‧ video T1, T2‧‧‧ length of time T3‧‧‧ long step K‧‧‧ Original image P‧‧‧Face area G0(t)‧‧‧ original signal G1(t)‧‧‧First signal G1m‧‧‧The first subsequence G1f‧‧‧First main frequency G2(t)‧‧‧Second signal G22‧‧‧Second signal after overlapping addition G3(t)‧‧‧third signal G3j‧‧‧Center frequency G3ju‧‧‧ Pass the upper limit frequency G3jd‧‧‧pass the lower limit frequency G3m‧‧‧second subsequence G3f‧‧‧second main frequency S34‧‧‧ frequency mask screening steps G4(t)‧‧‧Signal 4 G44‧‧‧Fourth signal after overlapping addition L1, LA‧‧‧ First curve L2, LB‧‧‧Second curve L3, LC‧‧‧third curve L4, LD‧‧‧ fourth curve

Figure 1 is a flowchart of the calculation method of the present invention. Figure 2 is a schematic diagram of the present invention. Figure 3 is a schematic diagram of the first overlapping cutting process of the present invention. Figure 4 is a small video of the present invention. Image schematic diagrams 5A and 5B are schematic diagrams of tracking the face area of the present invention respectively. FIG. 6 is a schematic diagram of the original signal of the present invention. FIGS. 7 and 8 are diagrams of the original signal of the present invention. Fig. 9 is the schematic diagram of the first signal of the present invention. Fig. 10 is the schematic diagram of the first singular spectrum analysis step of the present invention. Fig. 11 is the second signal of the present invention. Fig. 12 is a schematic diagram of the first overlap-add process of the present invention. Fig. 13 is a schematic diagram of the second overlap-cutting process of the present invention. Fig. 14 is a schematic diagram of the frequency mask construction process of the present invention. Fig. 15 Schematic diagram of the second singular spectrum analysis process of the present invention. FIG. 16 is a schematic diagram of the second overlapping addition process of the present invention. FIG. 17A is a schematic diagram of a static film of the present invention after performing a band-pass filtering process. FIG. 17B Figure 17A is the schematic diagram after the first singular spectrum analysis step and the double relationship screening processing step. Figure 17C is the schematic diagram of the second singular spectrum analysis and frequency mask screening step in Figure 17B. Figure 17D is the 17A, Comparison of Figure 17B and Figure 17C Figure 18A is a schematic diagram of a dynamic film of the present invention after performing a band-pass filter processing step. Figure 18B is the first singular spectrum analysis step of Figure 18A and the double relationship filtering processing step Schematic diagram 18C is a schematic diagram of the second singular spectrum analysis and frequency mask screening steps of FIG. 18B. FIG. 18D is a comparison diagram of the present invention after the final heart rate signal is extracted through 18A, 18B and 18C.

S1‧‧‧First overlapping cutting step

S2‧‧‧ First processing steps

S21‧‧‧Pretreatment steps

S22‧‧‧band pass filter processing steps

S23‧‧‧The first step of singular spectrum analysis

S24‧‧‧Double screening steps

S25‧‧‧Reconstruction steps

S3‧‧‧The first step of overlapping addition

S4‧‧‧Second overlapping cutting step

S5‧‧‧second processing steps

S51‧‧‧Frequency mask construction steps

S52‧‧‧ Second singular spectrum analysis step

S6‧‧‧The second overlapping and adding step

Claims (3)

A method for extracting and calculating the heart rate of a compressed image of a face includes the following steps: The first overlapping cutting step: obtain a face compressed video after video compression processing, overlap and cut the face compressed video into a plurality of small videos; the method of overlapping cutting is defined as the face compressed video At the beginning, capture the first video of the small segment, and then capture another video of the small segment every other step for a long time, repeat the aforementioned actions until the end of the face compression video, and then obtain a plurality of the small video; two. The first processing step is to perform the following steps one by one for each small video: [a] Pre-processing step: each small video contains N frame original images; face area tracking is performed for each frame original image. Obtain a face area with X times Y pixels, and average the green values of the X times Y complex pixels to obtain a scalar quantity, which is defined as the average green value of a single frame; and so on , The average green value of N frame can be obtained, and an original signal G0(t) is further obtained from the average green value of N frame, where t=1 to N; [b] Bandpass filter processing step: for the aforementioned original signal G0( t) Perform band-pass filtering and select a signal with a frequency between 0.8 Hz and 2.0 Hz to obtain a first signal G1(t), where t=1 to N; [c] The first singular spectrum analysis step: The first signal G1(t) is decomposed into a plurality of first sub-sequences, and a fast Fourier transform is performed on all the first sub-sequences to obtain a plurality of frequency spectra of the first sub-sequences, and then each of the first sub-sequences The frequency with the largest amplitude in the frequency spectrum is taken as its first main frequency; [d] Step of screening for double relationship: comparing the aforementioned first subsequences in pairs, if the two first main frequencies are doubled after comparison Relationship, the two first main frequencies are reserved, otherwise the two first main frequencies are discarded; if there is no one that satisfies this double relationship after pairwise comparison, all first subsequences are retained; and at least Two first reserved subsequences; [e] Reconstruction step: Reconstruct all the first reserved subsequences of the previous step to obtain a second signal G2(t), where t=1 to N, the first The horizontal axis of the second signal G2(t) is time, and the vertical axis is the intensity of the green value. The second signal G2(t) has the length of time; 3. The first overlapping and adding step: the plurality of second signals G2(t) obtained after the first processing step are processed by the existing overlapping and adding method of the raised cosine window method to obtain a second signal after overlapping and adding , Which has the length of time; the overlap-add technique is defined as between two adjacent second signals G2(t), which are overlapped and added in this step for a long time; 4. The second overlapping cutting step: obtaining the second signal after the overlapping addition has the time length, and overlappingly cutting the second signal after the overlapping addition into a plurality of small segment signals, each small segment signal having the time length ; The way of overlapping cutting is defined as the beginning of the second signal after the overlap is added, to capture a small segment of the signal, then every other step for a long time, and then capture another small segment of the signal, repeat the aforementioned Action until the second signal ends after the above-mentioned overlap addition, and then a plurality of small segment signals can be taken, and each small segment signal is defined as the third signal G3(t), where t=1 to N; 5. Second processing step: Perform the following steps one by one for each third signal G3(t): [f] Frequency mask construction step: Use the frequency with the largest amplitude of the third signal G3(t) as its center frequency , And using the center frequency as the center, set a passing upper limit frequency and a passing lower limit frequency to obtain a frequency mask range; [g] Second singular spectrum analysis step: decompose the third signal G3(t) into A plurality of second subsequences, and perform a fast Fourier transform on all second subsequences to obtain the spectrum of the plurality of second subsequences, and then use the frequency with the largest amplitude in the spectrum of each second subsequence as Its second main frequency; [h] frequency mask screening step: only the second subsequence with the second main frequency within the frequency mask range is retained, then at least one second reserved subsequence can be obtained by screening, It becomes a fourth signal G4(t), where t=1 to N, the horizontal axis of the fourth signal G4(t) is time, and the vertical axis is the intensity of the green value; if any second retainer is not obtained by screening Sequence, the fourth signal G4(t) is directly equal to the third signal G3(t); VI. The second overlapping addition step: the plurality of fourth signals G4(t) obtained after the second processing step are processed by the existing overlapping cosine window method of overlapping addition technique to obtain an overlapping addition The four signals, which have this length of time, are the final heart rate signals. The method for extracting the heart rate of a face compressed image as described in item 1 of the patent application scope, wherein in the step of obtaining the original image, two sets of video devices and a network connection device are provided, and the two sets of video devices pass through the Network connection device to achieve video contact. The method for extracting the heart rate of a face compressed image as described in item 1 of the patent application scope, wherein the compression method of the face compressed image is selected from one of x264, x265, vp8, and vp9.

Images