CN114818833A - Heart rate measuring method, device, electronic equipment, storage medium and program product - Google Patents

Abstract

The present disclosure relates to a heart rate measurement method, apparatus, electronic device, storage medium and program product. The method comprises the following steps: acquiring an image sequence corresponding to a target object, wherein the image sequence comprises a plurality of images; extracting a plurality of items of first characteristic information which are in one-to-one correspondence with the plurality of images respectively; processing the plurality of items of first characteristic information by adopting a time domain convolution network to obtain a plurality of items of second characteristic information which are in one-to-one correspondence with the plurality of images; and determining the heart rate value of the target object according to the plurality of items of second characteristic information.

Description

Heart rate measurement methods, devices, electronic equipment, storage media and program products

technical field

The present disclosure relates to the technical field of computer vision, and in particular, to a heart rate measurement method, apparatus, electronic device, storage medium and program product.

Background technique

Heart rate is one of the important indicators for the evaluation of human physiological activity and health status. At present, heart rate measurement technologies mainly include electrocardiogram (electrocardiogram, ECG) and wearable photoplethysmography (PPG). Both of these heart rate measurement technologies need to use contact methods for signal measurement, and the usage scenarios are limited, especially when it is more difficult to detect the daily health status of infants.

SUMMARY OF THE INVENTION

The present disclosure provides a technical solution for measuring heart rate.

According to an aspect of the present disclosure, there is provided a heart rate measurement method, comprising:

acquiring an image sequence corresponding to the target object, wherein the image sequence includes multiple images;

respectively extracting multiple items of first feature information corresponding to the multiple images one-to-one;

Using a temporal convolution network to process the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one;

The heart rate value of the target object is determined according to the plurality of pieces of second feature information.

By acquiring an image sequence corresponding to the target object, wherein the image sequence includes multiple images, multiple items of first feature information corresponding to the multiple images are extracted respectively, and a time domain convolution network is used to analyze the multiple items. The first feature information is processed to obtain multiple pieces of second feature information corresponding to the multiple images one-to-one, and the heart rate value of the target object is determined according to the multiple pieces of second feature information. A high-level time-domain convolutional network replaces the traditional time-series network structure (such as LSTM), and a lightweight time-domain convolutional network is used to model the time-series relationship to extract time-series information, which can reduce the dependence of model deployment on operators. The hardware threshold for model deployment is lowered, and the adaptability of the model to hardware is enhanced, that is, it can be deployed on hardware with lower cost, fewer neural network operators supported, and poor computing performance.

In a possible implementation manner, the determining the heart rate value of the target object according to the multiple pieces of second feature information includes:

predicting a remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of second feature information;

According to the rPPG signal, the heart rate value of the target object is determined.

In this implementation, by predicting the remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of second feature information, and determining the heart rate value of the target object according to the rPPG signal, it is possible to Determine the target subject's heart rate value more accurately.

In a possible implementation manner, the predicting the remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of second feature information includes:

Ordinal regression is performed on the multiple items of second feature information to obtain a remote photoplethysmography rPPG signal corresponding to the image sequence.

In this implementation, the remote photoplethysmography rPPG signal corresponding to the image sequence is obtained by performing ordinal regression on the multiple pieces of second feature information, and the ordinal regression method that is more suitable for predicting continuous values can be obtained. More accurate rPPG signal.

In a possible implementation manner, the determining the heart rate value of the target object according to the rPPG signal includes:

Wavelet analysis is performed on the rPPG signal to determine the heart rate value of the target object.

In this implementation, the heart rate value of the target object is determined by performing wavelet analysis on the rPPG signal, so that the calculation compatibility for the rPPG signal with a low signal-to-noise ratio is better, that is, even in the rPPG signal In the case of low signal-to-noise ratio, a more accurate heart rate value can also be determined.

In a possible implementation manner, the performing wavelet analysis on the rPPG signal to determine the heart rate value of the target object includes:

determining at least two widths of the wavelets of the wavelet;

performing wavelet transformation on the rPPG signal according to the at least two widths to obtain at least two wavelet responses corresponding to the at least two widths one-to-one;

determining the response intensity and the heart rate value corresponding to the at least two wavelet responses;

The heart rate value of the target object is determined according to the response intensity and the heart rate value corresponding to the at least two wavelet responses.

Since the signal-to-noise ratio of the rPPG signal is usually low, and the heart rate varies greatly with time, in this implementation manner, by determining at least two widths of the wavelet of the wavelet, according to the at least two widths, the The rPPG signal is subjected to wavelet transformation to obtain at least two wavelet responses corresponding to the at least two widths one-to-one, and the response intensity and heart rate value corresponding to the at least two wavelet responses are determined, and according to the at least two wavelet responses The corresponding response intensity and heart rate value are used to determine the heart rate value of the target object, thereby helping to more accurately determine the heart rate value of the target object.

In a possible implementation, the determining at least two widths of the wavelet of the wavelet include:

At least two widths of the wavelet of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence.

In this implementation manner, at least two widths of wavelets of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence, thereby helping to determine an appropriate width , thereby improving the accuracy of heart rate determination.

In a possible implementation manner, the determining the heart rate value of the target object according to the response intensity and the heart rate value corresponding to the at least two wavelet responses includes:

The heart rate value corresponding to the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range is determined as the heart rate value of the target object.

Since the response intensity of the noise may be greater than the response intensity of the heart rate in the case of a low SNR In response to the corresponding heart rate value, the heart rate value of the target object is determined, thereby improving the robustness and accuracy of the heart rate measurement.

In one possible implementation,

The extracting, respectively, multiple items of first feature information corresponding to the multiple images includes: sequentially extracting the first feature information corresponding to the images in the image sequence, and buffering the first feature information in a pre-set Set the length in the first-in-first-out queue;

The use of a time-domain convolution network to process the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one includes: using a time-domain convolution network for the advanced feature information. The multiple items of first feature information in the first-out queue are processed to obtain multiple items of second feature information corresponding to the multiple items of first feature information one-to-one.

In this implementation, by sequentially extracting the first feature information corresponding to the images in the image sequence, the first feature information is cached in a FIFO queue of preset length, and a time domain convolutional network is used to The multiple items of first feature information in the FIFO queue are processed to obtain multiple items of second feature information that correspond one-to-one with the multiple items of first feature information, thereby reducing redundant computation and saving hardware equipment costs. computing power.

According to an aspect of the present disclosure, there is provided a heart rate measurement method, comprising:

acquiring an image sequence corresponding to the target object, wherein the image sequence includes multiple images;

respectively extracting multiple items of first feature information corresponding to the multiple images one-to-one;

predicting a remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of first feature information;

Wavelet analysis is performed on the rPPG signal to determine the heart rate value of the target object.

In this embodiment of the present disclosure, by acquiring an image sequence corresponding to a target object, wherein the image sequence includes multiple images, multiple items of first feature information corresponding to the multiple images are extracted respectively, and according to the multiple images Items of first feature information, predict the rPPG signal corresponding to the image sequence, and perform wavelet analysis on the rPPG signal to determine the heart rate value of the target object, which can improve the calculation of the rPPG signal with low signal-to-noise ratio. Compatibility, that is, more accurate heart rate values can be determined even when the signal-to-noise ratio of the rPPG signal is low.

In a possible implementation manner, the performing wavelet analysis on the rPPG signal to determine the heart rate value of the target object includes:

determining at least two widths of the wavelets of the wavelet;

performing wavelet transformation on the rPPG signal according to the at least two widths to obtain at least two wavelet responses corresponding to the at least two widths one-to-one;

determining the response intensity and the heart rate value corresponding to the at least two wavelet responses;

The heart rate value of the target object is determined according to the response intensity and the heart rate value corresponding to the at least two wavelet responses.

Since the signal-to-noise ratio of the rPPG signal is usually low, and the heart rate varies greatly with time, in this implementation manner, by determining at least two widths of the wavelet of the wavelet, according to the at least two widths, the The rPPG signal is subjected to wavelet transformation to obtain at least two wavelet responses corresponding to the at least two widths one-to-one, and the response intensity and heart rate value corresponding to the at least two wavelet responses are determined, and according to the at least two wavelet responses The corresponding response intensity and heart rate value are used to determine the heart rate value of the target object, thereby helping to more accurately determine the heart rate value of the target object.

In a possible implementation, the determining at least two widths of the wavelet of the wavelet include:

At least two widths of the wavelet of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence.

In this implementation manner, at least two widths of wavelets of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence, thereby helping to determine an appropriate width , thereby improving the accuracy of heart rate determination.

In a possible implementation manner, the determining the heart rate value of the target object according to the response intensity and the heart rate value corresponding to the at least two wavelet responses includes:

The heart rate value corresponding to the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range is determined as the heart rate value of the target object.

Since the response intensity of the noise may be greater than the response intensity of the heart rate in the case of a low SNR In response to the corresponding heart rate value, the heart rate value of the target object is determined, thereby improving the robustness and accuracy of the heart rate measurement.

In a possible implementation manner, the predicting the remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of first feature information includes:

processing the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one;

According to the multiple pieces of second feature information, the rPPG signal corresponding to the image sequence is predicted.

In this implementation manner, by processing the multiple items of first feature information, multiple items of second feature information corresponding to the multiple images are obtained, and according to the multiple items of second feature information, the multiple items of second feature information are predicted. The rPPG signal corresponding to the image sequence can be obtained, so that a more accurate rPPG signal can be obtained.

In a possible implementation manner, the predicting the rPPG signal corresponding to the image sequence according to the multiple pieces of second feature information includes:

Ordinal regression is performed on the multiple items of second feature information to obtain a remote photoplethysmography rPPG signal corresponding to the image sequence.

In this implementation, the remote photoplethysmography rPPG signal corresponding to the image sequence is obtained by performing ordinal regression on the multiple pieces of second feature information, and the ordinal regression method that is more suitable for predicting continuous values can be obtained. More accurate rPPG signal.

According to an aspect of the present disclosure, there is provided a heart rate measurement device, comprising:

an acquisition module, configured to acquire an image sequence corresponding to the target object, wherein the image sequence includes multiple images;

an extraction module, configured to extract a plurality of pieces of first feature information corresponding to the plurality of images one-to-one;

a processing module, configured to process the multiple items of first feature information by using a time-domain convolutional network to obtain multiple items of second feature information corresponding to the multiple images one-to-one;

The first determination module is configured to determine the heart rate value of the target object according to the multiple pieces of second characteristic information.

In a possible implementation manner, the first determining module is used for:

predicting a remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of second feature information;

According to the rPPG signal, the heart rate value of the target object is determined.

In a possible implementation manner, the first determining module is used for:

Ordinal regression is performed on the multiple items of second feature information to obtain a remote photoplethysmography rPPG signal corresponding to the image sequence.

In a possible implementation manner, the first determining module is used for:

Wavelet analysis is performed on the rPPG signal to determine the heart rate value of the target object.

In a possible implementation manner, the first determining module is used for:

determining at least two widths of the wavelets of the wavelet;

performing wavelet transformation on the rPPG signal according to the at least two widths to obtain at least two wavelet responses corresponding to the at least two widths one-to-one;

determining the response intensity and the heart rate value corresponding to the at least two wavelet responses;

The heart rate value of the target object is determined according to the response intensity and the heart rate value corresponding to the at least two wavelet responses.

In a possible implementation manner, the first determining module is used for:

At least two widths of the wavelet of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence.

In a possible implementation manner, the first determining module is used for:

The heart rate value corresponding to the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range is determined as the heart rate value of the target object.

In one possible implementation,

The extraction module is used for: sequentially extracting the first feature information corresponding to the images in the image sequence, and buffering the first feature information in a FIFO queue of preset length;

The processing module is used for: using a time-domain convolutional network to process multiple items of first feature information in the first-in, first-out queue to obtain multiple items of second feature information one-to-one corresponding to the multiple items of first feature information information.

According to an aspect of the present disclosure, there is provided a heart rate measurement device, comprising:

an acquisition module, configured to acquire an image sequence corresponding to the target object, wherein the image sequence includes multiple images;

an extraction module, configured to extract a plurality of pieces of first feature information corresponding to the plurality of images one-to-one;

a prediction module, configured to predict the remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of first feature information;

The second determination module is configured to perform wavelet analysis on the rPPG signal to determine the heart rate value of the target object.

In a possible implementation manner, the second determining module is used for:

determining at least two widths of the wavelets of the wavelet;

performing wavelet transformation on the rPPG signal according to the at least two widths to obtain at least two wavelet responses corresponding to the at least two widths one-to-one;

determining the response intensity and the heart rate value corresponding to the at least two wavelet responses;

The heart rate value of the target object is determined according to the response intensity and the heart rate value corresponding to the at least two wavelet responses.

In a possible implementation manner, the second determining module is used for:

At least two widths of the wavelet of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence.

In a possible implementation manner, the second determining module is used for:

The heart rate value corresponding to the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range is determined as the heart rate value of the target object.

In a possible implementation, the prediction module is used to:

processing the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one;

According to the multiple pieces of second feature information, the rPPG signal corresponding to the image sequence is predicted.

In a possible implementation, the prediction module is used to:

Ordinal regression is performed on the multiple items of second feature information to obtain a remote photoplethysmography rPPG signal corresponding to the image sequence.

According to an aspect of the present disclosure, there is provided an electronic device comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the memory storage executable instructions to perform the above method.

According to an aspect of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the above method when executed by a processor.

According to an aspect of the present disclosure, there is provided a computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, when the computer-readable code is stored in an electronic device When running, the processor in the electronic device executes the above method.

In the embodiment of the present disclosure, by acquiring an image sequence corresponding to a target object, wherein the image sequence includes multiple images, a plurality of pieces of first feature information corresponding to the multiple images are extracted respectively, and a temporal volume The product network processes the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one, and determines the heart rate of the target object according to the multiple items of second feature information Therefore, a lightweight time-domain convolutional network is used to replace the traditional time-series network structure (such as LSTM), and the time-series relationship is modeled through a lightweight time-domain convolutional network to extract time-series information, which can reduce the model Deployment relies on operators, lowers the hardware threshold for model deployment, and enhances the adaptability of models to hardware, that is, it can be deployed on lower-cost hardware that supports fewer neural network operators and has poor computing performance superior.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the technical solutions of the present disclosure.

FIG. 1 shows a flowchart of a heart rate measurement method provided by an embodiment of the present disclosure.

FIG. 2 shows a schematic diagram of extracting first feature information by a single-frame encoding module in a heart rate measurement method provided by an embodiment of the present disclosure.

FIG. 3 shows a schematic diagram of the input and output of a time-domain convolutional network module in the heart rate measurement method provided by an embodiment of the present disclosure.

FIG. 4 shows a schematic diagram of performing ordinal regression on the second feature information in the heart rate measurement method provided by an embodiment of the present disclosure.

FIG. 5 shows a schematic diagram of a Ricker wavelet in the heart rate measurement method provided by an embodiment of the present disclosure.

FIG. 6 shows a schematic diagram of a wavelet response spectrum in a heart rate measurement method provided by an embodiment of the present disclosure.

FIG. 7 shows a schematic diagram of a curve corresponding to an rPPG signal and a time domain response curve corresponding to a wavelet response in the heart rate measurement method provided by an embodiment of the present disclosure.

FIG. 8 shows a schematic diagram of a heart rate measurement model in a heart rate measurement method provided by an embodiment of the present disclosure.

FIG. 9 shows another flowchart of a heart rate measurement method provided by an embodiment of the present disclosure.

FIG. 10 shows a block diagram of a heart rate measurement apparatus provided by an embodiment of the present disclosure.

FIG. 11 shows another block diagram of the heart rate measurement device provided by the embodiment of the present disclosure.

FIG. 12 shows a block diagram of an electronic device 1900 provided by an embodiment of the present disclosure.

Detailed ways

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

In addition, in order to better illustrate the present disclosure, numerous specific details are set forth in the following detailed description. It will be understood by those skilled in the art that the present disclosure may be practiced without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present disclosure.

In the related art, a non-contact heart rate measurement scheme based on RGB (Red-Green-Blue, red-green-blue) video stream is proposed. In the related art, the rPPG (remote photoplethysmography, remote photoplethysmography) signal is extracted by using the time series information of the change of the absorption rate of natural light by the content of red blood cells of the face, and then the heart rate is calculated. Existing solutions usually use LSTM (Long Short Term Memory) network or 3D-CNN (3Dimensions-Convolutional Neural Network, three-dimensional convolutional neural network) to model time series information, which has high performance requirements for the deployment platform. As a result, it cannot be deployed on many small hardware devices.

Embodiments of the present disclosure provide a heart rate measurement method, device, electronic device, storage medium, and program product, by acquiring an image sequence corresponding to a target object, wherein the image sequence includes multiple images, and extracting images corresponding to the multiple images respectively The multiple items of first feature information corresponding to the images one-to-one are processed by using the time domain convolution network to obtain multiple items of second feature information corresponding to the multiple images one-to-one. The multiple pieces of second feature information are used to determine the heart rate value of the target object, whereby a lightweight time-domain convolutional network is used to replace the traditional time-series network structure (such as LSTM), and a lightweight time-domain convolutional network is used The network models the timing relationship to extract timing information, which can reduce the dependence of model deployment on operators, reduce the hardware threshold for model deployment, and enhance the adaptability of the model to hardware. On hardware that supports fewer neural network operators and has poor computing performance.

The heart rate measurement method provided by the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

FIG. 1 shows a flowchart of a heart rate measurement method provided by an embodiment of the present disclosure. In a possible implementation manner, the execution subject of the heart rate measurement method may be a heart rate measurement device, for example, the heart rate measurement method may be executed by a terminal device or a server or other electronic devices. The terminal device may be User Equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, Personal Digital Assistant (PDA), handheld device, computing device, vehicle-mounted device, or wearable device. equipment, etc. In some possible implementations, the heart rate measurement method may be implemented by a processor invoking computer-readable instructions stored in a memory. As shown in FIG. 1 , the heart rate measurement method includes steps S11 to S14.

In step S11, an image sequence corresponding to the target object is acquired, wherein the image sequence includes multiple images.

In step S12, multiple items of first feature information corresponding to the multiple images one-to-one are extracted respectively.

In step S13, a time domain convolution network is used to process the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one.

In step S14, the heart rate value of the target object is determined according to the multiple pieces of second feature information.

In an embodiment of the present disclosure, the target object may represent an object of heart rate measurement. The target object can be a person or an animal.

In the embodiment of the present disclosure, an image may be extracted from a video stream corresponding to the target object to obtain an image sequence corresponding to the target object. The video stream corresponding to the target object may be a video stream captured on the face of the target object, that is, the image content of the video frame images in the video stream corresponding to the target object may include at least part of the face area of the target object.

In a possible implementation manner, the frame rate of the image sequence corresponding to the target object may be equal to the frame rate of the video stream corresponding to the target object. For example, the frame rate of the video stream corresponding to the target object may be 20FPS (Frames PerSecond, frames per second) to 25FPS. In this implementation manner, each video frame image in the video stream corresponding to the target object can be extracted, and based on the each video frame image, an image sequence corresponding to the target object can be obtained.

As an example of this implementation, for any video frame image in the video stream corresponding to the target object, an image of a preset size can be intercepted from the video frame image according to the position of the face region in the video frame image, as An image in the sequence of images corresponding to the target object. For example, the preset size may be 64×64 or 128×128, etc., which is not limited here. In this example, the image content of the images in the image sequence corresponding to the target object includes at least part of the face area of the target object.

As another example of this implementation, the video frame image in the video stream corresponding to the target object may be directly used as the image in the image sequence corresponding to the target object. In this example, the size of the image in the image sequence corresponding to the target object is equal to the size of the video frame image in the video stream corresponding to the target object.

In another possible implementation manner, the frame rate of the image sequence corresponding to the target object may be smaller than the frame rate of the video stream corresponding to the target object. In this implementation, a video frame image can be extracted every K frames from the video stream corresponding to the target object, and an image sequence corresponding to the target object can be obtained based on the extracted video frame images, where K is greater than or An integer equal to 1.

As an example of this implementation, for any extracted video frame image, an image of a preset size can be intercepted from the video frame image according to the position of the face region in the video frame image, as the image sequence corresponding to the target object an image in . In this example, the image content of the images in the image sequence corresponding to the target object includes at least part of the face area of the target object.

As another example of this implementation, each of the extracted video frame images may be directly used as images in the image sequence corresponding to the target object. In this example, the size of the image in the image sequence corresponding to the target object is equal to the size of the video frame image in the video stream corresponding to the target object.

In a possible implementation manner, the images in the image sequence corresponding to the target object may be RGB images. In other possible implementation manners, the image in the image sequence corresponding to the target object may also be an infrared image, etc., which is not limited in this embodiment of the present disclosure.

In this embodiment of the present disclosure, the image sequence corresponding to the target object may include N images, where N is an integer greater than or equal to 3. For example, N can be equal to 120. Of course, those skilled in the art can flexibly set the size of N according to actual application scenario requirements, which is not limited here.

The embodiments of the present disclosure determine the heart rate value of the target object based on the image sequence corresponding to the target object, thereby enabling non-contact heart rate measurement, thereby improving the convenience of heart rate measurement.

In this embodiment of the present disclosure, the first feature information may represent feature information corresponding to images in the image sequence. The first feature information is in one-to-one correspondence with images in the image sequence.

In a possible implementation manner, the first feature information corresponding to the images in the image sequence may be extracted through a single-frame encoding module. FIG. 2 shows a schematic diagram of extracting first feature information by a single-frame encoding module in a heart rate measurement method provided by an embodiment of the present disclosure. The input of the single-frame encoding module may be a three-channel image with a size of 64×64, and the output of the single-frame encoding module may be first feature information, and the first feature information may be a 120-dimensional feature vector.

As an example of this implementation, the single-frame encoding module may include 5 convolution modules, and each convolution module may include a 2D (2Dimensions, two-dimensional) convolution layer with a convolution kernel size of 3×3, a 2D batch normalization (BatchNormalization, BN) layer, 2D average pooling layer and Relu activation function layer.

Of course, the single-frame encoding module may also adopt other structures, which are not limited here.

In the embodiment of the present disclosure, the multiple items of first feature information are processed through a Temporal Convolutional Network (TCN) to obtain multiple items of second feature information corresponding to the multiple images one-to-one. Among them, the time-domain convolutional network is a lightweight deep neural network, so it can reduce the performance requirements of the hardware. In addition, the time-domain convolutional network uses dilated convolution to expand the receptive field. Compared with ordinary convolution operations, dilated convolution is more suitable for extracting long-term related features.

In this embodiment of the present disclosure, the multiple items of first feature information may be processed through an M-layer temporal convolutional network to obtain multiple items of second feature information corresponding to the multiple images one-to-one, where M is Integer greater than or equal to 1. The M-layer time-domain convolutional network can form a time-domain convolutional network module. For example, M can be equal to 4, and each layer of the temporal convolutional network can include a 2D dilated convolutional layer, an adaptive cropping layer, a Relu activation function layer, a dropout layer, a 2D dilated convolutional layer, an adaptive cropping layer, and a Relu layer. Activation function layer and dropout layer. Among them, the size of the convolution kernel of the 2D dilated convolution layer can be 1 × 2, and the jumper holes can be set.

FIG. 3 shows a schematic diagram of the input and output of a time-domain convolutional network module in the heart rate measurement method provided by an embodiment of the present disclosure. As shown in FIG. 3 , the input of the time domain convolutional network module may be N items of 120-dimensional first feature information, and the output may be N items of 80-dimensional second feature information.

In a possible implementation manner, the extracting a plurality of pieces of first feature information corresponding to the plurality of images respectively includes: sequentially extracting the first feature information corresponding to the images in the image sequence, and extracting the first feature information corresponding to the images in the image sequence The first feature information is cached in a FIFO queue with a preset length; the time domain convolution network is used to process the multiple items of first feature information to obtain multiple items corresponding to the multiple images one-to-one. Items of second feature information, including: using a time-domain convolution network to process multiple items of first feature information in the FIFO queue to obtain multiple items of second feature information one-to-one corresponding to the multiple items of first feature information characteristic information.

In the related art, the encoding module performs feature encoding on all images in the sliding window each time. When the step size of the sliding window is smaller than the size of the sliding window, the encoding module will repeatedly encode part of the images in the sliding window, resulting in redundant computation. In the above implementation manner, the heart rate measurement model may only include a single-frame encoding module, and the single-frame encoding module may perform feature extraction on one image at a time, and cache the extracted feature information.

In this implementation manner, the first feature information corresponding to the images in the image sequence may be sequentially extracted according to the order of the images in the image sequence, and the extracted first feature information may be cached in a first-in, first-out format with a preset length. in the queue. The first-in, first-out queue may represent a first-in, first-out queue for buffering the first feature information. In an example, the length of the FIFO queue may be N, that is, the FIFO queue may buffer the first feature information corresponding to N images. For example, N can be equal to 120.

In this implementation manner, the time-domain convolutional network can process all the first feature information in the FIFO queue to obtain the first feature information corresponding to each item of first feature information in the FIFO queue one-to-one. Two feature information.

In this implementation, by sequentially extracting the first feature information corresponding to the images in the image sequence, the first feature information is cached in a FIFO queue of preset length, and a time domain convolutional network is used to The multiple items of first feature information in the FIFO queue are processed to obtain multiple items of second feature information that correspond one-to-one with the multiple items of first feature information, thereby reducing redundant computation and saving hardware equipment costs. computing power.

In another possible implementation, feature extraction may be performed on at least two images in parallel. In this implementation, the heart rate measurement model may include at least two single-frame encoding modules.

In a possible implementation manner, the determining the heart rate value of the target object according to the multiple pieces of second feature information includes: predicting the distance corresponding to the image sequence according to the multiple pieces of second feature information Photoplethysmography rPPG signal; according to the rPPG signal, determine the heart rate value of the target object. In this implementation, by predicting the remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of second feature information, and determining the heart rate value of the target object according to the rPPG signal, it is possible to Determine the target subject's heart rate value more accurately.

As an example of this implementation, the predicting the remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple items of second feature information includes: performing ordinal regression on the multiple items of second feature information to obtain The remote photoplethysmography rPPG signal corresponding to the image sequence. In this example, ordinal regression may be performed on the multiple items of second feature information, respectively, to obtain multiple items of rPPG values that correspond one-to-one with the multiple items of second feature information, and may be composed of multiple items of rPPG values according to the multiple items of second feature information The rPPG signal corresponding to the image sequence.

FIG. 4 shows a schematic diagram of performing ordinal regression on the second feature information in the heart rate measurement method provided by an embodiment of the present disclosure. In the example shown in FIG. 4 , the image sequence includes 120 frames of images, and among the 120 items of second feature information corresponding to the 120 frames of images, each item of second feature information is an 80-dimensional feature vector. For any item of second feature information in the 120 items of second feature information, 80 dimensions corresponding to the second feature information can be combined in pairs to form 40 binary classifiers. Among them, the values of the two dimensions corresponding to any two classifiers can be recorded as A and B. If A is greater than B, the result of the second classifier is 1, and if A is less than or equal to B, the result of the two classifiers is 0. The rPPG value corresponding to the second feature information can be obtained by adding the results of the 40 binary classifiers. The rPPG value corresponding to the second feature information is greater than or equal to 0 and less than or equal to 40, and the rPPG value corresponding to the second feature information is an integer value. As shown in Figure 4, for the first item of second feature information, the 80 dimensions corresponding to the second feature information can be combined in pairs to form 40 binary classifiers, each of which represents a component of the rPPG value . The values of the two dimensions corresponding to any two-classifier can be recorded as A and B. If A is greater than B, the result of the two-classifier is 1, and if A is less than or equal to B, the result of the two-classifier is 0 . The rPPG value corresponding to the first item of the second feature information can be obtained by adding the results of the 40 binary classifiers. Ordinal regression is performed on the 120 items of second feature information respectively, and 120 items of rPPG values can be obtained, and the 120 items of rPPG constitute the rPPG signal corresponding to the image sequence.

In this example, the long-range photoplethysmography rPPG signal corresponding to the image sequence is obtained by performing ordinal regression on the multiple items of second feature information, so that an ordinal regression method that is more suitable for predicting continuous values can be used to obtain better Accurate rPPG signal.

Of course, other regression methods may also be used to regress the multiple pieces of second feature information, which are not limited here.

As an example of this implementation, the determining the heart rate value of the target object according to the rPPG signal includes: performing wavelet analysis on the rPPG signal to determine the heart rate value of the target object.

rPPG signals predicted by deep learning algorithms often have strong noise. In related technologies, the peaks and the peak spacing of the signal are usually detected to calculate the heart rate, or the Fast Fourier Transform (FFT) is used to extract the main frequency to calculate the heart rate. These methods are often greatly affected by noise. Relatively low rPPG signals are prone to larger errors. In the above example, the heart rate value of the target object is determined by performing wavelet analysis on the rPPG signal, so that the calculation compatibility for the rPPG signal with a low signal-to-noise ratio is better, that is, even in the signal-to-noise ratio of the rPPG signal. In the case of low noise ratio, a more accurate heart rate value can also be determined.

In an example, the performing wavelet analysis on the rPPG signal to determine the heart rate value of the target object includes: determining at least two widths of wavelets of the wavelet; according to the at least two widths, analyzing the rPPG Wavelet transform is performed on the signal to obtain at least two wavelet responses corresponding to the at least two widths one-to-one; the response intensity and the heart rate value corresponding to the at least two wavelet responses are determined; according to the responses corresponding to the at least two wavelet responses Intensity and heart rate value, determine the heart rate value of the target object.

In this example, the Ricker wavelet function may be used, and other wavelet functions may also be used, which is not limited herein. FIG. 5 shows a schematic diagram of a Ricker wavelet in the heart rate measurement method provided by an embodiment of the present disclosure. Among them, 2σ represents the width of the wavelet of the Ricker wavelet, and σ represents the half width of the wavelet of the Ricker wavelet. The Ricker wavelet function y(t,σ) can be expressed by Equation 1:

The rPPG signal can be denoted as s(t). Wavelet transform is performed on the rPPG signal to obtain the wavelet response I(t,σ), which can be expressed by Equation 2:

I(t,σ)=s(t)*y(t,σ) Formula 2,

Among them, * represents the convolution, that is, the wavelet response I(t,σ) is the convolution of the Ricker wavelet and the rPPG signal.

In this example, the wavelet responses corresponding to different widths may represent the intensities of waves with different widths in the curve corresponding to the rPPG signal. I(t=t ₀ , σ=σ ₀ ) represents the response intensity corresponding to the wavelet response whose half-width of the wavelet is σ ₀ at time t ₀ . At the peak position of the heartbeat, the response intensity of a wavelet of appropriate width will peak.

In this example, the response intensity corresponding to any one of the at least two wavelet responses may represent the overall response intensity corresponding to the wavelet response. The response intensity M(σ ₀ ) corresponding to the wavelet response whose half-width of the wavelet is σ ₀ can be determined by using Equation 3:

FIG. 6 shows a schematic diagram of a wavelet response spectrum in a heart rate measurement method provided by an embodiment of the present disclosure. In Fig. 6, the abscissa is time, the ordinate is the response intensity, and the brightness is positively correlated with the response intensity. It can be seen from Figure 6 that, corresponding to different wavelet widths, the response spectrum presents two frequency responses: the high-frequency response corresponds to heart rate, and the low-frequency response corresponds to noise.

FIG. 7 shows a schematic diagram of a curve corresponding to an rPPG signal and a time domain response curve corresponding to a wavelet response in the heart rate measurement method provided by an embodiment of the present disclosure. It can be seen from FIG. 7 that the peak position of the time domain response curve 22 corresponding to the wavelet response matches the peak position of the main peak of the curve 21 corresponding to the rPPG signal, and the small peaks are effectively filtered out.

其中，K为大于1的整数。那么，可以计算得到子波的半宽度为σ₀的小波响应对应的响应频率为

即，子波的半宽度为σ₀的小波响应对应的心率值为

In an example, the peaks of the time-domain response curve corresponding to the wavelet response with the half-width of the wavelet σ ₀ appear in sequence from t ₁ to t _K , then the average period is

Wherein, K is an integer greater than 1. Then, it can be calculated that the response frequency corresponding to the wavelet response whose half-width of the wavelet is σ ₀ is

That is, the heart rate corresponding to the wavelet response whose half-width of the wavelet is σ ₀ is

Since the signal-to-noise ratio of the rPPG signal is generally low, and the heart rate varies greatly with time, in the above example, by determining at least two widths of the wavelet of the wavelet, according to the at least two widths, the Perform wavelet transformation on the rPPG signal to obtain at least two wavelet responses corresponding to the at least two widths one-to-one, determine the response intensity and heart rate value corresponding to the at least two wavelet responses, and determine the corresponding response intensity and heart rate according to the at least two wavelet responses. The response intensity and heart rate value of the target object are determined, and the heart rate value of the target object is determined, thereby helping to determine the heart rate value of the target object more accurately.

In an example, the determining at least two widths of the wavelet wavelet includes: determining at least two widths of the wavelet wavelet according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence. kind of width.

For example, the maximum value of the normal heart rate may be used as the maximum preset heart rate value; the minimum value of the normal heart rate may be used as the minimum preset heart rate value.

In one example, Equation 4 can be used to determine the half-width σ of the wavelet of the wavelet:

Wherein, f represents the frame rate of the image sequence, and the unit is Hz; h _max represents the maximum preset heart rate value, h _min represents the minimum preset heart rate value, and the units of h _max and h _min are times/min.

The half-width of the wavelet of the wavelet may be an integer value or a decimal value. In one example, the integer values satisfying Equation 4 can be respectively determined as the half widths of the wavelets of the wavelet. For example, f=20, h _min =60, h _max =100, then, σ satisfying Equation 4 includes 6, 7, 8, 9, and 10, so that five kinds of widths of wavelets of the wavelet can be determined.

In this example, at least two widths of wavelets of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence, thereby helping to determine an appropriate width, Thereby, the accuracy of heart rate determination can be improved.

In an example, the determining the heart rate value of the target object according to the response intensity and the heart rate value corresponding to the at least two wavelet responses includes: determining the heart rate value of the target object in the wavelet response whose heart rate value belongs to a preset heart rate value range, the response intensity The heart rate value corresponding to the largest wavelet response is determined as the heart rate value of the target object.

确定出心率值属于预设心率值范围的小波响应中、响应强度最大的小波响应，并可以将

确定为目标对象的心率值。For example, the preset heart rate value range may be [h _min , h _max ], then, according to h _min ≤f(σ)≤h _max , wavelet responses whose heart rate values fall within the preset heart rate value range can be screened. For example, the half-width of the wavelet of the wavelet response whose heart rate value belongs to the preset heart rate value range can be denoted as σ′, then, according to

Determine the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range, and can use

Determines the heart rate value of the target subject.

Since the response intensity of the noise may be greater than the response intensity of the heart rate in the case of a low SNR The corresponding heart rate value is determined as the heart rate value of the target object, thereby improving the robustness and accuracy of the heart rate measurement.

In other examples, the rPPG signal may also be processed by means of Fourier transform, etc., to obtain the heart rate value of the target object.

In another possible implementation manner, the determining the heart rate value of the target object according to the multiple pieces of second feature information includes: inputting the multiple pieces of second feature information into a pre-trained heart rate extraction model , and the heart rate value of the target object is determined via the heart rate extraction model.

In another possible implementation manner, the determining the heart rate value of the target object according to the multiple pieces of second feature information includes: processing the multiple pieces of second feature information through a pre-designed function, Obtain the heart rate value of the target object.

In a possible implementation manner, multiple items of first feature information corresponding to the multiple images one-to-one are extracted by the single-frame encoding module, and the multiple items of first feature information are extracted by the rPPG signal prediction module using a time-domain convolutional network. A feature information is processed to obtain multiple items of second feature information corresponding to the multiple images one-to-one, and a remote photoplethysmography rPPG signal corresponding to the image sequence is predicted according to the multiple items of second feature information. That is, in this implementation manner, the step of "respectively extracting multiple items of first feature information corresponding to the multiple images one-to-one" can be performed by the single-frame encoding module, "using a time-domain convolutional network to The first feature information is processed to obtain multiple items of second feature information corresponding to the multiple images one-to-one, and based on the multiple items of second feature information, the remote photoplethysmography rPPG signal corresponding to the image sequence is predicted” The steps of can be performed by the rPPG signal prediction module, and the single-frame encoding module and the rPPG signal prediction module are deployed separately.

In the related art, an end-to-end prediction method is used to predict the rPPG signal from the video stream. This end-to-end prediction method has high requirements on the performance of the platform used, and is difficult to deploy on hardware devices with weak computing performance. In the above implementation manner, by adopting a manner of separating the single-frame encoding module and the rPPG signal prediction module, the performance requirement of the hardware device is lower, so it can be applied to more hardware devices. For example, adopting this partitioned deployment scheme makes it easier to deploy on mobile devices with weak computing performance.

The heart rate measurement method provided by the embodiments of the present disclosure can be applied to application scenarios such as computer vision, physiological index measurement, and health monitoring. The heart rate measurement method provided by the embodiments of the present disclosure can be deployed on hardware devices such as smart hardware equipped with vision sensors, home security cameras, and nursing instruments for children or the elderly. When the computing resources of these hardware devices are insufficient, the heart rate measurement method provided by the embodiments of the present disclosure can still be deployed. Therefore, the heart rate measurement method provided by the embodiments of the present disclosure helps to reduce the cost of hardware devices.

The heart rate measurement method provided by the embodiment of the present disclosure is described below through a specific application scenario. FIG. 8 shows a schematic diagram of a heart rate measurement model in a heart rate measurement method provided by an embodiment of the present disclosure. In the example shown in FIG. 8 , the heart rate measurement model includes a single frame encoding module, an rPPG signal prediction module and a wavelet analysis module.

Among them, the single-frame encoding module can include 5 convolution modules, and each convolution module can include a 2D convolution layer with a convolution kernel size of 3 × 3, a 2D batch normalization layer, a 2D average pooling layer, and a Relu activation function layer. . The single-frame encoding module can perform feature extraction on one image at a time, and cache the extracted feature information. For example, the single-frame encoding module can input an image with a size of 64×64 each time, extract the 120-dimensional first feature information corresponding to the image, and cache the first feature information in a first-in, first-out queue with a length of 120 .

The rPPG signal prediction module can include 4 layers of temporal convolutional network and ordinal regression head, wherein each layer of temporal convolutional network can include 2D dilated convolution layer, adaptive cropping layer, Relu activation function layer, drop layer, 2D dilation layer The convolutional layer, adaptive cropping layer, Relu activation function layer and dropout layer, and the size of the convolution kernel of the 2D dilated convolutional layer can be 1×2, and the jumping hole can be set. The rPPG signal prediction module can process the 120 items of first feature information in the FIFO queue to obtain 120 items of second feature information corresponding to the 120 items of first feature information, wherein the second feature information Can be an 80-dimensional feature vector. For any item of second feature information in the 120 items of second feature information, 80 dimensions corresponding to the second feature information can be combined in pairs to form 40 binary classifiers. Among them, the values of the two dimensions corresponding to any two classifiers can be recorded as A and B. If A is greater than B, the result of the second classifier is 1, and if A is less than or equal to B, the result of the two classifiers is 0. The rPPG value corresponding to the second feature information can be obtained by adding the results of the 40 binary classifiers. The rPPG value corresponding to the second feature information is greater than or equal to 0 and less than or equal to 40, and the rPPG value corresponding to the second feature information is an integer value. Ordinal regression is performed on the 120 items of second feature information respectively, and 120 items of rPPG values can be obtained, and the 120 items of rPPG constitute the rPPG signal corresponding to the FIFO queue.

The wavelet analysis module can use Equation 4 above to determine various half-widths σ of wavelets of the wavelet; use Equation 2 above to perform wavelet transformation on the rPPG signal to obtain wavelet responses corresponding to different widths; determine the wavelets of different widths Respond to the corresponding response intensity and heart rate value, and determine the heart rate value corresponding to the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range as the heart rate value of the target object.

FIG. 9 shows another flowchart of a heart rate measurement method provided by an embodiment of the present disclosure. In a possible implementation manner, the execution subject of the heart rate measurement method may be a heart rate measurement device, for example, the heart rate measurement method may be executed by a terminal device or a server or other electronic devices. The terminal device may be a user equipment, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant, a handheld device, a computing device, a vehicle-mounted device, a wearable device, or the like. In some possible implementations, the heart rate measurement method may be implemented by a processor invoking computer-readable instructions stored in a memory. As shown in FIG. 9 , the heart rate measurement method includes steps S21 to S24.

In step S21, an image sequence corresponding to the target object is acquired, wherein the image sequence includes multiple images.

In step S22, multiple items of first feature information corresponding to the multiple images one-to-one are extracted respectively.

In step S23, a remote photoplethysmography rPPG signal corresponding to the image sequence is predicted according to the multiple items of first feature information.

In step S24, wavelet analysis is performed on the rPPG signal to determine the heart rate value of the target object.

In an embodiment of the present disclosure, the target object may represent an object of heart rate measurement. The target object can be a person or an animal.

In the embodiment of the present disclosure, an image may be extracted from a video stream corresponding to the target object to obtain an image sequence corresponding to the target object. The video stream corresponding to the target object may be a video stream captured on the face of the target object, that is, the image content of the video frame images in the video stream corresponding to the target object may include at least part of the face area of the target object.

In a possible implementation manner, the frame rate of the image sequence corresponding to the target object may be equal to the frame rate of the video stream corresponding to the target object. For example, the frame rate of the video stream corresponding to the target object may be 20FPS to 25FPS. In this implementation manner, each video frame image in the video stream corresponding to the target object can be extracted, and based on the each video frame image, an image sequence corresponding to the target object can be obtained.

As an example of this implementation, for any video frame image in the video stream corresponding to the target object, an image of a preset size can be intercepted from the video frame image according to the position of the face region in the video frame image, as An image in the sequence of images corresponding to the target object. For example, the preset size may be 64×64 or 128×128, etc., which is not limited here. In this example, the image content of the images in the image sequence corresponding to the target object includes at least part of the face area of the target object.

As another example of this implementation, the video frame image in the video stream corresponding to the target object may be directly used as the image in the image sequence corresponding to the target object. In this example, the size of the image in the image sequence corresponding to the target object is equal to the size of the video frame image in the video stream corresponding to the target object.

In another possible implementation manner, the frame rate of the image sequence corresponding to the target object may be smaller than the frame rate of the video stream corresponding to the target object. In this implementation, a video frame image can be extracted every K frames from the video stream corresponding to the target object, and an image sequence corresponding to the target object can be obtained based on the extracted video frame images, where K is greater than or An integer equal to 1.

As an example of this implementation, for any extracted video frame image, an image of a preset size can be intercepted from the video frame image according to the position of the face region in the video frame image, as the image sequence corresponding to the target object an image in . In this example, the image content of the images in the image sequence corresponding to the target object includes at least part of the face area of the target object.

As another example of this implementation, each of the extracted video frame images may be directly used as images in the image sequence corresponding to the target object. In this example, the size of the image in the image sequence corresponding to the target object is equal to the size of the video frame image in the video stream corresponding to the target object.

In a possible implementation manner, the images in the image sequence corresponding to the target object may be RGB images. In other possible implementation manners, the image in the image sequence corresponding to the target object may also be an infrared image, etc., which is not limited in this embodiment of the present disclosure.

In this embodiment of the present disclosure, the image sequence corresponding to the target object may include N images, where N is an integer greater than or equal to 3. For example, N can be equal to 120. Of course, those skilled in the art can flexibly set the size of N according to actual application scenario requirements, which is not limited here.

The embodiments of the present disclosure determine the heart rate value of the target object based on the image sequence corresponding to the target object, thereby enabling non-contact heart rate measurement, thereby improving the convenience of heart rate measurement.

In this embodiment of the present disclosure, the first feature information may represent feature information corresponding to images in the image sequence. The first feature information is in one-to-one correspondence with images in the image sequence.

In a possible implementation manner, the first feature information corresponding to the images in the image sequence may be extracted through a single-frame encoding module. For example, the input of the single-frame encoding module may be a three-channel image with a size of 64×64, and the output of the single-frame encoding module may be the first feature information, and the first feature information may be a 120-dimensional feature vector.

As an example of this implementation, the single-frame encoding module may include 5 convolution modules, and each convolution module may include a 2D convolution layer with a convolution kernel size of 3×3, a 2D batch normalization layer, and a 2D average pooling layer. layer and Relu activation function layer.

Of course, the single-frame encoding module may also adopt other structures, which are not limited here.

In a possible implementation manner, the predicting the remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple items of first feature information includes: processing the multiple items of first feature information to obtain Multiple items of second feature information corresponding to the multiple images one-to-one; and predicting the rPPG signal corresponding to the image sequence according to the multiple items of second feature information.

As an example of this implementation, the multiple items of first feature information may be processed through a time-domain convolutional network to obtain multiple items of second feature information corresponding to the multiple images one-to-one. Among them, the time-domain convolutional network is a lightweight deep neural network, so it can reduce the performance requirements of the hardware. In addition, the time-domain convolutional network uses dilated convolution to expand the receptive field. Compared with ordinary convolution operations, dilated convolution is more suitable for extracting long-term related features.

In this example, the multiple items of first feature information may be processed through an M-layer temporal convolutional network to obtain multiple items of second feature information corresponding to the multiple images one-to-one, where M is greater than or An integer equal to 1. The M-layer time-domain convolutional network can form a time-domain convolutional network module. For example, M can be equal to 4, and each layer of the temporal convolutional network can include a 2D dilated convolutional layer, an adaptive cropping layer, a Relu activation function layer, a dropout layer, a 2D dilated convolutional layer, an adaptive cropping layer, and a Relu activation function layer and discard layers. Among them, the size of the convolution kernel of the 2D dilated convolution layer can be 1 × 2, and the jumper holes can be set.

In one example, the input of the time-domain convolutional network module may be N items of 120-dimensional first feature information, and the output may be N items of 80-dimensional second feature information.

In other examples, a neural network such as LSTM may also be used to process the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one.

In an example, the first feature information corresponding to the images in the image sequence may be sequentially extracted, and the first feature information may be cached in a FIFO queue of preset length; The multiple items of first feature information in the FIFO queue are processed to obtain multiple items of second feature information corresponding to the multiple items of first feature information one-to-one.

In the related art, the encoding module performs feature encoding on all images in the sliding window each time. When the step size of the sliding window is smaller than the size of the sliding window, the encoding module will repeatedly encode part of the images in the sliding window, resulting in redundant computation. In the above example, the heart rate measurement model may only include a single-frame encoding module, and the single-frame encoding module may perform feature extraction on one image at a time, and cache the extracted feature information.

In this example, the first feature information corresponding to the images in the image sequence may be sequentially extracted according to the order of the images in the image sequence, and the extracted first feature information may be cached in a first-in, first-out queue of preset length. middle. The first-in, first-out queue may represent a first-in, first-out queue for buffering the first feature information. In one example, the length of the FIFO queue may be N, that is, the FIFO queue may buffer the first feature information corresponding to N images. For example, N can be equal to 120.

In this example, the time-domain convolutional network can process all the first feature information in the FIFO queue, and obtain the second feature information corresponding to each item of first feature information in the FIFO queue one-to-one. characteristic information.

In this example, by sequentially extracting the first feature information corresponding to the images in the image sequence, the first feature information is cached in a FIFO queue of preset length, and a time-domain convolutional network is used to The multiple items of first feature information in the FIFO queue are processed to obtain multiple items of second feature information that correspond one-to-one with the multiple items of first feature information, which can reduce redundant calculations and save the calculation of hardware devices. force.

In another example, feature extraction may be performed on at least two images in parallel. In this implementation, the heart rate measurement model may include at least two single-frame encoding modules.

As an example of this implementation, the predicting the rPPG signal corresponding to the image sequence according to the multiple items of second feature information includes: performing ordinal regression on the multiple items of second feature information to obtain the image sequence Corresponding remote photoplethysmography rPPG signal. In this example, ordinal regression may be performed on the multiple items of second feature information, respectively, to obtain multiple items of rPPG values that correspond one-to-one with the multiple items of second feature information, and may be composed of multiple items of rPPG values according to the multiple items of second feature information The rPPG signal corresponding to the image sequence.

In the example shown in FIG. 4 , the image sequence includes 120 frames of images, and among the 120 items of second feature information corresponding to the 120 frames of images, each item of second feature information is an 80-dimensional feature vector. For any item of second feature information in the 120 items of second feature information, 80 dimensions corresponding to the second feature information can be combined in pairs to form 40 binary classifiers. Among them, the values of the two dimensions corresponding to any two classifiers can be recorded as A and B. If A is greater than B, the result of the second classifier is 1, and if A is less than or equal to B, the result of the two classifiers is 0. The rPPG value corresponding to the second feature information can be obtained by adding the results of the 40 binary classifiers. The rPPG value corresponding to the second feature information is greater than or equal to 0 and less than or equal to 40, and the rPPG value corresponding to the second feature information is an integer value. As shown in Figure 4, for the first item of second feature information, the 80 dimensions corresponding to the second feature information can be combined in pairs to form 40 binary classifiers, each of which represents a component of the rPPG value . The values of the two dimensions corresponding to any two-classifier can be recorded as A and B. If A is greater than B, the result of the two-classifier is 1, and if A is less than or equal to B, the result of the two-classifier is 0 . The rPPG value corresponding to the first item of the second feature information can be obtained by adding the results of the 40 binary classifiers. Ordinal regression is performed on the 120 items of second feature information respectively, and 120 items of rPPG values can be obtained, and the 120 items of rPPG constitute the rPPG signal corresponding to the image sequence.

In this example, the long-range photoplethysmography rPPG signal corresponding to the image sequence is obtained by performing ordinal regression on the multiple items of second feature information, so that an ordinal regression method that is more suitable for predicting continuous values can be used to obtain better Accurate rPPG signal.

Of course, other regression methods may also be used to regress the multiple pieces of second feature information, which are not limited here.

rPPG signals predicted by deep learning algorithms often have strong noise. In related technologies, the heart rate is usually calculated by detecting the peaks and the peak spacing of the signal, or by applying the fast Fourier transform to extract the main frequency to calculate the heart rate. These methods are often greatly affected by noise, and are easy to use for rPPG signals with low signal-to-noise ratios. produce larger errors. In this embodiment of the present disclosure, by acquiring an image sequence corresponding to a target object, wherein the image sequence includes multiple images, multiple items of first feature information corresponding to the multiple images are extracted respectively, and according to the multiple images Items of first feature information, predict the rPPG signal corresponding to the image sequence, and perform wavelet analysis on the rPPG signal to determine the heart rate value of the target object, which can improve the calculation of the rPPG signal with low signal-to-noise ratio. Compatibility, that is, more accurate heart rate values can be determined even when the signal-to-noise ratio of the rPPG signal is low.

In a possible implementation manner, the performing wavelet analysis on the rPPG signal to determine the heart rate value of the target object includes: determining at least two widths of wavelets of the wavelet; according to the at least two widths, Perform wavelet transformation on the rPPG signal to obtain at least two wavelet responses corresponding to the at least two widths one-to-one; determine the response intensity and heart rate value corresponding to the at least two wavelet responses; In response to the corresponding response intensity and heart rate value, the heart rate value of the target object is determined.

In this implementation manner, a Ricker wavelet function may be used, or other wavelet functions may be used, which is not limited herein. The rPPG signal can be denoted as s(t), and the wavelet function can be denoted as y(t,σ). The wavelet response obtained by the wavelet transform of the rPPG signal can be denoted as I(t,σ).

In this implementation manner, the wavelet responses corresponding to different widths may represent the intensities of waves with different widths in the curve corresponding to the rPPG signal. I(t=t ₀ , σ=σ ₀ ) represents the response intensity corresponding to the wavelet response whose half-width of the wavelet is σ ₀ at time t ₀ . At the peak position of the heartbeat, the response intensity of a wavelet of appropriate width will peak.

In this implementation manner, the response intensity corresponding to any one of the at least two wavelet responses may represent the overall response intensity corresponding to the wavelet response. The response intensity M(σ ₀ ) corresponding to the wavelet response whose half-width of the wavelet is σ ₀ can be determined by using Equation 3 above.

其中，K为大于1的整数。那么，可以计算得到子波的半宽度为σ₀的小波响应对应的响应频率为

即，子波的半宽度为σ₀的小波响应对应的心率值为

In an example, the peaks of the time-domain response curve corresponding to the wavelet response with the half-width of the wavelet σ ₀ appear in sequence from t ₁ to t _K , and the average period is

Wherein, K is an integer greater than 1. Then, it can be calculated that the response frequency corresponding to the wavelet response whose half-width of the wavelet is σ ₀ is

That is, the heart rate corresponding to the wavelet response whose half-width of the wavelet is σ ₀ is

Since the signal-to-noise ratio of the rPPG signal is usually low, and the heart rate varies greatly with time, in the above implementation manner, by determining at least two widths of the wavelet of the wavelet, according to the at least two widths, The rPPG signal is subjected to wavelet transformation to obtain at least two wavelet responses corresponding to the at least two widths one-to-one, and the response intensity and heart rate value corresponding to the at least two wavelet responses are determined, and according to the at least two wavelet responses The corresponding response intensity and heart rate value are used to determine the heart rate value of the target object, thereby helping to more accurately determine the heart rate value of the target object.

In an example, the determining at least two widths of the wavelet wavelet includes: determining at least two widths of the wavelet wavelet according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence. kind of width.

For example, the maximum value of the normal heart rate may be used as the maximum preset heart rate value; the minimum value of the normal heart rate may be used as the minimum preset heart rate value.

In one example, the above equation 4 can be used to determine the half width σ of the wavelet of the wavelet.

The half-width of the wavelet of the wavelet may be an integer value or a decimal value. In one example, the integer values satisfying Equation 4 can be respectively determined as the half widths of the wavelets of the wavelet. For example, f=20, h _min =60, h _max =100, then, σ satisfying Equation 4 includes 6, 7, 8, 9, and 10, so that five kinds of widths of wavelets of the wavelet can be determined.

In this example, at least two widths of wavelets of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence, thereby helping to determine an appropriate width, Thereby, the accuracy of heart rate determination can be improved.

In an example, the determining the heart rate value of the target object according to the response intensity and the heart rate value corresponding to the at least two wavelet responses includes: determining the heart rate value of the target object in the wavelet response whose heart rate value belongs to a preset heart rate value range, the response intensity The heart rate value corresponding to the largest wavelet response is determined as the heart rate value of the target object.

确定出心率值属于预设心率值范围的小波响应中、响应强度最大的小波响应，并可以将

确定为目标对象的心率值。For example, the preset heart rate value range may be [h _min , h _max ], then, according to h _min ≤f(σ)≤h _max , wavelet responses whose heart rate values fall within the preset heart rate value range can be screened. For example, the half-width of the wavelet of the wavelet response whose heart rate value belongs to the preset heart rate value range can be denoted as σ′, then, according to

Determine the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range, and can use

Determines the heart rate value of the target subject.

Since the response intensity of the noise may be greater than the response intensity of the heart rate in the case of a low SNR The corresponding heart rate value is determined as the heart rate value of the target object, thereby improving the robustness and accuracy of the heart rate measurement.

In a possible implementation manner, the single-frame encoding module extracts multiple pieces of first feature information that correspond to the multiple images one-to-one, and the rPPG signal prediction module predicts the first feature information according to the multiple pieces of first feature information. The rPPG signal corresponding to the image sequence described above. That is, in this implementation manner, the step of "respectively extracting multiple items of first feature information corresponding to the multiple images" may be performed by the single-frame encoding module, "according to the multiple items of first feature information, predict the The step of "rPPG signal corresponding to the image sequence" may be performed by the rPPG signal prediction module, and the single-frame encoding module and the rPPG signal prediction module are deployed separately.

In the related art, an end-to-end prediction method is used to predict the rPPG signal from the video stream. This end-to-end prediction method has high requirements on the performance of the platform used, and is difficult to deploy on hardware devices with weak computing performance. In the above implementation manner, by adopting a manner of separating the single-frame encoding module and the rPPG signal prediction module, the performance requirement of the hardware device is lower, so it can be applied to more hardware devices. For example, adopting this partitioned deployment scheme makes it easier to deploy on mobile devices with weak computing performance.

It can be understood that the above-mentioned method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic. Those skilled in the art can understand that, in the above method of the specific embodiment, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present disclosure also provides heart rate measurement devices, electronic equipment, computer-readable storage media, and computer program products, all of which can be used to implement any heart rate measurement method provided by the present disclosure, and the corresponding technical solutions and technical effects can be found in the method section The corresponding records will not be repeated.

FIG. 10 shows a block diagram of a heart rate measurement apparatus provided by an embodiment of the present disclosure. As shown in Figure 10, the heart rate measurement device includes:

an acquisition module 31, configured to acquire an image sequence corresponding to the target object, wherein the image sequence includes multiple images;

an extraction module 32, configured to extract a plurality of pieces of first feature information corresponding to the plurality of images one-to-one;

The processing module 33 is configured to process the multiple items of first feature information by using a time-domain convolutional network to obtain multiple items of second feature information corresponding to the multiple images one-to-one;

The first determination module 34 is configured to determine the heart rate value of the target object according to the multiple pieces of second characteristic information.

In a possible implementation manner, the first determining module 34 is used for:

predicting a remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of second feature information;

According to the rPPG signal, the heart rate value of the target object is determined.

In a possible implementation manner, the first determining module 34 is used for:

Ordinal regression is performed on the multiple items of second feature information to obtain a remote photoplethysmography rPPG signal corresponding to the image sequence.

In a possible implementation manner, the first determining module 34 is used for:

Wavelet analysis is performed on the rPPG signal to determine the heart rate value of the target object.

In a possible implementation manner, the first determining module 34 is used for:

determining at least two widths of the wavelets of the wavelet;

performing wavelet transformation on the rPPG signal according to the at least two widths to obtain at least two wavelet responses corresponding to the at least two widths one-to-one;

determining the response intensity and the heart rate value corresponding to the at least two wavelet responses;

The heart rate value of the target object is determined according to the response intensity and the heart rate value corresponding to the at least two wavelet responses.

In a possible implementation manner, the first determining module 34 is used for:

At least two widths of the wavelet of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence.

In a possible implementation manner, the first determining module 34 is used for:

The heart rate value corresponding to the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range is determined as the heart rate value of the target object.

In one possible implementation,

The extraction module 32 is configured to sequentially extract the first feature information corresponding to the images in the image sequence, and cache the first feature information in a FIFO queue of preset length;

The processing module 33 is configured to: use a time-domain convolutional network to process multiple items of first feature information in the FIFO queue to obtain multiple items of second feature information that correspond one-to-one with the multiple items of first feature information. characteristic information.

In the embodiment of the present disclosure, by acquiring an image sequence corresponding to a target object, wherein the image sequence includes multiple images, a plurality of pieces of first feature information corresponding to the multiple images are extracted respectively, and a temporal volume The product network processes the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one, and determines the heart rate of the target object according to the multiple items of second feature information Therefore, a lightweight time-domain convolutional network is used to replace the traditional time-series network structure (such as LSTM), and the time-series relationship is modeled through a lightweight time-domain convolutional network to extract time-series information, which can reduce the model Deployment relies on operators, lowers the hardware threshold for model deployment, and enhances the adaptability of models to hardware, that is, it can be deployed on lower-cost hardware that supports fewer neural network operators and has poor computing performance superior.

FIG. 11 shows another block diagram of the heart rate measurement device provided by the embodiment of the present disclosure. As shown in Figure 11, the heart rate measurement device includes:

an acquisition module 41, configured to acquire an image sequence corresponding to the target object, wherein the image sequence includes multiple images;

The extraction module 42 is used for extracting a plurality of pieces of first feature information corresponding to the plurality of images one-to-one respectively;

A prediction module 43, configured to predict the remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of first feature information;

The second determination module 44 is configured to perform wavelet analysis on the rPPG signal to determine the heart rate value of the target object.

In a possible implementation manner, the second determining module 44 is used for:

determining at least two widths of the wavelets of the wavelet;

performing wavelet transformation on the rPPG signal according to the at least two widths to obtain at least two wavelet responses corresponding to the at least two widths one-to-one;

determining the response intensity and the heart rate value corresponding to the at least two wavelet responses;

The heart rate value of the target object is determined according to the response intensity and the heart rate value corresponding to the at least two wavelet responses.

In a possible implementation manner, the second determining module 44 is used for:

At least two widths of the wavelet of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence.

In a possible implementation manner, the second determining module 44 is used for:

The heart rate value corresponding to the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range is determined as the heart rate value of the target object.

In a possible implementation, the prediction module 43 is used for:

processing the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one;

According to the multiple pieces of second feature information, the rPPG signal corresponding to the image sequence is predicted.

In a possible implementation, the prediction module 43 is used for:

Ordinal regression is performed on the multiple items of second feature information to obtain a remote photoplethysmography rPPG signal corresponding to the image sequence.

In this embodiment of the present disclosure, by acquiring an image sequence corresponding to a target object, wherein the image sequence includes multiple images, multiple items of first feature information corresponding to the multiple images are extracted respectively, and according to the multiple images Items of first feature information, predict the rPPG signal corresponding to the image sequence, and perform wavelet analysis on the rPPG signal to determine the heart rate value of the target object, which can improve the calculation of the rPPG signal with low signal-to-noise ratio. Compatibility, that is, more accurate heart rate values can be determined even when the signal-to-noise ratio of the rPPG signal is low.

In some embodiments, the functions or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the methods described in the above method embodiments, and the specific implementation and technical effects may refer to the above method embodiments. It is concise and will not be repeated here.

Embodiments of the present disclosure further provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented. Wherein, the computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

An embodiment of the present disclosure further provides a computer program, including computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method.

Embodiments of the present disclosure also provide a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are executed in an electronic device , the processor in the electronic device executes the above method.

Embodiments of the present disclosure further provide an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke executable instructions stored in the memory instruction to execute the above method.

The electronic device may be provided as a terminal, server or other form of device.

FIG. 12 shows a block diagram of an electronic device 1900 provided by an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. 12, electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource, represented by memory 1932, for storing instructions executable by processing component 1922, such as applications. An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a Microsoft server operating system (Windows Server ^™ ), a graphical user interface based operating system (Mac OSX ^™ ) introduced by Apple, a multi-user multi-process computer operating system ( Unix ^™ ), Free and Open Source Unix-like Operating System (Linux ^™ ), Open Source Unix-like Operating System (FreeBSD ^™ ) or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as memory 1932 comprising computer program instructions executable by processing component 1922 of electronic device 1900 to perform the above-described method.

The present disclosure may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for carrying out operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code, written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to the processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, and the same or similar points can be referred to each other, and for the sake of brevity, they will not be repeated herein.

If the technical solutions of the embodiments of the present disclosure involve personal information, before processing personal information, the products applying the technical solutions of the embodiments of the present disclosure have been clearly informed of the personal information processing rules, and the individual's voluntary consent has been obtained. If the technical solutions of the embodiments of the present disclosure involve sensitive personal information, the products applying the technical solutions of the embodiments of the present disclosure have obtained individual consent before processing sensitive personal information, and at the same time satisfy the requirement of "express consent". For example, at the personal information collection device such as a camera, set a clear and conspicuous sign to inform that the personal information has entered the scope of personal information collection, and the personal information will be collected. On the personal information processing device, if the personal information processing rules are informed by obvious signs/information, the personal authorization can be obtained by means of pop-up information or asking individuals to upload their personal information; among them, the personal information processing rules may include personal information Information processor, purpose of processing personal information, method of processing, and types of personal information processed.

Various embodiments of the present disclosure have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims (19)

1. a heart rate measurement method, is characterized in that, comprises: acquiring an image sequence corresponding to the target object, wherein the image sequence includes multiple images; respectively extracting multiple items of first feature information corresponding to the multiple images one-to-one; Using a temporal convolution network to process the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one; The heart rate value of the target object is determined according to the plurality of pieces of second feature information. 2. The method according to claim 1, wherein the determining the heart rate value of the target object according to the multiple pieces of second characteristic information comprises: predicting a remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of second feature information; According to the rPPG signal, the heart rate value of the target object is determined. 3. The method according to claim 2, wherein the predicting the remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of second feature information comprises: Ordinal regression is performed on the multiple items of second feature information to obtain a remote photoplethysmography rPPG signal corresponding to the image sequence. 4. The method according to claim 2 or 3, wherein the determining the heart rate value of the target object according to the rPPG signal comprises: Wavelet analysis is performed on the rPPG signal to determine the heart rate value of the target object. 5. The method according to claim 4, wherein the performing wavelet analysis on the rPPG signal to determine the heart rate value of the target object comprises: determining at least two widths of the wavelets of the wavelet; performing wavelet transformation on the rPPG signal according to the at least two widths to obtain at least two wavelet responses corresponding to the at least two widths one-to-one; determining the response intensity and the heart rate value corresponding to the at least two wavelet responses; The heart rate value of the target object is determined according to the response intensity and the heart rate value corresponding to the at least two wavelet responses. 6. The method according to claim 5, wherein the determining at least two widths of the wavelet of the wavelet comprises: At least two widths of the wavelet of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence. 7. The method according to claim 5 or 6, wherein the determining the heart rate value of the target object according to the response intensity and the heart rate value corresponding to the at least two wavelet responses, comprises: The heart rate value corresponding to the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range is determined as the heart rate value of the target object. 8. The method according to any one of claims 1 to 7, characterized in that, The extracting, respectively, multiple items of first feature information corresponding to the multiple images includes: sequentially extracting the first feature information corresponding to the images in the image sequence, and buffering the first feature information in a pre-set Set the length in the first-in first-out queue; The use of a time-domain convolution network to process the multiple items of first feature information to obtain multiple items of second feature information corresponding to the multiple images one-to-one includes: using a time-domain convolution network for the advanced feature information. The multiple items of first feature information in the first-out queue are processed to obtain multiple items of second feature information corresponding to the multiple items of first feature information one-to-one. 9. A heart rate measuring device, comprising: an acquisition module, configured to acquire an image sequence corresponding to the target object, wherein the image sequence includes multiple images; an extraction module, configured to extract a plurality of pieces of first feature information corresponding to the plurality of images one-to-one; a processing module, configured to process the multiple items of first feature information by using a time-domain convolutional network to obtain multiple items of second feature information corresponding to the multiple images one-to-one; The first determination module is configured to determine the heart rate value of the target object according to the multiple pieces of second characteristic information. 10. The apparatus according to claim 9, wherein the first determining module is used for: predicting a remote photoplethysmography rPPG signal corresponding to the image sequence according to the multiple pieces of second feature information; According to the rPPG signal, the heart rate value of the target object is determined. 11. The apparatus according to claim 10, wherein the first determining module is configured to: Ordinal regression is performed on the multiple items of second feature information to obtain a remote photoplethysmography rPPG signal corresponding to the image sequence. 12. The apparatus according to claim 10 or 11, wherein the first determining module is configured to: Wavelet analysis is performed on the rPPG signal to determine the heart rate value of the target object. 13. The apparatus according to claim 12, wherein the first determining module is configured to: determining at least two widths of the wavelets of the wavelet; performing wavelet transform on the rPPG signal according to the at least two widths to obtain at least two wavelet responses corresponding to the at least two widths one-to-one; determining the response intensity and the heart rate value corresponding to the at least two wavelet responses; The heart rate value of the target object is determined according to the response intensity and the heart rate value corresponding to the at least two wavelet responses. 14. The apparatus according to claim 13, wherein the first determining module is configured to: At least two widths of the wavelet of the wavelet are determined according to the frame rate, the maximum preset heart rate value and the minimum preset heart rate value corresponding to the image sequence. 15. The apparatus according to claim 13 or 14, wherein the first determining module is configured to: The heart rate value corresponding to the wavelet response with the largest response intensity among the wavelet responses whose heart rate value belongs to the preset heart rate value range is determined as the heart rate value of the target object. 16. The device according to any one of claims 9 to 15, characterized in that, The extraction module is used for: sequentially extracting the first feature information corresponding to the images in the image sequence, and buffering the first feature information in a FIFO queue of preset length; The processing module is used for: using a time-domain convolutional network to process multiple items of first feature information in the FIFO queue to obtain multiple items of second feature information one-to-one corresponding to the multiple items of first feature information information. 17. An electronic device, comprising: one or more processors; memory for storing executable instructions; wherein the one or more processors are configured to invoke executable instructions stored in the memory to perform the method of any one of claims 1-8. 18. A computer-readable storage medium on which computer program instructions are stored, wherein the computer program instructions implement the method of any one of claims 1 to 8 when executed by a processor. 19. A computer program product, characterized by comprising computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are executed in an electronic device, A processor in the electronic device performs the method of any one of claims 1 to 8.

Images