TWI841435B - Time series data fusion algorithm and home care system based on convolutional neural network - Google Patents

Abstract

This invention provides a time series data fusion algorithm and home care system based on convolutional neural network, the time series data fusion algorithm based on convolutional neural network includes the following steps: input multi-source time series data; use denoising autoencoder to denoise and reconstruct each time series data; input each time series data into a 1-D convolution neural network for training and learning; adding self-attention mechanism to deep convolutional neural network, it is used to improve the ability to capture long-distance information of time series data; and data fusion result output.

The time series data fusion algorithm based on convolutional neural network can improve the ability to capture long-distance information, compute the correlations within the entire time series, and has the advantage of high computational efficiency.

Description

Time series data fusion algorithm based on convolutional neural network and home care system

The present invention relates to the field of information fusion algorithms; in particular, it is an innovative technology revealer of a time series data fusion algorithm based on a convolutional neural network and a home care system.

Data fusion obtains simpler and more accurate judgments through multi-source sensor information. By fusing the real-time data of multiple sensors with the information in the relevant database, more accurate results and more specific inferences can be obtained. Data fusion methods can be used to mine potential information in the data.

The general processing flow of data fusion methods mainly includes: data acquisition, feature extraction, target description, data association and target consistency. Traditional data fusion algorithms have the problem that it is difficult to associate information between different sensors and are very likely to rely on prior experience and artificial feature extraction. It is necessary to use neural network-like methods to achieve automatic association and fusion of data. Machine learning and deep learning neural network-like algorithms have been widely used in data fusion. Among them, deep convolutional neural networks (DCNN) have outstanding fusion performance. It increases the depth of convolutional neural networks to achieve feature extraction, information association and decision judgment of multi-sensor information. However, when the above methods are fused, there are some shortcomings in network model design and input data feature processing, which makes it difficult to significantly improve the fusion performance.

The main purpose of this invention is to provide a time series data fusion algorithm and home care system based on convolutional neural network. The technical problem it aims to solve is to explore and innovate breakthroughs in how to develop a more ideal and practical new data fusion algorithm and its application.

Based on the above purpose, the technical features of the present invention for solving the problem mainly lie in that the time series data fusion algorithm based on the convolutional neural network includes the following steps: inputting multi-source time series data; using a denoising auto-encoder to denoise and reconstruct each of the time series data; inputting each of the time series data into a one-dimensional convolutional neural network for training and learning; adding a self-attention mechanism to the deep convolutional neural network to improve the ability to capture long-distance information of each of the time series data; and outputting the data fusion result.

The main effect and advantage of the present invention is that it can combine the convolutional neural network with the self-attention mechanism algorithm to improve the ability of the time series data fusion algorithm based on the convolutional neural network to capture long-distance information. The self-attention mechanism algorithm can calculate the internal correlation of the entire time series and has the advantage of high computational efficiency.

The home care system using the time series data fusion algorithm based on the convolution neural network includes a camera for capturing images of the care recipient, a processing device electrically connected to the camera, and a microprocessor for receiving the image and executing the time series data fusion algorithm based on the convolution neural network on the image to judge the posture and movement of the care recipient.

10: Camera

20: Processing device

22: Microprocessor

24: Wireless signal transmission and reception module

26: Memory module

28: Alarm device

Figure 1 is a structural diagram of the algorithm of the preferred embodiment of the present invention.

Figure 2 is a one-dimensional convolutional neural network structure diagram of the algorithm of the preferred embodiment of the present invention.

Figure 3 is a flow chart of the algorithm of the preferred embodiment of the present invention.

Figure 4 is a structural diagram of the noise reduction automatic encoder of the algorithm of the preferred embodiment of the present invention.

Figure 5 is an example diagram of the pooling operation of the algorithm of the preferred embodiment of the present invention.

Figure 6 is a structural diagram of the self-attention mechanism of the algorithm of the preferred embodiment of the present invention.

Figure 7 is a framework diagram of the self-attention mechanism of the algorithm of the preferred embodiment of the present invention.

Figure 8 is a circuit block diagram of a home care system of a preferred embodiment of the present invention.

Please refer to the drawings, which are preferred embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not limited to this structure in patent applications.

As shown in Figures 1 to 7, the time series data fusion algorithm based on the convolutional neural network includes the following steps: input multi-source time series data.

Denoising AutoEncoder is used to denoise and reconstruct the time series data: The original characteristics of the data have a great impact on the performance of the deep learning network. The denoising autoencoder has a good performance in data denoising and restoration. The main reason is that the denoising autoencoder network adds noise to the original time series data for training and learning, thereby forcing the learned data to be more robust and have better execution capabilities. Therefore, the denoising autoencoder network is used to denoise and reconstruct the original time series data. The denoising autoencoder is used to denoise and reconstruct the original time series data from multiple sources to obtain more robust data.

Based on the data fusion calculation that is more robust by partially destroying the input pattern to learn the representation data, the input time series data is first partially destroyed, and then the network is trained through unsupervised learning. This method is based on the human perception function of associative memory for knowledge, that is, people can combine multiple fragments of knowledge through associative methods to obtain complete memory, so when the data is damaged or lost, our brain can still recall it; this noise reduction autoencoder is a kind of artificial neural network method, which can resist the pollution and loss of original data.

The denoising and reconstruction processing of each time series data using the denoising automatic encoder is to add Gaussian white noise to the original time series data, then encode and decode, and use an unsupervised learning method to train the network, thereby restoring the real information and performing denoising and reconstruction processing.

Using the denoising auto-encoder to denoise and reconstruct each time series data, each time series data can be randomly set to zero, then encoded and decoded, and the network can be trained using an unsupervised learning method to restore the real information, thus becoming a transformation implementation option.

After using the denoising auto-encoder to denoise and reconstruct each time series data, the order of each time series data is randomly shuffled to speed up the network training speed, and then grouped at a fixed time. Each time series data of fixed size is input into a one-dimensional convolutional neural network (1D CNN) for training and learning. The one-dimensional convolutional neural network is a deep convolutional neural network.

Adding self-attention mechanisms (AM) to deep convolutional neural networks is used to improve the ability to capture long-distance information of each time series data.

Data fusion result output.

Each time series data enters the convolution layer, the convolution forward calculation is selected and the hyperbolic tangent function is used as the activation function to perform nonlinear representation, and then enters the pooling layer to perform average pooling operation, repeats the convolution layer and the pooling layer operation once, and then calculates the output of the self-attention mechanism, and then connects to the fully connected layer to aggregate key information again, and finally connects to the fully connected layer with the normalized exponential (Softmax) function to output the final decision value, and then outputs the fusion value result.

On this convolution layer, each convolution kernel slides down to perform convolution operations. This convolution layer extracts data features through convolution calculations, and more complex abstract features are obtained through multiple convolution layers. In addition, the convolution kernels of each convolution layer share weight values, which greatly reduces the computational complexity of the neural network.

In this pooling layer, the key points in the feature map are extracted through the pooling method and the parameter dimension is reduced. The pooling operation is then used to perform secondary sampling to form a new feature map.

The function of the fully connected layer is mainly to convert the feature map into a one-dimensional vector to enhance the original signal, reduce the dimension of the data and preserve the main information. The increase in the depth of the fully connected layer can effectively improve the nonlinear expression ability of the model.

The self-attention mechanism mainly uses the fully connected layer with the normalized exponential activation function as the similarity function, and uses its output as the weight value. The output of the fully connected layer is matrix multiplied with its input, and the result is the output of the self-attention mechanism.

The algorithm of the self-attention mechanism is weighted summation. We can regard the information in the input source (Source) as a series of key-value pairs (Key, Value). If we give a query element (Query) in the target output (Target), we can calculate the similarity between the query element and the key to obtain the weight coefficient of each key corresponding to the value. Finally, the elements in the input source are weighted and summed according to the weight coefficient to obtain the final self-attention (Attention) value.

In order to improve the deficiency that the convolutional neural network can only obtain local information of the sequence data, the time series data fusion algorithm based on the convolutional neural network combines the convolutional neural network with the self-attention mechanism algorithm to improve the ability of the time series data fusion algorithm based on the convolutional neural network to capture long-distance information. The main reason is that the self-attention mechanism algorithm can calculate the internal correlation of the entire time series and has the advantage of high computational efficiency.

As shown in FIG8 , the home care system using the aforementioned time series data fusion algorithm based on the convolution neural network includes a camera 10 and a processing device 20, wherein the camera 10 is used to capture the image of the care recipient, and the processing device 20 is electrically connected to the camera 10. When a network camera is used as the camera 10, the camera 10 and the processing device 20 can select to transmit signals bidirectionally through a wired network or a wireless network. The processing device 20 has a microprocessor 22, and the processing device 20 is used to receive the image and execute the aforementioned time series data fusion algorithm based on the convolution neural network on the image, thereby judging the posture and movement of the care recipient.

Specifically, after the camera 10 captures the image of the care recipient, the nodes of the care recipient's body joints in the image are marked by using the method of locating anatomical key points, and the nodes are connected to obtain the time series data for executing the time series data fusion algorithm based on the convolution neural network, thereby determining the posture and movement of the care recipient.

The posture of the care recipient can be divided into two categories: normal posture and fallen posture, which can provide effective reference for caregivers during monitoring. The key to defining the fallen posture is whether the human skeleton recognition diagram after connecting the nodes is completely vertical to the image screen. If it is not completely vertical, it may be a fall. The rest of the postures are defined as normal postures. The care recipient may present a variety of different postures when he/she is unconscious or falls, but most of the postures are falling or lying on the ground. When the care recipient is unconscious or falls, although the posture recognition status defined as a normal posture may be presented, it involves the safety risk of the care recipient and can still provide the caregiver with a reference for judging whether it is necessary to conduct an on-site visit or take emergency measures.

The home care system can be optionally combined with a physiological sensor. The posture judgment of the care recipient is combined with the physiological data of the care recipient captured by the physiological sensor to determine whether the alarm notification requirement is met. Specific examples of the physiological data may be heart rate, blood pressure, body temperature, etc.

The processing device 20 further includes a wireless signal transmission and reception module 24, a memory module 26 and an alarm 28, wherein the wireless signal transmission and reception module 24, the memory module 26 and the alarm 28 are electrically connected to the microprocessor 22 respectively, the memory module 26 is a readable and writable memory medium, the memory module 26 stores the notification message, and the microprocessor 22 controls the care recipient when the posture or action is determined to be abnormal. The wireless signal transmission and receiving module 24 sends an alarm message to the communication device held by the set user or fire rescue unit through the Internet, and controls the alarm device 28 to emit an alarm sound. Accordingly, when the caregiver is resting or handling other matters, the home care system can provide an immediate alarm when the caregiver is determined to be likely to fall into a coma or fall, avoiding delays in taking emergency measures for the caregiver.

Claims (4)

A time series data fusion algorithm based on a convolutional neural network includes the following steps: inputting multi-source time series data; using a denoising auto-encoder to denoise and reconstruct each of the time series data, adding Gaussian white noise to each of the time series data, then encoding and decoding, and using an unsupervised learning method to train the network to restore the real information; after denoising and reconstructing each of the time series data using the denoising auto-encoder, first randomly shuffle the order of each of the time series data to accelerate the network training speed, and then group them at a fixed time, and then input each of the time series data of a fixed size into a one-dimensional convolutional neural network for training and learning; adding a self-attention mechanism in the deep convolutional neural network to improve the attention to each of the time series data The self-attention mechanism mainly uses the fully connected layer with the normalized exponential function as the similarity function, and uses its output as the weight value. The output of the fully connected layer is matrix-multiplied with its input, and the result is the output of the self-attention mechanism. The time series data enters the convolution layer, and the forward calculation of the convolution uses the hyperbolic tangent function. The number is used as the activation function to represent it nonlinearly, and then it enters the pooling layer to perform the average pooling operation, repeats the operation of the convolution layer and the pooling layer, and then calculates the output of the self-attention mechanism, and then connects with the fully connected layer to aggregate the key information again, and finally connects with the fully connected layer with the normalized exponential function to output the final decision value; and the data fusion result is output. A home care system includes a camera for capturing images of a care recipient, a processing device electrically connected to the camera, the processing device having a microprocessor, the processing device for receiving the image and executing the time series data fusion algorithm based on the convolutional neural network described in claim 1 on the image, thereby determining the posture and movement of the care recipient. The home care system as described in claim 2, wherein the camera captures the image of the care recipient, and then uses the method of locating anatomical key points to mark the nodes of the care recipient's human joints in the image, and connects each of the nodes, thereby obtaining multi-source time series data for executing the time series data fusion algorithm based on the convolutional neural network, thereby determining the posture and movement of the care recipient. The home care system as described in claim 2, wherein the processing device further includes a wireless signal transmission and reception module, a memory module and an alarm, wherein the wireless signal transmission and reception module, the memory module and the alarm are electrically connected to the microprocessor respectively, the memory module is composed of a readable and writable memory medium, the memory module stores notification messages, and the microprocessor controls the wireless signal transmission and reception module to send an alarm message to a communication device held by a set user or fire rescue unit through the Internet when the posture or movement of the care recipient is determined to be abnormal, and controls the alarm to emit an alarm sound.

Images