TWI790715B - Intelligent image recognition drowning warning system - Google Patents

Abstract

The present invention discloses an intelligent image recognition drowning warning system, which includes a camera, a deep learning image recognition unit and a warning device. The camera is used to capture human body activity images of the water area at each preset interval time to form water area motion images. The deep learning image recognition unit includes a bone joint point labeling module and a drowning action recognition module. The skeletal joint point labeling module is used to extract human body features from the image features of water activities, and the image is processed into corresponding skeletal joint point features. The drowning action recognition module establishes a feature database, which is used to input the features of each skeletal joint point received immediately and several times into the feature database, so as to identify the matching probability of each skeletal joint point feature and the drowning action feature sample. When the probability is greater than the preset probability, it is judged as drowning, and a package warning signal is output. The warning device is used to output the warning signal as warning information including the current position of the human body, so that through the integration of human bone joint point labeling and deep learning image technology, information such as continuous action posture, moving speed and vector coordinates can be obtained. To judge whether the swimmer's action is drowning, it has the characteristics of shortening the recognition calculation time, effectively improving the success rate of image recognition and speeding up the speed of drowning rescue.

Description

Intelligent image recognition drowning warning system

The present invention relates to a smart image recognition drowning warning system, especially a device that can judge swimmers' actions through information such as continuous action posture, moving speed, and vector coordinates through the integration of human bone joint point labeling and deep learning image technology. Drowning lifesaving techniques for drowning or not.

Generally speaking, water areas for water entertainment or water sports, such as water parks, beaches or swimming pools, are already one of the best places for modern people to relax their bodies and minds. In addition to achieving the effect of relieving heat and relieving heat, the water area can also achieve the functions of game entertainment, swimming and fitness. However, in the process of playing in water, swimming or engaging in water sports, the swimmer or swimmer may have an accident of drowning due to various factors such as the body or the environment. Save the precious life of the drowning person; on the contrary, if the lifeguard fails to rescue the drowning person in the first time due to the large number of people in the swimming pool, their own negligence or other factors, it is difficult to save the life of the drowning person. The safety of swimmers or swimmers in water activities is undoubtedly a severe challenge for related industries. Therefore, how to develop a set of water image monitoring technology that can effectively improve the safety of swimmers or swimmers has become a reality. Technical issues that industry players in related technical fields are eager to solve and challenge.

In order to improve the above deficiencies, industry players in related technical fields have developed a patent as shown in Invention Publication No. 201308264 "Water Area Warning Lifesaving System". Image recognition technology is used to improve the safety of water players or swimmers in water activities; however, this patent still has the following deficiencies:

1. The patent does not use artificial intelligence-based deep learning image recognition technology, so in addition to consuming more memory resources and recognition computing time, the success rate of image recognition cannot be effectively improved, resulting in a higher probability of misjudgment for drowning identification .

2. In this patent, all swimmers must be equipped with a set of positioning sending devices, so as to effectively monitor the positioning information of each swimmer; however, wearing a positioning sending device will not only make swimmers feel uncomfortable, It will also increase the construction cost of the hardware equipment.

3. The patent uses various sensor devices of the wearable device to detect whether the swimmer has drowned, but the wearable device will cause the swimmer to feel a foreign body while swimming and hinder the swimmer from swimming. Sensor equipment, in addition to wearable devices, may cause damage and malfunction of sensors when swimmers swim and other strenuous actions.

In addition, there is another patent as shown in the new No. M608940 "Intelligent Drowning Rescue Device". Although the patent can receive the facial expression of the drowning person, it is used to receive and transmit the facial expression of the drowning person to the display screen for real-time display and image recognition; however, the patent only sets an array of camera modules on the water surface, and does not The camera module is installed under the water surface and underwater, so that the captured image will have poor image recognition accuracy due to insufficient camera angle of view, and the patent does not have human bone joint points marked to identify whether swimmers are drowning. , so that it is impossible to judge whether the swimmer's action is drowning through information such as the action posture, moving speed, and vector coordinates of the joint points of the human skeleton.

In view of this, the conventional technology and the above-mentioned patents have not yet had a patent or paper on a deep learning image recognition technology that uses human bone joint points to mark whether swimmers are drowning It was proposed, and based on the urgent needs of related industries, the inventor finally developed a set of the present invention which is different from the above-mentioned conventional technologies and patents through continuous efforts in research and development.

The main purpose of the present invention is to provide an intelligent image recognition drowning warning system, which is mainly based on the integrated setting of human bone joint point labeling and deep learning image technology, and can judge swimming through information such as continuous action posture, moving speed, and vector coordinates. Therefore, it has the characteristics of shortening the recognition calculation time, effectively improving the success rate of image recognition, and speeding up the speed of drowning rescue. The technical means adopted to achieve the main purpose of the present invention include cameras, deep learning image recognition units and warning devices. The camera is used to capture human body activity images of the water area at each preset interval time to form water area motion images. The deep learning image recognition unit includes a bone joint point labeling module and a drowning action recognition module. The skeletal joint point labeling module is used to extract human body features from the image features of water activities, and the image is processed into corresponding skeletal joint point features. The drowning action recognition module establishes a feature database, which is used to input the features of each skeletal joint point received immediately and several times into the feature database, so as to identify the matching probability of each skeletal joint point feature and the drowning action feature sample. When the probability is greater than the preset probability, it is judged as drowning, and a package warning signal is output. The warning device is used to output the warning signal as warning information including the current position of the human body.

10: Camera

20:Deep Learning Image Recognition Unit

21: Skeleton joint point labeling module

22: Drowning Action Recognition Module

220: Feature database

30:Warning device

40: Waters

I: Moving images of water areas

Fig. 1 is the implementation schematic diagram of installing the video camera in the water area of the present invention.

Fig. 2 is the schematic diagram of the control implementation of the system architecture of the present invention

Fig. 3 is a schematic diagram of the implementation of human skeleton joint point labeling in the present invention.

Fig. 4 is a schematic diagram of the implementation of the training process of the deep learning of the present invention.

Fig. 5 is a schematic diagram of the implementation of the data processing flow of the deep learning of the present invention.

Fig. 6 is a schematic diagram of implementing the image recognition process of deep learning in the present invention.

Fig. 7 is a schematic diagram of implementation of a drowning action feature sample in a perspective of the present invention.

Fig. 8 is a schematic diagram of implementation of a drowning action feature sample from another perspective of the present invention.

In order to allow your review committee to further understand the overall technical characteristics of the present invention and the technical means to achieve the purpose of the present invention, specific embodiments and accompanying drawings are hereby described in detail:

Please refer to FIGS. 1-8 . In order to achieve the main purpose of the present invention, the specific embodiment includes multiple cameras 10 , a deep learning image recognition unit 20 and a warning device 30 and other technical features. A plurality of cameras 10 are set in a water area 40 (such as a water park, a beach or a swimming pool), and are used to capture 1 Zhang; but not limited thereto) image captures at least one human body activity image in the water area 40 to form a water area activity image I. The deep learning image recognition unit 20 has a deep learning training function to perform the image recognition function. The deep learning image recognition unit 20 includes a skeletal joint point labeling module 21 and a drowning action recognition module 22. The skeletal joint point labeling module 21 is used to extract at least one human body feature image from the water area moving image, and process the at least one human body feature image into corresponding bone joint point features. The drowning action recognition module 22 establishes a feature database 220 including at least one built-in feature sample of a plurality of drowning actions, and is used to input the features of each skeletal joint point received immediately and several times before into the feature database 220 in sequence, so as to The image recognizes the recognition result information of the coincidence probability between the characteristics of each bone joint point and the drowning action feature sample. When the coincidence probability is greater than the preset probability, it is determined that the human body is drowning, and a warning signal including the current position of the human body is output. The warning device 30 (as shown in FIG. 2 with a display screen and an audio device marked with a warning frame to mark the posture of the drowning person's skeleton) is used to output the warning signal as warning information including the current position of the human body.

Please refer to FIG. 1 , the number of the cameras 10 is plural, and the plurality of cameras 10 are time-synchronously captured and imaged into a plurality of moving images I of the water area. A plurality of cameras 10 are installed on the water surface of the water area 40 to sequentially capture the water surface of the water area 40 A plurality of moving images I of waters; two cameras 10 are installed under the water surface of the waters 40 to capture moving images I of the underwater activities of the waters 40; three cameras 10 are installed at the bottom of the waters 40 , used to capture the multiple waters moving images I of the underwater activities in the waters 40 facing upwards; the deep learning image recognition unit 20 uses the Mapping coordinate positioning method to coordinate the multiple waters moving images I to obtain time synchronization and different viewing angles Multiple water moving images I of the same human body, and send the multiple water moving images I to the skeletal joint point labeling module 21 for related image processing.

Specifically, the identification result information may be one of the moving speed, moving direction, current position or action posture of the human body. The skeletal joint feature can be at least one of the multiple joint points of the human body, the serial number of each joint point (as shown in FIG. 3 ), the position of each joint point or the vector coordinates of the relationship between each joint point.

In a specific application embodiment, the characteristic sample of the drowning action is when the human body does not leave the water area 40 within 20 seconds, and the connecting line segments of the joint points of the head are located underwater or swing between the water surface and underwater. In the state of stagnation, there is a fast and irregular swinging motion of the connecting line segments of the joint points of the arms or legs. Specifically, the fast and irregular swinging action is selected from one of abnormal arm swinging and abnormal kicking; the fast and irregular swinging action lasts for more than 3 seconds and the action frequency is 1-2 seconds per second. 4 times.

In another specific application embodiment, the characteristic sample of the drowning action is when the connection line segment of each joint point of the head is upward relative to the water surface, and the staying time in the water area 40 exceeds 1 minute, the connection line segment is upward relative to the water surface and contains The frequency of floating load and sinking is more than 30 times per minute, the connecting line segment is raised relative to the water surface, and only about one-fifth of the small area of the face is exposed to the water surface.

In yet another specific application embodiment, the characteristic sample of the drowning action is that the connecting line segments of the joint points of the arms of the human body protrude from the water surface and the frequency of waving or trembling exceeds 30 times per minute, and the joints of the arms Hold the dot-connected line segment up high and swing left and right The frequency of waving or trembling when the connecting line segments of each joint point of one arm protrudes out of the water is more than 30 times per minute, and the connecting line segment of each joint point of one arm is higher.

In addition, as shown in Figures 7-8, they are pictures of continuous action drowning action feature samples from different perspectives in the present invention.

Please refer to FIG. 6, the skeletal joint point labeling module 21 is a skeletal joint point labeling network; the drowning action recognition module 22 is a drowning action recognition model; the deep learning image recognition unit 20 executes the image The identification includes the following steps:

Step 1: Input every five moving images I of the water area into the skeletal joint point labeling network in sequence.

Step 2, extracting the at least one human body image from the skeleton joint point labeling network feature, and labeling the numbers of multiple joint points in sequence, and judging whether the joint points numbered 1, 8, and 11 are marked, and when the judgment result is no, Then end after deducting all bone joint points belonging to the human body.

Step 3, when the judgment result is yes, it is judged whether all relevant nodes of the human body are marked completely.

Step 4, when the judgment result is yes, deduct the five joint points of the head, and calculate the coordinates and speed changes of the five joint points to obtain the bone joint features.

Step 5, when the judgment result is negative, use the left-right symmetry of the human body to fill in the missing joint point coordinates, and proceed to Step 4.

Step 6: Input the skeleton and joint features into the drowning action recognition model, and use the recurrent convolutional neural network to identify the drowning action. When it is determined that the probability of drowning is greater than the normal probability, it is determined as drowning; when it is determined that the probability of drowning is low When the probability is normal, it is judged as normal without drowning.

In addition, the present invention mainly uses the images returned by the camera 10 in real time to carry out deep learning motion recognition, to judge the motions of all swimmers in the image in real time, and to identify swimmers who are drowning through the motion recognition network, and immediately Sending a message to the lifeguard on the scene or sending out an emergency report in time... Follow-up rescue processing methods, in addition, the method of using images for drowning motion recognition, swimmers will not have the foreign body sensation of wearing sensors, and do not have to worry about damage to the wearable device Affect the judgment of drowning situation occurs.

Firstly, the placement position of the camera 10 is explained. The camera 10 has cameras 10 placed above and below the water and with different viewing angles and directions. As shown in FIG. Next, in the action recognition, the coordinates of the joint points of the bones of the human body are used for action recognition processing. In terms of network architecture, as shown in Figure 2, the system architecture diagram is mainly divided into two stages. First, to extract the information of the joint points of the bones as Subsequent action recognition network input data, so it is necessary to use the skeletal joint point labeling network to mark the position of the skeletal joint point of the characters in the screen. Use the OpenPose skeletal joint point labeling network of Bottom-up. In the OpenPose labeling network, replace VGG-19 with Thin-MobileNet's lighter labeling network. After this stage of skeletal joint point coordinate labeling, it will be Output the joint point positions of all swimmers in each picture, and the vector coordinate information of the relationship between the joint points and the joint points. In the feature extraction of skeletal joint points, the coordinates of all skeletal joint points of each swimmer and the velocity displacement vector of each joint point position are selected as the input data for the follow-up action recognition network. These data are called follow-up use in the present invention. The skeletal joint point features for action classification are used as input data for the subsequent action recognition network. Because each video frame of the drowning action has a correlation with time, so in the selection of the action recognition network, the present invention selects a recursive network model specially used to process sequence data for action recognition. The test of the present invention uses RNN, LSTM The three recursive network models, BiLSTM and BiLSTM, are used for the action recognition of drowning, and the recognition results are tested. Finally, the BiLSTM with the lowest error rate in judging the drowning action is selected as the final action recognition network. The skeletal joint point annotation network used is a method that can better achieve Realtime. In the second stage, the final action recognition is performed using a recurrent neural network that can extract more sequence information.

As shown in Figure 5, the present invention comprises a data preprocessing step, when executing this step, Because the motion recognition data is obtained from the pre-shot video, the video must first be cropped and imaged every second, and a folder with the same file name as the video file should be created to store these images. Next, name the pictures in the folder starting from 1, as shown in Figure 5. According to the category of the images in the folder, add the category name in front of the folder name to create a blank file, because each image may contain images that do not belong to the category, so it is necessary to mark from which image to the first Several pictures are the movies that the present invention wants, as shown in Figure 5, after completing the blank file, it means that the data preprocessing step is completed.

As shown in Figures 3 and 4, the present invention includes a training process step. When this step is executed, the training picture will first be input into the bone joint point labeling network every five pictures according to the previously prepared blank file. When the image If the number 1, 8, and 11 of the central joints (the numbering diagram is shown in the Openpose-18-keypoints picture) are not marked at the same time, it will be judged that no person is marked, so the positions of the joint points will not be saved. If all three joint points are marked, it will be judged that the person has been found, and the bone joint point labeling will continue, because if the camera 10 image captures the side image of the person, there will be joint points on the other side that cannot be obtained from the image. It is directly marked in , so the left-right symmetry of the human body will be used to map the joint points on one side of the character to the other side according to the center point of the human body, and the joint points on the other side will be filled. Because I am afraid that the length of the data input to the deep learning network is not the same, because sometimes the head of the swimmer will not be under the water surface, and the network of bone joint points cannot mark the joint points of the head above the water. Therefore, the 5 joint points of the head of each character are deleted; then, the change between the same joint points of the same human body is calculated at each time, and the velocity vector of the joint point change is calculated. Combining the coordinates of the joint points and the velocity change vector of the joint points is the input data to the next stage of the action recognition network. Here, the recurrent neural network BiLSTM is used for action recognition. After network training, a drowning action recognition model that can identify whether drowning will be obtained.

As shown in Figure 6, the present invention includes identification process steps. When this step is performed, after the network model is trained first, a drowning function specially used to identify the drowning action will be obtained. Action recognition model. Therefore, when the image is input, the image will be input to the skeleton joint point labeling network. After marking the bone joint point labeling network, the information such as the coordinate position of the joint point and the velocity vector can be obtained. These information It will be input into the trained network model, and then according to the trained drowning action recognition model (that is, the network model), what is the probability that each swimmer can be successfully identified as drowning based on the skeletal joint points?

The experimental results of the present invention show that no matter whether the swimmer is frontal (as shown in Figure 7) or sideways (as shown in Figure 8) in the image, it can be identified. Because drowning is a continuous action, it can be input to the network every five images. Judging in the road model, the supervised learning method is used in the training, telling the computer from which image to which image in the training data belongs to the action of drowning, and from which image to which image belongs to the normal action , so the deep learning image recognition unit (such as the deep learning software built into the computer) will mark the bone joint points of these images according to the information and obtain the feature data of the bone joint points. The deep learning software can learn these features and then learn when to grasp What kind of joint and bone point features will be drowned when you get what kind of bone joint point features are normal. Therefore, it is impossible to know what the skeletal joints look like when it is drowning, and what the skeletal joints look like is normal. The present invention can only teach the computer image from which image to which image is drowning or normal in the training images.

The above is only a feasible embodiment of the present invention, and is not intended to limit the patent scope of the present invention. Any equivalent implementation of other changes based on the content, characteristics and spirit of the following claims should be Included in the patent scope of the present invention. The structural features of the invention specifically defined in the claims are not found in similar items, and are practical and progressive, and have met the requirements of an invention patent. I file an application in accordance with the law. I would like to ask the Jun Bureau to approve the patent in accordance with the law to maintain this invention. The legitimate rights and interests of the applicant.

10: Camera

40: Waters

Claims (8)

An intelligent image recognition drowning warning system, which includes: a plurality of cameras, which are installed in a water area, and are used to capture at least one human body activity image in the water area at each preset interval time as a water area moving image; a deep learning image recognition A unit with a deep learning training function to perform image recognition. The deep learning image recognition unit includes a skeletal joint point labeling module and a drowning action recognition module. The skeletal joint point labeling module is used for moving images of water areas Extracting at least one human body feature image, and processing the at least one human body feature image into corresponding skeletal joint point features; the drowning action recognition module establishes a feature database including at least one built-in plurality of drowning action feature samples, and uses The features of the skeletal joint points received immediately and several times before are sequentially input into the feature database, and the identification result information of the probability of matching between the features of the skeletal joint points and the sample of the drowning action feature is identified by image, when the When the matching probability is greater than a preset probability, it is determined that the at least one human body is drowning, and a warning signal including the current position of the at least one human body is output; and a warning device is used to output the warning signal as including the Warning information on the current position of at least one human body; among them, one of the plurality of cameras is set on the water surface of the water area to sequentially capture the moving images of the plurality of water areas of the water surface activities in the water area; the second of the plurality of cameras is set Under the water surface of the water area, the plurality of moving images of the water area are used to capture the underwater activities of the water area; thirdly, the plurality of cameras are installed at the bottom of the water area, and are used to capture the plurality of underwater activities of the water area upwards Water area moving images; the deep learning image recognition unit performs coordinate positioning on the plurality of water area moving images using the Mapping coordinate positioning method to obtain the plurality of water area moving images of the same at least one human body in time synchronization and different angles of view, and the plurality of The moving image of the water area is sent to the bone joint point labeling module for related image processing. The intelligent image recognition drowning warning system as described in Claim 1, wherein, the plurality of cameras are time-synchronously captured and imaged into a plurality of moving images of the water area. The intelligent image recognition drowning warning system as described in Claim 1, wherein the recognition result information is one of the information selected from the moving speed, moving direction, current position and action posture of the at least one human body; the bone joint point feature is At least one feature information selected from the plurality of joint points of the at least one human body, the number label of each joint point, the position of each joint point, and the vector coordinates of the relationship between each of the joint points. The intelligent image recognition drowning warning system as described in claim 3, wherein the drowning action feature sample is when the at least one human body has not left the water area within 20 seconds, and the connecting line segments of the joint points of the head are located at Underwater or swinging between the water surface and underwater to stagnate, and the joint line segments of the arms or legs have fast and irregular swings. The intelligent image recognition drowning warning system as described in claim 4, wherein the fast and irregular swinging action is selected from one of abnormal arm swinging and abnormal kicking; the fast and irregular swinging action time is It lasts for more than 3 seconds and the action frequency is 1~4 times per second. The intelligent image recognition drowning warning system as described in claim 3, wherein the characteristic sample of the drowning action is when the connection line segments of the joint points of the head are raised relative to the water surface and stay in the water area for more than 1 minute, the The connecting line segment is raised relative to the water surface and the frequency of floating and sinking exceeds 30 times per minute, the connecting line segment is raised relative to the water surface and only about one-fifth of the small area of the face is above the water surface. The intelligent image recognition drowning warning system as described in Claim 3, wherein the characteristic sample of the drowning action is that the connecting line segments of the joint points of the arms of the at least one human body protrude from the water surface and the frequency of waving or trembling exceeds 30 times per minute More than 30 times per minute, when the connecting line segments of each joint point of both arms are held high and swinging left and right, when the connecting line segment of each joint point of one arm sticks out of the water, the frequency of waving or trembling is more than 30 times per minute, and each arm of a single arm The connecting line segment of the joint points is held high to perform left and right or forward and backward swinging motions. The intelligent image recognition drowning warning system as described in claim 1, wherein the bones and joints The point labeling module is a network for skeletal joint point labeling; the drowning action recognition module is a drowning action recognition model; the deep learning image recognition unit includes the following steps when performing the image recognition: step 1, sequentially The moving image of the water area is input into the skeletal joint point labeling network; Step 2, the skeletal joint point labeling network features extract the at least one human body image and mark the numbers of the multiple joint points in sequence, and determine the numbers 1, 8 11. Whether the joint points are marked, when the judgment result is no, then end after deducting all bone joint points belonging to the human body; step 3, when the judgment result is yes, then judge whether all the relevant nodes of the human body are fully marked ; Step 4, when the judgment result is yes, deduct the five joint points of the head, and calculate the coordinates of the five joint points and the change of the joint speed, so as to obtain the characteristics of the bone joint; Step 5, when the judgment result is no, use Fill in the missing joint point coordinates according to the left and right symmetry of the human body, and enter the step 4; and step 6, input the skeletal joint features into the drowning action recognition model, and use the recursive convolution neural network with the feature database to establish The network recognizes the drowning action. When the probability of drowning is determined to be higher than the normal probability, it is judged to be drowning; when the probability of drowning is judged to be lower than the normal probability, it is judged to be normal without drowning.

Images