TWI778872B - Sensor fusion method for detecting a person's condition - Google Patents

Abstract

Disclosed is a sensor fusion method for detecting a person's condition, including: locating the position of at least one moving person in a detection area with a millimeter wave radar; capturing an RGB image or an IR image of the at least one moving person with a depth sensor camera, and generating two-dimensional (2D) human skeleton point information corresponding to the at least one moving person; and utilizing an artificial intelligence computing platform to execute a sensor fusion process, thereby synthesizing a three-dimensional (3D) human skeleton time series based on the data derived from the two-dimensional human skeleton point information. The method further includes that when the number of the synthesized three-dimensional human skeleton time series is greater than a threshold N, a motion recognition module in the artificial intelligence computing platform determines whether the at least one moving person has fallen, so as to determine whether to send a notification.

Description

A Sensor Fusion Method for Personnel Condition Detection

The present invention relates to a method for detecting a person's condition, and more particularly, to a sensor fusion method for detecting a person's condition.

In the current technology, there are many non-contact fall detection products or solutions, but the following problems are generally faced. For example, in a scheme using radar, the fall recognition rate may decrease when there are too many people in the scene. In addition, in the solution using image-based devices (eg, RGB cameras, IR cameras), the field of view (FOV) coverage is too small, and the depth camera has a distance limit (5m). Furthermore, the solution of using an image-type device can easily keep the system under high load (ie, the computation amount is relatively complex) for a long time, which has the disadvantages of affecting product life and wasting energy.

To sum up, how to provide an effective and accurate method for detecting personnel status is an important issue that the industry needs to think about.

In other words, the embodiments of the present invention mainly provide a sensor fusion method for personnel condition detection, which can effectively improve product life and avoid energy waste. In addition, by combining data from different sources, the information obtained is less uncertain than using these sources alone. That is, the human condition can be more accurately discriminated.

In view of the above, an aspect of the present disclosure provides a sensor fusion method for personnel condition detection, including: locating the position of at least one moving person in a detection area with a millimeter-wave radar; using a depth-sensing camera Capture an RGB image or an IR image of the at least one moving person, and generate a two-dimensional human skeleton point information corresponding to the at least one moving person; perform a sensor fusion with an artificial intelligence computing platform fusion) program, using the data derived from the two-dimensional human skeleton point information to synthesize a three-dimensional human skeleton time series, wherein the artificial intelligence computing platform is coupled to the millimeter-wave radar and the depth-sensing camera; When the number of three-dimensional human skeleton time series is greater than a threshold N, a motion recognition module in the artificial intelligence computing platform is used to determine whether the at least one moving person has fallen, so as to determine whether to issue a notification.

According to one or more embodiments of the present disclosure, before executing the sensor fusion process, it further comprises: a mapping step, first converting the 2D human skeleton point information into a 3D human skeleton point information; and a signal The processing step is to obtain the average speed of a point cloud cluster from the signal of the millimeter-wave radar using a clustering algorithm; wherein, the three-dimensional human skeleton point information and the average speed of the point cloud cluster are combined in the sensor fusion process Then, the three-dimensional human skeleton time series is generated.

According to one or more embodiments of the present disclosure, the depth-sensing camera further utilizes Time of Flight (ToF) technology, structured light technology, or active stereo vision ) technology to obtain a depth information of the at least one moving person for obtaining the three-dimensional human skeleton point information in the mapping step.

According to one or more embodiments of the present disclosure, it further includes: an ID number generating step, when the RGB image or the IR image shows that the at least one moving person is a plurality of persons, assigning a corresponding ID number to each moving person ID number, and then performing the mapping step; an ID number comparison step, comparing each of the ID numbers with an ID number stored in a memory, wherein the memory is coupled to the artificial intelligence computing platform; a data concatenation step of concatenating the detected three-dimensional human skeleton time series and the time series of the same ID number when the result of the ID number comparison step shows the same; When the number is greater than the threshold N, the motion recognition module in the artificial intelligence computing platform is used to determine whether the at least one moving person has fallen, and then determine whether to issue the notification.

According to one or more embodiments of the present disclosure, between the ID number generation step and the ID number comparison step, the step further includes: a coordinate system conversion step of changing the origin of the coordinate system from the depth sensing camera The center is converted to the origin of the human skeleton, wherein the origin of the human skeleton is the intersection of the line connecting the shoulder and the head.

According to one or more embodiments of the present disclosure, when the result of the ID number comparison step is shown to be different, an ID number is added and the three-dimensional human skeleton time series storage space is created, and the detected The three-dimensional human skeleton time series is stored in the memory, and then returns to the ID number generating step.

According to one or more embodiments of the present disclosure, the depth-sensing camera, within the detection area of the millimeter-wave radar, pairs the at least one moving person at different positions one by one according to a preset priority order. Capture the RGB image or the IR image.

According to one or more embodiments of the present disclosure, the two-dimensional human skeleton point information is obtained by an attitude estimation and tracking module in the artificial intelligence computing platform according to an attitude estimation and tracking model, wherein The backbone network of the pose estimation and tracking model uses a convolutional neural network architecture.

According to one or more embodiments of the present disclosure, the motion recognition module determines whether the at least one moving person falls by using a deep learning model or a machine learning classifier.

According to one or more embodiments of the present disclosure, the millimeter-wave radar will repeatedly perform the detection action in the detection area until the at least one moving person appears and confirms the position, and then by working with the artificial The intelligent computing platform is coupled to a motor to adjust the shooting direction and angle of the depth-sensing camera.

In order to facilitate your reviewers to have a further understanding and understanding of the purpose, shape, structure and device features of the present invention and their effects, the following examples are given in conjunction with the drawings.

The following disclosure provides different embodiments or examples for implementing different features of the provided subject matter. The specific examples of components and arrangements described below are for the purpose of simplifying the present disclosure and are not intended to be limiting; the size and shape of the components are not limited by the disclosed ranges or values, but may depend on the processing conditions of the components or the desired characteristic. For example, technical features of the invention are described using cross-sectional illustrations that are schematic illustrations of idealized embodiments. Thus, variations in the shapes of the illustrations due to manufacturing processes and/or tolerances are foreseeable and should not be limited thereto.

Furthermore, spatially relative terms such as "below", "below", "below", "above" and "above" are used to facilitate the description of the elements depicted in the drawings or relationship between features; further, spatially relative terms encompass different orientations of elements in use or operation in addition to the orientation depicted in the illustrations.

First of all, it should be noted that the embodiment of the present invention utilizes sensor fusion technology, by combining data obtained by different sensors (eg, millimeter-wave radar, depth-sensing camera) to generate data that cannot be generated by a single sensor information provided by the sensor.

In the embodiment of the present invention, the millimeter wave radar is used to detect whether there is a person in a large area of the environment, and if there is a person, the position of the human body is located. Next, turn the depth-sensing camera to lock the human body. Then, an artificial intelligence (AI) computing platform is used to extract the three-dimensional (3D) human skeleton and track the target with AI deep learning technology. Finally, the movement speed of the center point of the human body detected by the millimeter-wave radar is used to identify whether a fall has occurred.

In the following, the sensor fusion method and the applied system of the personnel condition detection in the embodiments of the present application are described with the drawings.

First, please refer to FIG. 1 . FIG. 1 is a schematic diagram illustrating the appearance of a hardware system according to an embodiment of the present invention. As shown in FIG. 1 , the hardware system 100 includes an artificial intelligence computing platform 10 , a millimeter-wave radar 20 , a depth-sensing camera 30 , a motor 40 and a memory 50 . The artificial intelligence computing platform 10 is respectively coupled to the millimeter wave radar 20 , the depth sensing camera 30 , the motor 40 and the memory 50 .

Then, please refer to FIG. 2A , which is a schematic diagram illustrating the operation of the hardware system in the embodiment of the present invention. FIG. 2A is a top view illustrating an embodiment of the present invention that first uses the millimeter wave radar 20 to search for the position of the human body in the field, and then determines the rotation direction and angle of the depth-sensing camera 30 . In the embodiment of the present invention, the millimeter-wave radar 20 will repeatedly perform the detection action in the detection area 110 until at least one moving person 120 appears and confirms the position, and then uses a motor 40 coupled to the artificial intelligence computing platform 10 . Adjust the shooting direction and angle of the depth-sensing camera 30 .

As shown in FIG. 2A , the millimeter-wave radar 20 of the hardware system 100 divides its detection area 110 into four quadrants. When the millimeter-wave radar 20 detects a moving person 120 in the detection area 110 , it locates the moving person 120 . Person 120's position (x, y). For example, in an embodiment of the present invention, the mobile person 120 is located in the fourth quadrant. Next, a processor (not shown) in the AI computing platform 10 calculates the rotation angle α of the motor 40 according to the position (x, y) of the moving person 120 in the fourth quadrant, and substitutes it into the following (Equation 4). If the position (x, y) of the moving person 120 is in the first, second or third quadrant, then substitute the corresponding (Equation 1), (Equation 2) or (Equation 3) to calculate the rotation angle α of the motor 40 .

(式1) Quadrant 1:

(Formula 1)

(式2) Second quadrant:

(Formula 2)

(式3) The third quadrant:

(Formula 3)

(式4) Fourth quadrant:

(Formula 4)

Here, the rotation angle α is defined as the included angle between the line connecting the moving person 120 and the center of the hardware system 100 and the coordinate axis X.

In the embodiment of the present invention, the motor 40 is used to adjust the direction and angle of the depth-sensing camera 30 , so the depth-sensing camera 30 is rotated by the motor 40 by a rotation angle α toward the position of the moving person 120 . Then, the depth-sensing camera 30 performs a human skeleton detection operation and obtains depth information within its own field of view (FOV).

In addition, please refer to FIG. 2B . FIG. 2B is a schematic diagram of generating a four-dimensional point cloud by a millimeter-wave radar and performing a clustering algorithm according to an embodiment of the present invention. As shown in FIG. 2B, in the embodiment of the present invention, in step 200, data collection is performed. Then, in step 210, a single frame processing procedure is performed, that is, after the millimeter wave radar 20 is used to generate a four-dimensional (4D) point cloud (x, y, z, v), the The center point 220 of each cluster and the average velocity. Where (x, y, z) represents the position of each point, and v represents the velocity of each point. In the embodiment of the present invention, the generation of the four-dimensional (4D) point cloud (x, y, z, v) is to use a frequency modulated continuous wave radar (Frequency Modulated Continuous Waveform radar, FMCW radar) to transmit millimeter waves and record the Reflection of the scene, then compute a sparse point cloud and filter out points corresponding to static objects. In the embodiment of the present invention, the clustering algorithm adopts the density-based clustering (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) algorithm to cluster the point cloud and find the center point 220 and the center point of each cluster. average speed.

In addition, please refer to FIG. 2C , which is a schematic diagram illustrating the operation of the hardware system in the embodiment of the present invention. 2C is a top view illustrating that in a multi-person scene, since the field of view 130 of the depth-sensing camera 30 of the hardware system 100 cannot cover the detection area 110 of the millimeter-wave radar 20 at one time, the priority of locking the target is based on a preset conditions to resolve. For example, the moving person 230 located in the first quadrant has the first priority according to the preset conditions because of the sharp change of speed v in the Z-axis direction of the point cloud, so the depth-sensing camera 30 will rotate to the moving person first. 230 direction, carry out the relevant procedures. After the moving person 230 is processed, the moving person 240 that is located in the fourth quadrant and has obvious displacement in the X-axis or Y-axis direction of the point cloud is processed. According to the preset conditions, it has the priority of the second order, so the depth-sensing camera 30 will then rotate to the direction of the moving person 240, and perform related procedures. After the moving person 240 is processed, the third quadrant with the largest number of people detected in each range is then processed. This situation has the third priority according to the preset conditions, so the depth sensing camera 30 will then rotate to the moving group. 250 direction, carry out the relevant procedures. The above FIG. 2C is only used to illustrate an embodiment of determining the priority of the millimeter-wave radar 20 in a multi-person scene. Of course, in other embodiments, the preset conditions can also define different priorities according to different design requirements. order.

Next, please refer to FIG. 3A . FIG. 3A is a flowchart illustrating a sensor fusion method for human condition detection according to an embodiment of the present invention. As shown in FIG. 3A , the sensor fusion method for personnel condition detection according to an embodiment of the present invention includes steps S10 to S130 , and each step is described below with reference to FIGS. 1 and 2A to 2C .

In step S10, a millimeter-wave radar 20 is used to detect the moving human body, and the information is sent back to the processor in the artificial intelligence computing platform 10 (not shown). In various embodiments, the mobile person system is at least one mobile person, such as a mobile person or a mobile crowd, as in a real environment.

In step S20 , the processor (not shown) in the artificial intelligence computing platform 10 determines whether there is a person in the detection area 110 of the millimeter-wave radar 20 . If the determination result is no, that is, there is no one, then go back to step S10 to continue detecting the moving human body with the millimeter wave radar 20 . If the determination result is that there is a person, such as the mobile person 120 in FIG. 2A , the mobile personnel 230 , 240 or the mobile crowd 250 in FIG. 2C , then go to the next step S30 .

In step S30, the processor (not shown) in the artificial intelligence computing platform 10 locates the position of the detected moving human body and rotates the depth-sensing camera 30 by the motor 40 to align the detected movements one by one human body.

In step S40 , an RGB image or an IR image of the detected moving human body is captured one by one with the depth sensing camera 30 .

In step S50, two-dimensional human skeleton estimation and tracking are performed. The processor (not shown) in the artificial intelligence computing platform 10 generates a two-dimensional human skeleton point information corresponding to the detected moving human body according to the RGB image or the IR image in step S40.

In step S60 , the processor (not shown) in the artificial intelligence computing platform 10 determines whether there is a person within the field of view of the depth sensing camera 30 . If the determination result is no, that is, there is no one, then go back to step S10 to continue detecting the moving human body with the millimeter wave radar 20 . If the judgment result is yes, that is, there is someone, the process proceeds to the next step S70.

Next, in step S70 and step S80, a sensor fusion program is executed by the artificial intelligence computing platform 10, as described below.

In step S70, according to the two-dimensional human skeleton point information obtained in step S50, a mapping step S701 is performed by a mapping module (not shown) in the artificial intelligence computing platform 10, and the two-dimensional human skeleton point information is converted into Converted into three-dimensional human skeleton point information represented by (x _m , y _m , z _m ), where m is a natural number, as shown in FIG. 3B . In addition, a signal processing step S702 is executed by the processor (not shown in the figure) in the artificial intelligence computing platform 10, and the clustering algorithm is used to obtain the average velocity _of the point cloud cluster represented by v1 from the signal of the millimeter-wave radar 20, that is, the so-called millimeter Wave radar point cloud velocity extraction, as shown in Figure 3B. It will be further described in FIG. 3B later. In the embodiment of the present invention, the depth-sensing camera 30 further utilizes Time of Flight (ToF) technology, structured light technology or active stereo vision technology to obtain at least A depth information of a moving person is used to obtain the three-dimensional human skeleton point information in the mapping step.

In step S80, the three-dimensional human skeleton point information obtained in step S70 and the point cloud clustering average velocity are combined into a three-dimensional human skeleton time series, as shown in FIG. 3B. It will be further described in FIG. 3B later.

In step S90 , whether the number of the three-dimensional human skeleton time series synthesized in step S80 is greater than a threshold N is determined by the processor (not shown) in the artificial intelligence computing platform 10 . When the number of the three-dimensional human skeleton time series is not greater than a threshold N (ie, N pictures), the process returns to step S50. When the number of the three-dimensional human skeleton time series is greater than a threshold N, the next step S100 is executed.

In step S100, the artificial intelligence computing platform 10 calls a motion recognition module (not shown in the figure), and the motion recognition module is used to determine whether at least one mobile person falls.

In step S110, when the motion recognition module determines that no one has fallen, the process returns to step S50. When the motion recognition module determines that someone has fallen, the next step S120 is executed.

In step S120 , when the motion recognition module determines that the situation of someone falling is continuous and the number of times is greater than or equal to K, a notification is sent to notify that someone has fallen in step S130 . If the motion recognition module determines that the situation of someone falling does not meet the condition of continuous occurrence and the number of times is greater than or equal to K, the process returns to step S50.

Next, please refer to FIG. 3B . FIG. 3B illustrates the process of sensor fusion in FIG. 3A . As shown in FIG. 3B , in the embodiment of the present invention, the process of sensor fusion is to fuse the data of the depth-sensing camera 30 and the millimeter-wave radar 20 . As also mentioned above, according to the 2D human skeleton point information obtained from the RGB image or IR image of the depth sensing camera 30, the 3D human skeleton point information 300 is mapped in step S701, including (x ₁ , y ₁ , z ₁ ), (x ₂ , y ₂ , z ₂ ), (x ₃ , y ₃ , z ₃ ), …, (x _m-2 , y _m-2 , z _m-2 ), (x _m-1 , y _m-1 , z _m-1 ) and (x _m , y _m , z _m ) have m data points in total. In addition, according to the signal obtained from the millimeter-wave radar 20, signal processing is performed in step S702 to obtain the average velocity 310 _of the point cloud cluster represented by v1. Then, a three-dimensional human skeleton time series 320 is synthesized by combining the obtained three-dimensional human skeleton point information and the point cloud cluster average velocity through sensor fusion technology.

Next, please refer to FIG. 4 . FIG. 4 is a flowchart illustrating a fall recognition algorithm for a multi-person scene according to an embodiment of the present invention. As shown in FIG. 2C , FIG. 3A and FIG. 4 , in the case of a multi-person scene, the depth-sensing camera 30 pairs different positions one by one in the detection area 110 of the millimeter-wave radar 20 according to a preset priority order. The at least one moving person captures an RGB image or an IR image. Next, the algorithm flow of the fall recognition algorithm in the multi-person scene of steps S400 to S480 is performed.

In step S400, when the RGB image or the IR image shows that the at least one moving person is a plurality of persons, an ID number generation step is performed, and an ID number corresponding to each moving person is given, and the multi-person two-dimensional Human skeleton estimation and tracking to obtain 2D human skeleton point information.

In step S410, a mapping step is performed to map the two-dimensional human skeleton point information into three-dimensional human skeleton point information.

In step S420, coordinate system conversion is performed to convert the origin of the coordinate system from the center of the depth sensing camera 30 to the origin of the human skeleton, wherein the origin of the human skeleton is the intersection of the line connecting the shoulder and the head.

In step S430 , the ID numbers are compared, that is, each ID number is compared with the ID numbers stored in the memory 50 . If the comparison result shows that they are not the same, then in step S440, an ID number is added and a three-dimensional human skeleton time series storage space is created. Then, in step S450, the detected three-dimensional human skeleton time series is stored in the memory 50, and the process returns to step S400. If the comparison result in step S430 shows the same, then proceed to step S460 to perform data concatenation, concatenate the detected 3D human skeleton time series with the time series of the same ID number, and proceed to the next step S470.

In step S470, when the number of three-dimensional human skeleton time series is not greater than a threshold N, the process returns to step S400. When the number of three-dimensional human skeleton time series is greater than a threshold N, then go to step 480, use the motion recognition module (not shown in the figure) in the artificial intelligence computing platform 10 to determine whether the at least one moving person has fallen, and decide Whether to issue a notification to notify someone of a fall.

In addition, please refer to FIG. 5 . FIG. 5 is a functional schematic diagram of an attitude estimation and tracking module according to an embodiment of the present invention. As shown in FIG. 5 , the pose estimation and tracking module (not shown) in the artificial intelligence computing platform 10 is used to input the RGB image 502 or the IR image 504 into the human skeleton estimation and tracking model 508 to obtain a two-dimensional human skeleton The point information 510 is then combined with the depth information 506 to map the 3D human skeleton point information 512 .

In addition, please refer to FIG. 6 . FIG. 6 is a functional schematic diagram of a motion recognition module according to an embodiment of the present invention. As shown in FIG. 6, the motion recognition module (not shown) in the artificial intelligence computing platform 10 is used to input the three-dimensional human skeleton point information corresponding to the time series t-2, t-1, t into the motion recognition model 600, and then According to the estimation category 610 , the action of the at least one mobile person is identified, and an estimation result 620 is obtained. Here, the time series t-2, t-1, and t are only examples and are not used to limit the present invention. In fact, the number of time series input to the action recognition model 600 is determined according to actual training needs. In the embodiment of the present invention, the action recognition model 600 is, for example, a fall recognition model. In addition, the motion recognition module 600 may be a deep learning model (RNN, LSTM or GCN) architecture or a machine learning classifier (SVM).

In addition, please refer to FIG. 7 . FIG. 7 is a functional schematic diagram of a posture estimation and tracking module according to another embodiment of the present invention. As shown in FIG. 7 , using the posture estimation and tracking module (not shown) in the artificial intelligence computing platform 10, the IR images 720 and 710 corresponding to the time series t-1 and t and the center point of the time t-1 A heatmap 730 is input to the human skeleton estimation and tracking model 740, and images 750, 760 and 770 are obtained. In image 750, there are bounding boxes to estimate the number and location of detected human bodies in the frame. In image 760, the 2D skeleton estimates body joints and keypoints of important parts. In the image 770, the offset can be used to estimate or predict the coordinate displacement of the front and rear frames to track the human ID number. In an embodiment of the present invention, the backbone network of the pose estimation and tracking model 740 may be a different form of convolutional neural network (CNN) architecture. Moreover, different tasks share the backbone model, which reduces the computational burden of the system.

Next, please refer to FIG. 8 , which is a schematic diagram illustrating a time series of a three-dimensional human skeleton according to an embodiment of the present invention. As shown in FIG. 8 , frame n-4, frame n-3, frame n-2, frame n-1, and frame n represent consecutive frames to illustrate a three-dimensional human skeleton time series, where n is a natural number . In addition, the component symbol P is the origin of the human skeleton. As described in the previous step S420, when the origin of the coordinate system is converted from the depth-sensing camera 30 as the center to the human skeleton origin P as the center, the depth-sensing camera can be excluded. The influence of the 30 angle of view on the fall recognition model and other action recognition models.

The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced. Without departing from the spirit and scope of the technical solutions of the present invention.

10: AI computing platform 40: Motor 20: Millimeter-wave radar 50: Memory 30: Depth sensing camera 100: Hardware system 110: Detection area 750: Image 120: Mobile personnel 760: Image 130: Field of view 770: Image 200, 210: Steps S10~S130: Step 220: Center point S701, S702: Step 230: Moving people S400~S480: Step 240: Moving people P: Human skeleton origin 250: Moving crowd v: Speed 300: Three-dimensional human skeleton Point Info v ₁ : Point Cloud Cluster Average Velocity 310: Point Cloud Cluster Average Velocity X, Y, Z: Coordinate Axis 320: 3D Human Skeleton Time Series (x, y): Position 502: RGB Image 504: IR Image 506: Depth Information 508: Human Skeleton Estimation and Tracking Model 510: 2D Human Skeleton Point Information 512: 3D Human Skeleton Point Information 600: Action Recognition Model 610: Estimation Category 620: Estimated Results 710: IR Image 720: IR Image 730: Center point heatmap 740: Human skeleton estimation and tracking model (x, y, z, v): 4D point cloud t: time α: rotation angle

In order to make the above-mentioned and other objects, features, advantages and embodiments of the present invention more easily understood, the descriptions of the accompanying drawings are as follows: FIG. 1 is a schematic diagram showing the appearance of a hardware system according to an embodiment of the present invention. FIG. 2A is a schematic diagram illustrating the operation of the hardware system in the embodiment of the present invention. FIG. 2B is a schematic diagram illustrating a millimeter-wave radar to generate a four-dimensional point cloud and perform a clustering algorithm according to an embodiment of the present invention. FIG. 2C is a schematic diagram illustrating the operation of the hardware system in the embodiment of the present invention. FIG. 3A is a flowchart illustrating a sensor fusion method for human condition detection according to an embodiment of the present invention. FIG. 3B illustrates the process of sensor fusion in FIG. 3A . FIG. 4 is a flowchart illustrating an algorithm for identifying a fall in a multi-person scene according to an embodiment of the present invention. FIG. 5 is a functional schematic diagram of an attitude estimation and tracking module according to an embodiment of the present invention. FIG. 6 is a functional schematic diagram of a motion recognition module according to an embodiment of the present invention. FIG. 7 is a functional schematic diagram of a posture estimation and tracking module according to another embodiment of the present invention. FIG. 8 is a schematic diagram illustrating a three-dimensional human skeleton time series according to an embodiment of the present invention.

In accordance with common practice, the various features and elements in the drawings are not drawn to scale, but are drawn in order to best represent specific features and elements relevant to the present invention. Furthermore, the same or similar reference numerals are used to refer to similar elements and parts among the different figures.

S10~S130: Steps

Claims (9)

A sensor fusion method for human condition detection, comprising: locating the position of at least one moving person in a detection area with a millimeter-wave radar; capturing an RGB image of the at least one moving person with a depth-sensing camera or An IR image is generated, and a two-dimensional human skeleton point information corresponding to the at least one moving person is generated; an artificial intelligence computing platform is used to execute a sensor fusion (sensor fusion) program, using the two-dimensional human skeleton point information to derive Synthesizing the data into a three-dimensional human skeleton time series, wherein the artificial intelligence computing platform is coupled to the millimeter-wave radar and the depth sensing camera; and when the number of the three-dimensional human skeleton time series is greater than a threshold N, the artificial intelligence computing platform An internal motion recognition module determines whether the at least one moving person falls and decides whether to issue a notification; before executing the sensor fusion process, it further includes: a mapping step, first converting the two-dimensional human skeleton point information forming a three-dimensional human skeleton point information; and a signal processing step, using a clustering algorithm to obtain a point cloud cluster average velocity from the signal of the millimeter wave radar; wherein, the three-dimensional human skeleton point information and the point cloud cluster average velocity are in the sensor After the combination of the detector fusion program, the three-dimensional human skeleton time series is generated, which is used by the motion recognition module as a basis for judging whether the at least one moving person falls. The sensor fusion method for personnel condition detection as claimed in claim 1, wherein the depth-sensing camera further utilizes Time of Flight (ToF) technology, structured light technology or active The active stereo vision technology obtains a depth information of the at least one moving person, and is used to obtain the three-dimensional human skeleton point information in the mapping step. The sensor fusion method for personnel condition detection as described in claim 2, further comprising: An ID number generation step, when the RGB image or the IR image shows that the at least one moving person is a plurality of persons, assigning each moving person a corresponding ID number, and then performing the mapping step; an ID number comparison step, the Each of the ID numbers is compared with an ID number stored in a memory, wherein the memory is coupled to the artificial intelligence computing platform; in a data concatenation step, when the results of the ID number comparison step are shown to be the same, concatenate the data. The detected three-dimensional human skeleton time series and the time series with the same ID number; and when the number of the three-dimensional human skeleton time series is greater than the threshold N, use the motion recognition module in the artificial intelligence computing platform to determine the at least one Whether the mobile person falls, decide whether to issue the notification. The sensor fusion method for personnel condition detection according to claim 3, wherein between the ID number generation step and the ID number comparison step, further comprising: a coordinate system conversion step, converting the origin of the coordinate system from the The center of the depth-sensing camera is converted to the origin of the human skeleton, wherein the origin of the human skeleton is the intersection of the line connecting the shoulder and the head. The sensor fusion method for personnel condition detection according to claim 4, wherein when the result of the ID number comparison step is shown to be different, an ID number is added and a storage space for the three-dimensional human skeleton time series is created, and store the detected three-dimensional human skeleton time series in the memory, and then return to the ID number generating step. The sensor fusion method for personnel condition detection as claimed in claim 5, wherein the depth-sensing camera in the detection area of the millimeter-wave radar, one by one according to a preset priority order to the at least one of different positions A moving person captures the RGB image or the IR image. The sensor fusion method for personnel condition detection as claimed in claim 1, wherein the two-dimensional human skeleton point information is obtained by an attitude estimation and tracking module in the artificial intelligence computing platform according to a A pose estimation and tracking model is obtained, wherein the backbone network of the pose estimation and tracking model uses a convolutional neural network architecture. The sensor fusion method for personnel condition detection according to claim 1, wherein the motion recognition module determines whether the at least one moving person falls by using a deep learning model or a machine learning classifier. The sensor fusion method for personnel condition detection as claimed in claim 1, wherein the millimeter-wave radar repeatedly performs detection actions in the detection area until the at least one moving person appears and confirms the position, and then uses A motor coupled to the artificial intelligence computing platform adjusts the shooting direction and angle of the depth-sensing camera.

Images