WO2024090786A1 - Radar data-based fall detection model training method - Google Patents

Abstract

A fall detection model training method according to an embodiment of the present disclosure comprises the steps of: training an object tracking model on the basis of radar data; and training a fall detection model on the basis of a feature extracted from the radar data through the object tracking model, wherein the step of training the fall detection model comprises the steps of: acquiring a first training data set including label information that is related to whether a fall has occurred, and is allocated to the radar data and camera data matched to the radar data; storing one or more features extracted from the radar data through the object tracking model so as to acquire a feature queue; inputting the features stored in the feature queue to the fall detection model; acquiring a distribution prediction value related to whether a fall has occurred, the value being output through the fall detection model on the basis of the features; and updating the fall detection model on the basis of the distribution prediction value and the label information of the first training data set.

Description

Learning method of fall detection model based on radar data

The present disclosure relates to a method of learning a fall detection model based on radar data and a fall detection device that performs the same. Specifically, the present disclosure relates to a method of learning an object tracking model for tracking an object in radar data, a method of learning a fall detection model to detect whether an object has fallen, and a fall detection device that performs the same.

As artificial intelligence technology develops, it is being used in various industrial fields. In particular, as radar processing technology using artificial intelligence techniques becomes more advanced, technology that detects falling objects based on radar is attracting attention.

Conventionally, a fall of an object was detected using Doppler images. Doppler images are calculated as the change in position over a certain period of time. Conventionally, the speed of an object located at a specific location was measured using a Doppler image, and if the measured speed of the object was detected to be above a certain speed, it was judged to be a fall. However, fall detection technology using Doppler images has limitations in that it can only measure speed in a specific direction (range bin), and due to these limitations, there is a limitation in that falls in directions other than the specific direction cannot be detected. . Furthermore, fall detection technology using Doppler images had a limitation in that it could only analyze a certain area, not the entire area measured by radar.

Accordingly, there is a need to develop a new technology to detect the fall of an object based on radar data using artificial intelligence technology.

The problem that the present disclosure aims to solve is to provide a method of learning a neural network model for tracking an object based on radar data and detecting whether the object has fallen, and a fall detection device that performs the same.

The problem to be solved by the present disclosure is not limited to the above-mentioned problems, and problems not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings. .

A method for learning a fall detection model according to an embodiment of the present disclosure includes training an object tracking model based on radar data; and training a fall detection model based on features extracted from the radar data through the object tracking model, wherein training the fall detection model includes the radar data and a camera matched with the radar data. Obtaining a first learning data set consisting of label information related to whether or not a fall has been assigned to the data; acquiring a feature queue by storing at least one feature extracted from the radar data through the object tracking model; Inputting at least one feature stored in the feature queue into the fall detection model; Obtaining a classification value related to whether a fall has occurred output through the fall detection model based on the at least one feature; And it may include updating the fall detection model based on the classification value and the label information of the first learning data set.

A fall detection device according to an embodiment of the present disclosure is a processor configured to train an object tracking model based on radar data and train a fall detection model based on features extracted from the radar data through the object tracking model. Including, wherein the processor acquires a first learning data set consisting of label information related to whether a fall is assigned to the radar data and camera data matched with the radar data, and obtains a first learning data set from the radar data through the object tracking model. Obtain a feature queue by storing at least one extracted feature, input at least one or more features stored in the feature queue into the fall detection model, and output through the fall detection model based on the at least one or more features. It may be configured to train the fall detection model by obtaining a classification value related to whether a fall has occurred, and updating the fall detection model based on the classification value and the label information of the first learning data set.

The technical solutions of the present disclosure are not limited to the above-described technical solutions, and technical solutions not mentioned are clearly understood by those skilled in the art from this specification and the attached drawings. It could be.

According to the fall detection model learning method and fall detection device according to an embodiment of the present disclosure, it is possible to track an object and detect a fall based only on radar data. Through this, it is possible to provide the effect of tracking objects and detecting falls using only radar data, even in environments where cameras are difficult to install.

According to the fall detection model learning method and fall detection device according to an embodiment of the present disclosure, a fall detection model is trained using a learning data set configured to detect falls occurring in various postures, regardless of the fall posture. It can provide the effect of detecting falls.

According to the fall detection model learning method and fall detection device according to an embodiment of the present disclosure, the time required for learning the fall detection model is reduced by using features extracted through the object tracking model in learning the fall detection model. You can do it.

According to the fall detection model learning method and fall detection device according to an embodiment of the present disclosure, the accuracy of the fall detection task is improved while maintaining the accuracy of the object tracking task at a certain level by alternately learning the object tracking model and the fall detection model. It can be increased.

According to the fall detection model learning method and fall detection device according to an embodiment of the present disclosure, the object tracking model and the fall detection model are trained or executed using a two-dimensional heatmap, thereby reducing the relatively low performance radar ( The object tracking model and fall detection model can also be run on radar's own processor.

The effects of the present disclosure are not limited to the effects described above, and effects not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings.

1 is a schematic diagram of a fall detection device according to an embodiment of the present disclosure.

FIG. 2 is a diagram illustrating an aspect of training an object tracking model according to an embodiment of the present disclosure and an aspect of calculating location information of an object using the learned object tracking model.

FIG. 3 is a diagram illustrating an aspect of learning a fall detection model according to an embodiment of the present disclosure and an aspect of detecting a fall using a fall detection model that has completed learning.

FIG. 4 is a diagram illustrating an aspect of learning an object tracking model and a fall detection model according to an embodiment of the present disclosure.

Figure 5 shows an aspect of calculating an object's location information and a classification value related to whether a fall occurred from radar data using a fully learned object tracking model and a fully learned fall detection model according to an embodiment of the present disclosure. It is a drawing.

Figure 6 is a flow chart illustrating a method for training a fall detection model according to an embodiment of the present disclosure.

Figure 7 is a flow chart specifying the steps of training an object tracking model according to an embodiment of the present disclosure.

Figure 8 is a flow chart specifying the steps of training a fall detection model according to an embodiment of the present disclosure.

Figure 9 is a flowchart showing an aspect of calculating location information of an object using a fully trained object tracking model and detecting a fall using a fully trained fall detection model according to an embodiment of the present disclosure.

The above-described objects, features and advantages of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings. However, since the present disclosure can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail below.

Like reference numerals throughout the specification in principle refer to the same elements. In addition, components with the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals, and overlapping descriptions thereof will be omitted.

If it is determined that a detailed description of a known function or configuration related to the present disclosure may unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In addition, numbers (eg, first, second, etc.) used in the description of this specification are merely identifiers to distinguish one component from another component.

In addition, the suffixes “module” and “part” for components used in the following examples are given or used interchangeably only for the ease of writing the specification, and do not have distinct meanings or roles in themselves.

In the following examples, singular terms include plural terms unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have mean the presence of features or components described in the specification, and do not exclude in advance the possibility of adding one or more other features or components.

In the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and the present disclosure is not necessarily limited to what is shown.

If an embodiment can be implemented differently, the order of specific processes may be performed differently from the order described. For example, two processes described in succession may be performed substantially simultaneously, or may proceed in an order opposite to that in which they are described.

In the following embodiments, when components are connected, this includes not only the case where the components are directly connected, but also the case where the components are indirectly connected by intervening between the components.

For example, in this specification, when components, etc. are said to be electrically connected, this includes not only cases where the components are directly electrically connected, but also cases where components, etc. are interposed and indirectly electrically connected.

A method for learning a fall detection model according to an embodiment of the present disclosure includes training an object tracking model; and training a fall detection model, wherein the step of training the fall detection model includes acquiring a first learning data set consisting of label information related to whether or not a person has fallen assigned to radar data and camera data matched with the radar data. steps; Inputting the radar data into the fall detection model; Obtaining a classification prediction value related to whether a fall occurred output through the fall detection model; And it may include updating the fall detection model based on the classification prediction value and the label information of the first learning data set.

According to an embodiment of the present disclosure, training the object tracking model includes obtaining a second learning data set consisting of location information of an object calculated from the radar data and camera data matched with the radar data; Inputting the radar data into the object tracking model and obtaining an output value related to the location of the object output through the object tracking model; And it may further include updating the object tracking model based on the output value and location information of the object in the second learning data set.

According to an embodiment of the present disclosure, training the object tracking model further includes extracting features from the radar data through the object tracking model, wherein inputting the radar data to the fall detection model The step includes obtaining a feature queue by storing at least one feature extracted through the object tracking model; and inputting at least one feature stored in the feature queue into the fall detection model.

According to an embodiment of the present disclosure, the label information includes a first label indicating that a fall occurred in the first frame section of the camera data, and a label indicating that a fall did not occur in the second frame section of the camera data. It can be one of 2 labels.

According to an embodiment of the present disclosure, the radar data may be composed of a two-dimensional Range-Azimuth heat map and a two-dimensional Range-Elevation heat map.

According to an embodiment of the present disclosure, a computer-readable recording medium recording a program for executing the fall detection model learning method may be provided.

A fall detection device according to an embodiment of the present disclosure includes a processor configured to train an object tracking model and a fall detection model, wherein the processor determines whether a fall has occurred assigned to radar data and camera data matched with the radar data. Obtain a first learning data set consisting of related label information, input the radar data into the fall detection model, obtain a classification prediction value related to whether a fall occurred output through the fall detection model, and classify It may be configured to train the fall detection model by updating the fall detection model based on the prediction value and the label information of the first learning data set.

According to an embodiment of the present disclosure, the processor acquires a second learning data set consisting of location information of an object calculated from the radar data and camera data matched with the radar data, and uses the radar data to track the object. Enter the model, obtain an output value related to the location of the object output through the object tracking model, and update the object tracking model based on the output value and the location information of the object in the second learning data set. Can be configured to train an object tracking model.

According to an embodiment of the present disclosure, the processor extracts features from the radar data through the object tracking model, stores at least one or more features extracted through the object tracking model to obtain a feature queue, and It may be configured to input at least one or more features stored in a feature queue into the fall detection model.

According to an embodiment of the present disclosure, the label information includes a first label indicating that a fall occurred in the first frame section of the camera data, and a label indicating that a fall did not occur in the second frame section of the camera data. It can be one of 2 labels.

According to an embodiment of the present disclosure, the radar data may be composed of a two-dimensional Range-Azimuth heat map and a two-dimensional Range-Elevation heat map.

Hereinafter, a fall detection device according to an embodiment of the present disclosure and/or a method of learning an object tracking model and a method of learning a fall detection model performed by the fall detection device will be described with reference to FIGS. 1 to 9.

Figure 1 is a schematic diagram of a fall detection device 1000 according to an embodiment of the present disclosure.

The fall detection apparatus 1000 according to an embodiment of the present disclosure can track an object based on radar data and train a neural network model to detect whether the object has fallen. Furthermore, the fall detection device 1000 can track an object using a learned neural network model and perform an operation to detect or analyze whether the object has fallen.

The fall detection device 1000 according to an embodiment of the present disclosure may include a transceiver 1100, a memory 1200, and a processor 1300.

The transceiver unit 1100 of the fall detection device 1000 can communicate with any external device. For example, the fall detection device 1000 may receive radar data through the transceiver 1100. Additionally, the fall detection device 1000 may obtain execution data for executing a trained neural network model (eg, object tracking model and/or fall detection model) through the transceiver 1100. Here, execution data may mean encompassing arbitrary data for properly executing a trained neural network model, including layer information, operation information, structural information, and weight information of the trained neural network model. In addition, the fall detection device 1000, through the transceiver 1100, transmits any information, including location information of the object, bounding box information defining the object, and/or fall notification, to an external device (or can be transmitted to an external terminal).

The fall detection device 1000 can connect to a network and transmit and receive various data through the transmitting and receiving unit 1100. The transceiver unit 1100 may largely include a wired type and a wireless type. Since the wired type and the wireless type have their own advantages and disadvantages, in some cases, the fall detection device 1000 may be provided with both the wired type and the wireless type. Here, in the case of the wireless type, a WLAN (Wireless Local Area Network) type communication method such as Wi-Fi can be mainly used. Alternatively, in the case of the wireless type, cellular communication, such as LTE or 5G communication methods, can be used. However, the wireless communication protocol is not limited to the above-described example, and any appropriate wireless type of communication method can be used. In the case of the wired type, LAN (Local Area Network) or USB (Universal Serial Bus) communication are representative examples, but other methods are also possible.

The memory 1200 of the fall detection device 1000 can store various types of information. Various data may be temporarily or semi-permanently stored in the memory 1200. Examples of memory 1200 include hard disk drive (HDD), solid state drive (SSD), flash memory, read-only memory (ROM), random access memory (RAM), etc. This can be. The memory 1200 may be provided as built into the fall detection device 1000 or in a detachable form. The memory 1200 contains information necessary for the operation of the fall detection device 1000, including an operating program (OS: Operating System) for operating the fall detection device 1000 and a program for operating each component of the fall detection device 1000. Various data can be stored.

The processor 1300 may control the overall operation of the fall detection device 1000. For example, the processor 1300 trains an object tracking model to track an object based on radar data to be described later, calculates location information of an object through a trained object tracking model, and trains a fall detection model. The overall operation of the fall detection device 1000 can be controlled, including the operation of calculating a classification value related to whether a fall has occurred using a fall detection model for which training has been completed. Specifically, the processor 1300 may load and execute a program for the overall operation of the fall detection device 1000 from the memory 1200. The processor 1300 may be implemented as an application processor (AP), a central processing unit (CPU), a microcontroller unit (MCU), or a similar device depending on hardware, software, or a combination thereof. At this time, hardware may be provided in the form of an electronic circuit that processes electrical signals to perform a control function, and software may be provided in the form of a program or code that drives the hardware circuit.

Hereinafter, with reference to FIGS. 2 to 9, the operation of the fall detection device 1000, the learning method of the object tracking model, and/or the learning method of the fall detection model according to an embodiment of the present disclosure will be described in more detail. do.

FIG. 2 is a diagram illustrating an aspect of training an object tracking model according to an embodiment of the present disclosure and an aspect of calculating location information of an object using the learned object tracking model. Specifically, FIG. 2(a) is a diagram illustrating an aspect of learning an object tracking model according to an embodiment of the present disclosure, and FIG. 2(b) is a diagram illustrating a fully trained object tracking model according to an embodiment of the present disclosure. This is a diagram showing how to calculate the location information of an object from radar data.

The fall detection apparatus 1000 according to an embodiment of the present disclosure may perform an operation of training a neural network model (hereinafter referred to as an object tracking model) for tracking an object based on radar data. In order to detect a fall, the object that is the target of fall detection must be tracked from radar data, so the fall detection device 1000 tracks the object and provides location information of the object and/or a bounding box that defines the object. It may be configured to train an object tracking model for computing.

Specifically, the fall detection device 1000 may train an object tracking model using a learning data set of the object tracking model consisting of radar data and camera data matched with the radar data. At this time, the coordinates of the radar data and the coordinates of the camera data are calibrated, so conversion between each coordinate is possible.

As an example, the fall detection apparatus 1000 may train an object tracking model using a learning data set of the object tracking model consisting of location information of objects calculated based on radar data and camera data. As an example, location information of an object may be calculated using a first image obtained from a first camera installed at a first location and a second image obtained from a second camera installed at a second location. Specifically, the fall detection device 1000 extracts the skeletal information of the object through a pose estimation technique for each image, and establishes location information of the object and a bounding box that defines the object based on the skeletal information of the extracted object. You can obtain (bounding box). For example, the fall detection device 1000 acquires the intersection of the straight line connecting the object's skeletal information acquired through one camera and the corresponding camera and the straight line connecting the object's skeletal information and the corresponding camera through another camera, and obtains each Based on the intersections corresponding to the skeleton information, the location information of the object and the bounding box that defines the object can be obtained. For example, a bounding box that defines an object can be obtained to include intersections corresponding to each skeletal information. For example, the location information of an object can be calculated based on the coordinate information of the bounding box that defines the object. Meanwhile, in the above-mentioned example, each image was acquired using two cameras, and the location information of the object was calculated from each image. However, when two cameras are used, there may be cases where occlusion occurs between objects, and when occlusion occurs between objects, the location information of the calculated object may be inaccurate. Therefore, when occlusion occurs between objects, the first image obtained from the first camera installed at the first location, the second image obtained from the second camera installed at the second location, and the third camera installed at the third location It can be implemented to calculate the location information of the object using the acquired third image.

The fall detection device 1000 may input radar data into an object tracking model and obtain an output value related to the location of the object output through the object tracking model. At this time, the fall detection device 1000 may update the object tracking model based on the output value and the location information of the camera-based object included in the learning data set of the object tracking model. Specifically, the fall detection device 1000 is configured so that the output value is approximated to the position information of the camera-based object based on the difference between the output value and the position information of the camera-based object included in the learning data set of the object tracking model. , the object tracking model can be trained by updating the parameters (or weights) of the object tracking model.

The fall detection device 1000 according to an embodiment of the present disclosure performs an operation of calculating object location information (or bounding box defining the object) from radar data using a learned object tracking model. can do. Specifically, the fall detection device 1000 may be configured to acquire radar data to be analyzed and input the radar data into a learned object tracking model. At this time, the fall detection device 1000 may acquire the location information of the object output through the learned object tracking model. The object tracking model that has completed learning is trained to output an output value approximated to the position information of the object that is ground-truth from the radar data through a learning data set consisting of the position information of the object calculated based on radar data and camera data. Therefore, the location information of the object can be calculated from radar data.

Meanwhile, in Figure 2, the learning method of the object tracking model is explained, focusing on calculating the location information of the object. However, this is only an example, and the fall detection device 1000 defines the object as well as the location information of the object from the radar data using a learning data set consisting of bounding box information that defines the object based on the above-described radar data and camera data. You will be able to train an object tracking model to calculate the bounding box.

FIG. 3 is a diagram illustrating an aspect of learning a fall detection model according to an embodiment of the present disclosure and an aspect of detecting a fall using a fall detection model that has completed learning. Specifically, FIG. 3(a) is a diagram showing an aspect of learning a fall detection model according to an embodiment of the present disclosure, and FIG. 3(b) shows a fall detection model that has completed learning according to an embodiment of the present disclosure. This is a diagram showing how to calculate a classification value related to whether a fall has occurred from radar data.

The fall detection apparatus 1000 according to an embodiment of the present disclosure may perform an operation of training a neural network model (hereinafter, fall detection model) for detecting a fall of an object based on radar data. Specifically, the fall detection device 1000 may train a fall detection model using a fall detection model training data set consisting of radar data and label information related to whether or not a fall has been assigned to camera data matched with the radar data. In the frames belonging to the first frame section of the camera data corresponding to the section in which a fall occurred (e.g., a fall while standing, a fall while lying down, or a fall while sitting), there is a second frame indicating that the fall occurred. 1 label can be assigned. On the other hand, frames belonging to the second frame section of camera data corresponding to a section in which no fall occurred (e.g., a section corresponding to a state in which an object is sitting, a state in which an object is lying down, a state in which an object is walking, a state in which an object is running, etc.) , a second label indicating that a fall did not occur may be assigned to frames belonging to the second frame section. 9

The fall detection device 1000 may input radar data from the fall detection model's training data set to the fall detection model and obtain a classification prediction value related to whether a fall occurred, which is output through the fall detection model. At this time, the fall detection device 1000 may update the fall detection model based on the classification prediction value and label information included in the fall detection model training data set. Specifically, the fall detection device 1000 determines the classification prediction value as a fall condition based on the classification prediction value and the label information included in the learning data set of the fall detection model, if the label information of the corresponding frame includes the first label indicating that a fall occurred. If the label information of the corresponding frame includes a second label indicating that the fall did not occur, the classification prediction value is output as the second value corresponding to the fact that the fall was not detected. To do so, the fall detection model can be trained by updating the parameters (or weights) of the fall detection model.

The fall detection device 1000 according to an embodiment of the present disclosure may perform an operation of calculating a classification value indicating that a fall of an object has occurred from radar data using a fall detection model that has completed learning. Specifically, the fall detection device 1000 may be configured to acquire radar data to be analyzed and input the radar data into a fall detection model on which learning has been completed. At this time, the fall detection device 1000 may obtain a classification value output through a fall detection model that has completed learning. The fall detection model that has completed training is trained to output a classification prediction value corresponding to the label information from the radar data through a learning data set consisting of label information assigned according to whether or not there was a fall for the radar data and camera data. From the data, a classification value can be calculated according to whether a fall occurred. Specifically, the fall detection model that has completed learning outputs a first classification value indicating that a fall occurred when a fall was detected in the radar data, and outputs a first classification value indicating that a fall occurred when a fall was not detected in the radar data. It may be configured to output the second classification value that represents the second classification value.

Hereinafter, an aspect of training an object tracking model and a fall detection model in conjunction with one another according to an embodiment of the present disclosure will be described with reference to FIG. 4 . When describing FIG. 4, overlapping content described in relation to FIGS. 2 and 3 may be omitted. However, this is only for convenience of explanation and should not be construed as limiting.

FIG. 4 is a diagram illustrating an aspect of learning an object tracking model and a fall detection model according to an embodiment of the present disclosure.

The fall detection device 1000 according to an embodiment of the present disclosure may perform an operation of linking and training an object tracking model for tracking an object and a fall detection model for detecting a fall of an object, based on radar data. You can. Specifically, the fall detection device 1000 models an object tracking model using a learning data set consisting of radar data, label information related to whether a fall has been assigned to camera data matched with the radar data, and location information of the object calculated from the camera data. and a fall detection model can be trained.

The fall detection device 1000 may input radar data (eg, radar data in the form of a heatmap) into an object tracking model and obtain an output value related to the location information of the object output through the object tracking model. At this time, the fall detection device 1000 may train an object tracking model based on the output value and the location information (ground-truth) of the object calculated from the camera data of the learning data set. Specifically, the fall detection device 1000 may update the parameters of the object tracking model so that the object tracking model outputs an output value that approximates the location information of the object calculated from camera data.

Meanwhile, some layers of the object tracking model may be configured to extract features representing the characteristics of the frame for each frame of radar data. At this time, the fall detection device 1000 may be implemented to train a fall detection model based on features extracted through an object tracking model. Specifically, the fall detection apparatus 1000 may obtain a feature queue by sequentially storing at least one feature extracted from the object tracking model. For example, the fall detection device 1000 stores at least one feature, including a first feature extracted from the first frame of the radar data and a second feature extracted from the second frame of the radar data, for each frame to obtain a feature queue. You can.

At this time, the fall detection device 1000 may input at least one feature stored in the feature queue into the fall detection model and obtain a classification prediction value related to whether a fall occurred, which is output through the fall detection model. Furthermore, the fall detection device 1000 may train a fall detection model based on label information and classification prediction values assigned to each frame of camera data matched to radar data included in the learning data set. For example, when the fall detection device 1000 is assigned a first label indicating that a fall has occurred to a frame of camera data corresponding to the first frame of radar data, the fall detection model creates the first label based on the first feature. The parameters of the fall detection model may be updated to output the first classification prediction value corresponding to the label (that is, to classify the first frame as having occurred a fall). For example, when a second label indicating that a fall has not occurred is assigned to a frame of camera data corresponding to a second frame of radar data, the fall detection device 1000 creates a fall detection model based on the second feature. The parameters of the fall detection model may be updated to output a second classification prediction value corresponding to the second label (that is, to classify the second frame as having not occurred a fall).

Meanwhile, the fall detection device 1000 according to an embodiment of the present disclosure may be configured to alternately perform learning of the object tracking model and learning of the fall detection model. For example, the fall detection apparatus 1000 may be configured to repeatedly perform the operations of training an object tracking model a predetermined number of times and training a fall detection model a predetermined number of times. The fall detection device 1000 according to an embodiment of the present disclosure alternately trains an object tracking model and a fall detection model, so that the object tracking task of the object tracking model and the fall detection task of the fall detection model improve the performance of each other's tasks. The object tracking model and the fall detection model can be trained in a way that maximizes the accuracy of the fall detection model while maintaining the accuracy of the object tracking model at a certain level.

Figure 5 shows an aspect of calculating an object's location information and a classification value related to whether a fall occurred from radar data using a fully learned object tracking model and a fully learned fall detection model according to an embodiment of the present disclosure. It is a drawing.

The fall detection device 1000 according to an embodiment of the present disclosure calculates location information of the object and classification values related to whether a fall of the object has occurred from radar data using a learned object tracking model and a fall detection model. You can. Specifically, the fall detection device 1000 may be configured to input radar data to be analyzed (eg, radar data in the form of a heat map) into a learned object tracking model and/or a learned fall detection model. At this time, the fall detection device 1000 may acquire the location information of the object through the learned object tracking model. As described above, the learned object tracking model is based on the object's position information (ground-truth) calculated from the radar data and the camera data matched to the radar data, and the output value is approximated to the object's position information from the radar data. Since it has been trained to output, the location information of objects can be calculated from layer data.

Meanwhile, the learned object tracking model may be configured to extract features representing characteristic information of the frame for each frame of radar data to be analyzed. At this time, the fall detection apparatus 1000 may obtain a feature queue by storing at least one extracted feature for each frame. Furthermore, the fall detection device 1000 can input features stored in the feature queue for each frame into a fall detection model that has been trained, and obtain a classification value related to whether the object has fallen, which is output through the fall detection model that has completed learning. As described above, the fall detection model that has completed learning is derived from the features of the feature queue through a learning data set consisting of label information assigned according to whether or not there was a fall for the features extracted from radar data and the frames of camera data corresponding to the features. Since it has been learned to output a classification prediction value corresponding to the label information, a classification value according to whether a fall occurred can be calculated from the features of the feature queue. In summary, the fall detection device 1000 can obtain location information of an object and a classification value related to whether a fall of the object has occurred from radar data through a fully learned object tracking model and a fully learned fall detection model. .

Meanwhile, radar data input to the object tracking model and/or fall detection model according to an embodiment of the present disclosure may be in the form of a two-dimensional heat map. For example, radar data included in the learning data set may consist of a two-dimensional Range-Azimuth heat map and a two-dimensional Range-Elevation heat map.

To predict 3D information, a 3D heatmap (Range-Azimuth-Elevation heatmap) is needed. However, because a large amount of computation is required to analyze a 3D heat map, the processor within the radar, which has relatively low performance, uses radar data in the form of a 3D heat map to track objects and detect falls of objects. It is inappropriate to do so. Therefore, the fall detection device 1000 according to an embodiment of the present disclosure inputs the two-dimensional Range-Azimuth heat map and the two-dimensional Range-Elevation heat map of each frame into an object tracking model and/or a fall detection model, It may be implemented to train an object tracking model and/or a fall detection model, or calculate location information and/or classification values of an object. By using a two-dimensional heat map, the fall detection device 1000 according to an embodiment of the present disclosure can execute an object tracking model and a fall detection model even on the radar's own processor, which has relatively low performance.

Hereinafter, a method for learning an object tracking model and a fall detection model according to an embodiment of the present disclosure will be described with reference to FIGS. 6 to 8. In describing the learning method of each model, overlapping embodiments described in relation to FIGS. 2 to 5 may be omitted. However, this is only for convenience of explanation and should not be construed as limiting.

Figure 6 is a flow chart illustrating a method for training a fall detection model according to an embodiment of the present disclosure.

A method of training a fall detection model according to an embodiment of the present disclosure may include training an object tracking model (S1000) and training a fall detection model (S2000).

In the step of training an object tracking model (S1000), the fall detection device 1000 may train an object tracking model that performs a task for tracking an object based on radar data. Specifically, the fall detection device 1000 may track an object and train an object tracking model to calculate the location information of the object and/or a bounding box that defines the object. The details of training the object tracking model will be described in more detail in FIG. 7.

In the step of training a fall detection model (S2000), the fall detection device 1000 may train a fall detection model to detect a fall of an object based on radar data. As an example, the fall detection device 1000 may train a fall detection model to determine whether an object has fallen in the frame based on the frame-by-frame features of the radar data extracted through the object tracking model in step S1000. there is. The contents of training the fall detection model will be described in more detail in FIG. 8.

Figure 7 is a flow chart specifying the step (S1000) of training an object tracking model according to an embodiment of the present disclosure.

The step of training an object tracking model according to an embodiment of the present disclosure (S1000) is the step of acquiring a learning data set consisting of location information of the object calculated from radar data and camera data matched with the radar data (S1100), radar Inputting data into the object tracking model, obtaining an output value output through the object tracking model (S1200), and updating the object tracking model based on the output value and the location information of the object in the learning data set (S1300) ) may further be included.

In the step (S1100) of acquiring a learning data set consisting of location information of an object calculated from radar data and camera data matched with the radar data, the fall detection device 1000 acquires a learning data set for training an object tracking model. can do. Specifically, the learning data set may be composed of radar data and object location information calculated from camera data matched with the radar data for each frame. Here, the radar data may be radar data in the form of a heat map, for example, a two-dimensional Range-Azimuth heat map and a two-dimensional Range-Elevation heat map.

The location information of the object may include a first image obtained from a first camera installed at a first location, a second image obtained from a second camera installed at a second location, and/or a third image obtained from a third camera installed at a third location. 3 Can be calculated using images. Specifically, the fall detection device 1000 extracts the skeletal information of the object through a pose estimation technique for each image, and establishes location information of the object and a bounding box that defines the object based on the skeletal information of the extracted object. You can obtain (bounding box). For example, the fall detection device 1000 acquires the intersection of the straight line connecting the object's skeletal information acquired through one camera and the corresponding camera and the straight line connecting the object's skeletal information and the corresponding camera through another camera, and obtains each Based on the intersections corresponding to the skeleton information, the location information of the object and the bounding box that defines the object can be obtained.

In the step (S1200) of inputting radar data into the object tracking model and obtaining an output value output through the object tracking model, the fall detection device 1000 includes heat map data ( For example, two-dimensional Range-Azimuth heat map data and two-dimensional Range-Elevation heat map data) can be input into an object tracking model, and output values related to the location of the object output through the object tracking model can be obtained.

Meanwhile, as described above, the object tracking model may be configured to extract features representing characteristic information of the heatmap data from heatmap data through some layers included in the object tracking model. At this time, the fall detection device 1000 may be configured to obtain at least one or more extracted features and store them in a feature queue.

In the step (S1300) of updating the object tracking model based on the output value and the location information of the object in the learning data set, the fall detection device 1000 uses ground-truth information on the location of the object calculated from the camera data of the learning data set. ) and the object tracking model can be trained based on the output values output through the object tracking model. Specifically, the fall detection device 1000 may update the parameters of the object tracking model so that the object tracking model outputs an output value that approximates the location information of the object calculated from camera data.

The fall detection device 1000 may perform an operation of calculating the location information of an object (or a bounding box defining the object) from radar data using a learned object tracking model. Specifically, the fall detection device 1000 may be configured to acquire radar data to be analyzed (eg, radar data in the form of a heat map) and input the radar data into a learned object tracking model. At this time, the fall detection device 1000 may acquire the location information of the object output through the learned object tracking model. The object tracking model that has completed learning is trained to output an output value approximated to the position information of the object that is ground-truth from the radar data through a learning data set consisting of the position information of the object calculated based on radar data and camera data. Therefore, it is possible to calculate object location information indicating the location of the object from radar data.

Figure 8 is a flow chart specifying the step (S2000) of training a fall detection model according to an embodiment of the present disclosure.

The step of training a fall detection model according to an embodiment of the present disclosure (S2000) is the step of acquiring a learning data set consisting of label information related to whether a fall has been assigned to radar data and camera data matched with the radar data (S2100). , inputting radar data into a fall detection model (S2200), obtaining a classification prediction value related to whether a fall occurred output through the fall detection model (S2300), and labeling the classification prediction value and the learning data set. A step of updating the fall detection model based on the information (S2400) may be further included.

In the step (S2100) of acquiring a learning data set consisting of radar data and label information related to whether or not a fall has been assigned to the camera data matched with the radar data, the fall detection device 1000 uses a learning data set for training a fall detection model. can be obtained. Specifically, the learning data set may be composed of radar data and label information assigned to each frame of camera data matched for each frame with the radar data. For example, in the frames belonging to the first frame section of the camera data corresponding to the section in which a fall (e.g., a fall while standing, a fall while lying down, or a fall while sitting) occurred, it is indicated that a fall occurred. A first label representing the first label may be assigned. On the other hand, frames belonging to the second frame section of the camera data corresponding to the section in which no fall occurred (e.g., the section corresponding to the state in which the object is sitting, the state in which the object is lying, the state in which the object is walking, the state in which the object is running, etc.) A second label indicating that a fall did not occur may be assigned.

In the step of inputting radar data into the fall detection model (S2200), the fall detection device 1000 may input radar data consisting of a plurality of frames into the fall detection model. As an example, the fall detection device 1000 may be configured to input radar data in the form of a two-dimensional heat map into a fall detection model. For example, radar data may be composed of a two-dimensional Range-Azimuth heat map and a two-dimensional Range-Elevation heat map for each frame.

Meanwhile, according to an embodiment of the present disclosure, the step of inputting radar data into a fall detection model (S2200) includes obtaining a feature queue by storing at least one feature extracted through the above-described object tracking model; And it may further include inputting at least one feature stored in the feature queue into the fall detection model. Specifically, the fall detection device 1000 includes at least one feature including a first feature extracted from the first frame of the radar data and a second feature extracted from the second frame of the radar data through some layers of the object tracking model. You can obtain feature queues saved for each frame. At this time, the fall detection apparatus 1000 may be configured to input the first feature or the second feature stored in the feature queue into the fall detection model.

In the step (S2300) of acquiring a classification prediction value related to whether a fall has occurred output through the fall detection model, the fall detection device 1000 uses the fall detection model from the above-described radar data (e.g., features stored in the feature queue). It is possible to obtain a classification prediction value related to whether a fall occurred or not.

In the step (S2400) of updating the fall detection model based on the classification prediction value and the label information of the learning data set, the fall detection device 1000 uses the classification prediction value output through the fall detection model and the radar included in the learning data set. The parameters of the fall detection model can be updated based on the label information assigned to each frame of camera data matched to the data.

For example, when the fall detection device 1000 is assigned a first label indicating that a fall has occurred to a frame of camera data corresponding to the first frame of radar data, the fall detection model is configured to use the first label extracted from the first frame. Based on the feature, the parameters of the fall detection model may be updated to output the first classification prediction value corresponding to the first label (that is, to classify the first frame as having occurred a fall). For example, when the fall detection device 1000 is assigned a second label indicating that no fall occurred to the frame of camera data corresponding to the second frame of radar data, the fall detection model is extracted from the second frame. Based on the second feature, the parameters of the fall detection model may be updated to output a second classification prediction value corresponding to the second label (that is, to classify that a fall did not occur for the second frame).

The fall detection device 1000 according to an embodiment of the present disclosure detects that a fall of an object has occurred from radar data (e.g., a feature queue in which features extracted from frames of radar data are stored) using a fall detection model that has completed learning. An operation can be performed to calculate the classification value it represents. Specifically, the fall detection apparatus 1000 may be configured to input features stored for each frame in a feature queue obtained from radar data of an analysis target into a fall detection model on which learning has been completed. At this time, the fall detection device 1000 may obtain a classification value related to whether a fall of the object output has occurred through a fall detection model that has completed learning. The fall detection model that has completed learning is provided with label information from radar data through a learning data set consisting of radar data (e.g., at least one or more features extracted from radar data) and label information assigned according to whether or not there was a fall for camera data. Since it has been learned to output a classification prediction value corresponding to , it is possible to calculate a classification value according to whether a fall occurred from the radar data to be analyzed. Specifically, the fall detection model that has completed learning outputs a first classification value indicating that a fall occurred when a fall was detected in the radar data, and outputs a first classification value indicating that a fall occurred when a fall was not detected in the radar data. It may be configured to output the second classification value that represents the second classification value.

Figure 9 is a flowchart showing an aspect of calculating location information of an object using a fully trained object tracking model and detecting a fall using a fully trained fall detection model according to an embodiment of the present disclosure. The trained object tracking model and the trained fall detection model according to an embodiment of the present disclosure may be executed in the processor of the radar itself. Specifically, the radar processor acquires radar data (e.g., radar data in the form of a heat map), inputs the radar data into a learned object tracking model, and provides location information (or object location information) calculated through the learned object tracking model. You can obtain the corresponding bounding box. Meanwhile, the radar processor acquires features representing characteristic information of each frame of radar data extracted through some layers of the object tracking model for which learning has been completed, acquires a feature queue by storing features for each frame, and obtains features stored in the feature queue. It can be configured to input into the fall detection model on which learning has been completed. At this time, the radar processor acquires a classification value related to whether the object has fallen through the learned fall detection model, and if it is determined that a fall has occurred based on the classification value, it notifies the protection terminal or institutional terminal that a fall has occurred. Can be configured to send notifications. On the other hand, if it is determined that a fall did not occur based on the classification value, it may be configured to repeatedly perform the task of tracking the object and detecting the fall of the object based on radar data.

There is a need for the fall detection device 1000 according to an embodiment of the present disclosure to detect falls, but in environments where it is difficult to detect falls through a camera due to individual privacy (e.g., nursing homes, nursing hospitals, restrooms, etc.) It may be used in .

According to the fall detection model learning method and fall detection device according to an embodiment of the present disclosure, it is possible to track an object and detect a fall based only on radar data. Through this, it is possible to provide the effect of tracking objects and detecting falls using only radar data, even in environments where cameras are difficult to install.

According to the fall detection model learning method and fall detection device according to an embodiment of the present disclosure, a fall detection model is trained using a learning data set configured to detect falls occurring in various postures, regardless of the fall posture. It can provide the effect of detecting falls.

According to the fall detection model learning method and fall detection device according to an embodiment of the present disclosure, the time required for learning the fall detection model is reduced by using features extracted through the object tracking model in learning the fall detection model. You can do it.

According to the fall detection model learning method and fall detection device according to an embodiment of the present disclosure, the accuracy of the fall detection task is improved while maintaining the accuracy of the object tracking task at a certain level by alternately learning the object tracking model and the fall detection model. It can be increased.

According to the fall detection model learning method and fall detection device according to an embodiment of the present disclosure, the object tracking model and the fall detection model are trained or executed using radar data in the form of a two-dimensional heat map, thereby improving relative performance. Object tracking models and fall detection models can also be run on the processor of this low-level radar itself. Through this, the effect of tracking objects and detecting falls can be provided through the radar's own processor.

Various operations of the fall detection device 1000 described above may be stored in the memory 1200 of the fall detection device 1000, and the processor 1300 of the fall detection device 1000 performs the operations stored in the memory 1200. may be provided to do so.

The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present disclosure and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. illustrated in each embodiment can be combined or modified and implemented in other embodiments by a person with ordinary knowledge in the field to which the embodiments belong. Accordingly, contents related to such combinations and modifications should be construed as being included in the scope of the present disclosure.

In addition, although the above description focuses on the embodiment, this is only an example and does not limit the present disclosure, and those skilled in the art will be able to understand the above without departing from the essential characteristics of the present embodiment. You will see that various modifications and applications not illustrated are possible. In other words, each component specifically shown in the embodiment can be modified and implemented. And these variations and differences related to application should be construed as being included in the scope of the present disclosure as defined in the attached claims.

The method of learning a fall detection model based on radar data as described above and the fall detection device that performs the same can be applied to various fields such as sports, medical, smart home, and industrial fields.

Claims (9)

In a method for a fall detection device to learn a fall detection model, training an object tracking model based on radar data; and Comprising training a fall detection model based on features extracted from the radar data through the object tracking model, The step of training the fall detection model is, Obtaining a first learning data set consisting of label information related to a fall assigned to the radar data and camera data matched with the radar data; acquiring a feature queue by storing at least one feature extracted from the radar data through the object tracking model; Inputting at least one feature stored in the feature queue into the fall detection model; Obtaining a classification prediction value related to whether a fall occurred output through the fall detection model based on the at least one feature; and Comprising the step of updating the fall detection model based on the classification prediction value and the label information of the first learning data set, Learning method of fall detection model. According to claim 1, The step of training the object tracking model is, Obtaining a second learning data set consisting of location information of an object calculated from the radar data and camera data matched with the radar data; Inputting the radar data into the object tracking model and obtaining an output value related to the location of the object output through the object tracking model; and Further comprising updating the object tracking model based on the output value and the location information of the object in the second learning data set, The location information of the object is calculated based on the skeleton information of the object obtained from the camera data, Learning method of fall detection model. According to claim 1, The label information is, The first frame section of the camera data is a first label indicating that a fall has occurred, and the second frame section of the camera data is one of a second label indicating that a fall has not occurred, Learning method of fall detection model. According to claim 1, The radar data is, Consisting of a two-dimensional Range-Azimuth heatmap and a two-dimensional Range-Elevation heatmap, Learning method of fall detection model. A computer-readable recording medium recording a program for executing the method according to any one of claims 1 to 4 on a computer. In the fall detection device, A processor configured to train an object tracking model based on radar data and train a fall detection model based on features extracted from the radar data through the object tracking model, The processor, Obtain a first learning data set consisting of label information related to the presence or absence of a fall assigned to the radar data and camera data matched with the radar data, and store at least one or more features extracted from the radar data through the object tracking model. Obtain a feature queue, input at least one or more features stored in the feature queue into the fall detection model, and classify whether a fall occurred or not output through the fall detection model based on the at least one feature. configured to train the fall detection model by obtaining a prediction value, updating the fall detection model based on the classification prediction value and the label information of the first learning data set, Fall detection device. According to clause 6, The processor, Obtain a second learning data set consisting of location information of the object calculated from the radar data and camera data matched with the radar data, input the radar data into the object tracking model, and output through the object tracking model. Configured to train the object tracking model by obtaining an output value related to the location of the object, updating the object tracking model based on the output value and the location information of the object in the second learning data set, The location information of the object is calculated based on the skeleton information of the object obtained from the camera data. Fall detection device. According to clause 6, The label information is, The first frame section of the camera data is a first label indicating that a fall has occurred, and the second frame section of the camera data is one of a second label indicating that a fall has not occurred, Fall detection device. According to clause 6, The radar data is, Consisting of a two-dimensional Range-Azimuth heatmap and a two-dimensional Range-Elevation heatmap, Fall detection device.

Images