CN111476135B - Abnormal Human Activity Recognition System Using Wearable Electronic Devices - Google Patents

Abstract

The present invention mainly proposes a human body abnormal activity recognition system, which includes: a first wearable electronic device, a plurality of second wearable electronic devices, at least one camera module, and a human body abnormal activity judgment module; The activity recognition system is suitable for use in various work environments for use by a user (such as: an operator or a worker) to follow standard operating procedure (SOP) documents, operation manuals, and/or work safety The normative document monitors and judges whether the user has or makes abnormal body activities, and in this way simultaneously achieves multiple purposes of reducing the operating error rate of operators (workers), effectively improving work efficiency, and promoting productivity.

Description

Abnormal Human Activity Recognition System Using Wearable Electronic Devices

Technical field:

The invention relates to the technical field of application of wearable electronic devices, in particular to a human body abnormal activity recognition system using wearable electronic devices.

Background technique:

It is known that the company's new recruits must undergo standardized training and pass the assessment before they can perform technical work. In the process of instructing and training new recruits, in addition to providing standard operating procedure (SOP) documents and/or paper and electronic files of operation manuals to new recruits, senior staff will also follow the standard teaching manual The recorded teaching steps and methods give relevant guidance and training to new recruits. After that, new recruits can refer to the standard operating procedure documents and operation manuals and continue to practice repeatedly step by step until they are familiar with the working procedures and machine operation methods.

However, for the staff who have been familiar with the standard operating procedure documents and operation manuals and are familiar with the working procedures and machine operation methods, the unexpected emergencies and dangerous factors hidden in the actual working environment will still cause the staff's anxiety. Misoperation or improper operation may even lead to industrial safety accidents in severe cases. In view of this, human factors engineering has been researched, proposed and applied to manage the interaction and rational combination of workers, machines and the environment, thereby improving production efficiency. It is a pity that the current application of human factors engineering has its limitations, making it unsuitable for various working environments. For example, when designing production tools, mechanical engineers must design the production tools as a human-machine system (Human-Machine System) based on human factors engineering, so that the operators of the production tools can effectively reduce the occurrence of errors Improve work efficiency and productivity. However, in reality, not every production tool, tool, workstation, working method, or work-related hardware can be designed as a machine-machine system.

From the above description, it can be seen that it is necessary to redesign and develop a new system different from the known man-machine system, man-machine equipment, or man-machine work, so that this new system can be suitable for use in various working environments, and then At the same time, various purposes of reducing the operating error rate of operators (workers), effectively improving work efficiency, and promoting productivity are achieved. In view of this, the inventor of this case tried his best to research and invent, and finally developed a human body abnormal activity recognition system using wearable electronic devices of the present invention.

Invention content:

The main purpose of the present invention is to provide a system for identifying abnormal human body activities using wearable electronic devices, which includes: a first wearable electronic device, a plurality of second wearable electronic devices, at least one camera module, and a human body abnormality Activity judgment module. The abnormal human body activity recognition system of the present invention is suitable for application in various working environments, and is used for a user (for example: an operator or a staff member) to use according to standard operating procedures (SOP) documents, operation manuals, And/or the operation safety specification document monitors and judges whether the user has occurred or made abnormal body activities, so as to simultaneously reduce the operating error rate of the operator (worker), effectively improve work efficiency, and promote productivity. Purpose.

In the case of applying the abnormal human body activity recognition system 1 of the present invention, the back-end manager can monitor the online operation actions of the operator in real time to ensure that the operator is following the relevant provisions of the standard operating procedure file and/or the operation manual perform their duties under the circumstances. At the same time, since the abnormal human body activity recognition system 1 of the present invention has the function of posture (action) estimation, it can predict whether the next action of the operator is correct or not. If the operator's next action is predicted to be an abnormal human action, the back-end manager will receive a warning message, and then correct the operator's action earlier, so as to effectively prevent occupational injuries caused by operating errors.

In order to achieve the above purpose, the present invention proposes an embodiment of the abnormal human activity recognition system, which includes:

A first wearable electronic device is used to be worn on a user's head, and has a camera unit for capturing a first image of at least one object and a working environment, and for using the a head motion detection unit for head motion detection on the head of the patient, and then output a first object identification data, a first situation identification data and a head behavior identification data;

A plurality of second wearable electronic devices are used to be worn on the user's wrist, and have a wrist motion detection unit for performing a wrist motion detection on the user's wrist, And then output a wrist behavior recognition data;

At least one camera module is used to capture a second image of the user, the at least one object and the working environment, and then output a second object identification data and a second situation identification data, and the The camera module has a posture estimation unit, which is used to perform posture estimation according to a user's body posture obtained by capturing the second image to generate posture estimation data; and

An abnormal human activity judging module, coupled to the first wearable electronic device, the plurality of second wearable electronic devices, and the camera module, and has:

A data fusion unit, configured to receive the first object identification data, the first situation identification data and the head behavior identification data transmitted by the first wearable electronic device and the second wearable electronic device The wrist behavior recognition data transmitted by the device, and the data fusion unit simultaneously receives the second object recognition data, the second situation recognition data and the posture estimation data transmitted by the camera module, Furthermore, the first object recognition data, the first situation recognition data, the head behavior recognition data, the wrist behavior recognition data, the second object recognition data, the second situation recognition data, performing a data fusion process with the posture estimation data; and

A decision-making unit, coupled to the data fusion unit, is used for judging whether an abnormal human activity event occurs to the user according to a fusion data and at least one reference data transmitted by the data fusion unit.

In the aforementioned embodiments of the system for identifying abnormal human activities of the present invention, the first wearable electronic device may be any of the following: a mixed reality (MR) helmet or smart glasses.

In the aforementioned embodiments of the system for identifying abnormal human activities of the present invention, the second wearable electronic device may be any of the following: a smart bracelet or a smart watch.

In the aforementioned embodiments of the abnormal human activity recognition system of the present invention, the abnormal human activity judging module has a database for storing the reference data, and the reference data includes: standard operating procedure documents, operation manuals , and job safety specification documents.

In the aforementioned embodiment of the abnormal human activity recognition system of the present invention, the head motion detection unit and the wrist motion detection unit are both an inertial sensor.

In the aforementioned embodiments of the abnormal human activity recognition system of the present invention, the abnormal human activity judging module is a data processing device, and the data processing device can be any of the following: a central control system, an industrial computer, a server computer , desktop computer, laptop computer, tablet computer, or smartphone.

In the aforementioned embodiments of the abnormal human activity recognition system of the present invention, the head movement detection includes head rotation angle detection, head rotation number detection, head tilt angle detection, and head position detection .

In the aforementioned embodiments of the human body abnormal activity recognition system of the present invention, the wrist motion detection includes wrist rotation angle detection, wrist rotation frequency detection, wrist inclination angle detection, and wrist inclination angle detection. Measurement.

In a feasible embodiment, the abnormal human body activity recognition system of the present invention further includes a third wearable electronic device, which is coupled to the abnormal human body activity judging module and used to be worn on the user's hand, and have:

a hand motion detection unit, for performing a hand motion detection on the user's hand; and

A hand behavior recognition unit is used to perform a hand behavior recognition process based on the hand movement data obtained by the hand movement detection, so as to generate hand behavior recognition data and send it to the abnormal human body activity judgment module.

In the aforementioned embodiments of the abnormal human activity recognition system of the present invention, the hand motion detection includes fist clenched detection, palm open detection, finger motion detection, hand position detection, hand rotation angle detection, Hand rotation detection, hand tilt angle detection, and hand tilt angle detection.

In the aforementioned embodiments of the abnormal human activity recognition system of the present invention, the abnormal human activity judging module further includes: a data synchronization unit coupled to the first wearable electronic device, the plurality of second wearable electronic devices, device, the camera module, and the third wearable electronic device, for making the first wearable electronic device, the plurality of second wearable electronic devices, the camera module, and the third wearable electronic device The wearable electronic device synchronously transmits the first object recognition data, the first situation recognition data, the head behavior recognition data, the wrist behavior recognition data, the second object recognition data, the second Situation recognition data, the gesture estimation data, and the hand behavior recognition data are sent to the data fusion unit.

Description of drawings:

Fig. 1 shows a first perspective view of a human body abnormal activity recognition system using a wearable electronic device of the present invention;

Fig. 2 shows the functional block diagram of the first wearable electronic device and the abnormal human activity judging module of the abnormal human activity recognition system of the present invention;

Fig. 3 shows the functional block diagram of the second wearable electronic device, the camera module and the abnormal human activity judging module of the abnormal human activity recognition system of the present invention;

Fig. 4 shows the second perspective view of the abnormal human activity recognition system of the present invention;

FIG. 5 shows a perspective view of another implementation of the second wearable electronic device;

FIG. 6 shows a third perspective view of the abnormal human activity recognition system of the present invention; and

FIG. 7 shows a functional block diagram of the third wearable electronic device and the abnormal human activity judging module of the abnormal human activity recognition system of the present invention.

Reference signs:

1 Abnormal Human Activity Recognition System

11 The first wearable electronic device

111 camera unit

112 head motion detection unit

HOR First Object Recognition Unit

HIS First Context Recognition Unit

HHO Head Behavior Recognition Unit

12 Second wearable electronic device

121 wrist motion detection unit

WGAR Wrist Behavior Recognition Unit

13 camera module

FRO Second Object Recognition Unit

FIS Second Situational Recognition Unit

FPE pose estimation unit

14 Abnormal Human Activity Judgment Module

140 data synchronization unit

141 Data Fusion Unit

142 decision-making units

15 Third Wearable Electronic Devices

151 hand motion detection unit

HGAR hand behavior recognition unit

2 users

Detailed ways:

In order to more clearly describe a system for identifying abnormal human body activities using wearable electronic devices proposed by the present invention, preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

FIG. 1 shows a first perspective view of a system for identifying abnormal human activities using wearable electronic devices of the present invention. The present invention proposes a system 1 for identifying abnormal human activities, which includes various types of wearable electronic devices. In particular, the abnormal human activity recognition system 1 of the present invention is suitable for use in various working environments, for use by a user 2 (for example: operators or staff), and then according to standard operating procedures (SOP) Documents, operation manuals, and/or work safety specification documents monitor and judge whether the user 2 has occurred or made abnormal body activities, so as to reduce the operating error rate of operators (workers), effectively improve work efficiency, and various purposes to boost productivity.

As shown in Figure 1, the abnormal human body activity recognition system 1 of the present invention mainly includes: a first wearable electronic device 11, a plurality of second wearable electronic devices 12, at least one camera module 13, and a human body abnormal activity judgment module 14. Further, FIG. 2 shows a functional block diagram of the first wearable electronic device 11 and the abnormal human activity judgment module 14, and FIG. 3 shows the second wearable electronic device 12, the camera module 13 and the Describe the functional block diagram of the human body abnormal activity judging module 14. 1 and 2, the first wearable electronic device 11 may be a mixed reality (Mixedreality, MR) helmet for wearing on the head of the user 2, and it has: a camera Unit 111 , a head motion detection unit 112 , a first object recognition unit HOR, a first situation recognition unit HIS, and a head behavior recognition unit HHO.

According to the above description, the camera unit 111 captures a first image of at least one object and a working environment based on a first-person perspective of the user 2 . Here, it must be noted that the objects refer to the tools, parts, materials, equipment, machinery, or equipment that the user 2 needs to use when performing a specific job, and the working environment It may be an office, clean room, or production line. On the other hand, the head movement detection unit 112 is an inertial sensor for detecting the head movement of the head of the user 2, wherein the head movement detection includes : Head rotation angle detection, head rotation times detection, head tilt angle detection, and head position detection. To describe in more detail, the first object recognition unit HOR is used for performing a first object recognition process according to a first object data obtained by capturing the first image to generate a first object recognition data. Moreover, the first situation recognition unit HIS is used for performing a first situation recognition process according to a first working environment data obtained by capturing the first image to generate a first situation recognition data. Furthermore, the head behavior recognition unit HHO is used for performing a head behavior recognition process according to a head movement data obtained by the motion detection to generate a first situation recognition data.

It should be added that the first wearable electronic device 11 (for example: MR helmet) usually has a microprocessor. Therefore, in a feasible embodiment, the first object recognition unit HOR, the first situation recognition unit HIS and the head behavior recognition unit HHO can be controlled in the form of a function library, variables or operands compiled into at least one application program, and then created in the microprocessor of the first wearable electronic device 11 . It should be noted that although FIG. 1 shows that the first wearable electronic device 11 is an MR helmet, it does not limit the implementation of the first wearable electronic device 11 . FIG. 4 shows a second perspective view of the system for identifying abnormal human activities using wearable electronic devices of the present invention. In FIG. 4 , the first wearable electronic device 11 is in the form of smart glasses as another implementation.

Continue to refer to FIG. 1 and FIG. 3 . The plurality of second wearable electronic devices 12 are used to be worn on the wrist of the user 2 , and are, for example, inertial sensors of a smart wristband. According to the design of the present invention, the second wearable electronic device 12 has a wrist motion detection unit 121 and a wrist behavior recognition unit WGAR, wherein the wrist motion detection unit 121 is used to identify the user The wrist of 2 performs a wrist motion detection, and the wrist motion detection includes wrist rotation angle detection, wrist rotation frequency detection, wrist tilt angle detection, and wrist tilt angle detection. On the other hand, the wrist behavior recognition unit WGAR is used to perform a wrist behavior recognition process according to the wrist motion data obtained by the wrist motion detection, and finally generate wrist behavior recognition data. It should be noted that although FIG. 1 shows that the second wearable electronic device 12 is a smart bracelet, it does not limit the implementation of the second wearable electronic device 12 . FIG. 5 shows a perspective view of another embodiment of the second wearable electronic device of the abnormal human activity recognition system of the present invention. In FIG. 5 , the second wearable electronic device 12 is in the form of a smart watch as another implementation.

Likewise, the second wearable electronic device 12 (for example: a smart bracelet) usually has a microprocessor. Therefore, in a feasible embodiment, the wrist behavior recognition unit WGAR is edited as at least one application program in the form of a function library, variable or operand, and then created on the second wearable electronic device 12 of the microprocessors. As shown in FIGS. 1 and 3 , the camera module 13 is used to capture a second image of the user 2 , the at least one object and the working environment. Simply put, the camera module 13 is used to simultaneously record real-time images of the at least one object of the user 2 and the working environment. According to the design of the present invention, the camera module 13 has a second object recognition unit FRO, a second situation recognition unit FIS, and a posture estimation unit FPE. Wherein, the second object recognition unit FOR is used for performing a second object recognition process according to a second object data obtained by capturing the second image to generate the second object recognition data. Moreover, the second situation recognition unit FIS is used for performing a second situation recognition process according to a second working environment data obtained by capturing the second image to generate the second situation recognition data. In particular, the posture estimation unit FPE is used for performing posture estimation according to a body posture of the user obtained by capturing the second image to generate posture estimation data.

It is supplemented that the camera module 13 (for example: a digital camera or a smart phone) usually has a microprocessor. Therefore, in a feasible embodiment, the second object recognition unit FOR, the second situation recognition unit FIS and the posture estimation unit FPE are edited in the form of a function library, variables or operands as At least one application program is then created in the microprocessor of the camera module 13 . Described in more detail, the abnormal human body activity judging module 14 connects the first wearable electronic device 11, the plurality of second wearable electronic devices 12, and the camera module 13 through a wireless connection to perform data processing. transmission, and it has: a data synchronization unit 140 , a data fusion unit 141 , and a decision unit 142 . According to the design of the present invention, the data synchronization unit 140 is coupled to the first wearable electronic device 11, the plurality of second wearable electronic devices 12, and the camera module 13, so as to make the first The wearable electronic device 11, the plurality of second wearable electronic devices 12, and the camera module 13 synchronize the first object recognition data, the first situation recognition data, the head behavior recognition data , the wrist behavior recognition data, the second object recognition data, the second situation recognition data, and the posture estimation data are sent to the data fusion unit 141 .

According to the above description, the data fusion unit 141 is used to further identify the first object recognition data, the first situation recognition data, the head behavior recognition data, the wrist behavior recognition data, the second A data fusion process is performed on the object recognition data, the second situation recognition data, and the posture estimation data. Moreover, the decision-making unit 142 is coupled to the data fusion unit 141, and is used to determine whether an abnormal human activity event occurs in the user 2 according to a fusion data transmitted by the data fusion unit 141 and at least one reference data. . It is supplemented that the abnormal human activity judging module 14 is a data processing device. For example, FIG. 1 shows that the abnormal human activity judging module 14 is a laptop computer. Therefore, the abnormal human body activity judging module 14 has a storage unit for storing the reference data, and the reference data includes: a standard operating procedure file, an operation manual, and an operation safety code file (Safetycode). On the other hand, in a feasible embodiment, the data fusion unit 141 and the decision-making unit 142 are compiled into at least one application program in the form of a function library, variable or operand, and then created in the In a processor of the abnormal human activity judging module 14. However, it should be known that the feasible embodiment of the abnormal human activity judging module 14 is not limited to a notebook computer, it can also be other types of data processing equipment, such as: central control system, industrial computer, server computer, desk laptop, tablet, or smartphone.

FIG. 6 shows a third perspective view of the system for identifying abnormal human activities using wearable electronic devices of the present invention. Comparing FIG. 1 with FIG. 6, it can be seen that in a feasible embodiment, the abnormal human activity recognition system 1 of the present invention may further include a third wearable electronic device 15, which is used to be worn on the hand of the user 2 And it is coupled to the abnormal human activity judging module 14 through a wireless connection or a wired connection.

FIG. 7 shows a functional block diagram of the third wearable electronic device 15 and the abnormal human activity judging module 14 . According to the design of the present invention, the third wearable electronic device 15 is a smart glove, and it has a hand motion detection unit 151 and a hand behavior recognition unit HGAR. Wherein, the hand motion detection unit 151 is used to perform a hand motion detection on the hands of the user 2, and the hand behavior recognition unit HGAR is used to obtain A hand motion recognition process is performed on the hand movement data to generate a hand behavior recognition data and send it to the abnormal human body activity judging module 14 .

Following the above description, the hand motion detection includes fist clenched detection, palm open detection, finger motion detection, hand position detection, hand rotation angle detection, hand rotation times detection, hand tilt angle detection, and hand tilt angle detection. Moreover, it can be deduced that the data synchronization unit 140 of the abnormal human activity judging module 14 is simultaneously coupled to the first wearable electronic device 11, the plurality of second wearable electronic devices 12, the camera module 13, and The third wearable electronic device 15 is used to make the first wearable electronic device 11, the plurality of second wearable electronic devices 12, the camera module 13, and the third wearable electronic device 15 Synchronously combine the first object recognition data, the first situation recognition data, the head behavior recognition data, the wrist behavior recognition data, the second object recognition data, the second situation recognition The data, the gesture estimation data, and the hand behavior recognition data are sent to the data fusion unit 141 .

In this way, the above is a complete and clear description of a human body abnormal activity recognition system using wearable electronic devices of the present invention; and, from the above, it can be known that the present invention has the following advantages:

(1) In the present invention, a first wearable electronic device 11 , multiple second wearable electronic devices 12 , at least one camera module 13 , and an abnormal human activity judgment module 14 form an abnormal human activity recognition system 1 . The abnormal human body activity recognition system 1 of the present invention is suitable for use in various working environments, and is used for a user 2 (for example: an operator or a staff member) to use, and then operate according to the standard operating procedure (SOP) document, The manual and/or operation safety specification documents monitor and judge whether the user 2 has occurred or made abnormal body activities, so as to reduce the operating error rate of operators (workers), effectively improve work efficiency, and promote productivity of various purposes.

(2) In the case of applying the abnormal human body activity recognition system 1 of the present invention, the manager at the back end can monitor the online operation actions of the operator in real time to ensure that the operator is following the standard operating procedure file and/or the operation manual Carry out their responsibilities under the relevant regulations. At the same time, since the abnormal human body activity recognition system 1 of the present invention has the function of posture (action) estimation, it can predict whether the next action of the operator is correct or not. If the operator's next action is predicted to be an abnormal human action, the back-end manager will receive a warning message, and then correct the operator's action earlier, so as to effectively prevent occupational injuries caused by operating errors.

It must be emphasized that the above detailed description is a specific description of the feasible embodiments of the present invention, but the embodiments are not intended to limit the patent scope of the present invention, any equivalent implementation or implementation that does not depart from the technical spirit of the present invention All changes should be included in the patent scope of this case.

Claims (10)

1. A human abnormal activity recognition system, comprising:

the first wearable electronic equipment is used for being worn on the head of a user, and is provided with a camera unit used for capturing a first image of at least one object and a working environment and a head motion detection unit used for detecting the head motion of the head of the user, so as to output first object identification data, first situation identification data and head behavior identification data;

the second wearable electronic devices are used for being worn on the wrists of the users and are provided with a wrist motion detection unit used for detecting the wrist motion of the wrists of the users, so as to output wrist behavior identification data;

the camera module is used for capturing a second image of the user, the at least one object and the working environment, further outputting second object identification data and second situation identification data, and is provided with a gesture estimation unit used for estimating a gesture according to the body gesture of the user obtained by capturing the second image so as to generate gesture estimation data; and

a person abnormal activity judging module coupled to the first wearable electronic device, the plurality of second wearable electronic devices, and the camera module, and having:

the data fusion unit is used for receiving the first object identification data, the first situation identification data and the head behavior identification data transmitted by the first wearable electronic equipment and the wrist behavior identification data transmitted by the second wearable electronic equipment, and simultaneously receiving the second object identification data, the second situation identification data and the gesture estimation data transmitted by the camera module, so as to perform data fusion processing on the first object identification data, the first situation identification data, the head behavior identification data, the wrist behavior identification data, the second object identification data, the second situation identification data and the gesture estimation data; and

the decision unit is coupled with the data fusion unit and is used for judging whether a human abnormal activity event occurs to the user according to the fusion data and at least one reference data transmitted by the data fusion unit.

2. The abnormal human activity recognition system according to claim 1, wherein the abnormal human activity determination module is a data processing device, and the data fusion unit and the decision unit are edited as at least one application program in the form of a library, a variable or an operand, and further created in a processor of the data processing device.

3. The abnormal human activity recognition system according to claim 1, wherein the abnormal human activity determination module further comprises a storage unit for storing the reference data, and the reference datagram comprises: standard job program files, operating manuals, and job safety specification files.

4. The system for identifying abnormal activity of a human body according to claim 1, wherein the first wearable electronic device further comprises:

a first object identification unit for performing a first object identification process according to a first object data obtained by the first image capturing to generate the first object identification data;

a first situation recognition unit for performing a first situation recognition process according to a first working environment data obtained by the first image capturing to generate the first situation recognition data; and

A head behavior recognition unit for performing a head behavior recognition process based on the head motion data obtained by the motion detection to generate the first context recognition data.

5. The system for identifying abnormal activity of a human body according to claim 1, wherein the second wearable electronic device further comprises:

and the wrist behavior recognition unit is used for performing wrist behavior recognition processing according to the wrist motion data obtained by the wrist motion detection so as to generate wrist behavior recognition data.

6. The system of claim 4, wherein the first object recognition unit, the first context recognition unit, and the head behavior recognition unit are compiled as at least one application in the form of a library, variables, or operands, and are created in a microprocessor of the first wearable electronic device.

7. The system for recognizing abnormal human body activity according to claim 4, wherein the camera module further comprises:

a second object recognition unit for performing a second object recognition process according to a second object data obtained by the second image capturing to generate the second object recognition data; and

And a second situation recognition unit for performing a second situation recognition process according to a second working environment data obtained by the second image capturing to generate the second situation recognition data.

8. The system according to claim 7, wherein the second object recognition unit, the second context recognition unit, and the gesture estimation unit are edited as at least one application program in the form of a library, variables, or operands, and further created in a microprocessor of the camera module.

9. The abnormal human activity recognition system of claim 1, further comprising a third wearable electronic device coupled to the abnormal human activity determination module and configured to be worn on the hand of the user, and having:

a hand motion detection unit for detecting a hand motion of the user's hand; and

and the hand behavior recognition unit is used for performing hand behavior recognition processing according to the hand motion data obtained by the hand motion detection so as to generate hand behavior recognition data and transmitting the hand behavior recognition data to the abnormal human body activity judgment module.

10. The abnormal human activity recognition system of claim 9, wherein the abnormal human activity determination module further comprises:

the data synchronization unit is coupled to the first wearable electronic device, the plurality of second wearable electronic devices, the camera module, and the third wearable electronic device, and configured to enable the first wearable electronic device, the plurality of second wearable electronic devices, the camera module, and the third wearable electronic device to synchronously transmit the first object identification data, the first context identification data, the head behavior identification data, the wrist behavior identification data, the second object identification data, the second context identification data, the gesture estimation data, and the hand behavior identification data to the data fusion unit.

Images