TWI770720B - Production line monitoring method - Google Patents

Abstract

A monitoring method for a production line, comprising the following steps: obtaining an image of a work area with at least one camera and sending it to a server; identifying the image of the work area to determine whether an operator has reached the work area; identifying the posture of the operator, and When the operator's posture conforms to a start posture, the server records a start time, and the operator performs at least one process; and recognizes the operator's posture, and when the operator's posture conforms to an end posture, the server records an end time. , and count a man-hour from the start time to the end time. Thereby, it is possible to accurately know the man-hours required for the operation.

Description

Production line monitoring method

The present invention is related to production management; in particular, it refers to a monitoring method of a production line using image recognition technology.

With the advancement of science and technology, most of the current industrial production models have entered the era of fully automated production. However, for traditional industries, although automated equipment can be introduced to replace manpower, there are still some production lines that rely on manpower to operate and cannot be replaced by automated equipment.

In the conventional production method of objects, for objects that need to be produced by manpower, a paper work order is first generated according to the order. The paper work order records the information of the process site required to produce the object, and then transfers it to the process site. in the hands of workers. The process required for the object is carried out by the operator at the process station. After the process is completed, the operator fills in the process time (such as start time and end time) on the paper work order.

However, not all production line sites can rely on paper work orders to fill in the process time. For example, in the sand foundry site, which is full of sand dust, noise, and high heat, the operators cannot fill in the work orders in this place, but The work order cannot be filled until the overall job has come to an end, so the start time to the end time can only be filled approximately. For production managers, it is difficult to know the efficiency of workers actually producing objects.

In view of this, the purpose of the present invention is to provide a monitoring method of a production line, which can accurately know the man-hours spent in the operation.

In order to achieve the above object, the present invention provides a production line monitoring method, which is applied to a production system. The production system includes at least one camera and a server. The monitoring method includes the following steps:

A. Obtain an image of a work area with the at least one camera and send it to the server;

B. The server recognizes the image of the work area to determine whether the operator has reached the work area;

C. The server recognizes the posture of the operator, and when the posture of the operator conforms to a starting posture, the server records a start time, and the operator performs at least one process; and

D. The server recognizes the posture of the operator, and when the posture of the operator conforms to an end posture, the server records an end time, and counts a man-hour from the start time to the end time.

The effect of the present invention is that the server judges the working state of the operator in the work area by the image, and recognizes the start time of the start posture to the end time of the end posture, and can accurately know the man-hours spent in the operation.

In order to describe the present invention more clearly, preferred embodiments are given and described in detail with the drawings as follows. Please refer to FIG. 1 , which is a production system to which the monitoring method of the production line according to the first preferred embodiment of the present invention is applied. The production system includes a plurality of cameras 10 and a server 12 .

In this embodiment, the camera 10 can be an IP camera connected to the network 16 through a wireless router 14 (Access Point), or the camera 10 can also be directly connected to the network 16 in a wired manner. The camera 10 is disposed around a work area 18 shown in FIG. 2 to obtain an image of the work area 18 . In this embodiment, a process equipment 20 is provided in the work area 18 for performing a specific process. In addition, in this embodiment, one of the cameras 10 is used to capture the image of the process equipment 20 . In one embodiment, there may also be at least one camera 10 for acquiring images of the work area 18 including the process equipment 20 and transmitting the images to the server 12 .

The server 12 is connected to the network 16 to receive the images obtained by the camera 10 . The server 12 includes an image recognition module 122 , a processing module 124 and a database 126 . The image recognition module 122 is used to recognize the image from the camera 10 , the processing module 124 is used to record and calculate the information, and the database 126 is used to store the data.

In this embodiment, the production system may further include a signal conversion device 22, the signal conversion device 22 is connected to the server 12 via the network 16, and the signal conversion device 22 is wirelessly connected to a bone conduction earphone 24, and A warning light 26 is wired or wirelessly connected. The bone conduction earphone 24 is worn by the operator, and the warning light 26 is provided in the work area 18 (eg, provided on the process equipment 20 ). The processing module 124 of the server 12 can transmit the information to the signal conversion device 22 through the network 16 , and the signal conversion device 22 transmits the information to the bone conduction earphone 24 or the warning light 26 . When the message received by the signal conversion device 22 is audio, the audio is converted into a wireless signal such as a Bluetooth signal and sent to the bone conduction earphone 24; when the message received by the signal conversion device 22 is a control signal, then The control signal is transmitted to the warning light 26 to control the warning light 26 correspondingly. The production system may further include a throat vibration microphone 28 and is worn by the operator. The signal conversion device 22 is wirelessly connected to the throat vibration microphone 28 to transmit the operator's voice to the server 12 . The selection of the bone conduction earphone 24 and the throat vibration microphone 28 can effectively avoid environmental noise interference in the workplace.

The monitoring method of the production line of the present embodiment is described below with reference to FIG. 3 , and the monitoring method includes the following steps:

In step S11 , the camera 10 acquires the image of the work area 18 and transmits it to the server 12 . In this embodiment, the cameras 10 continue to capture images of the work area 18 and continuously transmit the images of the work area 18 to the server 12 . The server 12 divides the work area 18 into a plurality of areas, taking the first to third areas 182, 184, 186 as examples, wherein the first area 182 is closest to the process equipment, the second area 184 is next, and the third area 186 is the same The process tool 20 is located in the first area 182 further away from the process tool 20 than the second area 184 . In this embodiment, the division of the work area can be performed by the image recognition module 122 , but it is not limited thereto, and can also be performed by the processing module 124 . In this embodiment, a plurality of gestures posed by the operator in the third area 186 can be preliminarily performed, and the image recognition module 122 of the server 12 can recognize the gestures posed by the operator in the third area 186 and store them respectively. A plurality of human body poses are formed in the database 126 . As shown in FIG. 4 , the human body postures at least include a starting posture 30 and an ending posture 32 . In addition, the human body posture may further include an interrupted posture 34 and at least one abnormal posture 36 and the like. In this embodiment, the OpenPose algorithm is used to identify the human body posture. The OpenPose algorithm can identify multiple human bodies in the image, and detect the key points of each human body and the coordinates of each key point, as shown in FIG. 5 . As shown, the human body image 42 can be converted into 25 key points, and the positions of the limbs can be obtained to form a human body model 44 , and the human body posture can be recognized by the human body model 44 .

In step S12 , the server 12 recognizes the image of the work area 18 to determine whether the operator has reached the work area 18 . In this embodiment, the image recognition module 122 recognizes whether there is a human body image 42 in the image transmitted from the camera 10, and also recognizes whether there is a human body model 44 in the image of the work area 18. If so, it means that the operator has arrived at work. Zone 18; if not, continue to identify.

In step S13, the server 12 recognizes the posture of the operator, and when the posture of the operator conforms to the initial posture 30, the server records a start time, and the operator performs at least one process. In this embodiment, the image recognition module 122 uses the OpenPose algorithm to recognize the posture, and compares the recognized posture with the human body postures in the database 126 to identify whether the operator is in the second position of the work area 18 . The area 184 assumes the starting posture. If so, the processing module 124 records the starting time in the database 126. If it is recognized that the operator does not assume the starting posture, the starting time is not recorded; When the first area 182 or the third area 186 assumes the starting posture, the processing module does not record the starting time to avoid interference, and the first area 182 does not make a judgment to prevent the operator from being too close to the process equipment 20 and posing as a human body. Accident. In addition, in other embodiments, the server 12 may record the start time only when the operator's posture conforms to the initial posture 30 and maintains a first predetermined time (eg, 2 to 3 seconds), so as to exclude the operator Possibility to inadvertently assume starting pose 30.

Next, the worker performs one process or a plurality of processes. Taking a plurality of processes as an example, a process start posture 38 and a process end posture 40 corresponding to each process can be stored in the database 126 in advance (see FIG. 6 ). When the image recognition module 122 of the server 12 recognizes that the operator's posture conforms to the process start posture 38 of each process, the processing module 124 of the server 12 records a process start time in the database 126; the image of the server 12 When the identification module 122 identifies that the operator's posture conforms to the process end posture 40 of each process, the processing module 124 of the server 12 records a process end time in the database 126 . The processing module 124 can count the process man-hours from the start time of each process to the end time of each process, that is, the time spent by each process.

In addition, when the processing module 124 of the server 12 reaches a predetermined time from the start time of any process and the server 12 has not yet recognized that the operator's posture conforms to the process end posture 40 of the corresponding process, the server 12 Send a warning message. The warning message can be transmitted to the signal conversion device 22 through the network 16, and the signal conversion device 22 transmits the warning message (audio) to the bone conduction earphone 24, so that the bone conduction earphone 24 can issue a warning sound or a voice prompt. In addition, the warning message (control signal) can also be transmitted to the warning light 26, so that the warning light 26 emits a warning light. In this way, the operator is reminded that a process has exceeded the predetermined time, and the status should be reported. The operator can report the status to the server 12 through the throat vibration microphone 28 , so that the management personnel can communicate with the operator through the microphone connected to the server 12 . The operator can also assume the interruption gesture 34 in the work area 18. When the server 12 recognizes the interruption gesture 34, it will immediately send a notification message to the management personnel, for example, to an electronic device, so that the management personnel can go to the work area 18 to know the situation .

The image recognition module 122 of the server 12 recognizes the material 46 held by the operator, and when the processing module 124 determines that the material 46 does not match a predetermined material stored in the database 126 for the corresponding process, the server 12 issues a warning message, such as The aforementioned warning message is transmitted to the bone conduction earphone 24 or the warning light 26 to prompt the operator to replace the correct material 46 to avoid process errors. Material 46 can be identified by using the YOLOv3 algorithm. The YOLOv3 algorithm needs to collect a large number of image data of pre-identified objects first, and label the image data. After labeling the data, use YOLOv3 as the convolution weight model training to do For the identified weight file, as long as the error value converges to a stable low point, the training is completed and can be used for material identification.

When the image recognition module 122 of the server 12 recognizes that the operator's posture conforms to the abnormal posture 36, the server 12 sends out a warning message, and the warning message is transmitted to the bone conduction earphone 24 or the warning light 26 as described above to remind not to make an abnormality Attitude 36, to avoid injury to the operator due to incorrect posture during work, such as bending over to lift heavy objects or incorrect operation posture in the process. In other words, the abnormal posture 36 corresponds to a posture in which the worker is easily injured.

When the image recognition module 122 of the server 12 recognizes that the operator is in contact with a predetermined part of the process equipment 20, and the processing module 124 determines that the contact time reaches a warning time, the server 12 sends a warning message. As mentioned above, the warning message is transmitted to the bone conduction earphone 24 or the warning light 26, and can be transmitted to the electronic device of the manager, so that the manager can go to the work area to know the situation. The aforesaid predetermined position is the position where the mechanism of the process equipment is actuated, so as to prevent the operator's limbs from being caught or stuck by the process equipment 20 and unable to disengage, and management personnel can assist in handling.

In this embodiment, the process equipment 20 has at least one meter, such as the pointer meter 48 shown in FIG. 7 and/or the digital meter 50 shown in FIG. For environmental parameters, a camera 10 obtains the image of the header and transmits it to the server 12 . The image recognition module 122 of the server 12 recognizes the reading value of the header. When the processing module 124 judges that the reading value is abnormal, it corresponds to an abnormal state, and the server 12 sends an abnormal message. As mentioned above, the abnormal message is sent to the bone conduction earphone 24 or the warning light 26 to prompt the operator to perform the abnormal processing procedure, and the abnormal message can also be sent to the electronic device of the management personnel, and the management personnel can assist the abnormal processing procedure.

The pointer type head 48 can be identified by using the Circle Hough Transform (HoughCircles) algorithm to capture the circular features to obtain the outer frame of the head, and the Hough Line Transform (HoughLinesP) algorithm to capture the linear features to obtain the outer frame of the pointer. Then use the angle between the pointer and the center of the circle to convert the scale of the head, and cooperate with the number or text to identify the reading value of the head.

The digital meter head 50 needs to first encircle the range to be recognized and perform image processing, including binarization, erosion, and expansion; then, the numbers or characters are captured, and then the numbers or characters are recognized.

In step S14, the server 12 recognizes the posture of the operator, and when the posture of the operator conforms to the end posture 32, the server 12 records an end time, and counts a man-hour from the start time to the end time. In this embodiment, the image recognition module 122 uses the OpenPose algorithm to recognize the posture, and compares the recognized posture with the human body postures in the database 126 to identify whether the operator is in the second position of the work area 18 . The area 184 assumes the ending posture. If so, the processing module 124 records the end time in the database 126, and records the man-hours from the start time to the end time in the database 126. If it is recognized that the operator does not pose In the end posture 32 , the start time is not recorded; if the operator assumes the end posture 32 in the first area 182 or the third area 186 of the work area 18 , the processing module 124 does not record the end time. In addition, in other embodiments, the server 12 may record the end time only when the operator's posture conforms to the end posture 32 and maintains a second predetermined time (eg, 2 to 3 seconds), so as to prevent the operator from inadvertently posing Possibility of ending gesture.

With the monitoring method of the production line in this embodiment, the working hours of the operators can be counted and the real-time abnormality control can be performed.

9 is a monitoring method for a production line according to a second preferred embodiment of the present invention, which has substantially the same steps as those in the first embodiment, except that in step S23, the posture posed by the operator conforms to the initial posture 30, and is determined by The operator sends a first voice command (for example, "start operation") to the server 12 through the throat vibration microphone 28, and the image recognition module 122 of the server 12 recognizes that the operator's posture conforms to the initial posture 30 and the processing mode. When the group 124 receives the first voice command, the processing module 124 of the server 12 records the start time in the database.

In step S24, the posture of the operator conforms to the ending posture 32, and the operator sends a second voice command (for example, "end operation") to the server 12 through the throat-vibrating microphone 28, and the image recognition mode of the server 12. When the group 122 recognizes that the operator's posture matches the end posture and the processing module 124 receives the second voice command, the processing module 124 of the server 12 records the end time in the database 126, and counts the time spent from the start time to the end time The working hours are recorded in the database. In this way, the possibility of the operator taking the starting posture 30 and the ending posture 32 inadvertently can be excluded.

According to the above, the monitoring method of the production line of the present invention uses the server 12 to determine the working status of the workers in the work area, and can count the man-hours spent in the work, which can be applied to the production sites without automation equipment to perform work. Efficient management.

The above descriptions are only preferred feasible embodiments of the present invention, and any equivalent changes made by applying the description of the present invention and the scope of the patent application should be included in the patent scope of the present invention.

[this invention] 10: Camera 12: Server 122: Image recognition module 124: Processing Modules 126:Database 14: Wireless router 16: Internet 18: Workspace 182: The first area 184: Second Region 186: The third area 20: Process equipment 22: Signal conversion device 24: Bone conduction headphones 26: Warning lights 28: Throat vibration microphone 30: Starting Stance 32: Ending Pose 34: Interrupt Gesture 36: Abnormal posture 38: Process start posture 40: At the end of the process 42: Human Image 44: Mannequins 46: Materials 48: pointer head 50: digital meter S11~S14, S23, S24: Steps

FIG. 1 is a schematic diagram of a production system according to a first preferred embodiment of the present invention. FIG. 2 is a schematic diagram of the work area of the first preferred embodiment of the present invention. FIG. 3 is a flow chart of the monitoring method of the production line according to the first preferred embodiment of the present invention. 4 is a schematic diagram of a human body posture according to the first preferred embodiment of the present invention. FIG. 5 is a schematic diagram of converting a human body image into a human body model according to the first preferred embodiment of the present invention. FIG. 6 is a schematic diagram of a process start posture and a process end posture according to the first preferred embodiment of the present invention. FIG. 7 is a schematic diagram of a pointer-type meter according to the first preferred embodiment of the present invention. FIG. 8 is a schematic diagram of a digital watch head according to the first preferred embodiment of the present invention. FIG. 9 is a flow chart of the monitoring method of the production line according to the second preferred embodiment of the present invention.

Steps S11~S14

Claims (13)

A production line monitoring method is applied to a production system. The production system includes at least one camera and a server. The monitoring method includes the following steps: A. Using the at least one camera to obtain an image of a work area and sending it to the servo wherein, there is a process equipment in the work area; B. The server recognizes the image of the work area to determine whether the operator has reached the work area; C. The server recognizes the posture of the operator, and the operator The server records a start time and the operator performs at least one process when the posture conforms to a starting posture and maintains a first predetermined time to meet the starting posture; when the server identifies the operator's limb and the process equipment When the contact time of the moving parts of the mechanism reaches a warning time, the server sends out a warning message; and D. The server recognizes the operator's posture, and when the operator's posture conforms to an end posture and maintains a second predetermined time When the end posture is met, the server records an end time, and counts a man-hour from the start time to the end time. The monitoring method for a production line according to claim 1, wherein the server divides the working area into a plurality of areas, and the areas respectively include a first area, a second area and A third area; in step C, the server recognizes that the posture of the operator in the second area of the work area conforms to the initial posture and records the start time, and the server recognizes that the operator is in the first area Or the start time is not recorded when the third area conforms to the starting posture; in step D, the server recognizes that the operator records the end time when the posture of the second area of the work area conforms to the end posture, and the The server identification operator does not record the end time when the first area or the third area conforms to the end posture. The method for monitoring a production line as claimed in claim 2, before step C, comprising: the server recognizing a plurality of postures posed by the operator in the third area and storing them as a plurality of human postures respectively, and the human postures include the The starting posture and the ending posture; in steps C and D, the server compares the posture of the operator in the second area of the work area with the postures of the human body. The method for monitoring a production line according to claim 1, comprising storing a plurality of human body postures in the server, and the human body postures include the starting posture and the ending posture; in steps C and D, the server is The posture of the worker in the work area is compared with the postures of the human body. The monitoring method for a production line according to claim 4, wherein the human body postures further include at least one abnormal posture, and when the server recognizes that the operator's posture conforms to the abnormal posture, the server sends a warning message. The monitoring method for a production line as claimed in claim 5, wherein the production system includes a bone conduction earphone, which is worn by the operator; wherein the server transmits the warning message to the bone conduction earphone, so as to enable the bone conduction The headset emits a warning tone. The production line monitoring method as claimed in claim 5, wherein the production system includes a warning light; wherein the server transmits the warning message to the warning light, so that the warning light emits a warning light. The monitoring method for a production line as claimed in claim 1, wherein the number of the at least one process in step C is plural, and when the server recognizes that the operator's posture conforms to a process start posture of each process, the server Record the start time of a process in the server; when the server recognizes that the operator's posture conforms to the process end posture of each process, it records the end time of a process in the server. The method for monitoring a production line as claimed in claim 8, wherein in step C, when the server reaches a predetermined time from the start time of any process, and the server still does not recognize that the posture of the operator conforms to the corresponding When the process is at the end of the process, the server sends out a warning message. The method for monitoring a production line according to claim 8, wherein in step C, the server counts the man-hours spent from the start time of each process to the end time of each process. The monitoring method for a production line as claimed in claim 1, wherein the process equipment has at least one header; the at least one camera is plural, and at least one of the cameras obtains the image of the at least one header and transmits it to the server , when the server identifies that the at least one header matches an abnormal state, the server sends an abnormal message. The method for monitoring a production line as claimed in claim 1, wherein step C includes that when the server identifies that the material held by the operator is inconsistent with a predetermined material, the server sends a warning message. The monitoring method for a production line as claimed in claim 1, wherein the production system includes a throat-vibration microphone and is worn by an operator, and the throat-vibration microphone is used to transmit the operator's voice to the server; wherein step C When the posture of the operator conforms to the starting posture and a first voice command sent by the operator through the throat-vibration microphone is received, the server records the starting time; wherein in step D, the posture of the operator conforms to the When the starting posture is ended and a second voice command is received by the operator through the throat-vibrating microphone, the server records the ending time.

Images