TWI842635B - Method, computer program, computer-readable medium, and device for monitoring displacement of medical duct by labeling based on artificial intelligence technology - Google Patents

Abstract

A method for monitoring displacement of medical duct by labeling based on artificial intelligence technology is disclosed. The method includes: using a trained AI to recognize a first label, a second label, and a third label from a plurality of images in real time, wherein the first and second labels are fixed to a displaceable medical duct and is spaced apart from each other by a designated distance along a path in a work area, and the third label is fixed at a position external to the medical duct; continuously calculating a first pixel-scaled distance between the first and second labels in each image to define a datum line according to the first pixel-scaled distance; continuously calculating a second pixel-scaled distance between the third and first labels, or between the third and second labels, in each image to define a reference line according to the second pixel-scaled distance; and continuously calculating a displacement value of the reference line in each image along a lengthwise direction; determining whether the medical duct displaces along the path in the work space based on the correlation between the displacement value and the baseline. A computer program, computer-readable medium, and device for executing the aforesaid method are also disclosed.

Description

Method, computer program, and computer-readable medium and device for monitoring medical tube displacement using artificial intelligence through tags

The present invention relates to a method, computer program, computer-readable medium and device for monitoring the displacement of medical tubes through artificial intelligence using tags, particularly a method, computer program, computer-readable medium and device for automatically determining whether the medical tube is displaced by attaching two tags at a fixed distance to the medical tube and setting another tag outside the medical tube, and performing real-time recognition of images containing the above tags through artificial intelligence, calculating the change in pixel distance between each tag, and performing real-time recognition of images containing the above tags, and calculating the change in pixel distance between each tag.

During medical treatment, it is common to deliver gas, fluid, food, etc. into the body through medical tubes, such as using an endotracheal tube to assist in maintaining airway patency, using a nasogastric tube or orogastric tube to assist in injecting fluid food into the stomach, using a chest tube to drain air and blood from the pleural cavity, and using a drainage tube to drain blood, secretions, or fluid accumulated in the body cavity. After these medical tubes are inserted into the human body, they should be kept in a fixed position. Excessive movement may cause serious harm to the human body. For example, when using an endotracheal tube to assist breathing, if the endotracheal tube is displaced, it may cause respiratory failure and hypoxia in the patient, and may also cause lung collapse or cardiopulmonary arrest. Generally speaking, if the endotracheal tube is accidentally pulled out from the installation position by more than 1 cm, it is considered endotracheal tube displacement (ETD); if the endotracheal tube is accidentally pulled out from the installation position by more than 4 cm, it may put the patient's life in danger.

For the detection of endotracheal tube displacement, Elaanba et al. published Automatic detection Using Deep Convolutional Neural Networks for 11 Abnormal Positioning of Tubes and Catheters in Chest X-ray Images on pages 7 to 12 of the 2021 IEEE World AI IoT Congress, revealing the use of X-rays to scan the patient's chest and use convolutional neural networks (CNNs) to determine the position of the endotracheal tube in the displaced chest X-ray images. Ullah et al. proposed a sensitive, non-invasive, operator-free and cost-effective optical endotracheal intubation in the Real-Time Optical Monitoring of Endotracheal Tube Displacement paper in Biosensors, vol. 10, pages 1 to 10, 2020. In 2010, Lederman et al. published a new automatic endotracheal positioning confirmation system in Journal of Clinical Monitoring and Computing, vol. 24, pages 335-340. The system uses a miniature CMOS sensor installed at the front end of the endotracheal tube and uses a human model algorithm to identify any movement of the endotracheal tube. The WO2019113536A1 patent published by the World Intellectual Property Organization (WIPO) discloses an endotracheal tube protection device with an optional retention system and an optional sensor. The device mainly includes an endotracheal tube retainer and a sensor, wherein the sensor issues an alarm when the movement of the endotracheal tube protection device exceeds a preset movement threshold.

Although the above method can detect any displacement of the endotracheal tube, it requires changing the structure of the existing endotracheal tube and obtaining approval from the medical device authority, or adding complex wearable equipment. In addition, any modification of the structure of the endotracheal tube will increase medical expenses or charge additional fees, increasing the burden on patients.

In order to solve the above problems, the present invention provides a monitoring method for effectively and accurately monitoring the displacement of medical tubes without changing the existing structure of the medical tubes. The present invention uses a simple and effective marking method and artificial intelligence image recognition to detect the displacement of medical tubes in real time. The present invention is a method for monitoring the displacement of medical tubes by using artificial intelligence through labels, comprising the following steps: taking a plurality of images of a workspace; using a trained artificial intelligence to identify a first label, a second label and a third label in the plurality of images, wherein the first label and the second label are fixed on a movable medical tube, the first label and the second label are separated by a calibrated distance along a path in the workspace, and the third label is fixed outside the medical tube, the artificial intelligence is trained by using a label recognition algorithm based on a convolution neural network with a plurality of original label images; continuously calculating the plurality of labels; The pixel distance between the first label and the second label in the plurality of images is defined as a baseline; the pixel distance between the third label and the first label or between the third label and the second label in the plurality of images is continuously calculated, and the pixel distance is defined as a reference line; a reference line movement value is continuously calculated from the plurality of images, and the reference line movement value is the result of the change of the reference line in the pixel distance in the plurality of images; when the reference line movement value is greater than the baseline, the ratio value of the baseline or the multiple value of the baseline, it is determined that the medical tube is displaced on the path of the working space.

In the present invention, each time unit is used to shoot the image with the same number of frames. In the continuous images obtained in any time unit, the displacement result of the third tag on the pixel is defined as a third tag displacement value. When the third tag displacement value of any time unit is less than a threshold value and the third tag displacement value of the previous time unit is greater than the threshold value, it is judged that the third tag is in a stable state. The artificial intelligence recognition starts to perform recognition and calculation when the third tag is judged to be in the stable state.

The aforementioned artificial intelligence is trained based on a convolutional neural network using a marker identification algorithm (MIA) with multiple original label images. The artificial intelligence marks the first label with a first rectangle, the second label with a second rectangle, and the third label with a third rectangle for the target that meets the label characteristics in the multiple images, and calculates the aforementioned baseline and the aforementioned reference line with the center points of the first rectangle, the second rectangle, and the third rectangle, respectively.

The first label, the second label and the third label have different color characteristics or different shape characteristics respectively.

After determining that the medical tube is displaced, the present invention further issues a warning message. When the artificial intelligence cannot identify the first label, the second label or the third label in the plurality of images, a warning message is also issued. The warning message can be presented in the form of sound, light or digital warning pattern. The transmission method of the warning message can be wired transmission or wireless transmission.

The above-mentioned medical tubes can be one of the following: endotracheal tube, chest tube, drainage tube, nasogastric tube, orogastric tube, ECMO tube, etc.

The present invention is also a computer program for execution on a computer, which executes the above-mentioned method of monitoring the displacement of medical tubes by artificial intelligence through tags.

The present invention is also a computer-readable medium, which records the aforementioned computer program.

The present invention is also a device for monitoring the displacement of medical tubes by artificial intelligence through tags, comprising: a first tag, which is arranged on the medical tube; a second tag, which is arranged on the medical tube and is separated from the first tag by a calibrated distance along a path; a third tag, which is arranged at a reference position outside the medical tube; a camera unit, which takes a plurality of images, each of which includes the first tag, the second tag and the third tag; an artificial intelligence module, which is connected to the camera unit, and the artificial intelligence is trained by a tag recognition algorithm based on a convolution neural network with a plurality of original tag images; the artificial intelligence module receives the plurality of images and identifies the first tag, the second tag and the third tag. three labels; the artificial intelligence module continuously calculates the pixel distance between the first label and the second label in the plurality of images, and defines the pixel distance as a baseline; the artificial intelligence module continuously calculates the pixel distance between the third label and the first label or the third label and the second label in the plurality of images, and defines the pixel distance as a reference line; from the plurality of images, the artificial intelligence module continuously calculates a reference line movement value, and the reference line movement value is the result of the change in the pixel distance in the plurality of images; when the reference line movement value is greater than the baseline, the ratio value of the baseline, or the multiple value of the baseline, it is determined that the medical tube is displaced on the path of a working space.

The first label, the second label and the third label have different color characteristics or different shape characteristics respectively.

The present invention further includes a display unit, which is connected to the artificial intelligence module, and the display unit displays the plurality of images and the first label, the second label and the third label.

The present invention further includes a warning unit, which is connected to the artificial intelligence module signal. After the artificial intelligence module determines that the medical tube is displaced, the artificial intelligence module sends a warning message and transmits the warning message to the warning unit by wire or wireless means.

According to the above technical features, the present invention has the following effects:

1. Applicable to various medical tubes that need to monitor displacement. It is not limited by the differences in styles and sizes of medical tubes from different brands, nor will it affect the structure or configuration of existing medical tubes. It only needs to set the first tag and the second tag on the existing medical tube and set the third tag at a reference position outside the medical tube, and then take pictures. The artificial intelligence module is used to continuously identify the first tag, the second tag and the third tag, and continuously calculate the pixel distance between the first tag and the second tag, and the pixel distance between the second tag and the third tag. By the change of the pixel distance, it can instantly and accurately determine whether the medical tube has been displaced.

2. No additional customized medical tubes are required. The attached first label, second label and third label are easy to combine, remove, replace and adjust the distance. The cost of the first label, second label and third label is extremely low, which can effectively reduce the cost and manpower of monitoring and save medical expenses.

3. For the displacement of medical tubes, there is no need to rely on manual monitoring by nursing staff, and it can provide full-time and accurate monitoring.

4. Once the medical circuit is abnormally displaced, the artificial intelligence module can immediately judge and send a near-end or far-end warning message, so that the nursing staff can immediately eliminate the abnormal displacement of the medical circuit and effectively avoid the harm to the patient caused by the displacement of the medical circuit.

5. The camera unit and artificial intelligence module can monitor the displacement of medical tubes without any contact with the patient, which can effectively avoid the discomfort caused by the monitoring equipment to the patient.

1: First label

11: First rectangle

110: Center point

2: Second label

21: Second rectangle

210: Center point

3: Third label

31: The third rectangle

31: Base position

310: Center point

311: First endpoint

312: Second endpoint

313: The third endpoint

314: The fourth endpoint

4: Photography unit

41: Image

5: Artificial intelligence module

51: Wireless transmission unit

6: Display unit

7,7A,7B: Warning unit

71A, 71B: Warning pictures

8: Medical pipeline

81: Axis

9: Electronic devices

D ₁₂ : Baseline

D ₂₃ : Reference line

L: Calibration distance

D _total : displacement value of the third label

TH _totall : threshold

:n+k幀基準線

:n+k frame baseline

:n幀參考線

:n frame reference line

:n+k幀參考線

:n+k frame reference line

D _offset : reference line shift value

:n+k幀參考線移動值

:n+k frame reference line shift value

P: Patient prosthesis

S1: Phase 1

S2: The second stage

S3: The third stage

S4: The fourth stage

[Figure 1] is a schematic diagram of the hardware configuration of the device of the present invention that uses artificial intelligence to monitor the displacement of medical tubes through tags.

[Figure 1A] is a schematic diagram of the method of the present invention for monitoring the displacement of medical tubes by artificial intelligence through tags, and attaching the first tag, the second tag and the third tag to the medical tube and the reference position.

[Figure 2] is a schematic diagram of the camera unit and the artificial intelligence module of the present invention configured in a workspace, wherein the camera unit is used to capture images containing the first label, the second label, and the third label.

[Figure 3] is a schematic diagram of the first label, the second label and the third label in the artificial intelligence recognition image of the method of the present invention for monitoring the displacement of medical tubes by artificial intelligence through labels.

[Figure 4] is a schematic diagram of the operating steps of the method of the present invention for monitoring the displacement of medical tubes through artificial intelligence using tags.

[Fifth Figure] is a schematic diagram of the method of the present invention for monitoring the displacement of medical tubes through artificial intelligence using labels, in which artificial intelligence calculates the center points of the first label, the second label, and the third label in the image.

[Figure 6] is a schematic diagram of the present invention's method of using artificial intelligence to monitor the displacement of medical tubes through tags to calculate the pixel distance between the first tag and the second tag, and the pixel distance between the second tag and the third tag.

[Figure 7] is one of the images taken by the camera unit in the method of monitoring the displacement of medical tubes by artificial intelligence through tags of the present invention, showing a schematic diagram of a patient's prosthesis inserted into an endotracheal tube and the patient's prosthesis in a supine position.

與n+k幀參考線移動值

之間的畫素距離與影像幀數關係圖。 [Figure 7A] In the method of the present invention for monitoring the displacement of medical tubes by artificial intelligence through tags, the patient's prosthesis is supine. In the four stages of the simulation, the baseline D ₁₂ , the reference line D ₂₃ , the n+k frame reference line

The reference line shift value of n+k frames

The relationship between the pixel distance and the number of image frames.

與n+k幀參考線移動值

的示意圖。 [Figure 7B] In the method of the present invention for monitoring the displacement of medical tubes by artificial intelligence through tags, the patient's prosthesis is lying on his back, and the medical tubes are before and after displacement, and the baseline D ₁₂ , reference line D ₂₃ , and n+k frame reference line

The reference line shift value of n+k frames

Schematic diagram of .

[Figure 8] is the second image captured by the camera unit in the method of monitoring the displacement of medical tubes by artificial intelligence through tags of the present invention, showing a schematic diagram of a patient's prosthesis inserted into an endotracheal tube and the patient's prosthesis lying on its side. .

與n+k幀參考線移動值

之間的畫素距離與影像幀數關係圖。 [Figure 8A] shows the method of monitoring the displacement of medical tubes by artificial intelligence through tags in the present invention. The patient's prosthesis is turned from supine to side-lying. In the four stages of the simulation, the baseline D ₁₂ , the reference line D ₂₃ , and the n+k frame reference line

The reference line shift value of n+k frames

The relationship between the pixel distance and the number of image frames.

[Figure 9] is a schematic diagram of the warning unit of the present invention displaying a warning pattern.

[Figure 10] is a schematic diagram of another warning unit of the present invention displaying a warning pattern.

The medical tubes referred to in the present invention include but are not limited to endotracheal tubes, chest tubes, drainage tubes, nasogastric tubes, orogastric tubes, ECMO tubes, etc. The first label, second label and third label referred to in the present invention are intended for artificial intelligence to identify from images. The first label and second label are not necessarily independent objects. For example, the first label and the second label are printed on a label paper with different color rings at a fixed distance so that artificial intelligence can identify the first label and the second label from the image. These are also the first label and the second label defined in the present invention. The present invention will be explained by an example of inserting a patient's prosthesis into an endotracheal tube and fixing it with tape to monitor whether the endotracheal tube is pulled out and causes displacement.

Please refer to the first figure and the first A figure. The present invention includes a first label 1, a second label 2, a third label 3, a camera unit 4, an artificial intelligence module 5, a display unit 6 and a warning unit 7 on the hardware. The above hardware is used to implement the method of the present invention for monitoring the displacement of medical tubes through labels with artificial intelligence. When implementing the method of the present invention, the pre-operation is to stick and fix the first label 1, the second label 2, and the third label 3.

The first label 1 is a fixed-width red sticker, and its length is sufficient to wrap around the outer edge of the medical tube 8. The width of the fixed-width first label 1 is, for example, 0.5 cm. The first label 1 is attached and fixed around the outer edge of the medical tube 8. The first label 1 is attached in a wrap-around manner. The upper edge and the lower edge of the first label 1 are aligned at the wrap-around overlap, so that no matter how the medical tube 8 rotates along the axis 81 of the medical tube 8, the first label 1 can maintain a consistent height when being photographed. It is preferred that the color of the first label 1 is clearly different from the color of the medical tube 8, the color of the patient's skin color, hair color, clothing, etc., and the color of the objects around the bed, so that the first label 1 can be easily identified by artificial intelligence after shooting. In addition to color features, the first label 1 can also have shape features. It is preferred that the shape features are different from the shape of the patient's clothing and the shape of the objects around the bed.

The second label 2 is a blue sticker of fixed width, and its length is sufficient to wrap around the outer edge of the medical tube 8. The width of the second label 2 of fixed width is, for example, 0.5 cm. The second label 2 is adhered and fixed around the outer edge of the medical tube 8. The upper edge and the lower edge of the second label 2 are aligned at the overlapping part of the wrapping, so that no matter how the medical tube 8 rotates along the axis 81 of the medical tube 8, the second label 2 can maintain a consistent height when photographed. The center point of the second label 2 is separated from the center point of the first label 1 by a calibrated distance L in the direction of the axis 81. The calibrated distance L in this embodiment is 1 cm. Preferably, the color of the second tag 2 is different from the color of the first tag 1, and is also obviously different from the color of the medical tube 8, and is also different from the color of the patient's skin color, hair color, clothing, etc., and is also different from the color of the objects around the bed, so that the second tag 2 can be easily identified by artificial intelligence after shooting. In addition to color features, the second tag 2 can also have shape features. Preferably, the shape features are different from the shape of the patient's clothing and the shape of the objects around the bed.

The third label 3 is a green sticker of a fixed width, and its length is sufficient to be horizontally pasted on a reference position 31 outside the medical tube 8. The width of the third label 2 of the fixed width is, for example, 0.5 cm. The reference position 31 of the present embodiment is the nose tip of the patient's prosthesis P, so that the third label 3 can be photographed regardless of whether the patient's prosthesis P is lying on its back or on its side. It is preferred that the color of the third label 3 is different from the colors of the first label 1 and the second label 2, and is also obviously different from the color of the medical tube 8, and is also different from the color of the patient's skin color, hair color, clothing, etc., and is also different from the color of the objects around the bed, so that the third label 3 can be easily recognized by artificial intelligence after being photographed. In addition to color features, the third label 3 may also have shape features. Preferably, the shape feature is different from the shape of the patient's clothing and the shape of the objects around the bed.

Please refer to the first, second and third figures. The camera unit 4 is installed at the head of the bed where the patient prosthesis P lies, and shoots the first tag 1, the second tag 2 and the third tag 3 in real time, so that the first tag 1, the second tag 2 and the third tag 3 can be simultaneously presented in the shot image 41. The camera unit 4 of this embodiment is a camera, and the lens is aimed at the head of the patient prosthesis P to shoot the image 41 in real time. The shot image 41 is, for example, 20 frames per second, and the image 41 is transmitted to the artificial intelligence module 5 for recognition and calculation.

The artificial intelligence module 5 is signal-connected to the camera unit 4, and the image 41 captured by the camera unit 4 is transmitted to the artificial intelligence module 5 for identification through artificial intelligence. The artificial intelligence module 5 of this embodiment includes an artificial intelligence edge platform, which enables the artificial intelligence after deep learning to accurately identify the first label 1, the second label 2 and the third label 3 in the aforementioned image 41. The artificial intelligence module 5 can also have a built-in wireless transmission unit 52 for transmitting signals to other electronic devices 9 such as mobile phones, smart watches, tablet computers, laptops, personal computers, servers, cloud databases, etc. in a wireless manner such as Bluetooth and WiFi. The electronic device 9 is, for example, carried or worn by a nurse, a caregiver, or a patient's family member, or is configured in a nursing station or a nursing vehicle. The deep learning of artificial intelligence is based on a convolutional neural network trained with a plurality of original labeled images using a marker identification algorithm (MIA). The plurality of original labeled images can be taken from 200 images 41 of the bedridden patient or the patient prosthesis P taken by the camera unit 4. The images 41 are taken from different shooting angles and lighting conditions. The data enhancement method is further adopted, including random scaling, random rotation, random cropping, random erasure, random shifting, etc., to expand the 200 original labeled images to 2000 images, and the first label 1, the second label 2 and the third label 3 in the image are marked by an image annotation tool as training data for artificial intelligence. Then, the convolutional neural network is used to train artificial intelligence using a label recognition algorithm, and the above-mentioned split of training data is used, for example, 70% of the training data is used for training and 30% of the training data is used for verification, to establish an artificial intelligence prediction model.

The display unit 6 is connected to the artificial intelligence module 5 signal, and is used to display the image 41 taken by the camera unit 4 and the calculation result of the artificial intelligence module 5.

The warning unit 7 is connected to the artificial intelligence module 5 by signal, and is used to receive the warning message issued by the artificial intelligence module 5 to issue a warning. The warning unit 7 of this embodiment is a buzzer, which emits a sound as a warning. However, the warning unit 7 of the present invention can also be other components, such as using LED to emit light as a warning. Alternatively, the aforementioned display unit 6 is used as the warning unit 7A, and the display screen of the warning unit 7A is used to display a warning pattern and emit a warning sound as a warning. The warning unit 7 can also be a remote electronic device 7B, such as a mobile phone, an electronic watch, a tablet computer or a computer, etc.

Please refer to the second, third and fourth figures. The method of the present invention for monitoring the displacement of medical tubes by artificial intelligence through tags includes the following steps: Use the camera unit 4 to continuously shoot multiple images 41 of a workspace, the image 41 includes the head of the bed patient prosthesis P and the exposed section of the medical tube 8 inserted into the trachea of the patient prosthesis P through the mouth, and the image 41 also includes the first tag 1, the second tag 2 and the third tag 3.

Please refer to the second, fourth, fifth and sixth figures. The artificial intelligence module 5 uses artificial intelligence to identify the first label 1, the second label 2 and the third label 3 that meet the label characteristics in all images 41, and marks the first label 1 with a first rectangle 11, the second label 2 with a second rectangle 21, and the third label 3 with a third rectangle 31. The first rectangle 11, the second rectangle 21 and the third rectangle 31 all have four endpoints. The pixel of each image 41 is used as a coordinate unit, the horizontal direction is set as X, the vertical direction is set as Y, and a coordinate origin (0,0) is selected. For example, the upper left corner of the screen is used as the coordinate origin (0,0), and the image 41 is defined as a rectangular coordinate plane. In this way, the four endpoints of the first rectangle 11, the second rectangle 21 and the third rectangle 31 mentioned above will each have a coordinate point in the rectangular coordinate plane. Taking the third rectangle 31 as an example, the four endpoints of the third rectangle 31 are a first endpoint 311, a second endpoint 312, a third endpoint 313 and a fourth endpoint 314, wherein the coordinates of the first endpoint 311 are (X ₃₁₁ , Y ₃₁₁ ), the coordinates of the second endpoint 312 are (X ₃₁₂ , Y ₃₁₂ ), the coordinates of the third endpoint 313 are (X ₃₁₃ , Y ₃₁₃ ), and the coordinates of the fourth endpoint 314 are (X ₃₁₄ , Y ₃₁₄ ). The coordinates (X ₃₁₀ , Y ₃₁₀ ) of the center point 310 of the third rectangle 31 can be obtained by the following formula: X ₃₁₀ =(X ₃₁₁ +X ₃₁₄ )/2

Y ₃₁₀ =(Y ₃₁₁ +Y ₃₁₄ )/2 Similarly, the coordinates (X ₁₁₀ , Y ₁₁₀ ) of the center point 110 of the first rectangle 11 and the coordinates (X ₂₁₀ , Y ₂₁₀ ) of the center point 210 of the second rectangle 21 can be calculated.

After confirming the coordinates ( _X110 , _Y110 ) of the center point 110 of the first rectangle 11, the coordinates ( _X210 , _Y210 ) of the center point 210 of the second rectangle 21, and the coordinates ( _X310 , _Y310 ) of the center point 310 of the third rectangle 31, the pixel distance _D12 between the center point 110 and the center point 210, and the pixel distance _D23 between the center point 210 and the center point 310 can be obtained by Pisces theorem. That is:

上述D₁₂為影像41中的第一矩形11與第二矩形21之間的畫素距離，代表影像41中第一標籤1與第二標籤2之間的畫素距離，本發明將定義為一基準線D₁₂。D₂₃ 為影像41中的第二矩形21與第三矩形31之間的畫素距離，代表影像41中第二標籤2與第三標籤3之間的畫素距離，本發明將之定義為一參考線D₂₃。

The above _D12 is the pixel distance between the first rectangle 11 and the second rectangle 21 in the image 41, representing the pixel distance between the first label 1 and the second label 2 in the image 41, and the present invention defines it as a baseline _D12 . _D23 is the pixel distance between the second rectangle 21 and the third rectangle 31 in the image 41, representing the pixel distance between the second label 2 and the third label 3 in the image 41, and the present invention defines it as a reference line _D23 .

，其中n為正整數；所拍攝第n+k幀影像41中的參考線D₂₃的畫素長度定義為一n+k幀參考線

，其中k為正整數。根據下列算式：

可以獲得n+k幀參考線移動值

，該n+k幀參考線移動值

代表上述參考線移動值D_offset在第n+k幀影像41中的移動結果。如果n+k幀參考線移動值

大於該幀影像41中的n+k幀基準線

，判定在該n+k幀影像41的拍攝時間點，該醫療管路8產生移位，且實際移位距離超前述的標定距離L的1公分。此乃由於位於醫療管路8上的第一標籤1與第二標籤2，在實際的工作空間 中的標定距離L是不變的，當標定距離L設定為1公分，則影像41中的基準線D₁₂的畫素距離即代表上述標定距離L的1公分，在任一幀影像41中，當參考線移動值D_offset大於該基準線D₁₂，即表示位於醫療管路8上的第二標籤2與位於病人鼻尖處的第三標籤3之間的實際移動距離超過1公分。 In any image 41, when a reference line shift value D _offset of the reference line D ₂₃ is greater than the baseline D ₁₂ in the image 41, it means that the medical tube 8 has shifted more than 1 cm along the axis 81. Before calculating the reference line shift value D _offset , the present invention first confirms that the patient is in a stable state. For example, one second is used as a time unit, and 20 frames of images 41 are captured in one second. The sum of the linear displacements of the pixels of the third tag 3 in the one second is defined as the third tag displacement value D _total . A threshold value TH _total is set to confirm the movement of the patient's head. For example, the threshold value TH _total is set to 200 pixels. Compare the change of the displacement value D _total of the third tag between the current second and the previous second, for example, if D _total of the current second <TH _total and D _total of the previous second >TH _total , then determine that the patient's head changes from the moving state of the previous second to the stable state of the current second. Preferably, the present invention, based on the above-mentioned determination method, starts to identify the first tag 1, the second tag 2 and the third tag 3 when it is determined that the patient's head is in a stable state, and calculates the baseline D ₁₂ , the reference line D ₂₃ and the reference line displacement value D _offset . When the patient is in the aforementioned stable state, the camera unit 4 starts shooting, and the pixel length of the reference line D ₂₃ in the captured n-th frame image 41 is defined as an n-frame reference line.

, where n is a positive integer; the pixel length of the reference line D ₂₃ in the captured n+k frame image 41 is defined as an n+k frame reference line

, where k is a positive integer. According to the following formula:

You can get the n+k frame reference line movement value

, the n+k frame reference line shift value

represents the result of the reference line shift value D _offset in the n+k frame image 41.

Greater than the n+k frame baseline in the frame image 41

, it is determined that at the time point of shooting the n+k frame image 41, the medical tube 8 is displaced, and the actual displacement distance exceeds 1 cm of the aforementioned calibration distance L. This is because the calibration distance L between the first label 1 and the second label 2 on the medical tube 8 in the actual working space is constant. When the calibration distance L is set to 1 cm, the pixel distance of the reference line _D12 in the image 41 represents 1 cm of the aforementioned calibration distance L. In any frame image 41, when the reference line displacement value _Doffset is greater than the reference line _D12 , it means that the actual displacement distance between the second label 2 on the medical tube 8 and the third label 3 located at the nose tip of the patient exceeds 1 cm.

Once the reference line shift value D _offset is greater than the baseline D ₁₂ , the artificial intelligence in the artificial intelligence module 5 will send a warning message to the warning unit 7 .

Please refer to the first figure, the first A figure, and the second figure. The present invention inserts a medical tube 8 into the mouth of the patient's prosthesis P to simulate the state of inserting the tube into the patient's trachea through the mouth. The exposed part of the medical tube 8 is affixed with a first label 1 and a second label 2, and a calibrated distance L of 1 cm is maintained along the axis 81 of the medical tube 8. A third label 3 is affixed to the nose tip of the patient's prosthesis 9. The camera unit 4 continuously shoots the patient's prosthesis P, and transmits the shot image 41 to the artificial intelligence module 5. The display unit 6 displays the image 41 and the monitoring results of the artificial intelligence in real time. The monitoring effect of the present invention is tested through the following four simulation stages, which include:

Phase 1 S1: Initially during the intubation of the medical circuit 8, the patient prosthesis P lies supine and remains still for a period of time.

The second stage S2: Shake the patient prosthesis P to simulate the patient entering a mobile state who can move independently.

The third stage S3: The patient's prosthesis P stops shaking, the patient's prosthesis P returns to a stable state, and the patient's prosthesis P lies on its back or side.

Stage 4 S4: After a brief pause, the medical tube 8 was manually pulled out more than 1 cm.

與n+k幀參考線移動值

在畫素距離上的變化結果。在第一階段S1的20~125幀中，基準線D₁₂的畫素距離保持不變，參考線D₂₃的畫素距離也保持不變，n+k幀參考線

的畫素距離與上述參考線D₂₃的畫素距離相同，參考線n+k幀移動值

的畫素距離為零。在第二階段S2的126~205幀中，病人假體P非處於安定狀態，人工智慧暫停監測，基準線D₁₂、參考線D₂₃、n+k幀參考線

與n+k幀參考線移動值

的畫素距離均為零。在第三階段S3的205~268幀中，病人假體P處於安定狀態，人工智慧恢復監測，基準線D₁₂、參考線D₂₃、n+k幀參考線

與n+k幀參考線移動值

的畫素距離與第一階段S1相同。在第四階段S4的269~358幀中，請配合參閱第七圖、第七A圖與第七B圖，其中第七B圖為在偵測過程中該第一標籤1、第二標籤2與第三標籤3的相對移動示意圖，第七B圖中的虛線表示第一標籤1與第二標籤2移動路徑及移動後的結果。在第四階段S4中，基準線D₁₂與參考線D₂₃的畫素距離均維持不變，但隨著醫療管路8的被抽動而移位，n+k幀參考線

在每一幀中均開始變大，同時n+k幀參考線移動值

在每一幀中也都開始有變大，該n+k幀參考線移動值

在第282~358幀之間均大於該基準線D₁₂或n+k幀基準線

，人工智慧判定在該282~358幀的時間內，該醫療管路8產生移位。在本實施例中，上述基準線D₁₂與n+k幀基準線

相同，但如果在第四階段S4中，除了醫療管路8產生移位之外，病人假體P同時也在該第四階段S4有頭部搖擺或仰傾的動作，則基準線D₁₂與n+k幀基準線

可能會不同，因此，較佳是以n+k幀參考線移動值

大於n+k幀基準線

來判斷醫療管路8產生移位。 Please refer to FIG. 7 and FIG. 7A. The patient's prosthesis P is in a stable supine position in the first stage S1, the third stage S3 and the fourth stage S4. FIG. 7A is the detection result from the first stage S1 to the fourth stage S4. The X axis of FIG. 7A is the image frame number and the Y axis is the pixel distance. FIG. 7A shows that during the first stage S1, the second stage S2, the third stage S3 and the fourth stage S4, the baseline D ₁₂ reference line D ₂₃ and the n+k frame reference line

The reference line shift value of n+k frames

The result of the change in pixel distance. In the first stage S1, the pixel distance of the baseline _D12 remains unchanged, the pixel distance of the reference line _D23 also remains unchanged, and the pixel distance of the reference line n+k frame remains unchanged.

The pixel distance of the reference line D 23 is the same as the pixel distance of the reference line D ₂₃ , and the frame shift value of the reference line n+k is

The pixel distance is zero. In the second stage S2, from 126 to 205 frames, the patient's prosthesis P is not in a stable state, the artificial intelligence suspends monitoring, the baseline D ₁₂ , the reference line D ₂₃ , and the n+k frame reference line

The reference line shift value of n+k frames

The pixel distances of are all zero. In the third stage S3, from 205 to 268 frames, the patient's prosthesis P is in a stable state, and artificial intelligence resumes monitoring. The baseline D ₁₂ , the reference line D ₂₃ , and the n+k frame reference line

The reference line shift value of n+k frames

The pixel distance is the same as that in the first stage S1. In the 269-358 frames of the fourth stage S4, please refer to Figure 7, Figure 7A and Figure 7B, where Figure 7B is a schematic diagram of the relative movement of the first tag 1, the second tag 2 and the third tag 3 during the detection process, and the dotted line in Figure 7B represents the movement path of the first tag 1 and the second tag 2 and the result after the movement. In the fourth stage S4, the pixel distances of the baseline D ₁₂ and the reference line D ₂₃ remain unchanged, but shift as the medical tube 8 is pulled, and the reference line n+k frame

It starts to grow in each frame, and the reference line moves by the value of n+k frame.

In each frame, the reference line shift value of the n+k frame also starts to increase.

The 282nd to 358th frames are all greater than the baseline D ₁₂ or the n+k frame baseline

, the artificial intelligence determines that the medical tube 8 is displaced within the time of 282-358 frames. In this embodiment, the above-mentioned baseline D ₁₂ and the n+k frame baseline

However, if in the fourth stage S4, in addition to the displacement of the medical tube 8, the patient's prosthesis P also has a head sway or tilt in the fourth stage S4, then the baseline D ₁₂ and the n+k frame baseline

may be different, so it is better to use n+k frame reference line shift value

Greater than n+k frame baseline

To determine whether the medical tube 8 has shifted.

與n+k幀參考線移動值

在畫素距離上的變化結果。在第一階段S1的20~100幀中，基準線D₁₂的畫素距離保持不變，參考線D₂₃的畫素距離也保持不變，n+k幀參考線

的畫素距離與上述參考線D₂₃的畫素距離相同，n+k幀參考線移動值

的畫素距離為零。在第二階段S2的101~180幀中，病人假體P非處於安定狀態，人工智慧暫停監測，基準線D₁₂、參考線D₂₃、n+k幀參考線

與n+k幀參考線移動值

的畫素距離均為零。在第三階段S3的180~242幀中，病人假體P轉為側臥並處於安定狀態，人工智慧恢復監測，由於病人假體P由仰臥轉為側臥，第三階段S3的基準線D₁₂、參考線D₂₃、n+k幀參考線

與n+k幀參考線移動值

的畫素距離與第一階段S1大致相同，但其中n+k幀參考線

因影像41的拍攝角度不同而在第三階段S3中略有變小。在第四階段S4的243~358幀中，基準線D₁₂與參考線D₂₃的畫素距離均維持不變，但隨著醫療管路8的被抽動，n+k幀參考線

在每一幀中均開始有變化，同時n+k幀參考線移動值

在每一幀中也都開始有變化，其中該n+k幀參考線移動值

在第254~358幀之間均大於該基準線D₁₂，人工智慧判定在254~358幀的時間內，該醫療管路8產生移位。 Please refer to Figure 8 and Figure 8A. Figure 8 shows that the patient prosthesis P turns to side-lying in the third stage S3. Figure 8A is the detection result from the first stage S1 to the fourth stage S4. The X axis of Figure 8A is the image frame number and the Y axis is the pixel distance. Figure 8A shows that in the process of the first stage S1, the second stage S2, the third stage S3 and the fourth stage S4, the baseline D ₁₂ , the reference line D ₂₃ , the n+k frame reference line

The reference line shift value of n+k frames

The result of the change in pixel distance. In the first stage S1, the pixel distance of the baseline _D12 remains unchanged, the pixel distance of the reference line _D23 also remains unchanged, and the pixel distance of the reference line n+k frame

The pixel distance of the reference line D 23 is the same as the pixel distance of the reference line D ₂₃ , and the n+k frame reference line movement value

The pixel distance is zero. In the second stage S2, from 101 to 180 frames, the patient's prosthesis P is not in a stable state, the artificial intelligence suspends monitoring, the baseline D ₁₂ , the reference line D ₂₃ , and the n+k frame reference line

The reference line shift value of n+k frames

The pixel distances of are all zero. In the third stage S3, from 180 to 242 frames, the patient's prosthesis P turns to lie on the side and is in a stable state. The artificial intelligence resumes monitoring. As the patient's prosthesis P turns from lying on the back to lying on the side, the baseline D ₁₂ , reference line D ₂₃ , and n+k frame reference line D 23 of the third stage S3 are

The reference line shift value of n+k frames

The pixel distance is roughly the same as that of the first stage S1, but the n+k frame reference line

The pixel distance between the baseline _D12 and the reference line D23 remains unchanged in the fourth stage S4 from 243 to 358 frames due to the different shooting angles of the image 41. However, as the medical tube 8 is pulled, the pixel distance between the reference line D12 and the reference line _D23 in the n+k frame increases.

In each frame, the value of the reference line in frame n+k starts to change.

There are also changes in each frame, where the n+k frame reference line movement value

The values between the 254th and 358th frames are greater than the baseline D ₁₂ , and the artificial intelligence determines that the medical tube 8 is displaced during the 254th and 358th frames.

Please refer to the first A, fifth and sixth figures. Although the actual distance between the first label 1 and the second label 2 in the working space remains unchanged, the pixel distance between the first rectangle 11 and the second rectangle 21, i.e., the reference line D ₁₂ , in the rectangular coordinate plane of the image 41 will change with the imaging result caused by the movement, swing or tilt of the medical tube 8. Similarly, the pixel distance between the second rectangle 21 and the third label 13, i.e., the reference line D ₂₃ , will also change with the swing or tilt of the patient's head and the imaging result caused by the pulling of the medical tube 8. In the actual working space, the first tag 1 and the second tag 2 always move synchronously with the medical tube 8, so the actual distance between the first tag 1 and the second tag 2 in the working space will always be maintained at 1 cm. When the patient's head moves and drives the medical tube 8 to move, the third tag 3 on the nose tip and the first tag 1 and the second tag 2 on the medical tube 8 also move synchronously. When the patient's head is in a stable state, that is, the patient's head does not move, the first tag 1, the second tag 2 and the third tag 3 do not move. The patient's head is in a stable state, but when the medical tube 8 is pulled out of the patient's trachea for a distance, the calibration distance L between the first label 1 and the second label 2 on the medical tube 8 remains at 1 cm, but the distance between the third label 3 at the nose tip and the second label 2 on the medical tube 8 will change, or the distance between the third label 3 at the nose tip and the first label 1 on the medical tube 8 will change. Usually, when the medical tube 8 is pulled out of the patient's trachea, the medical tube 8 moves along the axis 81 of the medical tube 8, so it is preferred to set the calibration distance L between the first label 1 and the second label 2 along the axis 81 of the medical tube 8.

In the above embodiment, the calibration distance L between the first tag 1 and the second tag 2 in the direction of the axis 81 is set to 1 cm because after the patient has an endotracheal tube inserted, if the endotracheal tube is pulled out and displaced by more than 1 cm, it will be judged as endotracheal tube displacement, and if the endotracheal tube is pulled out and displaced by more than 4 cm, it will cause harm to the patient. In the embodiment, the reference line shift value D _offset is greater than the baseline D ₁₂ as the criterion for judging displacement, because the baseline D ₁₂ in the image 41 represents the above calibration distance L. Once the reference line shift value D _offset is greater than the baseline D ₁₂ , it means that the endotracheal tube has moved in the direction of the axis 81 by more than 1 cm of the calibration distance L. The embodiment of the present invention chooses to issue a warning before the displacement of the endotracheal tube reaches 4 cm, which helps medical staff to eliminate the displacement problem of the endotracheal tube in the first place. For the endotracheal tube, the above-mentioned calibration distance L can also be set to 0.5 cm, and when the reference line displacement value D _offset is greater than twice the baseline D ₁₂ , it is judged that the endotracheal tube is displaced, which is also a feasible embodiment. Or the above-mentioned calibration distance L is set to 2 cm, and when the reference line displacement value D _offset is greater than half of the baseline D ₁₂ , it is judged that the endotracheal tube is displaced, which is also a feasible embodiment. That is, the calibration distance L does not necessarily have to be actually equal to the displacement warning distance. After setting a certain calibration distance L, it is also feasible to use the ratio or multiple value between the reference line displacement value D _offset and the baseline D ₁₂ as a comparison value for judging the displacement of the endotracheal tube.

For medical tubes 8 other than endotracheal tubes, such as chest tubes, drainage tubes, nasogastric tubes, orogastric tubes, ECMO tubes, etc., the above-mentioned calibration distance L can be freely set according to clinical medical needs, and then artificial intelligence can be used to determine whether the medical tube 8 is displaced.

Please refer to the first, ninth and tenth figures. Once the artificial intelligence determines that the medical tube 8 is displaced, the artificial intelligence module 5 will send a warning message, which can be transmitted to the warning unit 7, 7A, 7B by wire or wireless. When the medical tube 8 is displaced, the buzzer in the warning unit 7 will emit a warning sound; the near-end warning unit 7A will also immediately display a warning pattern 71A to remind the nurse or caregiver to pay attention; at the same time, the remote warning unit 7B will also display a warning pattern 71B at the same time.

In FIG. 6 , the reference line D ₂₃ of this embodiment is the pixel distance between the second tag 2 and the third tag 3. However, if the pixel distance between the first tag 1 and the third tag 3 is used as the reference line instead, the present invention can also be implemented.

If the aforementioned first label 1, second label 2 or second label 3 is blocked or falls off, the artificial intelligence cannot simultaneously identify the first label 1, second label 2 or second label 3 from the image 41, and the artificial intelligence module 5 will also issue another warning message to remind medical staff to eliminate the obstacle.

The present invention is also a computer program for execution on a computer, which executes the above-mentioned method of monitoring the displacement of medical tubes by artificial intelligence through tags.

The present invention is also a computer-readable medium that records the above-mentioned computer program.

Please refer to the first figure, the first figure A, the fifth figure and the seventh figure B. The present invention is also a device for monitoring the displacement of medical tubes by artificial intelligence through tags, including the aforementioned first tag 1, the second tag 2, the third tag 3, the camera unit 4, the artificial intelligence module 5, the display unit 6 and the warning unit 7. Among them: the first tag 1 is provided for setting on a medical tube 8. The second tag 2 is provided for setting on the medical tube 8, and is separated from the first tag 1 by a calibrated distance L along a path, and the path is, for example, the extension direction of the axis 81 of the medical tube 8. The third tag 3 is provided for setting at a reference position 31 outside the medical tube 8, such as the tip of the patient's nose. The photographing unit 4 is used to shoot multiple images 41, each of which includes the first tag 1, the second tag 2 and the third tag 3. The artificial intelligence module 5 is connected to the photographing unit 4, and the artificial intelligence module 5 receives the multiple images 41 and identifies the first tag 1, the second tag 2 and the third tag 3. The display unit 6 is connected to the artificial intelligence module 5 signal. The warning unit 7 is connected to the artificial intelligence module 5 signal.

)。當該參考線移動值D_offset大於該基準線D₁₂、該基準線D₁₂的比例值或該基準線D₁₂的倍數值時，判斷為該醫療管路8在該工作空間的該路徑上產生移位。 The artificial intelligence module 5 continuously calculates the pixel distance between the first label 1 and the second label 2 in the plurality of images 41, and defines the pixel distance as a baseline D ₁₂ . The artificial intelligence module 5 continuously calculates the pixel distance between the second label 2 and the third label 3 in the plurality of images 41, and defines the pixel distance as a reference line D ₂₃ . From the continuous images 41 in the plurality of images 41, the artificial intelligence module continuously calculates a reference line shift value D _offset (indicated as in FIG. 7B ) of the reference line in length.

). When the reference line shift value D _offset is greater than the baseline D ₁₂ , the ratio of the baseline D ₁₂ , or the multiple of the baseline D ₁₂ , it is determined that the medical tube 8 is displaced on the path of the working space.

Combined with the description of the above embodiments, the operation, use and effects of the present invention can be fully understood. However, the above embodiments are only the preferred embodiments of the present invention and cannot be used to limit the scope of implementation of the present invention. That is, simple equivalent changes and modifications made according to the scope of the patent application and the content of the invention description are all within the scope of the present invention.

Claims (13)

A method for monitoring the displacement of medical tubes by using artificial intelligence through labels comprises the following steps: taking a plurality of images of a workspace; using a trained artificial intelligence to identify a first label, a second label and a third label from the plurality of images, wherein the first label and the second label are located on a movable medical tube, the first label and the second label are separated by a calibrated distance along a path in the workspace, and the third label is fixed outside the medical tube, the artificial intelligence is trained by using a label recognition algorithm based on a convolutional neural network with a plurality of original label images; continuously calculating the plurality of images; The pixel distance between the first label and the second label in the image is defined as a baseline; the pixel distance between the third label and the first label or the third label and the second label in the complex image is continuously calculated, and the pixel distance is defined as a reference line; a reference line movement value is continuously calculated from the complex image, and the reference line movement value is the result of the change of the reference line in the pixel distance in the complex image; when the reference line movement value is greater than the baseline, the ratio value of the baseline or the multiple value of the baseline, it is determined that the medical pipeline is displaced on the path of the working space. As described in claim 1, a method for monitoring the displacement of medical tubes by using artificial intelligence through tags, wherein each time unit is to shoot the same number of frames of the image, and in the continuous images obtained in any time unit, the displacement result of the third tag on the pixel is defined as a third tag displacement value, and when the third tag displacement value of any time unit is less than a threshold value and the third tag displacement value of the previous time unit is greater than the threshold value, it is judged that the third tag is in a stable state, and the artificial intelligence recognition starts to identify and calculate when the third tag is judged to be in the stable state. A method for monitoring the displacement of medical tubes by using artificial intelligence through labels as described in claim 1, wherein the artificial intelligence marks the first label with a first rectangle, the second label with a second rectangle, and the third label with a third rectangle for targets that meet the label characteristics in multiple images, and calculates the aforementioned baseline and the aforementioned reference line using the center points of the first rectangle, the second rectangle, and the third rectangle, respectively. A method for monitoring the displacement of medical tubes by using artificial intelligence through tags as described in claim 1, wherein the first tag, a second tag, and a third tag have different color characteristics or different shape characteristics. As described in claim 1, a method for monitoring the displacement of medical tubes by using artificial intelligence through tags, wherein after determining that the medical tube has been displaced, a warning message is further issued. A method for monitoring the displacement of medical tubes by using artificial intelligence through labels as described in claim 1, wherein when the artificial intelligence cannot identify the first label, the second label or the third label in the plurality of images, a warning message is issued. A method for monitoring the displacement of a medical tube by artificial intelligence through a tag as described in claim 1, wherein the medical tube is one of the following: an endotracheal tube, a chest tube, a drainage tube, a nasogastric tube, an orogastric tube, or an ECMO tube. A computer program for executing on a computer, the computer program executing the method of monitoring medical tube displacement by artificial intelligence through tags as described in any one of claim 1 to claim 7. A computer-readable medium recording the computer program described in claim 8. A device for monitoring the displacement of medical tubes by artificial intelligence through tags, comprising: a first tag, arranged on the medical tube; a second tag, arranged on the medical tube, and having a calibrated distance from the first tag along a path; a third tag, arranged at a reference position outside the medical tube; a camera unit, taking a plurality of images, each of which includes the first tag, the second tag and the third tag; an artificial intelligence module, connected to the camera unit, the artificial intelligence is trained based on a convolution neural network with a plurality of original tag images using a tag recognition algorithm; the artificial intelligence module receives the plurality of images and identifies the first tag, the second tag and the third tag. The artificial intelligence module continuously calculates the pixel distance between the first label and the second label in the plurality of images, and defines the pixel distance as a baseline; the artificial intelligence module continuously calculates the pixel distance between the third label and the first label or the third label and the second label in the plurality of images, and defines the pixel distance as a reference line; from the plurality of images, the artificial intelligence module continuously calculates a reference line movement value, and the reference line movement value is the result of the change in the pixel distance in the plurality of images; when the reference line movement value is greater than the baseline, the ratio value of the baseline, or the multiple value of the baseline, it is determined that the medical tube is displaced on the path of a working space. As described in claim 10, a device for monitoring the displacement of medical tubes by using artificial intelligence through tags, wherein the first tag, the second tag, and the third tag have different color characteristics or different shape characteristics respectively. The device for monitoring the displacement of medical tubes by artificial intelligence through labels as described in claim 10 further comprises a display unit connected to the artificial intelligence module, and the display unit displays the plurality of images and the first label, the second label and the third label. The device for monitoring the displacement of medical tubes by artificial intelligence through tags as described in claim 10 further comprises an alarm unit, which is connected to the signal of the artificial intelligence module. After the artificial intelligence module determines that the medical tube is displaced, the artificial intelligence module sends a warning message, and transmits the warning message to the warning unit by wire or wireless means.

Images