TWI897267B - Method and device for drug identification using deep learning object detection technology - Google Patents

Abstract

A method and device for using deep learning object detection technology to assist in drug identification. The device includes a camera for acquiring images, providing deep learning and object analysis, and an artificial intelligence server for performing image and picture analysis and comparison, and providing probability feedback to the user. This device should be connected to a medical information system. During operation, pharmacists access prescription data within the medical information system and send it to the AI server. Simultaneously, the AI server begins capturing video of the dispensing process captured by a camera and dynamically compares the camera image with the drug image database on the AI server. The comparison results and probabilities are then transmitted to the medical information system interface, where they are reported back to assist pharmacists in determining the correctness of the drug. Pharmacists then provide appropriate feedback to the medical information system to further train the AI server.

Description

Method and device for assisting drug identification using deep learning object detection technology

This invention relates to a method and device for utilizing deep learning object detection technology to assist in drug identification, particularly to assisting in the judgment of drug dispensing processes and reducing the incidence of human error.

Medication accuracy is one of the most important monitoring indicators in pharmaceutical operations. However, due to the large workload, near-misses are more likely to occur. Pharmacy departments are constantly striving to improve the accuracy of medication dispensing. Furthermore, medication errors by medical personnel are a common medical dispute, and in severe cases, can even endanger patients' lives.

According to Article 22 of Chapter 1 of the Code of Practice for Good Pharmaceutical Dispensing, "Pharmacy personnel shall verify the accuracy of the contents of the medicine bag or label, the type of medicine, the quantity of medicine, and the prescription instructions when delivering medicines." This regulation specifies that the person who performs the regulation is the pharmacy personnel, not the quality control or yield control restrictions. Legal liability rests with the person who performs the regulation, not the equipment. Therefore, all advanced equipment should be used to assist in judgment, rather than providing direct results of correctness or error.

Currently, patents related to drug identification, such as Taiwan Patent No. I779596, place control over drug authenticity determination in the hands of the system. However, the data obtained from feature matching should be a probability, not simply true or false. Furthermore, static photography is difficult to achieve in busy medical settings, especially when all drugs need to be photographed without overlapping. Therefore, this technology may be difficult to apply in actual medical settings. Another example is Taiwan Patent No. I730508. This technology similarly places the responsibility of determining drug authenticity in the hands of the system. However, the data obtained from feature matching should be a probability, not simply a true or false statement. Furthermore, identifying each medication bag individually would require significant changes to current pharmacist dispensing procedures, potentially reducing overall efficiency in exchange for the possibility of drug identification. Therefore, this technology may be difficult to apply in actual medical settings.

Given that current drug identification systems primarily rely on static identification, the process significantly conflicts with pharmaceutical practices. Given the increased workload, the system will be difficult to sustainably use and maintain. Therefore, the general user experience will not meet actual user needs.

The primary objective of this invention is to overcome the aforementioned problems encountered by learning technologies and provide a method and device for utilizing prescription information within existing hospital information systems (HISs). This method utilizes all drug items in the prescription as a basis for judgment, providing users with an object match rate. This method adds a layer of auxiliary verification without impacting the work style and efficiency of pharmacy personnel, thereby improving the accuracy of pharmacy personnel's dispensing operations and reducing the incidence of drug incidents in the Taiwan Patient-Safety Reporting System (TPR). Furthermore, the method and device utilize deep learning object detection technology to assist in drug identification, using user feedback as a validation benchmark for deep learning, to continuously optimize the device's judgment capabilities.

To achieve the above objectives, the present invention is a method for assisting in drug identification using deep learning object detection technology, which comprises at least the following steps: Step 1: After a pharmacy staff reads a barcode on a prescription on a computer of an HIS, the pharmacy staff displays the prescription data related to the prescription on the operation screen of the computer screen, and transmits the prescription data to an artificial intelligence server equipped with a deep learning recognition unit; Step 2: The pharmacy staff places at least one medicine bag containing at least one medicine according to the prescription bag in the visual area of a camera to manually inspect all the medicines in the medicine bag, and the camera transmits the image data of all the medicines in the medicine bag to the artificial intelligence server; Step 3: The artificial intelligence server uses the prescription bag to identify the medicine bag. The data is linked to a drug feature database to extract the image feature data of all drugs in the prescription. The deep learning recognition unit uses a deep learning algorithm to perform a probability analysis and comparison with the image data, calculates the matching rate of each drug and transmits it to the HIS, which displays the matching rate of all drugs in the medicine bag on the computer screen. Complete and match the rate; Step 4: The pharmacy personnel uses the match rate of each drug as an auxiliary judgment to determine whether the drug in the bag is correct, completing the review of the prescription data and the at least one bag; and Step 5: Based on the review results, the pharmacy personnel feedback the accuracy of the match rate to the deep learning recognition unit in the HIS to retrain the deep learning algorithm.

In the above-mentioned embodiment of the present invention, the medicine bag has a front side containing patient and medicine information and a back side made of a transparent plastic film, into which the medicine can be placed and which can be photographed by the camera to obtain a clear image.

In the above-mentioned embodiment of the present invention, the camera is a high-speed camera with a frame rate of 120 FPS or above.

In the above embodiment of the present invention, the first step is for the pharmacy staff to use a barcode reader connected to the HIS to read the barcode on the prescription.

In the above embodiment of the present invention, the image data of all drugs in step 2 is presented as an image or continuous image based on the quantity of the drugs.

In the above-mentioned embodiment of the present invention, step three uses the image or the entire continuous image to compare with all the drugs in the prescription to determine the probability of each drug in the prescription appearing in the image or continuous image.

In the above embodiment of the present invention, in step 4, when the matching rate after comparison is low, the HIS will prompt the drug item with the low matching rate.

In the above-mentioned embodiment of the present invention, step five involves the pharmacist providing positive or negative feedback to the artificial intelligence server based on the review results to retrain the deep learning recognition unit.

In the above-mentioned embodiment of the present invention, step five further includes the step of providing feedback to the deep learning recognition unit when the deep learning recognition unit makes an incorrect judgment or when the appearance of the drug partially changes, so that the pharmacy personnel can relearn and correct the feature file.

In the above-described embodiment of the present invention, the deep learning algorithm includes at least one of an algorithm based on an auto-encoder architecture, an algorithm based on a multi-stage architecture, an algorithm based on a U-Net architecture, an algorithm based on a GAN architecture, and the convolutional neural network (U-Net XY-Deblur) algorithm specifically for deblurring.

In the above embodiment of the present invention, the convolutional neural network U-Net XY-Deblur algorithm uses an encoder and two decoders as its architecture, and improves the deblurring performance through rotation and shared parameters. Its formula is as follows:

In the above embodiment of the present invention, the artificial intelligence server uses an object detection algorithm to detect the image region (ROI) where the drug is located in the image data of the drug to be identified, and uses the YOLACT (You Only Look At CoefficienTs) algorithm model to perform instance segmentation on the ROI area, calculates the category confidence of each anchor, the position of the bounding box, the coefficient of the mask, and the mask of each target object in the image, so as to separate the foreground drug area from the background image. Finally, a Siamese Convolutional Neural Network (SCNN) is used as the recognition architecture for comparison. The two CNNs share the same parameters and use metric learning to achieve the best classification accuracy. The CNN uses a neural network to improve feature resolution through deep neural network learning. The input is the image data of the drug to be identified and the image feature data of the prescribed drug. A connection function is used to connect the two CNNs, linking the image data of the drug to be identified with the image feature data of the prescribed drug. Finally, a cost function is used to calculate the similarity between the drug to be identified and the prescribed drug.

In the above-mentioned embodiment of the present invention, the object detection algorithm used in the YOLACT algorithm model includes at least one of Region-based CNN (R-CNN), Fast R-CNN, Faster R-CNN, Mask R-CNN, YOLO (You Only Look Once), YOLOv2, YOLOv3, YOLOv7, Single Shot MultiBox Detector (SSD), and RetinaNet.

To achieve the above objectives, the present invention further provides a device that utilizes deep learning object detection technology to assist in drug identification. The device comprises: a camera that captures images of pharmacists dispensing and reviewing medications at a frame rate of 120 FPS or higher, primarily used to collect image data of all medications in at least one medicine bag printed according to a prescription; and an artificial intelligence server connected to the camera and a drug feature database. The artificial intelligence server includes a deep learning recognition unit that receives and stores image data of all medications, analyzes and compares image feature data of a predetermined prescription medication in the medication feature database with the current medication to be identified. The deep learning recognition unit includes: a drug imaging and image deblurring module, which is processed in parallel with the camera and uses a deep learning algorithm to deblur the image data of the drug to be identified; a high-reflective image restoration module, which is connected to the drug imaging and image deblurring modules and uses a polarizing filter to improve the computing speed of the drug imaging and image deblurring modules so that they can match the working speed of the pharmacist; an object detection and instance segmentation module. The pharmaceutical image segmentation module is connected to the pharmaceutical image acquisition and image deblurring module. Before performing pharmaceutical comparison, the object detection algorithm is used to detect the ROI region where the pharmaceutical is located in the image data of the pharmaceutical to be identified, and the YOLACT algorithm model is used to perform instance segmentation on the ROI region. The pharmaceutical image re-identification module is connected to the target object detection and instance segmentation module. The module receives the segmented pharmaceutical image data and Image data of the drug to be identified, where the foreground drug region is separated from the background image, is compared using a SCNN consisting of two CNNs as the recognition architecture. The two CNNs share the same parameters and utilize metric learning to enhance feature resolution. A connection function is then connected to the two CNNs, linking the image data of the drug to be identified with the image feature data of the prescribed drug. Finally, a cost function is used to calculate the similarity between the drug to be identified and the prescribed drug.

In the above-mentioned embodiment of the present invention, the deep learning algorithm includes at least one of an algorithm based on an Auto-Encoder architecture, an algorithm based on a Multi-Stage architecture, an algorithm based on a U-Net architecture, an algorithm based on a GAN architecture, and a convolutional neural network (U-Net XY-Deblur) algorithm specifically for deblurring.

In the above-described embodiment of the present invention, the convolutional neural network (U-Net) XY-Deblur algorithm employs an encoder and two decoders as its architecture, further enhancing its deblurring performance through rotation.

In the above-mentioned embodiment of the present invention, the object detection algorithm used in the YOLACT algorithm model includes at least one of R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, YOLO, YOLOv2, YOLOv3, YOLOv7, SSD, and RetinaNet.

In the above-mentioned embodiment of the present invention, the ROI region instance segmentation utilizes the YOLACT algorithm model, with two parallel sub-network branches, a first sub-network branch and a second sub-network branch, to achieve real-time instance segmentation. The first sub-network branch is the prediction head, which calculates the class confidence, bounding box position, and mask coefficients for each anchor. The second sub-network branch is the prototype network, which generates a mask for each target object in the image, thereby separating the foreground drug area from the background image.

In the above-described embodiment of the present invention, the AI server receives prescription data from a HIS computer and reads the prescription data for the drugs contained in the prescription. Using this prescription data, the server links to the drug feature database to extract the image feature data of all drugs in the prescription. Using a deep learning algorithm, the server then compares the image feature data of the predetermined prescription drug with the image data of the drug to be identified, performing a probability analysis to calculate the match rate for all drugs and transmits it back to the HIS computer for display.

100: Device

1: Computer

11: Screen

2: Camera

21: Viewing area

3: Artificial Intelligence Server

31: Deep Learning Recognition Unit

311: Pharmaceutical Imaging and Image Deblurring Module

312: Highly reflective image restoration module

313: Target Detection and Instance Segmentation Module

314: Pharmaceutical Image Re-identification Module

4: Drug Characteristics Database

5: Prescription

6.6a: Medicine Bag

61, 61A, 61B, 61C: Pharmaceuticals

7: Barcode reader

Z:Pharmacy personnel

Figure 1 is a functional block diagram of the basic implementation of the present invention.

Figure 2 is a schematic diagram of the device architecture and method flow of the first embodiment of the present invention.

Figure 3 is a schematic diagram of the device architecture and method flow of the second embodiment of the present invention.

Figure 4 is a schematic diagram showing the present invention feeding back the matching rate judgment results to the HIS.

Please refer to Figure 1, which is a functional block diagram of a basic implementation of the present invention. As shown in the figure, the present invention is a device 100 that utilizes deep learning object detection technology to assist in drug identification. The device comprises a computer 1 connected to a hospital information system (HIS), a camera 2, and an artificial intelligence server 3.

The HIS mentioned above is an existing device that can provide medical information such as patient data and prescription data.

The camera 2 captures images of pharmacists dispensing and reviewing medications. It primarily collects image data of all medications 61 in at least one medicine bag 6 printed according to a prescription 5. During dynamic imaging, relative motion of the subject or photographer can cause image blur. This is primarily due to excessive camera exposure time, which causes the subject to continuously move during the exposure, leaving behind object pixels in the image. Therefore, the camera 2 of the present invention uses a high-speed camera with a frame rate of 120 FPS or higher, effectively reducing exposure time and object pixel artifacts.

The AI server 3 is connected to the camera 2 and a drug feature database 4. The AI server 3 includes a deep learning recognition unit 31, which receives and stores image data of all drugs 61 and analyzes and compares the image feature data of a predetermined prescription drug in the drug feature database 4 with the image data of the drug 61 to be identified. The deep learning recognition unit 31 includes a drug imaging and image deblurring module 311, a high-reflective image restoration module 312, an object detection and instance segmentation module 313, and a drug image re-identification module 314. Wherein: The drug imaging and image deblurring module 311 is processed in parallel with the camera 2, and the image data of the drug to be identified is deblurred using a deep learning algorithm. The deep learning algorithm can be any of the following: an algorithm based on an Auto-Encoder architecture, an algorithm based on a Multi-Stage architecture, an algorithm based on a U-Net architecture, an algorithm based on a GAN architecture, and a convolutional neural network U-Net XY-Deblur algorithm specifically for deblurring. Among them, the convolutional neural network U-Net XY-Deblur algorithm adopts an encoder and two decoders as the architecture, and improves the deblurring performance by rotating and sharing parameters. The formula is as follows:

The highly reflective image restoration module 312 is connected to the drug imaging and image deblurring module 311. Because this device must be configured to work within the pharmacist's actual working environment, and considering the pharmacist's operating speed, deep neural network processing may cause the device's computing speed to lag behind the pharmacist's. Therefore, a polarizing filter is used as a hardware solution to increase the computing speed of the drug imaging and image deblurring module 311, enabling it to keep pace with the pharmacist's working speed.

The target object detection and instance segmentation module 313 is connected to the drug imaging and image deblurring module 311. Before performing drug comparison, it is necessary to first use an object detection algorithm to detect the image region (Region of Interest, ROI) where the drug is located in the image data of the drug to be identified, and then compare the ROI area. To address the impact of background interference on the ROI region, the ROI region is instance-segmented to separate the foreground drug area from the background image. The YOLACT (You Only Look At CoefficienTs) algorithm model is used to achieve real-time results. Two parallel sub-network branches, the first sub-network branch and the second sub-network branch, are used to implement real-time instance segmentation. The first sub-network branch is the prediction head, responsible for calculating the category confidence, bounding box position, and mask coefficients of each anchor. The second sub-network branch is the prototype network, responsible for generating a mask for each target object in the image. The object detection algorithm may be any of the following: R-CNN, Fast R-CNN, Faster R-CNN, or Mask R-CNN from the Region-based CNNs (R-CNN) series; or YOLO, YOLOv2, YOLOv3, or YOLOv7 from the YOLO (You Only Look Once) series; or Single Shot MultiBox Detector (SSD); or RetinaNet.

The drug image re-identification module 314 is connected to the object detection and instance segmentation module 313. It receives the image data of the drug to be identified, which has been instance segmented to separate the foreground drug area from the background image. It then uses a Siamese Convolutional Neural Network (SCNN) consisting of two CNNs as the recognition architecture for comparison. The two CNNs share the same parameters and use metric learning to upgrade the low-level features obtained by the CNNs to high-level features, thereby increasing the feature resolution. The two CNNs are then connected to a connection function to connect the image data of the drug to be identified with the image feature data of the prescribed drug. Finally, a cost function is used to calculate the cost function. function) to calculate the similarity between the drug to be identified and the prescribed drug. The present invention uses an identification method to train the recognition model, which is more effective than verification training. After metric learning and ranking, a higher accuracy rate can be achieved.

Currently, most HISs have already built patient basic information and prescription data into relevant databases, and will call them out for manual re-comparison during the dispensing process. This device activates when a pharmacist calls up a patient's prescription data. The HIS displays the patient's prescription data on the original system page for manual review by the pharmacist. Simultaneously, the prescription data is sent to the artificial intelligence server 3. The deep learning recognition unit 31 provides identification data for all drugs in the prescription and performs a feature comparison with the camera 2 image. After the comparison is complete, the probability of matching is sent back to the HIS, which displays the drug matching completion and the matching rate on the work interface. Once the pharmacist completes the manual review of the drug, the deep learning recognition unit 31 also completes the matching process for each drug. Only after all drugs have completed manual and deep learning recognition verification can they be delivered to the HIS, achieving assisted identification.

Please refer to Figures 2 through 4, which respectively illustrate the first embodiment of the present invention, the second embodiment of the present invention, and the present invention's method of feeding back the match rate determination results to the HIS. As shown in the figures, the present invention is a method for assisting in drug identification using deep learning object detection technology. This method can be implemented by the device 100 shown in Figure 1. The following details the steps in Figures 2 through 4, using the components shown in Figure 1. The operation steps are as follows: First, in step s11, after the HIS computer 1 reads the barcode on prescription sheet 5, the prescription data associated with prescription sheet 5 is displayed on the work interface of computer 1's screen 11 and transmitted to the artificial intelligence server 3 equipped with a deep learning recognition unit 31.

Referring to Figure 2, in step s11, pharmacy staff member Z uses a barcode reader 7 connected to the HIS computer 1 to read the barcode on the prescription sheet 5. The prescription information is displayed on the screen 11 of the HIS computer 1. The HIS computer 1 then simultaneously sends the prescription information to the artificial intelligence server 3 for extracting the image feature data of the prescribed medication.

In step s12, the pharmacy staff member Z places at least one medicine bag 6 containing at least one medicine 61 according to the prescription 5 in the viewing area 21 of a camera 2 to inspect all the medicines 61 in the medicine bag 6. Simultaneously, the camera 2 transmits image data of all the medicines 61 in the medicine bag 6 to the artificial intelligence server 3.

Referring to Figure 2, in step s12, pharmacy staff member Z takes the medicine bag 6 and places it in the viewing area 21 of the camera 2 for manual inspection of the medicine (Med). Simultaneously, the camera 2 transmits the captured image data to the artificial intelligence server 3.

In step s13, the AI server 3 uses the prescription data to connect to a drug feature database 4 and extract the image feature data of all drugs 61 in the prescription bag 5. The deep learning recognition unit 31 uses a deep learning algorithm to perform a probability analysis and comparison of the image feature data, calculates the matching rate of each drug 61, and transmits it to the HIS, which then displays on the work screen 11 of the computer 1 that all drugs 61 in the medicine bag 6 have been matched and their matching rates.

Referring to Figure 2, in step s13, the AI server 3 receives the image data transmitted by the camera 2, performs a probability analysis on the image data and the image feature data of the prescription drug, and transmits the matching rate to the HIS computer 1.

Referring to Figure 3, in step s13, the AI server 3 receives prescription data from the HIS computer 1, potentially receiving multiple data sets simultaneously. In this figure, because this prescription 5 contains three medicine bags 6a, pharmacy staff member Z can manually inspect the medications (Med A, Med B, Med C) 61A, 61B, and 61C in the camera 2's viewing area 21, either sequentially or non-sequentially. Simultaneously, the AI server 3 receives the continuous image data transmitted by the camera 2 and performs a probability match analysis on it with the three medication image feature data in the prescription 5. The resulting match rate is then transmitted to the HIS computer 1.

In step s14, pharmacy staff member Z uses the matching rates of each drug 61A, 61B, and 61C as an auxiliary basis for judgment and completes the review of the prescription data and the medicine bag 6a.

Please refer to Figure 4. In step s14, the pharmacist Z can start the HIS computer 1 for matching rate judgment after reading the prescription 5, and use this data to assist in judging whether the medicines 61A, 61B, and 61C in the medicine bag 6a are correct. In this embodiment, the matching rates for drug 61A are 90%, for drug 61B is 20%, and for drug 61C is 70%. This means that in the continuous images of drugs 61A, 61B, and 61C, there is a 90% probability that drug 61A is present, a 20% probability that drug 61B is present, and a 70% probability that drug 61C is present. This does not mean that there is a 90% probability that drug 61A is present in drug bag 6a, a 20% probability that drug 61B is present in drug bag 6a, and a 70% probability that drug 61C is present in drug bag 6a.

In step s15, the pharmacist Z feeds back the accuracy of the matching rate in the HIS to the artificial intelligence server 3 to train the deep learning recognition unit 31 again.

Referring to Figure 4, in step s15, after pharmacy officer Z completes the review of the prescription and medicine bag, he or she provides appropriate feedback to the HIS computer 1 based on the review results. In this embodiment, if pharmacy officer Z re-inspects medications 61A, 61B, and 61C and confirms that all medications 61A, 61B, and 61C match the prescription sheet 5, i.e., the medicine bag 6a, then the HIS computer 1 will generate a "Drug Confirmed" message. This message will be fed back to the artificial intelligence server 3 to further train the deep learning recognition unit 31.

Referring to Figure 4 , in step s15, if pharmacy employee Z re-inspects drugs 61A, 61B, and 61C and discovers an error with the drug (second drug 61B) in drug bag 6a, the HIS computer 1 will generate a "drug error" message. This message will be reported back to the artificial intelligence server 3 to retrain the deep learning recognition unit 31.

In a preferred embodiment of the present invention, the medicine bag 6 has a front side containing patient and medicine information and a back side made of a transparent plastic film, into which the medicine 61 can be placed and which can be photographed by the camera 2 to obtain a clear image.

In a preferred embodiment of the present invention, if the match rate after drug feature comparison is low, the HIS will prompt the pharmacist, "Low match rate, please reconfirm whether the drug is correct?" If the deep learning recognition unit 31 makes a misjudgment or if the drug's appearance has partially changed, the pharmacist can provide feedback to the deep learning recognition unit 31, causing it to relearn and modify the feature file, thereby achieving continuous use and continuous maintenance.

In a preferred embodiment of the present invention, the primary purpose of this method is to provide users with an image-based diagnostic accuracy rate with minimal process changes, and to provide prompts for medications with low accuracy rates.

In a preferred embodiment of the present invention, the method is used in conjunction with the HIS. When used, a batch of drug data and image features are obtained from the HIS and matched with all drugs in a segment of image data. The matching rate of each drug's appearance is calculated and provided to the user.

In a preferred embodiment of the present invention, this method only provides a matching rate, and the final judgment should still be made by pharmaceutical personnel or medical personnel in accordance with regulations.

In a preferred embodiment of the present invention, the method retrains the deep learning recognition unit 31 each time the pharmacy or medical staff makes a decision and provides feedback, thereby improving the effectiveness of deep learning.

This invention provides a method and device that utilizes deep learning object detection technology to assist in drug identification. This method is intended to assist pharmacists in performing the dispensing practices outlined in Article 3 of the Guidelines for Good Drug Dispensing Practices, thereby reducing the incidence of human error. Its technical features are as follows:

1. This device will assist in the judgment of the drug mixing process, reducing the incidence of human error.

2. The matching rate demonstrated by this invention provides pharmacists with faster and more accurate diagnostic assistance, thereby reducing their workload and improving overall efficiency. This is particularly important and necessary in an aging society characterized by a shortage of medical personnel and limited capacity.

3. This device combines the hospital information management system with deep learning object detection technology. The hospital information management system narrows the judgment scope, and deep learning object detection technology is then used to complete the identification, improving its accuracy and efficiency. It can also achieve assisted identification without changing current pharmaceutical practices.

In summary, the present invention is a method and device for assisting in drug identification using deep learning object detection technology, which can effectively improve the various shortcomings of drug use. It uses the prescription information in the existing Hospital Information System (HIS) and all the drug items in the prescription as the basis for judgment, providing the user with the object matching rate. Without affecting the work style and efficiency of pharmacists, it adds a layer of auxiliary inspection, thereby improving the accuracy of pharmacists' dispensing operations and reducing the patient safety reporting system (Taiwan Patient-safety Reporting System). The incidence rate of drug events in the TPR system (TPR) is being measured, and user feedback is being used as a validation benchmark for deep learning to continuously optimize the device's judgment capabilities. This will ensure that the invention is more advanced, more practical, and better meets user needs. This invention has indeed met the requirements for an invention patent application, and a patent application has been filed in accordance with the law.

However, the above description is merely a preferred embodiment of the present invention and should not be used to limit the scope of implementation of the present invention. Therefore, any simple equivalent changes and modifications made within the scope of the patent application and the contents of the invention specification should still fall within the scope of coverage of the present patent.

100: Device

1: Computer

2: Camera

3: Artificial Intelligence Server

31: Deep Learning Recognition Unit

311: Pharmaceutical Imaging and Image Deblurring Module

312: Highly reflective image restoration module

313: Target Detection and Instance Segmentation Module

314: Pharmaceutical Image Re-identification Module

4: Drug Characteristics Database

Claims (17)

A method for assisting in drug identification using deep learning object detection technology comprises at least the following steps: Step 1: After a barcode on a prescription is read by a computer in a hospital information system (HIS), prescription data related to the prescription is displayed on the operating screen of the computer screen, and the prescription data is transmitted to an artificial intelligence server equipped with a deep learning recognition unit; Step 2: At least one medicine bag containing at least one medicine according to the prescription bag is placed in a camera viewing area to view all the medicines in the medicine bag. At the same time, the camera transmits the image data of all medicines in the medicine bag to the artificial intelligence server; Step 3: The artificial intelligence server uses the prescription data to link to a medicine feature database to extract the image feature data of all medicines in the prescription, and the deep learning recognition unit uses the deep learning algorithm to perform a probability analysis and comparison of the image data to calculate the probability of each medicine. The drug matching rate is transmitted to the HIS, so that it can be displayed on the operation screen of the computer screen that all drugs in the medicine bag have been matched and their matching rates; Step 4: Using the matching rate of each drug as an auxiliary judgment to determine whether the drug in the medicine bag is correct or not, the prescription data and the at least one medicine bag review operation are completed; and Step 5: Feedback the accuracy of the matching rate in the HIS based on the review status The deep learning recognition unit is retrained with the deep learning algorithm. In other words, the AI server is given positive or negative feedback based on the review status to retrain the deep learning recognition unit. Furthermore, when the deep learning recognition unit makes a misjudgment or when the appearance of a drug partially changes, the deep learning recognition unit is given synchronous feedback to relearn and revise the feature file. According to the method for using deep learning object detection technology to assist in drug identification as described in claim 1, the medicine bag has a front side containing patient and drug information and a back side made of transparent plastic film, into which the drug can be placed and which can be clearly imaged by the camera. According to the method for using deep learning object detection technology to assist in drug identification as described in claim 1, the camera is a high-speed camera with a frame rate of 120 FPS or more. According to the method for using deep learning object detection technology to assist in drug identification as described in Item 1 of the patent application, step 1 is to use a barcode reader connected to the HIS to read the barcode on the prescription. According to the method for using deep learning object detection technology to assist in drug identification as described in Item 1 of the patent application, in step 2, the image data of all drugs is presented as images or continuous images based on the number of drugs. According to the method for using deep learning object detection technology to assist in drug identification as described in Item 1 of the patent application, step three involves comparing an image or an entire continuous image with all the drugs on the prescription to determine the probability of each drug on the prescription appearing in the image or continuous image. According to the method for using deep learning object detection technology to assist in drug identification as described in Item 1 of the patent application, in step 4, when the matching rate after comparison is low, the HIS will prompt the drug item with the low matching rate. According to claim 1, the method for assisting drug identification using deep learning object detection technology comprises at least one of an algorithm based on an Auto-Encoder architecture, an algorithm based on a Multi-Stage architecture, an algorithm based on a U-Net architecture, an algorithm based on a GAN architecture, and a convolutional neural network (U-Net XY-Deblur) algorithm specifically for deblurring. According to the method for using deep learning object detection technology to assist in drug identification described in claim 8, the convolutional neural network U-Net XY-Deblur algorithm uses an encoder and two decoders as its architecture and improves deblurring performance through rotation and shared parameters. The formula is as follows: According to the method of using deep learning object detection technology to assist in drug identification as described in item 1 of the patent application, the artificial intelligence server uses an object detection algorithm to detect the image region (Region of Interest, ROI) where the drug is located in the image data of the drug to be identified, and uses the YOLACT (You Only Look At CoefficienTs) algorithm model to perform instance segmentation on the ROI area, calculate the category confidence of each anchor, the position of the bounding box, the coefficient of the mask, and the mask of each target object in the image, so as to separate the foreground drug area from the background image, and finally use the Siamese Convolutional Neural Network (Siamese Convolutional Neural Network) to perform segmentation on the ROI area. The two CNNs share the same parameters and utilize metric learning to enhance feature resolution. The inputs are the image data of the drug to be identified and the image feature data of a prescribed drug in the drug feature database. A connection function is used to connect the two CNNs, linking the image data of the drug to be identified with the image feature data of the prescribed drug. Finally, a cost function is used to calculate the similarity between the drug to be identified and the prescribed drug. According to claim 10, the method for using deep learning object detection technology to assist in drug identification, wherein the object detection algorithm used in the YOLACT algorithm model includes at least one of Region-based CNN (R-CNN), Fast R-CNN (Fast R-CNN), Faster R-CNN (Faster R-CNN), Mask R-CNN (Mask R-CNN), YOLO (You Only Look Once), YOLOv2, YOLOv3, YOLOv7, Single Shot MultiBox Detector (SSD), and RetinaNet. A device that uses deep learning object detection technology to assist in drug identification includes: a camera that captures images of pharmacists dispensing and reviewing drugs at a frame rate of 120 FPS or higher, primarily used to collect image data of all drugs in at least one medicine bag printed based on a prescription; and an artificial intelligence server connected to the camera and a drug feature database. The artificial intelligence server includes a deep learning recognition unit that receives and stores image data of all drugs, analyzes and compares image feature data of a predetermined prescription drug in the drug feature database with the current drug to be identified. The deep learning recognition unit includes: a drug imaging and image deblurring module, which is processed in parallel with the camera and uses a deep learning algorithm to deblur the image data of the drug to be identified; a high-reflective image restoration module, which is connected to the drug imaging and image deblurring modules and uses a polarizing filter to improve the computing speed of the drug imaging and image deblurring modules so that they can match the working speed of the pharmacist; an object detection and instance segmentation module. The pharmaceutical image segmentation module is connected to the pharmaceutical image acquisition and image deblurring module. Before performing pharmaceutical comparison, the object detection algorithm is used to detect the ROI region where the pharmaceutical is located in the image data of the pharmaceutical to be identified, and the YOLACT algorithm model is used to perform instance segmentation on the ROI region. The pharmaceutical image re-identification module is connected to the target object detection and instance segmentation module. The module receives the segmented pharmaceutical image data and Image data of the drug to be identified, where the foreground drug region is separated from the background image, is compared using a SCNN consisting of two CNNs as the recognition architecture. The two CNNs share the same parameters and utilize metric learning to enhance feature resolution. A connection function is then connected to the two CNNs, linking the image data of the drug to be identified with the image feature data of the prescribed drug. Finally, a cost function is used to calculate the similarity between the drug to be identified and the prescribed drug. According to claim 12, the device utilizes deep learning object detection technology to assist in drug identification, wherein the deep learning algorithm includes at least one of an auto-encoder architecture-based algorithm, a multi-stage architecture-based algorithm, a U-Net architecture-based algorithm, a GAN architecture-based algorithm, and a convolutional neural network (U-Net) XY-Deblur algorithm specifically for deblurring. According to claim 13, the device for assisting drug recognition using deep learning object detection technology comprises a convolutional neural network (U-Net) XY-Deblur algorithm that employs an encoder and two decoders, and further enhances its deblurring performance through rotation. According to claim 12, the device for assisting in drug identification using deep learning object detection technology comprises at least one of R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, YOLO, YOLOv2, YOLOv3, YOLOv7, SSD, and RetinaNet. According to claim 12, the device for assisting drug recognition using deep learning object detection technology utilizes the YOLACT algorithm model to perform instance segmentation in the ROI region. This algorithm utilizes two parallel sub-network branches, a first sub-network branch and a second sub-network branch, to achieve real-time instance segmentation. The first sub-network branch is the prediction head, which calculates the confidence level of each anchored category, the position of the bounding box, and the mask coefficient. The second sub-network branch is the prototype network, which generates a mask for each target object in the image, thereby separating the foreground drug region from the background image. According to claim 12 of the patent application, the device utilizes deep learning object detection technology to assist in drug identification. The artificial intelligence server receives prescription data from a HIS computer, reads the prescription data for the drugs contained in the prescription, links the prescription data to the drug feature database, extracts the image feature data of all drugs in the prescription, and uses a deep learning algorithm to perform a probability analysis on the image feature data of the predetermined prescription drug and the image data of the drug to be identified. The server then calculates the match rate for all drugs and transmits it back to the HIS computer for display.