TWM604031U - Artificial intelligence-based optical inspection system - Google Patents

Abstract

An artificial intelligence-based optical inspection system includes a transfer device, an image-capturing device, and a processing circuit. The processing circuit executes a first neural network model, a second neural network model, and the third neural network models corresponding to one to one the defects. The object image of the target object captured by the image-capturing device is fed into the first neural network model, the second neural network model and the third neural network models. When the first neural network model performs online detection on the object image, the second neural network model and the third neural network models are verified with the same object image. Therefore, it is possible to reduce the introduction time and manpower of the neural network model.

Description

Optical inspection system based on artificial intelligence

This new model relates to the detection technology of defective objects, in particular to an optical detection system based on artificial intelligence.

In automated production, human resources need to be spent on product testing to eliminate defective products. The industry is often unable to conduct 100% online product inspection through human resources. Therefore, Automated Optical Inspection (AOI) technology is introduced to replace manual visual inspection to greatly shorten the inspection processing time and thereby reduce production costs. Automatic optical inspection is a high-speed, high-precision optical image inspection system. In the product manufacturing process, automatic optical inspection can obtain the surface condition of the product, and then use computer image processing technology to detect defects such as foreign objects or abnormal patterns.

However, when traditional AOI equipment is performing defect detection, underkill (also known as Leakage, where no defect is detected, misjudges a bad sample as a good sample) or overkill (misjudges a good sample as a bad sample) may occur . For missed detection or manslaughter, traditionally rely on personnel to re-judge, but this will cause the cost of detection to not be greatly reduced. Although the detection efficiency can be improved after the introduction of artificial intelligence, the verification time for the introduction of artificial intelligence into the AOI equipment according to the current practice is still quite long.

In one embodiment, an optical inspection system based on artificial intelligence includes: a transfer device, an image capture device, and a processing circuit. The transfer device moves the plurality of test objects one by one to a detection position. The image capturing device is aligned with the detection position, and captures the object images of the object to be tested located at the detection position one by one. The processing circuit is coupled to the image capturing device, and executes the first neural network model, the second neural network model, and the complex third neural network model for one-to-one correspondence of multiple defects. Wherein, when the object image of any object to be tested is captured, the processing circuit analyzes and determines that the object image is one of a plurality of predetermined categories by the first neural network model to obtain a detection result. Here, the plurality of predetermined categories include: a flawed category representing a flawed image that has a flaw in the object image, and a flawless category that represents a flawed image that does not have a flaw in the object image.

Moreover, when capturing an object image of any object under test, the processing circuit also uses the second neural network model to analyze and determine that the object image is one of a plurality of predetermined categories to obtain a first output and feed the object image into To a plurality of third neural network models, analyze and identify whether there is a corresponding defect image in the object image with each third neural network model to obtain a second output, and generate a verification based on the first output and each second output result.

In one embodiment, an optical inspection system based on artificial intelligence includes: a transfer device, an image capture device, and a processing circuit. The transfer device moves the plurality of test objects one by one to a detection position. The image capturing device is aligned with the detection position, and captures the object images of the object to be tested located at the detection position one by one. The processing circuit is coupled to the image capturing device and executes the first neural network model. Wherein, when the object image of any object to be tested is captured, the processing circuit analyzes and determines that the object image is one of a plurality of predetermined categories by the first neural network model to obtain a detection result. Here, the plurality of predetermined categories include: a defective category representing a defective image of the object image, a flawless category representing a defective image of the object image without a defect, and an unknown category representing the existence of a defective image in the object image cannot be confirmed.

To sum up, in some embodiments, the optical inspection system based on artificial intelligence can detect the object to be tested online, and at the same time, it can check the new version of the new version by re-judgment of multiple third neural network models corresponding to multiple defects. The neural network model (that is, the second neural network model) is validated, thereby reducing the time and manpower for importing the neural network model. In some embodiments, the artificial intelligence-based optical detection system adds an unknown type of judgment that represents the inability to confirm whether the object image has a flawed image, as a training material for a new version of the neural network model in the future.

In order to clearly express each element, the insulating layer is sometimes presented in a transparent or omitted manner in the drawings, but this is not a limitation of the present invention. In addition, the terms "first", "second", "third" and "fourth" mentioned below are used to distinguish the referred elements, not to sort or limit the differences of the referred elements It is not used to limit the scope of the present invention.

1, an optical detection system 10 based on artificial intelligence (hereinafter referred to as the optical detection system 10) includes a transfer device 110, an image capture device 120 and a processing circuit 130. In some embodiments, the optical inspection system 10 can be implemented by an automatic optical inspection machine, as shown in FIGS. 2, 4, and 5.

Here, the optical inspection system 10 can be used to detect whether the manufactured test object has defects. In some embodiments, the object to be measured may be a semiconductor device, a mechanical component, or a particulate matter. In the first example, referring to FIGS. 2 and 3, the optical inspection system 10 based on artificial intelligence can be used to detect whether the packaged semiconductor device 210 has defects on the package. Among them, the defects on the package may be, for example, void welding, bubble, underfill overflow, backside chipping, or any combination thereof. For example, the semiconductor element 210 can be a packaged chip formed by chip on glass (COG) technology, a packaged chip formed by chip on film (COF) technology, or a plastic package. Packaged chips formed by Chip On Plastic (COP). In the second exemplary embodiment, referring to FIG. 4, the optical inspection system 10 based on artificial intelligence can be used to detect whether the packaged mechanical component 310 has defects. Among them, the defects of the mechanical component 310 can be, for example, foreign debris, dent, stain, deformation, contamination, trimming, overflow, cracks ( chipping) or any combination thereof. In some embodiments, the mechanism 310 may be, for example, a connector, a wire harness, a mobile phone case, etc., but is not limited thereto. In the third example, referring to FIG. 5, the optical inspection system 10 based on artificial intelligence can be used to detect whether the encapsulated particles 410 have defects. Among them, the particulate matter 410 may be coffee beans or medicines such as capsules and tablets. When the particulate matter 410 can be coffee beans, the defects of the particulate matter 410 can be, for example, Quaker beans, insect damage beans, sshell beans, broken/chipped/cut beans, meteorite beans (ie Baked defective beans), parchment beans (also known as shelled beans), dark beans, or any combination thereof. When the particulate matter 410 can be a tablet, the defects of the particulate matter 410 can be, for example, corner spots, black lines, spots at the lettering, spots on the three-layer ingot, missing characters, edge breakage, coating edge defect, coating top defect, top surface defect, Edge breakage, breakage, side spots, top cracks, side cracks, marking spots, non-marking spots, split lines spots, layer top defects, abnormal shapes, surface bulges, lettering defects, or any combination thereof. When the particulate matter 410 may be a capsule, the defect of the particulate matter 410 may be, for example, a surface spot, an empty capsule, a missing character, a surface cavity, a joint flaw, a dent, a capsule cap cavity, a corner spot, or any combination thereof.

The processing circuit 130 is coupled to the transfer device 110 and the image capture device 120, and is used to control the operations of the transfer device 110 and the image capture device 120.

In some embodiments, the transfer device 110 is disposed on the frame 101 of the optical detection system 10 corresponding to the detection position DA, so that the transfer device 110 can move the test objects to the detection position DA one by one. In the first exemplary example, referring to FIGS. 1 to 3, taking the semiconductor device 210 as the test object and each semiconductor device 210 as a packaged chip as an example, the semiconductor devices 210 to be tested are sequentially placed on a tape 220 (That is, forming the tape-reel semiconductor assembly 20), and the transfer device 110 rolls the tape 220 to move the plurality of semiconductor elements 210 to be inspected one by one to the inspection position DA. For example, the transfer device 110 may include two conveying wheels 112 and 114. Each semiconductor element 210 is a packaged chip, and a plurality of semiconductor elements 210 to be inspected are sequentially placed on a tape 220, as shown in FIG. 3. Here, referring to FIGS. 2 and 3, the web 220 has two opposite surfaces 220a and 220b. The plurality of semiconductor elements 210 to be inspected are located on the surface 220 b of the tape 220. The two ends of the coil 220 are respectively wound on the two conveying wheels 112 and 114. During detection, the processing circuit 130 controls the two conveying wheels 112 and 114 to rotate in the same direction (for example, clockwise or counterclockwise) to roll the reel 220 so that the semiconductor components 210 on the reel 220 pass through the image capture device one by one 120 facing detection position DA. Among them, the tape 220 may be a tape-and-reel flexible circuit board (FPC). It should be understood that, in FIG. 2, the semiconductor element 210 is located on the other side surface of the tape 220 opposite to the surface 220a, so part of the semiconductor element 210 is drawn with a dotted line for illustration. In the second exemplary example, referring to FIGS. 1 and 4, taking the object to be measured as the mechanical component 310 as an example, the transfer device 110 can be a bearing jig 116, and the gripping robot 103, the feeding rail 104 and the vibration plate 105 Place the mechanical component 310 to be tested on the supporting fixture 116. By rotating the bearing jig 116, the mechanism 310 on the bearing jig 116 is moved to the detection position DA where each image capturing device 120 is aligned. In the third exemplary embodiment, referring to FIGS. 1 and 5, taking the object to be measured as the particulate matter 410 as an example, the transfer device 110 may include a roller 117 and a funnel 118. Through the rotating drum 117, the particles 410 in the drum 117 are brought out of the drum 117 from the opening 117a on the drum 117 one by one and fall into the hopper 118. The particulate matter 410 passes along the funnel 118 through the detection position DA where the image capturing device 120 is aligned for the image capturing device 120 to capture the object image IM of the particulate matter 410. In other words, the lower half of the funnel 118 is a transparent pipe, and the inside of the transparent pipe is the detection position DA. When the particulate matter 410 passes through the transparent pipe, the image capturing device 120 can capture the object image IM of the particulate matter 410 passing through the transparent pipe.

Referring to FIGS. 1 to 5, the image capturing device 120 is aligned with the detection position DA and set on the frame 101 of the optical detection system 10. The sensing surface of the image capturing device 120 faces the detection position DA.

1 to FIG. 7, when the detection program is executed, the processing circuit 130 drives the transfer device 110 to cause the transfer device 110 to drive the multiple objects to be tested to the detection position DA one by one (step S11). In addition, the processing circuit 130 drives the image capturing device 120, so that the image capturing device 120 captures the object images IM of the test object located at the detection position DA one by one (step S11). In other words, when any one of the plurality of test objects to be detected moves to the detection position DA, the image capturing device 120 captures the object image IM of the test object located in the detection position DA. In some embodiments, the optical inspection system 10 is provided with a single image capture device 120 or multiple image capture devices 120 according to the inspection requirements. In one example, the optical inspection system 10 may have a plurality of image capturing devices 120, and these image capturing devices 120 capture the object image IM of the test object at the same angle relative to the test object. In another example, the optical inspection system 10 may have a plurality of image capturing devices 120, and these image capturing devices 120 capture the object image IM of the test object at different angles relative to the test object, as shown in FIG. 4 shown.

The processing circuit 130 downloads, installs and executes a first neural network model 132a, a second neural network model 132b, and a plurality of third neural network models 134, as shown in FIG. 6. Here, the first neural network model 132a, the second neural network model 132b, and the third neural network model 134 of the processing circuit 130 may be neural networks that have completed training (hereinafter referred to as the first-stage training) model. In addition, referring to FIG. 6, the processing circuit 130 further has an input unit 131, an output unit 135 and a verification unit 137. The output of the input unit 131 is coupled to the input of the first neural network model 132a and the input of the second neural network model 132b. The output of the first neural network model 132a is coupled to the output unit 135. The output of the second neural network model 132b is coupled to the input of the plurality of third neural network models 134. The input of the verification unit 137 is coupled to the output of the second neural network model 132 b and the output of the third neural network model 134. Here, the second neural network model 132b is used as an update program of the first neural network model 132a.

When the object image IM of any object under test is captured by the image capturing device 120, the object image IM captured by the image capturing device 120 is provided to the processing circuit 130. At this time, the input unit 131 of the processing circuit 130 receives the object image IM from the image capturing device 120 and feeds the object image IM to the first neural network model 132a, the second neural network model 132b, and all third neural networks Road model 134 (step S12).

After the object image IM is fed (step S12), the first neural network model 132a (that is, in the prediction mode) will analyze the object image IM of the object under test to determine whether the object under test has defects and determine the object accordingly The image IM is one of a plurality of predetermined categories to obtain a detection result (step S13). Among them, the plurality of predetermined categories include a defect-free category representing a defect image without any kind of defects in the object image IM and a defect category representing a defect image with at least one defect in the object image IM. For example, suppose that the predetermined categories are defective and non-defective. When the first neural network model 132a confirms that the object under test does not have a defect by analyzing the object image IM, the first neural network model 132a obtains the detection result that the object image IM of the object under test belongs to the flawless category, which means that The test object is flawless. When the first neural network model 132a confirms that the object under test has a defect by analyzing the object image IM, the first neural network model 132a obtains that the object image IM of the object under test belongs to the detection result of the defect category, which means the object to be tested The thing is flawed. In some embodiments, the plurality of predetermined categories may further include an unknown category representing that it is impossible to confirm whether the object image IM has a defective image. For example, suppose that the predetermined categories are defective, non-defective, and unknown. When the first neural network model 132a confirms that the object under test does not have a defect by analyzing the object image IM, the first neural network model 132a obtains the detection result that the object image IM of the object under test belongs to the flawless category, which means that The test object is flawless. When the first neural network model 132a confirms that the object under test has a defect by analyzing the object image IM, the first neural network model 132a obtains that the object image IM of the object under test belongs to the detection result of the defect category, which means the object to be tested The thing is flawed. When the first neural network model 132a cannot confirm whether the object under test has a defect by analyzing the object image IM, the first neural network model 132a obtains the detection result that the object image IM of the object under test belongs to an unknown category.

After the object image IM is fed (step S12), the second neural network model 132b (that is, in the prediction mode) will also analyze the object image IM of the object to be tested to confirm whether the object to be tested is flawed and determine it accordingly The object image IM is one of a plurality of predetermined categories to obtain a first output (step S14). In some embodiments, since the second neural network model 132b is used as an update program of the first neural network model 132a, the plurality of predetermined categories classified by the second neural network model 132b are the same as the first neural network model Multiple predetermined categories classified by 132a. That is, the plurality of predetermined categories classified by the second neural network model 132b also include the flawless category representing that the object image IM does not have any kind of flaws, and the flawed category representing the object image IM has at least one kind of flaws. category. For example, suppose that the predetermined categories are defective and non-defective. When the second neural network model 132b confirms that the object under test does not have a defect by analyzing the object image IM, the second neural network model 132b obtains the first output that the object image IM representing the object under test belongs to the category of flawless. When the second neural network model 132b confirms that the object under test has a defect by analyzing the object image IM, the second neural network model 132b obtains the first output representing the object image IM of the object under test belonging to the defect category. In some embodiments, the plurality of predetermined categories classified by the second neural network model 132b may further include an unknown category representing whether the object image IM has a defect image or not. For example, suppose that the predetermined categories are defective, non-defective, and unknown. When the first neural network model 132a confirms that the object under test does not have a defect by analyzing the object image IM, the first neural network model 132a obtains the detection result that the object image IM of the object under test belongs to the flawless category, which means that The test object is flawless. When the first neural network model 132a confirms that the object under test has a defect by analyzing the object image IM, the first neural network model 132a obtains that the object image IM of the object under test belongs to the detection result of the defect category, which means the object to be tested The thing is flawed. When the first neural network model 132a cannot confirm whether the object under test has a defect by analyzing the object image IM, the first neural network model 132a obtains the detection result that the object image IM of the object under test belongs to an unknown category.

Here, the multiple third neural network models 134 correspond to multiple defects one-to-one. After the object image IM is fed (step S12), each third neural network model 134 (that is, in the prediction mode) analyzes the object image IM of the object to be tested to determine whether the object to be tested has a corresponding defect, that is, identify Whether the object image IM of the test object has the corresponding defect image to obtain a second output (step S15). Among them, the defect image refers to an image block showing a corresponding defect in the object image IM. In some embodiments, when the third neural network model 134 determines that the object to be tested has a corresponding defect, the third neural network model 134 obtains a second output indicating that the object image IM has a corresponding defect image. When the third neural network model 134 determines that the object under test does not have a corresponding defect, the third neural network model 134 obtains a second output indicating that the object image IM does not have a corresponding defect image. In the first example, it is assumed that the object to be tested is a semiconductor device 210, each semiconductor device 210 is a packaged chip, and the types of defects of the packaged chip are solder joints, bubbles, glue creep, and burrs. At this time, there are four third neural network models 134, namely the third neural network models 134a-134d. Among them, the third neural network model 134a corresponds to the empty welding, that is, it is used to identify whether the object image IM of the object to be measured has an empty welding defect image. The third neural network model 134b corresponds to the bubble, that is, it is used to identify whether the object image IM of the test object has a defect image of the bubble. The third neural network model 134c corresponds to the creeping glue, that is, it is used to identify whether the object image IM of the test object has the creeping glue defect image. The third neural network model 134d corresponds to the burr, that is, it is used to identify whether the object image IM of the test object has a burr defect image. In the second example, it is assumed that the object to be tested is the mechanical component 310, and the types of defects of the mechanical component 310 include foreign matter, indentation, stain, deformation, pollution, trimming, overflow, and crack. At this time, there are eight third neural network models 134, and one-to-one corresponds to foreign matter, indentation, stain, deformation, pollution, trimming, overflow, and crack.

Then, for the object image IM of each object under test, the verification unit 137 will use the first output of the object image IM obtained by the second neural network model 132b and the object image obtained by all the third neural network models 134 The second output of IM generates the verification result of the object image IM (step S16). In some embodiments, the verification result includes pass (ok), missed (underkill), unqualified (NG) and overkill (overkill). For each object image IM, when the second neural network model 132b obtains the first output representing that the object image IM belongs to the flawless category, and each third neural network model 134 obtains the first output representing the object image IM without corresponding defects During the second output of the image, the verification unit 137 will generate a verification result that is qualified. When the second neural network model 132b obtains the first output representing the object image IM belonging to the flawless category but any third neural network model 134 obtains the second output representing the defect image corresponding to the object image IM, the verification unit 137 will produce verification results representing missed inspections. When the second neural network model 132b obtains the first output representing the object image IM belonging to the defect category and any third neural network model 134 obtains the second output representing the defect image corresponding to the object image IM, the verification unit 137 will produce a verification result representing failure. When the second neural network model 132b obtains the first output representing the object image IM belonging to the defect category, but each third neural network model 134 obtains the second output representing the object image IM with no corresponding defect image, The verification unit 137 generates a verification result representing manslaughter.

In this way, the optical inspection system 10 can verify the new version of the neural network model (ie, the second neural network model 132b) while detecting the object to be tested online.

It should be noted that although Fig. 7 and its corresponding descriptions record the defect determination procedure (i.e. step S13) and the verification procedure (i.e. steps S14, S15, S16) in sequence, this sequence is not a limitation of the present model. Those who are familiar with related skills It should be understood that under reasonable circumstances, the execution sequence of some steps can be carried out simultaneously or sequentially.

In some embodiments, the output of the output unit 135 can be coupled to the storage device 150, and the storage device 150 has a plurality of data cassettes 151-153. These data boxes 151-153 respectively represent different test results. The output unit 135 also stores the corresponding object image IM in the corresponding data box (one of 151-153) according to the generated detection result. In an example, the detection result may be that the object image IM belongs to the defective category and the object image IM belongs to the non-defective category. The storage device 150 has a data box 151 representing a defect and a data box 152 representing no defect. Therefore, the output unit 135 stores the object image IM belonging to the defective category in the data box 151 representing the defect, and stores the object image IM belonging to the non-defective category in the data box 152 representing no defect. In another example, the detection result may be that the object image IM belongs to a defective category, the object image IM belongs to a non-defective category, and the object image IM belongs to an unknown category. The storage device 150 has a data box 151 representing a defect, a data box 152 representing no defect, and a data box 153 representing an unknown. Therefore, the output unit 135 stores the object image IM belonging to the defective category in the data box 151 representing the defect, and stores the object image IM belonging to the non-defective category in the data box 152 representing no defect, and will be unknown. The object image IM of is stored in the unknown data box 152.

In some embodiments, the output of the verification unit 137 can also be coupled to the storage device 150, and the storage device 150 further has a plurality of data cassettes 154-158. These data boxes 154-158 respectively represent a plurality of different verification results. The verification unit 137 also stores the corresponding object image IM in the corresponding data box (one of 154-158) according to the generated verification result. In an exemplary case, the verification result can be qualified, missed, unqualified, and manslaughter. The storage device 150 has a data box 154 representing qualified, a data box 155 representing missed detection, a data box 156 representing unqualified, and a data box 157 representing manslaughter. Therefore, the verification unit 137 stores the object image IM whose verification result is qualified in the data box 154 representing qualified, the object image IM whose verification result is missed detection is stored in the data box 154 representing missed detection, and the verification result is The unqualified object image IM is stored in the data box 154 representing unqualified, and the object image IM whose verification result is manslaughter is stored in the data box 154 representing manslaughter. In another example, the verification result can be qualified, missed, unqualified, manslaughter, and unknown. The storage device 150 has a data box 154 representing qualified, a data box 155 representing missed detection, a data box 156 representing unqualified, a data box 157 representing manslaughter, and a data box 158 representing unknown. Therefore, the verification unit 137 stores the object image IM whose verification result is qualified in the data box 154 representing qualified, the object image IM whose verification result is missed detection is stored in the data box 154 representing missed detection, and the verification result is The unqualified object image IM is stored in the data box 154 representing unqualified, the object image IM whose verification result is manslaughter is stored in the data box 154 representing manslaughter, and the object image IM whose verification result is unknown is stored in the representative unknown In the data box 154.

In some embodiments, the processing circuit 130 may further include a marking unit 138. The marking unit 138 is coupled to the storage device 150 and the display device 160.

In an exemplary example, the marking unit 138 executes a marking procedure to read any object image IM stored in the data cassettes 154-158, and displays the read object image IM on the display device 160, and according to the user's The operation marks the displayed object image IM to mark the defect image in the object image IM. After marking, the marking unit 138 will store the object image IM back into the read-out data box (one of 154-158).

In another example, the marking unit 138 executes a review process to read any object image IM stored in the data boxes 151-153 and/or data boxes 154-158 and displays the read object image IM The device 160 displays it for the user to review and confirm whether the verification result is correct. For example, the user checks the object image IM in the data box 154 that represents the qualification to confirm whether the object image IM is indeed flawless. The user checks the object image IM in the data box 155 that represents the missed inspection to confirm whether the object image IM has defects. The user reviews the object image IM in the data box 156 representing the failure to confirm whether the object image IM has defects. The user reviews the object image IM in the data box 157 representing manslaughter to confirm whether the object image IM is indeed flawless. In some embodiments, when the marking unit 138 executes the review process, it can also note the review result of the displayed object image IM according to the user's operation. After reviewing, the marking unit 138 will store the annotated object image IM back into the read-out data box (one of 154-158). Here, the object image IM in the data box 154 representing the qualified and the data box 156 representing the unqualified only needs to be manually inspected, and the object image IM in the data box 155 representing the missed detection and the data box 157 representing manslaughter are then manually checked. Full inspection, in order to reduce the time required for manual full inspection during verification. For example, when the marking unit 138 executes the review process, it randomly reads from all the object images IM representing the qualified data cassette 154 and displays part of the object images IM one by one, and all object images from the non-compliant data cassette 156. The IM randomly reads and displays part of the object images IM one by one, and reads and displays all the object images IM in the missing data box 155 and the object images IM in the data box 157 representing manslaughter one by one.

In another exemplary embodiment, the marking unit 138 has the execution capability of the aforementioned marking procedure and the aforementioned review procedure at the same time.

In some embodiments, the optical detection system 10 further includes a warning device 140. The warning device 140 is coupled to the processing circuit 130 and its operation is controlled by the processing circuit 130. In other words, the output unit 135 of the processing circuit 130 is also coupled to the warning device 140. When it is determined that the detection result of the object to be measured is that the object image IM belongs to the defective category, the output unit 135 will output an enabling signal to cause the warning device 140 to issue an alarm. In some embodiments, when the detection result of the object under test is that the object image IM belongs to the defect category, it means that the output unit 135 will also suspend the transfer device 110 and the image capture device 120 when a defect is detected in the object under test. The operation of the inspector to confirm whether the object under test corresponding to the object image IM is indeed defective. After the confirmation is completed, the processing circuit 130 restarts the transfer device 110 and the image capture device 120.

In some embodiments, the optical inspection system 10 further includes a classification device 180 and a collection box 190 for one-to-one correspondence of multiple detection results. The classification device 180 is coupled to the processing circuit 130 and its operation is controlled by the processing circuit 130. The classification device 180 is also provided corresponding to the collection box 190. In other words, the output unit 135 of the processing circuit 130 is also coupled to the classification device 180, and outputs a control signal to the classification device 180 according to the detection result. When the detection result of the object to be detected determines that the object image IM belongs to the defective category, the classification device 180 responds to the control signal to sort the object to be detected to the collection box 190 corresponding to the detection result of the object image IM belonging to the defective category. When the detection result of the object to be tested determines that the object image IM belongs to the non-defective category, the classification device 180 responds to the control signal to sort the object to be tested to the collection box 190 corresponding to the detection result of the object image IM belonging to the non-defective category. In the third exemplary embodiment described above, the transfer device 110 may further include a conveyor belt 119. The lower outlet of the funnel 118 is coupled to one end of the conveyor belt 119, and the other end of the conveyor belt 119 is aligned with the sorting device 180. The particulate matter 410 falling into the hopper 118 enters the conveyor belt 119 through the lower outlet of the hopper 118 and is conveyed to the sorting device 180 through the conveyor belt 119. The classification device 180 classifies the particulate matter 410 into the corresponding collection box 190 according to the detection result. For example, when the detection result is that the particulate matter 410 belongs to the defect category, the classification device 180 classifies the corresponding particulate matter 410 to the defect collection box 190a. When the detection result is that the particulate matter 410 belongs to the non-defective category, the classification device 180 classifies the corresponding particulate matter 410 to the qualified collection box 190b. For example, when the particulate matter 410 is coffee beans, the classification device 180 sorts the coffee beans whose detection result is that the particulate matter 410 belongs to the defective category to the defective bean collection box (such as the defect collection box 190a), and the detection result is that the particulate matter 410 belongs to The coffee beans of the non-defective category are sorted into good bean collection boxes (such as qualified collection box 190b).

In an embodiment of step S13, the processing circuit 130 uses the first neural network model 132a to calculate the possibility of the object image IM in the defective category and the possibility of the non-defective category, and according to the calculated possibility of the defective category Determine the detection result of the test object represented by the object image IM with the possibility of the flawless category. In an exemplary example, the first neural network model 132a uses the predetermined category with the highest probability as the detection result. For example, when the possibility of the flawless category is the highest value, the first neural network model 132a determines that the object image IM belongs to the flawless category (that is, the detection result), which means that there is no flawed image in the object image IM. In other words, the object to be tested to which the object image IM belongs is a qualified product without defects. When the possibility of the defect category is the highest value, the first neural network model 132a determines that the object image IM belongs to the defect category (ie the detection result), which means that the object image IM has any kind of defect image, that is to say , The object under test to which the object image IM belongs has a corresponding defect. In another example, the first neural network model 132a compares the possibility of the defective category with the possibility of the non-defective category by a threshold (hereinafter referred to as the first threshold), and obtains the detection result according to the comparison result. For example, when the possibility of the flawless category is greater than the first threshold, the first neural network model 132a determines that the object image IM belongs to the flawless category (ie, the detection result). When the possibility of the defect category is greater than the first threshold, the first neural network model 132a determines that the object image IM belongs to the defect category (ie, the detection result). In another example, the first neural network model 132a uses two thresholds (hereinafter referred to as the first threshold and the second threshold) to compare the possibility of a defective category with the possibility of a non-defective category, and obtain the result according to the comparison. Test results. Wherein, the first threshold is greater than the second threshold. For example, when the possibility of the flawless category is greater than the first threshold, the first neural network model 132a determines that the object image IM belongs to the flawless category (ie, the detection result). When the possibility of the defect category is greater than the first threshold, the first neural network model 132a determines that the object image IM belongs to the defect category (ie, the detection result). When the possibility of the flawless category and the possibility of the flawed category are both less than the second threshold, the first neural network model 132a determines that the object image IM belongs to the unknown category (ie, the detection result). In some embodiments, the aforementioned threshold may be a predetermined value set according to the required accuracy. In an example, the aforementioned probability can be expressed as a percentage, that is, between 0% and 100%. Therefore, the aforementioned threshold can be any number greater than 0% and less than or equal to 100%.

In an embodiment of step S14, the processing circuit 130 uses the second neural network model 132b to calculate the possibility of the object image IM in the defective category and the possibility of the non-defective category, and according to the calculated possibility of the defective category Determine the detection result of the test object represented by the object image IM with the possibility of the flawless category. In an exemplary example, the second neural network model 132b uses the predetermined category with the highest probability as the detection result. For example, when the possibility of the flawless category is the highest value, the second neural network model 132b determines that the object image IM belongs to the flawless category (ie, the first output). When the possibility of the defect category is the highest value, the second neural network model 132b determines that the object image IM belongs to the defect category (ie, the first output). In another example, the second neural network model 132b compares the possibility of the defective category with the possibility of the non-defective category with a threshold (hereinafter referred to as the first threshold) and obtains the detection result according to the comparison result. For example, when the possibility of the flawless category is greater than the first threshold, the second neural network model 132b determines that the object image IM belongs to the flawless category (ie, the first output). When the possibility of the defect category is greater than the first threshold, the second neural network model 132b determines that the object image IM belongs to the defect category (ie, the first output). In another example, the second neural network model 132b uses two thresholds (hereinafter referred to as the first threshold and the second threshold) to compare the possibility of a defective category with the possibility of a non-defective category and obtain the result according to the comparison Test results. Wherein, the first threshold is greater than the second threshold. For example, when the possibility of the flawless category is greater than the first threshold, the second neural network model 132b determines that the object image IM belongs to the flawless category (ie, the first output). When the possibility of the defect category is greater than the first threshold, the second neural network model 132b determines that the object image IM belongs to the defect category (ie, the first output). When the possibility of the flawless category and the possibility of the flawed category are both less than the second threshold, the second neural network model 132b determines that the object image IM belongs to the unknown category (ie, the first output). In some embodiments, the aforementioned threshold may be a predetermined value set according to the required accuracy. In an example, the aforementioned probability can be expressed as a percentage, that is, between 0% and 100%. Therefore, the aforementioned threshold can be any number greater than 0% and less than or equal to 100%.

In some embodiments, each neural network model (ie, the first neural network model 132a, the second neural network model 132b, or any third neural network model 134) can be composed of an input layer, an output layer, and an intermediate layer ( Hidden layer) composition. The input layer, output layer, and middle layer all include one or more neurons (units). Among them, the intermediate layer may be one layer or two or more layers. In some embodiments, each neural network model may be a deep neural network model, that is, it includes more than two intermediate layers. Input data (such as object image IM) is input to each neuron of the input layer, each neuron of the middle layer is input to the output signal of the neuron of the previous or next layer, and each neuron of the output layer is input to the previous layer (ie The output signal of the neuron in the last middle layer. Note that each neuron can be connected to all neurons in the previous layer and the next layer (full connection), or connected to some neurons. In some embodiments, each neural network model can be implemented by a deep learning algorithm (for example, a convolutional neural network (CNN) algorithm, a recurrent neural network (RNN) algorithm, or a combination thereof).

In some embodiments, the second neural network model 132b can be switched to the training mode. The input unit 131 can feed the marked plural object images IM stored in the data boxes 154-158 into the second neural network model 132b in the training mode, so that the second neural network model 132b is trained with the marked object images IM (That is the second stage of training).

In some embodiments, after the verification process of the second neural network model 132b is completed, the second neural network model 132b and the plurality of third neural network models 134 can be removed, so that the first neural network model 132a is alone Perform an online defect determination procedure, as shown in Figure 9. In other words, when the optical inspection system 10 is online, the processing circuit 130 has the first neural network model 132a, but does not have the second neural network model 132b and the third neural network model 134, and executes the first neural network model. The model 132a performs the inspection of the object to be tested, that is, the defect determination procedure.

In some embodiments, the processing circuit 130 may further include an update unit 133. After completing the verification procedure of the second neural network model 132b, the update unit 133 can update the first neural network model 132a with the second neural network model 132b. In other words, the optical inspection system 10 uses the second neural network model 132b and the complex third neural network model 134 for one-to-one correspondence of multiple types of defects instead of manual verification. After the second neural network model 132b is verified by a complex third neural network model 134 corresponding to a plurality of types of defects, and the performance is confirmed manually to a satisfactory level, the verified second neural network model 132b is It can be officially launched and operated as a new version of the first neural network model 132a (that is, it is updated to replace the original first neural network model 132a). In some embodiments, after the update, the processing circuit 130 no longer retains the second neural network model 132b.

In some embodiments, the processing circuit 130 may further include a transmission unit 139. The transmission unit 139 is coupled to a networking module 170. The networking module 170 can communicate with the server 40 via the network 30. Here, after the server 40 completes the first stage training of the new version of the second neural network model 132b, the transmission unit 139 can download the second neural network that has completed the first stage training from the server 40 through the networking module 170 Road model 132b.

In some embodiments, after the server 40 completes the first stage training of the new version of the second neural network model 132b and the plural third neural network models 134, the transmission unit 139 may also be used to obtain the network module 170 from the server. The device 40 downloads the second neural network model 132b and the third neural network model 134 that have completed the first stage of training.

In some embodiments, referring to FIGS. 1 to 8, the transmission unit 139 is further coupled to the storage device 150. Here, the transmission unit 139 can also upload the image IM of each object after determining the detection result to the server 40 through the network module 170. For example, the transmission unit 139 uploads the entire data cassettes 151-153 (including the object images IM stored therein) to the server 40. In some embodiments, after the server 40 receives the object image IM after determining the detection result, it can calculate the missed detection rate and the manslaughter rate according to the object image IM after the determined detection result, and determine any one of the missed detection rate and the manslaughter rate. When it exceeds the corresponding threshold, a notification message is generated to indicate that the first neural network model 132a needs to be updated. Here, the server 40 may return the notification message to the optical inspection system 10, so as to notify the user to update the first neural network model 132a of the optical inspection system 10 (for example, use the second neural network model 132b Update the first neural network model 132a, or train the fourth neural network model with the object image IM whose detection result is an unknown category and load it into the processing circuit 130 after training to update the first neural network model 132a, etc.). For example, during the continuous operation of the optical inspection system 10, the server 40 will continue to analyze the miss-detection rate and the false-kill rate of the optical inspection system 10, and store the object image IM whose detection result is an unknown category. When the missed detection rate and the manslaughter rate are abnormal (such as exceeding their corresponding thresholds), the server 40 will alert the user to manually mark the object image IM of the unknown category to facilitate subsequent marking IM retraining of object images of unknown categories generates a new neural network model. In some embodiments, the threshold corresponding to the missed detection rate and the threshold corresponding to the manslaughter rate may be the same or different. In some embodiments, the threshold corresponding to the missed detection rate and the threshold corresponding to the manslaughter rate may be preset values greater than 0 based on actual requirements. In an exemplary embodiment, with respect to the optical inspection system 10, the server 40 may be a remote cloud server. In another example, with respect to the optical inspection system 10, the server 40 may be a local central control machine.

In some embodiments, the server 40 may train a fourth neural network model according to the object image IM of an unknown category after the server 40 receives the object image IM after the detection result is determined. Here, the object image IM belonging to the unknown category used for training may be the object image IM marked by the marking program. Moreover, the marking operation of these object images IM can be implemented by the aforementioned marking unit 138 executing the aforementioned marking program, or by the server 40 executing the aforementioned marking program. In an example, the fourth neural network model may be the same as the first neural network model 132a. In other words, the fourth neural network model and the first neural network model 132a can be the same program. For example, the fourth neural network model is a copy of the first neural network model 132a, or the first neural network model 132a is a copy of the fourth neural network model. In this way, the retraining timeliness of the first neural network model 132a can be improved.

In some embodiments, referring to FIGS. 1 to 8, the transmission unit 139 can also upload the image IM of each object after determining the verification result to the server 40 through the networking module 170. For example, the transmission unit 139 uploads the entire data cassettes 154-158 (including the object images IM stored therein) to the server 40.

In some embodiments, the server 40 may train a fourth neural network model based on the object image IM of the determined verification result after receiving the object image IM of the determined verification result. Here, the object image IM used for training may be the object image IM marked by the marking program. Moreover, the marking operation of these object images IM can be implemented by the aforementioned marking unit 138 executing the aforementioned marking program, or by the server 40 executing the aforementioned marking program. In an example, the fourth neural network model may be the same as the second neural network model 132b. In other words, the fourth neural network model and the second neural network model 132b can be the same program. For example, the fourth neural network model is a copy of the second neural network model 132b, or the second neural network model 132b is a copy of the fourth neural network model. In this way, the retraining timeliness of the second neural network model 132b can be improved. In another example, the fourth neural network model may be different from the second neural network model 132b, and also different from the first neural network model 132a, that is, compared to the second neural network model 132b, it is a new version Neural network model. In this way, the training timeliness of the new version of the neural network model can be improved.

In an embodiment, the processing circuit 130 may be a processor. In another embodiment, the processing circuit 130 can be implemented by a built-in processor and a pluggable processor. In an exemplary example, when the processing circuit 130 is implemented by a built-in processor and a pluggable processor, the generation and access of all the third neural network model 134, the verification unit 137, and the data cassettes 154-158 can be The pluggable processor executes, and other components of the processing circuit 130 and their functional operations are executed or implemented by the built-in processor.

In some embodiments, the warning device 140 may be a buzzer or a warning light.

In some embodiments, the storage device 150 is also used to store related software/firmware programs, data, data, and combinations thereof. The storage device 150 may be implemented by one or more memories.

In some embodiments, the networking module 170 may include a wired/wireless Internet module for network 30 access. The wireless Internet module supports wireless Internet technologies, such as: wireless LAN technology (WLAN), wireless broadband technology (Wibro), global interoperability microwave access technology (Wimax), high-speed downlink packet access technology (HSDPA), mobile Communication technology (such as 3G, 4G, 5G), etc. The wired Internet module supports wired Internet technology, such as: digital subscriber line technology (x Digital Subscriber Line, xDSL), fiber to the home technology (FTTH), power line communication technology (PLC), etc.

In some embodiments, the processing circuit 130, the storage device 150, and the networking module 170 may be disposed in the computer host 102, as shown in FIG. 2.

To sum up, in some embodiments, the optical inspection system 10 based on artificial intelligence can detect the object to be tested online, and at the same time, it can use multiple third neural network models 134 that correspond to multiple defects one-to-one. The new version of the neural network model (that is, the second neural network model 132b) is validated, thereby reducing the time and labor for importing the neural network model. In some embodiments, the artificial intelligence-based optical inspection system 10 adds an unknown category judgment that represents whether the object image IM has a flawed image or not, as a training material for a new version of the neural network model in the future. In some embodiments, the optical detection system 10 based on artificial intelligence can also improve the accuracy of detection and reduce the rate of missed detection and manslaughter, thereby increasing the detection speed. In some embodiments, the optical inspection system 10 based on artificial intelligence can also load a new version of the neural network model from the remote server 40 to facilitate the distribution of the neural network model. In some embodiments, the optical inspection system 10 based on artificial intelligence can also upload the inspection results and/or verification results to the remote server 40 for further data analysis and/or determination by the server 40 Does the neural network model need to be updated? In some embodiments, the optical inspection system 10 based on artificial intelligence can also upload the inspection results and/or verification results to the remote server 40 for the server 40 to train the neural network model.

10: Optical inspection system 101: Frame 103: Grab robotic arm 104: feeding track 105: Vibration plate 110: Transfer device 112, 114: conveyor wheel 116: bearing fixture 117: roller 117a: opening 118: Funnel 119: Conveyor Belt 120: Image capture device 130: processing circuit 131: input unit 132a: The first neural network model 132b: The second neural network model 133: update unit 134: The third neural network model 134a~134d: The third neural network model 135: output unit 137: Verification Unit 138: marking unit 139: Transmission Unit 140: warning device 150: storage device 151~158: data box 160: display device 170: Networking Module 180: Sorting device 190: Collection Box 190a: Defect Collection Box 190b: Qualified collection box 20: Tape and Reel Semiconductor Components 210: Semiconductor components 220: Reel 220a, 220b: surface 310: mechanical parts 410: Particulate 30: Internet 40: server IM: Object image DA: Detection position S11~S16: steps

Fig. 1 is a functional block diagram of an optical detection system based on artificial intelligence in an embodiment. FIG. 2 is a schematic diagram of a first exemplary example of the optical detection system based on artificial intelligence in FIG. 1. FIG. 3 is a schematic diagram of the semiconductor device of FIG. 2. 4 is a schematic diagram of a second exemplary embodiment of the optical detection system based on artificial intelligence in FIG. 1. 5 is a schematic diagram of a third exemplary embodiment of the optical detection system based on artificial intelligence in FIG. 1. FIG. 6 is a functional block diagram of an exemplary example of the processing circuit in FIG. 1. Fig. 7 is a flowchart of an artificial intelligence-based defect detection method according to an embodiment. FIG. 8 is a partial functional block diagram of another exemplary embodiment of the processing circuit of FIG. 1. FIG. 9 is a functional block diagram of another exemplary embodiment of the processing circuit of FIG. 1.

130: processing circuit

131: input unit

132a: The first neural network model

132b: The second neural network model

133: update unit

134: The third neural network model

134a~134d: The third neural network model

135: output unit

137: Verification Unit

138: marking unit

139: Transmission Unit

150: storage device

151~158: data box

160: display device

170: Networking Module

IM: Object image

Claims (19)

An optical inspection system based on artificial intelligence, including: A transfer device to move the plurality of objects to be tested to a detection position one by one; An image capturing device aimed at the detection position to capture object images of the object to be tested located at the detection position one by one; and A processing circuit, coupled to the image capturing device, executes a first neural network model, wherein when capturing the object image of any object under test, the processing circuit uses the first neural network model to analyze And determine that the object image is one of the plurality of predetermined categories to obtain a detection result, wherein the plurality of predetermined categories include: a defect category representing a defect image of at least one of the plurality of defects in the object image and representing the object A flawless category of the flawed image that does not have any one of the plural flaws; Wherein, the processing circuit further executes a second neural network model and a complex third neural network model for one-to-one correspondence of multiple defects; and Wherein, when the object image of any one of the test objects is captured, the processing circuit further analyzes and determines that the object image is one of the plurality of predetermined categories by the second neural network model to obtain a first output , Feed the object image to the plurality of third neural network models, analyze and identify whether the object image has the defect image corresponding to the defect with each of the third neural network models to obtain a second output, and A verification result is generated according to the first output and each of the second outputs. The artificial intelligence-based optical inspection system as described in claim 1, further including: A warning device, wherein when the detection result is that the object image of the object under test belongs to the defective category, the processing circuit outputs an enabling signal, and the warning device issues an alarm according to the enabling signal. The artificial intelligence-based optical inspection system as described in claim 1, further including: A networking module is coupled to the processing circuit and connected to a server via a network communication, wherein the processing circuit further downloads the second neural network model from the server through the networking module. The optical inspection system based on artificial intelligence according to claim 3, wherein the processing circuit further downloads the plurality of third neural network models from the server through the networking module. The artificial intelligence-based optical inspection system as described in claim 1, further including: A networking module is coupled to the processing circuit and connected to a server via a network communication, wherein the processing circuit further uploads images of each object after determining the detection result to the server through the networking module. The artificial intelligence-based optical inspection system as described in claim 1, further including: A networking module is coupled to the processing circuit and connected to a server via a network communication, wherein the processing circuit further uploads images of each object after determining the verification result to the server through the networking module. The optical inspection system based on artificial intelligence according to claim 1, wherein the processing circuit further updates the first neural network model with the second neural network model. The optical inspection system based on artificial intelligence according to claim 1, wherein the processing circuit further executes a marking procedure to mark each image of the object. The artificial intelligence-based optical inspection system as described in claim 8, further including: A networking module, coupled to the processing circuit, connects to a server via a network communication, wherein the processing circuit further uploads the marked object images to the server through the networking module for the server The device trains a fourth neural network model. The optical detection system based on artificial intelligence according to claim 1, wherein the processing circuit is a processor. The optical inspection system based on artificial intelligence according to claim 1, wherein the processing circuit is implemented by a built-in processor and a pluggable processor. The artificial intelligence-based optical inspection system according to claim 1, wherein the plurality of predetermined categories further includes: an unknown category, and wherein the processing circuit uses the first neural network model to calculate the image of the object in the defect category The detection result of the object image corresponding to the object image with the possibility and the possibility of the flawless category and the possibility of the flawed category and the possibility of the flawless category are both lower than a threshold Determined as the unknown category. The optical inspection system based on artificial intelligence according to claim 1, wherein the plurality of objects to be tested are a plurality of semiconductor components, a plurality of mechanical components, or a plurality of particulate matter. An optical inspection system based on artificial intelligence, including: A transfer device to move the plurality of objects to be tested to a detection position one by one; An image capturing device aimed at the detection position to capture object images of the object to be tested located at the detection position one by one; and A processing circuit, coupled to the image capturing device, executes a first neural network model, wherein when capturing the object image of any object under test, the processing circuit uses the first neural network model to analyze And determine that the object image is one of the plurality of predetermined categories to obtain a detection result, wherein the plurality of predetermined categories include: a defect category representing a defect image of at least one of the plurality of defects in the object image, representing the object A flawless category of the flawed image in which the image does not have any one of the plural flaws and an unknown category that represents the existence of the flawed image in the object image cannot be confirmed. The artificial intelligence-based optical inspection system as described in claim 14, further including: A warning device, wherein when the detection result is that the object image of the object under test belongs to the defective category, the processing circuit outputs an enabling signal, and the warning device issues an alarm according to the enabling signal. The artificial intelligence-based optical inspection system as described in claim 14, further including: A networking module is coupled to the processing circuit and connected to a server via a network communication, wherein the processing circuit further downloads a second neural network model from the server through the networking module. The artificial intelligence-based optical inspection system as described in claim 14, further including: A networking module is coupled to the processing circuit and connected to a server via a network communication, wherein the processing circuit further uploads images of each object after determining the detection result to the server through the networking module. The optical inspection system based on artificial intelligence according to claim 14, wherein the processing circuit uses the first neural network model to calculate the possibility of the object image in the defective category and the possibility of the non-defective category and combine the The detection result of the object to be tested corresponding to the object image in which the possibility of the defective category and the possibility of the non-defective category are both lower than a threshold value is determined as the unknown category. The optical inspection system based on artificial intelligence according to claim 14, wherein the plurality of objects to be tested are a plurality of semiconductor elements, a plurality of mechanical components, or a plurality of particulate matter.

Images