CN100518487C - A way to get multiple images in a grab device - Google Patents

Abstract

Improved component placement inspection and verification is performed by a pick and place machine (301). Improvements include stereovision imaging of the intended placement location; enhanced illumination to facilitate the provision of relatively high-power illumination in the restricted space near the placement nozzle(s) (208, 210, 212); optics (380) to allow image acquisition device (300, 302) to view the placement location (360) from an angle relative to a plane of the placement location; techniques for rapidly acquiring images with commercially available CCD arrays such that acquisition of before and after images does not substantially impact system throughput; and image processing techniques to provide component inspection and verification information.

Description

A way to get multiple images in a grab device

This application is a divisional application with the application number CN 02826679.X, the application date is 2002.11.12, the applicant is Saibo Optical Company, and the title is "Gripping Device with Component Layout Inspection Function". The above-mentioned earlier application claims priority, and of said priority-

Country of earlier application Date of earlier application Earlier application number

United States 2001.11.13 60/338,233

United States 2002.2.13 60/356,801

United States 2002.4.22 60/374,964

technical field

Pick and place equipment is often used to manufacture circuit boards. Typically, blank circuit boards are provided to pick-and-place equipment, which picks circuit components from a component supply and places them on the circuit board. The components are temporarily held on the board by the solder oil or adhesive until a subsequent step in which the melted solder oil or adhesive is completely cured.

Background technique

Handling of gripping equipment can be challenging. Since the speed of the equipment corresponds to the production efficiency, the faster the handling equipment can run, the less it will cost to manufacture the circuit board. In addition, the accuracy of placement is extremely important. Many electrical components such as chip capacitors and chip resistors are relatively small and must be placed precisely in equally small layout locations. Other components, although larger, have a certain number of leads and wires that are spaced apart from each other by a relatively narrow distance. Such components must also be accurately positioned to ensure that each lead is placed on a suitable pad. Therefore, not only is the pick-and-place device extremely fast to operate, but the component placement must also be extremely accurate.

To improve board manufacturing quality, fully populated or partially filled boards are inspected to identify incorrectly placed or missing components, usually after placement operations, and before and after flux reflow, Or various errors that may occur. Automated systems that perform such operations are very useful to help identify component layout issues before flux reflow, making rework modifications easier, and to help identify defective boards after flux reflow, such that boards are Candidate object for rework modification. An example of such a system is available under the trade designation model KS 200 from CyberOptics Corporation of GoldenVelley, Minnesota. Such a system can be used to identify problems such as alignment and rotation errors, missing and popped components, billboards, tombstones, component defects, polarity errors, and component errors. Identifying errors prior to flux reflow has a number of advantages: easier flux reflow, ease of closed-loop manufacturing control, and less effort in the production process between creating errors and fixing them. While these systems provide very beneficial inspections, they take up space within the facility, programming time, maintenance effort, and more.

A rather recent attempt to facilitate post-layout inspection inside a gripper device is disclosed in US Patent No. 6,317,972 to Asai et al. This reference reports a method for mounting electrical components in which an image of the mounting position is obtained before the component is to be placed, and this image is compared with the image of the mounting position after the component is placed in order to check the placement operation at the component level .

While the Asai et al. disclosure marks an attempt to employ component-level inspection within the machine, the Asai et al. disclosure is primarily concerned with turntable-type pick-and-place devices where the layout position does not move in the x and y directions , but simply move up and down. In such systems, a relatively large and cumbersome imaging system is located near the nozzle(s), images multiple placement events, and has little or no adverse impact on the speed of the placement equipment or design placement. In contrast, with bridge gripper devices (which Asai et al. pay relatively little attention to), the nozzle moves in at least one of the x and y directions. In this way, the optical path used to image multiple layout events also moves in the x and/or y direction. Therefore, the size and mass (inertial load) of the optical system itself may prevent the use of a bridge picker on a placement head in a bridge picker. Furthermore, since the placement head of the bridge picker moves in the x and/or y direction, it is important to reduce the size of the optical system to minimize collisions with other parts of the picker probability.

For a pick-and-place device with a placement head moving in x and/or y direction, the increased mass is a problem because of the increased inertia. Achieving a certain machine productivity depends in part on the acceleration of the placement head. The increase in mass will reduce the acceleration if some power is provided by the electro-mechanical system of the gripping device.

The size of the optical system fixed to the moving head, ie its volume and/or shape, can also be problematic for a number of reasons. One reason is that it is possible to design the moving head so that it just fits into its surroundings without colliding with anything as it moves around its workspace. Adding something that protrudes beyond the space in which the existing moving head structure exists must be done with great care, taking into account the possibility of physical collisions. Another reason the size and/or shape of the moving head can be an issue is that, typically, there will be a considerable amount of cables, pipes, motors, and other structures to be mounted to the moving head. It is usually not good to add something that might conflict with the installation or maintenance of the equipment.

To increase the viability of component-level layout inspection in pick-and-place devices, it is beneficial to make improvements in: optics, lighting, image acquisition, and image processing without compromising productivity and design Impact. In addition, it would be beneficial to provide enabling optics and techniques in both turntable and bridge gripping devices.

Contents of the invention

Various embodiments of the present invention provide improvements with respect to component level inspection performed by a pick and place device. Such improvements include: stereoscopic imaging of the intended layout location; enhanced illumination so as to provide relatively high power illumination in the confined space near said layout nozzle; enabling image acquisition equipment to be viewed from a plane relative to the layout location at a certain angle. Observe the optical path of the layout position at an angle, thereby reducing the possibility that the image is blocked by components; use the technology of quickly acquiring images with a commercially applicable CCD array, so that the acquisition of front and rear images does not significantly affect system production efficiency; Image processing technology that provides component inspection and validation information. These and other advantages of the invention will become apparent from the following description.

Description of drawings

Fig. 1 is a schematic diagram of a grabbing device that can implement an embodiment of the present invention;

FIG. 2 is a simplified schematic diagram of a stereoscopic image acquisition system according to an embodiment of the present invention;

Figure 3 is a flowchart illustrating a method of operating a grabbing device according to an embodiment of the present invention;

Figure 4 is a block diagram illustrating how various images may be used and/or combined to provide component layout inspection in accordance with one embodiment of the present invention;

Fig. 5 is a schematic diagram of a lighting system according to an embodiment of the present invention;

Figure 6 is an enlarged view of a light pipe mounted to a light emitting diode (LED) according to one embodiment of the present invention;

7 is a schematic diagram of the optical device of the image acquisition system of the grasping device using the Scheimpflug condition;

Fig. 8 is the schematic diagram of the used interline transmission CCD array of the embodiment of the present invention;

Fig. 9 is a schematic diagram of an image acquisition system using an electronic shutter according to an embodiment of the present invention;

10 is a flowchart of a method for acquiring multiple images using an interline transfer CCD array according to an embodiment of the present invention;

Fig. 11 is the schematic diagram of the used frame transmission CCD array of the embodiment of the present invention;

12 is a schematic diagram of an image acquisition system using multiple CCD arrays and a shared optical device passing through a beam splitter according to an embodiment of the present invention;

13 is a top view of a multiple image acquisition system for generating multiple images of a layout position according to an embodiment of the present invention;

14 is a schematic diagram of a multi-optical path imaging system;

Fig. 15 is a flowchart of a method for identifying a position in an obtained image that facilitates image comparison according to an embodiment of the present invention;

Fig. 16 is a schematic diagram of a pixel and its adjacent surroundings;

17 is a schematic diagram of a method for analyzing an image to provide component layout confirmation according to an embodiment of the present invention;

FIG. 18 is a schematic diagram of a method for analyzing an image to provide component layout confirmation according to another embodiment of the present invention;

Fig. 19 is a schematic diagram of a method for analyzing an image to provide component layout confirmation according to another embodiment of the present invention;

20 is a schematic diagram of a method for analyzing images for value/type verification, X, Y, θ position alignment measurements, and polarity determination, according to an embodiment of the present invention;

21 and 22 are schematic diagrams of calibration targets according to embodiments of the present invention;

23A-23G are representative images used and/or generated by embodiments of the present invention.

Detailed ways

Although embodiments of the present invention are described with respect to a bridge-type grab device, those of ordinary skill in the art will recognize that embodiments of the invention may also be applied to other forms of grab devices.

FIG. 1 is a schematic diagram of a typical grab device 201 to which embodiments of the present invention may be applied. Pick and place equipment 201 receives a workpiece, such as a circuit board 203 , via a conveyor system or belt 202 . The placement head 206 then obtains one or more electrical components to be mounted on the workpiece 203 from a component supply (not shown) and moves in the x, y and z directions to place the components on the workpiece in the correct orientation. Correct location on 203. The placement head 203 may include a sensor 200 that may pass under the component held by the nozzles 208, 210, 212 as the placement head 206 moves the component from the pick position to the placement position. The sensor 200 enables the picker device 201 to view the underside of the component held by the nozzles 208, 210, 212 so that component orientation can be achieved while the component is moving from the component pick position to the placement position, and in a certain Component inspection can be achieved to a certain extent. Other pick-and-place devices may use a placement head that can move over a stationary camera to image the component. The placement head 206 can also be an overhead camera 209, which is typically used to locate fiducial marks on the workpiece 203 so that the relative position of the placement head 206 relative to the workpiece 203 can be easily calculated.

Fig. 2 is a schematic diagram of a placement head according to an embodiment of the present invention. FIG. 2 shows a pair of image acquisition devices 300 , 302 for obtaining an image of the placement location 360 of the component 304 prior to placing the component 304 from the nozzle 210 at the location 360 . Before placing the component 304, the image acquisition devices 300 and 302 acquire an image of the placement position 360 of the workpiece 304 on the workpiece 203 and simplify the process thereafter. Compare these before and after placement images for component level inspection and validation. Since obtaining an image of the placement site is typically performed while a nozzle, such as nozzle 210, is holding the component 304 over the placement site, it is important to be able to image the placement site 360 while simultaneously capturing the image from the component itself, or possibly Interference from other nearby elements already mounted on the workpiece is minimized. Thus, it is preferred that the optical axes used by apparatuses 300 and 302 be viewed tilted at an angle θ with respect to the plane of workpiece 203 . To compensate for the angle [theta] at which devices 300 and 302 view placement location 360, devices 300 and 302 are preferably adapted to use Scheimpflug conditions. A special way in which Scheimpflug conditions can be used is to adapt the area array detectors to be arranged in each device 300 and 302 relative to the optical axis of the optics of the corresponding device 300, 302 at an angle. In this condition, the tilted object plane (positioning position) is correctly imaged to the tilted image plane (area array detector within the device 300, 302).

As shown in FIG. 2 , devices 300 , 302 are also preferably separated by an angle φ about the z-axis so that devices 300 , 302 can provide a stereoscopic image of placement location 360 . Such stereoscopic imaging has a number of benefits, which are described in more detail below. Stereoscopic imaging of the conditions in front of and behind the placement location can utilize the x-y data to generate the desired height profile. Two advantages of stereoscopic imaging are: it produces a depth map of expected component placement; and it minimizes, or at least reduces, the chance of component images being obscured by other taller components along the line of sight. If the component image is blocked a little, it can usually be observed with other cameras.

Each of the devices 300 , 302 preferably also includes a non-structural illuminant 306 and a structured illuminant 308 . While non-structural illuminators 306 are shown provided on each device 300, 302, in some embodiments it is also possible to provide a single non-structural illuminant mounted separately.

The system shown in Figure 2 provides a series of features that can be used alone or in combination to give superior component inspection. One way to use this system is to use structured light to reconstruct the height of any one image from the placed position 360. Alternatively, simply use structured light as the source of illumination, providing a pattern of light on the surface at the placement location 360, and then use known correlation algorithms such as those reported by Rosenfeld, A. and A.C. Kak (1982, "Digital Image Processing", Volume 2, Science Press, New York), correlating structured illumination between two images obtained from different after placement). As used herein, structured light includes: sinusoidal patterns, regular arrays of light spots or shapes, random patterns of light, and even pseudo-random patterns of light. Sometimes there are also nozzle configuration situations when parts of the workpiece surface do not appear in one of the two images. Additional cameras may be positioned to observe portions of the workpiece surface that may have been missed from one or more images. Alternatively, missing image parts can also be stitched from other images (before or after placement) to obtain the necessary images to either provide quality information for the layout, or provide the stereo information necessary for height information.

FIG. 3 is a flowchart of a method of obtaining images so that component level inspection can be implemented in a pick-and-place device. At step 350, one or more non-structural illuminators (such as illuminators 306) are activated to illuminate the intended placement location prior to component placement. At step 352, each device 300, 302 acquires a pre-placement image of the placement location when illuminating the devices 300, 302 with the non-structural illuminant. Preferably, the devices 300, 302 are caused to acquire their respective images simultaneously or substantially simultaneously with each other. As used herein, substantially simultaneously means close enough in time that the physical system does not move appreciably between image acquisitions. In this way, in step 352, the pre-placement stereoscopic image is obtained, and the excitation non-structural illuminator is removed. Preferably for pre-placement inspection, pre-placement stereoscopic image(s) of structured illumination at expected component layout locations may be used. Such operations include: solder paste being applied in place to accept the associated component leads; no bit debris or other undesired material being placed at the target placement; or any other inspection operations that may be desired.

Structured illumination images can be acquired from each image acquisition device in order to obtain three-dimensional image information of expected component layout positions before component placement. As soon as each image acquisition device uses its own structured illuminant, it should then acquire its corresponding image so that the structured illumination of one image acquisition device does not interfere with the structured illumination of another image acquisition device. However, if an appropriate structured illuminant is used so that the structured illumination of each illuminant is sharp during processing, each image acquisition device may acquire their corresponding structured illumination images simultaneously.

In step 354, the nozzle (such as nozzle 210) component is placed in its intended component layout position. In step 356, the image acquisition device 300, 302 acquires the stereoscopic image of the placement position again, the stereoscopic image after placement. Preferably at step 358, structured illuminator 308 may be activated to provide structured illumination on component 304 when component 304 is seated in its intended component layout position. The apparatus 300, 302 is then free to acquire the next set of stereoscopic images when the intended component layout location is illuminated by the structured illuminant (block 359). Optional steps 358 and 359 facilitate generating a placed 3D fiducial image of the desired component layout location.

Figure 4 is a schematic diagram of how various illuminators and image acquisition devices are used to generate various images and how these images are combined to give important inspection information. As shown in FIG. 4 , the device 300 includes a non-structural lighting body 306 , a camera 310 and a structured lighting body 308 . Device 302 includes non-structural illuminant 306 , camera 312 and structured illuminant 308 . As can be seen from the figure, the camera 310 of the device 300 and the unstructured illuminator 306 cooperate with each other to generate a pre-placement grayscale image 314 and a post-placement grayscale image 316, and then use these images to generate an image corresponding to the image of the device 300. Gray scale difference (Δ) image 318 . Additionally, camera 310 and structured illuminator 308 cooperate to generate structured image 320 of the image of device 300 .

Device 302 generates images 322, 324 and 326 similarly to those described above with respect to device 300, but from different images, thus facilitating stereoscopic imaging. In particular, image 322 is preferably obtained concurrently, or substantially simultaneously, with image 320. Similarly, images 324, 326 are preferably obtained at the same time, or substantially simultaneously, as images 316, 314 are obtained. The two structured illumination images 320 and 322 taken from different images of the devices 300 , 302 , respectively, may be combined to provide a 3-dimensional fiducial image 328 . Additionally, the grayscale images obtained by device 302 may be combined to provide grayscale difference image 330 . Each of the grayscale difference images 318, 330 for projection is corrected using the 3-dimensional fiducial point image 32, thereby forming projection-corrected first and second grayscale difference images 332, 334, respectively. The projection-corrected grayscale difference images 332 and 334 are then used for a series of component layout inspections. Examples of such inspections may include: component presence checks, as shown in block 336; recognition of graphics/characters, as shown in block 338; analysis of part geometry, as shown in block 340; image symmetry Sex analysis, as shown in block 342. The graphic/character recognition 338 on the element 304 itself facilitates confirmation of the correct type of value for the element 304 , as indicated by block 344 . Additionally, by analyzing the geometry of the part, x, y, and rotational angle θ alignment can be measured and confirmed, as indicated at block 346 . Finally, analyzing the symmetry of the image, as indicated at block 342 , provides a convenient processing method for analyzing component polarity, as indicated at block 348 .

illumination

The following schemes related to lighting provide additional advantages for embodiments of the present invention. In fact, aspects of illumination disclosed below encompass aspects of the present invention, but need not necessarily be implemented using the above-described stereoscopic imaging techniques.

Due to the need for high resolution, short image acquisition times, low reflectance of the target object towards the camera, and other reasons, relatively bright illumination is usually required to obtain a sufficiently good image quality. Light-emitting diodes (LEDs) are usually a good choice because they are generally reliable, efficient, inexpensive, and compact. However, in pick-and-place devices, the space around the point of placement is strictly limited, so it is extremely difficult to locate the required LEDs near the point of placement.

In such cases, it is beneficial to use light pipes, especially fiber optic guides, to deliver light from remotely located LEDs to the target area. The advantages of using light pipes in pick-and-place devices include: the space required for a sufficiently large number of LEDs for bright lighting to be installed near the placement location is very large; the removal of the LED light source from the placement location near the placement location can Reduced heating near the nozzle (which is a problem since thermal expansion can change the dimensions of the element, nozzle and any other heated material); LEDs can be arranged to optimize packaging; and optical fibers The guide tube is flexible and the fiber optic guide tube can be suitably arranged to deliver light with respect to a moving placement head with a stationary LED light source.

Many applications require properly placed lighting to reveal features or details necessary for further processing. In some cases, either for efficient use of space, or to better match certain inherent symmetries of the imaging geometry, it may be desirable to arrange the lighting to complement the symmetry. For example, use a large ring light to create a uniform circular feature on a spherical bead, or use a linear illuminator to greatly reduce the size of a line scan camera.

Commercially available lighting structures are usually simply rectangular or circular arrays or chip-level arrays of LEDs packaged separately. While any of these LED assemblies can be arranged in arbitrary configurations, the size of the discrete components limits this arrangement to roughly roughly spaced apart; chip-level illuminator arrays are usually limited to flat planes . In either case, implementing either arrangement can be a complex and expensive burden.

Light emitting diodes or other suitable light sources coupled to flexible fiber optic conduits provide a modifiable and simple technique for realizing almost arbitrary lighting configurations. The complication in getting the light source to have the desired position and orientation is how to fix the output end of the fiber. Since optical fibers suitable for use as light pipes are usually quite flexible and have relatively small diameters (0.25-1.0mm), by securing the output end of the fiber optic guide to one or more suitably fabricated members, the This arrangement can be implemented.

Efficient coupling of fiber optic conduits to LEDs generally requires a lens between the LED assembly or module and the fiber input. Each LED lens and optical fiber must maintain strict alignment. The whole assembly becomes somewhat bulky and not well suited for a pre-made arrangement, especially if a single-piece lens array is used.

One aspect of the lighting assembly according to an embodiment of the invention includes coupling the fiber optic conduit input directly to the light emitting diode.

Fig. 5 is a schematic diagram illustrating various aspects of an illumination component according to an embodiment of the present invention. FIG. 5 shows nozzle 210 and image acquisition device 300 positioned relative to circuit board 203 so that device 300 acquires an image of placement location 360 when it is illuminated by illumination system 306 . Illumination system 306 is preferably an array of individual point light sources, such as LED array 362 having a plurality of individual LEDs 364 . An appropriate number of light pipes 366 are coupled to a single light source 364 to deliver illumination from array 362 to a location proximate placement location 360 . Preferably, the output end 368 of the light pipe 366 is coupled adjacent to a fixture 370 that can mount the output end 368 of the light pipe 366 in any desired manner. output 368 may be beamed, aligned, and/or contoured so that illumination may be directed toward placement location 360 at an appropriate angle and at an appropriate location to provide appropriate brightness and/or contrast in the image; and A desired image feature shape or appearance may be provided. In addition, output ends 368 may be bundled, aligned, and/or contoured to accommodate the size, space, and other mechanical constraints of the placement head.

FIG. 6 is a schematic diagram of an input end 368 coupled to an LED fiber optic conduit 366 . A hole 369 is formed in a portion of the LED 364 assembly or package to receive the light pipe 366 . In this manner, the fiber end 368 can be placed in close proximity to the LED module, making the geometrical coupling efficient. Additionally, an index matching adhesive can be filled in the space 370 between the fiber end 368 and the LED acrylic package or assembly 372 . However, if an acrylic fiber is used for the light pipe 366, the fiber 366 and LED assembly can simply be fused together, resulting in reduced loss. This arrangement has a fairly high coupling efficiency and does not require a coupling lens. However, the alignment is fixed and stable, and can be designed for optimal packaging. For example, fibers are allowed to exit the assembly at a considerable number of angles. Furthermore, LED and fiber optic assemblies can be housed in the case for added robustness.

In some embodiments, all LEDs in the array 362 are of one color. Preferably, the image acquisition device may comprise a filter which rejects ambient light but passes light having a wavelength in the illumination band. However, it is clearly contemplated that several colors of light emitting diodes may be used along with one or more suitable filters for use therewith.

light path

Illumination systems according to embodiments of the present invention must not only be compact and adaptable, but must also accommodate strict constraints on the inspection camera itself in terms of size, shape and viewing access to the area to be imaged. While it is preferable to assemble both imaging elements and lighting elements into a single integrated and interoperable unit, such integration is not necessary to practice embodiments of the present invention.

As noted above, the image acquisition systems 300, 302 preferably use tilted image planes to enable the devices 300, 302 to use relatively large viewing angles and to accurately image these angles toward the imaging array. An example of using such an oblique image plane is the Scheimpflug condition. While this technique can be used with both telecentric and non-telecentric optics, embodiments of the invention preferably use non-telecentric optics. The reason for this is simply that one of the requirements for telecentric imaging optics is that the target element of the telecentric imaging optics must be larger than the field of view.

FIG. 7 is a schematic diagram of the preferred optical system employed in either or both devices 300,302. Optical system 380 is non-telecentric optics, allowing the diameter of objective lens element 382 to be smaller than the field of view. In addition, since the optical system 380 is a non-telecentric optical device, the height H can be reduced, and a compact camera can be provided compared with a telecentric optical system. In order to increase the oblique proximity of the line of sight to placement location 360 , and/or to minimize the diameter of the optical system, a stop 384 is inserted between objective lens element 382 and placement location 360 . Additionally, an aperture 384 may also be used to assist in blocking ambient light. The optical axis can be bent using a mirror 386 as shown in FIG. 7 . Preferably, stop 384 is positioned between objective element 382 and mirror 386 .

When using non-telecentric optics, it is recommended to perform a calibration process so that the x,y pixel addresses can be mapped to x,y locations on the intended component layout plane. To this end, calibration targets are placed on the layout area plane, said targets on this plane having a known series of indication values, preferably squares of known dimensions at known intervals. Then, a known calibration target on the plane of the layout area is observed to enable the system to solve for a 3x3 uniform projective transformation matrix. This matrix converts x,y pixel addresses to x,y positions on the layout plane, preferably in millimeters. This transformation also corrects for scale scale and projection distortions. Furthermore, the use of two or more cameras facilitates stereopsis correction in order to derive depth (z) data from features that can be viewed in the two or more cameras. Z data is used to project features with heights vertically down onto the layout plane. Figure 21 shows a preferred calibration target. Process the image of the calibration target into a binary image by thresholding the image. To locate these black squares, a connectivity performance analysis was performed. These lines are then fitted exactly to each side of each square in the grayscale image with sub-pixel accuracy. The intersection points of all adjacent lines are then calculated. If a selected number of squares are visible in the image (e.g. 30 squares), then the number of squares multiplied by the number of corners (30×4) provides a fairly large number of points (e.g. 120 points), at These points can compute x, y values with sub-pixel precision. These positions are then compared to known angular positions on the calibration target in millimeters to calculate a calibration matrix.

Figure 22 shows an image of the calibration target during processing, where the plus sign indicates the center of the "bubble" and the layout of the line fits is clearly visible on the sides of the rectangular grid. To handle extreme shadows, the algorithm starts with the largest light bubble (indicated by reference numeral 600) in the image. The calibration matrix is successively refined by including the closest largest square, and then more and more squares until all squares in the grid are used in the final calculation. The calibration procedure is described below. First, all light bubbles are extracted from the image. Clipped blebs are discarded, and the bleb with the largest area is located. Then, roughly locate the 4 corners of the largest area light bubble. Using line fitting, fit these lines to the four sides of the large square. The system then solves for a 3×3 uniform transformation matrix using transposition, scaling, and correction to convert pixel units to millimeter units. This is a fairly rough approximation, making the 2D array of squares orthogonal in the image. Then, a two-dimensional grid of bleb pointers pointing to previously found blebs is generated. This is based on known millimeter spacing in the calibration grid. Then, start a cycle with the (+/-) row, (+/-) column blob (there may be as many as 9 blobs) in the largest blob in the grid. For each blob in the loop, use that blob with the four corners of the largest blob to solve for a new 3x3 pixel-to-millimeter transformation. Using the refined transformation, the 2-dimensional grid of the light bubble pointers to the earlier discovered light bubbles can be reproduced. The available grid is then expanded around the blob currently used by the (+/-) row, (+/-) column buff. If the expansion of the grid is already filled, the loop ends. Otherwise, continue looping for the next light bubble. Once the loop is complete, the bubble center is transformed using the final calibration matrix computed during the loop execution. Line fits are then performed for all four sides of all light bubbles in the grid, and the angles are calculated for all light bubbles in the grid by solving for the intersection points of all adjacent fitted lines. Finally, using all the corners found in the image and their expected positions, knowing the square size and grid spacing in the calibration target, the final calibration matrix can be solved.

Calibration for stereo vision provides depth data from features observed using two or more cameras. Calibration of two or more cameras with overlapping fields of view can be achieved by performing the above calibration twice. After calibrating on a calibration target that coincides with the layout plane, use a taller calibration target (with a known height) and repeat the calibration. Depth is determined by analyzing the direction and magnitude of the spatial shift of image features caused by different positions of the calibration target in each camera image and comparing the shift between cameras. The distance from the camera is inversely proportional to the amount of image movement.

image acquisition

In general, embodiments of the present invention obtain two or more sequential images (ie, before placement and after placement) of desired component layout locations. Because placement occurs relatively quickly, and because it is not desirable to reduce the productivity of the equipment, it is sometimes necessary to acquire two successive images extremely quickly, since the cessation of relative motion between the placement head and the board is extremely brief. For example, the two images must be acquired within a time interval of about 10 milliseconds.

According to various aspects of the present invention, there are different ways to rapidly acquire multiple successive images. One approach is to use commercially available CCD devices and make them work in a non-standard way to acquire images at a faster rate than they can be read from the device. Another method is to use multiple CCD arrays to observe the desired component layout positions through a common optical device.

Currently, there are two common CCD array structures: interline transfer CCD array (as shown in Figure 8) and frame transfer CCD array (as shown in Figure 11) (referred to here as IT and FT). These two types can be combined into a device that includes an image area, which accumulates an amount of charge proportional to its exposure, and a storage area, which is shielded from light. The image area of a CCD is a two-dimensional array of image elements (hereinafter referred to as pixels). The storage part and how the images are transferred to and through the storage part is an important factor that differentiates the two device types. In the interline transfer CCD, the entire image is transferred from the image part to the storage part near the pixel within a few clock cycles, while the frame transfer CCD transfers the image from the image part to the storage part one row at a time.

The action of transferring CCD390 between lines usually occurs in the following order. First, the image area is reset, clearing the pixels of any residual charge, as shown at block 410 in FIG. 10 . Once de-reset, the pixel begins to accumulate charge in response to light, as indicated by block 412 . After a suitable exposure time, within a few clock cycles, the charge accumulated in the entire image portion of the array is transferred to the storage portion of the array, as shown in block 414 . Once the image is in the storage section, the image is protected from light and the next image can be exposed. Since the transfer of the first image frame into the storage location occurs within a few clock cycles, the time delay between acquiring each image frame may be less than 100 microseconds. Each row of the first image in the memory section is then transferred to the horizontal register 396 where each pixel is shifted out, one at a time (or several at a time for multi-drop devices), and each pixel is digitized, as in block 416 shown. Once the first image is timed out, the second image can be timed.

Essentially, the frame rate of a CCD increases as the number of lines in the image decreases. For the image geometry as shown in Fig. 7, the aspect ratio of the image is elongated in the oblique direction. Since the full image is not needed in this direction, the resolution (ie number of lines) in this direction can be reduced. The reduction in image line resolution is accomplished by transferring any undesired lines in the CCD storage section to the horizontal register 396, and the lines are not shifted. Once all undesired lines are clocked into horizontal register 396, horizontal register 396 is shifted its entire length to clear any residual charge prior to transferring the first line of the intended image from memory section 394 into horizontal register 396. Therefore, the frame time reduction is equal to the number of undesired lines multiplied by the time required to shift a line from the horizontal register.

Another common type of CCD device is the frame transfer CCD array. According to embodiments of the present invention, this type of CCD array may also be used. Operation of such a device (shown in Figure 11) occurs in the following order. First, the image area 452 is reset, clearing the residual accumulated charge therein. Once reset is released, the pixels making up the image area 452 begin to respond to light to accumulate charge. After an appropriate exposure time, the charge accumulated in each pixel moves vertically into storage area 454, one row at a time. It is not necessary to move all the lines from the image area to the storage area, the first line to be moved must be the bottom line. Also, it is important to state that the pixels comprising the unmoved portion of the image continue to be exposed to incident light and are therefore subject to erosion by this light, in this case in the vertical direction. Therefore, frame transfer devices are most often used in environments where there is some external device that can control when and whether light reaches the detector.

The image in storage area 454 is suitable for readout, much like an interline transfer CCD structure. The key difference, however, is that since each row of the stored image is shifted vertically, thereby loading one or more horizontal readout registers 456, the photosensitive portion of the device is also shifted vertically. This means that reading of a stored image cannot take place while the photosensitive area of the device is being exposed to an image which is to be retained for later readout. For further details on the high-speed acquisition of multiple images on a frame-transfer CCD array, see co-pending U.S. Patent Application No. 09/522,519 (filed March 10, 2000, entitled "Inspection System Using Vibration-Resistant Video Capture ").

As mentioned above, along one axis of the image, it is not necessary to use the full image resolution. Therefore, the number of rows required to cover the layout area is less than the number of columns in the image array. By properly orienting the array, only the rows representing the layout positions need be transferred into the memory area, further reducing the time required between image acquisitions. With known CCD devices it is possible to obtain multiple images within a time interval which is less than the normally prescribed frame period. This image acquisition is conveniently used in conjunction with a method of controlling the light falling on the device after exposing the first image.

In order to prevent encroachment on the second image, it must be ensured that only negligible light reaches the CCD while the second image is in the photosensitive portion of the first image waiting for the first image to be read from the memory area. In this case, the exposure of the second image can start at any time after the first image has been transferred to the storage area, including very short times.

One way to solve this problem is to use electronically controlled optical shutters to control the light falling on the CCD array. Electronically controlled optical shutters include, but are not limited to, liquid crystal light valves, microchannel spatial light modulators, and electro-optic optical switches.

FIG. 9 shows one such electrical shutter 402 disposed between the interline transfer CCD array 390 and the lens 404 . The following describes the working process of the system shown in FIG. 9 with reference to the flowchart shown in FIG. 10 .

At step 410, when the system is initialized, a control signal is also sent to the electrical shutter to open the shutter, thereby allowing light to pass, and a signal to reset the pixels of the CCD 390. After the first image has been transferred to the storage portion of the array 390, as indicated by block 414, the photosensitive pixels are ready to begin acquiring a new image. After the appropriate integration period, a signal is sent to the electrical shutter 402, causing the shutter 402 to close, preventing light from passing, as indicated by block 422. At block 424 , the readout of the first image from the array 390 is complete and control passes to block 426 where the second image is transferred from the pixels 392 to the memory array 394 . Finally, at step 428 , the second image is read out from array 400 .

Alternatively, using an electrical shutter with a frame-transfer CCD array, two full-resolution (all lines) images can also be acquired in rapid succession. To do this, reset said region 452; expose region 452 to the first image; block additional light falling on photosensitive region 452; move the resulting image row by row as fast as possible into storage region 454, said This final image of the device is only limited by the vertical transfer cycle speed of the device; preferably reset the photosensitive area 452 again; expose the photosensitive area 452 to the second image; again block additional light falling on the photosensitive area 452; and then pixel by pixel, And two images are read out line by line.

It can be seen that this process can be extended to rapidly acquire multiple images of more than two, but to achieve this requires each image to have less than 100% of the available rows. For example, to obtain 3 images of the same size in quick succession, the images are limited to half the size of the full image.

Additional solutions for controlling CCD exposure include: controlling the lighting and light environment of the imaging system itself. One way to control the light environment is to control the wavelength of the illuminator to ensure that this wavelength is within a very narrow wavelength range, thereby minimizing the effect of ambient light, and setting a filter in front of the CCD. This filter is designed to reject light that is not in the wavelength range of the illuminator. When using a system according to this embodiment, the environment in which the camera is placed should not include ambient light falling within the wavelength of this bandpass filter. Thus, the pixels within the array are only sensitive to the external illuminator, which is controlled by the camera and flashes synchronously with the exposure of the CCD.

Another way to control ambient light is not to control the wavelength of the illuminator, but to provide an aperture of sufficiently small size, or a neutral filter placed in the optics series, so that the ambient light can be reduced to be fairly inconspicuous s level. Because the ambient light is reduced, the illuminants can be gated at the proper time and for the proper duration, providing enough light to expose the pixels even in the presence of small apertures or neutral filters.

Figure 12 shows an alternative arrangement that may be used with either or both of the devices 300, 302 to obtain multiple images at a time interval, which may have a frame rate that exceeds that of commercially available CCD cameras frame rate. Specifically, system 470 includes a pair of CCD arrays 472 , 474 that are both coupled to circuitry 476 that also controls an illuminator 478 . The two CCD arrays 472 , 474 are part of a train of optics that uses a beam splitter 480 to allow the two CCD arrays 472 , 474 to view the target plane through a lens 482 . Although similar structures are used in modern color CCD cameras, which typically include 3 different CCD arrays, each receiving a different wavelength band of light, it is not believed that such a device has been used to obtain multiple Basically monochrome images that are extremely close to each other, but not at the same time. Instead of using each CCD array to acquire an image simultaneously (as in the case of a color CCD camera), each CCD array can also be triggered sequentially to capture images before, after, and at any time during when a target is placed in the field of view . As shown, circuit 476 is coupled to high speed data bus 484 and to trigger information source 486; said trigger information source 486 preferably comprising a suitable position encoder. It should be noted that various aspects of this embodiment can be combined with the high-speed image acquisition described above, so that the operation of each CCD array can acquire images at a rate faster than the readout rate from each array, effectively and quickly capture Multiple images of more than two images.

Although the system described with reference to Figure 2 has a pair of cameras for stereoscopic imaging, additional cameras and lighting systems may be used for additional benefit. Accordingly, one aspect of the present invention involves the use of multiple complementary images of an inspection target, with both overlapping and non-overlapping regions in the images.

Multiple images can completely cover the target. The non-overlapping regions of any image are filled with information that all other images do not. This is especially important for inspection applications where the image is usually not taken from directly above, as oblique images are partially blocked.

Multiple different images of the same target can be used to generate unique information not available from just one image. (1) The overlapping portions of the images provide redundant data to improve the identification of features in the common area. (2) Unique information, such as height, can be derived directly or through some specific preparation phase (eg structured lighting).

Multiple images are given by:

1) Completely separate illuminator, optical device and detector;

2) completely separate optics and detectors with common illumination; or

3) Separate detectors with common optics and illumination.

The simplest configuration consists of multiple identical "cameras" (detectors, optics, and illumination) in full suite, whose fields of view essentially overlap and completely cover the intended target from all sides.

Fig. 13 is a schematic view of a 4-camera configuration viewed from above. Cameras 300, 302, 500, 502 capture images and may also provide lighting. Note that only a small portion of each corner of the target is visible in one of the 4 images. There is also a significant area of overlap between the two images.

Using multiple images to improve recognition results has a number of advantages. Rather than relying on complex optics to overlay the desired target in a single image, this method uses a small number of fairly simple cameras that work together. Not only is the coverage complete, but additional information can be derived from the redundant overlap of different camera images. This information is not available from a single camera. In addition, there is some reliability provided and increased by redundancy, since, with multiple images, areas occluded in one image can be modified or replaced with similar parts of another image .

With the use of suitable optics, a single detector can also be used in order to collect multiple images by using an imaging system with multiple optical paths. Figure 14 is a schematic diagram of such an imaging system. Each individual image 504, 506 is directed by appropriate optics 508 to a subset of the detector surface. Thus, the image obtained by the detector will contain both images as shown at 509 . This may be particularly useful where detectors are prohibitively expensive or where space constraints require image transmission to the detector by fiber optic means or other means.

Image Processing

As described above, in order to confirm the layout of components on a printed circuit board, two images before and after placement of the intended placement of the components on the circuit board are acquired, and then the other image is subtracted from one image. In this way, portions of the image that do not change between the pre-placement and post-placement images are suppressed, and the artifacts of the newly placed components are clearly revealed in the final image. However, the pre- and post-placement images are often not perfectly aligned due to inaccurate mechanical vibrations, mechanical movement, or because the printed circuit board and/or camera are still moving while the image is being acquired. When the two images (before and after placement) do not align, artifacts appearing in the final differential image may be a false indication that the component is present. One technique for estimating the misalignment of two images is to use a correction (such as normalized grayscale correction). However, this correction can only be performed if vertical edges appear on the template (area to be corrected). If the edges do not appear, or if the edges only appear in one direction, the correction will not produce a unique (x,y) aligned position.

If CAD data is available and the vision system knows where on the board it is looking at for the CAD description, a vision system, such as the one used by embodiments of the present invention, can pick up good locations for the stencil. Alternatively, the vision system may want to predetermine the starting position or template if it is informed offline by every possible field of view being displayed. However, if these conditions are not met, then the vision system needs to quickly determine the good template position in runtime.

An aspect of an embodiment of the present invention provides a computationally efficient method for determining good locations in one image for selecting a template for correcting another image. Figure 15 illustrates this approach, which begins at block 510 where a Sobel edge filter is computed for the image prior to placement, where the magnitude and direction of the strongest edge around each pixel is computed. (Alternatively, simpler gradient operations can be used in a 3x3 neighborhood around each pixel). At block 512, edge values are thresholded and edge directions are rounded to one of the following eight degrees: 0, 45, 90, 135, 180, 225, 270, 315 degrees.

Figure 16 shows a pixel 514 and its 8 neighbors. Each arrow indicates a normal edge direction. At block 516, the 8 directions of each pixel are encoded into an 8-bit byte (edge encoded pixel). The eight directions listed in this step can be described by their compass directions as follows: East (E)-7, Northeast (EN)-6, North (N)-5, Northwest (NW)-4, West (W )-3, Southwest (SW)-2, South (S)-1, Southeast (SE)-0. At block 518, OR boxcar filtering is performed on the encoded data. Unlike a normal boxcar filter, which computes an average value similarly to a bandpass filter, this OR boxcar filter performs a bit-by-bit OR operation on the pixels in the vicinity of the aperture. The aperture stop may be 5x5 pixels or 7x7 pixels (or some other suitable size of pixels). In the final "image" of edge-encoded pixels, each pixel indicates which edge direction occurs in its 5x5 or 7x7 neighborhood. At block 520, each 8-bit edge-coded pixel in the image is scored using a predetermined look-up table. In general, only a small number of fractions (such as 4) are needed. Because the 8-bit code is programmed into the lookup table, the lookup table only needs ²⁸ or 256 elements. The higher the score, the better the 5×5 or 7×7 represented neighborhood is for correcting templates. Of the 256 8-bit edge codes, most are symmetrical, so only a few sample fractions are shown in the table below.

Table 1

Fraction Sample 8-bit code 0 (), (E), (E, W), (E, SE), (E, NW, W, SE) 1 (E, NE, SE), (E, NE, NW, W, SW, SE), (E, NE, W, SE) 2 (E, NE, S, SE), (E, W, SW, S, SE), (E, NW, W, SW, S, SE) 3 (E, NE, N, NW, W, SW, S, SE), (E, N, NW, W, SW, S, SE), (E, N, W, S), (E, NE, N , W, S)

At block 522, the size of the image is reduced by summing the scored 4x4 neighborhoods. (In other words, build a physical image). Alternatively, additional boxcar filtering can be performed on images with aperture stops of 4x4 or larger. Finally, at block 524, the final image scored is scanned for a high score. A high score indicates a good location in the original image and can be used as a correction template. Since the component will be placed somewhere in the central part of the image, finding high scores should be limited to outside the central part of the image. (In fact, in order to be efficient, all the above processing must avoid the central area of the image)

Figure 17 illustrates a method of analyzing acquired images to determine the presence or absence of a component (block 336) in accordance with an embodiment of the present invention. At block 540, two images before and after placement are acquired. Examples of two images before and after placement are shown in Figures 23A and 23B, respectively. Dashed box 544 represents an optional step of aligning the two images using a one-dimensional or two-dimensional correction. The alignment method described above with reference to Figures 15 and 16 is preferably used. At block 546, the aligned pre-placement and post-placement images are subtracted from each other. gives the absolute value of each pixel. This step highlights the changes in the image, thus highlighting the difference between the two images. A typical differential image is shown in Figure 23C. At block 548, thresholding is performed on the difference image, providing an image that may resemble the image shown in FIG. 23D. Thresholding on the difference image produces a binary image that distinguishes changed regions from non-changed regions.

At block 549a, regions in the thresholded image that are known to vary between the two images, such as nozzles and nozzle reflections, are masked. This masking is accomplished by drawing filled black polygons in an otherwise white masked image over the area known to vary. The masked image of known variation is ANDed with the thresholded image. Optionally, if element data including element length, width, and height are available at block 549b, the system generates a white mask that covers a two-dimensional projection on the image plane of a volume, said The volume of is expected to be occupied by parts in an otherwise black image. This masking is achieved by drawing a white filled polygon in the otherwise black masking image in the area where the part is placed. Then, an AND operation is performed on the desired part mask image and the thresholded image.

At block 550, one or more optional morphological erosion operations may be performed, as indicated by the cycle labeled "N". This erosion is useful for removing noise or image misalignments that cause the final result. At block 552, another optional morphological operation is provided. Specifically, block 552 provides an optional enlargement operation, which is performed one or more times, as indicated by the loop labeled "M". The dilation operation may facilitate the ablation of individual objects that have been separated by the erosion operation or separated using features of elements matched to the background image. An example of an image after such an enlargement operation is given in Fig. 23E. At block 554, a connectivity analysis is performed to find objects in the image. This analysis can find many useful metrics for each object in the image, such as the object's center, length, width, area, corner, etc. (Gleason et al. Vision Systems, Proceedings of the 9th Symposium on Industrial Robotics, March 1979, pp. 57-570). Finally, at block 556, the component placement is confirmed by comparing the target metric values found in the image to the expected target metric values. This comparison can include (but is not limited to) metrics such as center position, width, area, corners, and the like.

Figure 18 is a schematic diagram of another method for analyzing an image to determine the presence or absence of a component according to an embodiment of the present invention. The method shown in FIG. 18 also includes steps 540 , 542 , 544 , 546 described with reference to FIG. 17 . Accordingly, these steps are neither described nor represented in FIG. 18 , which begins at step 560 , which is performed after step 546 . At block 560, a pixel change count for the layout location is determined. This is achieved by counting pixels whose value is greater than a certain threshold and whose value is within the area of the image where the component is expected to be placed. At block 562, a total pixel count is determined by counting pixels whose value is greater than a certain threshold and whose value is not within an area of the image where the component is expected to be placed. At block 564, component layout validation is performed by comparing the layout location pixel change count (block 560) to the total pixel change count (block 562).

FIG. 19 is a schematic diagram of yet another method of analyzing an image to determine the presence or absence of a component (block 336 ), according to an embodiment of the present invention. The method shown in FIG. 19 also includes steps 540 , 542 , 544 , 546 described with reference to FIG. 17 . Accordingly, these steps are neither described nor represented in FIG. 19 , which begins at step 566 , which is performed after step 546 . At block 566, a "layout location pixel difference intensity sum" is calculated by summing pixels whose values are greater than a certain threshold and whose values are within the region of the image where the component is expected to be placed. At block 568, a "total pixel difference intensity sum" is calculated by summing the pixel intensity values throughout the difference image. At block 570, the component layout is confirmed by comparing the "layout position pixel difference intensity sum" and the "total pixel difference intensity sum".

Figure 20 is a schematic representation for analyzing images for "value/type confirmation" (block 344 in Figure 4), "x, y, theta alignment measurements" (block 346 in Figure 4) and "polar "Property Determination" (block 348 in Figure 4). The method shown in FIG. 20 includes steps 540, 542, 544, 546, 548, 549a, 549b described with reference to FIG. Accordingly, these steps are neither described nor represented in FIG. 20, which begins with step 582, which is performed after step 549b.

The procedure of steps 544-549b and 582-584 generates a masked image. Steps 544-549b are described above. Step 582 performs multiple times (eg, 5 times) of 3×3 binary expansion on the thresholded difference image. Step 584 is a large boxcar filter (typically 19x19 pixels) that blurs the expanded difference image, producing an image similar to Figure 23F. At step 586 , the blurred mask image produced by step 584 is multiplied by the "post-placement image" obtained at step 540 . This isolates the components placed in the image, erasing the non-changing parts of the image. Multiplying the blurred masked image by the placed image is better than simply ANDing the unblurred masked image and the placed image, because ANDing the masked image with the grayscale image may result in Creates an artificial edge. A typical image derived from this multiplication operation is shown in Figure 23G.

Step 588 places various metrology tools, such as wire staplers and "rulers," or "calipers," in order to locate the leads of the component, or in the case where the leads (actually solder beads) are hidden beneath the part. 4 sides. After the leads and/or sides of the component are located, the x, y and theta position of the component can be calculated.

Similarly, Optical Character Recognition (OCR) or corrections may be used on the image produced by block 586 for "Value/Type Confirmation". Additional metrology tools can also be applied to the image to determine polarity.

Although the present invention has been described with reference to preferred embodiments, workers skilled in the art will recognize that changes may be made in form and detail without departing from the spirit and scope of the invention.

Claims (1)

1. A method for obtaining a plurality of images in a grabbing device, wherein the method comprises the steps of: Reset the pixels of the photosensitive array; get the first image in terms of pixels; transmitting the first image to a non-photosensitive area of the array; resetting the pixels of the photosensitive array to clear the first image; and A controlled exposure is produced so that the second image is acquired before the first image is fully read out from the non-photosensitive area of the array.

Images