HK1190195B - Fast auto-focus in microscopic imaging - Google Patents

Description

Fast autofocus in microscopy imaging

Technical Field

The present invention relates to electronic imaging systems.

Background

Many imaging and scanning applications acquire images in an automated fashion. The images obtained should be properly focused to facilitate meaningful analysis and interpretation. To acquire a properly focused image, the focal length, e.g., focal height or length, should be properly determined prior to acquiring the image. This can be done by, for example, acquiring several images at a given location using different focal lengths, and then selecting the best focus image. Acquiring multiple images can be time consuming, especially in applications where a large number of images must be captured within a short time span.

There may be local variations in the imaged object or plane that require different focal lengths for different locations on the object or plane to obtain an acceptably focused image. For example, the surface of a glass microscope slide or a biological sample deposited thereon may not be perfectly flat, and simply determining the focal plane to represent the focal distance at different locations across the slide does not take into account such local variations. In such a case, to obtain a well-defined and sharp focused image, it may be necessary to determine a focal length for each of a plurality of locations on the slide. Fine-tuning the focus distance at each location by, for example, conducting a full focus search is time consuming and may not be feasible in many applications.

Disclosure of Invention

The imaging process for capturing images of objects on a substrate can be greatly accelerated using automated and efficient focusing methods and systems. Well-focused images are typically acquired in tandem at multiple locations along the substrate using the system and method of the present invention. The present invention is based, at least in part, on the discovery that: instead of performing a time-consuming overall focus search at each new location, the focus distance can be reliably estimated from the focus data of previously imaged locations or objects at each new location to be imaged. As more sites are imaged, the associated estimation error is progressively reduced by combining the previous estimates in a manner that averages such errors.

In one aspect, the disclosure features a method of automatically focusing a plurality of images of one or more objects on a substrate. The method includes acquiring, by a processor, a representative focal distance for a first location on the substrate based on a set of focal distances at known locations on the substrate. The method also includes acquiring, by the image acquisition device, a set of at least two images of the first location. Each image is acquired using a different focal length offset from the representative focal length. The method further includes estimating, by the processor, an ideal focal distance corresponding to the first location based on the comparison of the focus quality of each image, and storing the estimated ideal focal distance and the first location in the set of focal distances at known locations.

In another aspect, the disclosure features an imaging system including an image acquisition device and a processor. The image acquisition device is configured to acquire a set of at least two images of a first location on the substrate using a different focal distance for each image offset from the representative focal distance. The processor is coupled to the image acquisition device and configured to calculate a representative focal distance for the first location based on a set of focal distances at known locations on the substrate. The processor is further configured to estimate an ideal focal distance corresponding to the first location based on the comparison of the focus quality of each image, and store the estimated ideal focal distance and the first location in the set of focal distances at known locations. The processor is further configured to provide the calculated representative focal distance to the image acquisition device.

In another aspect, the disclosure features a computer-readable storage device having computer-readable instructions encoded thereon. The instructions, when executed by the processor, cause the processor to calculate a representative focal distance for a first location on the substrate based on a set of focal distances at known locations on the substrate, and estimate an ideal focal distance corresponding to the first location based on a comparison of the focus quality of each image. The instructions may further cause the processor to store the estimated ideal focal distance and the first location in the set of focal distances at known locations, and provide the calculated representative focal distance to the image acquisition device. The image acquisition device is configured to acquire a set of at least two images of the first location using a different focal distance for each image offset from the representative focal distance.

These implementations may include one or more of the following:

the processor may retrieve the set of focal distances at known locations. The image acquisition device may acquire a further set of at least two images of the second location, wherein each image in the further set is acquired using a different focal length offset from a second representative focal length calculated for the second location from the set of already focused lengths at the known location. The processor may estimate the ideal focal distance corresponding to the second location based on a comparison of the focus quality of each image in the additional set.

The representative focal distance may be estimated as a weighted average of at least two focal distances from the set of focal distances at known locations, wherein the weight for a given focal distance is calculated as a function of the distance of the respective location from the first location. Comparing the focus quality may include calculating a focus score for each image by quantifying differences between adjacent pixels. The focus score may be a Brenner focus score. Estimating the ideal focal distance may include calculating a difference between logarithms of focus scores of the images, estimating an offset from the calculated difference, and estimating the ideal focal distance from the offset.

At least two images may be acquired under illumination of substantially the same color of light, examples of illumination colors including green, yellow, blue, and red. The set of images may be acquired under illumination of at least two different colors, and at least two images may be acquired under illumination of each color. Estimating the ideal focal distance corresponding to the first location may include calculating a focal distance offset for each of the different colors, determining an average focal distance offset from the focal distance offsets calculated for the different colors, and estimating the first ideal focal distance from the average focal distance offset.

The present invention provides a number of benefits and advantages, including the following (some of which may be obtained only in some of its various aspects and implementations). In general, the present invention provides for fast acquisition of images by providing a fast auto-focusing process. By estimating the ideal or true focal distance at each imaging location, the disclosed systems and methods account for variations caused by local irregularities on the surface of the object without compromising the speed of the imaging process. For each location, an ideal focal distance is calculated from an image acquired using the focal distance calculated from the initial representative focal distance. Since the representative focal distance is based on a large number of previously known and/or estimated ideal focal distances at other locations, the estimation error is gradually minimized, thereby gradually yielding a more accurate estimate of the focal distance at successive imaging locations.

With respect to biological sample imaging, certain applications require that a large number of images be acquired in tandem. Some applications may also require processing of the acquired images quickly, possibly in near real-time. In such cases, the imaging process can be greatly accelerated by implementing methods and systems that facilitate fast and accurate autofocus on the imaging site. For example, in the case of imaging a sample containing blood cells, a precisely focused image aids in detecting, identifying, and classifying various cell types. In addition, properly focused images help to distinguish between normal and abnormal cells. Similarly, precise focusing also plays a role in distinguishing between mature and immature cells, fetal and maternal cells, and healthy and diseased cells. Furthermore, acquiring and processing images in tandem ensures that multiple blood samples, e.g., from the same or different patients, can be managed within a given time period, thereby improving the throughput of the system.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although some suitable methods and apparatus are described below, methods and apparatus similar or equivalent to those described herein can also be used in the practice or testing of the present invention. All publications, patent applications, patents, and other references mentioned herein are incorporated by reference in their entirety. In case of conflict, the present specification, including definitions, will control. In addition, the apparatus, methods, and examples are illustrative only and not intended to be limiting.

Other features and advantages of the invention will be apparent from the following detailed description, and from the claims.

Drawings

FIG. 1 is a schematic diagram of one embodiment of an imaging system implementing the fast auto-focus method described herein;

FIG. 2 is a flow diagram depicting one embodiment of a series of operations for estimating an ideal focal length;

FIG. 3 is a set of curves illustrating an example of a function used as a focus score;

FIG. 4 is a graph depicting an embodiment of a weighting function;

FIG. 5 is a graphical representation of an estimated focal length in one plane; and

FIG. 6 is a schematic diagram of a computing device and system.

Detailed Description

The present disclosure describes fast auto-focusing in an imaging device based on estimating an ideal focal distance at each imaging location rather than performing a time-consuming full focus search for each new location. The ideal focal distance at a given location is estimated from the focus scores of multiple images obtained using focal distances that are offset from each other and from the initial representative focal distance. The representative focal distance is calculated from one or more sets of training data, or taking into account knowledge of previously estimated ideal focal distances at other locations. Such cumulative estimation gradually averages the associated estimation error as more locations are imaged, and gradually yields a more accurate estimate of the ideal focal length.

Imaging system

FIG. 1 illustrates one embodiment of an imaging system 100 employing the fast auto-focus method described herein. The imaging system 100 includes imaging hardware 105 controlled by a computer 190. Computer 190 typically includes a central processing unit 192, a hard disk drive 194, and a random access memory 196.

In the imaging system shown in fig. 1, a light source 110 illuminates a slide 130 containing a biological sample 135. The slide is mounted on a first motorized stage 120, the first motorized stage 120 being movable in a horizontal plane (parallel to the surface of the slide being imaged) so that any portion of the slide 130 can be placed under the objective lens 140. The second motorized stage 150 moves the objective lens 140 up and down to facilitate focusing on the specimen 135 deposited on the slide 130. The distance between the slide 130 and the objective lens 140 is called the "focal length". Shortening the focal length means in this example moving the objective lens 140 vertically towards the slide 130. Alternatively, the focal length may also be adjusted by moving the slide 130 vertically towards the objective lens 140 (e.g., by moving the first motorized stage 120). In some implementations, both the slide 120 and the objective lens 140 can be moved to facilitate adjusting the focal length. The axes in which the first motorized stage 120 moves in the horizontal direction are commonly referred to as the X and Y axes. The vertical axis along which the second motorized stage 150 moves the objective lens 140 is commonly referred to as the Z-axis. These three axes define a coordinate system that the system 100 uses to image any (x, y, z) spatial point associated with the slide 130.

Light from the light source 110 passes through the slide 130 and is projected by the objective lens 140 onto the sensor of the camera 160. The sensor may be, for example, a Charge Coupled Device (CCD) array. FIG. 1 depicts an example of "bright field" microscopy in which objects on a slide are visible because they absorb light and are therefore darker in the image produced by the camera. The imaging hardware 105 may include one or more additional lenses. Microscopic modes such as fluorescence, dark field, or phase contrast can also generate images to which the fast auto-focusing methods described herein can be applied.

If the image is obtained at a non-optimal focal length, the image is blurred and generally not suitable for many image processing applications. If the surface of the slide 130 is perfectly flat, the system 100 can simply acquire an in-focus image by determining the appropriate z-height corresponding to the in-focus image at the three (x, y) locations and then fitting a plane to those three (x, y, z) locations. The plane equation will then provide an in-focus z-height for any other (x, y) location on the slide. In practice, however, the focal length at a given location may not be accurately determined from the fitted plane as described above due to irregularities on the surface of the slide 130 and/or stage 120. Thus, in general, it may be necessary to adjust the focal distance slightly for each image obtained at a different (x, y) location on the slide.

After image acquisition, camera 160 sends the image to computer 190 for processing. If the image is processed quickly enough, the focus data from one image location can be used to adjust the focus at the next location to capture the image. This allows the system to adjust for variations in focal length corresponding to different locations, which in turn results in a more accurately focused image for display. For example, if one region of the slide 130 is slightly thicker than the other, and ten locations are imaged in that region, then the thickness can be found to have changed after the first image is obtained, and additional images taken at other locations within that region can be acquired at slightly adjusted focal distances using the methods described herein to compensate for the thickness change.

Fast automatic focusing method

Fig. 2 is a flow chart 200 depicting a general method of auto-focusing, which includes a series of operations performed by a system that estimates ideal focal distances at different imaging locations on a substrate, such as a microscope slide. These operations include acquiring a set of images at an imaging location using a focal distance that is a predetermined offset from a representative focal distance (step 210), and calculating a focus score for all or a small set of images in the set (step 220). The operations also include estimating and storing the ideal focal distance for the imaging location from the calculated focus scores (step 230), verifying whether the system requires additional locations on the imaging substrate (step 240), and based on such determination, moving the objective lens and/or stage hardware to the new imaging location (step 260). The representative focal distance for the new location may be estimated as a weighted average of the estimated ideal focal distances stored from the previous imaging locations (step 270). The system repeats these steps for the new imaging location, and this process continues until the system obtains images of all locations and/or objects of interest located on the slide 130. These steps are described in more detail below.

Image acquisition

The set of images acquired in step 210 may be referred to as a "pile" of images. Acquiring the set or stack of images may include acquiring a first image and a portion of the target on an imaging plane of a different focal length than the first image or (x,_y) At least one second image of the location. The focal lengths of the different images in a stack can be taken togetherTo be determined as an offset from the representative focal length of the stack. In certain embodiments, a representative focal length corresponding to the stack is specified for a predetermined point in the stack, and the focal length of each image within the stack is calculated as a relative offset from such predetermined point. For example, for a stack of four images with respective focal lengths equally spaced from each other, a representative focal length of the stack may correspond to a midpoint between the second and third images, as measured from the top of the stack. Other conventions that specify a representative focal length for the stack may also be used in practicing the invention. For example, a representative focal length for a stack may correspond to a focal length corresponding to a particular image in the stack.

Different stacks typically correspond to different imaging locations and may include the same number of images from one location to another. Further, the components of the system may be preprogrammed to acquire the stack of images so that the entire stack of images is acquired in response to a single command from the computer. Such pre-programming, whether using hardware or software, can reduce latency compared to the case where separate commands are provided for each image.

The images in a stack may be acquired under different color illumination. For example, some images in a stack may be captured under blue illumination, while other images may be captured under yellow, green, or red illumination. Illuminating the specimen with different colored lights may result in different information being extracted from the acquired images. For example, in the case of imaging a specimen containing blood cells, the cells may behave differently under different color illumination, thereby facilitating easier identification, classification, or differentiation. For example, red blood cells absorb a significant amount of blue light due to the presence of hemoglobin, and cell nuclei stained with a standard romanostiki (Romanowsky) stain absorb yellow light.

Each stack may contain more than one image obtained under illumination with a given color light with a corresponding focus offset, as well as sets of images obtained under illumination with different color lights. For example, a stack may contain two images each obtained under illumination by one or more of blue, green, red and yellow light. The relative focus offset between images obtained at the same or different colors may depend on other parameters like resolution. For example, for low resolution (or low magnification) images of blood cells, the focus offset between two images obtained under the same type or wavelength of illumination may be about 1 to 10 microns, e.g., 2, 4, 5, 6, or 7 microns, while for high resolution images the offset may be as small as 0.1 to 1.0 microns, e.g., 0.4, 0.5, 0.6, 0.7, or 0.8 microns.

As described further below, when the estimated values of ideal focal distances at multiple locations on the substrate are known, the representative focal distance for the new imaging location may be determined as a weighted average of the known or estimated ideal focal distances for the previous imaging locations. However, when the first stack (or the first few stacks) is acquired, there may not be enough focus information available for other locations on the substrate. In such a case, the system may first image multiple locations on the slide to determine the degree of tilt of the slide plane. This can be done, for example, by determining the actual focal lengths of three or more locations on the slide (e.g., a fine-tuning process), and fitting a plane through the determined points to estimate the tilt of the focal plane. Imaging various locations in the plane may take into account or correct for the effects of tilt throughout, and an initial representative focal distance may be calculated from the tilt measurements.

Calculation of focus score

The operations also include calculating a focus score for all or a small set of images in the acquired set or stack (step 220). The focus score of an image represents the deviation of a representative focal length of a given point on the imaging plane from an ideal or true focal length. Thus, in some implementations, the ideal focal distance may be estimated from the calculated focus scores from the multiple images. The focus score may be calculated using a function that quantitatively evaluates the focus quality of a given image. If the focus score is calculated for each acquired image using one such "autofocus function," the focal distance corresponding to the highest scoring image may be selected as the ideal focal distance. Some autofocus functions operate on the basis of image differences, i.e. quantized differences between adjacent pixels, in general, images that are blurred due to focus differences have smaller differences between adjacent pixels, while well-defined focused images have higher differences between adjacent pixels.

Different types of image difference functions are used to measure the relative focus quality of an image. For example, the brenner score may be used to measure the focus quality of an image. Calculation of the brenner score is described, for example, in the following publications: brenneretal, "Automated Microscopefor cytologicalResearch," J.Histochem.Cytocchem., 24: 100-; which is hereby incorporated by reference in its entirety.

The brenner score is a measure of texture in the image. The in-focus image has a high brenner score and has a texture with a smaller scale than the out-of-focus image. In contrast, out-of-focus images have low brenner scores and have textures that are not smaller in scale than in-focus images. The variation of the brenner score with focal length can be represented using a brenner function drawn experimentally by taking several images at different focal lengths and drawing their brenner scores as a function of focal length. The brenner function has a peak at the ideal focal length, decreasing as the focal length varies in either direction relative to the ideal focal length. Thus, in general, the brenner function starts from a low value when an image is acquired at a focal distance lower than the ideal focal distance, reaches a peak when an image is acquired at the ideal focal distance, and then decreases as the focal distance increases above the ideal focal distance.

Curve 310 in fig. 3 depicts a brenner function showing the bell-shaped nature of the function. The focus point was taken as the reference point (0 μm) and identified by the peak of the bell curve. In this example, the Brenner function is symmetric about the focal point, decreasing to almost zero at a point about 10 μm from the focal point along either direction.

The brenner score may be calculated by measuring the average gray level variation between pairs of points separated by a given number of pixels. For example, the brenner score (B) for a given image may be calculated using the following equation:

where R and C are the number of rows and columns of pixels in the image, and P_i,jIs the value of the pixel in row i and column j. The value of n can then be chosen experimentally, for example depending on the resolution and magnification of the optical system, and the pixel size of the camera. For example, n may be the minimum distance that the optical system can resolve.

Any image acquired by the sensor contains some noise in the pixel values, for example, due to variations in the number of photons striking the pixel site, or noise of electronics associated with the sensor. Such noise can cause two pixels in a given image to differ in value even though the two pixels represent the same intensity level of the image. The impact of such noise may be reduced by thresholding the terms considered in calculating the summation of the brenner scores. For example, the squared value between two adjacent pixels is added to the sum of the brenner scores only if the difference is above a predetermined threshold. An example of such a thresholding process is described in the following pseudo-code:

wherein B represents the Brenner score, P_i,jThe pixel values of the pixels representing the ith row and the jth column, R and C represent the row and column numbers, respectively, and T represents the threshold. As described in the pseudo-code above, only when the difference in intensity values between two pixels separated by n pixels exceeds a threshold T, is the difference squared and added to the most current value of the brenner score. Thus, the brenner score for a given image is calculated by summing such differences over the entire image.

Estimation of ideal focal length

Referring again to fig. 2, the operations further include estimating and storing an ideal focal distance for a given location based on the calculated focus scores (step 230). In some embodiments, a function derived from the brenner score is used to estimate the ideal focal distance from the calculated focus score. In addition, the difference between the logarithms of the brenner scores can be used to estimate the ideal focal length.

The logarithm of the brenner score may be represented as a parabola. An example of such a parabola is shown by curve 320 depicted in fig. 3. Estimating the ideal focal distance from the given set of focus scores may include fitting a parabola of the focus scores with a parabola as a function of the focal distance. Since the general shape of the parabola is known, and using the curve 320 in fig. 3, for example, the parabola can be mathematically represented by the following equation:

y=Ax²+Bx+C（4）

where Y = f (X) = log (b (X)) represents the logarithm of the brenner score, and A, B and C represent constants that govern the shape and position of the parabola on the X-Y plane. Therefore, three equations are needed to solve A, B and C. The peak value y of the parabola can be adjusted_max(or the values at the vertices) are preprogrammed into the curve fitting algorithm. The peak value may be determined from experimental data and provides a first equation solving for parameters A, B and C. Two other equations are implemented using the logarithm of the brenner scores computed for two different images. The parameters A, B and C can be determined from three equations to give a complete description of the parabola. The focal length corresponding to the peak position of the parabola is then used as an estimate of the ideal focal length.

Other ways of fitting a parabola to the calculated focus scores are within the scope of the present description. For example, one of parameters A, B and C may be fixed based on historical or experimental data. Equation (4) can thus be simplified to have two unknown parameters that can be solved using the calculated brenner scores of two different images. In the case where three images are required for a given illumination color, rather than two, three equations may be implemented from the respective calculated focus scores, and the parameters A, B and C may be solved from these equations. In some applications this may be more accurate than an estimate based on only two points, but requires additional images to be acquired and processed. Other focus score functions that may be suitable for use in the equation may also be used to estimate the ideal focal length without departing from the scope of the present disclosure.

With respect to the embodiment utilizing the brenner score, the difference between the logarithms of the brenner scores of a pair of images taken with a fixed focus offset from each other is a linear function (in the example shown in fig. 3, curve 330), the value of which is substantially proportional to the deviation or offset from the ideal focal length represented by the peak position of the brenner function. Thus, the difference in focal length per unit of the logarithm of the brenner score (Δ f) is assumed to be known to calculate the estimated ideal focal length offset for the image obtained at a given illumination. Such an assumption may be derived from, for example, the slope of a linear curve such as curve 330 shown in FIG. 3. The offset may then be calculated as follows:

where Δ f is the difference in focal length per unit of difference for the logarithm of the brenner score, and is the Z-spacing between the image pairs. The estimated shift amount of the pile of images can be calculated as an average of the shift amounts of all four colors in general. The ideal focal length for the site is stored after adding the average offset to the representative focal length used to acquire the stack. It should be noted that whether the calculated offset is added to or subtracted from the representative focal distance depends on the sign convention followed in the particular application.

Once the ideal focal distance is estimated for a given location, the estimated ideal focal distance is stored in a storage location (e.g., in a database) associated with the relative location on the substrate. The stored values can then be used in calculating a representative focal distance for an image pile at a subsequent imaging location, as described below.

Two or more images of the same location or object may be used to evaluate a focus score that is curve fit to estimate the ideal focal distance. For example, multiple images may be acquired at different focal lengths and the focus scores from the multiple images fitted with a bell curve equation. The peak position of the bell-shaped curve can be interpolated from the focus scores. The image and corresponding focus scores may be collected at a relatively large focus offset (e.g., 4 μm), but the ideal focal distance may be estimated at a much finer resolution.

Estimation of representative focal distance at new locations

These operations may include verifying that the imaging system needs to acquire additional images at other locations on a given substrate (step 240), and then going to the imaging process. If no further sites remain, the imaging process may be terminated for a given subject (step 250) and restarted for a new subject, e.g., a different slide containing a different biological sample. Conversely, if there are more sites to image, the imaging hardware may move to a new imaging site (step 260). The site to be imaged may be preprogrammed into a control module that controls the imaging hardware.

The operations further include estimating an initial representative focal distance at the new location (step 270). The representative focal distance may be estimated or calculated as a weighted average of known estimated ideal focal distances for other locations. The known estimated ideal focal distance may be assigned a weight based on, for example, a weighting function determined based on the distance of the previous imaging location relative to the new imaging location. An example of a representative focus estimate is illustrated in fig. 4. In the figure, several locations on the imaging plane 405 are represented as (x, y) coordinate pairs, and the corresponding estimated ideal focal distances at these locations are represented using z values. For example, the ideal focal length at location (5, 3) is estimated to be z =3 in this example. Similarly, the ideal focal lengths at locations (1, 8), (5, 8), (8, 4), and (8, 3) are estimated as z = -1, z = -3, z = -2, and z =1, respectively. The estimated ideal focal distance is then used to estimate a representative focal distance at the new imaging location represented by the point (6, 6).

In general, the weighted average assigns a higher weight to the estimated ideal focal distance for the closest location to the most current imaging location and a lower weight to the focal distance for imaging locations far from the current imaging location. The weighting function may be linear or a function of one or more parameters. In some embodiments, the weight assigned to an estimated (or known) ideal focal length at a given location is calculated as follows:

weight=(d²+n²)^-(s/2)（6）

where the parameter d is the distance of a given location from the most current location, the parameter n representing the neighborhood controls the horizontal scale of the weighting function, and the parameter s representing the sharpness controls the shape of the curve corresponding to the weighting function. In the present example, the (x, y) distance is calculated in millimeters. Thus, when assuming a neighborhood of 1000 μm (i.e., n =1000 μm) and a sharpness of 4 (i.e., s =4), the weights are calculated in this example as follows:

weight=(d²+1)^(-2)（7）

the calculation of the representative focal length in the above example of the new imaging location (6, 6) is illustrated in table 1:

table 1: an example of calculating a representative focal distance is shown

Sum of all: 0.055-0.079

Weighted average of z: -1.451 μm

As illustrated above, the representative focal length at locations (6, 6) was estimated to be-1.451 microns. In general, the representative focal distance is different from the true focal distance or estimated ideal focal distance of the previous imaging location. For example, an estimate of the ideal focal length at locations (6, 6) can be demonstrated to be 1 micron. The estimated ideal focal distance is stored for a given location, rather than a representative focal distance, and used in estimating a representative focal distance at one or more new imaging locations. By cumulatively averaging the previous estimates of ideal focal length from multiple locations, the error in the estimates can be smoothed, resulting in progressively better estimates of focal length at the new imaging locations.

A weighted average of the estimated ideal focal distance that estimates the focal distance at a given location may also result in a progressively more accurate estimate of the slope of the fitted parabola at the given location. The slope of the parabola generally depends on one or more parameters of the imaging field. For example, in the case of imaging a slide containing a sample of stained blood cells, the slope may depend on the darkness and density of the cells.

Referring to fig. 5, a graph 500 shows an example of a curve f (x, n, s) illustrating how the shape of the weighting function f (x) varies with the parameters n and s according to equation (6). For example, curve 510 represents a weighting function of n =1000 μm and s = 2. Curve 520 represents a weighting function for n =1554 μm and s = 4. Curve 530 represents the weighting function for n =1961 μm and s =6, and curve 540 represents the weighting function for n =2299 μm and s = 8. In some implementations, the appropriate weighting function is determined experimentally by manipulating the parameters n and s. Other weighting functions with or without additional or fewer parameters are also within the scope of the present disclosure.

Overview of computing System

FIG. 6 is a schematic diagram of a computer system 600 that may be used to control the operations described in connection with any of the computer-implemented methods described herein, according to one implementation. The system 600 includes a processor 610, a memory 620, a storage device 630, and an input/output device 640. Each of the components 610, 620, 630, and 640 are interconnected using a system bus 650. The processor 610 is capable of processing instructions for execution within the system 600. In one implementation, the processor 610 is a single-threaded processor. In another implementation, the processor 610 is a multi-threaded processor. The processor 610 is capable of processing instructions stored in the memory 620 or on the storage device 630 to display graphical information for a user interface on the input/output device 640.

Memory 620 stores information within system 600. In some implementations, the memory 620 is a computer-readable medium. The memory 620 may include volatile memory and/or nonvolatile memory.

The storage device 630 is capable of providing mass storage for the system 600. In general, the storage device 630 may include a non-transitory tangible medium configured to store computer readable instructions. In one implementation, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive.

Input/output device 640 provides input/output operations for system 600. In some implementations, the input/output devices 640 include a keyboard and/or pointing device. In some implementations, the input/output device 640 includes a display unit that displays a graphical user interface.

The described features may be implemented in digital electronic circuitry, or in computer hardware, firmware, or in combinations of them. The features can be implemented in a computer program product tangibly embodied in an information carrier, e.g., in a machine-readable storage device, for execution by a programmable processor, the features being executable by the programmable processor executing program instructions to perform functions of the described implementations by operating on input data and generating output. The described features can be implemented in one or more computer programs that are executable on a programmable system including at least one programmable processor configured to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program comprises a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program may be abbreviated in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

Various software architectures may be used to implement the methods and systems described in the present application. For example, a publish/subscribe messaging schema may be used in implementing the methods and systems described herein. In the case of publish/subscribe messaging, the system includes several hardware and software modules that communicate only via the messaging module. Each module may be configured to perform a particular function. For example, the system may include one or more of a hardware module, a camera module, and a focusing module. The hardware module may send commands to imaging hardware that implement fast auto-focus, which in turn triggers the camera to acquire images.

The camera module may receive images from the camera and determine parameters such as shutter time or focus. The images may also be buffered in the memory of the computer before being processed by the camera module. The camera module may also send a message to interrupt the hardware module when an image is seen that is sufficient to determine the appropriate shutter time or focus when an initial search is made for the tilt of the slide.

The system may also include a focusing module, which may be implemented in software, hardware, or a combination of software and hardware. In some implementations, the focus module examines all frames in a stack and estimates how far the stack is from an ideal or ideal focal distance. The focus module may also be responsible for assigning a focus score to each frame in a stack of images.

Processors suitable for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one of multiple processors of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. A computer includes a processor that executes instructions and one or more memories that store instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and an optical disc. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user and a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer. Alternatively, the computer may have no keyboard, mouse, or monitor attached thereto, and may be remotely controlled by another computer.

The features can be implemented in a computing system that includes a back-end component, such as a data server, that includes a middleware component, such as an application server or an Internet server, that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include, for example, a LAN (local area network), a WAN (wide area network), and computers and networks forming the internet.

The computing system may include clients and servers. A client and server are generally remote from each other, and typically interact through a network such as the one described. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

The processor 61 executes instructions related to a computer program. The processor 610 may include hardware such as logic gates, adders, multipliers and counters. The processor 610 may further include a separate Arithmetic Logic Unit (ALU) that performs arithmetic and logical operations.

Examples of the present invention

The following example describes the application of the present invention to imaging a biological sample containing blood cells using the system 100 described in connection with fig. 1. The following example, continuing with the example described above in connection with fig. 4, is for illustrative purposes only and is not limiting upon the scope of the invention as described in the claims.

An automated sample preparation system, for example, an embodiment of the system disclosed in co-pending U.S. application No. 12/430,885, prepares a biological sample for imaging by depositing a thin monolayer of blood cells on a slide 130. Thereafter, embodiments of the system disclosed in co-pending U.S. application No. 12/943,687 fix, stain, rinse, and/or dry specimens on the slide 130. This process involves staining the cells with methylene blue and azure B. However, the methods disclosed in this document and in the examples below may also be used when preparing an imaging specimen using a romano-based stain or other stains and/or dyes. An automated transport mechanism (not shown) of the system 100 then loads the slide 130 onto the first motorized stage 120.

Prior to imaging the cells deposited on the slide 130, the system 100 determines the degree of tilt to which the slide 130 is loaded onto the stage 120, as described further below. The computer 190 measures the focus scores at three different (x, y) locations on the slide 130 by issuing commands to move the stage 120 in the x and y directions and to move the objective lens 140 in the z direction using the second motorized stage 150. At each of the three locations, the computer 190 causes the imaging hardware to capture multiple images of the slide 130 at different focal lengths and searches for a peak in the focus score using the brenner focus function. These three different (x, y) locations on the slide 130 are labeled a (0, 0), B (0, 10), and C (10, 0) in fig. 4. The focal lengths corresponding to the peak focus scores at each of the three locations are used to provide a preliminary estimate of the focal plane of the slide 130. Computer 190 determines that the peak focus score for each of locations a, B, and C corresponds to the following stage coordinates listed in table 2:

table 2: an example of calculating a representative focal distance is shown

Location of a site	X	Y	Z
				A	0	0	0.5
B	0	10	-0.5
				C	10	0	2.5

When using stage coordinates for each peak focus score location on the slide 130, the computer 190 calculates the focal plane of the slide 130 as represented by the following equation:

z=0.2x-0.1y+0.5（8）

after the focal plane is calculated for the slide 130, the system 100 initiates a series of image capture steps for the slide 130 at several (x, y) locations containing cells of interest. At the first new (x, y) location (5, 3) and for the other four (x, y) locations on the slide 130 listed above in table 1, the camera 160 acquires an 8-image stack of cells of interest. Specifically, the camera acquires one image at each of the four illumination colors (i.e., 635, 598, 525, and 415 nm) at a first focal length corresponding to the focal plane calculated for the slide 130. The motorized stage 150 then changes the focal length by 0.7 microns, allowing the camera 160 to take four additional images, one for each of the four illumination colors, at the same (x, y) location (5, 3). As described in connection with equation 5, computer 190 uses the ideal focal distance offset calculation to estimate the ideal focal distance for each of locations (5, 3), (8, 4), (5, 8), and (1, 8) as shown in Table 1.

On the next site to be imaged (6, 6), computer 190 calculates a representative focal distance of-1.451 microns using the weights of equation (7) for the site (6, 6) calculations based on the estimated ideal focal distances corresponding to the five previously imaged sites on slide 130 as shown in table 1. In contrast, the z-height at location (6, 6) calculated using equation 8 is z = 1.1.

Computer 190 then determines from the focal plane z value of 1.1 and a representative focal distance to the focal plane of-1.451 that the z stage coordinate at the center of the 8-image stack at (x, y) stage coordinates (6, 6) is-0.351, i.e., z =1.1+ (-1.451) = -0.351 microns. Computer 190 instructs imaging hardware 105 to acquire two sets of four images at sites (6, 6). The imaging hardware acquires two sets of four images under different color illumination at ± 0.35 microns at first and second z locations relative to the center of the imaging stack. With respect to the focal plane, at the location (6, 6), z = -1.101 for the first set of four images and z = -1.801 for the second set of four images.

When using the focus scores from the eight image piles acquired at locations (6, 6), computer 190 calculates an estimated ideal focal distance for the location as described in the table below.

Table 3: example of ideal focal length estimation for an 8 image pile

Averaging: 0.429 micron

The difference in focal length per unit of difference for the logarithm of the brenner score was taken as 2 microns. The focal distances of the image pairs for each color are 0.7 microns apart and computer 190 calculates an estimated ideal focal distance offset using equation 5. Thus, computer 190 determines that the focal length of the acquired stack at location (1, 1) is 0.429 microns below the ideal focal length as shown in Table 3. Computer 190 then applies the offset value to the representative focal distance, deriving an estimated ideal focal distance for location (6, 6) as follows: 0.429+ (-1.451) =1.022 microns. This estimated ideal focal distance is then added to table 1 for use when the computer 190 calculates a representative focal distance for the next (x, y) location on the slide 130 to be imaged.

Computer 190 then proceeds to calculate a representative focal distance and estimate an ideal focal distance for each new (x, y) location imaged after location (6, 6). The system 100 does not complete this process until after the system 100 has acquired image piles for all cells of interest on the slide 130. Upon receiving a stack of images from a new (x, y) location, computer 190 analyzes the images and updates the model of the focal plane by the time imaging hardware 105 presents the next cell of interest to camera 160. As the system images additional locations along the slide 130, the model makes the focal length estimate more and more accurate while accounting for local variations on the surface of the slide 130. This allows images to be acquired as quickly as using simple plane fitting, but improves focus measurements.

Other embodiments

The invention described herein may be implemented in a number of ways. Some useful implementations are described above. The description of these implementations is not intended to be limited to the detailed implementations described herein, but is to be used in a broader sense to describe the invention in the claims. It is to be understood that while the invention has been described in conjunction with the detailed description thereof, the foregoing description is intended to illustrate and not limit the scope of the invention, which is defined by the scope of the appended claims. The methods and systems described herein may be used to achieve fast autofocus in other imaging systems, for example, in various medical imaging applications and fast focus photography. Any imaging system that requires fast on-line estimation of focal length is within the scope of the present application. Other aspects, advantages, and modifications are within the scope of the following claims.

Claims (17)

1. A method of automatically focusing a plurality of images of one or more objects on a substrate, the method comprising:

a) obtaining, by a processor, a representative focal distance for a first location on a substrate from a set of focal distances at known locations on the substrate, wherein the representative focal distance is estimated as a weighted average of at least two focal distances from the set of focal distances at known locations, wherein a weight for a given focal distance is calculated from a distance of the corresponding location from the first location;

b) acquiring, by an image acquisition device, a set of at least two images of a first location, wherein each image is acquired using a different focal length offset from a representative focal length;

c) estimating, by the processor, an ideal focal distance corresponding to the first location based on the comparison of the focus quality of each image; and

d) the estimated ideal focal distance and the first location are stored in the set of focal distances at known locations.

2. The method of claim 1, further comprising:

retrieving, by the processor, the set of focal distances at the known location;

acquiring, by the image acquisition device, a further set of at least two images of the second location, wherein each image in the further set is acquired using a different focal length offset from a second representative focal length of the second location calculated from the set of known focal lengths at the known location; and

an ideal focal distance corresponding to the second location is estimated by the processor based on the comparison of the focus quality of each image in the further set.

3. The method of claim 1, wherein comparing focus quality further comprises calculating a focus score for each image by quantifying differences between adjacent pixels.

4. The method of claim 3, wherein the focus score is a Brenner focus score.

5. The method of claim 3, wherein estimating an ideal focal length further comprises:

calculating a difference between logarithms of focus scores of the images;

estimating an offset from the calculated difference; and

and estimating the ideal focal distance according to the offset.

6. The method of claim 1, wherein at least two images are acquired under illumination by substantially the same color of light, wherein the color of light is at least one of green, yellow, blue, and red.

7. The method of claim 1, wherein the set of images is acquired under illumination of at least two different colors, and at least two images are acquired under illumination of each color.

8. The method of claim 7, wherein estimating the ideal focal distance corresponding to the first location further comprises:

calculating a focus offset for each different color;

determining an average focus offset based on the calculated focus offsets for the different colors; and

an ideal focal distance corresponding to the first location is estimated based on the average focal distance offset.

9. An imaging system, comprising:

an image acquisition device configured to acquire a set of at least two images of a first location on the substrate using a different focal distance for each image offset from the representative focal distance; and

a processor coupled to the image acquisition device, the processor configured to:

calculating a representative focal length for the first location from a set of focal lengths at known locations on the substrate, wherein the representative focal length is estimated as a weighted average of at least two focal lengths from the set of focal lengths at the known locations, wherein a weight for a given focal length is calculated from a distance of the corresponding location from the first location;

estimating an ideal focal distance corresponding to the first location based on the comparison of the focus quality of each image;

storing the estimated ideal focal distance and the first location in the set of focal distances at known locations; and

the calculated representative focal distance is provided to the image acquisition device.

10. The system of claim 9, wherein:

the processor is further configured to retrieve the set of focal distances at known locations; and

the image acquisition device is further configured to acquire a further set of at least two images of the second location, wherein each image in the further set is acquired using a different focal length offset from a second representative focal length of the second location calculated from the set of known focal lengths at known locations.

11. The system of claim 10, wherein the processor is further configured to estimate an ideal focal distance corresponding to the second location based on a comparison of the quality of focus of each image in the additional set.

12. The system of claim 9, wherein comparing focus quality further comprises calculating a focus score for each image by quantifying differences between adjacent pixels.

13. The system of claim 12, wherein the focus score is a brenner focus score.

14. The system of claim 12, wherein the processor is further configured to:

calculating a difference between logarithms of focus scores of the images;

estimating an offset from the calculated difference; and

and estimating the ideal focal distance according to the offset.

15. The system of claim 9, wherein at least two images are acquired under illumination by substantially the same color of light, wherein the color of light is at least one of green, yellow, blue, and red.

16. The system of claim 9, wherein the set of images is acquired under illumination of at least two different colors, and at least two images are acquired under illumination of each color.

17. The system of claim 16, wherein the processor is further configured to:

calculating a focus offset for each different color;

determining an average focus offset based on the calculated focus offsets for the different colors; and

an ideal focal distance corresponding to the first location is estimated based on the average focal distance offset.