TW201816723A - Recovery of pixel resolution in scanning imaging - Google Patents

Abstract

This invention describes a technique to enhance pixel resolution of high-speed laser scanning imaging by the means of sub-pixel sampling, applicable to one-dimensional (1D), two-dimensional (2D), and three-dimensional (3D) imaging.

Description

Pixel resolution recovery in scanning imaging

The field of the invention relates to methods of data acquisition in imaging systems.

The present invention is a pixel super-resolution (pixel SR) technique driven by the dilemma of high sample rate operation, which enhances high speed laser scanning imaging at lower sampling rates (eg, time stretch imaging or free space corners). Enhanced pixel delay (FACED) imaging (ie anti-aliasing), the technology is easily supported by any commercial grade digitizer. This technique is based on the general concept of a pixel SR in which high resolution (HR) image information can be recovered from a plurality of sub-pixel shifted low resolution (LR) images captured at a lower sampling rate. Unlike the prior art of pixel SR technology, the present invention takes advantage of the fact that a continuous line scan (naturally generated) is generated due to a mismatch between the laser scanning repetition frequency and the sampling frequency of the back end digitizer. Pixel shifting, which is a feature that occurs in all types of laser scanning imaging modes. Therefore, this technique does not require active synchronization control of illumination or detection for precise sub-pixel shift operations at ultra-fast rates. Unlike any classical pixel SR imaging technique, any additional hardware not required by the present invention controls sub-pixel shifting motion (eg, detector translation, illumination beam steering) or for uncontrolled due to highly accurate and reconfigurable pixel drift. Motion complex image pixel registration algorithm. The present invention can be beneficial for any laser scanning imaging applied to ultra-fast or/and high-flux imaging applications without sacrificing spatial resolution at high-speed scanning rates, ranging from industrial manufacturing (eg, for network inspection). Surface detection and quality control in machine vision, semiconductor VLSI chip manufacturing, non-contact metrology, single cell analysis and clinical diagnosis in basic life sciences (in biomedical and environmental research) (eg cell-based assays and organization) Microarray (TMA), Full Slice Imaging (WSI).

For ultra-fast imaging (at line scan rates beyond sub-MHz), the pixel resolution of high-speed laser scanning imaging is enhanced or restored, even at lower sampling rates without affecting spatial resolution. General-Pixel SR technology, in which the mismatch between the laser scan repetition frequency and the digitizer's sampling frequency naturally produces sub-pixel shifts of continuous line scans (during imaging) for 1D, 2D, and 3D laser scanning imaging strategies . Full passive pixel SR technology, does not require any additional hardware for controlling sub-pixel shift motion or complex image pixel registration algorithms for uncontrolled motion. Highly accurate and reconfigurable pixel shifting and a unique pixel registration algorithm.

The present invention is a pixel SR technique that enhances pixel resolution (ie, anti-aliasing) of high speed laser scanning imaging at lower sampling rates, such as time stretch imaging or free space angle 啁啾 enhanced delay (FACED) imaging. Technology is easily supported by any commercial grade digitizer. It can be applied to the 1D laser scan 1 strategy, 2D laser scan 5 strategy and 3D laser scan 6 strategy (Figure 1). In one embodiment, the time interleaving measurements are made by sampling clock shifts due to the fact that the digitizer sampling clock is not locked according to the laser scanning frequency (eg, pulsed laser source 7 or typically a scanning element). By utilizing this effect at a lower sampling rate, the present invention is capable of extracting multiple LR images with high temporal precision for each LR image (eg, tens of picoseconds in time stretch imaging or FACED imaging) Automatic subpixel shifting. In one embodiment, a 1D line scan 1 is performed on a unidirectional motion of a sample 3 of biological cells, such as in microfluidic flow 2. The 2D image 11 is reconstructed by the line scan 1 captured by the digital stack such that the fast axis 4 of the resulting 2D image 11 is the line scan direction and the slow axis 2 corresponds to the sample motion direction. In one embodiment, the 2D line scan 5 is performed by scanning the line scan beam along the slow axis 8, while the sample 3 is in a fixed position or at a slower motion than the line scan speed along the slow axis 8. The 2D image 11 is reconstructed by a line scan 5 captured by the digital stack. In one embodiment, the 3D line scan 6 is performed by scanning the line scan beam in a 2D manner, i.e., along the slow axis 6 and the axial axis 8. The sample 3 is in a fixed position or is moving at a slower speed than the line scan speed along the slow axis 6 and the axial axis 8. For the sake of demonstration, we consider the most common form of laser scanning imaging of 1D line scan 1 imaging. Its wide range of applications from flow cytometry to surface inspection has been demonstrated by instant line scan imaging of specimen 3 (Figure 2). In one embodiment of imaging with a 1D line scan 1, the imaged sample 3 is a biological cell in the microfluidic stream 2. In this case, the pixel resolution along the fast axis 4 is a linear flow rate. And laser pulse repetition rate Product of . On the other hand, the pixel resolution along the slow axis 2 is independently determined by the resolution and scanning speed of the imaging setting, ie ,among them, Is the scanning speed of the system (or basically a space-to-time conversion factor); Is the sampling rate of the digitizer. When operating at a low sampling rate, laser scanning imaging of ultrafast stream 2 produces elongated pixels 9 (Fig. 2). For example, we found that it was defined as The aspect ratio of the original LR pixel 9 is as small as typical ultra-fast laser scanning imaging configuration operations (such as time stretch imaging or FACED imaging) beyond MHz Order of magnitude. Ideally, if the digitizer's sampling clock frequency Locked to laser pulse repetition rate , the line scan will be perfectly aligned along the slow axis. In practice, the average number of pixels per line scan ( ) is not an integer. Line scan 1 appears to "drift" along slow axis 2, and thus 2D image 11 appears to be highly warped, especially at low sampling rates (Figure 2). Specifically, because of the sampling rate No repetition rate based on laser pulse Locked, so the pixel drift between stretch line scans 1 is observed in adjacent time and can be expressed as Where integer The number of pixels per line scan is rounded to the nearest integer. As can be seen . Therefore, the warp angle is given as ,as shown in picture 2. A common and straightforward method of de-warping an image is to realign the 2D image 11 (Fig. 2). However, as shown later, at lower sample rates, this will result in particularly severe image aliasing and artifacts (Fig. 2). Furthermore, digital up-sampling does not add additional image information and therefore does not provide an improvement in resolution along the fast axis. The present invention utilizes the warping effect to create a relative "sub-pixel shift" on both the fast axis 4 and the slow axis 2, thereby restoring the high resolution 2D image 12 (Fig. 2). The invention is unique in that the image warping registration method. We first register the exact warp angle of mesh 13 (Figure 2). This can be done by using the intentional non-uniform illumination background of the line scan 1 as a reference (eg, the laser spectrum in time stretch imaging, or changing/modulating the illumination intensity during scanning). The accuracy of the measured warp angle severely affects the performance of the pixel SR algorithm (Figure 3). Specifically, it is necessary to compensate for the warpage for accurate extraction of the non-uniform illumination distribution introduced in the system, which is estimated to be: among them, Is the number of line scans of the warped image, function Is the angle The image on the go warp filter. Note that in the case of 3D laser scanning, the warpage angle of the plane along the axial and slow axes (i.e., 4 and 8 in Fig. 1) should also be evaluated. Ideally, a "clean" foreground can be obtained by subtracting the background directly from the warped image 11. Warp angle The error of the value causes distortion in the estimated background, resulting in a banding artifact superimposed on the foreground specimen 3. However, by maximizing the "cleanness" of the extracted foreground, ie by minimizing the energy of the foreground, this property can be used to obtain accurate Value, expressed as Where integer Is the number of pixels scanned per line. Figure 3 depicts a graphical representation of image warpage registration. After this step, the non-uniform illumination background can be easily suppressed by subtracting the high bandwidth 1D reference illumination signal background from the intermediate "de-warped" image, which in turn passes through the interlaced head LR line scan 1 (Fig. 3) is restored. The 1D interleaving operation is based on a fast shift superposition algorithm along with a rational approximation. Finally, the image is denoised and resampled into the conventional high resolution grid 13 to reveal high resolution information. A complete step of pixel SR laser scanning imaging is depicted in Figure 3, which depicts a method used in time stretch imaging. The same workflow can be applied to any other laser scanning imaging mode. Note that interpolating adjacent line scans 1 effectively amplifies the pixel size along the slow axis and reduces the effective imaging line scan rate. As shown in Figure 2, the size of the interpolated pixel along the warp direction is given as When we consider the ratio of pixel size reduction, given In this demonstration, the resolution is improved for highly elongated pixels (Equation (1) and Figure 2) and significant warping ( )especially important. In both cases, the ultra-fast laser scan rate can be used (ie High repetition rate is achieved. In addition, the magnified pixel size of the slow axis after interleaving still far exceeds the optical diffraction limit, which takes advantage of the ultra-fast scan rate. As previously stated, the present invention is applicable to any laser scanning imaging, and to demonstrate the principle, we present herein a pixel SR for ultra-fast laser scanning time stretch imaging with improved spatial resolution. We chose a type of phytoplankton, scenedesmus 3 (American Carolina Bio) because of its unique morphological properties. In the experiment, a single genus To The ultra-fast linear flow rate is loaded into channel 8. Then by having versus A time-stretched waveform is digitized by a real-time oscilloscope with an adjustable sample rate. At the highest possible sampling rate ( Under the cell image, the cell image has a clear outline and visible intracellular content (Fig. 4, second column). However, at a lower sampling rate Next, the pixel size changes to . Since the diffraction limited resolution is estimated to be approximately The image of the cells captured at such a low sampling rate becomes highly aliased (Fig. 4, third column). In contrast, the present invention can achieve a 10-fold resolution improvement in principle. In our current setup, the actual resolution improvement is roughly limited to 4x, ie in terms of its pixel size and its effective sample rate. Next (the fourth column in Figure 4). This is not inherent to the pixel SR algorithm. In contrast, the restored image resolution is currently subject to the built-in signal conditioning filter at the cutoff frequency in the oscilloscope. The limitation of this means that additional sub-pixel measurements do not provide a 10x improvement in spatial resolution. With HR image restoration, ultra-fast pixel SR laser scanning imaging (such as time stretch imaging) is particularly useful for achieving morphological-based label-free, high-throughput cell sorting and analysis, as in standard flow cytometry The next is impossible. Here we are through our Down-sampling) optical flow control pixel SR time stretch imaging system to image the genus genus 3 ( Subtypes perform classification. In high performance clustering, images of a single colony are reconstructed by a pixel SR algorithm. We first retrieve two unmarked metrics for a single cell from the recovered pixel SR frame: opacity and area. These spatial average metrics represent the optical density (attenuation) and the size of the phylum 3 of the genus. Cell images were automatically divided into three groups by K-means clustering. Based on the scatter plot of these two parameters (Fig. 4), the fragments can be easily distinguished from living cells because they are significantly smaller and more transparent (see also the image in Figure 4). However, these spatial averaging measures (or essentially LR metrics) do not take into account the subtle morphological differences between the two sub-sets and the four sub-sets, which exhibit high variability in area and opacity. Next, new morphological metrics are adaptively extracted from the set of pixel SR images and plotted from the cell regions in the scatter plot (Fig. 5). We encode the morphological features of each image as a histogram (HoG) of the directional gradient and then project it into the most important component using principal component analysis (PCA). Essentially, this metric provides a measure of the complexity of the cell body structure of the genus Quercus. Using this new morphological metric instead of the opacity metric, we were able to separate the clusters of the two sub-sets from the four sub-sets (Figure 5), demonstrating the pixel SR strategy in automatic classification based on high-throughput imaging. importance. As mentioned earlier, the practical advantage of the pixel SR for high speed laser scanning is that it relaxes the very high sampling rate ( Or higher), this can only be provided by the most advanced and expensive oscilloscopes. Ultra-fast data acquisition using such high-end oscilloscopes often presents problems with limited memory buffers. It not only hinders continuous real-time real-time data storage, but also affects high-throughput post-processing and analysis. The present invention provides an efficient method to address this limitation by capturing ultra-fast laser scanned images at lower sampling rates without affecting image resolution. More importantly, unlike previous high-end oscilloscopes, the lower sample rate digitizer can be easily equipped with an FPGA that can stream a large amount of time stretched image raw data continuously and reconfigurably to distributed computer storage. cluster. To demonstrate the applicability of pixel SR to such high-throughput data processing platforms, we generate water-in-water emulsion droplets in microfluidic channel devices 8 (up to speed) ) Perform continuous real-time monitoring. Take The sampling rate continuously records the time stretched image signal (Figure 6). Similar to previous experiments using an oscilloscope, we also observed that the original time stretched image was highly warped due to the unlocked clock between the laser and the digitizer (Figure 6B-C). It should be reminded that the method often used is to warp the image by aligning each line scan. As shown in Figure 6C, each line scan 1 is digitally up-sampling at least 8 times to realign. Although this strategy is used to stretch the signal over time oversampled over bandwidth, it does not perform well at low sample rates due to signal aliasing (Figure 6C). Another approach is to interleave multiple line scans to resolve high bandwidth 1D time waveforms (Fig. 6D-E). This is often referred to as equivalent time sampling. However, the fusion of multiple line scans results in a reduction in pixel resolution along the slow axis, which also introduces image aliasing (Fig. 6D). In contrast, our pixel SR algorithm is capable of recovering HR time stretched images at any point in time. The resolution improvement is 5 times the apparent pixel size, ie in terms of its effective sampling rate and its pixel resolution (Fig. 6E-F). As previously mentioned, asynchronous sampling of ultrafast image signals results in relative sub-pixel alignment of 1D line scan 1, which can be accurately determined by our pixel registration algorithm. The invention can also be applied to a synchronous digitizer that locks to an ultra-fast line scan rate such that sub-pixel alignment can be accurately predetermined in a reconfigurable manner. This can be done by using a phase-locked loop (PLL) to digitize the sampler clock to a fractional ratio. Locked to the laser pulse trigger to achieve. For example, if the digitizer clock is in ratio Locked to the pulsed laser, the apparent number of pixels per line scan is . However, the first The relative sub-pixel alignment of the line scan can be Is accurately determined to provide five times the number of apparent pixels per line scan (ie ). Show different in lock phase The cumulative sub-pixel position of the value. This ratio can be precisely adjusted in hardware (eg, by a PLL) to control the density of sub-pixel locations. It is worth noting that when genlocking is achieved, the energy that can cross the foreground is minimized (Equation 4). It is therefore to be understood that the intent of the present invention, as well as the <RTIgt;</RTI><RTIgt;</RTI><RTIgt;</RTI><RTIgt; In the present invention, the description is intended to be illustrative and not restrictive. It is also to be understood that the appended claims are intended to cover all such claims and claims

1‧‧1D laser scanning

2‧‧‧microfluidic flow

3‧‧‧ sample

4‧‧‧ fast axis

5‧‧‧2D laser scanning

6‧‧‧3D laser scanning

7‧‧‧pulse laser source

8‧‧‧ slow axis

9‧‧‧ pixels

11‧‧‧2D image

12‧‧‧2D image

13‧‧‧Grid

1 shows a schematic diagram of 1D, 2D and 3D laser scanning strategies; FIG. 2 shows a schematic diagram of real-time line scanning imaging of a specimen 3 according to an embodiment of the invention; FIG. 3 depicts an image warping according to an embodiment of the invention. Figure 4 shows a schematic diagram of temporal stretching images and their recovery at different sampling rates of an oscilloscope according to an embodiment of the invention; Figure 5 shows automatic classification based on high throughput imaging An embodiment of mid-pixel SR imaging. 6 shows a schematic diagram of an improvement in pixel resolution in accordance with an embodiment of the present invention, in which a pixel SR algorithm recovers an HR time stretched image; FIG. 7 illustrates a different phase locked loop ratio P according to an embodiment of the present invention. Schematic diagram of the first six sub-pixel samples under /Q.

Claims (13)

A method of enhancing pixel resolution of high speed laser scanning imaging for enhancing pixel resolution of high speed laser scanning imaging by means of sub-pixel sampling, the method being suitable for one-dimensional (1D), two-dimensional (1D) 2D) and three-dimensional (3D) imaging. The method of claim 1, wherein the pixel resolution of the high-speed laser scanning imaging is enhanced by means of sub-pixel sampling, including by means of a loss between the repetition frequency of the high-speed laser scanning imaging and the sampling rate of the digitizer A sub-pixel shift with a continuous line scan that is naturally generated during imaging, recovering high resolution (HR) images from low-resolution (LR) images shifted by multiple sub-pixels captured at a lower sampling rate information. The method of claim 1 or 2, wherein the high speed laser scanning imaging comprises performing a line scan imaging of the sample, wherein the instant line scanning refers to the relative motion between the sample and the laser line scanning beam. The method of claim 3, wherein performing a line scan imaging of the sample comprises applying a 1D line scan to the one-way moving sample and reconstructing the 2D image by digitally capturing the captured 1D line scan, and wherein The fast axis of the resulting 2D image corresponds to the line scan direction, and the slow axis corresponds to the sample motion direction. The method of claim 3, wherein performing a line scan imaging of the sample comprises applying a 2D line scan to the sample by scanning the line scan beam along a slow axis, and digitally stacking the captured 1D A line scan is used to reconstruct the 2D image, and the fast axis of the resulting 2D image corresponds to the line scan direction, and the slow axis corresponds to the line scan beam motion direction. The method of claim 3, wherein performing a line scan imaging of the sample comprises applying a 3D line scan to the sample by scanning the line scan beam along a slow axis and an axial axis, and by digitally stacking along A 1D line scan captured by the slow axis and the axial axis is used to reconstruct the 3D image, and the fast axis of the resulting 3D image corresponds to the line scan direction, and the slow axis and the axial axis correspond to the direction of the line scan beam motion. According to the method of claim 4, the method further comprises using a warping effect of the 2D image or the 3D image to create a relative sub-pixel shift on the fast axis, the slow axis and the axial axis, thereby recovering the high resolution. 2D or 3D image, where when the digitizer's sample rate Laser pulse repetition rate not based on high speed laser scanning imaging When locked, pixel drift between adjacent line scans causes the warping effect. The method according to any one of claims 5 to 7, wherein the warping effect of the obtained 2D image or 3D image is used to create relative sub-pixel shifts on the fast axis and the slow axis by the following steps. Implementation: Introducing a predetermined non-uniform illumination background of a 1D line scan as a reference for compensating for warpage of 2D images or 3D images; aligning warp angles in high resolution grids of multiple captured line scan images , including: with a given warp angle De-warping filter de-wars the image; obtains accurate values by maximizing the "cleanliness" of the extracted foreground, ie minimizing foreground energy or banding artifacts ; subtract 1D reference illumination background from the warped image; restore the 2D or 3D image by interlacing the LR line scan image; and denoise the recovered 2D or 3D image and resample it to the high resolution grid , thus revealing high-resolution information. The method of claim 8, wherein the predetermined non-uniform illumination background of the 1D line scan comprises a laser spectrum in time stretch imaging or a change/modulation of the illumination intensity during the scan. According to the method of claim 9, wherein the accurate energy is obtained by minimizing the energy of the foreground Value, expressed as: Where integer Is the number of pixels scanned per line, function Is the angle The image on the warp filter, and It is the estimated non-uniform illumination distribution introduced in the system. The method of claim 2, wherein the lower sampling rate of the digitizer is as low as one million samples per second and the laser scanning repetition frequency of the high speed laser scanning imaging . The method of claim 1, wherein enhancing the pixel resolution of the high speed laser scanning imaging by means of sub-pixel sampling comprises using a phase locked loop (PLL) to digitize the digitizer sampling clock at a fractional ratio Lock to the laser pulse trigger. A system for enhancing pixel resolution of high-speed laser scanning imaging, which enhances pixel resolution of high-speed laser scanning imaging by means of sub-pixel sampling, the method being applicable to one-dimensional (1D), two-dimensional (2D) and three-dimensional (3D) imaging, the system comprising: a high speed laser scanning imager; a digitizer for capturing low resolution (LR) images at a lower sampling rate; and a processor and a memory, wherein the memory stores a computer program, The program is executed by the processor to implement the method of any of claims 1-12.