GB2596099A

GB2596099A - Image sensors

Info

Publication number: GB2596099A
Application number: GB2009253.2A
Authority: GB
Inventors: Lindner Scott; Stoppa David
Original assignee: Ams Sensors Asia Pte Ltd
Current assignee: Ams Sensors Asia Pte Ltd
Priority date: 2020-06-17
Filing date: 2020-06-17
Publication date: 2021-12-22
Also published as: WO2021256990A1; GB202009253D0

Abstract

An image sensor 1 comprising a detector unit 3 a, b, c, d comprising a plurality of single photon detectors; a processing circuit 5 configured to receive signals from the plurality of single photon detectors, to determine a time of arrival (ToA) of each signal, and to code shift the ToA of signals from different detectors by different amounts; and a memory configured to store the time shifted ToA of each signal in a histogram. The imaging sensor preferably includes an emitter 7, the emitter preferably being a pulsed laser. Each detector of the plural detectors preferably has a position address and the processing circuit comprises an address determination unit to determine the position address of the detector from which the signal has been received. The processing circuit also preferably configured to determine the amount of cade shift based on the position address of the detector from which the signal originated.

Description

Image Sensors

Field

The invention relates to image sensors and in particular to 3D image sensors.

Background

Image sensors including arrays of single photon avalanche diodes (SPADs) can be used for direct time-of-flight ranging applications to image objects in 3D. An emitter illuminates the object with pulsed illumination and the detectors measure the reflected light. The time of arrival (ToA) of individual photons at the detectors, relative to a trigger synchronised with the emitter, is measured. A histogram of photon counts over time can be generated for each detector. Each histogram represents the intensity of the reflected light over the emitter illumination time period, which can be used to recover the 3D geometry of the object.

A SPAD is a single photon detector, in which a photon-generated carrier can trigger a short-duration but relatively large avalanche current. SPADs are able to distinguish the arrival times of photons with a timing uncertainty (jitter) of a few tens of picoseconds.

Summary

In existing image sensors, there is a 110-1 correspondence of histogram memory to spatial sample in the field-of-view, which can result in a prohibitively large silicon area occupation, particularly for sensors with large formats, e.g. 320 x 240 pixels. For such sensors, 1-to-1 correspondence can be achieved by time-multiplexing multiple spatial samples to a single memory. However, this can negatively impact sensor performance in reduced range and/or lower framerate.

According to a first aspect of the invention there is provided an image sensor comprising a detector unit comprising a plurality of single photon detectors; a processing circuit configured to receive signals from the plurality of single photon detectors, to determine a time of arrival (ToA) of each signal, and to code shift the ToA of signals from different detectors by different amounts; and a memory configured to store the code shifted ToA of each signal in a histogram.

This configuration may be particularly suited to a system where the ranging performance is limited by the light reflected from the target, as opposed to limited by the background light present in the environment. An example of such a system could involve a receiver array imaging a scene illuminated in a scanning fashion, e.g. by a segmented VCSEL array or mirror. Each pixel captures events only for the period where the corresponding region in the field-of-view is illuminated and this period is short enough that the background light is negligible in the acquired histogram. In such a case, the increased integration time in comparison to a time-multiplexing configuration, in which each detector only acquires events for a fraction of the total integration time, results in a greater number of collected photons from the target for each detector, and thus increased detection range.

By storing/accumulating the ToAs from multiple detectors in a single histogram the amount of required memory space and dedicated chip area can be reduced. The histogram can then be decoded by an image processing unit to generate an image. The signal from a detector is generally a transition in voltage level (e.g. from 0 to 1V, or any other common positive voltage such as 3.3 or 5V) generated by a front-end circuit at the time of an event (i.e. a trigger of the single photon detector such as an avalanche current in a SPAD). Typically, a large number of signals from each detector is accumulated in an acquisition and the corresponding ToAs added to corresponding bins in the histogram. Curve fitting (e.g. polynomial fitting) can be used together with calibration data to perform a distance measurement on the histogram. Alternatively, the weighted mean of histogram bins may be used to perform the distance measurement.

The processing unit can be configured to receive further signals from the plurality of single photon detectors, and to determine the ToA of each further signal. The memory can then be configured to store the ToA of each further signal (having zero code shift) in a reference histogram. That is, two acquisition may be performed with the detector unit. The ToAs of signals from the first acquisition are code shifted, while the ToAs of further signals from the second acquisition are not code shifted (e.g. by applying a zero code shift to each of the further signals). The processing circuit may be configured to overwrite the histogram stored in the memory with the reference histogram. Once the histogram from the first acquisition has been retrieved, for example by an image processing unit, the memory can be overwritten with the reference histogram from the second acquisition. Alternatively, the reference histogram may be generated in parallel, instead of in a subsequent acquisition. In this case, two histograms are required to be stored in memory.

During one acquisition the code-shift unit can update the histogram stored in the memory by incrementing the appropriate bins of the histogram (i.e. the bins corresponding to the code-shifted ToAs). The code shift is typically a time shift, i.e. a shifting in time of the apparent time of arrival indicated by the signal.

The memory can be one of a flip-flop based ripple counter and a static random access memory (SRAM). Alternatively, the memory may be an analogue memory, which can save space on the sensor chip compared to a digital memory.

The image sensor may further comprise an image processing unit configured to decode the histogram in order to perform a distance measurement and to generate an image.

The image processing unit may perform curve fitting on the histogram and associate peaks in the histogram with the specific detectors from which the signals originated. The image processing unit may take the weighted mean of the bins of the histogram(s). To achieve this, the image processing unit can be configured to retrieve the histogram and the reference histogram from the memory and to use the reference histogram to decode the histogram in order to determine the peaks associated with each single photon detector.

The image processing unit may also be configured to determine when the histogram is empty or flat (no statistically significant peaks), and in response to such determination not perform a distance measurement. For example, if there is no object to reflect light from the emitter, then only the ToAs of background light and dark counts will be accumulated in the histogram, which would generally produce a flat profile. In such an event, no meaningful distance measurement can be performed.

The processing circuit may comprise a front-end circuit for each detector of the plurality of single photon detectors configured to receive and forward the signal from the detector. That is, the front-end circuits can provide a channel from the single photon detectors to other components of the processing circuit. The processing circuit may also comprises a ToA determination unit configured to determine the ToA of each signal, and the ToA determination unit may comprise a time to digital converter (TDC). The TDC can be configured to output, for each single photon detector, a bit code comprising a number of "1"s, which indicate the ToAs of an equal number of signals from that detector. That is, the ToA of an event is the position of the corresponding "1" in the bit code. The bit code may comprise multiple "1"s representing the ToAs of multiple events at one detector. The bit code can be a 16-bit code. For example, a 16-bit code may cover 16 ns of time and the single photon detector (e.g. a SPAD) may have a dead time of 2 ns, in which case the bit code may comprise up to eight ToAs (e.g. eight "1"s in the bit code) representing eight separate events. Each ToA in the bit code can be added to a corresponding bin of the histogram in the memory. The time covered by the bit code (16 ns in this example) may be equal to the period of the emitter (e.g. the time between subsequent laser pulses). The acquisition time, i.e. the time during which ToAs are added to the histogram, may be in the order of microseconds. The output from the TDC may also comprise a plurality of binary codes, where each binary code represents the ToA from a single photon detection. For example, each binary code may be a 4 bit binary code.

Each detector of the plurality of single photon detectors may have a position address. The processing circuit can be configured to, for each signal, determine said amount of code shift based on said position address of the detector from which the signal originated. For example, the processing circuit can comprise an address determination unit configured to, for each signal, determine the position address of the detector from which the signal originated. The position address is typically a bit code numbering the single photon detector from 0 to N -1, where N is the number of single photon detectors in the plurality of single photon detectors. For example, if the detector unit comprises a 2 x 2 array of detectors, then the detectors could be numbered 0, 1, 2 and 3 respectively in bit code. The address determination unit may be configured to determine the position address associated with a signal by identifying the front-end circuit from which the signal was received.

The processing circuit may further comprise a code-shift unit configured to receive the ToA of each signal from the ToA determination unit and to receive the position address associated with each signal from the address determination unit. Then, for each signal, the processing unit is configured to use the position address associated with the signal to code shift the ToA of the signal and to send the code shifted ToA of each signal to the memory. Typically, the code-shift unit is configured to use a 1 to 1 mapping between the position address and the amount of code shift applied to the ToA of a signal originating from the detector having that position address. The code shift can be specified as a number of bits to shift the ToAs in the bit code provided by the TDC. This corresponds to a shift of an equal number of bins in the histogram in which a ToA is counted. In general, the number of bins in the histogram may be equal to the number of bits in the bit code from the TDC (i.e. the range of the TDC). Each bin may store, for example, 12 bits of data, in order that a sufficiently large number of ToAs can be counted. Circular wrapping can be used, so that a "1" shifted n steps past the last bit of the bit code is moved to the n-th bit in the bit code. The code-shift unit can be configured to apply a zero code-shift to each further signal (i.e. to apply a zero code shift to signals from the second acquisition used in the reference histogram). Alternatively, the ToAs of the further signals may bypass the code shift unit and be stored directly in the memory in the reference histogram.

The code-shift unit may be a first code-shift unit, wherein the processing circuit comprises a second code-shift unit configured to receive the ToA of each signal from the ToA determination unit and receive the position address associated with each signal from the address determination unit. Then, for each signal, the second code-shift unit is configured to use the position address associated with the signal to apply a code shift to the ToA of the signal, wherein at least one code shift applied by the second code-shift unit to one of the signals is different from the code-shift applied to that signal by the first code-shift unit. The image sensor comprises a second memory configured to store the code shifted ToA of each signal from the second code-shift unit in a second histogram. In this case, only a single acquisition is required, which enables the full frame to be utilised for capturing light with the image sensor. In comparison, when two separate acquisitions are made, only half the frame may be used. The image processing unit may be configured to use the first and second histograms to decode the ToAs, and thereby generate an image.

Typically, the plurality of single photon detectors comprises an array of single photon avalanche diodes (SPADs). Alternatively, the plurality of single photon detectors may comprise an array of superconducting nanowire single photon detectors. The array may be one of a 2 x 1, 2 x 2, 3 x 1, 3 x 2, 3 x 3, 4 x 1, 4 x 2, 4 x 3 and 4 x 4 array. The detector unit is generally a sub-array, which together with further sub-arrays make up the full array of detectors of the image sensor. Typically the image sensor comprises a plurality of detector units and a plurality of processing circuits, wherein each detector unit of the plurality of detector units is connected to a respective processing unit of the plurality of processing units. For example, the image sensor may comprise 32 x 32 single photon detectors (1024 pixels) divided into 8 x 8 detector units each comprising a 4 x 4 array of single photon detectors.

The image sensor may comprise a 3D stacked structure in which the detector unit is located on a first die and at least a part of the processing circuit is located on a second die, and wherein one of the first and second die is flipped and bonded to the other die so as to connect the detector unit to the at least a part of the processing circuit. Such a structure may increase component density and reduce the overall size of the image sensor compared to other designs. For example, the ToA determination unit, the address determination and the code-shift unit of the processing circuit may be located on the second die, whereas the front-end circuits of the single photon detectors may be located on the first die together with the detector unit.

The image sensor may comprise an emitter for illuminating an object (i.e. the object to be imaged with the image sensor). The emitter may comprises a pulsed laser, which may provide short (e.g. 100 ps width) laser pulses. The emitter is synchronised with a reference trigger signal, which can be used to set a starting point for the ToA measurements. The ToA of each signal is generally measured relative to the reference trigger. For example, a single trigger source may be used to trigger both the emitter and the ToA measurement with an electrical signal. In some applications it may be preferable to detect the optical output from the laser and use this detection to trigger the ToA measurement According to a second aspect of the present invention, there is provided a mobile phone (e.g. a smartphone) comprising an image sensor according to the first aspect of the invention.

According to a third aspect of the present invention there is provided a method of imaging an object in 3D using an image sensor according to the first aspect of the present invention. The method comprises illuminating the object and receiving with the detector unit light reflected from the object.

According to a fourth aspect of the invention there is provided a method of imaging an object in 3D, comprising detecting with a detector unit comprising a plurality of single photon detectors photons reflected from the object; receiving with a processing circuit signals from the plurality of single photon detectors; determining with the processing unit a time of arrival (ToA) of each signal; and code shifting with the processing unit the ToA of signals from different detectors by different amounts. The method further comprises storing in a memory the code shifted ToA of each signal in a histogram.

Brief Description of Drawings

Figure 1 is a schematic diagram of an image sensor according to an embodiment; Figure 2 is a schematic diagram of a mobile phone comprising an image sensor according to an embodiment; Figure 3 is a schematic diagram of a part of an image sensor according to an embodiment: Figure 4 illustrates two histograms which may be generated by an image sensor according to an embodiment; Figure 5 is a schematic diagram of a part of an image sensor according to another embodiment in which different code shifts are applied to the ToA of the same signals to generate two different histograms stored in two memories; and Figure 6 illustrates different histograms generated by applying different code shifts to the ToA of the same signals.

Detailed description

Figure 1 shows a schematic diagram of an image sensor 1 according to an embodiment. The image sensor comprises a 3D stacked structure having a first die 2 comprising detector units 3a, 3b, 3c and 3d, each detector unit comprising a plurality of single photon detectors (not shown), and a second die 4 comprising a processing circuit 5. The first and second die are bonded by a dense matrix of bonds 6 being low Ohmic connections (e.g. hybrid bonds) so that the detector units 3a to 3d are connected to the processing circuit 5. The image sensor also comprises an emitter 7 being a pulsed laser. The emitter 7 is shown illuminating an object 8, which scatters the light. The scattered light 9 is detected by the detector units 3a to 3d. Each detector unit 3a to 3d sends a plurality of signals (e.g. one for each SPAD in the detector unit) to the processing circuit 5. The processing circuit 5 is configured to determine the time of arrival (ToA) relative to a reference trigger signal synchronised with the emitter 7. The processing circuit 5 then shifts the ToA of the signals circularly by different amounts, wherein signals from one of the single photon detectors in a detector unit may have a zero code shift. The code shifted ToAs of the signals are saved in a histogram in a memory 10. That is, each code shifted ToA from one detector unit increments a corresponding bin of a histogram in memory 10. An image processing unit 11 can retrieve the histogram from the memory 10, and from the histogram generate an image of the object 8.

Figure 2 shows a schematic diagram of a mobile phone 12 comprising an image sensor 1 such as the image sensor 1 illustrated in Figure 1.

Figure 3 shows a schematic diagram of a part of an image sensor, which may be an image sensor as illustrated in Figure 1, comprising a detector unit 3, a processing circuit 5 and a memory 10. The detector unit 3 comprises a 2 x 2 array of single photon detectors 13a, 13b, 13c and 13d being single photon avalanche diodes (SPADs), having position addresses A, B, C and D respectively. The processing circuit 5 comprises a front end-circuit 14 for each SPAD directly connected to the SPAD for routing signals from the SPAD to other components of the processing circuit 5. The processing circuit further comprises a ToA determination unit 15, being a time to digital converter TDC, for determining the ToA of each signal. In the illustrated case, the detector unit detects photons scattered from a flat region, such that the ToA of each signal is the same. The TDC outputs a 16 bit code 16 comprising a '1', which indicates the ToA of the signals. The processing unit 5 has an address determination unit 17, being a pixel address decoder and latch, for determining the position address associated with each signal (i.e. the position address belonging to the SPAD from which the signal originated). The processing circuit 5 also comprises a code-shift unit 18 for applying a circular code-shift to the ToA of signals received from the TDC based on the position address received from the pixel address decoder and latch. In the illustrated case, the code-shift unit applies a 1 to 1 mapping between the amount of shifting and the position address, as shown in Table 1. The code-shift unit 18 updates a single histogram with the code-shifted ToAs from all four SPADs in the memory 10, which stores the histogram. The embodiment allows the memory allocation for a single histogram to be shared by multiple detectors.

To determine the correspondence between the SPADs and peaks in the histogram, a second acquisition can be performed, where the code shift is 0 for all SPADs. With the knowledge of the single peak position with zero code shift, and the code shift applied to each SPAD, the peak-SPAD correspondence can be determined.

Table 1

Position address Code shift (bits) A 0 B 4 C 8 D 12 Figure 4 shows a first histogram 20 with code shifted ToA peaks 21a, 21b, 21c and 21d, which may be the histogram stored in the memory illustrated in Figure 3, and a second histogram 22 (which can be referred to as a reference histogram) with no or zero code shift applied to the ToAs. The histograms 20 and 22 comprise 16 bins numbered from 0 to 15, which corresponds to the number of bits in the code indicating the ToA from the ToA determination unit. The histograms 20 and 21 record the number of counts/frequency of signals from a detector unit having a particular ToA, i.e. the number of events at that time. In this case, an approximately equal number of photons was received by four single photon detectors of the detector unit. The second histogram can be used to decode the first histogram to determine which counts (i.e. which peaks) correspond to which detector.

Figure 5 is a schematic diagram of a part of an image sensor according to another embodiment. Similar or corresponding features to those of the image sensor in Figure 3 have been given the same reference numerals for clarity, and are not intended to limit the features of the illustrated embodiments. The part of the image sensor comprises a detector unit 3, a processing circuit 5 memories 10a and 10b. The memories may be two separate memory locations 10a and 10b in the same memory. The detector unit 3 comprises a 2 x 2 array of single photon detectors 13a, 13b, 13c and 13d being single photon avalanche diodes (SPADs), having position addresses A, B, C and D respectively. The processing circuit 5 comprises a front end-circuit 14 for each SPAD directly connected to the SPAD for routing signals from the SPAD to other components of the processing circuit 5. The processing circuit further comprises a ToA determination unit 15, being a time to digital converter TDC, for determining the ToA of each signal. The TDC outputs a 16 bit code indicating the ToA of each signal. The processing unit 5 has an address determination unit 17, being a pixel address decoder and latch, for determining the position address associated with each signal. The processing circuit 5 also comprises two code-shift units 18a and 18b for applying different circular code-shifts to the ToA of signals received from the TDC based on the position address received from the pixel address decoder and latch. In the example given, the circular shifts are A = 0 (0), B = 4 (3), C = 8 (6), D = 12 (9) for the first code-shift unit 18a and the second code-shift unit 18b respectively. Table 2 shows the mapping of the position address to the applied code shift for both circular shifters. The shifted codes are then used to increment the corresponding histogram bins in the memories 10a and 10b, which stores the two histograms from respective code-shift units 18a and 18b. The unique code difference between peaks from the same SPAD in the two memories 10a and 10b allows the histograms to be decoded by determining the correspondence between specific SPADs and peaks in the histograms.

Table 2

Position address Code shift A (bits) Code shift B (bits) A 0 0 B 4 3 C 8 6 D 12 9 Figure 6 shows a reference histogram 22 from ToAs from four single photon detectors with zero code-shift, giving rise to a single peak 23 (e.g. imaging a flat surface/region), and two histograms 24 and 25 where the ToAs have been code shifted in different ways based on the position addresses of the single photon detectors. The histograms 24 and 25 have four separate peaks, each separated from its neighbouring peaks. The two code-shifted histograms can be stored in different memories during the same acquisition, which enables the whole frame to be used for sampling. The two code-shifted histograms can then be used to determine the peak to detector correspondence.

Although the invention has been described in terms of preferred embodiments as set out above, these embodiments are illustrative only and the claims are not limited to those embodiments. Those skilled in the art will be able to make modifications and alternatives in view of the disclosure which fall within the scope of the claims. Each feature disclosed or illustrated in the present specification may be incorporated in the invention, whether alone or in any appropriate combination with any other feature disclosed or illustrated herein.

Claims

CLAIMS: 1. An image sensor comprising: a detector unit comprising a plurality of single photon detectors; a processing circuit configured to receive signals from the plurality of single photon detectors, to determine a time of arrival (ToA) of each signal, and to code shift the ToA of signals from different detectors by different amounts; and a memory configured to store the time shifted ToA of each signal in a histogram.
2. An image sensor according to claim 1, wherein each detector of the plurality of single photon detectors has a position address and the processing circuit comprises an address determination unit configured to, for each signal, determine the position address of the detector from which the signal originated.
3. An image sensor according to claim 2, wherein the processing circuit is configured to, for each signal, determine said amount of code shift based on said position address of the detector from which the signal originated.
4. An image sensor according to claim 1, 2 or 3, wherein the processing unit is configured to receive further signals from the plurality of single photon detectors, and to determine the ToA of each further signal; and the memory is configured to store the ToA of each further signal with zero code shift in a reference histogram.
5. An image sensor according to claim 4, wherein the processing circuit is configured to overwrite the histogram stored in the memory with the reference histogram.
6. An image sensor according to any one of the preceding claims, wherein the memory is one of a flip-flop based ripple counter and a static random access memory (SRAM)
7. An image sensor according to any one of claims 1 to 5, wherein the memory is an analogue memory.
8. An image sensor according to any one of the preceding claims, wherein the code shift is a time shift.
9. An image sensor according to any one of the preceding claims, and further comprising an image processing unit configured to decode the histogram in order to determine a distance measurement associated with each single photon detector in order to generate an image.
10. An image sensor according to any one of the preceding claims, wherein the processing circuit comprises a front-end circuit for each detector of the plurality of single photon detectors configured to receive and forward the signal from the detector.
11. An image sensor according to any one of the preceding claims, wherein the processing circuit comprises a ToA determination unit configured to determine the ToA of each signal.
12. An image sensor according to claim 11, wherein the ToA determination unit comprises a time to digital converter (TDC).
13. An image sensor according to claim 12, wherein the TDC is configured to output, for each single photon detector, a bit code comprising a number of "1"s, which indicate the ToAs of an equal number of signals from that detector.
14. An image sensor according to claim 13, wherein the bit code is a 16-bit code.
15. An image sensor according to claim 12, wherein the TDC is configured to output, for each single photon detector, a binary code.
16. An image sensor according to any one of the preceding claims, wherein the processing circuit comprises a code-shift unit configured to: receive the ToA of each signal from the ToA determination unit; receive the position address associated with each signal from the address determination unit; for each signal, use the position address associated with the signal to code shift the ToA of the signal; and send the code shifted ToA of each signal to the memory.
17. An image sensor according to claim 16, wherein the code-shift unit is configured to apply a zero code-shift to each further signal.
18. An image sensor according to claim 16 or 17, wherein the code-shift unit is a first code-shift unit and the processing circuit comprises a second code-shift unit configured to: receive the ToA of each signal from the ToA determination unit; receive the position address associated with each signal from the address determination unit; for each signal, use the position address associated with the signal to apply a code shift to the ToA of the signal, wherein at least one code shift applied by the second code-shift unit to one of the signals is different from the time-shift applied to that signal by the first code-shift unit; and and wherein the image sensor comprises a second memory configured to store the code shifted ToA of each signal from the second code-shift unit in a second histogram.
19. An image sensor according to any one of the preceding claims, wherein the plurality of single photon detectors comprises an array of single photon avalanche diodes (SPADs).
20. An image sensor according to any one of claims 1 to 18, wherein the plurality of single photon detectors comprises an array of superconducting nanowire single photon detectors.
21. An image sensor according to claim 19 or 20, wherein the array is one of a 2 x 1, 2 x 2, 3 x 1, 3 x 2, 3 x 3, 4 x 1, 4 x 2, 4 x 3 and 4 x 4 array. 30
22. An image sensor according to any one of the preceding claims, wherein the image sensor comprises a 3D stacked structure in which the detector unit is located on a first die and at least a part of the processing circuit is located on a second die, and wherein one of the first and second die is flipped and bonded to the other die so as to connect the detector unit to the at least a part of the processing circuit.
23. An image sensor according to any one of the preceding claims, wherein the image sensor comprises a plurality of detector units and a plurality of processing circuits, and wherein each detector unit of the plurality of detector units is connected to a respective processing unit of the plurality of processing units.
24. An image sensor according to any one of the preceding claims, wherein the image sensor comprises an emitter for illuminating an object.
25. An image sensor according to claims 24, wherein the emitter comprises a pulsed laser.
26. A mobile phone comprising an image sensor according to any one of claims 1 to 25.
27. A method of imaging an object in 3D using an image sensor according to any one of claims 1 to 25, comprising: illuminating the object and receiving with the detector unit light reflected from the object
28. A method of imaging an object in 3D, comprising: detecting with a detector unit comprising a plurality of single photon detectors photons reflected from the object; receiving with a processing circuit signals from the plurality of single photon detectors; determining with the processing unit a time of arrival (ToA) of each signal; code shifting with the processing unit the ToA of signals from different detectors by different amounts; and storing in a memory the code shifted ToA of each signal in a histogram.