TWI801572B - Image sensor, imaging unit and method to generate a greyscale image - Google Patents

Abstract

An image sensor includes a time-resolving sensor and a processor. The time-resolving sensor outputs a first signal and a second signal pair in response detecting one or more photons that have been reflected from an object. A first ratio of a magnitude of the first signal to a sum of the magnitude of the first signal and a magnitude of the second signal is proportional to a time of flight of the one or more detected photons. A second ratio of the magnitude of the second signal to the sum of the magnitude of the first signal and the magnitude of the second signal is proportional to the time of flight of the one or more detected photons. The processor determines a surface reflectance of the object where the light pulse has been reflected based on the first signal and the second signal pair and may generate a grayscale image. An imaging unit and a method to generate a greyscale image are provided.

Description

Image sensor, imaging unit and method for generating grayscale images [Cross-reference to related applications]

This application claims priority to U.S. Provisional Application No. 62/702,891, filed July 24, 2018, the disclosure of which is incorporated herein by reference in its entirety.

The subject matter disclosed herein relates generally to image sensors. More specifically, the subject matter disclosed herein relates to a Time-of-Fight (TOF) image sensor that can also generate grayscale images from accumulated photon detection events.

Three-dimensional (3D) imaging systems are increasingly used in a variety of applications such as industrial production, video games, computer graphics, robotic surgery, consumer displays, surveillance video, 3D modeling, real estate sales wait. Existing 3D imaging technologies may, for example, include time-of-flight (TOF) based range imaging, stereo vision systems, and structured light (SL) methods.

In the TOF method, the distance to the 3D object is resolved based on the known speed of light—by measuring the light signal between the camera and the 3D object for each point in the image The round-trip time it takes to travel. TOF cameras can use a scanless approach to capture the entire scene with every laser pulse or light pulse. Some exemplary applications of TOF methods may include advanced automotive applications such as active pedestrian safety or pre-collision detection based on real-time range imagery; such as tracking human movement during interaction with a game on a video game console; object detection in industrial machine vision Categorize and explain where the robot found items (like items on a conveyor belt), etc.

In a stereo imaging system, or stereo vision system, two cameras horizontally displaced from each other are used to obtain two different views of the scene or of 3D objects in the scene. By comparing the two images, relative depth information of the 3D object can be obtained. Stereo vision is very important in fields such as robotics to extract information about the relative position of 3D objects in the vicinity of an autonomous system/robot. Other applications of robotics include object discrimination, where stereo depth information enables robotic systems to separate occluded image components that the robot might not otherwise be able to distinguish as two separate objects—such as one object being in front of another causing the other object to be partially or completely hidden. 3D stereoscopic displays are also used in entertainment systems and automation systems.

In the SL method, the projected light pattern and an imaging camera can be used to measure the 3D shape of an object. In the SL method, a known light pattern (usually a grid or horizontal strips or a pattern formed by parallel strips) is projected onto the scene or 3D objects in the scene. The projected pattern may be deformed or displaced when it hits the surface of the 3D object. This deformation enables the SL vision system to calculate the depth information and surface information of the object. Thus, projecting a narrow band of light onto a 3D surface can generate illumination lines that can appear distorted from perspectives other than that of the projector and can be used to geometrically reconstruct the shape of the illuminated surface. SL-based 3D imaging can be used for different applications, e.g. by police officers for taking fingerprints in 3D scenes, in-line inspection of components during production processes, in healthcare for on-site measurements of human body shape and/or microstructure of human skin.

technical problem

It is an object of the present invention to provide an apparatus and method for sensing light reflected from an object in a pulsed manner to generate both 3D and two-dimensional (2D) images.

Exemplary embodiments provide an image sensor that may include a time resolved sensor and a processor. The time-resolved sensor may include at least one pixel and may output a pair of a first signal and a second signal in response to detection by the at least one pixel of one or more photons corresponding to a light pulse projected toward the object. Two signals, the one or more photons are reflected from the object, wherein the amplitude of the first signal in the pair is proportional to the amplitude and the amplitude of the first signal in the pair The first ratio of the sum of the amplitudes of the second signals in a pair may be proportional to the detected time-of-flight of the one or more photons, and wherein the second signal in the pair A second ratio of the amplitude to the sum of the amplitude of the first signal of the pair and the amplitude of the second signal of the pair may be related to the detected The time-of-flight of the one or more photons is proportional. The processor can determine a reflectivity of a surface of the object reflecting the light pulse based on the pair of first and second signals. The processor can also determine a distance to the object based on the pair of first and second signals. In one embodiment, the time-resolved sensor is responsive to a plurality of light pulses projected towards the object at the detecting one or more photons reflected from the object at the pixel to output a plurality of pairs of first and second signals, wherein each pair of the first and second signals may correspond to a corresponding light pulse, and The processor can determine a reflectivity of a surface of the object reflecting the light pulse based on pairs of first and second signals.

Exemplary embodiments provide an imaging unit that may include a light source, a time-resolved sensor, and a processor. The light source may illuminate the object with a series of light pulses projected towards the surface of the object. The time-resolved sensor may include at least one pixel, may be synchronized with the light source and output a pair of first a signal and a second signal, the one or more photons being reflected from the surface of the object, wherein the amplitude of the first signal in the pair is proportional to the first signal in the pair A first ratio of the amplitude of the amplitude of the second signal in the pair to the sum of the amplitudes of the second signal in the pair may be proportional to the time-of-flight of the detected one or more photons, and wherein in the pair A second ratio of said amplitude of said second signal to said sum of said amplitude of said first signal in said pair and said amplitude of said second signal in said pair The time-of-flight of the one or more photons detected may be proportional. The processor can determine the distance to the object based on the pair of first and second signals and can determine the distance at which the light pulse is reflected based on the pair of first and second signals. The reflectivity of the surface of the object. In one embodiment, the time-resolved sensor may output pairs of first and second signals in response to detecting one or more photons reflected from the object at the pixel, where each A pair of the first signal and the second signal may correspond to a corresponding one of the plurality of light pulses projected toward the object. The processor may also determine a plurality of surface reflections of the object reflecting each corresponding pulse of light based on the corresponding pair of first and second signals. fire rate. In one embodiment, the processor may further generate a grayscale image of the object based on the plurality of surface reflectances.

Exemplary embodiments provide a method of generating a grayscale image of an object, wherein the method may include: projecting a series of light pulses from a light source toward a surface of the object; detecting one or more photons corresponding to the light pulses at a pixel, The one or more photons are reflected from the surface of the object; a pair of first and second signals is generated by the time-resolved sensor in response to detecting the one or more photons, wherein the The time-resolved sensor can be synchronized with the light source, wherein the amplitude of the first signal of the pair is related to the amplitude of the first signal of the pair and the amplitude of the first signal of the pair. The first ratio of the sum of the amplitudes of the second signals may be proportional to the time-of-flight of the one or more detected photons, and the amplitude of the second signals in the pair is proportional to the A second ratio of the sum of the amplitude of the first signal in the pair and the amplitude of the second signal in the pair may be related to the detected one or more photons the time of flight is proportional; the distance to the object is determined by the processor based on the pair of first and second signals; and the distance to the object is determined by the processor based on the pair of first and second signals. signal to determine a surface reflectance of the object reflecting the one or more photons. In one embodiment, the method may further comprise detecting, at the pixel, one or more photons reflected from the object for a plurality of light pulses projected toward the object, wherein each pair of first a signal and a second signal may correspond to one of the plurality of light pulses; and determining, by the processor, a position of the object reflecting the light pulse based on at least one pair of the first signal and the second signal Surface reflectivity. In one embodiment, the method may further comprise generating, by the processor, at least one histogram of arrival times of photons detected by a predetermined one of the plurality of pixels to generate the gray scale image .

Beneficial effect of the invention

According to the invention, the amount of transferred charge is converted into a first signal by projecting a light pulse towards the object to sense the light pulse reflected from said object. The remaining charge amount is converted into a second signal. The time-of-flight of the light pulse is calculated based on the first signal and the second signal, and the distance is calculated according to the time-of-flight to generate a 3D image. The power of the reflected light pulse is calculated based on the first signal and the second signal, and the reflectivity of the object is calculated according to the calculated power and the calculated distance to generate a 2D gray scale image of the object.

15: System/Imaging System/Time-of-Flight (TOF) System

17:Module/imaging module

19: Module/processor/processor module/host

20:Module/Memory/Memory Module/System Memory/Memory Unit

22:Light source/module/projector module/light source module

24:Module/image sensor unit

26: Objects/3D Objects

28: Pulse/projected pulse/light signal/light pulse/pulse light/optical field of view

29: Optical field of view

30, 31: Irradiation path

33:Laser/light source/laser light source/irradiation source/laser source

34:Laser controller

35: Projection optics/focusing lens

36, 38, 39: collection path

37: Reflected pulse/collection path/return light pulse/return pulse/received pulse/received light/return light

42: Image sensor/array/pixel array/two-dimensional (2D) pixel array

43, 601, 602, 603, 604, 621, 622, 623, 624, 700, 1000, 1202, 1300: pixels

44: Collection Optics / Focusing Lens

46:Pixel processing unit/pixel processing circuit

50, 1100, 2800: flow chart

52, 54, 56, 58, 60, 1101, 1102, 1103, 1004, 1105, 1106, 1107: Operation

62, 64: Angular motion

66: Scanning line SR

68: scan line SR+1

70, 72, 73: light spot

71: Spot/spot

75: column R/pixel column R

76: Column R+1

78: Irradiation

80: light spot

82: Line C _i

84: depth/distance

86: axis/X axis

275: peripheral storage unit

277: output device/display unit

278: Network Interface / Network Interface Unit

280: Power supply unit/onboard power supply unit

501:SPAD core/SPAD core part

502:PPD core/PPD core part

503, 1302, 1603, 2311a, 2311n: SPAD

504: the first control circuit

505: Incoming light

506: Output/SPAD output/digital SPAD output/SPAD exclusive digital output/signal/output signal

507: the second control circuit

508, 1801, 2101: PPD

510: Pixel exclusive analog output/pixel exclusive output data line/pixel output data line/pixel exclusive output/PIXOUT signal/PIXOUT data line/PIXOUT line /Pixout line

600A: architecture/2×2 pixel array architecture

600B: Architecture/Pixel Array Architecture/SPAD Shared Configuration

600C: pixel array architecture/4×4 pixel array architecture/pixel array configuration

605, 641, 1001, 1311: PPD core

606, 607, 608, 609, 625, 642, 643, 644, 645, 646, 647, 648, 649, 650, 1002, 1003, 1004, 1005: SPAD core

701: shutter / shutter signal / electronic shutter / electronic shutter signal

702, 1319: logic unit

703: transistor/first N-channel metal oxide semiconductor field effect transistor (NMOSFET)/first NMOS transistor/first transistor

704: transistor/second NMOS transistor/second transistor

705: transistor/third NMOS transistor/third transistor

706: The fourth NMOS transistor/the fourth transistor/source follower

707: Fifth NMOS transistor/fifth transistor

708: Transfer enable (TXEN) signal

709: Reset (RST) signal/RST pulse

710: transfer voltage (VTX) signal

711: TX signal/TX voltage

712: Floating Diffusion (FD) Node / Floating Diffusion Junction

713: VPIX signal/pixel voltage (VPIX) signal

715: Selection (SEL) signal

800, 900, 1400: timing diagram

801, 802: waveform

901: time delay / delay time T _dly / flight time T _tof duration

902: pixel-specific TOF value/time of flight T _tof

903: Time period/electronic shutter on or active period T _sh

904: Shutter ON cycle

905:PPD preset event

906: The first floating diffusion reset event

907: The second FD reset event

908, 909: Reference number

910, 1402: events

1006, 1007, 1008, 1009: box/F(x,y) box

1010, 1011, 1012: Pulse

1200: Image sensor unit

1201: pixel array/2D pixel array

1203: Column decoder/column driver/processing unit

1204: row decoder/processing unit

1205: pixel row unit/processing unit

1206, 1207, 1208: row-specific pixout signal/pixel receiving PIXOUT signal

1209, 1210, 1211: column-specific collection

1212: input/column address input/control input

1213: P1 and P2 values

1214: row address input/control input

1301A, 1301N: SPAD core/SPAD

1303:SPAD working voltage/VSPAD voltage/signal

1304: Resistive elements/resistors

1305: Capacitor/coupling capacitor

1306, 1316: Inverter

1307: Transistor/PMOS transistor

1308: Electronic shutter signal/shutter input/shutter/electronic shutter

1309: VDD/power supply voltage VDD

1310: Output/Output Line/SPAD Output/SPAD Core/SPAD Core Exclusive Output

1312:SPAD/Core Proprietary SPAD

1313: resistance element

1315: coupling capacitor

1317: PMOS transistor

1318: Output/SPAD output/SPAD core exclusive output

1320: transistor/first NMOS transistor/first transistor/NMOS transistor

1321: transistor/NMOS transistor/second NMOS transistor/TX transistor

1322: transistor/NMOS transistor/third NMOS transistor

1323: transistor/NMOS transistor/fourth NMOS transistor

1324: transistor/NMOS transistor/fifth NMOS transistor

1325: TXEN signal/TXEN input/internal input TXEN

1326:RST signal/external input RST signal

1327: VTX signal

1328:TX signal/TX waveform/TX input

1329:VPIX signal

1330: SEL signal

1331: floating diffusion (FD) node/floating diffusion junction/FD signal/floating diffusion voltage waveform

1333: Second TXEN signal/TXENB signal

1334: transistor/NMOS transistor/sixth NMOS transistor

1335: Ground (GND) potential

1336: Storage Diffusion (SD) Capacitor

1337: transistor/NMOS transistor/seventh NMOS transistor

1338: SD node

1339: The second transfer (TX2) signal

1401: transfer mode (TXRMD) signal

1403:PPD default event

1404: delay time T _dly

1405: TOF cycle T _tof

1406: Shutter off interval

1407: The shutter is turned on or the active cycle T _sh

1408: shutter on cycle/shutter on or active cycle T _sh

1409: FD reset event

1412: the first read cycle

1413: the second read cycle

1500, 2000: time-resolved sensors

1501, 2001, 2301a, 2301n: SPAD circuit

1503, 2003, 2303: logic circuits

1505: PPD circuit

1601, 2313a, 2313n: Resistors

1605: Capacitor

1607, 2317a, 2317n: p-type MOSFET transistor

1609, 2319a, 2319n: buffer

1701: Latches

1703: Double input OR gate

1803, 2103, 2351: the first transistor

1805, 2105, 2353: second transistor

1807, 2107, 2355: the third transistor

1809, 2109, 2357: the fourth transistor

1811, 2111, 2359: fifth transistor

1900, 2200, 2400: relative signal timing diagram

2005: The second PPD circuit

2113, 2361: the sixth transistor

2115, 2363: The seventh transistor

2117, 2365: Eighth transistor

2119, 2367: Ninth Transistor

2300: pixel/time-resolved sensor

2305: The third PPD circuit

2315a, 2315n: capacitor

2369: tenth transistor

2371: Eleventh Transistor

2373: Twelfth Transistor

2375: Thirteenth Transistor

2500: method

2501, 2502, 2503, 2504, 2505, 2506, 2507, 2508, 2509: steps

2600: trigger waveform

2601, 2602, 2603, 2700: bar chart

2602a: Window width

2701: Number of events M

2702: Number of events N

2801, 2802, 2803, 2804, 2805, 2806: steps

2900: scene

2901: Depth map

2902: grayscale image

a, b, c, d: output/input

C _i : row i/row

DE: Detection Event

d : offset distance

h : distance

PIXA, PIXB: pixel output line

PIXOUT: pixel-specific analog output/pixel-specific output

PIXOUT1: Pixel output 1/signal/voltage

PIXOUT2: Pixel output 2/signal/voltage

q : offset distance/offset/position/parameter

R, R+1: column

SC: capacitive device

S _R, S _R+1 : line/scanning line

T _dly : value/delay time parameter/delay time/parameter/delay/time delay period

T _sh : parameter/period/electronic shutter on or active cycle/shutter on cycle/shutter on or active cycle/electronic shutter time

T _tof : value/parameter/time of flight/TOF cycle

V _DD : voltage/power supply voltage/universal power supply voltage

VSPAD: voltage/signal/SPAD operating voltage

X: axis/direction/horizontal direction

X _R,i : light spot

x, y: input

Y: axis/direction/vertical direction

Z: axis/depth/distance

α, β : Angle

θ: parameter/scan angle/beam angle

In the following sections, various aspects of the subject matter disclosed herein are set forth with reference to exemplary embodiments that are illustrated in the drawings, in which:

FIG. 1 illustrates a highly simplified partial configuration of an image sensor system according to the subject matter disclosed herein.

FIG. 2 illustrates an exemplary operational configuration of the image sensor system of FIG. 1 in accordance with the subject matter disclosed herein.

3 illustrates a flowchart of an exemplary embodiment of how 3D depth measurement may be performed in accordance with the subject matter disclosed herein.

4 illustrates how an exemplary point scan may be performed for 3D depth measurement in accordance with the subject matter disclosed herein.

5 illustrates a block diagram of an exemplary embodiment of a pixel according to the subject matter disclosed herein.

6A to 6C illustrate three different examples of pixel array architectures according to the subject matter disclosed herein.

FIG. 7 illustrates the circuitry of an exemplary embodiment of a pixel according to the subject matter disclosed herein. detail.

8 is an exemplary timing diagram providing an overview of a modulated charge transfer mechanism in the pixel shown in FIG. 7 in accordance with the subject matter disclosed herein.

9 is an illustration of different signals in the image sensor system shown in FIGS. 1 and 2 when TOF values are measured using pixels in the embodiment shown in FIG. 7 in a pixel array according to the subject matter disclosed herein. Timing diagram of the sequence.

FIG. 10 illustrates how logic units may be implemented in a pixel in accordance with the subject matter disclosed herein.

11 depicts an exemplary flowchart illustrating how TOF values may be determined in the image sensor system shown in FIGS. 1 and 2 in accordance with the subject matter disclosed herein.

12 is an exemplary layout of a portion of an image sensor unit according to the subject matter disclosed herein.

FIG. 13 illustrates another exemplary embodiment of a pixel according to the subject matter disclosed herein.

14 is a graph of various signals in the image sensor system shown in FIGS. 1 and 2 when TOF values are measured using pixels in the embodiment depicted in FIG. Timing diagram for example timing.

15 illustrates a block diagram of an exemplary embodiment of a time-resolved sensor in accordance with the subject matter disclosed herein.

16 is a schematic diagram of an exemplary embodiment of a single-photon avalanche diode (SPAD) circuit of the time-resolved sensor shown in FIG. 15 according to the subject matter disclosed herein.

FIG. 17 is a schematic diagram of an exemplary embodiment of a logic circuit for the time-resolved sensor shown in FIG. 15 according to the subject matter disclosed herein.

FIG. 18 illustrates the time-resolved sensor of FIG. 15 in accordance with the subject matter disclosed herein A schematic diagram of an exemplary embodiment of a pinned photodiode (PPD) circuit.

FIG. 19 illustrates an exemplary relative signal timing diagram for the time-resolved sensor shown in FIG. 15 in accordance with the subject matter disclosed herein.

20 illustrates a block diagram of another exemplary embodiment of a time-resolved sensor according to the subject matter disclosed herein.

21 is a schematic diagram of an exemplary embodiment of a second PPD circuit for the time-resolved sensor shown in FIG. 20 according to the subject matter disclosed herein.

FIG. 22 illustrates an exemplary relative signal timing diagram for the time-resolved sensor shown in FIG. 20 in accordance with the subject matter disclosed herein.

23 illustrates a block diagram of yet another exemplary embodiment of a time-resolved sensor according to the subject matter disclosed herein.

FIG. 24 illustrates an exemplary relative signal timing diagram for the time-resolved sensor shown in FIG. 23 in accordance with the subject matter disclosed herein.

25 is a flowchart illustrating a method for resolving time using the time resolving sensor shown in FIG. 23 according to the subject matter disclosed herein.

FIG. 26A depicts an exemplary trigger waveform output from a SPAD.

FIG. 26B illustrates an exemplary histogram that may be formed by photon detection times for an exemplary pixel according to the subject matter disclosed herein.

26C depicts an exemplary bar graph indicating a window width representing the full width at half-maximum (FWHM) of a projected pulse (not shown), in which it can be determined, in accordance with the subject matter disclosed herein. Event count maximum.

26D depicts an exemplary histogram in accordance with the subject matter disclosed herein, wherein the trigger waveform output from the SPAD (FIG. 26A) is convolved with the histogram to determine events. Maximum piece count.

27 illustrates an exemplary histogram of exemplary pixels according to the subject matter disclosed herein.

28 illustrates a flowchart of an exemplary method of generating a depth or range map and a grayscale image of a scene in accordance with the subject matter disclosed herein.

Figure 29A depicts an exemplary scenario.

29B and 29C illustrate an exemplary depth map and an exemplary grayscale image, respectively, of the scene depicted in FIG. 29A according to the subject matter disclosed herein.

FIG. 30 illustrates an exemplary embodiment of a general layout of the imaging system depicted in FIGS. 1 and 2 in accordance with the subject matter disclosed herein.

In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the disclosure. However, it will be understood by those skilled in the art that the various disclosed aspects may be practiced without these specific details. In other instances, well-known methods, procedures, components and circuits have not been described in detail so as not to obscure the subject matter disclosed herein. Additionally, the various aspects set forth can be implemented to perform low-power 3D depth measurements in any imaging device or system, including but not limited to smartphones, User Equipment (UE), and/or laptops. desktop computer.

Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may include the embodiments disclosed herein. In at least one embodiment. Hence, the phrase "in a In one embodiment" or "in an embodiment" or "according to one embodiment" (or other phrases of similar meaning) may not all refer to the same implementation Example. Furthermore, in one or more embodiments, particular features, structures or characteristics may be combined in any suitable manner. In this regard, the word "exemplary" as used herein means "serving as an example, example or exemplary". Any embodiment described herein as "exemplary" should not be construed as necessarily preferred or advantageous over other embodiments. Additionally, in one or more embodiments, any suitable Specific features, structures, or characteristics are combined in a specific manner. In addition, depending on the context of the discussion herein, singular terms may include the corresponding plural forms and plural terms may include the corresponding singular forms. Similarly, hyphenated terms (such as , "two-dimensional", "pre-determined", "pixel-specific", etc.) can occasionally be compared with the corresponding unhyphenated version (e.g., "two-dimensional (two dimensional", "predetermined", "pixel specific", etc.) are used interchangeably, and capitalized terms (e.g., "Counter Clock", "Column selection ( Row Select), "PIXOUT", etc.) are used interchangeably with the corresponding non-capitalized versions (e.g., "counter clock", "row select", "pixout", etc.). Occasional interchangeable uses should not be considered inconsistent with each other.

In addition, singular terms may include corresponding plural forms and plural terms may include corresponding singular forms, depending on the context of the discussion herein. It should also be noted that the various figures shown and discussed herein, including elemental figures, are for illustrative purposes only and are not drawn to scale. Similarly, various waveforms and timing diagrams are shown for illustrative purposes only. For example, for clarity, relative to other elements The item exaggerates the size of some of the elements. Further, where appropriate, reference numerals have been repeated among the figures to indicate corresponding and/or analogous elements.

The terminology used herein is for the purpose of describing some exemplary embodiments only and is not intended to be limiting of the claimed subject matter. As used herein, the singular forms "a, an" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that when the word "comprises and/or comprising" is used in this specification, it indicates the presence of stated features, integers, steps, operations, elements and/or components, but does not exclude one or more The presence or addition of other features, integers, steps, operations, elements, elements and/or groups thereof. As used herein, the terms "first", "second", etc. are used as labels for nouns that follow the terms, and the terms do not imply any type of order unless explicitly defined (eg, spatial, temporal, logical, etc.). Furthermore, the same reference numerals may be used in two or more figures to refer to components, elements, blocks, circuits, units or modules having the same or similar functions. However, this usage is only for brevity of description and ease of discussion; the usage does not imply that construction or architectural details of such elements or units are the same in all embodiments or that these commonly mentioned The components/modules are the only way to implement some of the example embodiments disclosed herein.

It will be understood that when an element or layer is referred to as being on, "connected to," or "coupled to" another element or layer, the element can be directly on, directly connected to, or directly connected to the other element or layer. to or directly coupled to another element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element or layer, there are no intervening elements or layers present. Like numbers refer to like elements throughout pieces. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The terms "first", "second", etc. are used herein as labels for the nouns that follow the term, and unless explicitly defined otherwise, the term does not imply any type of order (e.g., spatial, temporal) , logical, etc.). Furthermore, the same reference numerals may be used in two or more figures to refer to components, elements, blocks, circuits, units or modules having the same or similar functions. However, this usage is only for brevity of description and ease of discussion; the usage does not imply that construction or architectural details of such elements or units are the same in all embodiments or that these commonly mentioned The components/modules are the only way to implement some of the example embodiments disclosed herein.

In this text, for convenience of explanation, such as "beneath", "below", "lower", "above" may be used , "upper (upper)" and other spatially relative terms are used to describe the relationship between one element or feature and another (other) element or feature shown in the drawings. It will be understood that these spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the term "below" can encompass both an orientation of above and below. The device may be at other orientations (rotated 90 degrees or at other orientations), and the spatially relative descriptors used herein interpreted accordingly.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the subject matter belongs solutions have the same meaning. It should also be understood that terms (such as those defined in commonly used dictionaries) should be interpreted to have a meaning consistent with their meanings in the context of the relevant art, and should not be construed as having Idealistic or overly formal meaning.

As used herein, the term "module" refers to any combination of software, firmware, and/or hardware used in conjunction with a module to provide the functionality described herein. The software may be implemented as software packages, code, and/or instruction sets or instructions, and the term "hardware" as used in any implementation described herein may include, alone or in any combination, for example, hardware circuitry, programmable circuitry, state machine circuitry, and/or firmware storing instructions for execution by the programmable circuitry. The modules may be implemented collectively or individually as circuitry that forms part of a larger system such as, but not limited to, an integrated circuit (IC), system on-chip , SoC) and so on.

The 3D technology mentioned earlier has many disadvantages. For example, TOF-based 3D imaging systems may require high power to operate the shutters or switches. These systems typically work in the range of a few meters to tens of meters, but the resolution of these systems degrades for measurements at short distances, making 3D imaging at distances of about a meter almost impractical. Therefore, TOF systems may be undesirable for cell phone camera applications where pictures are primarily taken at close range. TOF sensors may also require proprietary pixels with large pixel sizes (typically greater than 7 micrometers (μm)). These pixels may also be susceptible to ambient light.

Stereo imaging methods are generally only effective on textured surfaces. Stereoscopic imaging methods have high computational complexity due to the need to match features and find correspondences between stereoscopic image pairs of objects. This requires high system power, which is This is an undesired property in applications that are not capable, such as in smartphones. In addition, stereoscopic imaging requires two conventional high-resolution sensors and two lenses, making the whole assembly unsuitable for use in portable devices such as cell phones or tablets where device footprint is at a premium.

The SL method introduces distance ambiguity and also requires high system power. For 3D depth measurements, SL methods may require multiple images with multiple patterns—all of which increase computational complexity and power consumption. In addition, SL imaging may also require conventional image sensors with high bit resolution. Therefore, systems based on structured light may not be suitable for low-cost low-power small image sensors in smartphones.

Compared with the 3D technologies mentioned above, some embodiments disclosed herein are used to implement low-power 3D imaging systems on portable electronic devices such as smart phones, tablet computers, and UEs. The 2D imaging sensor according to some embodiments disclosed herein can simultaneously capture 2D red, green, blue (RGB) images and 3D depth measurements for both. It should be noted that although the following discussion may frequently refer to visible light lasers as the light source for point scanning and 2D RGB sensors as image/light capture devices, this is for illustrative purposes and discussion purposes only. consistent. The examples discussed below based on visible laser and RGB sensors can be applied in low power consumer mobile electronic devices with cameras such as smartphones, tablets or UEs. However, it should be understood that the subject matter disclosed herein is not limited to the examples mentioned below based on visible laser RGB sensors. Specifically, point scan-based 3D depth measurement and ambient light suppression methods can be performed using many different combinations of 2D sensors and laser light sources (for point scans), according to some embodiments disclosed herein. Group Examples include (but not limited to): (i) 2D color (RGB) sensor and visible light laser source, where the laser source can be red (R), green (G) or blue (B) laser A laser source that emits light, or a combination of these; (ii) a visible light laser with a 2D RGB color sensor with an infrared (Infrared, IR) cut filter; (iii) a near infrared (Near Infrared, NIR) Laser with 2D IR sensor; (iv) NIR laser with 2D NIR sensor; (v) NIR laser with 2D RGB sensor (without IR cut filter); (vi) NIR laser with 2D RGB sensor (without NIR cut filter); (vii) 2D RGB-IR sensor with visible laser or NIR laser; (viii) 2D red, green, blue, white (red, green, blue, white, RGBW) with visible laser or NIR laser; etc.

During 3D depth measurement, the entire sensor can work as a binary sensor in conjunction with laser scanning to reconstruct the 3D content. In some embodiments, the pixel size of the sensor can be as small as 1 μm. In addition, due to the lower bit resolution, the processing power required by the analog-to-digital converter (ADC) unit in the image sensor according to some embodiments disclosed herein can be significantly lower than conventional The processing power required by high-resolution sensors in 3D imaging systems. Since less processing power is required, 3D imaging modules according to the subject matter disclosed herein may require lower system power and, therefore, may be well suited for inclusion in low power devices such as smartphones.

In some embodiments, the subject matter disclosed herein uses triangulation and point scanning that utilizes a set of line sensors for 3D depth measurement using a laser light source. The laser scan plane and imaging plane are oriented using epipolar geometry. An image sensor according to an embodiment disclosed herein can use timestamps to disambiguate triangulation methods, thereby reducing depth Degree of calculation and system power. The same image processor—that is, each pixel in the image processor—can be used in normal 2D (RGB color or non-RGB) imaging mode and in 3D laser scanning mode. However, in laser scanning mode, the resolution of the ADC in the image sensor is reduced to a binary output (only 1-bit resolution), thereby increasing the readout speed and making it difficult for example due to the Reduced power consumption due to switching in dies of sensors and associated processing units. The point-scan approach allows the system to perform all measurements in one pass, reducing depth measurement latency and reducing motion blur.

As noted above, in some embodiments, the entire image sensor can be used for routine 2D RGB color imaging with, for example, ambient light and 3D depth imaging with visible laser scanning. This dual use of the same camera unit saves space and cost on the mobile device. In certain applications, visible lasers for 3D applications may be more eye-safe for users than near-infrared (NIR) lasers. The quantum efficiency of the sensor in the visible spectrum may be higher than in the NIR spectrum, resulting in reduced power consumption of the light source. In one embodiment, the dual-purpose image sensor can operate in a linear mode of operation as a conventional 2D sensor for 2D imaging. However, for 3D imaging, the sensor can operate in linear mode under moderate lighting conditions and in logarithmic mode under high ambient light to facilitate continued use of visible laser sources by suppressing high ambient light. Ambient light suppression may also be required in the case of NIR lasers, for example, if the passband bandwidth of the IR cut filter employed with the RGB sensor is not narrow enough.

In summary, the present disclosure uses a pinned photodiode (PPD) in a pixel as a time-to-charge converter (TCC) whose amplitude modulates charge to determine TOF. The transfer operation is done via the Pixel to control the output of multiple adjacent SPADs. When ambient light is high, the likelihood that the SPAD can be triggered by ambient photons rather than reflected photons (eg, in reflected pulse 37 ) is high. Relying on this trigger can lead to range measurement errors. Therefore, in the present invention, the PDD charge transfer is stopped to record TOF only when two or more SPADs are triggered within a very short predefined time interval (eg when the electronic shutter is turned on). Thus, an all-weather autonomous navigation system in accordance with the teachings of the present disclosure can provide drivers with improved navigation under difficult driving conditions such as, for example, low light, fog, bad weather, high ambient light, etc. vision. In some embodiments, a navigation system according to the teachings of the present disclosure may have a high ambient light rejection level of up to 100 kilolux (100 kLux). In some embodiments, a high spatial resolution pixel architecture with a smaller pixel size may be provided with a 1:1 SPAD/PPD ratio. In some embodiments, a SPAD can be biased below its breakdown voltage and can be used in avalanche photodiode (APD) mode.

FIG. 1 depicts a highly simplified partial configuration of an imaging system 15 in accordance with the subject matter disclosed herein. System 15 may include an imaging module 17 coupled to and in communication with a processor module or host 19 . The system 15 may further include a memory module 20 coupled to the processor module 19 for storing information content such as image data received from the imaging module 17 . In some embodiments, the entire system 15 may be packaged in a single integrated circuit (IC) or die. Alternatively, each of modules 17, 19 and 20 may be implemented in a separate die. Memory module 20 may include more than one memory chip, and processor module 19 may also include multiple processing chips. Details about the packaging of the module in FIG. 1 and how the module is made or implemented—in a single die or in multiple discrete dies—are not relevant to this discussion, and therefore, such details are not provided herein. detail.

System 15 may be any low power electronic device configured for 2D camera applications and 3D camera applications in accordance with the subject matter disclosed herein. System 15 may be portable or non-portable. Some examples of portable versions of system 15 may include popular consumer electronics such as (but not limited to) mobile devices, cell phones, smartphones, user equipment (UE), tablet computers, digital cameras, laptop computers Or a desktop computer, an electronic smart watch, a machine-to-machine (M2M) communication unit, a virtual reality (Virtual Reality, VR) device or module, a robot, etc. On the other hand, some examples of non-portable versions of the system 15 may include game consoles in electronic arcades, interactive video terminals, automobiles, machine vision systems, industrial robots, VR equipment, driver side mounted cameras (for example, to monitor driver sobriety), etc. The 3D imaging capabilities disclosed herein can be used in many applications, such as (but not limited to), automotive applications (such as 24/7 autonomous navigation and driver assistance in low light or adverse weather conditions), human-machine interface and gaming applications, machine vision and robotics applications.

In some embodiments disclosed herein, the imaging module 17 may include a projector module (or light source module) 22 and an image sensor unit 24 . The light source in the projector module 22 may be an infrared (IR) laser, such as, for example, a near infrared (NIR) laser or a short wave infrared (SWIR) laser, for unobtrusive illumination. In other embodiments, the light source can be a visible light laser. The image sensor unit 24 may include a pixel array and auxiliary processing circuits as shown in FIG. 2 .

In one embodiment, the processor module 19 may be a central processing unit (CPU), which may be a general-purpose microprocessor. As used herein, the terms "processor" and "CPU" are used interchangeably. However, it should be understood that instead of or in addition to the CPU, the processor module 19 may contain any other type of processor, For example (but not limited to) microcontrollers, digital signal processors (Digital Signal Processor, DSP), graphics processing units (Graphics Processing Unit, GPU), application-specific integrated circuit (Application Specific Integrated Circuit, ASIC) processors, etc. . In one embodiment, processor module/host 19 may include more than one CPU, which may operate in a distributed processing environment. The processor module 19 can be configured according to a specific instruction set architecture (Instruction Set Architecture, ISA) (such as (but not limited to), x86 instruction set architecture (32-bit version or 64-bit version), PowerPC® ISA, or without Interlocked pipeline stage microprocessor (Microprocessor without Interlocked Pipeline Stages, MIPS) instruction set architecture, said microprocessor instruction set architecture without interlocked pipeline stage relies on reduced instruction set computer (Reduced Instruction Set Computer, RISC) ISA) to execute instructions and process data. In one embodiment, the processor module 19 may be a System-on-Chip (SoC) having functions in addition to CPU functions.

In some embodiments, the memory module 20 can be a dynamic random access memory (Dynamic Random Access Memory, DRAM) (such as (but not limited to) synchronous dynamic random access memory (Synchronous DRAM, SDRAM)) or based on Three-dimensional stack (Three-Dimensional Stack, 3DS) memory module of DRAM (such as (but not limited to) high bandwidth memory (High Bandwidth Memory, HBM) module or hybrid memory cube (Hybrid Memory Cube, HMC) memory module Group). In other embodiments, the memory module 20 can be a solid-state drive (Solid-State Drive, SSD), a non-3DS DRAM module or any other semiconductor-based storage system, such as (but not limited to) SRAM (Static Random Access Memory, SRAM), Phase-Change Random Access Memory (Phase-Change Random Access Memory, PRAM or PCRAM), electrical Resistive Random Access Memory (RRAM or ReRAM), Conductive-Bridging Random Access Memory (Conductive-Bridging RAM, CBRAM), Magnetic Random Access Memory (Magnetic RAM, MRAM) or Spin Transfer Torque Magnetic Random Access Memory (Spin-Transfer Torque MRAM, STT-MRAM).

FIG. 2 illustrates an exemplary operational configuration of imaging system 15 in FIG. 1 in accordance with the subject matter disclosed herein. System 15 may be used to obtain range information or depth information (along the Z- axis) of objects such as object 26, which may be individual objects or objects within a scene (not shown). System 15 may be a direct TOF imager in which a single pulse may be used per image frame (of an array of pixels). In some embodiments, short pulses may be transmitted onto object 26 . In one embodiment, the range/depth information may be determined by the processor module 19 based on scan data received from the image sensor unit 24 . In another embodiment, the range/depth information may be determined by the image sensor unit 24 . In some embodiments, the depth information may be used by the processor module 19 as part of a 3D user interface to enable a user of the system 15 to interact with or use the 3D image of an object as an object running on the system 15. Part of a game or another application (such as an autonomous navigation application). 3D imaging in accordance with the subject matter disclosed herein can also be used for other purposes or applications, and can be applied to virtually any scene or 3D object.

In FIG. 2 , the X- axis is viewed horizontally along the front of the system 15, the Y- axis is vertical (off the page in this view), and the Z- axis is away from the system 15 in the general direction of the object 26 being imaged. extend. For depth measurement, the optical axis of module 22 and the optical axis of module 24 may be parallel to the Z axis. Other optical arrangements may be used to implement the principles set forth herein, and such alternative arrangements are considered within the scope of the herein disclosed subject matter.

Projector (or light source) module 22 may illuminate object 26 within optical fields of view (field of view, FOV) 28 and 29 as indicated by the arrows, which are associated with corresponding illumination paths 30 and 31, The dashed lines indicate the illumination paths 30 and 31 of the beam or optical radiation that may be used to spot scan the object 26 . Column-by-column point scanning of the object surface may be performed using an optical radiation source, which may be a laser light source 33 operated and controlled by a laser controller 34 in one embodiment. Under the control of laser controller 34 , the beam from laser source 33 may be spot-scanned across the surface of object 26 in XY direction by projection optics 35 . Spot scanning may project spots of light on the surface of an object along scan lines, as discussed in more detail with reference to FIG. 4 . The projection optics 35 can be focusing lenses, glass/plastic surfaces or other cylindrical optical elements that focus the laser beam from the laser 33 into points or spots on the surface of the object 26 . In the embodiment depicted in FIG. 2 , the protruding structure is shown as a focusing lens 35 . However, any other suitable lens design may be chosen for the projection optics 35 . The object 26 can be placed at a focus position where the illumination light from the light source 33 is focused into a light spot by the projection optical device 35 at the focus position. Thus, in point scanning, points or narrow regions/spots on the surface of the object 26 may be sequentially illuminated by the focused light beam from the projection optics 35 .

In some embodiments, the light source (or illumination source) 33 can be a diode laser, or a light emitting diode (Light Emitting Diode, LED) that emits visible light, NIR laser, point light source, monochromatic light in the visible spectrum Illumination source (for example, a combination of white lamp and monochromator), or any other type of laser light source. The laser 33 may be fixed in one position within the housing of the system 15, but rotatable in the XY direction. Laser 33 may be XY addressable (eg, via laser controller 34 ) to perform point scanning of 3D object 26 . In one embodiment, the visible light may be substantially green light. Visible light illumination from laser source 33 may be projected onto the surface of 3D object 26 using a mirror (not shown), or the point scan may be completely mirrorless. In some embodiments, the light source module 22 may include more or fewer elements than those shown in the exemplary embodiment depicted in FIG. 2 .

In the embodiment shown in FIG. 2, light reflected from a point scan of object 26 may travel along collection paths 36, 37, 38, and 39 indicated by arrows and dashed lines. The light collection path may carry photons that are reflected from or scattered by the surface of object 26 upon receipt of illumination from laser source 33 . Here, it should be noted that the use of solid arrows and dashed lines to depict various propagation paths in FIG. 2 (and, if applicable, in FIG. Any actual optical signal propagation path. In practice, the illuminating and collecting signal paths may differ from the paths shown in FIG. 2 and may not be as clearly defined as depicted in FIG. 2 .

Light received from illuminated object 26 may be focused by collection optics 44 in image sensor unit 24 onto one or more pixels of 2D pixel array 42 . Like projection optics 35 , collection optics 44 may be focusing lenses, glass/plastic surfaces, or other cylindrical optics that focus reflected light received from object 26 onto one or more pixels in array 42 . In the embodiment depicted in FIG. 2 , the protruding structure is shown as a focusing lens 44 . However, any other suitable lens design may be chosen for collection optics 44 . Although pixel array 42 is shown as merely a 3x3 pixel array in FIG. 2, it should be understood that modern pixel arrays may contain thousands or even millions of pixels. The pixel array 42 can be an RGB pixel array, wherein different pixels can collect light signals of different colors. In some embodiments, pixel array 42 can be any 2D sensor, such as (but not limited to) 2D RGB sensor with IR cut filter, 2D IR sensor, 2D NIR sensor, 2D RGBW sensor, 2D RGB-IR sensor. System 15 can use the same pixel array 42 to map object 26 (or a scene containing said object) for 2D RGB color imaging and for 3D imaging of the object 26 (involving depth measurements).

Pixel array 42 may convert the received photons into corresponding electrical signals, which are then processed by associated pixel processing unit 46 to determine a 3D depth image of object 26 . In one embodiment, pixel processing unit 46 may use triangulation for depth measurement. The triangulation method is subsequently discussed with reference to FIG. 4 . Pixel processing unit 46 may also include circuitry for controlling the operation of pixel array 42 .

The processor 19 can control the operations of the light source module 22 and the image sensor unit 24 . For example, system 15 may have a mode switch (not shown) that is controllable by a user to switch from a 2D imaging mode to a 3D imaging mode. If the user uses the mode switch to select the 2D imaging mode, the processor 19 can activate the image sensor unit 24 but not activate the light source module 22 because 2D imaging can use ambient light. On the other hand, if the user selects the 3D imaging mode using the mode switch, the processor 19 may activate both modules 22 and 24 and, for example, if the ambient light is too strong to be suppressed by the linear mode (as further explained below), The processor 19 can also trigger the level change of the reset (RST) signal in the pixel processing unit 46 to switch from the linear mode to the logarithmic imaging mode. The processed image data received from the pixel processing unit 46 can be stored in the memory 20 by the processor 19 . The processor 19 can also display a 2D image or a 3D image selected by the user on a display screen (not shown) of the system 15 . Processor 19 may be programmed with software or firmware to perform various processing tasks as set forth herein. Alternatively or in addition, processor 19 may include programmable hardware logic circuitry for implementing some or all of processor 19's functions. In some embodiments, the memory 20 can store program codes, look-up tables and/or intermediate calculation results so that the processor 19 can provide the functions of the processor 19 .

FIG. 3 illustrates a flowchart 50 of an exemplary embodiment of how 3D depth measurement may be performed in accordance with the subject matter disclosed herein. The various operations depicted in FIG. 3 may be performed by a single module or a combination of modules or system elements in system 15 . Recitation of specific tasks as being performed by specific modules or system components is by way of example only. Other modules or system components may be suitably configured to perform such tasks.

In FIG. 3, at operation 52, system 15 (and more specifically, processor 19) may use a light source (such as light source module 22) to perform a one-dimensional (one-dimensional, 1D) point scan. As part of the point scanning, the light source module 22 may be configured, eg by the processor 19, to project a series of points of light on the surface of the 3D object 26 in a column-by-column manner. At operation 54, pixel processing unit 46 in system 15 may select a row of pixels in an image sensor (eg, 2D pixel array 42). Image sensor 42 may have a plurality of pixels arranged in a 2D array to form an image plane on which a selected row of pixels forms an epipolar line of a scan line (at operation 52 ). A brief discussion of the epipole geometry is provided below with reference to FIG. 4 . At operation 56, the pixel processing unit 46 is operatively configured by the processor 19 to detect each point of light using a corresponding pixel of the row of pixels. It should be noted that, for example, if the light reflected from the illuminated spot is focused by collection optics 44 onto two or more adjacent pixels, then the light reflected from the illuminated spot can be detected by a single pixel or by more than one pixel. of light. It is also possible that light reflected from two or more light points may be collected at a single pixel in the 2D pixel array 42 . Ambiguities related to depth calculations caused by the imaging of two different blobs by the same pixel or the imaging of a single blob by two different pixels can be removed using a timestamp-based approach. At operation 58, the pixel processing unit 46 (as suitably configured by the processor 19) may respond to the series of light points (at operation 52 Pixel-specific detection (at operation 56) of the corresponding light point in the point scan at ) generates a pixel-specific output. Accordingly, at operation 60, pixel processing unit 46 may determine the surface to the 3D object based at least on the pixel-specific output (at operation 58) and the scan angle used by the light source to project the corresponding point of light (at operation 52). The 3D distance (or depth) of the corresponding light point on . Depth measurement is discussed in more detail with reference to FIG. 4 .

4 illustrates how an exemplary point scan may be performed for 3D depth measurement in accordance with the subject matter disclosed herein. In FIG. 4, the XY rotational capability of the laser source 33 is indicated by arrows 62 and 64, which depict the angular movement of the laser in the X direction (with angle β ) and the Y direction (with angle α ). In one embodiment, the laser controller 34 may control the XY rotation of the laser source 33 based on scanning instructions/inputs received from the processor 19 . For example, if the user selects a 3D imaging mode, the processor 19 may configure and control the laser controller 34 to enable 3D depth measurement of the object surface facing the projection optics 35 . In response, the laser controller 34 can initiate a 1D XY point scan of the surface of the object through the XY movement of the laser light source 33 . As shown in FIG. 4 , the laser 33 can target the object 26 by projecting a spot along a 1D horizontal scan line (two scan lines _SR 66 and _SR+1 68 of which are indicated by dashed lines in FIG. 4 ). The surface is point-scanned. Due to the curvature of the surface of object 26, points of light 70 to 73 may form scan line S _R 66 in FIG. 4 . The light spots forming scan line _SR+1 68 are not indicated with reference numerals. For example, laser 33 may scan object 26 one spot at a time in a left-to-right direction along columns R, R+1, and so on. The values of R, R+1, etc. refer to the columns of pixels in the 2D pixel array 42 and are known. For example, in 2D pixel array 42 in FIG. 4 , pixel column R is indicated using reference numeral 75 and column R+1 is indicated using reference numeral 76 . It should be understood that columns R and R+1 are selected from the plurality of pixel columns for illustrative purposes only.

The plane containing the columns of pixels in the 2D pixel array 42 may be referred to as the image plane, and the plane containing the scan lines (eg, lines S _R and S _R+1 ) may be referred to as the scan plane. In the embodiment depicted in FIG. 4 , the image plane and scan plane are oriented using epipolar geometry such that each pixel column R, R+1, etc. in the 2D pixel array 42 forms a corresponding scan line S The epipolar lines of _R , S _R+1 , etc. If the projection of the illuminated spots (in a scan line) onto the image plane can form distinct points along a line (i.e. the column R itself), then the pixel column R can be considered as the antipode of the corresponding scan line S _R . For example, in FIG. 4 , arrow 78 indicates spot 71 illuminated by laser 33 , and arrow 80 indicates spot 71 imaged or projected by focusing lens 44 along row R 75 . Although not shown in FIG. 4 , all of the light spots 70 to 73 will be imaged by the corresponding pixel in column R . Thus, in one embodiment, the physical arrangement (e.g., position and orientation) of laser 33 and pixel array 42 may be such that pixels on the surface of object 26 may be captured or detected by pixels in corresponding columns in pixel array 42. The illuminated spot in the scan line of which the pixel column forms the epipolar line of the scan line.

Pixels in 2D pixel array 42 may be arranged in columns and rows. An illuminated spot may be referenced by a corresponding column and row in pixel array 42 . For example, in FIG. 4 , spot 71 in scan line _SR is designated X _R,i to indicate that spot 71 is imageable by column R and row i ( C _i ) in pixel array 42 . Row C _i is indicated by dashed line 82 . Other illuminated spots can be identified in a similar manner. As previously mentioned, it is possible that light reflected from two or more points of light may be received by a single pixel in a row, or alternatively, light reflected from a single point of light may be received by a column of pixels More than one pixel in receive. Ambiguities in depth calculations caused by such multiple or overlapping projections can be removed using a timestamp-based approach.

In the illustration shown in FIG . 4 , the arrow with reference numeral 84 indicates the depth or distance Z ( along Z axis). In FIG. 4 , the dashed line with reference number 86 represents such an axis, which can be conceived to be contained in a vertical plane that also contains projection optics 35 and collection optics 44 . However, for ease of explanation of the triangulation-based approach, the laser source 33 is shown in FIG. 4 as being located on the X-axis 86 rather than the projection optics 35 . In triangulation-based methods, the value of Z can be determined using the following equation:

where h is the distance along the Z -axis between the collection optics 44 and the image sensor 42, which is assumed to be located in a vertical plane behind the collection optics 44; d is the distance between the light source 33 and the image sensor The offset distance between the collection optics 44 associated with the unit 24; q is the offset distance between the collection optics 44 and the pixel detecting the corresponding spot (in the example shown in FIG. 4, the detection/imaging pixel i is denoted by the row C _i associated with the spot X _R,i 71); and θ is the scan angle or beam angle. Alternatively, q can also be regarded as the shift of the light spot within the field of view of the pixel array 42 . The parameters in equation (1) are also indicated in FIG. 4 .

It should be seen from equation (1) that only the parameters θ and q are variable for a given point scan, and that h and d are substantially predetermined or fixed based on the physical geometry of the system 15 . Since the column R 75 is the epipolar line of the scan line _SR , the depth difference or depth profile of the object 26 can be reflected by the image shift element in the horizontal direction (as represented by the value of q of the different light spots being imaged) . A time-stamp based approach can be used to find the correspondence between the pixel position of the captured light spot and the corresponding scan angle of the laser source 33 . That is, the timestamp can represent the association between the value of q and the value of θ. Thus, from a known value of scan angle Θ and the corresponding position of the imaged spot (as denoted by q ), the distance Z to this spot can be determined using triangulation equation (1). Triangulation for distance measurement is also described in related literature including, for example, US Patent Application Publication No. 2011/0102763 Al (Brown) to Brown et al. Accordingly, the disclosures of the Brown Disclosure relating to triangulation-based distance measurement are hereby incorporated by reference in their entirety.

FIG. 5 is a block diagram of an exemplary embodiment of a pixel, such as pixel 43 in pixel array 42 shown in FIG. 2 , in accordance with the subject matter disclosed herein. For TOF measurements, the pixels 43 can work as time-resolved sensors. As depicted in FIG. 5 , pixel 43 may include a SPAD core portion 501 electrically connected to a PPD core portion 502 . Different exemplary configurations of SPAD core arrangements and PPD core arrangements in a pixel as disclosed herein are shown in FIGS. 6A-6C . The SPAD core portion 501 may include two or more SPADs 503 operatively connected to a first control circuit 504 . One or more of the SPADs 503 can receive the incoming light 505 and generate corresponding SPAD-specific electrical signals that are processed by the first control circuit 504 to generate a SPAD-specific digital output. All such SPAD-proprietary digital outputs are collectively, symbolically depicted by arrow 506 in FIG. 5 . PPD core 502 may include a second control circuit 507 coupled to PPD 508 . A second control circuit 507 may receive the SPAD output 506 and in response control charge transfer from the PPD 508 to generate a pixel specific analog output (PIXOUT) data line 510 . More specifically, as discussed in more detail below, a (reflected) photon in incoming light 505 is detected by two or more of adjacent SPADs 503 in a pixel 43 within a predetermined time interval, and the output from the PPD 508 is stopped by the second control circuit 507 in order to record the TOF value and the corresponding range to the 3D object 26 . In other words, between the outputs of at least two adjacent SPADs 503 The spatiotemporal correlation of is used to control the operation of the PPD 508 . For pixel 43, the SPAD 503 performs the light sensing function, while the PPD 508 acts as a TCC rather than a light sensing element. Reflected photons (of the return light pulse 37) are correlated with the transmitted pulse 28 (compared to uncorrelated ambient photons), so controlling charge transfer from the PPD 508 is based on triggering two or more The proximity to the SPAD provides improved performance of the image sensor unit 24 under high ambient light conditions by suppressing ambient photons, thereby substantially preventing range measurement errors.

6A to 6C illustrate three different examples of pixel array architectures according to the subject matter disclosed herein. Any one of the pixel array architectures shown in FIGS. 6A to 6C can be used to implement the pixel array 42 shown in FIG. 2 . An exemplary 2×2 pixel array architecture 600A is shown in FIG. 6A , where each pixel 601 through 604 (which in some embodiments may represent pixel 43 in FIG. 5 ) includes a pixel-specific PPD core and four Pixel proprietary SPAD core. For simplicity, only the PPD core and the SPAD core of pixel 601 are identified, where the PPD core is indicated by reference numeral 605 and the SPAD core is indicated by reference numerals 606-609.

Since each pixel occupies physical space on a semiconductor die of a given size, the architecture 600A depicted in FIG. 6A may be considered a low (spatial) resolution architecture. Thus, a relatively smaller number of pixels can be formed in an on-die pixel array compared to the example architecture 600B depicted in FIG. 6B , which provides a higher resolution 3×3 pixel array. array schema. In the higher resolution architecture 600B in Figure 6B, one SPAD core is shared by four (2x2) adjacent PPD cores. For example, in FIG. 6B, SPAD core 625 is shown as being shared by PPD cores adjacent to pixels 621-624 (in some embodiments, each of pixels 621-624 may represent of pixels 43). For simplicity, without reference numbers Other components in pixel array architecture 600B in FIG. 6B are identified. The configuration of pixel array architecture 600B in FIG. 6B provides an effective ratio of 1:1 between the PPD in a pixel and the SPAD associated with that pixel, in which four adjacent pixels Share a SPAD between them.

As shown in pixel array architecture 600C in FIG. 6C , this sharing can be extended to 3×3 sharing or more. The SPAD sharing configuration 600B depicted in FIG. 6B provides a high (spatial) resolution architecture for the pixel array because if each SPAD is shared between adjacent pixels on the die, the pixel array More pixels can be formed in the chip, thus making more available space on the die to accommodate more pixels. Additionally, since the pixels in the pixel array architecture 600B in FIG. 6B have a single PPD core associated with four SPAD cores in a 2×2 configuration, each pixel can detect up to four coincident photons (i.e., one photon per SPAD).

6A and 6B illustrate exemplary pixel array architectures in which PPDs and SPADs can be implemented in a single die. In other words, SPAD and PPD are at the same level in the bare die. In contrast, FIG. 6C illustrates an exemplary 4x4 pixel array architecture 600C in which pixels may be implemented in stacked die. For example, a SPAD core can be implemented in the upper die, and a PPD core (and readout circuitry) can be implemented in the lower die. Therefore, the PPD and SPAD can be on two different dies, the two different die can be stacked and the circuit elements (PPD, SPAD, transistors, etc.) connect. Like architecture 600B in FIG. 6B, pixel array architecture 600C in FIG. 6C can also provide a high-resolution architecture in which a single SPAD core can be shared by nine (3×3) adjacent PPD cores. Equivalently, as shown in FIG. 6C, a single PPD core (such as PPD core 641) can communicate with nine SPADs Cores (eg, SPAD cores 642-650) are associated to form a single pixel. SPAD cores 642 to 650 can also be shared by other pixels. For simplicity, the other pixels, their PPD cores and associated SPAD cores are not indicated with reference numbers in Figure 6C. Additionally, since the pixels in the pixel array architecture 600C in FIG. 6C have a single PPD core associated with nine SPAD cores in a 3×3 configuration, each pixel can detect up to nine coincident photons (i.e., one photon per SPAD).

FIG. 7 illustrates circuit details of an exemplary embodiment of a pixel 700 according to the subject matter disclosed herein. Pixel 700 depicted in FIG. 7 may be an example of more general pixel 43 depicted in FIGS. 2 and 5 . An electronic shutter signal 701 may be provided to each pixel (as discussed in more detail subsequently with reference to the timing diagrams in FIGS. The pixel is caused by proprietary optoelectronics. More generally, pixel 700 can be viewed as having a charge transfer triggering portion, a charge generation and transfer portion, and a charge collection and output portion. The charge transfer trigger part may include a SPAD core 501 and a logic unit 702 . The charge generation and transfer part may include PPD 508, a first N-channel Metal Oxide Semiconductor Field Effect Transistor (N-channel Metal Oxide Semiconductor Field Effect Transistor, NMOSFET or NMOS transistor) 703, a second NMOS transistor 704 and a third NMOS transistor 705 . The charge collecting and outputting part may include a third NMOS transistor 705 , a fourth NMOS transistor 706 and a fifth NMOS transistor 707 . In some embodiments, the PPD core in the pixel 700 in FIG. 7 and the pixel 1300 in FIG. Transistors) or other different types of transistors or charge transfer devices. Additionally, the various portions of pixel 700 described herein are for illustrative and discussion purposes only. In some embodiments, the portions may include more, fewer and/or different circuit elements than those described herein.

PPD 508 can store charge similar to a capacitor. In one embodiment, the PPD 508 may be covered and thus not responsive to light. Therefore, the PPD 508 can be used as a TCC instead of a photosensitive element. However, as mentioned above, the photosensitive function can be realized by the SPAD in the SPAD core 501 . In some embodiments, a shutter or other semiconductor device (with suitable modifications) may be used in place of the PPD in the pixel configurations shown in FIGS. 7 and 13 .

The charge transfer trigger part can generate a transfer enable (Transfer Enable, TXEN) signal 708 under the control of the electronic shutter signal 701 to trigger the transfer of charges stored in the PPD 508 . A SPAD can detect photons in pulses of light emitted and reflected from an object (such as object 26 in FIG. Latch under control for subsequent processing by the logic unit 702 . The logic unit 702 may include logic circuitry for processing all digital SPAD outputs 506 to generate an output 506 when, for example, outputs 506 are received from at least two adjacent SPADs within a predefined time interval while the shutter signal 701 is active. TXEN signal 708 .

In the charge generation and transfer portion, the reset (RST) signal 709 may be used in conjunction with the third transistor 705 to first set the PPD 508 to its full well capacity. The first transistor 703 may receive a transfer voltage (VTX) signal 710 at a drain terminal of the first transistor 703 and a TXEN signal 708 at a gate terminal of the first transistor 703 . The TX signal 711 is available at the source terminal of the first transistor 703 and applied to the gate terminal of the second transistor 704 . As shown, the source terminal of the first transistor 703 may be connected to the gate terminal of the second transistor 704 . VTX signal 710 (or equivalently, TX signal 711 ) may be used as an amplitude modulation signal to control the charge to be transferred from PPD 508 , which may be connected to the source terminal of transistor 704 . The second transistor 704 can transfer the charge on the PPD 508 from the source terminal of the second transistor 704 to the drain terminal of the second transistor 704, which can be connected to the fourth transistor 706 and forms a charge “collection site” referred to herein as a floating diffusion (FD) node/junction 712 . In some embodiments, the charge transferred from PPD 508 may depend on the modulation provided by amplitude modulation signal 710 (or equivalently, TX signal 711 ). In the embodiments shown in Figures 7 and 13, the transferred charges are electrons. However, the subject matter disclosed herein is not limited to this disclosure, and PPDs with different designs can be used where the transferred charges can be holes.

In the charge collection and output section, the third transistor 705 may receive the RST signal 709 at the gate terminal of the third transistor 705 and the pixel voltage (VPIX) signal 713 at the drain terminal of the third transistor 705 . The source terminal of transistor 705 may be connected to floating diffusion node/junction 712 . In one embodiment, the voltage level of the VPIX signal 713 may be equal to the voltage level of the universal power supply voltage VDD, and may be in the range of 2.5 volts (V) to 3.0V. The drain terminal of the fourth transistor 706 can also receive the VPIX signal 713 . In some embodiments, the fourth transistor 706 can operate as an NMOS source follower to act as a buffer amplifier. The source terminal of the fourth transistor 706 may be connected to the drain terminal of the fifth transistor 707 which may be cascoded with the source follower 706 at the gate terminal of the fifth transistor 707 A select (SEL) signal 715 is received. The charge transferred from the PPD 508 and collected at the floating diffusion node/junction 712 can appear as a pixel specific output (PIXOUT) data line 510 at the source terminal of the fifth transistor 707 .

Charge transfer from PPD 508 to FD node/junction 712 is controlled by VTX signal 710 (and TX signal 711). The amount of charge reaching the floating diffusion node/junction 712 is modulated by the TX signal 711 . In one embodiment, transfer voltage (VTX) signal 710 (and TX signal 711 ) may be ramped to gradually transfer charge from PPD 508 to floating diffusion node/junction 712 . Thus, the amount of charge transferred can be a function of the amplitude modulated TX signal 711, and the ramp of the TX signal 711 can be a function of time. Therefore, the amount of charge transferred from PPD 508 to floating diffusion node/junction 712 is also a function of time. If during charge transfer from PPD 508 to floating diffusion node/junction 712, second transistor 704 is turned off due to logic unit 702 generating TXEN signal 708 when at least two adjacent SPADs in SPAD core 501 have a photon detection event, The transfer of charge from PPD 508 to floating diffusion node/junction 712 then ceases. Thus, the amount of charge transferred to floating diffusion node/junction 712 and the amount of charge remaining in PPD 508 are both a function of the TOF of the incoming photon. The result is time-to-charge conversion and single-ended-to-differential signal conversion. Therefore, PPD 508 operates as a time-to-charge converter (TCC). The more charge transferred to the floating diffusion node/junction 712 , the more the voltage on the floating diffusion node/junction 712 decreases and the more the voltage on the PPD 508 increases.

The voltage at the floating diffusion node/junction 712 can then be transferred to an analog-to-digital converter (ADC) unit (not shown) via the fifth transistor 707 as the PIXOUT signal and converted to an appropriate Digital signal/value for further processing. More details on the timing and operation of the various signals in FIG. 7 are provided with reference to the discussion of FIG. 9 . In the embodiment shown in FIG. 7, the fifth transistor 707 can receive the SEL signal 715 for selecting the pixel 700 to sense the charge in the floating diffusion node/junction 712 as the PIXOUT1 (or pixel output 1) voltage and The PPD is read after the remaining charge in the PPD 508 is fully transferred to the floating diffusion node/junction 712 The remaining charge in 508 is used as the PIXOUT2 (or pixel output 2) voltage, wherein the floating diffusion node/junction 712 converts the charge on the PPD 508 into a voltage, and the pixel output data line 510 sequentially outputs the PIXOUT1 signal and the PIXOUT2 signal, such as This is discussed subsequently with reference to FIG. 10 . In another embodiment, either the PIXOUT1 signal or the PIXOUT2 signal can be read, but not both.

8 is an exemplary timing diagram 800 that provides an overview of the modulated charge transfer mechanism in pixel 700 shown in FIG. 7 in accordance with the subject matter disclosed herein. The waveforms shown in Figure 8 (as well as Figures 9 and 14) are simplified in nature and are for illustrative purposes only; actual waveforms may differ in timing and shape depending on circuit implementation. Signals that are common between FIGS. 7 and 8 are identified using the same reference numbers and include VPIX signal 713 , RST signal 709 , electronic shutter signal 701 , and amplitude modulation signal 710 . Two additional waveforms 801 and 802 are also depicted in FIG. 8 to illustrate the state of charge in PPD 508 and the state of charge in floating diffusion node/junction 712 when amplitude modulated signal 710 is applied during charge transfer, respectively. . In the embodiment shown in FIG. 8, VPIX signal 713 may start at a low logic voltage (eg, logic 0 or 0V) to initialize pixel 700, and switch to a high logic voltage (eg, logic 0 or 0V) during operation of pixel 700. For example, logic 1 or 3V)). Reset (RST) signal 709 may begin with a high logic voltage pulse (eg, a pulse from logic 0 to logic 1 and back to logic 0) during initialization of pixel 700 to set the charge in PPD 508 to its full well capacity and sets the charge in the floating diffusion node/junction 712 to zero coulombs (0C). The reset voltage level of the floating diffusion node/junction 712 may be a logic 1 level. The more electrons the floating diffusion node/junction 712 receives from the PPD 508 during a range (TOF) measurement operation, the lower the voltage on the floating diffusion node/junction 712 becomes. Electronic shutter signal 701 may be at a low logic voltage during initialization of pixel 700 (eg, logic 0 or 0V), switch to a logic 1 level (eg, 3V) at times corresponding to the minimum measurement range during operation of the pixel 700 to enable the SPAD 503 in the SPAD core 501 to detect The photons in the light pulse 37 are returned and then switched to a logic zero level (eg, 0V) at a time corresponding to the maximum measurement range. Thus, the duration of the logic 1 level of the shutter signal 701 may provide a predefined time interval/window such that the outputs received from neighboring SPADs during this time interval are spatiotemporally correlated. The charge in PPD 508 starts out fully charged during initialization and decreases as VTX signal 710 ramps from 0V to a higher voltage, preferably in a linear fashion. The PPD charge level under the control of the amplitude modulated VTX signal 710 is depicted in FIG. 8 by the waveform with reference numeral 801 . The PPD charge reduction may be a function of the ramp time of the VTX signal, which causes an amount of charge to be transferred from the PPD 508 to the floating diffusion node/junction 712 . Thus, as depicted by the waveform with reference numeral 801 in FIG. This is increased by a high voltage, which in part transfers a certain amount of charge from the PPD 508 to the floating diffusion node/junction 712 . The charge transfer is a function of the ramp time of the VTX signal 710 .

The pixel-specific output (PIXOUT) on data line 510 is derived from the PPD charge transferred to floating diffusion node/junction 712, as previously described. Therefore, the PIXOUT signal 510 can be viewed as being amplitude modulated over time by the amplitude modulated VTX voltage 710 (or equivalently, the TX voltage 711 ). In this way, TOF information is provided by amplitude modulating the pixel-specific output 510 using the amplitude modulated VTX signal 710 (or equivalently, the TX signal 711 ). In some embodiments, the modulation function used to generate the VTX signal 710 may be monotonic. In the exemplary embodiments shown in FIGS. 8, 9 and 14, a ramp function may be used to generate the amplitude modulated signal, and since Here, the amplitude modulated signal is shown as having a ramp-type waveform. However, in other embodiments, different types of analog waveforms/functions may be used as modulation signals.

In one embodiment, the ratio of one pixel output (for example, PIXOUT1 ) to the sum of the two pixel outputs (here, PIXOUT1+PIXOUT2 ) may be proportional to the time difference between T _tof and T _dly values , T _tof and T _dly values are shown, for example, in FIG. 9 and subsequently discussed in more detail below. In the case of pixel 700, for example, the parameter T _tof can be the pixel-specific TOF value of the light signal received by two or more SPADs in the SPAD core 501, and the delay time parameter T _dly can be The time from when optical signal 28 is first transmitted until VTX signal 710 begins ramping. The delay time Tdly _may be negative if the light pulse 28 is emitted after the VTX signal 710 begins ramping (this typically occurs when the electronic shutter 701 opens). The proportional relationship can be expressed by:

However, the subject matter disclosed herein is not limited to the relationship shown in equation (2). As discussed below, the ratio in equation (2) can be used to calculate the depth or distance of an object and is less sensitive to pixel-to-pixel variation if Pixoutl + Pixout2 are not always the same.

For convenience, the term "P1" may be used to refer to " Pixout1 " as used herein and the term "P2" may be used to refer to " Pixout2 " as used herein. From the relationship in equation (2), it can be seen that the pixel-specific TOF value can be determined as the ratio of the pixel-specific output values P1 and P2. In some embodiments, once the pixel-specific TOF value is thus determined, the pixel-specific distance D to an object (such as object 26 in FIG. 2 ) or a specific location on the object can be given by or range R :

where c is the speed of light. Alternatively, in some embodiments where, for example, the modulating signal (e.g., VTX signal 710 (or TX signal 711) in FIG. 7) is linear within the shutter window, the range/distance can be calculated as follows:

Accordingly, TOF system 15 may generate a 3D image of an object (eg, object 26 ) based on the pixel-specific range values determined by the equations given above.

Amplitude modulation based manipulation or control of the PPD charge distribution within a pixel makes range measurement and resolution controllable as well. Pixel-level amplitude modulation of the PPD charge can work with electronic shutters such as rolling shutters in complementary metal oxide semiconductor (CMOS) image sensors or For example, a global shutter in a charge coupled device (CCD) image sensor. Although the disclosure herein may be provided primarily in the context of a single-pulse TOF imaging system such as system 15 in FIGS. 1 and 2 , the principles of the pixel-level internal amplitude modulation method discussed herein may be adapted This is implemented (if desired) in a continuous wave modulated TOF imaging system or a non-TOF system and pixel 43 (FIG. 5).

FIG. 9 is a diagram of the pixel 700 shown in FIG. 1 and FIG. 2 when using the pixel 700 in the embodiment shown in FIG. Timing diagram 900 of exemplary timing of various signals in system 15 . The same reference numbers are used in FIG. 9 to identify the various signals depicted in the embodiments shown in FIGS. 2 and 7, such as transmitted pulse 28, VPIX signal 713, TXEN signal 708, and so on. Before discussing FIG. 9, it should be noted that in the context of FIG. 9 (and in the case of FIG. 14), the parameter T _dly refers to the time instance between the rising edge of the projected pulse 28 and when the VTX signal 710 begins to ramp. , as indicated by reference number 901; the parameter T _tof refers to the pixel-specific TOF value measured by the delay between the rising edge of the projected pulse 28 and the rising edge of the received pulse 37, as indicated by reference number 902 indicated; and the parameter Tsh designates the time period between the opening and closing of the electronic shutter, as indicated _by reference numeral 903 and through the assignment (for example, logic 1 or on) and deassignment of the shutter signal 701 (or deactivate) (for example, logic 0 or shutdown) is given. Therefore, electronic shutter 701 is considered active during period T _sh , which is also identified using reference numeral 904 . In some embodiments, the delay T _dly may be predetermined and fixed regardless of operating conditions. In other embodiments, the delay T _dly may be adjusted at execution time, depending eg on external weather conditions. Here, it should be noted that the high signal level or the low signal level is related to the design of the pixel 700 . The signal polarities or bias levels shown in FIG. 9 may be different in other types of pixel designs based on, for example, the type of transistors or other circuit elements used.

As previously mentioned, the waveforms shown in Figure 9 (and Figure 14) are simplified in nature and are for illustrative purposes only; actual waveforms may differ in timing and shape depending on circuit implementation. As shown in FIG. 9 , return pulse 37 may be a time-delayed version of projected pulse 28 . In some embodiments, the projected pulse 28 may be of very short duration, such as, for example, in the range of about 5 nanoseconds (ns) to about 10 ns. Return pulse 37 may be sensed using two or more SPADs in pixel 700 . The electronic shutter signal 701 may enable the SPAD to capture pixel-specific photons in the received light 37 . The electronic shutter signal 701 may have a gating delay (cf. projected pulse 28 ) to avoid light scatter reaching the pixel array 42 . Scattering of light from the projected pulse 28 may occur, for example, due to bad weather.

Timing diagram 900 in FIG. 9, in addition to various external signals (e.g., VPIX signal 713, RST signal 709, etc.) The following events or time periods are also identified: (i) PPD default event 905 when the RST signal, VTX signal, TXEN signal, and TX signal are high while the VPIX signal 713 and shutter signal 701 are low; (ii) from the TX signal to Low until first floating diffusion reset event 906 when RST signal goes from high to low; (iii) delay time T _dly 901; (iv) flight time T _tof 902; (v) electronic shutter on or active period T _sh 903; and (vi) a second FD reset event 907 for the duration of the second time RST signal 709 is logic 1. 9 also illustrates when the electronic shutter is first closed or turned off (this is indicated by reference numeral 908), when the electronic shutter is opened or switched on (this is indicated by reference numeral 904), and is first transferred to the floating diffusion node/junction 712. When the charge of the PIXOUT data line 510 is read out (this is indicated by reference number 909), when the voltage of the floating diffusion node/junction 712 is reset a second time at 907, and when the remaining charge in the PPD 508 is transferred to Node/junction 712 is floated and read out again at event 910 (eg, as output to PIXOUT 510). In one embodiment, the shutter turn-on period T _sh may be less than or equal to the ramp time of the VTX signal 710 .

Referring to FIG. 9, in the case of pixel 700 in FIG. 7, PPD 508 may be charged to its full well capacity at an initialization phase (eg, PPD preset event 905). During PPD default event 905, RST signal 709, VTX signal 710, TXEN signal 708, and TX signal 711 can be high, while VPIX signal 713 and shutter signal 701 can be low, as shown. Next, the VTX signal 710 (and TX signal 711 ) can go low to turn off the second transistor 704 and the VPIX signal 713 can go high to start charge transfer from the fully charged PPD 508 . in some real In one embodiment, all pixels in a row of pixels in pixel array 42 may be selected together at one time, and the PPDs in all pixels in the selected row may be reset together using RST signal 709 . Each pixel in a selected column of pixels can be read individually, and the analog-based pixout signal can be converted to a digital value by a corresponding row ADC unit (not shown). In one embodiment, the RST line may be held high or on for unselected pixel columns to prevent blooming.

In the embodiment shown in FIG. 9, all signals except TXEN signal 708 start with a logic 0 or low level, as shown. First, the PPD 508 is preset when the RST signal 709 , VTX signal 710 , TXEN signal 708 , and TX signal 711 go to a logic 1 level and the VPIX signal 713 remains low. Thereafter, when VTX signal 710 and TX signal 711 become logic 0 and VPIX signal 713 becomes high (or logic 1), floating diffusion node/junction 712 is reset while RST signal 709 is logic 1. For convenience, the same reference number 712 is used to refer to the floating diffusion nodes/junctions in FIG. 7 and the associated voltage waveforms in the timing diagram shown in FIG. 9 . After floating diffusion node/junction 712 is reset high (eg, 0C in the charge domain), VTX signal 710 ramps while TXEN signal 708 is logic 1. Time-of-flight T _tof duration 901 is from when pulsed light 28 is emitted until return light 37 is received, and is also the time during which charge is partially transferred from PPD 508 to floating diffusion node/junction 712 . VTX signal 710 (and TX signal 711 ) may be ramped while shutter 701 is on or open. This may cause an amount of charge in PPD 508 to be transferred to floating diffusion node/junction 712, which amount may be a function of the ramp time of VTX. When the transmitted pulse 28 is reflected from the object 26 and received by at least two SPADs in the SPAD core 501 of the pixel 700, the resulting SPAD output 506 can be processed by the logic unit 702, which in turn can cause the TXEN signal 708 to become Static logic 0. Thus, detection of return pulse 37 by at least two adjacent SPADs in a time-correlated manner (ie, when the shutter is on or active) can be indicated by a logic 0 level of TXEN signal 708 . A logic low on TXEN signal 708 turns off transistor 703 and transistor 704 , which stops charge transfer from PPD 508 to floating diffusion node/junction 712 . When ESL signal 701 goes to logic 0 and SEL signal 715 (not shown in FIG. 9 ) goes to logic 1, the charge in floating diffusion node/junction 712 is output on PIXOUT line 510 as voltage PIXOUT1. Next, the floating diffusion node/junction 712 can be reset again with a logic high RST pulse 709 (as indicated by reference numeral 907). Thereafter, when TXEN signal 708 becomes a logic 1, substantially all of the remaining charge in PPD 508 is transferred to floating diffusion node/junction 712 and output onto PIXOUT line 510 as voltage PIXOUT2. As mentioned earlier, the PIXOUT1 and PIXOUT2 signals may be converted by appropriate ADC units (not shown) into corresponding digital values P1 and P2. In some embodiments, these P1 and P2 values may be used in equation (3) or equation (4) to determine the pixel-specific distance/pixel-specific range between pixel 700 and object 26 .

In one embodiment, logic unit 702 may include logic circuitry (not shown) to generate an output based on the G( ) function (shown and discussed with reference to FIG. Signals similar to the TXRMD signal 1401 shown in FIG. 14 are ORed to obtain the final TXEN signal 708 . This internally generated signal can be held low while the electronic shutter is on, but can be assigned high to cause the TXEN signal 708 to become a logic 1, thereby facilitating the transfer of residual charge in the PPD ( event 910). In some embodiments, the TXRMD signal or similar signal may be externally supplied.

FIG. 10 illustrates a logical unit (eg, logic How unit 702 (FIG. 7) or logic unit 1319 (FIG. 13) may be implemented in a pixel such as pixel 700 (FIG. 7) or pixel 1300 (FIG. 13)). FIG. 10 shows a pixel 1000 having a PPD core 1001 associated with four SPAD cores 1002 to 1005 in a 2×2 architecture configuration as depicted in FIG. 6A or FIG. 6B (pixel 1000 may represent pixel 700 or any one in 1300 pixels) highly simplified diagram. The availability of four SPADs enables the detection of up to four temporally and spatially correlated coincident photons. In some embodiments, logic units (not shown) in pixel 1000 may include logic circuits (not shown) that implement the functions F(x,y) and G(a,b,c,d) depicted in FIG. Shows). Blocks 1006 to 1009 in FIG. 10 depict the inputs and outputs of the logic circuit implementing the F(x,y) function. Accordingly, blocks 1006 through 1009 may be considered to represent such logic circuits and together form part of the logic unit of pixel 1000 . For ease of discussion, these boxes may be referred to as F(x,y) boxes. Although blocks 1006 to 1009 are shown external to PPD core 1001 for convenience, it should be understood that logic circuits implementing the functions of blocks 1006 to 1009 may be part of logic units (not shown) in PPD core 1001 .

As shown, each F(x,y) block 1006 through 1009 may receive two inputs x and y, one from each of its two associated SPAD cores. In the context of FIGS. 5 and 7 , such input may be in the form of output signal 506 from SPAD core 501 . In the context of FIG. 13 , SPAD outputs 1310 and 1318 may represent the x,y inputs necessary for such F(x,y) boxes in logic unit 1319 . For pixels with more than four SPAD cores associated with PPD cores (such as, for example, pixel array configuration 600C in FIG. 6C ), similar two-input )F(x,y) box. In some embodiments, all of the F(x,y) blocks 1006 to 1009 may be integrated and implemented by a single F(x,y) unit in the PPD core 1001, which contains the logic , the Logic circuits are configured to operate on different pairs of SPAD outputs (as its x and y inputs) to implement the functions of individual F(x,y) blocks 1006 to 1009 . As previously mentioned, the TOF measurements disclosed herein may be based on the detection of spatially and temporally correlated photons by at least two SPADs in a pixel. Therefore, as shown in FIG. 10, each F(x, y) block 1006 to 1009 (more specifically, the logic circuit in the F(x, y) block) can be configured to perform the following predefined operations: ( i) perform a logical AND (NAND) operation (given by (x*y)) on its respective inputs x and y to detect two or four coincident photons, and (ii) on its respective inputs x and y A logical OR (NOR) operation (given by (x+y)) is performed to detect three coincident photons. Thus, F(x,y) blocks 1006 through 1009 are implemented when signals 506 (FIG. 5) from SPAD cores 1002 through 1005 indicate that both (or all four) SPADs detected photons during the shutter on period. The logic circuit can perform logical NAND operations. Similarly, a logical NOR operation may be selected when the signal 506 from the SPAD cores 1002 through 1005 indicates that the three SPADs detected photons during the shutter on period. Three pulses 1010 to 1012 are shown in the exemplary depiction in FIG. )) for the detection of three coincident photons.

Referring back to FIG. 10 , the output of each F(x,y) block 1006 to 1009 is depicted using corresponding reference letters a, b, c, and d. The logic units (not shown) in the PPD core 1001 may also include additional logic circuits (not shown) for receiving and processing the outputs a through d. A logic circuit may receive all four of these outputs as inputs to the logic circuit and operate on them according to a predefined logic function G(a,b,c,d). For example, as shown in FIG. 10, in the case of detection of two coincident photons, the G( ) function may perform a logical NAND operation on all four of its inputs a through d (by (a*b*c*d) gives). On the other hand, in the case of detection of three or four coincident photons, the G( ) function performs a logical NOR operation on all four of its inputs a to d (given by (a+b+c+d) ). In one embodiment, the TXEN signal (such as the TXEN signal 708 in FIG. 7 or the TXEN signal 1325 in FIG. 13 ) can be an output of a logic circuit implementing the G( ) function. In another embodiment, an OR operation may be performed on the output of the logic circuit for the G( ) function and an internally generated signal (such as the TXRMD signal 1401 in FIG. 14 ) to obtain the final TXEN signal.

FIG. 11 depicts an exemplary flowchart 1100 illustrating how TOF values may be determined in the system 15 shown in FIGS. 1 and 2 in accordance with the subject matter disclosed herein. The various steps indicated in FIG. 11 may be performed by a single module or a combination of modules or system elements in system 15 . In the discussion herein, describing particular tasks as being performed by particular modules or system components is by way of example only. Other modules or system components may also be suitably configured to perform such tasks. As described at operation 1101, first, system 15 (and more specifically, projector module 22) may project a laser pulse (eg, pulse 28 in FIG. 2) onto an object (eg, object 26 in FIG. 2) superior. At operation 1102, processor 19 (or, in some embodiments, pixel processing unit 46) may apply an amplitude modulated signal (eg, VTX signal 710 in FIG. 7 ) to a PPD (eg, FIG. 7 PPD 508 in pixel 700 in ). The pixel 700 can be any one of the pixels 43 in the pixel array 42 in FIG. 2 . At operation 1103 , pixel processing unit 46 may initiate the transfer of a portion of the charge stored in PPD 508 based on the modulation received from amplitude modulation signal 710 . To initiate this charge transfer, pixel processing circuit 46 may provide various external signals to pixel 700, such as electronic shutter signal 701, VPIX signal 713, and RST, at logic levels shown in the exemplary timing diagram shown in FIG. Signal 709. At operation 1104, multiple SPADs in the pixel 700 may be used to explore Measure return pulses, such as return pulse 37. As mentioned earlier, the return pulse 37 may be the projected pulse 28 reflected from the object 26, and each SPAD in the pixel 700 (in the SPAD core 501) is operable to convert the illumination received from the return pulse into Corresponding (SPAD proprietary) electrical signal.

For each SPAD receiving illumination, the first control circuit 504 in the SPAD core 501 in the pixel 700 may process the corresponding (SPAD-specific) electrical signal to generate a SPAD-specific digital output therefrom (operation 1105). All such SPAD-specific digital outputs are collectively represented by arrows with reference number 506 in FIGS. 5 and 7 . As described with reference to the discussion of FIG. 9, logic unit 702 may process output 506 and may place TXEN signal 708 in a logic 0 (low) state as long as the outputs are temporally and spatially correlated. A logic 0 level of TXEN signal 708 turns off first transistor 703 and second transistor 704 in pixel 700 , which stops charge transfer from PPD 508 to floating diffusion node/junction 712 . Accordingly, at operation 1106, the second control circuit 507 may terminate the earlier initiated charge-to-charge when at least two SPAD-specific digital outputs are generated at predetermined time intervals (eg, within the shutter-on period 904 in FIG. 9 ). Transfer of the portion (at operation 1103).

As discussed earlier with reference to FIG. 7, the portion of the charge transferred to the floating diffusion/junction 712 (until the transfer is terminated at operation 1106) can be read out as the Pixout1 signal and converted to the appropriate digital value P1, bit The value P1 can be used together with the subsequently generated digital value P2 (of the Pixout2 signal) to obtain TOF information from the ratio P1/(P1+P2). Accordingly, at operation 1107, pixel processing unit 46 or processor 19 in system 15 may determine a TOF value for return pulse 37 based on the portion of analog charge transferred at termination (at operation 1106).

FIG. 12 is a diagram of an image sensor unit 1200 according to the subject matter disclosed herein. An example layout of a section. The image sensor unit 1200 may correspond to the image sensor unit 24 shown in FIGS. 1 and 2 . The portion of the image sensor unit 1200 shown in FIG. 12 can communicate with and provide the signals necessary to capture the return light and generate P1 and P2 values for subsequent calculation of the TOF value (from equation (2)) and generate The 3D image of the object 26 is related. As in the case shown in FIG. 2 , for convenience, the pixel array 1201 in the image sensor unit 1200 in FIG. 12 is shown as having nine pixels arranged in a 3×3 array. In practice, pixel arrays may contain hundreds of thousands or millions of pixels in multiple columns and rows. In some embodiments, each pixel in pixel array 1201 may have the same configuration, and thus, as shown in FIG. 12, the same reference number 1202 is used to identify each pixel. In the embodiment shown in FIG. 12 , the 2D pixel array 1201 can be a complementary metal oxide semiconductor (CMOS) array, and each pixel 1202 can be the pixel 1300 shown in FIG. 13 . Although the example layout in FIG. 12 refers to the pixel configuration shown in FIG. 13, it should be understood that when each pixel 1202 has the configuration shown in FIG. to be modified. In some embodiments, the pixels 1202 may have different configurations than those shown in FIGS. Suitably modified to run with the desired pixel configuration.

In addition to the pixel array 1201, the image sensor unit 1200 in the embodiment shown in FIG. 12 may also include a column decoder/driver 1203, a row decoder 1204, and a pixel row unit 1205. Circuitry for Correlated Double Sampling (CDS) and row-specific Analog-to-Digital Converters (ADCs) to be used during 2D and 3D imaging. In one embodiment, there may be one ADC per row of pixels. In some embodiments, the processing units 1203, 1204 and 1205 may be part of the pixel processing unit 46 shown in FIG. 2 . In the embodiment shown in FIG. 12 , column decoder/driver 1203 is shown providing six different signals as inputs to each pixel 1202 in a column of pixels to control the pixels in pixel array 1201. operation and thereby enable generation of row-specific pixout signals 1206-1208. Each of the arrows 1209 to 1211 in FIG. 12 illustrates a column-specific set of these signals to be applied as input to each pixel 43 in the corresponding column. These signals may include: reset (RST) signal, second transfer (TX2) signal, electronic shutter (SH) signal, transfer voltage (VTX) signal, pixel voltage (VPIX) signal and column select (SEL) signal. Figure 13 shows how these signals may be applied to the pixels. FIG. 14 shows an example timing diagram including many of these signals.

In one embodiment, a row select (SEL) signal may be assigned to select the appropriate row of pixels. Column decoder/driver 1203 may receive address or control information for a column to be selected via column address/control input 1212 , eg, from processor 19 . The column decoder/driver 1203 can decode the received input 1212 to enable the column decoder/driver 1203 to select the appropriate column using the SEL signal, and also provide the corresponding RST signal, VTX signal, and other signals to the selected/ The decoded column. A more detailed discussion of these signals (when applied as pixel inputs) is provided below with reference to the discussion of FIGS. 13 and 14 . In some embodiments, column decoder/driver 1203 may also receive control signals (not shown), such as from processor 19, to configure column decoder/driver 1203 to respond to the SEL signals indicated at arrows 1209 to 1211, the RST signal, VTX signal, SH signal, and various other signals to apply appropriate voltage levels.

Pixel row unit 1205 may receive PIXOUT signals 1206-1208 from pixels in a selected column and process them to generate pixel-specific signal values from which TOF measurements may be obtained. The signal values can be P1 and P2 value, as indicated by arrow 1213 in FIG. 12 . Each row-specific ADC unit processes the received input (pixout signal) to generate a corresponding digital data output (P1/P2 value). More details of the CDS operation and the ADC operation provided by the CDS circuit and the ADC circuit (not shown) in the pixel row unit 1205 are provided below with reference to FIG. 14 . In the embodiment depicted in FIG. 12 , row decoder 1204 is shown coupled to pixel row unit 1205 . Row decoder 1204 may receive a row address/control input 1214 from, for example, processor 19 for a row to be selected in conjunction with a given column select (SEL) signal. Row selection may be sequential, enabling sequential receipt of pixel output from each pixel in the column selected by the corresponding SEL signal. Processor 19 may provide an appropriate column address input to select a column of pixels, and may also provide an appropriate row address input to row decoder 1204 to enable pixel row unit 1205 to select an individual picture from the selected column. Pixel receive output (pixout).

FIG. 13 illustrates another exemplary embodiment of a pixel 1300 according to the subject matter disclosed herein. Pixel 1300 in FIG. 13 is another example of the more general pixel 43 depicted in FIG. 2 . The pixel 1300 may include a plurality of SPAD cores (ie, SPAD core 1 to SPAD core N, where N≧2) as part of the SPAD cores of the pixel 1300 . Some circuit details of two such SPAD cores 1301A and 1301N are shown in FIG. 13 . It should be noted that in some embodiments, similar circuitry may be employed for the SPAD core in pixel 700 in FIG. 7 . The SPAD core 1301A may include a SPAD 1302 that receives a SPAD operating voltage VSPAD 1303 through a resistive element (eg, a resistor) 1304 . However, the configuration of the SPAD may not be limited to the configuration depicted in FIG. 13 . In one embodiment, resistor 1304 and SPAD 1302 are interchangeable. In SPAD core 1301A, SPAD 1302 responds to light. When SPAD 1302 receives a photon, the output of SPAD 1302 changes from the level of VSPAD to 0V and Change back to VSPAD pulse. The output from SPAD 1302 can be filtered by capacitor 1305 and applied to inverter 1306 (inverter 1306 can be used as a combination buffer and latch). In one embodiment, capacitor 1305 may be omitted. The SPAD core 1301A may include a PMOS transistor 1307 that receives the electronic shutter signal 1308 at its gate terminal, while the drain terminal of the transistor 1307 is connected to a capacitor (and the input of the inverter 1306), and the The source terminal of crystal 1307 may receive supply voltage VDD 1309 (or in some embodiments the VPIX voltage). When the electronic shutter signal 1308 is turned off (for example, a logic 0 or low level), the transistor 1307 is turned on and the output 1310 of the inverter 1306 can be maintained at a fixed voltage level (for example, at a logic level low or logic 0 state), regardless of the state of any output received from SPAD 1302. The output from SPAD core 1301A can be applied to PPD core 1311 only when electronic shutter signal 1308 is turned on or active. When the shutter is active (eg, logic 1 level), transistor 1307 is turned off and the output generated by the SPAD can be transmitted (via coupling capacitor 1305) to inverter 1306 and can be on output line 1310 Appears as a positive pulse (low to high).

SPAD core 1301N may be identical in circuit details to SPAD core 1301A, and thus, operational details of SPAD core 1301N are not provided. As shown, SPAD core 1310N may include core-specific SPAD 1312, resistive element 1313, coupling capacitor 1315, inverter 1316, and PMOS transistor 1317, VSPAD voltage 1303 is supplied to SPAD 1312 through resistive element 1313, inverter 1316 is used to lock and output the output generated by SPAD 1312 and PMOS transistor 1317 is used to control the operation of inverter 1316 through shutter input 1308 . Output 1318 of inverter 1316 may be provided to PPD core 1311 for further processing. In some embodiments, the signals VSPAD 1303, VDD 1309, and shutter 1308 can be obtained from an external unit such as Column decoder/driver 1203 as shown in FIG. 12) or any other module (not shown) in pixel processing unit 46 (or processor 19) in FIG. 2 is supplied to each SPAD core 1301A and 1301N. All of the SPAD core specific outputs 1310 and 1318 may collectively form the signal identified using reference numeral 506 in FIG. 5 .

Thus, the electronic shutter signal 1308 ensures that the outputs 1310 and 1318 from the SPAD cores 1301A and 1301N are not only spatially correlated due to the proximity of the SPAD cores 1301A and 1301N in the pixel 1300, but also temporally (or aspects) related. Additional pixel geometries are shown in the exemplary embodiment shown in FIGS. 6A-6C .

Like pixel 700 in FIG. 7, pixel 1300 in FIG. 13 also includes PPD 508, logic unit 1319, first NMOS transistor 1320, second NMOS transistor 1321, third NMOS transistor 1322, fourth NMOS transistor Crystal 1323, fifth NMOS transistor 1324; generates internal input TXEN 1325; receives external input RST signal 1326, VTX signal 1327 (and TX signal 1328), VPIX signal 1329 and SEL signal 1330; has a floating diffusion (FD) node/junction 1331 ; and output PIXOUT signal 510 . However, unlike pixel 700 in FIG. 7, pixel 1300 in FIG. 13 also generates a second TXEN signal (TXENB) 1333, which may be the complement of TXEN signal 1325 and may be supplied to the gate terminal of the sixth NMOS transistor 1334 . A drain terminal of the sixth NMOS transistor 1334 may be connected to a source terminal of the first transistor 1320 , and a source terminal of the sixth NMOS transistor 1334 may be connected to a ground (GND) potential 1335 . TXENB signal 1333 may be used to bring GND potential to the gate terminal of TX transistor 1321 . In the absence of TXENB signal 1333, when TXEN signal 1325 goes low, The gate of TX transistor 1321 may be floating and charge transfer from PPD 508 may not be completely terminated. The TXENB signal 1333 can be used to improve this situation. In addition, the pixel 1300 may further include a storage diffusion (SD) capacitor 1336 and a seventh NMOS transistor 1337 . SD capacitor 1336 may be connected at the junction of the drain terminal of transistor 1321 and the source terminal of transistor 1337, and SD node 1338 may be formed at the junction. NMOS transistor 1337 may receive as input a different second transfer (TX2) signal 1339 at its gate terminal. The drain of transistor 1337 may be connected to FD node 1331 as shown.

In some embodiments, the RST signal, VTX signal, VPIX signal, TX2 signal and SEL signal may be supplied to the pixel 1300 from an external unit such as the column decoder/driver 1203 shown in FIG. 12 . In some embodiments, the SD capacitor 1336 may not be an additional capacitor, but may simply be the junction capacitance of the SD node 1338 . A comparison of FIG. 5 with FIG. 13 shows that in pixel 1300, all SPADs in SPAD cores 1301A, 1301N, etc. can collectively form SPAD 503 in FIG. 5; all non-SPAD circuits from each SPAD core 1301A, 1301N, etc. The elements may collectively form the first control circuit 504 in FIG. 5 ; and all non-PPD circuit elements in the PPD core 502 may form the second control circuit 507 in FIG. 5 .

In pixel 1300 , the charge transfer triggering portion may include SPAD cores 1301A and 1301N (and other such cores) and logic unit 1319 . The charge generation and transfer section may include PPD 508 , NMOS transistors 1320 to 1322 , 1334 and 1337 , and SD capacitor 1336 . The charge collection and output section may include NMOS transistors 1322 to 1324 . Here, it should be noted that the division of various circuit elements into respective parts is for illustrative and discussion purposes only. In some embodiments, such portions may include more, less or different circuit elements than those listed here element.

As previously mentioned, the pixel configuration in FIG. 13 is substantially similar to that in FIG. 7 except for the CDS-based charge collection and output portion. Therefore, for convenience, circuit portions and signals common between the embodiments in FIG. 7 and FIG. 13 , such as transistors 1320 to 1324 and associated inputs such as RST, SEL, VPIX, etc., are not discussed here. It should be understood that CDS is a noise reduction technique for measuring electrical values such as pixel/sensor output voltage (pixout) in a manner that enables removal of undesired offsets. In some embodiments, a row-specific CDS unit (not shown) may be employed in the pixel-row unit 1205 (FIG. 12) to perform correlated double sampling. In CDS, the output of a pixel (such as pixel 1300 in Figure 13) can be measured twice; once under known conditions and once under unknown conditions. The value measured from the known condition can then be subtracted from the value measured from the unknown condition to generate a value having a known relationship to the measured physical quantity (ie, the PPD charge representing the pixel-specific portion of received light). Using CDS, noise can be reduced by removing the pixel's reference voltage (such as, for example, the voltage of the pixel after it is reset) from the pixel's signal voltage at the end of each charge transfer. Therefore, in CDS, a reset/reference value is sampled before the pixel's charge is transferred as an output, and then subtracted from the value after the pixel's charge is transferred.

In the embodiment shown in FIG. 13 , SD capacitor 1336 (or associated SD node 1338 ) stores PPD charge before it is transferred to floating diffusion node 1331, thereby enabling The appropriate reset value was previously established (and sampled) at floating diffusion node 1331 . Therefore, each pixel-specific output (Pixout1 and Pixout2) can be specified in the row-specific CDS unit (not shown in the pixel row unit 1205 (FIG. 12) shown) are processed to obtain a pair of pixel-specific CDS outputs. The pixel-specific CDS output may then be converted to digital values (eg, P1 and P2 values indicated by arrow 1213 in FIG. 12 ) by corresponding row-specific ADC units (not shown) in pixel row unit 1205 . Transistors 1334 and 1337 and TXENB signal 1333 and TX2 signal 1339 in FIG. 13 provide the auxiliary circuit elements needed to facilitate CDS-based charge transfer. In one embodiment, the P1 and P2 values may be generated in parallel using, for example, a pair of identical ADC circuits as part of a row-specific ADC unit. Therefore, the difference between the reset level of the pixout1 signal and the pixout2 signal and the corresponding PPD charge level of the pixout1 signal and the pixout2 signal can be converted into a digital value by the row-parallel ADC and used as the pixel-specific signal value (ie, P1 and P2) is output so that the pixel-specific TOF value of return pulse 37 can be calculated for pixel 1300 based on equation (2). Such calculations may be performed by pixel processing unit 46 or by processor 19 in system 15, as described earlier. Thus, the pixel-specific distance to object 26 (FIG. 2) can also be determined using, for example, equation (3) or equation (4). The pixel-by-pixel charge collection operation may be repeated for all pixel columns in the pixel array 42 . All pixel-specific distance values or pixel-specific range values based on pixels 43 in pixel array 42 may, for example, be generated by processor 19 and displayed on a suitable display or user interface associated with system 15 3D image of object 26 . For example, when a range value is not calculated or when a 2D image is required regardless of the availability of range values, a 2D image of object 26 may be generated by simply adding the P1 and P2 values. In some embodiments, such a 2D image may simply be a grayscale image, for example when using an IR laser.

It should be kept in mind that the pixel configurations shown in Figures 7 and 13 are examples only. Other types of PPD-based pixels with multiple SPADs can also be used to implement the subject matter disclosed herein. Such pixels may include, for example, pixels with a single output (e.g., Figure 7 and PIXOUT line 510 in the embodiment shown in FIG. 13) or a pixel with dual outputs, where the Pixout1 and Pixout2 signals can be output through different outputs in the pixel.

FIG. 14 is a timing diagram 1400 illustrating when pixel 1300 in the embodiment depicted in FIG. 13 is used in a pixel array, such as pixel array 42 in FIGS. 2 and 12 , in accordance with the subject matter disclosed herein Exemplary timings of different signals in the system 15 shown in FIGS. 1 and 2 when measuring the TOF value. The timing diagram 1400 in FIG. 14 is similar to the timing diagram 900 in FIG. 9 , especially in the waveforms of the VTX signal, the shutter signal, the VPIX signal, and the TX signal and for various timing intervals or events such as, for example, the PPD reset setting event, shutter turn-on period, time delay period Tdly _{, etc.} ). Due to the extensive discussion earlier on timing diagram 900 in FIG. 9 , only a brief discussion of the distinguishing features in timing diagram 1400 in FIG. 14 is provided for convenience.

In FIG. 14, various externally supplied signals (such as VPIX signal 1329, RST signal 1326, electronic shutter signal 1308, amplitude modulation signal VTX 1327, and TX2 signal 1339) and the internally generated TXEN signal 1325 are used in accordance with FIG. 13 are identified by the same reference numbers as those used in 13. Similarly, the same reference number 1331 is used to refer to the floating diffusion node 1331 in FIG. 13 and the associated voltage waveforms in the timing diagram shown in FIG. 14 for convenience. The transfer mode (TXRMD) signal 1401 is shown in FIG. 14 but is not shown in FIG. 13 or earlier in the timing diagram shown in FIG. 10 . In some embodiments, the TXRMD signal 1401 may be internally generated by the logic unit 1319 or externally supplied to the logic unit 1319 by a column decoder/driver (eg, column decoder/driver 1203 in FIG. 12 ). In one embodiment, logic unit 1319 may include logic circuitry (not shown) to generate an output based on the G( ) function (FIG. 10) and then compare that output with an internally generated signal (eg, TXRMD signal 1401). logical OR operation to obtain the final TXEN message No. 1325. As shown in FIG. 14, in one embodiment, this internally generated TXRMD signal 1401 can be held low while the electronic shutter is on, but can then be assigned high to cause the TXEN signal 1325 to go logic 1, thereby facilitating the transfer of residual charge in the PPD (at event 1402 in Figure 14).

It should be noted that the PPD preset event 1403, delay time T _dly 1404, TOF period T _tof 1405, shutter off interval 1406 and shutter on or active period T _sh 1407 or 1408 and FD reset event 1409 in FIG. The corresponding events or time periods shown in Figure 9 are similar. Therefore, no additional discussion of these parameters is provided. First, the FD reset event 1409 causes the FD signal 1331 to go high, as shown. After PPD 508 is preset low, SD node 1338 is reset high. More specifically, during PPD default event 1403, TX signal 1328 may be high, TX2 signal 1339 may be high, RST signal 1326 may be high, and VPIX signal 1329 may be low to fill PPD 508 with electrons and The PPD 508 is preset to zero volts. Thereafter, TX signal 1328 may go low, but TX2 signal 1339 and RST signal 1326 may briefly remain high, which together with high VPIX signal 1329 may reset SD node 1338 high and remove electrons from SD capacitor 1336 . At the same time, FD node 1331 is reset (after FD reset event 1409). The voltage at SD node 1338 or the SD reset event is not shown in FIG. 14 .

In contrast to the embodiment in FIGS. 7 and 9 , in the embodiment depicted in FIGS. 13 and 14 , when shutter 1308 is active and VTX signal 1327 ramps up (as shown on TX waveform 1328 ), the PPD charge is amplitude modulated and first transferred to SD node 1338 (via SD capacitor 1336). When at least two SPADs in pixel 1300 (FIG. 13) detect photons during shutter-on period 1408, TXEN signal 1325 goes low and initial charge from PPD 508 to SD node 1338 Transfer stopped. During the first readout period 1412, the transferred charge stored at SD node 1338 may be read out on Pixout line 510 (output as Pixout1). During the first readout period 1412, the RST signal 1326 may be assigned high briefly after the electronic shutter 1308 is deactivated or turned off to reset the floating diffusion node 1331. Thereafter, TX2 signal 1339 may be pulsed high to transfer charge from SD node 1338 to floating diffusion node 1331 while TX2 1339 signal is high. The floating diffusion voltage waveform 1331 illustrates the charge transfer operation. The transferred charge (as the Pixout1 voltage) may then be read out through Pixout line 510 using SEL signal 1330 (not shown in FIG. 14 ) during first readout period 1412 .

During the first readout interval 1412, after the initial charge is transferred from the SD node to the FD node and the TX2 signal 1339 returns to a logic low level, the TXRMD signal 1401 can be assigned (pulsed) high to switch on the TXEN input A high pulse is generated on 1325, which in turn generates a high pulse on TX input 1328 to enable transfer of remaining charge in PPD 508 to SD node 1338 (via SD capacitor 1336), as indicated by reference numeral 1402 in FIG. Thereafter, when the RST signal 1326 is briefly assigned high again, the FD node 1331 can be reset again. A second RST high pulse may define a second readout period 1413, where the TX2 signal 1339 may be pulsed high again to transfer the remaining charge on the PPD 508 from the SD node 1338 while TX2 is high (at event 1402 ) to the floating diffusion node 1331. The floating diffusion voltage waveform 1331 illustrates the second charge transfer operation. The transferred remaining charge (as the Pixout2 voltage) can then be read out through Pixout line 510 using SEL signal 1332 (not shown in FIG. 14 ) during second readout period 1413 . As mentioned earlier, the PIXOUT1 and PIXOUT2 signals may be converted by appropriate ADC units (not shown) into corresponding digital values P1 and P2. In some embodiments, These values P1 and P2 can be used in equation (3) or equation (4) to determine the pixel-specific distance/pixel-specific range between pixel 1300 and object 26 . The SD-based charge transfer depicted in FIG. 14 enables generation of a pair of pixel-specific CDS outputs, as described earlier with reference to the discussion of FIG. 13 . Signal processing based on CDS achieves additional noise reduction.

FIG. 15 illustrates a block diagram of an exemplary embodiment of a time-resolved sensor 1500 in accordance with the subject matter disclosed herein. Time resolved sensor 1500 may include SPAD circuit 1501 , logic circuit 1503 and PPD circuit 1505 .

The SPAD circuit 1501 may include a SPAD, a first input, a second input, a third input and an output, the SPAD is used to detect photons, the first input is used to receive the VSPAD voltage, and the second input is used to receive the shutter signal to The opening and closing of the electronic shutter is controlled, the third input is used to receive the V _DD voltage, and the output is used to output a detection event (DE) signal. In response to receiving a photon, SPAD circuit 1501 outputs a pulse signal that quickly changes from the VSPAD voltage to a voltage below the SPAD breakdown voltage and then more slowly returns to the VSPAD voltage.

The logic circuit 1503 may include a first input connected to the DE signal output from the SPAD circuit 1501, a second input for receiving the TXRMD signal in the PPD of the PPD circuit 1505, and an output. The remaining charge is fully transferred to the FD node, which is used to output the TXEN signal.

The PPD circuit 1505 may include a first input, a second input, a third input, a fourth input, a fifth input, and a PIXOUT output, the first input is connected to the TXEN signal output from the logic circuit 1503, and the second input is connected to In receiving the VTX signal to transfer charge partially or completely from the PPD of the PPD circuit 1505 to the FD node in the PPD circuit 1505, the third input is used to receive the RST signal to transfer the charge to the FD node in the PPD circuit 1505. The charge in the node is reset and the charge in the PPD is preset, the fourth input is used to receive the VPIX voltage of the PPD circuit 1505, and the fifth input is used to receive the SEL signal to enable the readout of the PIXOUT1 signal (represented by charge on the FD node) or the PIXOUT2 signal (representing the remaining charge in the PPD), the PIXOUT output is used to output the PIXOUT1 signal and the PIXOUT2 signal in response to the SEL signal.

FIG. 16 shows a schematic diagram of an exemplary embodiment of a SPAD circuit 1501 of a time-resolved sensor 1500 according to the subject matter disclosed herein. In one embodiment, SPAD circuit 1501 may include resistor 1601 , SPAD 1603 , capacitor 1605 , p-type MOSFET transistor 1607 and buffer 1609 . Resistor 1601 may include a first terminal for receiving a VSPAD voltage and a second terminal. SPAD 1603 may include an anode connected to ground potential and a cathode connected to the second terminal of resistor 1601 . In another embodiment, resistor 1601 and SPAD 1603 may be swapped. SPAD 1603 can respond to light. In response to receiving photons, SPAD 1603 outputs a pulse signal that quickly changes from the VSPAD voltage to a voltage below the breakdown voltage and then more slowly returns to the VSPAD voltage. In one example, the breakdown voltage may be a certain threshold voltage.

Capacitor 1605 may include a first terminal connected to the cathode of SPAD 1603 and a second terminal. In alternative embodiments, capacitor 1605 may be omitted. The p-type MOSFET transistor 1607 may include a first S/D (source/drain) terminal, a gate, and a second S/D terminal, the first S/D terminal being connected to the second terminal of the capacitor 1605, so The gate is used to receive the shutter signal, and the second S/D terminal is used to receive the VPIX voltage (V _DD ). The buffer 1609 may include an input connected to the second terminal of the capacitor 1605 and an output for outputting the DE signal. The DE signal may correspond to the DE output of the SPAD circuit 1501 . In an alternate embodiment, buffer 1609 may be an inverter.

FIG. 17 is a schematic diagram of an exemplary embodiment of a logic circuit 1503 of a time-resolved sensor 1500 according to the subject matter disclosed herein. The logic circuit 1503 may include a latch 1701 and a two-input OR gate 1703 .

The latch 1701 may include an input connected to the DE signal output from the SPAD circuit 1501 and an output. In response to the DE signal, the latch 1701 outputs a logic signal such as changing from a logic 1 to a logic 0 and remaining at a logic 0. In other words, the latch 1701 converts a pulse-type signal into a signal that changes from a logic 1 to a logic 0 and remains at a logic 0 without returning to a logic 1 until reset. The latch output can be triggered by the leading edge of the DE signal, where the leading edge can be either positive or negative depending on the design of the SPAD circuit 1501 .

The two-input OR gate 1703 may include a first input, a second input and an output, the first input is connected to the output of the latch 1701, the second input is used to receive the TXRMD signal, and the output is used to output the TXEN signal . The two-input OR gate 1703 performs a logical OR function and outputs the result as a TXEN signal. Specifically, if the photon is received by the SPAD circuit 1501 when the shutter signal is a logic 1 or if the TXRMD signal is a logic 1, the output of the two-input OR gate 1703 becomes a logic 1, which is the PPD of the PPD circuit 1505. Occurs when the remaining charge will be completely transferred to the FD node to be read out as the PIXOUT2 signal.

FIG. 18 shows a schematic diagram of an exemplary embodiment of a PPD circuit 1505 of a time-resolved sensor 1500 according to the subject matter disclosed herein. The PPD circuit 1505 may include a PPD 1801 , a first transistor 1803 , a second transistor 1805 , a third transistor 1807 , a fourth transistor 1809 and a fifth transistor 1811 .

PPD 1801 may include an anode and a cathode, the anode being connected to ground potential. The PPD 1801 can store charge in a similar manner to a capacitor. In one embodiment, the PPD 1801 can be covered and thus not responsive to light, and can be used as a TCC rather than a photosensitive element.

The first transistor 1803 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is connected to the TXEN signal output of the logic circuit 1503, and the first S/D terminal is used for Receive VTX signal. The first transistor 1803 can receive the VTX signal and enable the VTX signal to pass through the first transistor 1803 under the control of the TXEN signal to output the TX signal at the second S/D terminal of the first transistor 1803 .

The second transistor 1805 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is connected to the second S/D terminal of the first transistor 1803, the first S/D terminal The /D terminal is connected to the cathode of the PPD 1801. The second transistor 1805 can receive the TX signal on the gate terminal and transfer the charge on the PPD 1801 on the source terminal to the drain terminal connected to the FD node. There may be parasitic capacitance between the FD node and ground, which is not indicated in FIG. 18 . In an embodiment, a physical capacitor may also be connected between the FD node and the ground.

The third transistor 1807 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is used to receive the RST signal, the first S/D terminal is used to receive the VPIX voltage, The second S/D terminal is connected to the second S/D terminal of the second transistor 1805 .

The fourth transistor 1809 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is connected to the second S/D terminal of the second transistor 1805, the first S/D terminal The /D terminal is connected to the first S/D terminal of the third transistor 1807 .

The fifth transistor 1811 may include a gate terminal, a first S/D terminal and a first Two S/D terminals, the gate terminal is used to receive the SEL signal, the first S/D terminal is connected to the second S/D terminal of the fourth transistor 1809, and the second S/D terminal is PPD PIXOUT output of circuit 1505 . The fifth transistor 1811 can receive the SEL signal to select a pixel to read out the charge in the FD node (as PIXOUT1 ) or the remaining charge in the PPD 1801 (as PIXOUT2 ).

Charge transfer from PPD 1801 to FD node is controlled by TX signal. In one embodiment, the VTX signal is coupled through the first transistor 1803 to become a TX signal. The VTX signal ramps up to transfer charge from the PPD 1801 to the FD node faster and faster. The amount of charge transferred from the PPD 1801 to the FD node can be a function of the level of the TX signal, and the ramp of the TX signal can be a function of time. Thus, charge transferred from PPD 1801 to FD node may be a function of time. If the second transistor 1805 is turned off in response to detection of incoming photons by the SPAD circuit 1501 during the transfer of charge from the PPD 1801 to the FD node, the transfer of charge from the PPD 1801 to the FD node ceases. Both the amount of charge transferred to the FD node and the amount of charge remaining in the PPD 1801 can be related to the TOF of the incoming photon. The transfer of charge from the PPD 1801 to the FD node based on the TX signal and upon detection of incoming photoelectricity can be considered to provide single-ended to differential conversion of charge to time.

The fourth transistor 1809 operates to convert the charge stored on the FD node into a voltage at the second S/D terminal of the fourth transistor 1809 . The SEL signal is used to select the pixel to read out the PIXOUT1 signal corresponding to the charge that has been transferred to the FD node or subsequently read out the PIXOUT1 signal corresponding to the charge remaining in the PPD 1801 after it has been transferred to the FD node PIXOUT2 signal. In one embodiment, the ratio of the PIXOUT1 signal to the sum of the PIXOUT1 signal plus the PIXOUT2 signal is proportional to the difference between the TOF of the light signal received by the pixel and the delay time, e.g. Expressed by the ratio in equation (2). In embodiments where the light pulse is emitted after the VTX signal begins ramping up, the delay time may be negative.

For time-resolved sensor 1500, the ratio expressed in equation (2) can be used to determine the depth or extent of an object, and if PIXOUT1 + PIXOUT2 does not vary from measurement to measurement, then the ratio is is less sensitive to changes. In one embodiment, the VTX signal may ideally be linear, and may ideally be uniform across all the different pixels of the TOF pixel array. In practice, however, the VTX signal that can be applied to different pixels of a TOF pixel array can vary from pixel to pixel, thereby introducing errors in range measurements that depend on the pixel-to-pixel distance between pixels. VTX signal changes and can also vary from measurement to measurement.

In one embodiment, the first transistor 1803 , the second transistor 1805 , the third transistor 1807 , the fourth transistor 1809 and the fifth transistor 1811 can be n-type MOSFETs or p-type MOSFETs respectively. However, the subject matter disclosed herein is not limited to the use of n-type MOSFETs or p-type MOSFETs, as any other suitable transistor may be used.

FIG. 19 illustrates an exemplary relative signal timing diagram 1900 for the time-resolved sensor 1500 of FIG. 15 in accordance with the subject matter disclosed herein. In FIG. 19, the RST, VTX, and TX signals each go high (logic 1) and then return to 0 (logic 0) to reset the PPD circuit 1505 during the shutter turn-off (initialization) cycle. TXEN signal is high. At this initialization period, PPD 1801 may be filled to its full well capacity with charge. The VTX signal and the TX signal go low to turn off the second transistor 1805 of the PPD circuit 1505 . The VPIX voltage goes high, which resets the FD node. When the RST signal returns to 0 or shortly thereafter, a light pulse is emitted towards the object. The VTX signal then begins ramping up and the shutter signal goes high to begin the shutter on cycle.

As the VTX signal ramps up, the TX signal also ramps up and the charge on the FD node begins to decrease in response to the TX signal. The returning light pulse causes the TXEN signal to go low (logic 0), thereby stopping the transfer of charge between the FD node and the PPD 1801 .

The delay time T _dly represents the time between when the light pulse starts to be emitted and when the TX signal starts to ramp up. The time-of-flight T _tof represents the time between when the light pulse is started to be emitted and when the return signal is received. The electronic shutter time T _sh represents the time from when the electronic shutter is opened to when the electronic shutter is closed (shutter ON period). In one embodiment, the electronic shutter time T _sh may be less than or equal to the ramp time of the VTX signal.

The transferred charge is read out as the PIXOUT1 signal during the read charge transfer cycle. While the shutter signal is low, the RST signal goes high a second time to reset the charge on the FD node, then the TXRMD, TXEN and TX signals go high to transfer the remaining charge on the PPD 1801 to the FD node for It is read as PIXOUT2 signal.

FIG. 20 shows a block diagram of another exemplary embodiment of a time-resolved sensor 2000 according to the subject matter disclosed herein. The time-resolved sensor 2000 may include a SPAD circuit 2001 , a logic circuit 2003 and a second PPD circuit 2005 .

The SPAD circuit 2001 may include a SPAD, a first input, a second input, a third input and an output, the SPAD is used to detect photons, the first input is used to receive the VSPAD voltage, and the second input is used to receive the shutter signal to The opening and closing of the electronic shutter is controlled, the third input is used to receive a VDD voltage (V _DD ), and the output is used to output a detection event (DE) signal. In response to receiving a photon, SPAD circuit 2001 outputs a pulse signal that goes from VSPAD to 0 quickly and slowly back to VSPAD. In one embodiment, the SPAD circuit 2001 may be the same as the SPAD circuit 1501 shown in FIG. 15 .

The logic circuit 2003 may include a first input connected to the DE output of the SPAD circuit 2001, a second input for receiving the TXRMD signal to be placed in the PPD of the second PPD circuit 2005, and an output. The remaining charge is completely transferred and the output is used to output the TXEN signal. In one embodiment, the logic circuit 2003 may be the same as the logic circuit 1503 shown in FIG. 15 .

The second PPD circuit 2005 may include a first input, a second input, a third input, a fourth input, a fifth input and a sixth input, the first input is connected to the TXEN signal output from the logic circuit 2003, the first input Two inputs are connected to the second input of the logic circuit 2003 to receive the TXRMD signal, and the third input is used to receive the VTX signal to transfer charge partially or completely from the PPD of the second PPD circuit 2005 into the second PPD circuit 2005 The first floating diffusion (FD1) node, the fourth input is used to receive the RST signal to reset the charge in the FD1 node and preset the charge in the PPD, and the fifth input is used to receive the second PPD VPIX voltage of circuit 2005, the sixth input for receiving the SEL signal to enable reading on the PIXOUT1 output the PIXOUT1 signal corresponding to the charge on the FD1 node and to enable reading on the PIXOUT2 output corresponding to the charge on the second PPD circuit The PIXOUT2 signal corresponding to the remaining charge in the PPD of 2005.

FIG. 21 is a schematic diagram of an exemplary embodiment of a second PPD circuit 2005 of a time-resolved sensor 2000 according to the subject matter disclosed herein. The second PPD circuit 2005 may include a PPD 2101, a first transistor 2103, a second transistor 2105, a third transistor 2107, a fourth transistor 2109, a fifth transistor 2111, a sixth transistor 2113, and a seventh transistor 2115, eighth transistor 2117 and ninth transistor 2119.

PPD 2101 may include an anode and a cathode, the anode being connected to ground potential. The PPD 2101 can store charge in a similar manner to a capacitor. In one embodiment, the PPD 2101 can be covered and thus not responsive to light, and can be used as a TCC rather than a photosensitive element.

The first transistor 2103 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is connected to the output of the logic circuit 2003 to receive the TXEN signal output, the first S/D The terminal is used to receive the VTX voltage to control the transfer of charge from the PPD 2101 .

The second transistor 2105 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is connected to the second S/D terminal of the first transistor 2103 to receive the TX signal from The PPD 2101 transfers the charge, the first S/D terminal is connected to the cathode of the PPD 2101, the second S/D terminal is connected to the first floating diffusion node FD1, and the charge is transferred from the PPD 2101 to the first floating diffusion (FD1) node. The FD1 node may have a first capacitance. There may be parasitic capacitance between the FD1 node and ground, which is not indicated in FIG. 21 . In an embodiment, a physical capacitor may also be connected between the FD1 node and the ground. The charge transferred from the PPD 2101 to the FD1 node through the second transistor 2105 is controlled by the TX signal.

The third transistor 2107 may include a gate terminal connected to the FD1 node and connected to the second S/D terminal of the second transistor 2105, a first S/D terminal and a second S/D terminal, The first S/D terminal is used to receive the VPIX voltage. The third transistor 2107 is operable to convert the charge stored on the FD1 node to a voltage at the second S/D terminal of the third transistor 2107 .

The fourth transistor 2109 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is used to receive the RST signal to assign the charge level of the FD1 node, the first The S/D terminal is used to receive the VPIX voltage, the first The second S/D terminal is connected to the second S/D terminal of the second transistor 2105 .

The fifth transistor 2111 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is used to receive the SEL signal to read the charge on the FD1 node, the first S/D The D terminal is connected to the second S/D terminal of the third transistor 2107, and the second S/D terminal is connected to the pixel output PIXOUT1 data line to output a voltage corresponding to the charge on the FD1 node as a PIXOUT1 signal.

The sixth transistor 2113 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is used to receive the TXRMD signal to completely transfer the charge remaining in the PPD 2101 to the second floating diffusion Node FD2, the first S/D terminal is connected to the cathode of the PPD 2101, and the second S/D terminal is connected to the FD2 node. The FD2 node may have a second capacitance. There may be parasitic capacitance between the FD2 node and ground, which is not indicated in FIG. 21 . In one embodiment, a physical capacitor may also be connected between the FD2 node and the ground. In one embodiment, the second capacitance of the FD2 node may be equal to the first capacitance of the FD1 node. Any remaining charge in the PPD 2101 can be transferred to the FD2 node through the sixth transistor 2113 .

The seventh transistor 2115 may include a gate terminal connected to the second S/D terminal of the sixth transistor 2113 and connected to the FD2 node, a first S/D terminal and a second S/D terminal, The first S/D terminal is used to receive the VPIX voltage. The seventh transistor 2115 is operable to convert the charge stored on the FD2 node to a voltage at the second S/D terminal of the seventh transistor.

The eighth transistor 2117 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is used to receive the RST signal to assign the charge level of the FD2 node, the first The S/D terminal is used to receive the VPIX signal, and the second S/D terminal is connected to the second S/D terminal of the sixth transistor 2113 .

The ninth transistor 2119 may include a gate terminal, a first S/D terminal, and a second S/D terminal, and the gate terminal is used to receive the SEL signal to select a pixel to read out a voltage corresponding to the charge in the FD2 node , the first S/D terminal is connected to the second S/D terminal of the seventh transistor 2115, and the second S/D terminal is connected to the pixel output PIXOUT2 data line to output the charge corresponding to the FD2 node voltage as PIXOUT2 signal.

In one embodiment, the VTX signal (and the TX signal) can be ramped up to transfer charge from the PPD 2101 to the FD1 node. The amount of charge transferred from the PPD 2101 to the FD node can be a function of the level of the TX signal, and the ramp of the TX signal can be a function of time. Thus, charge transferred from PPD 2101 to FD1 node may be a function of time. If during charge transfer from PPD 2101 to FD1 node, second transistor 2105 is turned off in response to detection of an incoming photon by SPAD circuit 2001, transfer of charge from PPD 2101 to FD1 node stops, and charge transferred to FD1 node Both the amount of charge and the amount of charge remaining in the PPD 2101 are related to the TOF of incoming photons. The transfer of charge from the PPD 2101 to the FD1 node based on the TX signal and based on the detection of incoming photons provides a single-ended to differential conversion of charge to time.

For time-resolved sensor 2000, the ratio expressed in equation (2) can be used to determine the depth or range of an object, and if PIXOUT1 + PIXOUT2 does not vary from measurement to measurement, then the ratio is is less sensitive to changes. In one embodiment, the VTX signal may ideally be linear, and may ideally be uniform across all the different pixels of the TOF pixel array. In practice, however, the VTX signal that can be applied to different pixels of a TOF pixel array can vary from pixel to pixel, thereby introducing errors in range measurements that depend on the pixel-to-pixel distance between pixels. VTX signal changes and can also vary from measurement to measurement.

In one embodiment, the first transistor 2103, the second transistor 2105, the third transistor 2107, the fourth transistor 2109, the fifth transistor 2111, the sixth transistor 2113, the seventh transistor 2115, the eighth transistor Transistor 2117 and ninth transistor 2119 may each be an n-type MOSFET or a p-type MOSFET; however, any other suitable transistor may be used.

FIG. 22 illustrates an exemplary relative signal timing diagram 2200 for a time-resolved sensor 2000 in accordance with the subject matter disclosed herein. The signal timing diagram shown in FIG. 22 is similar to the signal timing diagram shown in FIG. 19 and the similarities have been explained with reference to FIG. 19 . The difference in the signal timing diagram shown in FIG. 22 is that the FD2 signal is included, and at the end of the shutter on period, the remaining charge on the PPD 2101 is transferred to the FD2 node through the operation of the TXRMD signal. In addition, PIXOUT1 signal and PIXOUT2 signal can be read out at the same time.

It should be noted that the second PPD circuit 2005 relies on constant full well capacity to determine the maximum range; however, actual implementations of the time-resolved sensor 2000 may experience differences based on thermal noise between different second PPD circuits 2005. Full well variation of PPD 2101. Additionally, the VTX signal can have different slopes (slopes) based on the position of the pixels in the pixel array. That is, the slope (slope) of the VTX signal at a pixel may vary depending on the proximity of the pixel to the source of the VTX signal.

FIG. 23 illustrates a block diagram of yet another exemplary embodiment of a time-resolved sensor 2300 in accordance with the subject matter disclosed herein. Time-resolved sensor 2300 may include one or more SPAD circuits 2301 a through 2301 n , logic circuit 2303 , and third PPD circuit 2305 .

In one embodiment, each of the one or more SPAD circuits 2301a through 2301n may include SPADs 2311a through 2311n, resistors 2313a through 2313n, capacitors 2315a through 2315n, p-type MOSFET transistors 2317a through 2317n and buffers 2319a through 2319n. SPADs 2311a through 2311n may include anodes and cathodes, the anodes being connected to ground potential. Resistors 2313a through 2313n may include a first terminal for receiving the VSPAD voltage and a second terminal connected to the cathodes of SPADs 2311a through 2311n. In another embodiment, the SPADs 2311a-2311n and the resistors 2313a-2313n are interchangeable. SPADs 2311a through 2311n may respond to light. In response to receiving photons, SPADs 2311a-2311n output pulse signals that quickly change from the VSPAD voltage to a voltage below the breakdown voltage and then return more slowly to the VSPAD voltage.

Capacitors 2315a through 2315n may include a first terminal connected to a cathode of SPAD 2311a through 2311n and a second terminal. In alternative embodiments, capacitors 2315a-2315n may be omitted. P-type MOSFET transistors 2317a through 2317n may include a first S/D terminal connected to a second terminal of capacitors 2315a through 2315n, a gate and a second S/D terminal, the gate Used to receive the shutter signal, the second S/D terminal is used to receive the VPIX voltage (V _DD ). Buffers 2319a through 2319n may include inputs connected to second terminals of capacitors 2315a through 2315n and inverting outputs that may output DE signals corresponding to outputs of SPAD circuits 2301a through 2301n. In an alternate embodiment, buffers 2319a-2319n may be non-inverting.

The logic circuit 2303 may include an input connected to a DE signal output of each of the one or more SPAD circuits 2301a through 2301n and an output, the output outputs a TXEN signal and a TXENB signal, the TXENB signal may be The inversion of the TXEN signal.

The third PPD circuit 2305 may include a capacitive device SC, a first transistor 2351, a second transistor 2353, a third transistor 2355, a fourth transistor 2357, a Five transistors 2359, sixth transistors 2361, seventh transistors 2363, eighth transistors 2365, ninth transistors 2367, tenth transistors 2369, eleventh transistors 2371, twelfth transistors 2373 and Thirteen transistors 2375.

The capacitive means SC may comprise a first terminal and a second terminal, the first terminal being connected to ground potential. The capacitive means SC can store charge in a similar manner to a capacitor. In one embodiment, the capacitive means SC may be a capacitor. In another embodiment, the capacitive means SC may be a PPD which may be covered so that it does not respond to light. In either of these two embodiments, capacitive means SC may be used as part of the TCC.

The first transistor 2351 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is connected to the RST signal, the first S/D terminal is connected to ground potential, the The second S/D terminal is connected to the second terminal of the capacitive means SC.

The second transistor 2353 may include a gate terminal connected to the TXA signal, a first S/D terminal connected to the first floating diffusion FD1, and a second S/D terminal. node, the second S/D terminal is connected to the second S/D terminal of the first transistor 2351 and the second terminal of the capacitive means SC. The first floating diffusion FD1 node is represented by a capacitor symbol in FIG. 23 . There may be parasitic capacitance between the FD1 node and ground, which is not indicated in FIG. 23 . In an embodiment, a physical capacitor may also be connected between the FD1 node and the ground.

The third transistor 2355 may include a gate terminal connected to the FD1 node and the first S/D terminal of the second transistor 2353, a first S/D terminal and a second S/D terminal, the The first S/D terminal is connected to the VPIX voltage. The third transistor 2355 is operable to convert the charge on the FD1 node to a voltage at the second S/D terminal of the third transistor 2355 .

The fourth transistor 2357 may include a gate terminal connected to the RST signal, a first S/D terminal connected to the VPIX voltage, and a second S/D terminal connected to the second S/D terminal. The S/D terminal is connected to the second S/D terminal of the first transistor 2351 and the second terminal of the capacitive means SC.

The fifth transistor 2359 may include a gate terminal connected to the TXEN signal, a first S/D terminal connected to the VTX signal, and a second S/D terminal. The second S/D terminal is connected to the gate terminal of the second transistor 2353 .

The sixth transistor 2361 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is connected to the TXENB signal, the first S/D terminal is connected to ground potential, the The second S/D terminal is connected to the gate terminal of the second transistor 2353 and the second S/D terminal of the fifth transistor 2359 .

The seventh transistor 2363 may include a gate terminal, a first S/D terminal connected to the SEL signal, and a second S/D terminal connected to the third transistor 2355. The second S/D terminal of the second S/D terminal is connected to the pixel output line PIXA.

The eighth transistor 2365 may include a gate terminal connected to the TXB signal, a first S/D terminal connected to the TXB signal, and a second S/D terminal connected to the second floating diffusion FD2 node, the second S/D terminal is connected to the second S/D terminal of the first transistor 2351 , the second terminal of the capacitive means SC and the second S/D terminal of the second transistor 2353 . The second floating diffusion FD2 node is represented by a capacitor symbol in FIG. 23 . There may be parasitic capacitance between the FD2 node and ground, which is not indicated in FIG. 23 . In one embodiment, a physical capacitor may also be connected between the FD2 node and the ground.

The ninth transistor 2367 may include a gate terminal connected to the FD2 node and the first S/D terminal of the eighth transistor 2365, a first S/D terminal and a second S/D terminal, the The first S/D terminal is connected to the VPIX voltage. The ninth transistor 2367 is operable to convert the charge on the FD2 node to a voltage at the second S/D terminal of the ninth transistor 2367 .

The tenth transistor 2369 may include a gate terminal connected to the RST signal, a first S/D terminal connected to the VPIX voltage, and a second S/D terminal. The second S/D terminal is connected to the second S/D terminal of the first transistor 2351 , the second terminal of the capacitive means SC and the second S/D terminal of the eighth transistor 2365 .

The eleventh transistor 2371 may include a gate terminal, a first S/D terminal and a second S/D terminal, the gate terminal is connected to the TXENB signal, and the first S/D terminal is connected to the VTX signal, so The second S/D terminal is connected to the gate terminal of the eighth transistor 2365.

The twelfth transistor 2373 may include a gate terminal connected to the TXEN signal, a first S/D terminal connected to a ground potential, and a second S/D terminal, so that The second S/D terminal is connected to the gate terminal of the eighth transistor 2365 and the second S/D terminal of the eleventh transistor 2371.

The thirteenth transistor 2375 may include a gate terminal, a first S/D terminal connected to the SEL signal, and a second S/D terminal, the first S/D terminal connected to the ninth transistor The second S/D terminal of 2367, the second S/D terminal is connected to the pixel output line PIXB.

FIG. 24 illustrates an exemplary relative signal timing diagram 2400 for a time-resolved sensor 2300 in accordance with the subject matter disclosed herein. The signal timing diagram shown in Figure 24 is consistent with Figure 19 and The signal timing diagram shown in FIG. 22 is similar and similarities have been explained with reference to FIG. 19 . The signal timing diagram shown in FIG. 24 is different from the signal timing diagram shown in FIG. 22 in that it does not include the TXRMD signal and the TX signal, but includes the TXENB signal, the TXA signal and the TXB signal.

In the signal timing diagram shown in FIG. 24, the TXENB signal is the inversion of the TXEN signal. When the shutter signal is active high, the TXEN signal is active and the VTX signal passes through the fifth transistor 2359, thereby making the TXA signal active. The charge on the capacitive device SC is transferred to the FD1 node through the second transistor 2353 . At the same time, the ground potential passes through the twelfth transistor 2373, thereby deactivating the TXB signal.

When a detection event (DE) occurs, the TXEN signal becomes inactive and the TXENB signal becomes active. When the TXEN signal becomes inactive, the TXA signal also becomes inactive and charge transfer from the capacitor device SC to the FD1 node through the second transistor 2353 stops. When the TXENB signal becomes active, the TXB signal becomes active and charge is transferred from the capacitor device SC to the FD2 node through the eighth transistor 2365 .

When the shutter signal ends, the TXB signal becomes inactive and charge transfer from the capacitive device SC to the FD2 node through the eighth transistor 2365 stops. The respective voltages associated with the charge on the FD1 node and the FD2 node are read from the PIXA output line and the PIXB output line.

It should be noted that variations in the slope of the VTX signal and variations in the capacitance of the capacitive device SC from pixel to pixel will not cause range measurement errors as long as the second transistor 2353 (TXA) and the eighth transistor 2365 (TXB) are at It is enough to operate in linear mode during the active shutter signal period.

FIG. 25 shows a flowchart of a method 2500 of resolving time using a time-resolving sensor 2300 according to the subject matter disclosed herein. The method starts at 2501 . At 2502, an active shutter signal is generated. At 2503, one or more photons (detection events (DE)) incident on at least one SPAD circuit 2301a through 2301n are detected during the active shutter signal, wherein the one or more detected photons are from the object reflection. At 2504, an output signal is generated based on the detection event (DE). At 2505, a first enable signal (eg, TXEN) and a second enable signal (eg, TXENB) are generated based on the output signal related to the detection event (DE). In one embodiment, the first enable signal becomes active in response to the start of the active shutter signal and becomes inactive in response to the output signal, and the second enable signal becomes active in response to the output signal and responds to It becomes inactive at the end of the active shutter signal.

At 2506, if the first enable signal is active, the charge on the capacitive device SC is transferred to the first floating diffusion (FD1) node to form a first charge on the first floating diffusion (FD1) node. At 2507, if the second enable signal is active, the remaining charge on the capacitive device SC is transferred to the second floating diffusion (FD2) node to form a second charge on the second floating diffusion (FD2) node. At 2508, a first voltage based on the first charge and a second voltage based on the second charge are output. The first ratio of the first voltage to the sum of the first voltage and the second voltage is proportional to the time-of-flight of the one or more detected photons, and the ratio of the second voltage to the sum of the first voltage and the second voltage The second ratio is proportional to the time-of-flight of the one or more detected photons. At 2509, the method ends.

In one embodiment, transferring the first charge and the second charge further includes varying the VTX signal (or drive signal) according to a ramp function, wherein the VTX signal (or drive signal) is responsive to detecting the detected one or more The start time of the light pulse of photons begins to change until the end of the active shutter signal. In addition, the electric The charge on the capacitor device is transferred to the first floating diffusion (FD1) node to form the first charge on the first floating diffusion (FD1) node may also be based on the VTX signal (or driving signal) when the first enable signal is active level, and transferring the remaining charge on the capacitive device to the second floating diffusion (FD2) node to form a second charge on the second floating diffusion (FD2) node can also be based on the VTX signal when the second enable signal is active (or drive signal) level.

In another embodiment, the first ratio of the first voltage to the sum of the first voltage and the second voltage may also be proportional to the time of flight of the one or more detected photons minus the delay time. Similarly, the second ratio of the second voltage to the sum of the first voltage and the second voltage may also be proportional to the time-of-flight of the one or more detected photons minus the delay time comprising the photon The time between the start of the transmission time of the pulse and the time when the VTX signal (or drive signal) starts to change.

LiDAR (light detection and range measurement) systems using SPADs typically do not offer light intensity based imaging capabilities. Intensity imaging together with range information can significantly improve object discrimination performance in advanced driver-assistance system (ADAS) and autonomous driving applications. The subject matter disclosed herein provides an imaging system that provides both range information and intensity imaging information. Both range and intensity images are generated from the same source, so there are no image alignment issues and/or complex fusion algorithms are required. Embodiments of pixels disclosed herein are configured as TCCs. Alternatively, pixels configured as a time-to-digital converter (TDC) may also be used, but may provide an image with a smaller spatial resolution than pixels configured as a TCC. That is, pixels configured as TCC are smaller and may provide a pixel array with higher resolution than pixel arrays using pixels configured as TDC.

One exemplary embodiment of an imaging system that can provide range information and light intensity information is imaging system 15 depicted in FIG. 1 . Pixel array 42 may include embodiments of TCC pixels disclosed herein, such as pixel 43 depicted in FIG. 5 , pixel 700 depicted in FIG. 7 , pixel 1300 depicted in FIG. 13 , the pixel 2100 depicted in FIG. 21 and/or the pixel 2300 depicted in FIG. 23 . The light source 22 is controlled to provide a point scan (such as disclosed in connection with FIGS. 3 and 4 ), which is synchronized with the pixel processing unit 46 . Multiple point scans may be repeated to provide a statistical average of both the range information and the light intensity information. The light intensity information may be used to determine the reflectivity of objects in the field of view of imaging system 15 . In an alternative embodiment, light source 22 may be controlled to illuminate the entire scene. If multiple light pulses are projected and captured, a histogram can be formed for each pixel, and by summing the bins around the peaks in the histogram, a grayscale image can be produced, which High-order imagery can be used to significantly improve object discrimination performance in ADAS and autonomous driving applications. It should be kept in mind that alternative embodiments may use pixels configured to provide a TDC output.

The reflectance of each spot captured by the image sensor unit 24 can be determined based on the distance and grayscale value of the spot. Grayscale images can also be generated in the absence of laser pulses. By summing multiple frames together, image sensor unit 24 can operate as a quantum image sensor that captures at most one photon per pixel per frame. High dynamic range imaging is achieved when multiple bit planes (ie frames) are added together. By using both 3D images and 2D images generated from the same image sensor unit for object recognition, complicated image fusion processing can be avoided and the recognition performance can be improved.

In one embodiment, the peak value of the histogram of the arrival time (ie detection time) of photons detected by the pixel can be directly used to generate the gray scale value of the pixel. The window width may be the same as the projected laser or light, the full width at half maximum (FWHM) of the pulse. A histogram of photon detection times can be formed and the group corresponding to the peak number of detected photons can be used to estimate the surface reflectivity S of the object at the point where the light pulse has been reflected. Alternatively, the histogram of pixels can be convolved with the trigger waveform output from the SPAD, and then the maximum detected photon count selected.

FIG. 26A depicts an exemplary trigger waveform 2600 output from a SPAD. The abscissa of FIG. 26A is relative time (without units), and the ordinate of FIG. 26A is relative amplitude (without units). FIG. 26B illustrates an example bar graph 2601 that may be formed by photon detection times of an example pixel according to the subject matter disclosed herein. The abscissa of Figure 26B is the relative normalized time, and the ordinate is the photon detection event count.

26C depicts an example bar graph 2602 in which the window width representing the FWHM of the projected pulse (not shown) is indicated at 2602a to indicate the window within which the maximum event count can be determined. Figure 26D depicts an exemplary histogram 2603 with which the trigger waveform output from the SPAD (Figure 26A) is convolved to determine the event count maximum.

The surface reflectance S can be estimated from the following equation: P = α * S * L (5)

where P is the pixel value; α is the system dependence constant that converts lux to pixel value; S is the surface reflectance, and L is the light intensity. The light intensity L can be represented by Lamb + L _laser , which are _the ambient light intensity and the laser intensity reaching the sensor, respectively.

Therefore, the surface reflectance S can be estimated as

The ambient light intensity L _amb can be expressed as L _amb =[( β * M / D )/ N ]* L _laser (7)

where N is the number of events detected from the light pulse, M is the number of events detected before the light pulse was detected (i.e. from the environment), D is the estimated distance, and β is another calibrated system dependent variables.

FIG. 27 illustrates an exemplary bar graph 2700 of exemplary pixels in accordance with the subject matter disclosed herein. The abscissa of the bar graph 2700 is relative time (unitless), and the ordinate of the bar graph 2700 is the count of photon detection events. At 2701 the number of detected events M is indicated before a light pulse is detected. The number of events N detected within the light pulse is indicated at 2702 .

The relationship between the transmitted laser power (represented by I _laser ) and the received laser power L _lesar can be: L _laser = γ *(1/ D ² )* I _laser (8)

Where γ is a system constant.

Then, the estimate of reflectivity S at position ( x,y ) can be expressed as: S ( x,y )= f ( D (x,y))* P / I _laser (9)

Where f ( D ) is a distance-dependent function, that is, f =1/[ α *(1+ β * M / D * N )* γ *(1/ D ² )] exported above. (10)

28 illustrates a flowchart 2800 of an exemplary method of generating a depth or range map and a grayscale image of a scene in accordance with the subject matter disclosed herein. The method starts at 2801 . At 2802, the scene is point-scanned by, for example, the imaging system 15 depicted in FIG. 1 . Pixel array 42 of imaging system 15 may include exemplary pixels 43 (FIG. 5), 700 (FIG. 7), 1300 (FIG. 13), 2100 (FIG. 21), and/or 2300 (FIG. 23). The point scan may be performed only once, but it will be appreciated that better results may be obtained by repeating the point scan multiple times. At 2803, photon detection events are accumulated for pixels of the pixel array 42. At 2804, a depth map or range map is generated as disclosed herein. Range information may be determined by pixel processing unit 46 and/or processor 19 . At 2805, a grayscale image of the scene is generated based on the estimate of the reflectivity of the scene. The grayscale image may be determined by the pixel processing unit 46 and/or the processor 19 . At 2806, the method ends.

FIG. 29A depicts an exemplary scenario 2900. 29B and 29C illustrate an exemplary depth map 2901 and an exemplary gray scale image 2902 respectively of the scene depicted in FIG. 29A according to the subject matter disclosed herein. The scale on the right side of Figure 29B is in meters.

FIG. 30 illustrates an exemplary embodiment of the general layout of the imaging system 15 depicted in FIGS. 1 and 2 in accordance with the subject matter disclosed herein. The imaging module 17 may include the desired hardware shown in the exemplary embodiments shown in FIGS. 2, 5, 7 (or 13) to achieve 2D/3D imaging and TOF according to various inventive aspects of the present disclosure Measurement. Processor 19 may be configured to interface with a number of external devices. In one embodiment, imaging module 17 may act as an input device that provides data logs to processor 19 for further processing in the form of processed pixel output (eg, P1 and P2 values in FIG. 12 ). . Processor 19 may also receive input from other input devices (not shown) that may be part of system 15 . Some examples of such input devices include computer keyboards, trackpads, touchscreens, joysticks, physical "tappable buttons" or virtual "tapable buttons," and/or computer mice/pointing devices. In FIG. 30 , processor 19 is shown coupled to system memory 20 , peripheral storage unit 275 , one or more output devices 277 , and network interface unit 278 . In FIG. 30 , a display unit is shown as the output device 277 . In some embodiments, system 15 may include more than one instance of the devices shown. Some examples of system 15 include computer systems (desktop or laptop), tablet computers, mobile devices, cell phones, video game units or video game game consoles, machine-to-machine (M2M) communication units, robots, automobiles, virtual reality devices, stateless thin client systems, vehicle dashcam or rearview camera systems, autonomous navigation systems, or any other type of computing or data processing device. In various embodiments, all of the components shown in Figure 30 may be housed within a single housing. Accordingly, system 15 may be configured as a freestanding system or any other suitable form factor. In some embodiments, system 15 may be configured as a client system rather than a server system.

In some embodiments, system 15 may include more than one processor (eg, in a distributed processing configuration). When system 15 is a multi-processor system, there may be more than one instance of processor 19, or there may be multiple processors coupled to processor 19 through respective interfaces (not shown). Processor 19 may be a system on chip (SoC), and/or may include more than one central processing unit (CPU).

System memory 20 may be any semiconductor-based storage system such as, for example, DRAM, SRAM, PRAM, RRAM, CBRAM, MRAM, STT-MRAM, and the like. In some embodiments, the memory unit 20 may include a combination of at least one 3DS memory module and one or more non-3DS memory modules. Non-3DS memory can include DDR-SRAM or DDR-2 DRAM, DDR-3 DRAM or DDR-4 DRAM (Double Data Rate or Double Data Rate 2, 3, or 4 Synchronous Dynamic Random Access Memory, DDR/DDR2/DDR3/DDR4 SDRAM), or Rambus®DRAM, Flash memory, various types Read Only Memory (Read Only Memory, ROM) etc. In addition, in some embodiments, the system memory 20 may include various types of semiconductor memories, and Not a single type of memory. In other embodiments, the system memory 20 can be a non-transitory data storage medium.

In various embodiments, peripheral storage unit 275 may include support for magnetic storage media, optical storage media, magneto-optical storage media, or solid-state storage media, such as hard disk drives, optical disks (such as compact disks (CDs) or Digital Versatile Disk (DVD)), non-volatile Random Access Memory (RAM) devices, and the like. In some embodiments, peripheral storage unit 275 may comprise a more complex storage device/system, such as a disk array (which may be in a suitable Redundant Array of Independent Disks (RAID) configuration) or a storage area network. (Storage Area Network, SAN), and the peripheral storage unit 275 can pass through standard peripheral interfaces (such as Small Computer System Interface (Small Computer System Interface, SCSI), Fiber Channel interface (Fibre Channel interface), Firewire® (IEEE 1394) interface, An interface based on the Peripheral Component Interface Express (PCI Express ^TM ) standard, an interface based on the Universal Serial Bus (USB) protocol or another suitable interface) is coupled to the processor 19 . Various such storage devices may be non-transitory data storage media.

The display unit 277 may be an example of an output device. Other examples of output devices may include graphics devices/display devices, computer screens, alarm systems, Computer Aided Design/Computer Aided Machining (CAD/CAM) systems, video game stations, smartphone display screens, A dashboard-mounted display or any other type of data output device in a car. In some embodiments, the input device (such as the imaging module 17) and the output device (such as the display unit 277) can be coupled through an input/output (I/O) interface or a peripheral interface. Combined with the processor 19.

In one embodiment, network interface 278 may communicate with processor 19 to enable system 15 to be coupled to a network (not shown). In another embodiment, the network interface 278 may not exist at all. Network interface 278 may include any device, media, and/or protocol suitable for connecting system 15 to a network, whether wired or wireless. In various embodiments, the network may include a Local Area Network (LAN), a Wide Area Network (WAN), wired or wireless Ethernet, a telecommunications network, a satellite link, or other suitable types of networks .

System 15 may include an onboard power supply unit 280 to provide power to the various system components shown in FIG. 30 . The power supply unit 280 may be a battery or may be connected to an alternating current (AC) power outlet or a car-based power outlet. In one embodiment, the power supply unit 280 may convert solar or other renewable energy into electricity.

In one embodiment, imaging module 17 may be integrated with a high-speed interface (such as, for example, a Universal Serial Bus 2.0 or 3.0 (USB 2.0 or 3.0) interface or higher) that plugs into any PC (Personal Computer, PC) or laptop computer. A non-transitory computer readable data storage medium such as, for example, system memory 20 or a peripheral data storage unit (eg, CD/DVD) may store program code or software. Processor 19 and/or pixel processing unit 46 (FIG. 2) in imaging module 17 may be configured to execute code so that system 15 is operable to perform 2D imaging (eg, grayscale images of 3D objects ), TOF and range measurement, and use of pixel-specific distance/pixel-specific range values to generate a 3D image of an object, such as the operations discussed earlier with reference to FIGS. 1-29 . For example, in some embodiments, when executing the program code, the processor 19 and/or the pixel processing unit 46 may suitably configure (or activate) relevant circuit elements (eg, FIG. 12 column decoder/driver 1203 and pixel row unit 1205) to apply appropriate input signals (such as shutter signals, RST signals, VTX signals, SEL signals, etc.) to the pixels 43 in the pixel array 42, so that Pixel output that enables capture of light from the return laser pulse and subsequent processing of pixel-specific P1 and P2 values required for TOF and range measurements. The code or software may be proprietary software or open source software that, when executed by an appropriate processing entity (e.g., processor 19 and/or pixel processing unit 46), enables the processing entity to process various pixel-specific ADCs Output (P1 and P2 values), determine range values, render results in multiple formats (including, for example, displaying 3D images of distant objects based on TOF-based range measurements). In some embodiments, pixel processing unit 46 in imaging module 17 may perform some processing on the pixel output data before it is sent to processor 19 for further processing and display. In other embodiments, the processor 19 may also perform some or all of the functions of the pixel processing unit 46 . In this case, the pixel processing unit 46 may not be a part of the imaging module 17 .

As will be appreciated by those skilled in the art, the innovative concepts described herein can be modified and varied over a wide range of applications. Accordingly, the scope of claimed subject matter should not be limited to any of the specific exemplary teachings discussed above, but is instead defined by the claims that follow.

43: Pixel

501:SPAD core/SPAD core part

502:PPD core/PPD core part

503:SPAD

504: the first control circuit

505: Incoming light

506: Output/SPAD output/digital SPAD output/SPAD exclusive digital output/signal/output signal

507: the second control circuit

508:PPD

510: Pixel exclusive analog output/pixel exclusive output data line/pixel output data line/pixel exclusive output/PIXOUT signal/PIXOUT data line/PIXOUT line/Pixout line

Claims (20)

An image sensor comprising: a time-resolved sensor comprising at least one pixel responsive to detection by the at least one pixel of one or more outputting a pair of first and second signals for photons, the one or more photons being reflected from the object, a first ratio of the first amplitude to the sum of the first and second amplitudes and the first ratio of the first amplitude to the sum of the first and second amplitudes The time-of-flight of the detected one or more photons is proportional, and a second ratio of the second amplitude to the sum of the first amplitude and the second amplitude is proportional to the detected one or more photons. or multiple photons, wherein the first amplitude is the amplitude of the first signal in the pair, and the second amplitude is the second signal in the pair an amplitude of the signal; and a processor to determine, based on the pair of first and second signals, a reflectivity of a surface of the object reflecting the light pulse. The image sensor according to claim 1, wherein the processor further determines the distance to the object based on the pair of first and second signals. The image sensor of claim 1, wherein the time-resolved sensor is responsive to detecting one or a plurality of photons to output a plurality of pairs of first and second signals, the one or more photons being reflected from the object, each pair of the first signal of the plurality of pairs of first and second signals corresponds to the second signal and the corresponding light pulse, and Wherein the processor determines a surface reflectance of the object reflecting the plurality of light pulses based on pairs of the first signal and the second signal. The image sensor of claim 3, wherein the time-resolved sensor is responsive to detecting at each of the plurality of pixels a corresponding light pulse projected toward the object. outputting each pair of the first signal and the second signal of the plurality of pairs of the first signal and the second signal, the one or more photons being reflected from the object, and wherein the processing The device further generates a gray scale image of the object based on the plurality of pairs of the first signal and the second signal. The image sensor of claim 4, wherein said processor generates at least one histogram of arrival times of photons detected by a predetermined one of said plurality of pixels to generate said grayscale image. The image sensor according to claim 3, wherein the time-resolved sensor further includes a plurality of pixels, at least one pixel in the plurality of pixels includes: at least one single photon avalanche two a polar body, each of said at least one single photon avalanche diode responding to the active shutter signal based on detection of incident on said each single photon avalanche diode and from said One or more photons reflected by the object to generate an output signal; a logic circuit coupled to the output signal of the at least one single photon avalanche diode, the logic circuit is used to generate a first enabling signal and a second enabling signal an enable signal, the first enable signal being active in response to initiation of the active shutter signal and inactive in response to the output signal of the at least one single photon avalanche diode enabled, and the second enable signal is active in response to the output signal of the at least one SPAV diode and inactive in response to the end of the active shutter signal; and differential a time-to-charge converter circuit coupled to the first enable signal and the second enable signal, the differential time-to-charge converter circuit comprising: a capacitive device having a first terminal and a second terminal, the a second terminal coupled to a ground voltage; first switching means having a first terminal, a second terminal and a third terminal, the first terminal of the first switching means being coupled to the first terminal of the capacitive means , the second terminal of the first switching device is coupled to a first floating diffusion node, the third terminal of the first switching device is coupled to the first enable signal, and the first switching device transferring a first charge on the capacitive device to the first floating diffusion node in response to the first enable signal; a second switching device having a first terminal, a second terminal and a third terminal, the The first terminal of the second switching means is coupled to the first terminal of the capacitive means, the second terminal of the second switching means is coupled to a second floating diffusion node, the second switching means the third terminal is coupled to the second enable signal, and the second switching means transfers residual charge on the capacitive means to the second floating diffusion node in response to the second enable signal and an output circuit for outputting the pair of first and second signals, the first signal comprising a first voltage and the second signal comprising a second voltage, the first voltage being based on the The first charge on the first floating diffusion node, the second voltage is based on the remaining charge on the second floating diffusion node. The image sensor of claim 6, further comprising a drive signal that changes based on a ramp function, the drive signal starting to change in response to a start time of a light pulse from which the one or more photons are detected until the active shutter signal ends, if the first enable signal is active, the drive signal is connected to the third terminal of the first switching device, and if the second enable signal is Now, the driving signal is connected to the third terminal of the second switching device. The image sensor according to claim 7 of the patent claims, wherein the first ratio of the first voltage to the sum of the first voltage and the second voltage is more related to the ratio of the one or more photons said time of flight minus delay time, and a second ratio of said second voltage to said sum of said first voltage and said second voltage is more proportional to said time of flight of said one or more photons Subtracting the delay time is proportional to the delay time comprising the time between the start of the transmission time of the light pulse and the start change of the drive signal. The image sensor according to claim 6, wherein the capacitive device includes a capacitor or a pinned diode. An imaging unit comprising: a light source illuminating an object with a series of light pulses projected toward a surface of the object; a time-resolved sensor comprising at least one pixel, the time-resolved sensor being synchronized with the light source and responsive outputting a pair of first and second signals upon detection of one or more photons corresponding to a light pulse at said at least one pixel, said one or more photons coming from said surface of said object reflection, the first amplitude for the A first ratio of the sum of the first amplitude and the second amplitude is proportional to the detected time-of-flight of the one or more photons, and the second amplitude is proportional to the sum of the first amplitude and the second amplitude. A second ratio of the sum is proportional to the time-of-flight of the one or more detected photons, wherein the first amplitude is the amplitude of the first signal in the pair, and the the second amplitude is the amplitude of the second signal in the pair; and the processor determines the distance to the object based on the first and second signals of the pair and based on the pair The first signal and the second signal are used to determine the reflectivity of the surface of the object reflecting the light pulse. The imaging unit of claim 10, wherein the time-resolved sensor outputs pairs of first photons in response to detecting one or more photons reflected from the object at the at least one pixel. a signal and a second signal, each pair of the first signal and the second signal corresponding to a corresponding one of the plurality of light pulses projected toward the object, And wherein the processor further determines a plurality of surface reflectivities of the object based on a corresponding pair of first and second signals. The imaging unit according to claim 11, wherein the processor further generates a grayscale image of the object based on the plurality of surface reflectances. The imaging unit of claim 12, wherein the processor generates at least one histogram of arrival times of photons detected by a predetermined one of the plurality of pixels to generate the grayscale image. The imaging unit according to claim 11, wherein the time-resolved sensor further includes a plurality of pixels, at least one of the plurality of pixels includes: at least one single photon avalanche diode , each of the at least one SPAD responds to an active shutter signal based on detection of incident on said each SPAD and reflection from said object one or more photons to generate an output signal; a logic circuit coupled to the output signal of the at least one single photon avalanche diode, the logic circuit is used to generate a first enable signal and a second enable signal , the first enable signal is active in response to initiation of the active shutter signal and inactive in response to the output signal of the at least one SPAV diode, and the second enable signal an enable signal being active in response to the output signal of the at least one SPAV diode and being inactive in response to the end of the active shutter signal; and a differential time-to-charge converter circuit, coupled To the first enable signal and the second enable signal, the differential time-to-charge converter circuit outputs the first signal and the second signal. The imaging unit of claim 14, wherein the differential time-to-charge converter circuit comprises: a capacitive device having a first terminal and a second terminal, the second terminal being coupled to a ground voltage; a first switch means having a first terminal, a second terminal and a third terminal, the first terminal of the first switching means being coupled to the first terminal of the capacitive means The second terminal of the first switching device is coupled to a first floating diffusion node, the third terminal of the first switching device is coupled to the first enable signal, and the first switch means for transferring a first charge on the capacitive means to the first floating diffusion node in response to the first enable signal; and second switching means having a first terminal, a second terminal and a third terminal, The first terminal of the second switching means is coupled to the first terminal of the capacitive means, the second terminal of the second switching means is coupled to a second floating diffusion node, the second switch The third terminal of the device is coupled to the second enable signal, and the second switching device transfers the remaining charge on the capacitive device to the second floating device in response to the second enable signal. diffusion node, wherein the first signal includes a first voltage and the second signal includes a second voltage, the first voltage is based on the first charge on the first floating diffusion node, the second A voltage is based on the remaining charge on the second floating diffusion node, wherein a drive signal varying according to a ramp function begins varying in response to a start time of a first light pulse from which the one or more photons are detected until the said active shutter signal ends, if said first enable signal is active, said drive signal is connected to said third terminal of said first switching device, and if said second enable signal is active, Then the driving signal is connected to the third terminal of the second switching device. The imaging unit according to claim 15, wherein the capacitive device comprises a capacitor or a pinned photodiode. A method for generating a grayscale image of an object, wherein the method includes: Projecting a series of light pulses from a light source toward a surface of an object; detecting at a pixel one or more photons corresponding to the light pulses, the one or more photons being reflected from the surface of the object; a detector generating a pair of first and second signals in response to detecting the one or more photons, the time-resolved sensor being synchronized with the light source, a first amplitude relative to the first amplitude and the second signal A first ratio of the sum of the two amplitudes is proportional to the detected time-of-flight of the one or more photons, and the first amplitude is proportional to the ratio of the sum of the first amplitude and the second amplitude. Two ratios are proportional to the time-of-flight of the one or more detected photons, wherein the first amplitude is the amplitude of the first signal in the pair, and the second amplitude is an amplitude of the second signal in the pair; determining, by the processor, a distance to the object based on the first and second signals of the pair; and determining, by the processor, a distance to the object based on the pair of The first signal and the second signal are used to determine the surface reflectivity of the object. The method of claim 17, further comprising: detecting one or more photons reflected from the object at the pixel for a plurality of light pulses projected toward the object, pairs of each pair of the first signal and the second signal corresponds to a corresponding one of the series of light pulses; and based on the plurality of pairs of the first signal and the second signal by the processor At least one pair of the two signals is used to determine the surface reflectivity of the object. The method described in item 17 of the scope of the patent application further includes: detecting one or more photons at each pixel in the plurality of pixels, for from Each of the one or more photons detected is from the object reflection; generating, by the time-resolved sensor, a pair of first and second signals for each pixel in response to detecting the one or more photons; and generating, by the processor based on the pairs of The first signal and the second signal are used to generate the grayscale image of the object. The method according to claim 19, further comprising generating, by the processor, at least one histogram of arrival times of photons detected by a predetermined pixel of the plurality of pixels to generate the gray step image.

Images