JP2022067623A - Photoelectric conversion element, control method of photoelectric conversion element, and information processing device. - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a photoelectric conversion element for detecting an address event signal by using a photon count type sensor.
A photoelectric conversion element according to the present invention that solves the above problems is a pixel provided with a conversion means that outputs a signal in response to an incident of a photon, and measures the signal output from the pixel. 1 measuring means, a second measuring means for measuring the time required for the signal measured by the first measuring means to satisfy the first threshold value, and past measurement of the second measuring means. The first storage means for storing the result as the first time, the first time stored by the first storage means, and the second time measured by the second measuring means. It is characterized by having a comparison means for comparison.
[Selection diagram] Fig. 5

Description

The present invention relates to a photoelectric conversion element.

A photoelectric conversion element (hereinafter, photon count type sensor) is known, which digitally counts the number of photons arriving at an avalanche photodiode (hereinafter, APD) and outputs the counted value as a photoelectrically converted digital signal from a pixel. (See Patent Document 1).

U.S. Pat. No. 9,210,350

An object to be solved by the present invention is to detect an address event signal using a photon count type sensor.

The photoelectric conversion element according to the present invention, which solves the above-mentioned problems, is a pixel provided with a conversion means for outputting a signal in response to an incident of photons, and is a first measuring means for measuring the signal output from the pixel. A second measuring means for measuring the time required for the signal measured by the first measuring means to satisfy the first threshold value, and a first measurement result of the second measuring means in the past. A comparison means for comparing the first storage means stored as the time, the first time stored by the first storage means, and the second time measured by the second measuring means. And, characterized by having.

The address event signal can be detected using a photon count type sensor.

Block diagram showing a functional configuration example of an information processing device including a photoelectric conversion element The figure which shows an example of the laminated structure of the photoelectric conversion element of Embodiment 1. The figure which shows an example of the sensor chip in the element of Embodiment 1. The figure which shows an example of the circuit chip in the element of Embodiment 1. The figure which shows an example of the pixel of Embodiment 1. Timing chart for explaining the driving method of the element of the first embodiment The figure which shows an example of the pixel of Embodiment 2. Flow chart explaining the processing executed by the information processing device A block diagram showing an example of the hardware configuration of an information processing device

Hereinafter, the photoelectric conversion element according to the embodiment of the present invention will be described with reference to the drawings. At that time, those having the same function in all the figures are given the same number, and the explanation of the repetition is omitted.

<Embodiment 1>
<Information processing equipment>
FIG. 1 is an example of a functional configuration of an information processing apparatus 1000 including a photoelectric conversion element according to the present embodiment. The information processing device 1000 includes a light receiving lens 10001, a photoelectric conversion element 100, a control unit 1002, a storage unit 1003, an image processing unit 1004, a display unit 1005, an analysis processing unit 1006, and an acquisition unit 1007. The information processing device 1000 is specifically an image pickup device or a measurement device. The light receiving lens 1001 receives the incident light and forms an image on the photoelectric conversion element 100. The photoelectric conversion element 100 outputs a signal corresponding to the received incident light. Details will be described later. The control unit 1002 controls the focus of the light receiving lens 1001, the aperture, the photoelectric conversion element 100, and the like. The storage unit 1003 stores, for example, a signal output by the photoelectric conversion element 100 and image data processed by the image processing unit 1004. It also stores various settings related to the information processing device. Image data is processed based on the signal output by the photoelectric conversion element 100. The image processing unit 1004 determines a pixel value according to a signal and a measured time for each pixel of the image, and generates image data. Although an example of an information processing device used by converting an address event signal from the photoelectric conversion element 100 into image data by the image processing unit 1004 is shown, the information processing device 1000 does not necessarily have to generate image data. good. The display unit 1005 displays an image and various information generated by the image processing unit 1004. Functions such as the storage unit 1003 and the display unit 1005 may be outside the information processing device. Further, as an example of other functional configurations (not shown), there may be an operation unit for the user to input various instructions such as imaging, an audio output unit for outputting the sound of a moving image as an output other than the image display, and the like. .. The analysis processing unit 1006 executes a predetermined analysis processing based on the address event signal output by the photoelectric conversion element. Specifically, it is subject detection, motion detection, and the like. Detailed processing will be described later. The acquisition unit 1007 acquires the address event signal from the photoelectric conversion element 100.

FIG. 2 is a diagram showing a configuration example of a photoelectric conversion element according to the present embodiment. The photoelectric conversion element 100 is configured by stacking two chips of a sensor chip 11 and a circuit chip 21 and electrically connecting them. The sensor chip 11 includes a pixel area 12. The circuit chip 21 includes a pixel circuit area 22 for processing a signal detected in the pixel area 12, and a read circuit area 23 for reading a signal from the pixel circuit area 22.

<Pixel board>
FIG. 3 is a diagram showing a configuration example of the sensor chip 11. The pixel region 12 of the sensor chip 11 includes a plurality of pixels 101 arranged two-dimensionally. Pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (APD). FIG. 3 shows 6 rows from the 0th row to the 5th row and 36 pixels 101 arranged in the 6 columns from the 0th column to the 5th column together with a row number and a code indicating the column number. ing. For example, the unit pixels 11 arranged in the first row and the fourth column are designated by the reference numeral "P14". The number of rows and columns of the pixel array forming the pixel region 12 is not particularly limited.

<Circuit board>
FIG. 4 is a diagram showing a configuration example of the circuit chip 21. The circuit chip 21 includes a pixel circuit area 22 and a read circuit area 23. The pixel circuit area 22 includes a plurality of signal processing units 103 arranged two-dimensionally corresponding to each pixel 103 of the sensor chip 11. In FIG. 4, 6 rows from the 0th row to the 5th row and 36 signal processing units 103 arranged in the 6 columns from the 0th column to the 5th column are represented by row numbers and reference numerals indicating column numbers. Is shown with. For example, the signal processing unit 103 arranged in the first row and the fourth column is designated by the reference numeral "S14". The number of rows and columns of the signal processing unit array forming the pixel circuit area 22 is not particularly limited. The read circuit area 23 includes a vertical arbiter circuit 110, a column circuit 112, a horizontal read circuit 111, and a signal output circuit 117.

<Vertical arbiter circuit>
Each line of the signal processing unit array of the pixel circuit area 22 extends in the first direction (horizontal direction in FIG. 4), and a request signal output line 114VEQU and a response input line 115VACT are arranged. The request signal output line 114VEQU and the response input line 115VACT are connected to the signal processing units 103 arranged in the first direction, respectively, to form a signal line. The extending first direction of the request signal output line 114VEQU and the response input line 115VACT may be expressed as a row direction or a horizontal direction. Note that FIG. 4 shows the control lines VRQ and VACT together with the reference numerals indicating the line numbers. For example, the request signal line on the first line is designated by the reference numeral "VRQ [1]".

The control lines VRQ and VACT of each line are connected to the vertical arbiter circuit 110. The vertical arbiter circuit 110 supplies a control signal for driving the signal processing unit 103 to the signal processing unit 103 via the request signal output line 114VEQU and the response input line 115VACT.

The signal processing unit 103 outputs a request signal requesting the output of the address event data to the vertical arbiter circuit 110 via the request signal output line 114VEQU. The vertical arbitration circuit 110 arbitrates the request from the signal processing unit 103 of each pixel, and returns a response indicating permission or disapproval of output of the address event data to the signal processing unit 103 via the response input line 115VACT. ..

<Signal output>
A signal line 116 is arranged in each row of the signal processing unit array of the pixel circuit region 22 so as to extend in a second direction (longitudinal direction in FIG. 4) intersecting with the first direction. The signal lines 116 are connected to the signal processing units 103 arranged in the second direction, respectively, and form a common signal line. The extending second direction of the signal line 116 may be referred to as a column direction or a vertical direction. Note that FIG. 4 shows the signal line 116 together with a reference numeral indicating a column number. For example, the signal line 116 in the fourth column is designated by the reference numeral "POUT4".

Each signal processing unit 103 receives the permission response of the output from the vertical arbiter circuit 110, and outputs the address event data from the signal processing unit 103 to the column circuit 112 via the signal line 116. The column circuit 112 is provided corresponding to each column of the signal processing unit array of the pixel circuit unit region 22, and is connected to the signal line 116 of the corresponding column. The column circuit 112 includes a latch function for holding a signal read from the signal processing unit 103 via the signal line 116 of the corresponding column.

<Horizontal reading>
The horizontal read circuit 111 supplies a control signal for reading a signal from the column circuit 112 to the column circuit 112, and receives address event data from the column circuit 112 in each column. The signal output circuit 117 outputs the address event data measured by each pixel as an output signal SOUT. The address event data includes coordinate information of a unit pixel in which a change in the number of incident photons per unit time as an event occurs, and time information in which a change in the number of incident photons per unit time occurs. In addition, the polarity (positive or negative) of the change in the number of incident photons per unit time can be included. The counting method of address event data will be described later. By using address event data, it is possible to handle high-speed processing that was not possible with conventional synchronous photoelectric conversion elements in use cases that require high-speed processing such as robot and vehicle control. .. (The merit of minute change in brightness of SPAD will be described later)
<Pixel part>
FIG. 5 is an example of an equivalent circuit and a block diagram of the pixel 101 of FIG. 3 and the signal processing unit 103 of FIG.

The pixel 101 in the sensor chip 11 includes an APD 201 which is a photoelectric conversion unit. When light is incident on the APD201, charge pairs corresponding to the incident light are generated by photoelectric conversion. A voltage VL (first voltage) is supplied to the anode of the APD201. Further, a voltage VH (second voltage) higher than the voltage VL supplied to the anode is supplied to the cathode of the APD201. A reverse bias voltage is supplied to the anode and cathode so that the APD201 operates in an avalanche multiplication operation. By supplying such a voltage, the electric charge generated by the incident light causes an avalanche multiplication, and an avalanche current is generated.

The signal processing unit 103 in the sensor chip 21 includes a quench element 202, a waveform shaping unit 210, a first counter circuit 211, a second counter circuit 212, a first determination circuit 213, a memory 214, a comparator 215, and a second counter. The determination circuit 216, the response circuit 217, and the selection circuit 218 are included.

The quench element 202 is connected to the power supply that supplies the voltage VH and the APD 201. The quench element 202 has a function of replacing the change in the avalanche current generated in the APD 201 with a voltage signal. The quench element 202 functions as a load circuit (quenching circuit) when the signal is multiplied by the avalanche multiplication, suppresses the voltage supplied to the APD 201, and has a function of suppressing the avalanche multiplication (quenching operation).

The waveform shaping unit 210 shapes the potential change of the cathode of the APD 201 obtained at the time of photon detection, and outputs a pulse signal. For the waveform shaping unit 210, for example, an inverter circuit or a buffer circuit is used.

<Pixel counter unit>
The first counter circuit 211 counts the pulse signal (output signal from the pixel) output from the waveform shaping unit 210. That is, the first counter circuit 211 is a counter circuit that counts the number of photons incident on each APD. Further, the first counter circuit 211 receives the reset signal from the first determination circuit 213 and resets the counter value.

On the other hand, the second counter circuit 212 counts the time for counting photons in the first counter circuit 211 using a clock supplied from the outside of the sensor (or a clock generated in the sensor using the clock). do. That is, the second counter circuit 212 is a time-digital conversion circuit Time to Digital Converter (hereinafter, TDC). The output of the second counter circuit 212 may be output as time (seconds) or the number of clocks.

The first determination circuit 213 resets the count value of the first counter circuit 211 when the number of photons counted by the first counter circuit 211 reaches the first threshold value. Further, after the comparison by the comparator 215, which will be described later, is completed, the counter value (second time count value) in the second counter circuit 212 is overwritten on the memory 214, and the count value of the second counter circuit 212 is also obtained. Reset. That is, the first determination circuit includes a first counter circuit that counts the number of photons each time the number of photons incident on the APD reaches the first threshold value, and a second counter circuit that counts the time that the photons are incident. It has the role of resetting the counter circuit.

The counter value (first time count value) of the second counter circuit 212 in the past is stored in the memory 214, and the counter value of the memory 214 is stored every time the count value of the second counter circuit 212 is reset. It will be overwritten. In the comparator 215, the count value of the difference between the counter value (second time count value) of the current second counter circuit 212 and the counter value (first time count value) of the second counter circuit 212 in the past. Ask for 219. That is, in the comparator 215, the number of photons exceeds the threshold value at the N + 1th time as well as the first time required for the number of photons measured in the first counting circuit to exceed the threshold value at the Nth time. A predetermined signal (for example, a difference count value) is output according to the comparison result with the required second time. The comparator 215 may detect a change in luminance by using the ratio of the counter value of the first counter circuit to the counter value of the second counter circuit.

<Cooperation with peripheral circuits>
In the second determination circuit 216, when the difference count value 219 is equal to or greater than the second threshold value, the request signal is sent to the vertical arbiter circuit 110 via the request signal output line 118VEQU. Then, the response circuit 217 receives a response from the vertical arbiter circuit 110 via the response input line 119VACT, indicating permission or disapproval of the output of the address event data. On the other hand, when the difference count value 219 is less than the second threshold value, the request signal is not sent.

When the response circuit 217 receives a response indicating that the output is permitted, the selection circuit 218 switches the connection between the memory 214 and the signal line 113 by the control signal VSEL. As a result, the counter value (second time count value) of the second counter circuit 212 held in the first memory 314 is output to the column circuit 112.

<Summary>
The difference counter value 219 calculated by the comparator 215 is the time interval at which the second counter value is reset. As described above, when the number of photons incident on the APD reaches the first threshold value, the second counter value is reset. Therefore, the difference counter value is such that the number of incident photons reaches the first threshold value. The time interval reached. That is, the difference counter value 218 corresponds to the reciprocal of the incident frequency of photons. Therefore, the photoelectric conversion element 100 has a function of measuring "change in incident frequency of photons", that is, change in luminance.

Then, using the second determination circuit 216, the address event data is output only when the difference in the time interval at which the number of incident photons reaches the first threshold value is equal to or greater than the second threshold value. That is, it is a photoelectric conversion element that outputs the incident frequency when the difference in incident frequency is large and does not output the incident frequency when the difference is small. With the above configuration, it is possible to realize an asynchronous photoelectric conversion element that detects a change in luminance as an address event in real time for each pixel address. In this way, by realizing DVS in a photon count type, it becomes possible to detect an address event of a change at the level of one photon and to set the detection condition of the address event in more detail.

<Timing chart>
FIG. 6 is a timing chart illustrating a driving method for detecting an address event using the photoelectric conversion element 100. FIG. 6 shows a case where both the first threshold value and the second threshold value have 4 counts. 250 is a clock, 251 is the timing when a photon is detected by the first counter circuit, 211 is the count value of the first counter circuit, and 214 is the counter value of the second counter circuit stored in the memory. ..

At the first time T0, the photon counting in the first counter circuit 211 is started. At this time, the memory 214 stores the first time count value counted by the second counter circuit at the timing before that. (In FIG. 6, it is assumed that the first time T0 = 10 is stored as the initial time count value).
After starting the counting of photons in the first counter circuit 211, the count value of the first counter circuit 211 reaches the first threshold value = 4 at the second time T1. The second time T1 is counted by the second counter circuit 212, and FIG. 6 shows a case where the time count value counted by the second counter circuit is T1 = 8. The first determination circuit 213 determines that the count value of the first counter circuit 211 has reached the first threshold value = 4, and resets the first counter circuit 211.

At the same time, the comparator 215 obtains the counter value ΔT1 = 2 of the difference between T1 and T0. The second determination circuit 216 compares ΔT1 = 2 counts with the second threshold value = 4 counts. At time T1, since ΔT1 is less than the second threshold value, T1 is not output. When the determination by the comparator 215 is completed, the time count value T1 counted by the second counter circuit 212 is written to the memory 214, and the count value of the second counter circuit 212 is reset.

Then, the photon counting is restarted from the time T1. After starting the photon counting, the count value of the first counter circuit 211 reaches the first threshold value = 4 again at time T2. FIG. 5 shows the case where the time count value T2 = 16. As in the case of time T1, the first determination circuit 213 resets the first counter circuit 211, and the comparator 215 obtains the counter value ΔT2 = 6 count of the difference between T2 and T1. Since ΔT2 is equal to or higher than the second threshold value at time T2, the request signal for outputting T2 is transmitted to the vertical arbiter circuit 110. Then, in response to the response signal from the vertical arbiter circuit 110, the time count value T2 of the second counter circuit 212 is output to the column circuit 112.

<Advantages of photon count type>
In the conventional photoelectric conversion element, since the change in the brightness is detected as the analog signal as the change in the amount of light incident on the photodiode, it is difficult to measure the change in the brightness at the level of one photon. On the other hand, the photoelectric conversion element of the present embodiment measures the change in luminance by using the first counting circuit 211 for counting the number of photons and the second counter circuit 212 for counting the time. Therefore, it is possible to detect a change in brightness at the level of one photon. Specifically, by setting the first threshold value to 1, it is possible to detect a change in brightness at the level of one photon. By detecting the change in brightness at the level of one photon, the address event signal can be acquired even in a night-vision state such as at night.

Further, in the photoelectric conversion element of the present embodiment, the presence or absence of a change in luminance is determined by using the comparator 215 and the second determination circuit 216. Since the address event detection condition can be set by a digital signal by the second threshold value, the photoelectric conversion element of the present embodiment can set the address event detection condition in detail. By setting the address event detection conditions in detail, it is possible to remove the brightness change due to photon shot noise and acquire only the necessary address event information according to the scene. Specifically, if it is desired to capture only a large luminance change, a large threshold value may be set, and if it is desired to capture a small luminance change, the threshold value may be set small.

<Variation 1: Photon count>
Although FIG. 5 shows a case where the APD is used as the photoelectric conversion unit that realizes the photon count, another photoelectric conversion unit may be used. For example, a photoelectric conversion unit that suppresses read noise to about 0.5 electron or less, such as QIS (Quanta Image Sensor) disclosed in Reference 1, may be used to realize photon counting. However, reading noise is less when APD is used as shown in FIG. (Reference 1: Jiaju Ma, et. Al, "Photon-number-resolving megapixel image sensor at room temperature temperature without avalanche gain", Optica, 2017)
<Variation 2: First threshold>
The first threshold value is preferably set to a value equal to the number of saturated bits of the first counter circuit. Specifically, it may be configured to reset the first count value and the second count value when the most significant bit of the first counter circuit becomes 1.

The smaller the first threshold value, the higher the frequency of determining whether or not there is a change in the difference counter value 219, and the change in luminance can be captured instantaneously, which is preferable. On the other hand, the larger the first threshold value, the more the incident frequency of photons is estimated with a large number of photons, so that the estimation accuracy is improved and the change in luminance can be accurately measured, which is preferable. Therefore, it is more preferable that the first threshold value is configured so that it can be appropriately changed depending on the use case, the subject, and the like.

DVS has two main merits. One is to capture high-speed phenomena such as in-vehicle, and the other is to reduce the amount of data by not outputting output when there is no change in brightness, such as fixed-point monitoring in parking lots. Is. When it is desired to capture a high-speed phenomenon as in the former case, it is preferable to set the first threshold value small so that the change in brightness can be captured instantaneously. On the other hand, as in the latter case, when it is desired to suppress the output when there is no change in brightness, the first threshold value is set large and the change in brightness is accurately measured to obtain the output when there is no change in brightness. It is preferable to suppress it surely.

<Variation: Second threshold>
The second threshold value for determining whether to output the difference counter value 219 may be a fixed value or may be temporally changeable. Since the difference counter value 218 corresponds to the reciprocal of the incident frequency of photons, it is preferable to change the second threshold value according to the difference counter value 219. Specifically, it is preferable to change the second threshold value to be inversely proportional to the difference counter value 219 each time the first count value and the second count value are reset. With such a configuration, the second threshold value can be determined so that the ratio (ΔL / L) of the change ΔL of the luminance to the original luminance L is constant. Further, it is more preferable to configure the second threshold value so that it can be changed for each pixel. Since the incident frequency of photons differs depending on the pixel, it is necessary to be able to change the second threshold value for each pixel in order to keep ΔL / L constant.

The second threshold value may be configured to have different threshold values when the difference counter value increases and when the difference counter value decreases. By having different threshold values, it is possible to selectively detect only the increase in luminance or selectively detect only the decrease in luminance.

<Variation: Pixel value is output when there is no photon count>
When the incident frequency of photons is extremely low, the time until the count value of the first counter circuit reaches the first threshold value becomes long, and the detection frequency of address events may decrease. Therefore, if the count value of the first counter circuit does not reach the first threshold value even if the count value of the second counter circuit reaches the third threshold value, the count of the first counter circuit at that time is counted. It is preferable to have a configuration that outputs a value. Specifically, when the count value of the second counter circuit reaches the third threshold value, it is determined whether the count value of the first counter circuit is equal to or more than the first threshold value or less than the first threshold value. A third determination circuit is provided. Further, the third determination circuit is connected to the request signal line, the response input line, the selection circuit, and the column circuit. With such a configuration, it is possible to detect the presence or absence of an address event even when the incident frequency of photons is low.

<Variation: Peripheral circuit>
FIG. 4 shows a case where the address event of each pixel is read by using the vertical arbiter circuit and the horizontal read circuit, but other configurations may be used. Specifically, it has a three-layer configuration in which a memory chip 31 is further added to the sensor chip 11 and the circuit chip 21, and the time count value measured by each pixel is transferred to the memory chip for each pixel to store the memory in the memory chip. After holding it once, it may be output sequentially from the memory chip.

<Variation: Analysis processing using address event signal>
An information processing apparatus that executes various analysis processes based on the address event signal output by the photoelectric conversion element described above will be described. FIG. 9 is a block diagram showing a hardware configuration example of the information processing device. The CPU 1100 reads and executes an OS and other programs stored in the memory 1200, controls each connected configuration, and performs calculations and logical determinations for various processes. The process executed by the CPU 1100 includes information processing of the embodiment. The memory 1200 is a hard disk drive, an external storage device, or the like, and stores programs and various data related to information processing of the embodiment. FIG. 8 is a flowchart illustrating a process executed by an information processing apparatus that acquires an address event signal output by the photoelectric conversion element described in the present embodiment and executes various analysis processes. The process shown in the flowchart of FIG. 8 is executed by the CPU 1100 of FIG. 9, which is a computer, according to a computer program stored in the memory 1200. In the following description, the notation of the process (step) is omitted by adding S at the beginning of each process (step). In S801, the acquisition unit 1007 acquires the address event signal detected by the photoelectric conversion element 100. In S802, the analysis processing unit 1006 executes a predetermined analysis processing based on the acquired address event signal. The analysis process is, for example, various analysis processes such as motion detection and vibration detection. In S830, the analysis processing unit 1006 outputs the analysis result of the address event signal. For example, in the case of motion detection, image data that can display the time and place where the motion is detected is generated and output to the display unit.

Further, a trained neural network that takes an address event signal as an input and outputs a control signal may be provided, and data to be used for post-processing may be learned from the address event signal. For example, the address event signal may be used as it is as a recognition signal for machine learning, and analysis processing such as subject detection and subject classification may be performed. Further, the result of calculating the motion vector of the subject from the address event signal may be output to the control unit 1002 and used for vibration isolation, tracking, motion prediction, motion analysis and the like of the subject. Further, it may be used for position estimation or failure analysis of a vibrating object by analyzing the frequency of the blinking light source from the temporal change of the address event.

<Variation: Luminance output>
In the above, the second determination circuit 216 is used to output the second counter value when the difference in the time interval at which the number of incident photons reaches the first threshold value is equal to or greater than the second threshold value. Was there. However, it is more preferable to output the brightness information of the subject in addition to the second count value.

Specifically, when the difference in the time interval at which the number of incident photons reaches the first threshold value is equal to or greater than the second threshold value, the second counter is reset once. After that, once again, the first memory unit 220 for storing the second count value until the difference in the time interval at which the number of incident photons reaches the first threshold value becomes equal to or higher than the second threshold value, and the second. It suffices to have a second memory unit 221 for storing the number of times the counter of 1 is reset.

For example, when the number of times the first counter is reset is C, the second count value is T, and the first threshold value is N, the luminance L of the subject can be expressed as follows.
L = K (C × N / T)
However, K is an arbitrary coefficient.

By doing so, the pixel value can be acquired by using the change in light intensity as a trigger. As a result, for example, it is possible to acquire an image in which the influence of camera shake is reduced or an image at the moment when the subject starts to move.

<Embodiment 2>
The photoelectric conversion element 300 of the second embodiment differs from the photoelectric conversion element 100 shown in the first embodiment only in the configuration of the signal processing unit. FIG. 7 is an example of an equivalent circuit and a block diagram of the pixels of the photoelectric conversion element 300 and the signal processing unit 301. Since the configuration of the pixel portion is the same as that of the photoelectric conversion element 100 shown in FIG. 5, the description thereof will be omitted.

Similar to the signal processing unit 103 shown in FIG. 5, the signal processing unit 301 includes a quench element 302, a waveform shaping unit 310, a first counter circuit 311 and a second counter circuit 312, a first determination circuit 313, and a second. The determination circuit 316, the response circuit 317, and the selection circuit 318 are included. Further, in addition to the first memory 314 and the first comparator 315, the second memory 324 and the second comparator 325 are included. With such a configuration, the "degree of change" in brightness can be detected as an address event. Since the "change in brightness" is the first derivative of the luminance, the "degree of change in the brightness" corresponds to the second derivative of the luminance. By using the second derivative of the brightness as an address event, it is possible to detect only an abnormal phenomenon, for example, detecting an abnormality of an object whose brightness changes uniformly.

Since the roles and configurations of the quench element 302, the waveform shaping unit 310, the first counter circuit 311 and the second counter circuit 312 are the same as those in FIG. 5, they are omitted.

The first determination circuit 313 resets the count value of the first counter circuit 311 when the number of photons counted by the first counter circuit 311 reaches the first threshold value. Further, after the comparison by the first comparator 315, which will be described later, is completed, the counter value (second time count value) in the second counter circuit 312 is overwritten on the first memory 314 to overwrite the second counter. The count value of the circuit 312 is also reset.

The counter value (first time count value) of the second counter circuit 312 in the past is stored in the first memory 314, and the first counter value is reset every time the count value of the second counter circuit 312 is reset. The counter value of the memory 314 is overwritten. In the first comparator 315, the difference between the counter value (second time count value) of the current second counter circuit 312 and the counter value (first time count value) of the second counter circuit 312 in the past. The count value of 319 is obtained.

The past difference count value 329 (first difference count value) is stored in the second memory, and each time the count value of the second counter circuit 312 is reset, the counter value of the second memory 324 is stored. Is overwritten. In the second comparator 325, the count value 339 of the difference between the counter value of the current difference (second difference count value) and the counter value of the past difference (first difference count value) is obtained.

In the second determination circuit 316, when the difference count value 339 is equal to or greater than the fourth threshold value, the request signal is sent to the vertical arbiter circuit via the request signal output line. Then, the response circuit 317 receives a response indicating permission or disapproval of output of the address event data from the vertical arbiter circuit via the response input line.

When the response circuit 317 receives a response indicating that the output is permitted, the selection circuit 318 switches the connection between the second memory 324 and the signal line. As a result, the difference count value 339 held in the second memory 324 is output to the column circuit.

<Summary>
The difference counter value 339 is the difference between the time intervals at which the second counter value is reset. Therefore, the difference counter value 339 corresponds to the degree of change in the incident frequency of photons. Therefore, the photoelectric conversion element 100 of the present invention measures the second derivative of the luminance. With the above configuration, it is possible to realize an asynchronous photoelectric conversion element that detects the second derivative of the luminance as an address event in real time for each pixel address.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

11 Sensor chip 21 Circuit chip 101 pixel 103 Signal processing unit 201 Avalanche photodiode 202 Quench element 210 Waveform shaping unit 211 First counter circuit 212 Second counter circuit 213 First judgment circuit 214 Memory 215 Comparer 216 Second Judgment circuit 217 Response circuit 218 Selection circuit

Claims (22)

It is a photoelectric conversion element equipped with a pixel that outputs a signal according to the incident of a photon.
A first measuring means for measuring the signal output from the pixel, and
A second measuring means for measuring the time required for the signal measured by the first measuring means to reach the first threshold value, and a second measuring means.
A first storage means for storing the measurement result of the second measurement means at the first time point as a first time, and a first storage means.
The first time stored by the first storage means is compared with the second time measured by the second measuring means at a second time point after the first time point. As a means of comparison,
A photoelectric conversion element comprising: an output means for outputting a signal corresponding to a comparison result by the comparison means. The photoelectric conversion element according to claim 1, wherein the output means outputs the signal when the difference between the first time and the second time is larger than the second threshold value. The output means outputs the second time when the difference between the second time and the first time is equal to or greater than the second threshold value, and outputs the second time when the difference is less than the second threshold value. The photoelectric conversion element according to claim 2, wherein the time of 2 is not output. The photoelectric conversion element according to claim 3, wherein the second threshold value is configured to be changeable. The photoelectric conversion element according to claim 3 or 4, wherein the second threshold value is configured to be changeable for each pixel. The claim is characterized in that the second threshold value is different depending on whether the second time is larger than the first time or the second time is smaller than the first time. The photoelectric conversion element according to any one of 3 to 5. The photoelectric conversion element according to any one of claims 1 to 6, wherein the first measuring means measures the number of photons output from the pixel. The photoelectric conversion element according to any one of claims 1 to 7, further comprising a conversion means composed of an avalanche photodiode. The photoelectric conversion element according to any one of claims 1 to 8, wherein the first threshold value is equal to the number of saturated bits of the first measuring means. The photoelectric conversion element according to any one of claims 1 to 9, wherein the first threshold value is configured to be changeable. Any of claims 1 to 10, wherein the second measuring means resets the measurement by the second measuring means when the first time reaches a preset third threshold value. The photoelectric conversion element according to claim 1. The eleventh aspect of the present invention, wherein the comparison means outputs the measurement result of the first measurement means when the measurement result of the second measurement means reaches the third threshold value. Photoelectric conversion element. One of claims 1 to 12, further comprising a second storage means for storing the difference between the second time and the first time based on the comparison result by the comparison means. The photoelectric conversion element according to item 1. The comparison means further compares the difference in time stored by the second storage means with the difference between the measurement result by the second measuring means and the time stored by the first storage means. 13. The photoelectric conversion element according to claim 13. 17. The photoelectric conversion element according to any one item. When the measurement result of the first measuring means satisfies the first threshold value, the determination means outputs the measurement result of the first time by the second measuring means to the first storage means. The photoelectric conversion element according to claim 15, further comprising resetting the measurement result of the second measuring means to the first time. The photoelectric conversion element according to claim 15, wherein the determination means outputs the luminance of the subject based on the number of times the measurement by the first measurement means is reset. The second measuring means is characterized in that by measuring the number of clocks, the time required for the signal measured by the first measuring means to reach the first threshold value is measured. Item 1. The information processing apparatus according to item 1. An information processing apparatus including the photoelectric conversion element according to any one of claims 1 to 17.
An acquisition means for acquiring the signal output by the photoelectric conversion element, and
An information processing apparatus comprising: an analysis means for performing a predetermined analysis process based on the acquired signal. It is a control method of a photoelectric conversion element having a pixel that outputs a signal according to the incident of a photon.
The first measurement step of measuring the signal output from the pixel, and
A second measurement step of measuring the time required for the signal measured by the first measurement step to reach the first threshold value, and a second measurement step.
A first storage step of storing the measurement result of the second measurement step at the first time point as a first time, and a first storage step.
The first time stored by the first storage step is compared with the second time measured by the second measurement step at a second time point after the first time point. Comparison process and
A control method comprising: an output step of outputting a signal corresponding to a comparison result in the comparison step. A program for causing a computer to execute the control method according to claim 20. A computer-readable storage medium that stores a program for causing a computer to execute the control method according to claim 20.

Images