DE102023207294A1 - Storage-efficient device and storage-efficient method for determining measurement information - Google Patents

Abstract

A device for determining distance according to an embodiment is provided. The device comprises a receiving interface (110) for receiving a plurality of timestamps of a LiDAR detection. The device further comprises a processing unit (120) for carrying out one or more processing operations. The processing unit (120) is designed to determine a distance depending on the one or more processing operations. The processing unit (120) is designed to group the plurality of timestamps when carrying out the one or more processing operations such that two or more timestamps become the plurality of timestamps; and/or the processing unit (120) is designed to assign one or more additional data to more than one timestamp of the plurality of timestamps when carrying out the one or more processing operations.

Description

The application relates to a memory-efficient device and a memory-efficient method for determining measurement information and a corresponding LiDAR device.

In the field of light detection and ranging (LiDAR) technology, the direct time-of-flight (TOF) measurement principle with single photon avalanche diode (SPAD) detectors is widely used due to its simplicity and effectiveness. The technology transforms the flight time of a laser pulse into timestamps for distance measurement. The distance is calculated from the timestamps and the speed of light. However, a single timestamp is not reliable due to noise and false trigger events. To reduce these influences, most such LiDAR systems accumulate a large number of these timestamps in a so-called time-correlated histogram.

2 shows an example of a time-correlated histogram of a LIDAR system. In particular, 2 an example of a histogram with a range of 0 - 60 m (400 ns) and a TDC resolution of 312.5 ps, with a number of laser pulses per histogram of 400 and a background photon rate of 5 MHz. The x-axis in 2 corresponds to the arrival time of the photons. It is discrete and the resolution (also called bin width in a histogram) corresponds to the resolution of the time-to-digital converter (TDC) used. Each time stamp can assume N _B = 1280 different values (400 ns / 312.5 ps). In order to form the complete histogram, an additional counter is required for each of these values, the bit width of which depends on the number of individual measurements carried out. The y-axis is the count value that indicates how many time stamps are in the corresponding bin. If, as in the example mentioned above, 400 individual measurements are carried out, each of these counters requires at least a bit width of N _Z = [log ₂ 400] = 9 bits.

To improve the reliability of the distance measurement of a LiDAR system, noise reduction, interpolation and other algorithms are often applied to the measured histogram. Storing a single histogram already requires a large amount of storage space, which poses major challenges in terms of storage space and transmission bandwidth, for example for an embedded system. In addition, if algorithms are used that use information from consecutive histograms, the demands on the embedded systems increase to such an extent that most current systems can no longer meet them.

If you look at 2 As an example, assuming that a LiDAR sensor has N _pix = 12000 pixels, each count is stored in N _Z = 9 bits, and N _FPS = 25 deep images are generated per second, the total amount of data with a factor of F _S = 1.3 as an assumed safety margin is: $$M_{Hist}=F_S{N}_{FPS}{N}_{pix}{N}_B{N}_Z\approx \mathrm{4,18}Gbit/s$$

As the number of pixels, resolution and range of future LiDAR systems increase, this problem will become even more serious.

The large amounts of data generated by a LiDAR system cannot be stored on an embedded system or transmitted in real time.

So far, this problem has been solved mainly in three ways:

One way to address the problem is to create partial histograms: In this approach, the distance is determined by creating partial histograms with different resolutions in several measurement steps.

A variant of the device and method for creating partial histograms is presented in [1]. This variant includes several measurement stages. The measurements in the first stage start with a low distance resolution and the full range. Then, according to the result of the last stage, a partial range is selected and the range resolution is increased. After several iterations, the method finally reaches the maximum resolution only in a small sub-range. The approach can reduce the amount of data to be stored compared to storing the complete histogram with full resolution.

Another variant of generating partial histograms is presented in [2] as a device. The device contains a window manager and a controller that is responsible for time gating the SPADs and generating partial histograms. The process can be divided into two steps: In a coarse detection phase, the controller first samples the time gating interval (gating) over the detection period and collects the information from different gating intervals. With this information, the distance with coarse resolution and the optimal detection window are determined. In the fine detection phase, the sensor then only detects the signal within the specified time window with high resolution and the fine distance is determined accordingly. Compared to the first variant, which measures the entire range in the first step, this variant only measures the partial range in both steps. Thus, the variant could have better noise tolerance, since it does not have to significantly reduce the distance resolution to achieve a promising data reduction rate. A disadvantage is that the control of the time gating scheme is complex. In addition, the frame rate is low because several cycles are required to determine a distance.

Another way to address the above problem is compressive single-photon histograms (CSPH). In this concept, the timestamps are converted into an encoded data form using an encoding matrix in three steps:

wobei T die Reichweite bezeichnet und Res die Auflösung von TDCs bezeichnet. Das Element A_i ist berechnet durch $$A_i=\{\begin{array}{cc}\mathrm{1,}& i=\lfloor \frac{T_{TS}}{Res}\rfloor \\ \mathrm{0,}& sonst\end{array},$$

In a first step, the timestamp T _TS is converted into a one-hot key A: $$A=\left\{{\mathrm{<figure-callout\; id="1"\; label="A"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">A</figure-callout>}}_1,\dots A_i,\dots A_N\right\}mitN=\lfloor \frac{T}{Res}\rfloor ,$$

where T is the range and Res is the resolution of TDCs. The element A _i is calculated by $$A_i=\{\begin{array}{cc}\mathrm{1,}& i=\lfloor \frac{T_{TS}}{Res}\rfloor \\ \mathrm{0,}& sonst\end{array},$$

durchgeführt, wobei B eine kodierte Darstellung eines Zeitstempels bezeichnet und C eine M × N Kodierungsmatrix bezeichnet.In one step, the coding is done by: $$B=C\times A$$

where B denotes an encoded representation of a timestamp and C denotes an M × N encoding matrix.

wobei N_T die Anzahl der Zeitstempel bezeichnet. Die Standard-Histogrammisierung kann als ein spezielles CSPH betrachtet werden, wobei C eine N × N Einheitsmatrix ist. Eine vielversprechende Kodierungsmatrix muss mit einem kleinen M eine gute Rauschtoleranz haben und die gewünschte Abstandsinformation beibehalten. Ein Konstruktionsbeispiel mit einer auf Gray-Coding (deutsch: Gray-Kodierung) basierenden Kodierungsmatrix wird in [3] vorgestellt.In a third step, accumulation occurs through: $$\widehat{B}={\displaystyle \sum _{j=0}^{Nr}{B}_j},$$

where N _T is the number of timestamps. The standard histogramization can be considered as a special CSPH, where C is an N × N identity matrix. A promising coding matrix must have a good noise tolerance with a small M and preserve the desired distance information. A design example with a coding matrix based on Gray coding is presented in [3].

Finally, another way to address the above problem is to store subpixels: This concept reduces data by reducing the number of pixels per image.

When forming macro pixels, the information of the neighboring pixels is combined and only one distance value is output. In this case, the data reduction rate is inversely proportional to the spatial resolution of the LiDAR system, and there is only a low spatial resolution.

With partial field-of-view (FOV), the system captures only a portion of the FOV in each frame, but in a dynamic measurement environment this approach can cause large asynchrony.

In [4], DE 10 2020 203 796 A1 , "Method and apparatus for determining measurement information and LiDAR device" provides a method for determining measurement information based on a plurality of measurement values from a measurement value range. The method reduces the amount of data used for distance determination by feature extraction. However, this is a bar chart processing method, which means that the system used must still have the necessary storage space for the complete bar chart.

A device for determining distance according to one embodiment is provided. The device comprises a receiving interface for receiving a plurality of timestamps of a LiDAR detection. The device further comprises a processing unit for carrying out one or more processing operations. The processing unit is designed to determine a distance depending on the one or more processing operations. When carrying out the one or more processing operations, the processing unit is designed to group the plurality of timestamps such that two or more timestamps of the plurality of timestamps are aggregated; and/or the processing unit is designed to assign one or more additional data to more than one timestamp of the plurality of timestamps when carrying out the one or more processing operations.

• - Empfangen einer Mehrzahl von Zeitstempeln einer LiDAR Detektion,
• - Durchführung von ein oder mehrerer Verarbeitungsoperationen durch eine Verarbeitungseinheit. Und:
• - Bestimmen einer Distanz abhängig von den ein oder mehreren Verarbeitungsoperationen durch die Verarbeitungseinheit.

Furthermore, a method for determining distance according to an embodiment is provided. The method comprises:

• - Receiving a plurality of timestamps of a LiDAR detection,
• - Execution of one or more processing operations by a processing unit. And:
• - Determining a distance depending on the one or more processing operations by the processing unit.

When performing the one or more processing operations, the processing unit groups the plurality of timestamps such that two or more timestamps of the plurality of timestamps are aggregated. And/or when performing the one or more processing operations, the processing unit assigns one or more additional data to more than one timestamp of the plurality of timestamps.

Furthermore, a computer program according to an embodiment is provided with a program code for carrying out the method described above.

Preferred embodiments of the invention are described below with reference to the drawings.

• 1 zeigt eine Vorrichtung zur Distanzbestimmung gemäß einer Ausführungsform.
• 2 zeigt ein Beispiel für ein zeitkorreliertes Histogramm eines LIDAR Systems.
• 3 zeigt Ausführungsvarianten der LiDAR Datenverarbeitung gemäß bestimmter Ausführungsbeispiele.
• 4 zeigt eine Sortierung von Zeitstempeln in aufsteigender Reihenfolge und eine Erweiterung um Zusatzwissen gemäß einem Ausführungsbeispiel.
• 5 zeigt einen Prozess der Erstellung neuer Merkmale bei der ersten Stufe der Selbstgruppierung gemäß einer Ausführungsform.
• 6 zeigt ein Verfahren der Aktualisierung einer bestehenden Merkmalsgruppe bei der ersten Stufe der Selbstgruppierung gemäß einer Ausführungsform.
• 7 zeigt ein Verfahren der Erstellung weiterer Merkmale bei der ersten Stufe der Selbstgruppierung gemäß einer Ausführungsform.
• 8 zeigt das Verfahren der Anwendung einer weiteren Aktualisierungsregel bei der ersten Stufe der Selbstgruppierung gemäß Ausführungsformen.
• 9 zeigt einen Prozess der zweiten Stufe der Selbstgruppierung gemäß einer Ausführungsform.
• 10 zeigt einen Aufbau und das Training eines stacked-FNN gemäß einer Ausführungsform.

The drawings show:

• 1 shows a device for determining distance according to an embodiment.
• 2 shows an example of a time-correlated histogram of a LIDAR system.
• 3 shows variants of LiDAR data processing according to certain embodiments.
• 4 shows a sorting of timestamps in ascending order and an extension with additional knowledge according to an embodiment.
• 5 shows a process of creating new features at the first stage of self-grouping according to one embodiment.
• 6 shows a method of updating an existing feature group at the first stage of self-grouping according to one embodiment.
• 7 shows a method of creating additional features in the first stage of self-grouping according to one embodiment.
• 8 shows the method of applying another update rule to the first stage of self-grouping according to embodiments.
• 9 shows a second stage self-grouping process according to one embodiment.
• 10 shows a structure and training of a stacked FNN according to an embodiment.

1 shows a device according to the invention which is suitable for distance determination according to an embodiment.

The device comprises a receiving interface 110 for receiving a plurality of timestamps of a LiDAR detection.

Furthermore, the device comprises a processing unit 120 for performing one or more processing operations.

The processing unit 120 is configured to determine a distance depending on the one or more processing operations.

The processing unit 120 is configured to group the plurality of timestamps such that two or more timestamps become the plurality of timestamps when performing the one or more processing operations; and/or the processing unit 120 is configured to assign one or more additional data to more than one timestamp of the plurality of timestamps when performing the one or more processing operations.

• Gemäß einer Ausführungsform kann die Verarbeitungseinheit 120 z.B. ausgebildet sein, bei der Durchführung der ein oder mehreren Verarbeitungsoperationen die Mehrzahl von Zeitstempeln umzusortieren, um eine Mehrzahl umsortierter Zeitstempel zu erhalten, und eine Datenstruktur zu erzeugen, in der die Mehrzahl umsortierter Zeitstempel in ihrer umsortierten Reihenfolge angeordnet sind. Dabei kann die Verarbeitungseinheit 120 z.B. ausgebildet sein, die Distanz unter Verwendung der Datenstruktur zu bestimmen.

The following embodiments relate to data integration according to particular embodiments:

• According to one embodiment, the processing unit 120 can be designed, for example, to reorder the plurality of timestamps when performing the one or more processing operations in order to obtain a plurality of reordered timestamps and to generate a data structure in which the plurality of reordered timestamps are arranged in their reordered order. In this case, the processing unit 120 can be designed, for example, to determine the distance using the data structure.

In one embodiment, the processing unit 120 can be designed, for example, to assign the one or more additional data to the plurality of time stamps when carrying out the one or more processing operations. The processing unit 120 can be designed, for example, to generate the data structure such that the data structure includes the one or more additional data.

According to one embodiment, the processing unit 120 can, for example, be designed to generate the data structure such that the one or more additional data are arranged only after the majority of all rearranged time stamps in the data structure, or are arranged before the majority of all rearranged time stamps in the data structure.

In one embodiment, the processing unit 120 may, for example, be configured to use a machine learning method for the distance estimation and to use the data structure as input of the machine learning method.

According to one embodiment, the processing unit 120 may, for example, be configured to assign the one or more additional data to the plurality of timestamps when performing the one or more processing operations, wherein the one or more additional data may, for example, comprise a background intensity and/or a distance determination of a previous frame and/or a temperature value and/or calibration data and/or a photon rate and/or a distance to previous images and/or one or more features of neighboring pixels.

• In einer Ausführungsform kann die Verarbeitungseinheit 120 z.B. ausgebildet sein, bei der Durchführung der ein oder mehreren Verarbeitungsoperationen die Mehrzahl von Zeitstempeln derart zu gruppieren,
  • - dass ein erster Zeitstempel der Mehrzahl von Zeitstempeln eine erste Merkmalsgruppe von ein oder mehreren Merkmalsgruppen bildet, und
  • - dass ein oder mehrere weitere Zeitstempel der Mehrzahl von Zeitstempeln entweder einer der ein oder mehreren Merkmalsgruppen zugeordnet werden oder eine weitere Merkmalsgruppe der ein oder mehreren Merkmalsgruppen bilden.

The following embodiments relate to a first stage of self-grouping according to particular embodiments:

• In one embodiment, the processing unit 120 may be configured, for example, to group the plurality of timestamps when performing the one or more processing operations such that
  • - that a first time stamp of the plurality of time stamps forms a first feature group of one or more feature groups, and
  • - that one or more further time stamps of the plurality of time stamps are either assigned to one of the one or more feature groups or form a further feature group of the one or more feature groups.

According to one embodiment, the processing unit 120 may, for example, be configured to determine a count value for each feature group of the one or more feature groups, which count value depends on how many timestamps of the plurality of timestamps have been assigned to this feature group.

In one embodiment, the processing unit 120 may, for example, be configured to determine for each feature group of the one or more feature groups a central point whose value depends on those timestamps of the plurality of timestamps assigned to this feature group and/or whose value depends on derived values, wherein each of the derived values depends on one of the timestamps assigned to this feature group (e.g., a distance value for a timestamp may depend on a time value of the timestamp).

According to one embodiment, the processing unit 120 can be designed, for example, to determine the central point for each feature group of the one or more feature groups such that the central point results from a combination of the values of those time stamps that are assigned to this feature group, or to determine it such that the central point results from a combination of the derived values of those time stamps that are assigned to this feature group.

In one embodiment, the processing unit 120 can be configured, for example, to group the plurality of timestamps when performing the one or more processing operations such that the one further timestamp of the plurality of timestamps forms a further feature group of the one or more feature groups if and only if an amount of a difference between the value of the central point and a value of the one further timestamp is greater than a threshold value, or if an amount of a difference between the value of the central point and a derived value of the one further timestamp is greater than a threshold value.

According to one embodiment, the processing unit 120 can be designed, for example, to no longer create any new feature groups when a number of the one or more feature groups reaches a predefined upper limit.

In one embodiment, the processing unit 120 may, for example, be configured to overwrite a less significant feature group of the one or more feature groups when a number of the one or more feature groups reaches a predefined upper limit.

• Gemäß einer Ausführungsform kann die Verarbeitungseinheit 120 z.B. ausgebildet sein, bei der Durchführung der ein oder mehreren Verarbeitungsoperationen jeden der Mehrzahl von Zeitstempeln genau einem Cluster von ein oder mehreren Clustern zuzuordnen.

The following embodiments relate to clustering with macro bins according to particular embodiments:

• According to one embodiment, the processing unit 120 may, for example, be configured to assign each of the plurality of timestamps to exactly one cluster of one or more clusters when performing the one or more processing operations.

In one embodiment, the processing unit 120 may, for example, be configured to increment a count value associated with a cluster when assigning another timestamp to a cluster of the one or more clusters to which a timestamp has already been assigned.

According to one embodiment, the processing unit 120 may, for example, be configured, when assigning a further timestamp to the cluster to which a timestamp has already been assigned, to not change any other values or properties assigned to this cluster other than increasing the count value of this cluster.

• In einer Ausführungsform kann z.B. die ein oder mehreren Merkmalsgruppen wenigstens zwei Merkmalsgruppen aufweisen. Dabei kann die Verarbeitungseinheit 120 z.B. ausgebildet sein, die wenigstens zwei Merkmalsgruppen zu einer geringeren Anzahl Merkmalsgruppen abhängig von einer Eigenschaft der wenigstens zwei Merkmalsgruppen zusammenzufassen.

The following embodiments relate to a second stage of self-grouping according to particular embodiments:

• In one embodiment, the one or more feature groups may comprise at least two feature groups. The processing unit 120 may be designed to, for example, to combine two feature groups into a smaller number of feature groups depending on a property of at least two feature groups.

According to one embodiment, the processing unit 120 can, for example, be designed to combine the at least two feature groups into the smaller number of feature groups depending on differences between the center points of the at least two feature groups.

In one embodiment, the one or more clusters can have at least two clusters. The processing unit 120 can, for example, be designed to combine the at least two clusters into a smaller number of clusters depending on a property of the at least two clusters.

• Gemäß einer Ausführungsform kann die Verarbeitungseinheit 120 z.B. ausgebildet sein, abhängig von den ein oder mehreren Verarbeitungsoperationen unter Einsatz eines maschinellen Lernverfahrens die Distanz zu bestimmen.

The following embodiments concern the distance determination according to special embodiments:

• According to one embodiment, the processing unit 120 may, for example, be configured to determine the distance depending on the one or more processing operations using a machine learning method.

In one embodiment, the processing unit 120 may, for example, be configured to use a neural network as a machine learning method.

According to one embodiment, the processing unit 120 may, for example, be configured to use the neural network as a regressor.

In one embodiment, the processing unit 120 may, for example, be configured to use the neural network as a classifier.

According to one embodiment, the processing unit 120 may, for example, be configured to use the data structure as the input of the machine learning method.

In one embodiment, the processing unit 120 may, for example, be configured to use information about the one or more feature groups as input to the machine learning method; or the processing unit 120 may, for example, be configured to use information about the one or more clusters as input to the machine learning method.

According to one embodiment, the processing unit 120 may, for example, be configured to determine the distance depending on the one or more processing operations without using a machine learning method.

Specific embodiments of the invention are described below.

The processing of SPAD-based LiDAR data typically involves three forms of data: timestamps, histograms and point clouds. Histogram data requires a large amount of storage space, which is a major challenge in embedded systems. To solve this problem, embodiments provide a new way of processing timestamps without taking histograms into account. In certain embodiments, the data processing can be divided into the areas of "conversion of timestamps" and "distance determination".

3 represents five variants of LiDAR data processing. Other variants are possible.

In variant 1, the timestamps are converted using data integration and the distance is determined using an AI regressor.

In variant 2, the conversion of the timestamps is carried out by means of a first stage of self-grouping, and the distance determination is carried out by means of digital processing or by means of an AI regressor or by means of an AI classifier.

In variant 3, the conversion of the timestamps is carried out by means of a first stage of self-grouping and a second stage of self-grouping, and the distance determination is carried out by means of digital processing or by means of an AI regressor or by means of an AI classifier.

In variant 4, the conversion of the timestamps is carried out by means of clustering with macro bins, and the distance determination is carried out by means of digital processing or by means of an AI regressor or by means of an AI classifier.

In variant 5, the conversion of the timestamps is carried out by means of clustering with macro bins and then by means of a second stage of self-grouping, and the distance determination is carried out by means of digital processing or by means of an AI regressor or by means of an AI classifier.

The conversion of timestamps according to specific embodiments is described below.

The aim of converting timestamps is to create an efficient data format instead of the histogram. The data format should retain all important information of the original data as far as possible. In addition, the conversion process should be efficient given the limited resources of embedded systems. The four blocks (data integration, 1st / 2nd level of self-grouping and clustering with macro bins) are presented in detail below. Examples of possible combinations of these four blocks within the scope of the implementation variants are the 3 and already described above. Other design variants are possible.

First, data integration according to a specific embodiment is described.

First, in some embodiments, data integration is carried out by sorting. The time stamps are arranged in a random order. This random factor hinders the subsequent distance prediction. A sorting is applied to the time stamps. The sorting can be either ascending or descending, for example.

• Zu diesen zusätzlichen Information zählen beispielsweise ein oder mehrere der folgenden Informationen:
  • - Hintergrundintensität: Sie wird durch die Photonenrate dargestellt, die neben den Messungen der Zeitstempel aufgezeichnet wird. Dieser Wert ist positiv mit der Wahrscheinlichkeit einer falschen Zeitstempelgenerierung korreliert. In bestimmten Ausführungsformen wird der Wert daher beispielsweise normalisiert als zusätzlicher Parameter verwendet.
  • - Temporäre Informationen: Da die Zeit zwischen zwei Frames der LiDAR-Aufnahme sehr kurz ist (< 0,04s) wird davon ausgegangen, dass ein entdecktes Objekt seine Position allmählich ändert und nicht plötzlich in eine andere zufällige Entfernung springt. Daher wird in bestimmten Ausführungsformen beispielsweise die Distanzbestimmung des vorherigen Frames als weiterer zusätzlicher Parameter verwendet. Und/oder, es werden Merkmale von Nachbarpixeln als zusätzliche Informationen verwendet.
  • - In bestimmten Ausführungsformen werden beispielsweise weitere/andere Informationen wie Temperaturwerte und Kalibrierungsdaten (ebenfalls) berücksichtigt.

Then, when integrating data in certain embodiments, additional knowledge is added. In LiDAR measurements, there is additional information in addition to the time stamps that could be beneficial for determining distances:

• This additional information includes, for example, one or more of the following information:
  • - Background intensity: It is represented by the photon rate recorded alongside the timestamp measurements. This value is positively correlated with the probability of false timestamp generation. In certain embodiments, the value is therefore used, for example, normalized as an additional parameter.
  • - Temporary information: Since the time between two frames of the LiDAR recording is very short (< 0.04s), it is assumed that a discovered object changes its position gradually and does not suddenly jump to another random distance. Therefore, in certain embodiments, for example, the distance determination of the previous frame is used as another additional parameter. And/or, features of neighboring pixels are used as additional information.
  • - In certain embodiments, for example, additional/other information such as temperature values and calibration data are (also) taken into account.

The final data form after the data has been compiled according to a particular embodiment is 4 shown.

This shows 4 a sorting of timestamps in ascending order and an extension with additional knowledge according to an embodiment. In particular, 4 The new design includes seven timestamps, a value for the background photon rate, and a distance to previous images in nanoseconds.

The in 4 The shape F shown can, in one embodiment, be used, for example, as input to a machine learning method (e.g., a neural network).

wobei N_I die Anzahl der Zusatzwissen(-Elemente) bezeichnet.With the definition that N _D is the required number of bits for each element in F, the total data set M _integration results from $$M_{Integration}=F_S{N}_{FPS}{N}_{pix}\left(N_T+N_l\right)N_D,$$

where N _I denotes the number of additional knowledge (elements).

• N_T = Anzahl der Zeitstempel
• N_pix = Pixel des LiDAR Sensors
• N_Z = Anzahl Bits, in denen ein Zählwert gespeichert wird
• N_FPS = Anzahl der Tiefbilder pro Sekunde.

There are:

• N _T = number of timestamps
• N _pix = pixels of the LiDAR sensor
• N _Z = Number of bits in which a count value is stored
• N _FPS = number of deep frames per second.

The data reduction rate R _{Integration-Hist} is then given by $$R_{Hist-Integration}=\frac{N_B{N}_Z}{\left(N_T+N_I\right)N_D}.$$

Since N _D and N _Z typically cannot be changed in most LiDAR systems, R _{Hist integration} is inversely proportional to (N _T + N _I ).

For this reason, data integration is particularly suitable for systems that have a small N _T , such as scanning LiDAR systems.

Next, the first stage of self-grouping according to particular embodiments is described.

For example, a “self-grouping” algorithm calculates important features or updates them when a new timestamp arrives.

For example, when creating new features, the result of self-grouping can be a list of feature groups. Each feature group refers to a data pair that includes a central point (distance value) and a corresponding count value. In the initialization phase, the maximum number of feature groups M can be specified, for example, and thus the required storage space can also be defined. If the list is empty, a generated timestamp from laser emission and echo detection is entered directly into the list as a central point with a count value of 1. The procedure is described in 5 shown.

This shows 5 a method of creating new features at the first stage of self-grouping according to an embodiment.

This is how 5 For example, a first time stamp is received that corresponds to a (e.g. distance) value of 5.00 m. A new, first feature group is created with the central point 5.00 m and the count value 1.

1. 1. Der neue Zentralpunkt wird durch $$D^{\prime }=f_1\left(D,Z,T_{TS}\right)$$
   
   berechnet, wobei D' den neuen Zentralpunkt angibt, D den alten Zentralpunkt angibt, T_TS die eingehenden Zeitstempel angibt und Z den Zählwert angibt. Ein Beispiel für f₁ könnte ein gewichteter Durchschnitt sein wie $$f_1\left(D,Z,TS\right)=\frac{D\cdot Z+T_{TS}}{Z+1}.$$
   
2. 2. Der entsprechende Zählwert Z' wird aktualisiert durch $$Z^{\prime }=f_2\left(D,Z,T_{TS}\right).$$
   
   Ein einfaches Beispiel für f₂ ist: $$f_2\left(Z\right)=\mathrm{<figure-callout\; id="1"\; label="Z\; +"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">Z</figure-callout>}+1.$$
   
   Das Verfahren ist in 6 dargestellt.

When updating the features, for example, if there is at least one feature group in the list, a newly generated timestamp is first compared with the central points contained in the list. If the difference between the timestamp and one of the central points is smaller than a certain interval d _E , the feature group corresponding to this central point is updated. The update rule can, for example, be structured as follows:

1. 1. The new central point is $$D^{\prime }=f_1\left(D,Z,T_{TS}\right)$$
   
   where D' indicates the new center point, D indicates the old center point, T _TS indicates the incoming timestamps and Z indicates the count value. An example of f ₁ could be a weighted average like $$f_1\left(D,Z,TS\right)=\frac{D\cdot Z+T_{TS}}{Z+1}.$$
   
2. 2. The corresponding count value Z' is updated by $$Z^{\prime }=f_2\left(D,Z,T_{TS}\right).$$
   
   A simple example for f ₂ is: $$f_2\left(Z\right)=Z+1.$$
   
   The procedure is in 6 shown.

This shows 6 a method of updating an existing feature group in the first stage of self-grouping according to one embodiment. Thus, a feature group exists with a central point of 5.00 m and a count value of 1. A new time stamp is received which corresponds to a (e.g. distance) value of 5.05 m. The value of this further time stamp is sufficiently close to the previous central point of 5.00 m, for which the count value is 1. This forms a new central point: 5.025 m = 1 * 5.00 m + 1 * 5.05 m, which has the new count value of 2. The method is continued accordingly for further time stamps.

The first stage of self-grouping can also result in the creation of additional features. For example, if the difference between the timestamp and all the center points in the list is greater than a certain interval d _E , a new feature group can be added.

For example, D and Z can be written according to the 7 described method, whereby 7 shows the process of creating additional features in the first stage of self-grouping according to one embodiment.

So in 7 the arrival of a new time stamp is shown, which corresponds to 3.02 m. The previous central point has a value of 5.025 m. Since 3.02 m lies outside the interval d _E around the previous central point, a new feature is created with the central point 3.02 m and the count value 1.

In addition to the rules mentioned above, further update rules are possible as follows, which, for example, define procedures when the feature group is full.

For example, if the number of features reaches the upper limit M, no new feature group will be added thereafter.

For example, a new, added timestamp that is to be stored as a new feature group could be discarded in one embodiment.

berechnet werden. Als Beispiel könnte f₃ durch $$f_3\left(Z\right)=Z$$

definiert werden.Or, a new, added timestamp could, for example, overwrite the least significant feature group. The significance S for each feature group can, for example, be determined by $$S=f_3\left(D,Z\right)$$

As an example, f ₃ could be calculated by $$f_3\left(Z\right)=Z$$

be defined.

8 shows the methods described herein of applying another update rule to the first stage of self-grouping according to embodiments that applies an implementation-specific strategy when the feature group is full.

The total data volume M _SG results from $$M_{SG}=F_S{N}_{FPS}{N}_{pix}M\left(N_D+N_Z\right).$$

The data reduction rate R _Hist-SG is thus given by $$R_{Hist-SG}=\frac{N_B{N}_Z}{M\left(N_D+N_Z\right)}.$$

Compared to the concept of timestamp conversion, R _Hist-SG is not directly affected by N _T . Therefore, self-grouping is a promising data reduction method for LiDAR systems, especially with large N _T , such as the Flash LiDAR system. However, it should be noted that the algorithm is usually executed every time a timestamp arrives. If the time interval between the arrival of the two timestamps is shorter than the processing time of the algorithm, this would lead to an additional time delay.

Clustering with macro-bins according to specific embodiments is described below. Instead of storing the timestamps in full resolution and updating the center point, such a macro-bin method can be applied to save further storage space and operations at lower distance accuracy.

The idea behind this concept is to create only a limited number of histogram bins. A new cluster is created as soon as a timestamp is recorded that has not yet been stored. The bit depth per timestamp (resolution of the TDC) is not necessarily reduced.

A timestamp occupies $$\text{ld}\left(\frac{t_{\text{Hist}}}{t_{\text{Bin}}}\right)\text{Bits}.$$

Each entry has a counter with depth N _Z in which the successful detections can be stored.

When creating the histogram, a maximum of k _bin histogram entries are created. The total amount of data for one pixel is $$B_{\text{Pix}}=k_{\text{Bin}}\cdot \text{ld}\left(N_Z\right)=\lceil \frac{t_{\text{Hist}}}{t_{\text{Bin}}}\rceil \cdot \text{ld}\left(N_Z\right).$$

This results in the total data volume of the histogram $$B_{\text{Hist}}=N_{\text{Pix}}\cdot k_{\text{Bin}}\cdot \text{ld}\left(N_Z\right)$$

mit k_Cluster < k_Bin.If the self-clustering approach is used, this data volume is reduced to $$B_{\text{Cluster}}=N_{\text{Pix}}\cdot k_{\text{Cluster}}\cdot \left(\text{ld}\left(N_{\text{Cluster}}\right)+\text{ld}\left(N_Z\right)\right)$$

with k _clusters < k _bins .

Here, N _Cıuster describes the set of quantized times resulting from the reduction of N _R of the original bin width to the width of a macrobin. $$N_{\text{Cluster}}=\lceil \frac{t_{\text{Hist}}}{t_{\text{Cluster}}}\rceil =\lceil \frac{t_{\text{Hist}}}{N_{\text{R}}{t}_{\text{Bin}}}\rceil <\lceil \frac{t_{\text{Hist}}}{t_{\text{Bin}}}\rceil$$

The individual cluster is given an index i _cluster with which the corresponding mapped time of the cluster can be calculated using the quantization width of the cluster t _cluster : $$t_{\text{F}}=i_{\text{Cluster}}\cdot t_{\text{Cluster}}.$$

Assuming that the height of the counts is the same in the case of clusters and bins, a data reduction by the following factor can be achieved: $$r=\frac{k_{\text{Bin}}\text{ld}\left(N_Z\right)}{k_{\text{Cluster}}\left(\text{ld}\left(N_{\text{Cluster}}\right)+\text{ld}\left(N_Z\right)\right)}.$$

If all clusters are occupied, an incoming new timestamp cannot be stored directly.

Logic to handle this case may include decrementing all clusters until a cluster reaches a count of 0. Then all clusters with a count of 0 are removed and are available for reassignment.

Alternatively, since this condition indicates a distribution with high variance, which may be caused by strong background light, a mismeasurement indicator bit can be set.

Furthermore, the cluster with the lowest count can be removed to provide new space for storing additional clusters.

• 0 cm bis 20 cm; 20,1 cm bis 40 cm; 40,1 cm bis 60 cm; 60,1 cm bis 80 cm, usw.

jeweils demselben Cluster zugeordnet werden. Geht beispielsweise ein erster Zeitstempel mit dem entsprechenden Abstandswert 45 cm ein, so wird ein erster Cluster mit Zählwert 1 gebildet. Dieser Cluster kann dann den Zentralpunkt 45 cm dauerhaft aufweisen (da dies der Wert des ersten, eingegangenen Zeitstempels ist oder aber er weist den ca. Mittelwert des Intervalls auf, hier 50 cm als Mittelwert des Intervalls 40,1 cm bis 60 cm.For example, clustering with Marko Bins can be implemented in such a way that the size of a cluster is predefined. For example, it can be defined that the (distance) values in an interval of 20 cm are assigned to the same cluster. For example, it can be defined in advance that timestamps with a (distance) value of:

• 0 cm to 20 cm; 20.1 cm to 40 cm; 40.1 cm to 60 cm; 60.1 cm to 80 cm, etc.

are each assigned to the same cluster. For example, if a first time stamp is received with the corresponding distance value 45 cm, a first cluster with a count value of 1 is formed. This cluster can then have the central point 45 cm permanently (since this is the value of the first time stamp received) or it can have the approximate mean value of the interval, here 50 cm as the mean value of the interval 40.1 cm to 60 cm.

Further incoming timestamps of this cluster interval from 40.1 cm to 60 cm then only increase the count value of the cluster; a recalculation of the central point then does not take place.

The second stage of self-grouping is described below. The second stage of self-grouping is optional in any case. Furthermore, the second stage of self-integration complements the previous concepts in many embodiments.

und $$Z^{\prime }=f_5\left(D_1,Z_1,\dots ,D_n,Z_n\right).$$

Thus, the process of (eg first stage of) self-grouping can produce a large number of feature groups whose central points are very close to each other. These feature groups are very likely to originate from the same object. Therefore, the second grouping (eg second stage of self-grouping) is applied as an option, which summarizes these feature groups. For example, the summary can be done by: $$D^{\prime }=f_4\left({\mathrm{<figure-callout\; id="1"\; label="D"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">D</figure-callout>}}_1,{\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1,\dots ,D_n,Z_n\right)$$

and $$Z^{\prime }=f_5\left({\mathrm{<figure-callout\; id="1"\; label="D"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">D</figure-callout>}}_1,{\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1,\dots ,D_n,Z_n\right).$$

und f₅ durch $$f_5\left(Z_1,Z_2\right)=Z_1+Z_2\text{, }\left|D_1-D_2\right|<\Delta d$$

definiert werden, wobei Δd die zulässige Differenz zwischen den Zentralpunkten ist.For example, f ₄ can be $$f_4\left({\mathrm{<figure-callout\; id="1"\; label="D"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">D</figure-callout>}}_1,{\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1,{\mathrm{<figure-callout\; id="2"\; label="D"\; filenames="DE102023207294A1\_0035.png,DE102023207294A1\_0036.png"\; state="\{\{state\}\}">D</figure-callout>}}_2,Z_2\right)=\frac{{\mathrm{<figure-callout\; id="1"\; label="D"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">D</figure-callout>}}_1\cdot {\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="D"\; filenames="DE102023207294A1\_0035.png,DE102023207294A1\_0036.png"\; state="\{\{state\}\}">D</figure-callout>}}_2\cdot Z_2}{{\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="Z"\; filenames="DE102023207294A1\_0035.png,DE102023207294A1\_0036.png"\; state="\{\{state\}\}">Z</figure-callout>}}_2},\left|{\mathrm{<figure-callout\; id="1"\; label="D"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">D</figure-callout>}}_1-{\mathrm{<figure-callout\; id="2"\; label="D"\; filenames="DE102023207294A1\_0035.png,DE102023207294A1\_0036.png"\; state="\{\{state\}\}">D</figure-callout>}}_2\right|<\Delta d$$

and f ₅ by $$f_5\left({\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1,Z_2\right)={\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="Z"\; filenames="DE102023207294A1\_0035.png,DE102023207294A1\_0036.png"\; state="\{\{state\}\}">Z</figure-callout>}}_2\text{, }\left|{\mathrm{<figure-callout\; id="1"\; label="D"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">D</figure-callout>}}_1-{\mathrm{<figure-callout\; id="2"\; label="D"\; filenames="DE102023207294A1\_0035.png,DE102023207294A1\_0036.png"\; state="\{\{state\}\}">D</figure-callout>}}_2\right|<\Delta d$$

where Δd is the allowable difference between the central points.

An example of the second grouping (the second stage of self-grouping) is in 9 shown.

According to the self-grouping algorithm, M feature groups are calculated. These feature groups may still contain redundant or unimportant information.

Therefore, feature extraction is performed as an option to extract K feature groups from M feature groups, where K < M. The extraction can be done e.g. by: $$\left\{\left({\mathrm{<figure-callout\; id="1"\; label="D"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">D</figure-callout>}}_1^{\ast },{\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1^{\ast }\right)\dots ,\left(D_K^{\ast },\left({\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1^{\ast }\right)\right)\right\}=f_6\left(\left({\mathrm{<figure-callout\; id="1"\; label="D"\; filenames="DE102023207294A1\_0036.png,DE102023207294A1\_0038.png"\; state="\{\{state\}\}">D</figure-callout>}}_1,Z_1\right)\dots ,\left(D_M,Z_M\right)\right).$$

mit $$D_{\text{Min}}+\frac{\left(D_{\text{Max}}-D_{\text{Min}}\right)\left(k-1\right)}{K}<D_i<D_{\text{Min}}+\frac{\left(D_{\text{Max}}-D_{\text{Min}}\right)k}{K},$$

wobei
i∈M, k∈K und D_Min und D_Max
der minimale bzw. maximale Erfassungsabstand sind.An example for f ₆ is: $$D_k=argma{x}_{Z_i}\left(D_i\right),$$

with $$D_{\text{Min}}+\frac{\left(D_{\text{Max}}-D_{\text{Min}}\right)\left(k-1\right)}{K}<D_i<D_{\text{Min}}+\frac{\left(D_{\text{Max}}-D_{\text{Min}}\right)k}{K},$$

where
i∈M, k∈K and D _Min and D _Max
the minimum and maximum detection distances respectively.

In the following, concepts of distance determination according to embodiments are described.

• Zur Distanzbestimmung können beispielsweise eingesetzt werden:
  • - Klassische digitale Verarbeitung
  • - Neuronales Netz als Regressor
  • - Neuronales Netz als Klassifikator.

According to embodiments, the above-mentioned variants convert the timestamps into several features. These features are then further processed using the following data processing methods to determine distance:

• For example, the following can be used to determine distance:
  • - Classic digital processing
  • - Neural network as regressor
  • - Neural network as classifier.

First, concepts of classical digital processing are described, which can be combined with the above embodiments.

The distance evaluation can be carried out using proven methods. This can include, for example, a simple mode detection or an interpolation of neighboring filled clusters to achieve a sub-cluster resolution.

In mode detection, the most frequent count is evaluated. First, the cluster with the largest count must be identified $$a=\text{argmax}\left(N_c\right).$$

This information can then be used to read the time-of-flight of the associated cluster.

The equation for determining the distance in this detection mode is expressed as follows $$d=c\cdot t_{\text{ToF}}=c\cdot \text{argmax}\left(N_c\right).$$

Interpolation can be used to generate a higher depth resolution, especially when the reflected laser information is located between two macro bins. This approach is somewhat less efficient when using clustering than when fully capturing histograms, since the limited number of clusters means that it cannot be guaranteed that macro bins with a low fill level are included in the capture. To further avoid the introduction of errors due to unrecorded macro bins, the approach should also be combined with a mode finding. The determined distance is then calculated according to $$d=c\cdot \frac{{\displaystyle {\sum }_{i=argmax\left(N_{Clusters}\right)-1}^{argmax\left(N_{Clusters}\right)+1}\frac{N_{C,i}}{N}{t}_{Cluster}\left(i_{Cluster}+\frac{1}{2}\right)}}{{\displaystyle {\sum }_{i=argmax\left(N_{Clusters}\right)-1}^{argmax\left(N_{Clusters}\right)+1}\frac{N_{C,i}}{N}}}$$

Now, a use of a neural network as a regressor according to specific embodiments is described.

A neural network structure is used here, for example, as a regressor. The input data of the network could be F from the data integration described above, or a combination of D and Z (possibly with preprocessing, e.g. normalization) from the first stage of self-grouping described above, from the clustering with macro bins described above, or from the second stage of self-grouping described above.

1. 1. Fully-connect feed-forwarded neural network (FNN; deutsch: vollständig verbundenes, vorwärts einspeisendes, neuronales Netzwerk)
2. 2. Stacked-FNN: Ein stacked-FNN (deutsch: gestapeltes FNN) bezieht sich auf zwei einzelne FNNs. Das erste FNN übernimmt z.B. die Daten aus der oben beschriebenen Datenintegration als Eingabe, während das zweite FNN die Ausgabe des ersten FNN als Eingabe übernimmt. Der Aufbau und das Training eines stacked-FNN gemäß einer Ausführungsform ist in 10 dargestellt. Der Zweck dieser Struktur besteht darin, aus zwei schwach Lernenden einen stark Lernenden zu machen, indem man sie hintereinanderschaltet. Als Sonderfall könnte das zweite FNN auch ein Duplikat des ersten FNN sein. In diesem Fall werden nur die Parameter des ersten FNN auf einem eingebetteten System gespeichert.
3. 3. Recurrent neural network (RNN, deutsch: rekursives neuronales Netzwerk): Hier kann ein Simple-RNN oder ein LSTM (long short term memory; deutsch: Lang-Kurzzeit-Speicher) verwendet werden.

For example, one or more of the following three network structures can be used:

1. 1. Fully-connect feed-forwarded neural network (FNN)
2. 2. Stacked FNN: A stacked FNN refers to two individual FNNs. The first FNN takes the data from the data integration described above as input, while the second FNN takes the output of the first FNN as input. The construction and training of a stacked FNN according to one embodiment is described in 10 The purpose of this structure is to make one strong learner out of two weak learners by connecting them in series. As a special case, the second FNN could also be a duplicate of the first FNN. In this case, only the parameters of the first FNN are stored on an embedded system.
3. 3. Recurrent neural network (RNN): A simple RNN or an LSTM (long short term memory) can be used here.

The output precision of a regressor depends on the number of digits but is independent of the sensor resolution. This means that a regressor has the potential to achieve higher precision than the resolution of the sensor. A regressor is therefore suitable for short range, low noise and high precision requirements.

In the following, a neural network as a classifier according to specific embodiments is described.

This method is similar to those mentioned in [4]. However, the approach mentioned in [4] is still based on histograms. Therefore, the use of machine learning should be slightly modified and adapted.

wobei O_k für den k-ten Ausgangswert und O_H für den höchste Ausgangswert des FNN steht. Der Objektdistanz ist dann der entsprechende D_H.An FNN is constructed as a classifier and uses the output from the data integration described above as input. Accordingly, K output neurons (classes) are constructed. Finally, the result is $$O_{\text{H}}=argma{x}_k\left(O_k\right),$$

where O _k is the k-th output value and O _H is the highest output value of the FNN. The object distance is then the corresponding D _H .

A classifier takes several possible distance values into account and assigns a corresponding quality to each distance. The output resolution corresponds to the sensor resolution. This approach is therefore suitable for long ranges, high noise, lower precision requirements and the case of multiple reflections.

In other embodiments, other machine learning algorithms, binary FNN, SVM, Bayes classifiers, decision tree or random forest can be used for distance estimation.

Embodiments can be used, for example, for a distance measuring system based on the time-of-flight principle (e.g. LiDAR systems), or for LiDAR applications, e.g. in the automotive sector, which require real-time processing of the data on embedded systems, or for applications in the industrial sector, e.g. stand-alone monitoring systems for object detection and tracking.

Certain embodiments are characterized by two particular advantages.

A first advantage is data reduction: The amount of LiDAR data can be reduced so that it can be processed on embedded systems with limited resources. To give a rough impression of different variants, two concrete scenarios are assumed, which are shown in Table 1. The corresponding data volumes that must be stored for different variants are listed in Table 2. It can be seen that embodiments in Scenario 1 and Scenario 2 achieve a similar level of accuracy and precision as with classical histogramization and processing.

Table 1 shows a parameter configuration for two concrete scenarios: Table 1 Scenario 1 Scenario 2 Number of timestamps for a distance determination ≤ 10 ≤ 400 range / m 0-8 0-60 TDC resolution / m 0.0468 0.0468 number of depth images per second 25 25 safety margin factor 1.3 1.3 number of feature groups (or clusters) 10 32 number of bits for a timestamp 8 11 number of additional knowledge 2 2 Number of bits for macro bin width 4 7 number of bits for a count value 4 9 reduction factor bin depth clustering 16 16 number of pixels 12,000 12,000

Table 2 shows the amount of data to be stored for different variants: Table 2 data volume / Gbits Scenario 1 Scenario 2 histogramization 0.25 4.18 data integration 0.03 - self-grouping 0.04 0.23 clustering 0.03 0.19

Classical digital processing methods based on histograms are effective but computationally intensive. Most of them cannot be implemented on an embedded system. Therefore, the most commonly used method is global maximum detection, which, although quite simple lags behind more complex methods in terms of precision, accuracy and robustness. A further advantage of some embodiments can thus be seen in the fact that the algorithms provided are relatively simple and efficient. Therefore, there is the possibility that embodiments can be implemented on an embedded system.

Although some aspects have been described in the context of a device, it is to be understood that these aspects also represent a description of the corresponding method, so that a block or component of a device can also be understood as a corresponding method step or as a feature of a method step. Analogously, aspects described in the context of or as a method step also represent a description of a corresponding block or detail or feature of a corresponding device. Some or all of the method steps can be performed by a hardware apparatus (or using a hardware apparatus), such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some or more of the key method steps can be performed by such an apparatus.

Depending on particular implementation requirements, embodiments of the invention may be implemented in hardware or in software, or at least partially in hardware or at least partially in software. The implementation may be carried out using a digital storage medium, for example a floppy disk, a DVD, a BluRay disc, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, a hard disk or another magnetic or optical memory on which electronically readable control signals are stored that can interact or interact with a programmable computer system in such a way that the respective method is carried out. Therefore, the digital storage medium may be computer readable.

Some embodiments according to the invention thus comprise a data carrier having electronically readable control signals capable of interacting with a programmable computer system such that one of the methods described herein is carried out.

In general, embodiments of the present invention may be implemented as a computer program product having a program code, wherein the program code is operable to perform one of the methods when the computer program product is run on a computer.

The program code can, for example, also be stored on a machine-readable medium.

Other embodiments include the computer program for carrying out one of the methods described herein, wherein the computer program is stored on a machine-readable medium. In other words, an embodiment of the method according to the invention is thus a computer program that has a program code for carrying out one of the methods described herein when the computer program runs on a computer.

A further embodiment of the methods according to the invention is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program for carrying out one of the methods described herein is recorded. The data carrier or the digital storage medium or the computer-readable medium is typically tangible and/or non-transitory.

A further embodiment of the method according to the invention is thus a data stream or a sequence of signals which represents or represents the computer program for carrying out one of the methods described herein. The data stream or the sequence of signals can, for example, be configured to be transferred via a data communication connection, for example via the Internet.

A further embodiment comprises a processing device, for example a computer or a programmable logic device, which is configured or adapted to carry out one of the methods described herein.

A further embodiment comprises a computer on which the computer program for carrying out one of the methods described herein is installed.

A further embodiment according to the invention comprises a device or a system which is designed to transmit a computer program for carrying out at least one of the methods described herein to a recipient. The transmission can be carried out electronically or optically, for example. The recipient can be, for example, a computer, a mobile device, a storage device or a similar device. The device or system can, for example, comprise a file server for transmitting the computer program to the recipient.

In some embodiments, a programmable logic device (e.g., a field programmable gate array, an FPGA) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may interact with a microprocessor to perform any of the methods described herein. In general, in some embodiments, the methods are performed by any hardware device. This may be general-purpose hardware such as a computer processor (CPU) or hardware specific to the method such as an ASIC.

The embodiments described above are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the following claims and not by the specific details presented in the description and explanation of the embodiments herein.

Literature:

1. [1] C. Zhang, S. Lindner, IM Antolovic, J. Mata Pavia, M. Wolf, and E. Charbon, “A 30-frames/s, 252×144 SPAD Flash LiDAR With 1728 Dual-Clock 48.8-ps TDCs, and Pixel-Wise Integrated Histogramming,” IEEE J. Solid-State Circuits, vol. 54, no. 4, pp. 1137-1151, 2019, doi: 10.1109/JSSC.2018.2883720 .
2. [2] Anup K. Sharma, Arnaud Laflaquière, Gennadiy A. Agranov, Gershon Rosenblum, and Shingo Mandai, “SPAD array with gated histogram construction “, US 2017/0052065 , published: 2017.
3. [3] F. Gutierrez-Barragan, A. Ingle, T. Seets, M. Gupta, and A. Velten, “Compressive Single-Photon 3D Cameras,” 2022 .
4. [4] " Method and device for determining measurement information and LiDAR device “, DE 10 2020 203 796 A1 , published: 2021.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• DE 10 2020 203 796 A1 [0018, 0152]
• US 2017/0052065 [0152]

Cited non-patent literature

• C. Zhang, S. Lindner, I. M. Antolovic, J. Mata Pavia, M. Wolf, and E. Charbon, “A 30-frames/s, 252×144 SPAD Flash LiDAR With 1728 Dual-Clock 48.8-ps TDCs, and Pixel-Wise Integrated Histogramming,” IEEE J. Solid-State Circuits, vol. 54, no. 4, pp. 1137-1151, 2019, doi: 10.1109/JSSC.2018.2883720 [0152]
• Anup K. Sharma, Arnaud Laflaquière, Gennadiy A. Agranov, Gershon Rosenblum, and Shingo Mandai, “SPAD array with gated histogram construction [0152]
• F. Gutierrez-Barragan, A. Ingle, T. Seets, M. Gupta, and A. Velten, “Compressive Single-Photon 3D Cameras,” 2022 [0152]
• Method and device for determining measurement information and LiDAR device [0152]

Claims (28)

Device for determining distance, the device comprising: a receiving interface (110) for receiving a plurality of timestamps of a LiDAR detection, a processing unit (120) for carrying out one or more processing operations, wherein the processing unit (120) is designed to determine a distance depending on the one or more processing operations, wherein the processing unit (120) is designed to group the plurality of timestamps when carrying out the one or more processing operations such that two or more timestamps of the plurality of timestamps are aggregated; and/or wherein the processing unit (120) is designed to assign one or more additional data to more than one timestamp of the plurality of timestamps when carrying out the one or more processing operations. device according to claim 1 , wherein the processing unit (120) is configured to reorder the plurality of timestamps when performing the one or more processing operations in order to obtain a plurality of reordered timestamps and to generate a data structure in which the plurality of reordered timestamps are arranged in their reordered order, wherein the processing unit (120) is configured to determine the distance using the data structure. device according to claim 2 , wherein the processing unit (120) is designed to assign the one or more additional data to the plurality of time stamps when carrying out the one or more processing operations, wherein the processing unit (120) is designed to generate the data structure such that the data structure comprises the one or more additional data. device according to claim 3 , wherein the processing unit (120) is designed to generate the data structure such that the one or more additional data are only arranged behind the majority of all rearranged time stamps in the data structure, or are arranged before the majority of all rearranged time stamps in the data structure. Device according to one of the Claims 2 until 4 , wherein the processing unit (120) is designed to use a machine learning method for the distance estimation and to use the data structure as input of the machine learning method. Device according to one of the preceding claims, wherein the processing unit (120) is designed to assign the one or more additional data to the plurality of time stamps when carrying out the one or more processing operations, wherein the one or more additional data comprise a background intensity and/or a distance determination of a previous frame and/or a temperature value and/or calibration data and/or a photon rate and/or a distance to previous images and/or one or more features of neighboring pixels. device according to claim 1 , wherein the processing unit (120) is designed to group the plurality of time stamps when carrying out the one or more processing operations such that a first time stamp of the plurality of time stamps forms a first feature group of one or more feature groups, and that one or more further time stamps of the plurality of time stamps are either assigned to one of the one or more feature groups or form a further feature group of the one or more feature groups. device according to claim 7 , wherein the processing unit (120) is designed to determine a count value for each feature group of the one or more feature groups, which count value depends on how many timestamps of the plurality of timestamps have been assigned to this feature group. device according to claim 7 or 8 , wherein the processing unit (120) is designed to determine a central point for each feature group of the one or more feature groups, the value of which depends on those time stamps of the plurality of time stamps assigned to this feature group. sen, and/or whose value depends on derived values, where each of the derived values depends on one of the timestamps assigned to this feature group (e.g. a distance value for a timestamp may depend on a time value of the timestamp). device according to claim 9 , wherein the processing unit (120) is designed to determine the central point for each feature group of the one or more feature groups such that the central point results from a combination of the values of those time stamps that are assigned to this feature group, or to determine it such that the central point results from a combination of the derived values of those time stamps that are assigned to this feature group. device according to claim 9 or 10 , wherein the processing unit (120) is designed to group the plurality of time stamps when carrying out the one or more processing operations such that the one further time stamp of the plurality of time stamps forms a further feature group of the one or more feature groups if and only if an amount of a difference between the value of the central point and a value of the one further time stamp is greater than a threshold value, or if an amount of a difference between the value of the central point and a derived value of the one further time stamp is greater than a threshold value. Device according to one of the Claims 7 until 11 , wherein the processing unit (120) is designed to no longer create any new feature groups when a number of the one or more feature groups reaches a predefined upper limit. Device according to one of the Claims 7 until 11 , wherein the processing unit (120) is designed to overwrite a less significant feature group of the one or more feature groups when a number of the one or more feature groups reaches a predefined upper limit. device according to claim 1 , wherein the processing unit (120) is designed to assign each of the plurality of timestamps to exactly one cluster of one or more clusters when carrying out the one or more processing operations. device according to claim 14 , wherein the processing unit (120) is designed to increase a count value associated with a cluster when assigning a further time stamp to a cluster of the one or more clusters to which a time stamp has already been assigned. device according to claim 15 , wherein the processing unit (120) is designed, when assigning a further timestamp to the cluster to which a timestamp has already been assigned, not to change any other values or properties assigned to this cluster other than increasing the count value of this cluster. Device according to one of the Claims 7 until 13 , wherein the one or more feature groups have at least two feature groups, and wherein the processing unit (120) is designed to combine the at least two feature groups into a smaller number of feature groups depending on a property of the at least two feature groups. device according to claim 17 , wherein the processing unit (120) is designed to combine the at least two feature groups into the smaller number of feature groups depending on differences between the central points of the at least two feature groups. Device according to one of the Claims 14 until 16 , wherein the one or more clusters comprise at least two clusters, and wherein the processing unit (120) is configured to combine the at least two clusters into a smaller number of clusters depending on a property of the at least two clusters. Device according to one of the preceding claims, wherein the processing unit (120) is designed to determine the distance depending on the one or more processing operations using a machine learning method. device according to claim 20 , wherein the processing unit (120) is designed to use a neural network as a machine learning method. device according to claim 21 , wherein the processing unit (120) is designed to use the neural network as a regressor. device according to claim 21 , wherein the processing unit (120) is designed to use the neural network as a classifier. Device according to one of the Claims 20 until 23 , wherein the device is further claim 5 and the processing unit (120) is configured to use the data structure as the input of the machine learning method. Device according to one of the Claims 20 until 23 , wherein the device is further claim 7 and the processing unit (120) is designed to use information about the one or more feature groups as input of the machine learning method; or wherein the device is further comprised of claim 14 and the processing unit (120) is configured to use information about the one or more clusters as input to the machine learning method. Device according to one of the preceding claims, wherein the processing unit (120) is designed to determine the distance depending on the one or more processing operations without using a machine learning method. Method for determining distance, the method comprising: receiving a plurality of timestamps of a LiDAR detection, performing one or more processing operations by a processing unit (120), and determining a distance depending on the one or more processing operations by the processing unit (120), wherein the processing unit (120) groups the plurality of timestamps in such a way that two or more timestamps of the plurality of timestamps are aggregated when performing the one or more processing operations; and/or wherein the processing unit (120) assigns one or more additional data to more than one timestamp of the plurality of timestamps when performing the one or more processing operations. Computer program with a program code for carrying out the method according to claim 27 .

Images