CN114415163A

CN114415163A - Camera-based distance measurement

Info

Publication number: CN114415163A
Application number: CN202111193075.2A
Authority: CN
Inventors: I·雷切尔高兹
Original assignee: Altberui Technology Co ltd
Current assignee: Altberui Technology Co ltd
Priority date: 2020-10-13
Filing date: 2021-10-13
Publication date: 2022-04-29
Also published as: US20220114756A1; US12142005B2

Abstract

A system, method, and computer-readable medium for storing instructions for distance measurement, the method may include acquiring an image of the environment surrounding the vehicle from a camera of the vehicle; searching for an anchor within the image, wherein the anchor is related to the at least one physical dimension of known values; and when the anchor is found, based on (a) said at least one physical dimension of known values, (b) said at least one physical dimension of known values in said image the appearance of dimensions, and (c) a distance-appearance relationship that maps appearance to distance to determine the distance between the camera and the anchor, wherein the distance-appearance relationship is generated by a calibration process that includes One or more calibration images of the anchor are obtained, and one or more distance measurements from the anchor are obtained.

Description

Camera-based distance measurement

Background

The three main types of Autonomous Vehicle (AV) sensors are cameras, radar and lidar.

These three sensors work together, providing the AV with information about its surroundings and helping the AV to detect the speed and distance of nearby objects, as well as their three-dimensional shape.

Lidar based systems provide real-time visualization of the AC ambient based on the distance of each laser spot. The computer of the lidar based system converts the laser points into a three-dimensional (3D) presentation and is able to identify other vehicles, people, roads, buildings, etc. as a means to enable the AV to navigate in its surroundings in a secure manner.

The cost of the lidar based system is very high-several orders of magnitude higher than the cost of the camera-making the lidar based system impractical to install in most vehicles.

There is an increasing need to provide a system that is cost effective and has the ability to generate depth information.

Disclosure of Invention

Systems, methods, and computer-readable media may be provided as shown in the specification.

Drawings

Embodiments of the present disclosure will be understood and appreciated more fully from the following detailed description, taken in conjunction with the accompanying drawings, in which:

FIG. 1 illustrates an example of a method;

FIG. 2 illustrates an example of a method;

FIG. 3 shows an example of an image and dimensions of an anchor;

FIG. 4 shows an example of an image and dimensions of an anchor;

FIG. 5 shows an example of a calibration vehicle and its surroundings; and

fig. 6 shows an example of a vehicle and its surroundings.

Detailed Description

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, however, both as to organization and method of operation, together with objects, features, and advantages thereof, may best be understood by reference to the following detailed description when read with the accompanying drawings.

It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to identify corresponding or analogous elements.

Because the illustrated embodiments of the present invention may, for the most part, be implemented using electronic components and circuits known to those skilled in the art, the details will not be explained in any greater extent than that considered necessary as illustrated above, for the understanding and appreciation of the underlying concepts of the present invention and in order not to obfuscate or distract from the teachings of the present invention.

Any reference in the specification to a method is to be taken as a reference to an apparatus or system and/or non-transitory computer-readable medium adapted to carry out the method and storing instructions for performing the method.

Any reference in the specification to a system or apparatus should be taken to apply to a method executable by the system and/or to a non-transitory computer-readable medium storing instructions for execution by the system.

Any reference in the specification to a non-transitory computer readable medium should be taken to apply to a device or system capable of executing the instructions stored in the non-transitory computer readable medium and/or to a method that may be adapted to execute the instructions.

Any combination of any of the modules or units listed in any of the figures, any part of the specification and/or any claim may be provided.

The description and/or drawings may refer to images. Any reference to an image should be taken to apply to the video stream or one or more images.

The specification and/or drawings may refer to a processor. The processor may be a processing circuit. The processing circuit may be implemented as a Central Processing Unit (CPU) and/or one or more other integrated circuits, such as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a fully custom integrated circuit, etc., or a combination of such integrated circuits.

Any combination of any steps of any of the methods shown in the specification and/or drawings may be provided.

Any combination of any subject matter of any claim may be provided.

Any combination of systems, units, components, processors, sensors shown in the description and/or the figures may be provided.

The analysis of the content of the media units may be performed by generating an identification of the media units and by comparing the identification with a reference identification. The reference signatures may be arranged in one or more conceptual configurations or may be arranged in any other manner. The signature may be used for object detection or any other purpose.

The term "substantially" refers to insignificant deviation, e.g., no more than a few percent of the value, less than the difference in accuracy and/or resolution associated with the face recognition process. The content of "substantially" may be defined in any way.

An active sensor is a sensor that emits radiation during sensing. The radiation may be optical radiation (lidar), radio frequency radiation (radar), acoustic waves (sonar), etc.

The terms first camera and second camera are used to indicate that the camera used in the calibration process may be different from the camera used after the calibration process, or may be the same camera.

For simplicity of explanation, it is assumed that the active sensor is a lidar.

A system, method, and non-transitory computer-readable medium may be provided that obtains depth information using images acquired by a camera. The camera is a two-dimensional camera and should not be a three-dimensional camera, such as a stereo camera.

The method obtains a mapping of anchors and their appearance in the image to the camera distance. An anchor is an object associated with at least one physical dimension of known value.

The association may mean that the at least one physical dimension is at least one dimension of the object.

The at least one dimension of the object may be any dimension, such as a height, length, width, radius, diagonal, etc., of the object or any portion thereof. For example, the known value of the dimension of the zebra crossing may be its width.

The association may mean that the at least one physical dimension is a spatial relationship (e.g. distance) related to a portion of the object. For example, the known value of the size of the zebra crossing may be the pitch of the stripes of the same color.

The association may mean that the at least one physical dimension is indicative of a spatial relationship between the object and another object (e.g., a distance, overlap, etc. between the two). For example, the known value of the zebra stripe size may be the pitch of the same color stripe. As another example, the distance between adjacent road dividers.

The at least one physical dimension of known value may be a known accurate value (e.g., the width of the object may be 40 centimeters).

The at least one physical dimension of known value may be within a certain range (e.g., a white adult may be between 160 and 210 centimeters in height). A larger range may reduce the accuracy of the distance determination, but may be better than not estimating the distance at all.

The anchors may be determined in any manner, e.g., defined, determined from a predefined list based on metadata associated with a description of the anchor, etc.

Non-limiting examples of anchors are traffic signs, different types of traffic lights (type refers to different model, different number, etc.), zebra crossings, road spacers, vehicles of a particular model, different types of pedestrians (children, adults, race), etc.

A system, method, and non-transitory computer-readable medium may be provided that performs a calibration procedure or receives results of the calibration procedure.

The calibration process involves the use of a camera and an active sensor, such as a lidar.

One or more images of the camera are then obtained, and distance information is determined based on the results of the calibration process and the appearance of one or more anchors in the one or more images.

Fig. 1 illustrates a calibration method 100.

Method 100 may begin at step 110 with receiving information about one or more anchors. The information may include an identifier for identifying each object that serves as an anchor and at least one physical dimension of a known value associated with each anchor.

Step 110 may be followed by step 120.

Step 120 may include obtaining a plurality of calibration images by a second camera and obtaining range information associated with the calibration images by an active sensor, such as a lidar.

The distance information may be a lidar image or any other information about distance. The lidar image and the calibration image may be taken simultaneously or at least before changing the distance between the camera and its environment.

Obtaining the distance information in the presence of a known spatial relationship between the second camera and the lidar. The second camera and the lidar may be co-located, but this is not essential.

The second camera is a two-dimensional camera rather than a 3D camera.

The plurality of calibration images acquire the one or more anchors. Each calibration image may acquire at least one of the one or more anchors. Given that there are multiple anchors, any of the multiple calibration images may include one, some, or all of the multiple anchors. For example, one calibration image may include a first anchor, a second calibration image may include a second anchor, a third calibration image may include the first and second anchors, and so on.

At least some of the plurality of calibration images may be acquired at different distances from the camera. Thus, one or more calibration images may be acquired at the same distance.

Step 120 may be followed by step 130 of detecting the one or more anchors within the plurality of calibration images and obtaining range measurements (via lidar) from the one or more anchors within the plurality of calibration images.

Step 130 may include making one or more distance measurements by an active sensor while utilizing a spatial relationship (aligned or at least aligned with a known mapping) between pixels of the image acquired by the active sensor and pixels of the one or more calibration images to detect the anchor in the image acquired by the active sensor.

The lidar and the second camera may be aligned such that the fields of view of the lidar and the second camera are the same.

In this case, once an anchor is detected within the calibration image (acquired by the second camera), its location within the lidar image is also known, and a range measurement associated with that anchor can be acquired from the lidar image.

This alignment may eliminate the need to detect the anchor in the lidar image.

Step 130 may include making one or more distance measurements by the active sensor and searching for the anchor in one or more images acquired by the active sensor and searching for the anchor in the one or more calibration images by applying an object detection algorithm.

The lidar and the second camera may not be aligned, but the mapping between the pixels of the calibration image and the pixels of the lidar image should be known so that the position of the anchor in the lidar image can be derived from the position of the anchor in the calibration image and the mapping.

This mapping may eliminate the need to detect the anchor in the lidar image.

According to another example, the second camera and the lidar may operate independently of each other, even without aligning known mappings between pixels.

In this case, the lidar image should be processed to detect the anchor within the lidar image.

This is followed by correlating the range information provided from the lidar image with the anchor in the calibration image.

Step 130 may be followed by step 140 of generating a distance-appearance relationship that maps appearance to distance.

The appearance of a dimension of a known value in an image may be the size of the dimension within the image, e.g. the number of pixels in the image representing the dimension. For example, a traffic signal of known width is displayed in an image as a box of H x W pixels, so the appearance is W pixels.

The result of step 120 is a plurality of tuples (appearance of anchors with known value sizes, distance from the anchors) and the distance-appearance relationship can be generated during step 130 by any function capable of generating a mapping from instances of the plurality of tuples.

Fig. 2 illustrates a method 200.

The method 200 may begin at step 210 by obtaining an image of the vehicle surroundings from a camera of the vehicle.

The surroundings of the vehicle are anything within the field of view of the camera.

Step 210 may be followed by step 220 of searching for an anchor within the image, wherein the anchor is associated with at least one physical dimension of known value.

Step 220 may be followed by step 230 of determining (when the anchor is found) a distance of the camera from the anchor based on (a) the at least one physical dimension of known value, (b) an appearance of the at least one physical dimension of known value in the image, and (c) a distance-appearance relationship that maps the appearance to a distance.

The distance-appearance relationship may be generated by a calibration process, which may include obtaining one or more calibration images of the anchor, and obtaining one or more distance measurements from the anchor.

The method 100 is an example such as a calibration process.

Method 200 may include method 100, for example, as a preliminary step.

The one or more calibration images are a plurality of calibration images acquired at different distances from the second camera, and wherein the one or more distance measurements indicate different distances. The performing of the calibration process may comprise calculating the distance-appearance relationship based on the appearance of the at least one physical dimension of known values in the plurality of calibration images and the different distances.

The one or more distance measurements may be performed by an active sensor, wherein the performing of the calibration process comprises searching for the anchor in one or more images acquired by the active sensor, and searching for the anchor in one or more calibration images by applying an object detection algorithm.

The one or more distance measurements may be performed by an active sensor, wherein the performing of the calibration process includes utilizing a spatial relationship between pixels of the image acquired by the active sensor and pixels of the one or more calibration images to detect the anchor in the image acquired by the active sensor.

Step 220 may include approximating the anchor by a bounding box and measuring at least one dimension of the bounding box to provide a value indicative of an appearance of the at least one physical dimension of a known value in the image.

Examples of defining the bounding box are shown in U.S. patent application US 16/544,940, filed on 2019, 8, 20, which is incorporated herein by reference.

Fig. 3 and 4 show various images of some dimensions including various anchors and known values.

Fig. 3 shows a first image 11 in which the width of the way-giving sign 21 is known (for example 26cm), which appears in the first image as a line of N1 pixels. In the second image 12, the same yield marking 21 of known same width appears in a line of N2 pixels.

Assuming that the first and second images are obtained during the calibration process and at different distances from the yield sign (as measured by an active sensor), these different distances and different appearances may be used to calculate the distance-appearance relationship.

Alternatively, assuming that the first and second images are obtained after the calibration process, the appearance of the yield sign may provide an estimate of the distance of the yield sign from the vehicle at each moment in time.

It should be noted that if the image includes a plurality of anchors, the distance from a plurality of points within the image may be calculated, for example by extrapolation, based on the distances of these anchors in the image and their locations. It should be noted that the latter may also be based on the desired optical properties of the camera, such as distortion.

Fig. 4 shows an example of possible dimensions of known values, such as the diameter of the no entry sign 22, the width of the lane-giving sign 21, the height of an adult, the width of a zebra crossing stripe, the pitch of zebra crossing stripes of the same colour.

Fig. 5 shows a calibration process vehicle 90 that performs a calibration process. The active sensor 91 makes a distance measurement 93 of a yield sign also within the field of view 94 of the camera 92.

Fig. 6 shows a vehicle 90' without an active sensor.

The vehicle 90' may also include a camera 92, a processor 99, a vehicle controller 98, a storage unit 97, one or more vehicle computers (e.g., an autonomous driving controller or ADAS computer 96), and a communication unit 95 for communicating with other vehicles and/or remote computer systems (e.g., cloud computers).

The storage unit may store any data structure, such as a distance-appearance relational database, or the like.

The figure also shows a yield sign with a width shown as N3 pixels indicating its distance from the camera as D3 from the distance-appearance relationship database 89.

While the foregoing written description of the invention enables one of ordinary skill to make and use what is presently considered to be the best mode thereof, those of ordinary skill will understand and appreciate the existence of variations, combinations, and equivalents of the specific embodiments, methods, and examples herein. Accordingly, the present invention should not be limited by the above-described embodiments, methods and examples, but by all embodiments and methods within the scope and spirit of the invention as claimed.

In the foregoing specification, the invention has been described with reference to specific examples of embodiments of the invention. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.

Furthermore, the terms "front," "back," "top," "bottom," "over," "under," and the like in the description and in the claims, are used for description and not necessarily for describing permanent relative positions. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in other orientations than those illustrated or otherwise described herein.

Furthermore, the terms "valid" or "set" and "invalid" (or "deassert" or "clear") are used when referring to setting a signal, status bit, or similar device to its logically true or logically false state, respectively. If the logically true state is a logic level 1, the logically false state is a logic level 0. And if the logically true state is a logic level 0, the logically false state is a logic level 1.

Those skilled in the art will recognize that the boundaries between logic blocks are merely illustrative and that alternative embodiments may merge logic blocks or circuit elements or impose an alternate decomposition of functionality upon various logic blocks or circuit elements. Thus, it is to be understood that the architectures depicted herein are merely exemplary, and that in fact many other architectures can be implemented which achieve the same functionality.

Any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as "associated with" each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being "operably connected," or "operably coupled," to each other to achieve the desired functionality.

Furthermore, those skilled in the art will recognize that the boundaries between the above described operations merely illustrative. Multiple operations may be combined into a single operation, single operations may be distributed in additional operations, and performance of the operations may overlap, at least in part, in time. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be altered in various other embodiments.

As another example, in one embodiment, the illustrated examples may be implemented as circuitry located on a single integrated circuit or within a same device. Alternatively, these examples may be implemented as any number of separate integrated circuits or separate devices interconnected with one another in a suitable manner.

However, other modifications, variations, and alternatives are also possible. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of other elements or steps than those listed in a claim. Furthermore, the terms "a" or "an," as used herein, are defined as one or more. Furthermore, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be construed to imply that the introduction of another claim element by the indefinite articles "a" or "an" limits any particular claim containing such introduced claim element to inventions containing only one such element, even if the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an". The same holds true for the use of definite articles. Unless otherwise specified, terms such as "first" and "second" are used to arbitrarily distinguish between the elements such terms describe. Thus, these terms are not necessarily intended to indicate temporal or other prioritization of such elements. The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage.

While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents will now occur to those of ordinary skill in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

It is appreciated that various features of the embodiments of the disclosure which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the embodiments of the disclosure that are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination.

It will be appreciated by persons skilled in the art that the embodiments of the present disclosure are not limited by what has been particularly shown and described hereinabove. But the scope of the embodiments of the present disclosure is defined by the appended claims and equivalents thereof.

Claims

1. A method for distance measurement, the method comprising:

obtain an image of the surroundings of the vehicle from a camera of the vehicle;

searching for an anchor within the image, wherein the anchor is associated with at least one physical dimension of known value;

When the anchor is found, an appearance based on (a) the at least one physical dimension of known values, (b) the appearance of the at least one physical dimension of known values in the image, and (c) mapping the appearance A distance-appearance relationship to distance determines the distance between the camera and the anchor, wherein the distance-appearance relationship is generated by a calibration process that includes obtaining one or more of the anchors calibration images, and obtaining one or more distance measurements from the anchor.

2. The method of claim 1, comprising performing the calibration process.

3. The method of claim 2, wherein the camera is a first camera, the one or more calibration images are a plurality of calibration images acquired at different distances from a second camera, and wherein the one or a plurality of distance measurements indicating different distances; wherein said performing of said calibration process comprises calculating said different distances based on the appearance of said at least one physical dimension of known values in said plurality of calibration images and said different distances. The distance-appearance relationship is described.

4. The method of claim 3, wherein the one or more distance measurements are performed by an active sensor, wherein the performing of the calibration process is included in one or more images acquired by the active sensor The anchor is searched and found in the one or more calibration images by applying an object detection algorithm.

5. The method of claim 3, wherein the one or more distance measurements are performed by an active sensor, wherein the performing of the calibration process comprises using pixels of the image acquired by the active sensor to match The one or more calibrates the spatial relationship between the pixels of the image to detect the anchor in the image acquired by the active sensor.

6. The method of claim 1, wherein the at least one physical dimension of the known value of the anchor is the size of the anchor.

7. The method of claim 1, wherein the at least one physical dimension of the known value of the anchor is the distance between the anchor and another anchor.

8. The method of claim 1, comprising approximating the anchor by a bounding box, and measuring at least one dimension of the bounding box to provide a value indicative of the known value in the image The appearance of the at least one physical dimension.

9. A non-transitory computer readable medium for storing the following instructions:

Obtain an image of the vehicle's surroundings from the vehicle's camera; search for an anchor within the image, wherein the anchor is associated with at least one physical dimension of known value; when the anchor is found, based on (a ) the at least one physical dimension of known values, (b) the appearance of the at least one physical dimension of known values in the image, and (c) a distance-appearance relationship that maps appearances to distances to determine the the distance between the camera and the anchor, wherein the distance-appearance relationship is generated by a calibration process that includes obtaining one or more calibrated images of the anchor, and obtaining a distance from the anchor one or more distance measurements.

10. The non-transitory computer readable medium of claim 9 storing instructions for performing the calibration process.

11. The non-transitory computer-readable medium of claim 10, wherein the camera is a first camera and the one or more calibration images are a plurality of calibration images acquired at different distances from a second camera, and wherein the one or more distance measurements are indicative of different distances; wherein the performing of the calibration process includes the appearance and all of the at least one physical dimension based on known values in the plurality of calibration images. The different distances are used to calculate the distance-appearance relationship.

12. The non-transitory computer-readable medium of claim 11, wherein the one or more distance measurements are performed by an active sensor.

row, wherein the performing of the calibration process includes searching for the anchor in one or more images acquired by the active sensor, and by applying an object detection algorithm, in the one or more calibration images Search for the anchor.

13. The non-transitory computer-readable medium of claim 11, wherein the one or more distance measurements are performed by an active sensor, wherein the performing of the calibration process comprises utilizing the acquisition by the active sensor The spatial relationship between the pixels of the image and the pixels of the one or more calibration images to detect the anchor in the image acquired by the active sensor.

14. The non-transitory computer-readable medium of claim 9, wherein the at least one physical dimension of the known value of the anchor is a size of the anchor.

15. The non-transitory computer-readable medium of claim 9, wherein the at least one physical dimension of the known value of the anchor is a distance between the anchor and another anchor.

16. The non-transitory computer-readable medium of claim 9, storing instructions for approximating the anchor by a bounding box and measuring at least one dimension of the bounding box to provide an indication of what is in the image. The value of the appearance of at least one physical dimension of the known value.

17. A computerized system comprising a processor configured to: receive an image of an environment surrounding the vehicle from a camera of the vehicle; search for an anchor within the image, wherein the anchor is related to a known at least one physical dimension of the value is associated; when the anchor is found, based on (a) the at least one physical dimension of the known value, (b) the at least one physical dimension of the known value in the image appearance, and (c) a distance-appearance relationship that maps appearance to distance to determine the distance between the camera and the anchor, wherein the distance-appearance relationship is generated by a calibration process that includes One or more calibration images of the anchor are obtained, and one or more distance measurements from the anchor are obtained.

18. The computerized system of claim 17, storing instructions for performing the calibration process.

19. The computerized system of claim 18, wherein the camera is a first camera, the one or more calibration images are a plurality of calibration images acquired at different distances from a second camera, and wherein the One or more distance measurements are indicative of different distances; wherein said performing of said calibration process includes determining the appearance of said at least one physical dimension of known values in said plurality of calibration images and said different distances. The distance-appearance relationship is calculated.