RU2785988C1 - Imaging system and image processing method - Google Patents

Abstract

FIELD: images capturing and processing.

SUBSTANCE: invention relates to systems for capturing and processing images. The effect is achieved in that the imaging system comprises an optical system, an image sensor and an image processing unit, wherein the optical system is configured to form an intermediate image. The image processing unit is configured to restore the intermediate image to obtain an output image with a desired quality level. To restore the intermediate image, optimized parameters are introduced into the image processing unit, obtained on the basis of minimizing the difference in pairs of images formed by two or more models of the optical system with different levels of abstraction. Abstraction levels differ in the number and type of parameters used in describing the passage of light through an optical system. The imaging method is implemented by the imaging system. A method for designing the parameters of an imaging system based on optimizing the parameters by said reduction of differences in image pairs is also provided.

EFFECT: improving the quality of the image restored or processed based on the optimized parameters of the optical system.

17 cl, 6 dwg

Description

The technical field to which the invention belongs

The present invention relates to the field of optical systems, and more particularly to a method and system for imaging based on optimized optical system parameters obtained using optical system models with different levels of abstraction.

State of the art

A real camera (hereinafter also referred to in the context of this application as a "physical" camera, as opposed to a "virtual" camera or model camera) forms a two-dimensional (2D) image on its image sensor based on a three-dimensional (3D) scene of the real world. Then, the two-dimensional image registered by the image sensor (receiver) is processed by the image processing unit to improve the image quality. In the development of advanced imaging cameras, developers face the challenge of simultaneously developing and optimizing the optical system of the camera with the image processing unit (so-called end-to-end optical system design), for which it is preferable to have a model based on the physical camera. This would make it possible to exclude expensive work on creating prototypes of the camera in the process of its design and refinement, as well as to debug the image processing algorithm before creating a prototype of the optical system of a physical camera.

The efforts of many developers of optical systems are currently focused on solving this problem. However, in the prior art, technologies for computer simulation of a 3D scene, formation of a 2D image of the scene by an optical system, and subsequent processing of the 2D image are still mainly based on a pinhole model or a thin lens model. However, these models are not able to properly model the optical system of a physical camera and lens (in particular, such parameters of a physical optical system as depth of field, blur, distortion). In particular, to model the optical system of the physical camera and the image processing unit, machine learning (ML) algorithms based on data sets in the form of image sets representing the reference (ground truth) images of the scene and images after processing by the image processing unit can be used. In this regard, machine learning algorithms that simulate the operation of an image processing unit currently have a number of problems associated, in particular, with unnatural blurring of the background and / or foreground (“bokeh” effect), ghosting and blurring in images generated by the simulated image processing unit. Unnatural blur is expressed in the image of point objects in the form of halos in the form of ovals, rings or semi-rings - "donuts" (in English literature known as donut-shaped bokeh).

Optical systems with a wide field of view used in virtual reality (VR) devices also have a problem that consists in the need to provide high image resolution precisely in the part of the image that the subject’s gaze is directly directed at (in other words, around the “fixation” point of the gaze). subject; in the English literature, such imaging is called foveated imaging) with a lower image resolution at the periphery of the visual field.

In addition, at present, it is problematic to obtain a set of image data for developing a machine learning algorithm based on the parameters of a physical optical system, based on a three-dimensional scene, taking into account the propagation of optical radiation from the imaged scene, the features of image transformation through a specific optical system, an image sensor and a block image processing without the need to manufacture a real (physical) optical system.

US 6,028,606 (Leland Stanford Junior University, published Feb. 22, 2000) discloses a camera simulation system for rendering images in computer graphics. The known solution uses, in particular, camera-based models to synthesize a two-dimensional image from data representing a three-dimensional scene. Also, the known solution relates to a method for modeling a physical camera in a computer graphics system. Based on the characteristics of the lenses of a physical camera, including their sizes, refractive indices, aperture values and shutter characteristics, as well as the orientation of the film surface relative to the lens system and the orientation of the camera within the imaged scene, the known model reproduces the physical principles of imaging, creating an image that approximates the image formed by a real physical camera. However, this well-known solution has a number of disadvantages, among which it should be noted the lack of information about the method of developing an optical system (camera) model, the use of only ray approximation as the basis for the disclosed camera model (without taking into account the wave properties of light), and the inability to generate a data set generated images.

US 20090102841 (Sony Electronics Inc., published 04/23/2009) discloses a virtual camera and lens system in a 3D computer graphics simulation environment that is based on the use of a nominal focal length. The system calculates the true focal length. An optical axis object is created that represents the optical axis of the virtual camera and lens system. Attached to the optical axis is an object located at a location that reproduces the location of the virtual camera system and lens, determined based on the true focal length. The focal length of the virtual camera system and lens is set to the calculated true focal length. In this case, the focal length and aperture value can be determined based on the near and far focus points. The disadvantages of this well-known solution include the fact that it is based on a model of an ideal optical system, and not a real (physical) optical system. In addition, only the focal lens is used for imaging in the prior art system.

US 7,616,842 (Ricoh Co. Ltd., published Nov. 10, 2009) discloses an end-to-end optical system design method in which imaging performance is predicted using a spatial model for a source object and models for an optical subsystem, a detector subsystem, and a digital imaging subsystem. . The optical subsystem and the digital imaging subsystems are designed together taking into account the parameters of the entire system. The intermediate image generated by the optical subsystem does not require high quality, since the quality can be corrected by the digital image processing subsystem. The disadvantages of this known solution include the fact that, from the point of view of determining the performance indicators of the optical system, the source object is only an ideal representation (in particular, a camera with a pinhole), while other options for the components of the optical system (lenses with specific parameters , cameras, etc.) are not taken into account.

WO 2002044650 (Xygent (UK) Ltd., published 06/06/2002) discloses a device for simulating the measurement of a part using a visual measurement system, the device comprising: a computer executing a program that receives a digital model of the part and renders (draws) an image of the model the part, wherein the image is rendered based on conditions simulating the measurement system and evaluates the image to obtain a value representing a physical characteristic of the part. The disadvantages of this well-known solution include the fact that it is based on a model based on the parameters of an ideal optical system, and not a real (physical) optical system.

The known solution according to US 7616842 can be considered as the closest analogue (prototype) for the claimed invention.

Disclosure of invention

This section, which discloses various aspects and embodiments of the claimed invention, is intended to provide a brief description of the claimed objects of the invention and embodiments. A detailed description of technical means and methods that implement combinations of features of the claimed inventions is given below. Neither this disclosure nor the following detailed description and accompanying drawings should be construed as defining the scope of the claimed invention. The scope of legal protection of the claimed invention is determined solely by the attached claims.

Taking into account the prior art solutions discussed above, the objective technical problem solved by the invention is to optimize the parameters of the optical system without the need to manufacture prototypes of the optical system to optimize the parameters.

The problem solved by the present invention is to provide an imaging method and system with improved quality of the final image, as well as a method for designing such an imaging system.

The technical result achieved by using the present invention is to improve the quality of the image restored or processed based on the optimized parameters of the optical system.

In the first aspect of the present invention, this problem is solved by an imaging system that includes an optical system, an image sensor and an image processing unit. In this case, the optical system is configured to form an intermediate image. The image processing unit is configured to restore said intermediate image. At the same time, the imaging system is implemented by jointly optimizing the parameters of the optical system and the image processing unit by minimizing the difference in pairs of images, and the difference in pairs of images is formed by at least two models of the optical system, and at least two models of the optical system contain two or more models of an optical system with different levels of abstraction, made with the possibility of forming images, the modeling of which is carried out by calculating the path of rays or the propagation of a light wave through the corresponding optical system. Abstraction levels of at least two models of an optical system differ in the number and type of parameters used in describing the passage of light through an optical system. In one or more embodiments of the invention, at least one image in said pairs of images is formed using a light wave propagation algorithm (which may otherwise be called wave approximation) and/or an algorithm for calculating the path of rays (which may otherwise be called ray or geometric approximation) from objects three-dimensional scene. The ray tracing algorithm is a forward ray tracing algorithm or a reverse ray tracing algorithm. In one or more embodiments of the invention, said at least two models of the optical system comprise a low abstraction level model that more fully takes into account the behavior of the physical optical system, and at least one high abstraction level model that less fully takes into account the behavior of the physical optical system. The high abstraction model can be a pinhole model, a thin lens model, a foveolar lens model, an achromatic optical system model, a non-blurring distortion optical system model. In one or more embodiments, the imaging pairing may additionally employ at least two image sensor models with different levels of abstraction, wherein at least one of the two image sensor models is a noise model, a spectral selectivity model, and/or a model that takes into account sampling parameters such as sampling, pixel-by-pixel integration, and quantization.

и

с использованием по меньшей мере двух моделей оптической системы с различными уровнями абстракции; и минимизируют различия между изображениями в наборе пар изображений

и

путем оптимизации параметров оптической системы, датчика изображения и/или блока обработки изображений.In the second aspect of the present invention, a method is provided for designing the parameters of an imaging system comprising an optical system, an image sensor and an image processing unit, comprising the steps of: forming at least one set of image pairs

and

using at least two models of the optical system with different levels of abstraction; and minimize differences between images in a set of image pairs

and

by optimizing the parameters of the optical system, image sensor and/or image processing unit.

(также называемую в контексте настоящей заявки нормой невязки, где невязка - разность между эталонным изображением и изображением, полученным физической моделью) для оценки качества изображения, формируемого системой формирования изображений, и получают оптимальные параметры оптической системы, оптимальные параметры датчика изображения и/или оптимальные параметры блока обработки изображений на основании минимизации метрики

. По меньшей мере одно изображение в упомянутых парах изображений может формироваться с использованием алгоритма распространения световой волны и/или алгоритма расчета хода лучей от объектов трехмерной сцены.The abstraction levels of the at least two models of the optical system differ in the number and type of parameters used in describing the passage of light through the optical system. In one or more embodiments of the invention, at least two models of the optical system comprise a low abstraction level model that more fully takes into account the behavior of the physical optical system, and at least one high abstraction level model that less fully takes into account the behavior of the physical optical system. In one or more embodiments, the step of minimizing differences between images comprises a step at which a metric is applied

(also referred to in the context of this application as the residual norm, where the residual is the difference between the reference image and the image obtained by the physical model) to evaluate the image quality generated by the imaging system, and obtain the optimal parameters of the optical system, the optimal parameters of the image sensor and/or the optimal parameters image processing unit based on metric minimization

. At least one image in said pairs of images can be generated using a light wave propagation algorithm and/or an algorithm for calculating the path of rays from objects in a three-dimensional scene.

и

с использованием по меньшей мере двух моделей оптической системы с различными уровнями абстракции; применяют метрику

для оценки качества изображения, формируемого системой формирования изображений, и получают оптимальные параметры оптической системы, оптимальные параметры датчика изображения и оптимальные параметры блока обработки изображений на основании минимизации метрики

; применяют полученные оптимальные параметры оптической системы и оптимальные параметры датчика изображения для формирования промежуточного изображения; вводят промежуточное изображение в блок обработки изображений; применяют оптимальные параметры блока обработки изображений в блоке обработки изображений; и выполняют восстановление изображения на основании оптимальных параметров блока обработки изображений для получения изображения высокого качества. При формировании пар изображений могут дополнительно применяться по меньшей мере две модели датчика изображения с различными уровнями абстракции, причем по меньшей мере одна из двух моделей датчика изображения представляет собой модель, учитывающую шум, модель, учитывающую спектральную селективность, и/или модель, учитывающую параметры выборки.In a third aspect of the present invention, an imaging method is provided by an imaging system comprising an optical system, an image sensor, and an image processing unit. The method comprises the steps of: generating at least one set of image pairs

and

using at least two models of the optical system with different levels of abstraction; apply the metric

to evaluate the quality of the image generated by the imaging system, and obtain the optimal parameters of the optical system, the optimal parameters of the image sensor and the optimal parameters of the image processing unit based on the minimization of the metric

; applying the obtained optimal parameters of the optical system and the optimal parameters of the image sensor to form an intermediate image; introducing an intermediate image into an image processing unit; applying the optimal parameters of the image processing unit in the image processing unit; and performing image reconstruction based on the optimal parameters of the image processing unit to obtain a high quality image. When generating image pairs, at least two image sensor models with different levels of abstraction can be additionally applied, wherein at least one of the two image sensor models is a noise model, a spectral selectivity model, and/or a sampling parameter model. .

As will be apparent to those skilled in the art, the claimed invention may also be embodied in the form of other objects such as, but not limited to, an imaging device, an optical system, an augmented reality (AR) or virtual reality (VR) system, a display device. etc.

Brief description of the drawings

The drawings are given in this document to facilitate understanding of the essence of the present invention. The drawings are schematic and not to scale. The drawings are for illustrative purposes only and are not intended to define the scope of the present invention.

On FIG. 1 is a block diagram of an imaging system according to the invention.

On FIG. 2 illustrates an example of parameter sets for a low abstraction level optical system model (View A) and a high abstraction level optical system model (View B), respectively.

On FIG. 3 shows a flowchart of an imaging method according to the invention.

On FIG. 4 schematically illustrates a model of a pinhole optical system according to one embodiment of the invention.

On FIG. 5 schematically illustrates a model of an "ideal" lens according to one embodiment of the invention.

On FIG. 6 schematically illustrates an exemplary lens profile with a foveolar zone and corresponding zones in an image formed according to one embodiment of the invention.

Implementation of the invention

The present invention is based on a two-level model for obtaining optimized parameters of the optical system and compensating for image imperfections that are characteristic of the result of the operation of a real ("physical") optical system, such as blur (insufficient sharpness), geometric and chromatic aberrations, distortion, ghosting, noise, and etc. The result of the application of the present invention is the formation of a reconstructed image with the desired quality through the use of optimized parameters of the optical system, image sensor and image processing unit.

To do this, the concept of the present invention is based on an approach based on the so-called "end-to-end" (called end-to-end in the English literature) design of optimized parameters of the imaging system (device) using a three-dimensional (3D) object as a source of initial data and/or scene, rather than a two-dimensional image, as in the prior art approaches discussed above. In the general case, in the context of the present invention, a three-dimensional scene is used for this, containing geometric models of one or more three-dimensional objects with specified materials and surface properties, and one or more light sources with specified characteristics.

A 3D scene for the purpose of implementing the present invention may be, as a non-limiting example, a scene previously created using a specific 3D modeling computer program, or a finished scene selected from one or more databases of 3D models and/or scenes. As a non-limiting example, a specific 3D scene can be created and/or selected for use in implementing the invention based on the purpose and required characteristics of the imaging system (device) for which the parameters are being designed in accordance with the invention. As a non-limiting example, and solely for illustrative purposes, for an imaging device that is a front camera of a telephone primarily for capturing a person's face that is 200 to 700 mm away, the 3D scene may include a 3D model corresponding to the face. person, as well as one or more models of three-dimensional objects representing a typical background, such as trees, buildings, etc., which can be located behind the person's face at a distance of more than 1000 mm. The mutual arrangement of these objects in a three-dimensional scene is also set in accordance with the specific problem being solved, an example of which is given above.

At the same time, according to the inventive concept, for the design of optimized parameters of the imaging system, it is not required to manufacture a prototype of a real (“physical”) optical system and/or the imaging system as a whole in order to assess the quality of the resulting image for the considered set of parameters of the optical system. Instead, optical system models with different levels of abstraction are applied and the problem of optimizing the parameters of the optical system is solved, and an image processing algorithm for the image processing unit is developed, as will be described below.

As a result, in addition to designing the parameters of the imaging system to obtain an image with the desired level of quality based on the intermediate image, the proposed technical solution can also provide image data sets for use in a machine learning algorithm designed to optimize the parameters of the image processing unit, which in such cases embodiments of the present invention are the parameters of a neural network or an image processing algorithm, and the like. The work of the proposed technical solution will be described below.

It should be noted that in the context of the present invention, the designed imaging system (device) contains three main components - an optical system (in particular, one or more lenses, one or more lenses or groups of lenses, etc.), an image sensor and a processing unit images. When designing the parameters of the imaging device with the desired quality of the resulting images, the characteristics of some or all of the three above-mentioned components of the imaging device are taken into account.

With regard to the quality indicators of the images generated by the image forming apparatus after optimizing the parameters in accordance with the invention, the following should be noted. In the context of the present invention, quality indicators in various embodiments can be understood as various parameters, including those that are subject to objective quantitative assessment. However, the scope of the present invention is not limited to the specific selection of image quality parameters generated by the imaging apparatus (system) as a result of parameter optimization in accordance with the invention. As non-limiting examples and to clarify the semantic content of the concept of "image quality indicators", the following criteria to be quantified, which are most often used in this field of technology, can be given:

- image contrast (contrast ratio) - a contrast of less than 30% is considered to indicate poor image quality, more than 70% - indicating high image quality;

- change in illumination across the field - for high quality images, it is desirable that the image be illuminated evenly, that is, that the fall in illumination does not exceed 10%;

- distortion less than 2% indicates high image quality. However, for wide-angle lenses, the allowable amount of distortion can be much larger (>20%);

- chromatism - for high quality images it should be less than half a pixel.

- PSNR (peak signal to noise ratio) - for high quality images, this figure is usually greater than 30 dB.

In addition to the above non-limiting examples of objective image quality metrics, mention should also be made of non-numerical (qualitative) image quality metrics that can also affect the perception of the final image generated by the imaging system. As a non-limiting example, the “donut” bokeh mentioned above in the prior art overview, which gives the viewer the impression of an unnatural blur of the foreground and/or background that is outside the focus area. In addition, in the context of the present invention, mention can be made of such quality factors that affect the assessment of the final image by the observer, such as minimizing "ghosting" or blurring, expressed, for example, in the fact that the side lobes in the plot of the scatter function, caused by parasitic diffraction orders from display elements in the camera under the display (UDC), do not exceed, for example, 10% of the maximum of the spread function. In addition to the above, specialists in the art will be aware of other possible criteria for assessing the quality of the final image, both numerical and non-numerical (qualitative), affecting the visual perception of the final image by the observer.

On FIG. 1 schematically illustrates an imaging system in accordance with the first aspect of the present invention, indicated generally by reference numeral 100. The system includes an optical system 110, an image sensor 120, and an image processing unit 130. The optical system 110 is configured to form an intermediate image on the image sensor 120 . The image processing unit 130 is configured to restore said intermediate image, resulting in an output image 300.

At the same time, the imaging system 100 is configured to jointly optimize the parameters of the optical system 110 and the image processing unit 130 by minimizing the difference in pairs of images formed by at least two models 201 and 202 of the optical system, with at least two models 201 and 202 of the optical system The systems are models of an optical system with different levels of abstraction capable of generating images that are modeled by calculating the path of rays or the propagation of a light wave through the corresponding model 201 and 202 of the optical system 110. The abstraction levels of the at least two models 201 and 202 of the optical system differ in the number and type of parameters used in describing the passage of light through the optical system 110. In the following, the low-level abstraction model will also be referred to in this description as the physical model of the optical system, and the high-level model abstraction level - an ideal optical system. It should be noted that, in the general case, an ideal optical system in optics is commonly understood as an aberration-free optical system. However, in the context of the present description, the concept of an “ideal” optical system is not limited to an aberration-free optical system, and, as will be detailed below, various specific high-level abstraction models, which will also be referred to as ideal optical system models, are provided in various embodiments of the present invention. It should be understood that the higher the level of abstraction of the model, the fewer parameters of the “real” (physical) optical system are taken into account in it, and vice versa.

для оценки качества изображения и получают оптимальные параметры системы 100 формирования изображений, минимизируя эту метрику

, которые применяют к оптической системе 110, датчику 120 изображения и блоку 130 обработки изображений для восстановления промежуточного изображения с получением выходного изображения 300 с желаемым уровнем качества.To minimize the difference in pairs of images, a metric is used

to evaluate image quality and obtain the optimal parameters of the imaging system 100 by minimizing this metric

, which are applied to the optical system 110, the image sensor 120, and the image processing unit 130 to reconstruct the intermediate image to obtain an output image 300 with a desired level of quality.

Those skilled in the art will appreciate that the imaging system 100 may be practiced using various combinations of hardware and software to implement the respective blocks and/or perform the respective functions in operation of the system. As a non-limiting example, in addition to the optical system 110, which may be in the form of one or more lenses, lens groups, and the like. depending on the specific embodiment of the practical implementation of the present invention, and the image sensor 120, which, depending on the specific implementation of the system, can be any image sensor known in the art, such as, as a non-limiting example, a CMOS or CCD sensor, the system may involve at least one processing unit, such as a general purpose processor or graphics processing unit (GPU), microprocessor, field-programmable logic integrated circuit (ASIC), or the like. In addition, the system may contain at least one storage device (RAM, ROM, etc.), which stores software and/or firmware that implements the appropriate data processing functions within the framework of the system implementation, in particular, the mentioned models of the optical system , image modeling algorithms, etc.

It should be noted that the scope of the claimed legal protection is not limited to specific logistical means that implement the imaging system or the corresponding methods described in the materials of this application. Specialists in the art will be aware of various combinations of hardware and/or software that allow you to implement the described invention, depending on specific private options for its practical implementation. In addition, it should be noted that at least software and/or firmware that implements the respective functions and/or steps of the inventive methods may be stored on at least one computer-readable medium located physically in the same location as the imaging system or remote from it and connected to it through one or more wired or wireless communication networks.

, параметрами оптических материалов

, положениями оптических элементов

, габаритными характеристиками L), модель датчика изображения (характеризуемого такими параметрами, как спектральная селективность

, параметры «шума»

и параметры выборки (дискретизация, попиксельное интегрирование S, квантование), и модель блока обработки изображения (характеризуемая параметрами цифрового фильтра или весовыми коэффициентами слоев

нейронной сети). В отношении оптической системы, помимо модели «физической» («реальной») оптической системы с низким уровнем абстракции, применяется также модель «идеальной» оптической системы с высоким уровнем абстракции, как будет более подробно описано ниже. Каждая из упомянутых двух моделей оптических систем с различными уровнями абстракции дает по одному изображению, т.е. получается пара изображений одного и того же трехмерного объекта или одной и той же трехмерной сцены с различным качеством, поскольку на изображение, получаемое моделью низкого уровня абстракции (моделью «физической» оптической системы, максимально приближенной к реальной оптической системе) влияют особенности данной оптической системы, в то время как модель высокого уровня абстракции (т.е. «идеальной» оптической системы) не имеет соответствующих особенностей и формирует изображение, лишенное ряда недостатков (таких как размытие, дисторсия, аберрации, шум), вносимых в изображение «физической» оптической системой.The end-to-end design of the optical system, image sensor and image processing unit is carried out using a two-level model of the optical system, in which a low-level abstraction model (corresponding to the "physical" optical system) is used, characterized, in particular, by the parameters of optical surfaces

, parameters of optical materials

, positions of optical elements

, overall characteristics L ), image sensor model (characterized by such parameters as spectral selectivity

, "noise" parameters

and sampling parameters (sampling, pixel-by-pixel integration S , quantization), and an image processing unit model (characterized by digital filter parameters or layer weights

neural network). With regard to the optical system, in addition to the "physical"("real") optical system model with a low level of abstraction, the model of the "ideal" optical system with a high level of abstraction is also applied, as will be described in more detail below. Each of the above two models of optical systems with different levels of abstraction gives one image, i.e. a pair of images of the same three-dimensional object or the same three-dimensional scene with different quality is obtained, since the image obtained by a model of a low level of abstraction (a model of a “physical” optical system that is as close as possible to a real optical system) is affected by the features of this optical system, while a model of a high level of abstraction (i.e., an “ideal” optical system) does not have the corresponding features and forms an image devoid of a number of shortcomings (such as blur, distortion, aberrations, noise) introduced into the image by a “physical” optical system .

More specifically, the optical system and image sensor in the low abstraction model, as in the real optical system of the imaging device, introduce some blurring, distortion, and "noise" into the image. In particular, on the image sensor, said distortions can be caused by non-linearities, saturation unevenness, losses at the stage of converting the analog image formed by the optical system into digital image data.

In accordance with the invention, the low abstraction model generates an intermediate image. Next, the image processing unit reconstructs the image using the aforementioned pairs of images generated by the optical system models with different levels of abstraction.

In addition, in some embodiments, as will be described in more detail below, the designed imager may be used in configurations such as under display camera (UDC) or extended depth of field (EDoF) camera, where at the entrance to the optical system an amplitude-phase mask is also present, which enhances the functionality of the imaging device, but also contributes to image quality degradation, causing undesirable phenomena such as blurring and ghosting (in the case of UDC) or blurring and unnatural "bokeh" (in case with EDoF).

трехмерного объекта

. Математически процесс формирования изображения

может быть представлен в виде оператора от параметров

, который отображает объект

в изображение

, причем функция имеет следующий вид:In other words, the proposed invention is based on a "two-level" model of the optical system. The first "level" uses a model of the "physical" optical system (in other words, a low-level model of abstraction, close to the "real" optical system), which forms the image

three-dimensional object

. Mathematically, the imaging process

can be represented as an operator on parameters

The that renders the object

into the image

, and the function has the following form:

(1),

(one),

являются параметрами оптической системы, такими как радиусы кривизны поверхностей линз, расстояния между линзами, толщина линз и т.п.where

are parameters of the optical system, such as the radii of curvature of lens surfaces, distances between lenses, lens thickness, etc.

получают с использованием алгоритма распространения света, который будет более подробно рассмотрен ниже.Operator

obtained using a light propagation algorithm, which will be discussed in more detail below.

в изображение

, характеризуемое следующим оператором:The image sensor of the "physical" optical system transforms the image

into the image

, characterized by the following operator:

(2),

(2)

являются параметрами датчика изображения.where

are the image sensor parameters.

в изображение

с применением следующего оператора:The image processing algorithm transforms the image

into the image

using the following operator:

(3),

(3)

являются параметрами алгоритма.where

are parameters of the algorithm.

объекта

с помощью оператора:A simplified model of an optical system (that is, a model of an “ideal” optical system with a high level of abstraction) builds a reference (reference) image

object

using the operator:

(4),

(4),

- параметры упрощенной модели.where

- parameters of the simplified model.

и изображение

, полученное на основе модели низкого уровня абстракции.As a result of the operation of the two-level model mentioned above, two images are formed: a reference (reference) image

and image

, derived from the low-level abstraction model.

, полученным на основе модели низкого уровня абстракции, и опорным изображением

, полученным моделью высокого уровня абстракции, получается выходное изображение

, в котором компенсированы такие недостатки, присущие различным конфигурациям «физических» оптических систем и блоков обработки изображения, как размытие изображения, монохроматические и/или хроматические аберрации, двоение изображения, дисторсия, «шум» и т.п., Таким образом решается задача оптимизации для прогнозирования характеристик проектируемого «сквозным» методом устройства формирования изображений в соответствии с изобретением.Based on minimizing differences (residuals) between the image

, obtained on the basis of the low-level abstraction model, and the reference image

obtained by the high-level abstraction model, the output image is obtained

, which compensates for such shortcomings inherent in various configurations of "physical" optical systems and image processing units as blurring of the image, monochromatic and / or chromatic aberrations, ghosting, distortion, "noise", etc., Thus, the optimization problem is solved for predicting the characteristics of the image forming device designed by the "through" method in accordance with the invention.

, полученное на основе модели высокого уровня абстракции, и изображение

, полученное на основе модели низкого уровня абстракции, формируются моделями с различными уровнями абстракции в одной и той же системе. Изображения

используются в метрике

для оценки качества изображения, формируемого камерой. Путем минимизации различия (выражаемого, например, нормой невязки) между упомянутыми двумя изображениями могут быть получены такие параметры проектирования

, которые обеспечивают компенсацию размытия, аберраций и «шума» на получаемом изображении. Это позволяет спроектировать улучшенную систему формирования изображений, в которой уменьшено влияние размытия и «шума» на качество получаемого изображения. Сама по себе упомянутая выше норма невязки может рассматриваться в качестве критерия качества для сравнения итогового изображения, получаемого в соответствии с изобретением после оптимизации параметров оптической системы, с эталонным изображением. В качестве неограничивающего примера, при оптимизации параметров сравниваются один или более доступных алгоритмов обработки изображения для блока обработки изображений, при этом лучшим (и, следовательно, дающим наиболее высокое качество итогового изображения) считается тот алгоритм обработки изображения, который дает меньшее значение нормы невязки. В одном или более неограничивающих вариантах выполнения изобретения, норма невязки (L2 - евклидова) может быть связана со среднеквадратической ошибкой между эталонным изображением и получаемым изображением, при этом чем меньше упомянутая среднеквадратическая ошибка, тем выше качество получаемого изображения.Reference (reference) image

, obtained on the basis of a high-level abstraction model, and the image

, obtained on the basis of a low-level abstraction model, are formed by models with different levels of abstraction in the same system. Images

used in metric

to evaluate the quality of the image generated by the camera. By minimizing the difference (expressed, for example, by the residual norm) between the two images mentioned, the following design parameters can be obtained

, which provide compensation for blurring, aberrations and "noise" in the resulting image. This allows you to design an improved imaging system that reduces the effect of blur and "noise" on the quality of the resulting image. By itself, the above-mentioned residual norm can be considered as a quality criterion for comparing the final image obtained in accordance with the invention after optimizing the parameters of the optical system with a reference image. As a non-limiting example, parameter optimization compares one or more of the available image processing algorithms for an image processing unit, with the image processing algorithm that yields the smallest value of the residual norm being considered the best (and therefore giving the highest quality of the final image). In one or more non-limiting embodiments of the invention, the residual norm (L2 - Euclidean) may be related to the rms error between the reference image and the resulting image, wherein the smaller the rms error, the higher the quality of the resulting image.

It should be noted that the concept of residual is a mathematical term (see, for example, https://ru.wikipedia.org/wiki/%D0%9D%D0%B5%D0%B2%D1%8F%D0%B7%D0 %BA%D0%B0), which in the context of the present invention essentially means the difference between the reference and the received image. This parameter is dimensionless, and the lower its value, the higher the quality of the resulting image and, thus, the better one or another image processing algorithm for the image processing unit.

The discrepancy is considered to be minimized if it ceases to decrease significantly during the optimization process. As a non-limiting example, this condition can be considered satisfied if, at the next optimization step, the residual norm changes by no more than 1% compared to the previous optimization step.

), служит для прогнозирования параметров проектирования устройства формирования изображений

что может быть записано следующим образом:An optimization algorithm aimed at minimizing the differences (residual norms) between two images (the metric mentioned above

), serves to predict the design parameters of the imaging device

which can be written as follows:

(5).

(five).

объекта

. Математически процесс формирования этого изображения может быть описан как следующий оператор от параметров

оптической системы, которая отображает объект

в изображение

:As stated above, a model of a "physical" optical system (a model of a low level of abstraction that approximates a real optical system) forms an intermediate image

object

. Mathematically, the process of forming this image can be described as the following operator from the parameters

optical system that displays an object

into the image

:

, (6)

, (6)

- параметры амплитудно-фазовой маски (если она имеется),

- параметры оптической системы. Датчик изображения преобразует промежуточное изображение

в изображение

:where

- parameters of the amplitude-phase mask (if any),

- parameters of the optical system. The image sensor converts the intermediate image

into the image

:

(7)

(7)

- параметры датчика изображения.where

- image sensor parameters.

в изображение на основе модели низкого уровня абстракции

:The image processing algorithm transforms the image

to an image based on a low-level abstraction model

:

(8)

(eight)

являются параметрами алгоритма.where

are parameters of the algorithm.

объекта

:A simplified model of the optical system (a model of a high level of abstraction) forms a reference (reference) image

object

:

(9)

(nine)

являются параметрами упрощенной модели.where

are parameters of the simplified model.

As a simplified model (also referred to herein as an ideal optical system model or a high level of abstraction model), various non-limiting embodiments of the invention can use, for example, a thin lens model, a foveolar zone lens model, a pinhole model (in the English literature called term pinhole). The use of these models will be described below for various non-limiting embodiments of the present invention.

на основе модели высокого уровня абстракции и изображение на основе модели низкого уровня абстракции

.As a result of the above operations, two images are formed: a reference image

based on a high-level abstraction model and an image based on a low-level abstraction model

.

путем решения задачи оптимизации. В результате выполнения этой операции могут быть найдены параметры проектирования

обеспечивающие оптимальную компенсацию вышеупомянутых недостатков изображения (размытия, аберраций, дисторсии, шума и т.п.), вносимых оптической системой, датчиком изображения и (при ее наличии) амплитудно-фазовой маской. Эти найденные параметры проектирования далее могут использоваться при проектировании усовершенствованного устройства формирования изображений, способного получать изображения, в которых вышеупомянутые недостатки минимизированы.After that, for the above two images, the difference (residual norm) between the two images is minimized based on the metric

by solving the optimization problem. As a result of this operation, design parameters can be found

providing optimal compensation for the above-mentioned image imperfections (blurring, aberrations, distortion, noise, etc.) introduced by the optical system, image sensor and (if available) amplitude-phase mask. These found design parameters can then be used in designing an improved imaging apparatus capable of obtaining images in which the aforementioned disadvantages are minimized.

Based on the two models of the optical system described above (the so-called low abstraction level model and the high abstraction level model), the method according to the invention essentially forms a "virtual" camera, which is used to generate an image based on a three-dimensional scene in the stage of generating an image processing algorithm. The mentioned models of the optical system differ in levels of abstraction - from the lowest level of abstraction, which most closely matches the "real" optical system, to the highest level of abstraction, which corresponds to the model of the "ideal" optical system, or, in other words, the "simplified" model of the optical system.

The levels of abstraction of models determine the amount of information contained in the model, and the higher the level of abstraction, the less information the model contains, and the further this model is from the “real” optical system and closer to the “ideal” optical system, respectively. The amount of information in the model decreases as the level of abstraction increases.

As non-limiting examples of optical system models with different levels of abstraction, one or more of the following models can be used in various embodiments of the invention:

- model of a camera with a pinhole aperture ("camera obscura");

- model of an ideal (thin) lens;

- model of achromatic optical system;

- model of the optical system with distortion without image blur;

- model of a realistic optical system.

In addition, one or more embodiments of the invention may also use one or more image sensor models with different levels of abstraction, such as:

- image sensor model without noise formation;

- image sensor model with wavelength selectivity (spectral selectivity);

is an image sensor model that takes into account the sampling parameters.

To model the performance of an optical system model according to the invention, the following ray path calculation techniques can be used:

- direct ray tracing;

- reverse ray tracing;

- algorithm of wave propagation of light; and

- hybrid modeling using a combination of the above techniques.

The use of these models with different levels of abstraction or different modeling methods allows you to create pairs of images with blurring, distortion, chromatic aberrations, diffraction effects, dependence on depth of field, or without these features. These pairs of images can be used for machine learning (ML) purposes.

датчика изображения выполняется трассировка лучей

в направлении, ограниченном выходным зрачком оптической системы. Для каждого луча, пока луч не вышел за пределы оптической системы и не превысил установленное количество отражений повторяются следующие операции для каждого линзового элемента: находят точки

встречи лучей на оптическими поверхностями линз, находят направление преломленного

, отраженного

и рассеянного

лучей. Затем вычисляют энергию, связанную с соответствующими лучами.The reverse ray tracing algorithm is as follows. For every pixel

image sensor is ray tracing

in the direction limited by the exit pupil of the optical system. For each beam, until the beam has gone beyond the limits of the optical system and has not exceeded the set number of reflections, the following operations are repeated for each lens element: find points

meeting of rays on the optical surfaces of the lenses, find the direction of the refracted

reflected

and scattered

rays. The energy associated with the respective beams is then calculated.

лучей с поверхностями объектов, находят направления преломления

, отражения

и рассеяния

лучей, и вычисляют величины энергии, связанной с упомянутыми выше лучами.For each ray, until it hits the light source, the number of its reflections does not exceed a certain number, or it does not pass by the objects in the scene for each object present in the scene, the following steps are repeated: find the intersection points

rays with the surfaces of objects, find the directions of refraction

, reflection

and scattering

beams, and calculate the energy values associated with the above beams.

от источника света, пока луч не попал во входной зрачок оптической системы или не прошел мимо объектов, или количество его отражений не превысило определенное число, или он не поглощен поверхностями объектов, для каждого объекта в сцене повторяют следующие этапы: находят точки

падения лучей на поверхности объектов; находят направления преломления

, отражения

и рассеяния

лучей. Затем вычисляются величины энергии, связанной с упомянутыми лучами.The direct ray tracing algorithm is as follows. For every beam

from the light source until the beam hits the entrance pupil of the optical system or passes by the objects, or the number of its reflections does not exceed a certain number, or it is not absorbed by the surfaces of the objects, the following steps are repeated for each object in the scene: points are found

falling rays on the surface of objects; find directions of refraction

, reflection

and scattering

rays. Then the energy values associated with said beams are calculated.

лучей с оптическими поверхностями линз; находят направления преломления

, отражения

и рассеяния

лучей. Затем вычисляют величины энергии, связанной с упомянутыми выше лучами.For each beam that enters the entrance pupil of the optical system, until this beam hits the image sensor or a certain number of reflections is exceeded, or it is not absorbed by the surfaces of objects, the following steps are repeated for each lens element: the intersection points are found

rays with optical surfaces of lenses; find directions of refraction

, reflection

and scattering

rays. The energy values associated with the beams mentioned above are then calculated.

The following methods are used to calculate the energy values associated with said beams, by way of non-limiting example.

To estimate the propagation of light through a medium, the Bouguer-Lambert-Beer law is used, on the basis of which the attenuation of light due to its absorption in the material through which the light propagates is estimated:

(10)

(10)

вдоль определенного направления

наблюдения, учитывая функцию для падающего света и двулучевую функцию отражательной способности (BRDF):For an opaque surface, a rendering equation is used that describes the total amount of light emitted from a point

along a certain direction

observations, taking into account the function for the incident light and the two-beam reflectivity function (BRDF):

(11)

(eleven)

where:

- полная спектральная плотность энергетической яркости, направленная во внешнем направлении

;

- total spectral density of radiance directed in the outer direction

;

- излучаемая спектральная плотность энергетической яркости в направлении

;

- radiated spectral density of radiance in the direction

;

- спектральная плотность энергетической яркости, обусловленная отражением излучения от поверхности в точке

;

- spectral density of energy brightness, due to the reflection of radiation from the surface at a point

;

- BRDF (определяемая свойствами непрозрачной поверхности);

- BRDF (determined by the properties of an opaque surface);

- координаты точки пересечения с поверхностью, в которой свет падает на поверхность;

- coordinates of the point of intersection with the surface, in which the light falls on the surface;

- координаты точки, в которой свет выходит из материала или поверхности;

- coordinates of the point at which light exits the material or surface;

- направление выходящего света;

- the direction of the outgoing light;

- направление падающего света;

- the direction of the incident light;

- длина волны;

- wavelength;

- нормаль к поверхности в точке

;

- normal to the surface at a point

;

Ω - hemisphere;

- скалярное произведение (равно косинусу угла падения света);

- dot product (equal to the cosine of the angle of incidence of light);

- коэффициент ослабления; и

- attenuation coefficient; and

- длина оптического пути в среде.

is the length of the optical path in the medium.

In one or more alternative embodiments of the invention, a polarimetric BRDF may also be used to also take into account the effects of polarization.

Fig. 2 illustrates an example of two models of an optical system with different levels of abstraction - in view A of FIG. 2 schematically illustrates the low-level abstraction model using a "real" lens described by a set of 5 parameters as an example, and view B of FIG. 2 schematically illustrates a model of a high level of abstraction using the example of a thin lens characterized by only 1 parameter. Note that the example shown in FIG. 2 is only a schematic simplified example intended to facilitate understanding of the invention, and in fact a low-level abstraction model may be based on a set of characteristics from more than one lens, as shown in FIG. 2, but at least one group of lenses with a certain relative position, with the ability to move to focus and / or change the focal length, etc., as is usually the case in the lenses of real imaging devices.

As stated above, the "physical" optical system model is characterized by a low level of abstraction and a set of at least 5 parameters. This model generates an image characterized by blur, aberrations, distortion, and so on.

The image modeling according to the invention in said low abstraction model is performed based on the light propagation algorithm.

(фокусным расстоянием), данная модель формирует «идеальное» изображение, на котором отсутствуют размытие, аберрации, дисторсия и т.п., свойственные изображению, формируемому моделью «физической» оптической системы, как указано выше. Фокусное расстояние идеальной оптической системы определяется по формуле (12):As for the model of the "ideal" optical system, which, as mentioned above, is characterized by a high level of abstraction and is described by only one parameter -

(focal length), this model generates an “ideal” image, on which there is no blur, aberration, distortion, etc., inherent in the image generated by the model of the “physical” optical system, as indicated above. The focal length of an ideal optical system is determined by formula (12):

(12)

(12)

- радиус первой поверхности;where

- radius of the first surface;

- радиус второй поверхности;

- radius of the second surface;

- толщина;

- thickness;

- показатель преломления;

- refractive index;

- фокусное расстояние.

- focal length.

Ray propagation modeling in one or more non-limiting embodiments of the present invention may be described as follows. The simulation is carried out on the basis of calculating the energy value for propagating rays. In particular, for polished surfaces of objects in a three-dimensional scene, the Fresnel equation can be applied, which describes the reflection and transmission of light incident on the separation surface of various optical media:

(13)

(13)

where R _S is the reflectance for s -polarized light;

- коэффициент отражения для

-поляризованного света;

- reflection coefficient for

- polarized light;

- коэффициент пропускания для s-поляризованного света;

- transmittance for s -polarized light;

- коэффициент пропускания для

-поляризованного света;

- transmittance for

- polarized light;

- показатели преломления сред, из которой падает излучение и в которую оно проходит, соответственно;

are the refractive indices of the media from which the radiation falls and into which it passes, respectively;

- угол падения луча;

- beam incidence angle;

- угол преломления луча;

- beam refraction angle;

- угол отражения луча;

- beam reflection angle;

- коэффициент пропускания;

- transmittance;

- коэффициент отражения;

- reflection coefficient;

- интенсивность пропускаемого света;

- intensity of transmitted light;

- интенсивность отражаемого света;

- intensity of reflected light;

- интенсивность падающего света.

is the intensity of the incident light.

или

поляризованного света, соответственно.To take into account the effects of polarization when calculating the light intensity, it is necessary to use the corresponding components of the reflection and transmission coefficients for

or

polarized light, respectively.

In the event that reflective coatings are applied to the mentioned objects in a three-dimensional scene, instead of the above Fresnel equations for each polarization state, the following dependences of the transmittance and reflectivity on the wavelength and angle of incidence of the light beam are used:

в точке

может быть выражено через следующую формулу дифракции Рэлея-Зоммерфельда (см. Michael Bass “Handbook of optics”, volume I, second edition, New York: McGraw-Hill, 1995):Modeling based on the light propagation algorithm, as a non-limiting example, can be based on the Huygens-Fresnel principle; field

at the point

can be expressed in terms of the following Rayleigh-Sommerfeld diffraction formula (see Michael Bass “Handbook of optics”, volume I, second edition, New York: McGraw-Hill, 1995):

(14)

(fourteen)

(15)

(fifteen)

- величины для поля на апертуре

при

;where

are the values for the field at the aperture

at

;

- расстояние между точкой в апертуре

и точкой наблюдения

- distance between a point in the aperture

and observation point

The following formulas can be used for approximate calculations.

Fraunhofer approximation:

(16)

(16)

(17)

(17)

Fresnel approximation:

(18)

(eighteen)

In addition, in one or more non-limiting embodiments of the invention, it is possible to use the so-called "hybrid" simulation, which is the most commonly used algorithm for estimating the performance of an optical system and is essentially a combination of a ray tracing algorithm and a light propagation algorithm. In this algorithm, at the first stage, ray tracing is performed to the exit pupil. The result of the first stage is the pupil function, which describes how a light wave is affected by its passage through the optical system. The second stage uses the wave propagation of light from the exit pupil to the image plane. One of the above diffraction formulas is used. For each point of the object, ray tracing is performed through the points of the entrance pupil.

As stated above, the first step is to calculate the function for the pupil with respect to the propagation of the rays:

(19)

(nineteen)

- координаты в пределах входного зрачка,

- пропускание вдоль оптического пути

- волновая аберрация, i - мнимое число.where

- coordinates within the entrance pupil,

- transmission along the optical path

- wave aberration, i - imaginary number.

The field in the exit pupil is proportional to the function of the pupil:

At the second stage, the propagation of light is estimated in the wave approximation. One of the above diffraction formulas is applied here. For quick evaluation, the diffraction formula can be approximated via the Fourier transform:

(20)

(twenty)

представляет собой функцию рассеяния:where F is the Fourier transform operator, square

is the scattering function:

(21)

(21)

где x, y - координаты на плоскости датчика изображения или на плоскости изображения. Как указано выше,

определяется формой зрачка и аберрационными свойствами такими как волновые аберрации, выражаемыми как разность оптических путей (OPD) лучей в пучке. PSF описывает реакцию оптической системы на точечный объект. Изображение

формируемое на части датчика изображения (что соответствует полю, в пределах которого PSF может считаться инвариантной), может быть представлено в виде свертки распределения интенсивностей для объектов

и PSF: In addition, in accordance with the invention, image modeling can perform fast image estimation for a planar object. It should be noted that the level of image blur in optics is described by the scattering function

where x, y are coordinates on the image sensor plane or on the image plane. As mentioned above,

is determined by the shape of the pupil and aberrational properties such as wave aberrations, expressed as the optical path difference (OPD) of the beams in the beam. PSF describes the response of an optical system to a point object. Image

formed on a part of the image sensor (which corresponds to the field within which PSF can be considered invariant) can be represented as a convolution of the intensity distribution for objects

and PSF:

(22)

(22)

означает зависимость от длин волн. where subscript

means dependence on wavelengths.

формируемое изображение вычисляется следующим образом:Given the spectral sensitivity (selectivity)

the generated image is calculated as follows:

(23)

(23)

, формируемое на датчике изображения, интегрируется по пикселям датчика с добавлением шума, который характеризуется величиной

и в результате получается изображение, характеризуемое как:Image

, formed on the image sensor, is integrated over the pixels of the sensor with the addition of noise, which is characterized by the value

and the result is an image characterized as:

(24)

(24)

where S is the sample and integration operator.

The resulting image is transferred to the image processing unit.

и

с использованием по меньшей мере одной модели оптической системы с различными уровнями абстракции. На этапе S2 применяют метрику

для оценки качества изображения, формируемого системой формирования изображений, и получают оптимальные параметры оптической системы, оптимальные параметры датчика изображения и оптимальные параметры блока обработки изображений на основании минимизации метрики

.On FIG. 3 schematically illustrates an imaging method according to the invention in one or more non-limiting embodiments. In step S1, at least one set of image pairs is generated

and

using at least one optical system model with different levels of abstraction. In step S2, the metric is applied

to evaluate the quality of the image generated by the imaging system, and obtain the optimal parameters of the optical system, the optimal parameters of the image sensor and the optimal parameters of the image processing unit based on the minimization of the metric

.

In step S3, the obtained optimal parameters of the optical system and the optimal parameters of the image sensor are applied. At step S4, an intermediate image is formed based on the optimal parameters of the optical system and the optimal parameters of the image sensor. At step S5, an intermediate image is introduced into the image processing unit, and the optimal parameters of the image processing unit in the image processing unit are applied. In step S6, image reconstruction is performed based on the optimal parameters of the image processing unit to obtain a high quality image.

Some non-limiting embodiments of the present invention will be described below. It should be understood that, unless expressly stated otherwise, the listed embodiments should not be construed as preferable or otherwise limiting the scope of the claimed legal protection. In addition, it should be understood that in addition to the explicitly described embodiments characterized by certain combinations of features, other embodiments are possible that combine the features of various embodiments of those that are explicitly described, unless otherwise indicated explicitly or does not follow explicitly from the technical nature of specific features.

может использоваться модель точечной диафрагмы (в англоязычной литературе называется pinhole), которую также можно охарактеризовать как модель «камеры-обскура», в которой свет, представляющий изображаемую сцену, проходит сквозь отверстие очень малого диаметра (0,01-5 мм), и создает перевернутое изображение на некотором экране (в качестве которого в контексте изобретения может рассматриваться, например, плоскость поверхности датчика изображения, и также называется в данном контексте плоскостью анализа).In one or more embodiments, as a model of an "ideal" optical system for imaging pairs

a pinhole model can be used (called a pinhole in the English literature), which can also be described as a “camera obscura” model, in which the light representing the imaged scene passes through a hole of a very small diameter (0.01-5 mm), and creates an inverted image on a certain screen (which in the context of the invention can be considered, for example, the surface plane of the image sensor, and is also called the analysis plane in this context).

The advantage of this model is that it allows you to generate an image based on a 3D scene that is sharp throughout the entire depth of the depicted scene (in other words, the entire 3D scene is “in focus”), as opposed to a 2D image generated based on the same 3D scene. conventional optical system, which is, for example, a group of lenses. The principle of image formation in this model is schematically illustrated in Fig. 4, where the 3D scene is shown schematically on the left and the analysis plane on the right.

As illustrated in FIG. 4, the pinhole model works as follows. Light from the rendered 3D scene passes through the aperture and projects an inverted image onto the analysis plane. For simulation, the following parameters are used, illustrated in FIG. 4:

- поле зрения (FoV) или угловой размер, соответствующий сцене, регистрируемой на плоскости анализа;

- высота объекта в сцене;

- высота изображения в плоскости анализа;

- расстояние от объекта трехмерной сцены до апертуры;

- расстояние от апертуры до плоскости анализа;

- увеличение, при этом модель может быть описана следующим выражением:

- field of view (FoV) or angular size corresponding to the scene recorded on the analysis plane;

- the height of the object in the scene;

- image height in the analysis plane;

- distance from the object of the three-dimensional scene to the aperture;

- distance from the aperture to the plane of analysis;

- increase, while the model can be described by the following expression:

(25)

(25)

может использоваться модель «идеальной» линзы, схематично проиллюстрированная на Фиг. 5. В соответствии с данным вариантом выполнения изобретения модель «идеальной» линзы описывается следующими параметрами: (i) фокусным расстоянием

или оптической силой

, и (ii) величиной апертуры, ограниченной диаметром линзы D. Для любого луча, попадающего в апертуру, выполняется трассировка с использованием следующих формул:In another embodiment, as a model of a high level of abstraction in the formation of image pairs

the "ideal" lens model illustrated schematically in FIG. 5. In accordance with this embodiment of the invention, the ideal lens model is described by the following parameters: (i) focal length

or optical power

, and (ii) the aperture value, limited by the lens diameterD. Any ray entering the aperture is traced using the following formulas:

- угол падения;where

- angle of incidence;

- высота луча на линзе;

- height of the beam on the lens;

- угол за линзой;

- angle behind the lens;

- высота луча в плоскости анализа;

- beam height in the analysis plane;

d is the distance between the lens and the analysis plane.

A feature of this model is that any light beam that does not enter the aperture of an ideal lens is considered to be blocked. At the same time, in this model, in contrast to the embodiment of the pinhole diaphragm model according to Fig. 4, lens diameter is taken into account (for illumination estimation). In addition, this model also takes into account the depth of field of the imaged space (DoF).

может использоваться модель ахроматической оптической системы. В отличие от модели низкого уровня абстракции, которой свойственны хроматические аберрации, проявляющиеся в снижении резкости изображения, в появлении цветных контуров, полос, пятен - артефактов изображения, в модели ахроматической оптической системы эти хроматические аберрации отсутствуют. Известно, что хроматические аберрации обусловлены зависимостью показателя преломления среды от длины волны проходящего через нее света. В данном варианте выполнения изобретения модель ахроматической оптической системы не учитывает упомянутую выше зависимость показателя преломления от длины волны. Благодаря этому, в отличие от изображения, формируемого моделью низкого уровня абстракции, модель ахроматической оптической системы формирует изображение без хроматических аберраций и, соответственно, получаемые пары изображений могут использоваться для проектирования оптических систем и/или устройств формирования изображений с исправленными хроматическими аберрациями, а также для разработки алгоритмов обработки изображений для компенсации хроматических аберраций, вносимых оптической системой.In another embodiment, as a model of an "ideal" optical system for imaging pairs

the model of an achromatic optical system can be used. In contrast to the low-level abstraction model, which is characterized by chromatic aberrations, which manifest themselves in a decrease in image sharpness, in the appearance of color contours, stripes, spots - image artifacts, these chromatic aberrations are absent in the achromatic optical system model. It is known that chromatic aberrations are caused by the dependence of the refractive index of the medium on the wavelength of the light passing through it. In this embodiment of the invention, the model of the achromatic optical system does not take into account the above-mentioned dependence of the refractive index on the wavelength. Due to this, in contrast to the image generated by the low-level abstraction model, the achromatic optical system model forms an image without chromatic aberrations and, accordingly, the resulting image pairs can be used to design optical systems and/or imaging devices with chromatic aberrations corrected, as well as for development of image processing algorithms to compensate for chromatic aberrations introduced by the optical system.

In yet another non-limiting embodiment of the invention, a model that excludes aberrations in the generated image and preserves only distortion can be used as a model of a high level of abstraction. In this model of a high level of abstraction, a pinhole is used as the aperture diaphragm, and the system itself is an ordinary lens or a group of lenses, similar to a "physical" optical system. Such a high-level abstraction model allows images to be formed without blur, but with a certain amount of distortion (which may be desirable, for example, in the case of modeling a wide-angle or ultra-wide-angle lens). The resulting pairs of images can be used in the design of optical systems with minimization of image blur and for the development of algorithms for the image processing unit to eliminate blur in the resulting intermediate image. Moreover, in this embodiment, blur and distortion can be processed separately (for example, blur can be minimized by optimizing the optical system, and distortion can be compensated by generating an appropriate algorithm for the image processing unit).

In yet another embodiment of the invention, the model of the optical system can be created based on a combination of light wave propagation and ray tracing algorithms. Both of these techniques have already been described above. It should be noted that the method based on the wave propagation of light algorithm takes into account the diffraction of light, and in the scattering function (PSF) there are side lobes (peaks or maxima) caused, for example, by the presence of a grating in front of the lens in the path of light propagation. At the same time, in the ray tracing technique, light diffraction is not taken into account, and the mentioned side lobes (peaks) are absent. Thus, in this embodiment, the light propagation algorithm takes into account the effects caused by the presence of a grating in the path of light in front of the lens, and the ray tracing technique does not take into account the effects caused by the presence of a grating. This makes it possible to form images in which there is or is not present the effect of light diffraction caused by the presence of a grating in front of the lens, and in practice this can be used, for example, in the design of an imaging device such as an Under Display Camera (English Under Display Camera, UDC) , where the display can be viewed as a grill located in front of the lens of the imaging device.

, моделируемое с применением алгоритма распространения света, а «опорное» изображение

может моделироваться посредством алгоритма трассировки лучей. В таком случае в промежуточном изображении могут присутствовать упомянутые выше «боковые лепестки» или «пики», что вызывает размытие, двоение изображения и т.п., а алгоритм обработки изображения в блоке обработки изображений может быть разработан таким образом, чтобы минимизировать упомянутые «боковые лепестки» (пики), и таким образом в итоговом изображении, формируемом разрабатываемым устройством формирования изображений, будет минимизирован эффект затемнения и дифракции, вызываемый наличием дисплея перед устройством формирования изображений.As a non-limiting example, for designing a UDC, an image generated by a low-level abstraction model can be an image

, modeled using the light propagation algorithm, and the "reference" image

can be modeled using a ray tracing algorithm. In such a case, the "sidelobes" or "spikes" mentioned above may be present in the intermediate image, which causes blurring, ghosting, etc., and the image processing algorithm in the image processing unit can be designed to minimize the mentioned "sidelobes". petals (peaks), and thus the final image generated by the imaging device under development will minimize the effect of shading and diffraction caused by having the display in front of the imaging device.

In yet another non-limiting embodiment, the model of the optical system may or may not take into account, respectively, the polarization of the light entering the designed imaging device. To do this, the optical system can be provided with a polarization-dependent component (for example, a polarizer or a phase plate, in particular half-wave or quarter-wave) located in the path of light at the entrance to the optical system (in front of the lens). The principle of operation of the polarization-dependent component can be based on birefringence, dichroism, or polarization of light that occurs at the interface between media. This component can selectively pass and block, respectively, s-polarized or p-polarized optical radiation, as a result of which the above-mentioned pair of images can be formed, for example, with or without "natural" polarization (for example, sky polarization or glare on the surface water), respectively. This allows the simulation of pairs of images that take into account and do not take into account the influence of polarization, respectively, which can be used to develop an algorithm for modeling a polarizing filter in the image processing unit without the need for a real "physical" polarizing filter in the designed imager.

получают путем моделирования получения изображения обычным датчиком изображения и датчиком изображения, выполненным по технологии «линза на чипе» (в англоязычной литературе называется On-Chip Lens (OCL)), в которой, в качестве неограничивающего примера, два или четыре соседних пикселя имеют общую линзу. Следует отметить, что в технологии OCL, в которой два или четыре пикселя имеют общую линзу, если изображение расфокусировано, датчик изображения обнаруживает двойное изображение (в случае с линзой на два пикселя) или четыре изображения одного и того же объекта (в случае с линзой на четыре пикселя), при этом датчики OCL с линзой на два или четыре пикселя используются, как правило, в системах фазовой автофокусировки (AF) и автофокусировка выполняется посредством минимизации упомянутых двоений изображений объектов. Моделирование датчика изображения с OCL позволяет моделировать работу системы автофокусировки в оптической системе, а формируемые пары изображений могут использоваться для компенсации расфокусировки.In yet another embodiment of the invention, said simulated image pairs

is obtained by simulating the acquisition of an image by a conventional image sensor and a lens-on-a-chip image sensor (called On-Chip Lens (OCL) in the English literature) in which, as a non-limiting example, two or four adjacent pixels share a lens . It should be noted that in OCL technology, in which two or four pixels share a common lens, if the image is out of focus, the image sensor detects a double image (in the case of a two-pixel lens) or four images of the same object (in the case of a lens on four pixels), while OCL sensors with a lens of two or four pixels are used, as a rule, in phase-detection autofocus (AF) systems and autofocusing is performed by minimizing the said ghost images of objects. Image sensor simulation with OCL allows you to simulate the operation of the autofocus system in the optical system, and the generated image pairs can be used to compensate for defocus.

In other embodiments of the present invention, an image data set may be generated using more than two different models with different levels of abstraction. So, for example, in addition to the low and high abstraction model, a "medium" abstraction model can be additionally used, which is, as a non-limiting example, a model that generates images with distortion but without blur, as described above. As a result, different pairs of images can be used to design different components of the imaging device (system). So, for example, a pair from an image obtained by a low abstraction level model (image 1) and an image obtained by a high abstraction level model (image n) can be used to design an optical system with aberration (blurring) compensation, and a pair from an image obtained by the model " middle" level of abstraction (images n-1), and images n can be used to design an image processing unit with distortion compensation.

In addition, mention should also be made of possible non-limiting embodiments related to the correction of the unnatural "bokeh" effect mentioned above. This can be implemented in a manner including designing an optical system and developing an algorithm for an image processing unit, which in particular comprises the steps of:

- forming a set of image data using models with different levels of abstraction (in particular, the model of the "physical" optical system and the model of the "ideal" lens or pinhole, described above);

- minimize the difference (discrepancy rate) between the images of the data set by optimizing the characteristics of the optical system, sensor and image processing unit.

полученное моделью «физической» оптической системы имеет неестественный эффект «боке» в виде «бубликов», который отсутствует на опорном изображении

получаемом посредством модели идеальной оптической системы, как описано выше. В результате в реконструированном изображении

проявляется естественно выглядящий эффект «боке» с сохранением качества изображения как в областях, которые находятся «в фокусе», так и в тех областях, которые находятся за пределами глубины резкости.At the same time, the image

obtained by the model of the "physical" optical system has an unnatural "bokeh" effect in the form of "donuts", which is absent in the reference image

obtained through the model of an ideal optical system, as described above. As a result, in the reconstructed image

a natural-looking “bokeh” effect is produced while maintaining image quality both in areas that are “in focus” and in those areas that are outside the depth of field.

In one or more non-limiting embodiments, the invention makes it possible to implement a method of so-called "retinal imaging" (the term foveated imaging is used in the English literature), which can be used when applying the invention in virtual reality (VR) systems. In this case, as a model with a low level of abstraction, a model of any lens with any specific level of abstraction, depending on the specific problem being solved, can be used. At the same time, as a model with a high level of abstraction, a special composite lens with a foveolar zone, which has an area (in particular, an area of a certain size in the center of a circular lens) with a power different from that of the rest of the lens, can be used. Such a lens configuration makes it possible to form an image in which the sharpness and resolution of the optical system in the central part of the image, where the eye of the observer is directed, is higher than in the peripheral parts of the image. On FIG. 6 schematically illustrates an exemplary lens profile with a foveolar zone and shows two zones in the image - a more densely shaded circular zone in the center corresponds to an area with increased sharpness and resolution, and a less densely shaded zone represents a peripheral region where sharpness and resolution are less. This configuration is more close to the perception of the image by the human eye, when in the center of attention (where the observer's eye is focused) the sharpness and resolution are higher than at the periphery. This embodiment of the invention enables "retinal imaging" imaging, which is particularly advantageous for augmented reality (AR) and virtual reality (VR) applications.

, где

- изображение, формируемое на сетчатке моделью с низким уровнем абстракции, а

- «идеализированное» изображение, формируемое на сетчатке моделью с высоким уровнем абстракции. Применение изобретения при проектировании системы VR обеспечивает полезные эффекты, состоящие, в частности, в компенсации размытия изображения, минимизации «эффекта решетки», повышении разрешения изображения, демонстрируемого пользователю системой VR, обеспечение «визуализации на сетчатке».In yet another embodiment, for designing an imaging device in a VR system, optimization of the parameters of the image processing unit in accordance with the invention can be performed based on a pair of images, the first of which is created by a lens model commonly used in VR systems (low abstraction model), and the second - with a model of a lens with a foveolar zone (a model with a high level of abstraction) described above, while the optimization of the parameters of the image processing unit is carried out on the basis of minimizing the metric

, where

- an image formed on the retina by a model with a low level of abstraction, and

- "idealized" image formed on the retina by a model with a high level of abstraction. The use of the invention in the design of a VR system provides beneficial effects, consisting, in particular, in compensating for image blur, minimizing the "lattice effect", increasing the resolution of the image shown to the user by the VR system, providing "visualization on the retina".

In one or more non-limiting embodiments of the invention, the proposed technical solution can be used to design an augmented reality (AR) system display. At the same time, according to the invention, a set of image data is formed using AR optical system models with different levels of abstraction, and various methods for estimating the parameters of the optical system are used in modeling - the wave propagation algorithm of light and the ray tracing algorithm, as described above. Then, optimization of the parameters is carried out by minimizing the differences in the pairs of images obtained by the simulation, by optimizing the projection lens, the combiner, which is a device for combining the image of the real world and the image generated by the AR device, and depending on the specific implementation, which does not limit the scope of the invention, can be made in the form of a waveguide or a beam-splitting cube or a prism with a translucent coating and a compensator, and an image processing unit. Further, the optimized parameters of the projection lens, combiner and image processing unit are applied in the reconstruction of the corresponding images formed on the display of the AR system, as well as images of the real world observed by the user through the AR system. This makes it possible to obtain high image quality in the AR system for both the image of the real world observed through the head unit of the AR system, and augmented reality images formed on the display of the AR system.

When implementing the above-described embodiments of the present invention, at least the following technical effects are achieved. When applying the invention in cameras with an increased depth of field of the imaged space (in the English literature, the term Extended Depth of Field Camera, EDoF is used) in such devices as front and rear cameras of smartphones, tablet computers, security cameras, the invention makes it possible to compensate for image blur in areas scenes that are out of focus. In "Under Display Cameras" (UDC) the invention makes it possible to compensate for the effects of diffraction and obscuration caused by having the display in front of the imaging device. In virtual reality (VR) display optical systems, the invention allows aberrations to be minimized and image pre-processing parameters to be corrected by the image processing unit. In an augmented reality (AR) system, the application of the invention makes it possible to minimize the aberrations introduced by the lens and the effects of diffraction introduced by the combiner (for example, based on a waveguide with diffraction gratings). It should be noted that the above examples do not limit the range of possible applications of the invention, and those skilled in the art will recognize various other examples of imaging devices and systems that can be designed by optimizing the parameters in accordance with the invention and reconstruct images with an increased level of quality and compensation for shortcomings. introduced into images by known optical systems, image sensors and image processing units.

The present invention can be used in the design of imaging devices and systems for various applications, in particular, as a non-limiting example, in portable user devices such as a smartphone, tablet computer, laptop computer, laptop, etc., in augmented systems ( AR) and virtual reality (VR), in various head-mounted and helmet-mounted display devices, in projection displays, including displays on the windshield, etc.

Other embodiments that may be within the scope of the present invention may be contemplated by those skilled in the art upon careful reading of the foregoing description with reference to the accompanying drawings, and all such obvious modifications, alterations and/or equivalent substitutions are deemed to be within the scope of the present invention. While the various features of the invention, characterizing private variants of its implementation, are set forth in the dependent claims, the combination of the features of various dependent claims in other combinations not explicitly described in this document is not excluded.

All prior art references cited and discussed in this document are hereby incorporated into this specification by reference as far as applicable.

While the present invention has been described and illustrated with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes may be made to its form and specific details without departing from the scope of the present invention, which is defined only the following claims and their equivalents.

Claims (30)

1. An imaging system comprising an optical system, an image sensor and an image processing unit, wherein: the optical system is configured to form an intermediate image; the image processing unit is configured to restore said intermediate image; wherein the imaging system is implemented by jointly optimizing the parameters of the optical system and the image processing unit by minimizing the difference in pairs of images, wherein the difference in pairs of images is formed by at least two models of the optical system, and at least two models of the optical system contain two or more models of the optical system with different levels of abstraction, the modeling of which is carried out by calculating the path of rays or the propagation of a light wave through the corresponding optical system, abstraction levels of at least two models of the optical system differ in the number and type of parameters used in describing the passage of light through the optical system. 2. The system of claim. 1, in which at least one image in said pairs of images is formed using a light wave propagation algorithm and/or a ray tracing algorithm from objects of a three-dimensional scene. 3. The system of claim 2, wherein the ray tracing algorithm is a forward ray tracing algorithm or a reverse ray tracing algorithm. 4. The system of claim 1, wherein said at least two optical system models comprise a low abstraction level model more fully taking into account the behavior of the physical optical system, and at least one high abstraction level model less fully taking into account the behavior of the physical optical system. 5. The system of claim 4, wherein the high abstraction model is a pinhole model. 6. The system of claim 4, wherein the high abstraction model is a thin lens model. 7. The system of claim 4, wherein the high abstraction model is a foveal lens model. 8. The system of claim 4, wherein the high abstraction model is a model of an achromatic optical system. 9. The system of claim. 4, wherein the high abstraction model is a model of a distorting optical system without blurring. 10. The system of claim. 1, in which the formation of pairing images additionally used at least two models of the image sensor with different levels of abstraction, and at least one of the two models of the image sensor is a model that takes into account noise, a model with spectral selectivity and /or a model that takes into account sampling parameters such as sampling, pixel-by-pixel integration, and quantization. 11. A method for designing the parameters of an imaging system containing an optical system, an image sensor and an image processing unit, comprising the steps of: form at least one set of image pairs

and

using at least two optical system models with different levels of abstraction and minimize differences between images in a set of image pairs

by optimizing the parameters of the optical system, image sensor and/or image processing unit. 12. The method of claim 11, wherein the levels of abstraction of the at least two optical system models differ in the number and type of parameters used in describing the passage of light through the optical system. 13. The method of claim 11, wherein said at least two optical system models comprise a low abstraction level model more fully taking into account the behavior of the physical optical system and at least one high abstraction level model less fully taking into account the behavior of the physical optical system. 14. The method of claim 11, wherein the step of minimizing differences between images comprises the step of applying a metric in the form of a residual norm

to evaluate the quality of the image generated by the imaging system, and obtain the optimal parameters of the optical system, the optimal parameters of the image sensor and/or the optimal parameters of the image processing unit based on the minimization of the metric

. 15. The method according to claim 11, wherein at least one image in said pairs of images is formed using a light wave propagation algorithm and/or an algorithm for calculating the path of rays from objects in a three-dimensional scene. 16. A method for imaging by an imaging system comprising an optical system, an image sensor and an image processing unit, the method comprising the steps of: form at least one set of image pairs

and

using at least two models of the optical system with different levels of abstraction; apply the metric

to evaluate the quality of the image generated by the imaging system, and obtain the optimal parameters of the optical system, the optimal parameters of the image sensor and the optimal parameters of the image processing unit based on the minimization of the metric

applying the obtained optimal parameters of the optical system and the optimal parameters of the image sensor to form an intermediate image; introducing an intermediate image into an image processing unit; apply the optimal parameters of the image processing block in the image processing block, and performing image restoration based on the optimal parameters of the image processing unit to obtain a high quality image. 17. The method of claim 16, wherein the image pairing further employs at least two image sensor models with different levels of abstraction, wherein at least one of the two image sensor models is a noise-aware model, a spectral selectivity model, and /or a model that takes into account sampling parameters such as sampling, pixel-by-pixel integration, and quantization.

Images