TWI612334B - Photometric compensation method and system for a see-through device - Google Patents

Abstract

A photometric compensation method for a see-through device first provides a photometric model whose total response is equal to the sum of the response of the device light from the see-through device and the response of the scene light from the scene. The calibration phase is performed in the conversion domain, which is only relevant to the characteristics of the image capture device of the projector and the see-through device. During the compensation phase, the original image is responded to in the darkroom, and the response of the image is determined based on the response of the original image and the response of the scene light. The compensated image is generated based on the response of the compensated image.

Description

Photometric compensation method and system for see-through device

The present invention relates to photometric compensation, and more particularly to a photometric compensation method and system for a see-through device.

Smart glasses are a tool for augmented reality, which allows users to receive additional information from the surrounding world by projecting images from embedded projectors. The user can see the projected image and the real world scene. Because the amplified visual message is digitally operated, it can produce an interesting and interactive user experience.

The tiny projectors of most smart glasses have much less power than traditional projectors. When the projected image is mixed with the scene and is placed on the user's retina, if the irradiance of the projector is the same or less than the illumination of the scene light, it is easy to cause photometric distortion. This luminosity distortion is a major determinant of the quality of smart glasses.

Although the scene light causes luminosity distortion, in order to eliminate the luminosity distortion, we must consider the characteristics of the scene light, the reflectance of the projector and the smart glasses, which can be achieved by using the camera and a set of calibration patterns. To reach. The projector projects the amplified or calibrated image to the user's eyes, while the camera is responsible for capturing the image of the scene.

Whenever the scene of the smart glasses' field of view changes or the user moves, the photometric calibration must be re-executed, thus interrupting the user's interaction. Another determinant of the quality of smart glasses is efficiency. It takes time to project and process the calibration graphic, which typically takes seconds to tens of seconds, so it cannot be accepted by real time applications. Therefore, it is urgent to propose a novel mechanism to overcome the lack of traditional methods.

In view of the above, one of the objects of embodiments of the present invention is to provide a photometric compensation method and system for a see-through device. In one embodiment, an algorithm for photometric compensation based on distorted images is proposed that requires only one photometric calibration. Based on the distorted image captured at each time point for each subsequent compensation operation, no recalibration is required, thereby achieving instant photometric compensation.

In accordance with an embodiment of the invention, a photometric model is provided with a total response equal to the sum of the response of the device ray from the fluoroscopy device and the response of the scene ray from the scene. The calibration phase is performed in the conversion domain, which is only relevant to the characteristics of the image capture device of the projector and the see-through device. During the compensation phase, the original image is responded to in the darkroom, and the response of the image is determined based on the response of the original image and the response of the scene light. The compensated image is generated based on the response of the compensated image.

100‧‧‧Photometric compensation system

11‧‧‧Projector

12‧‧‧Image capture device/camera

13‧‧‧ calibration device

131‧‧‧Channel decoupling unit

14‧‧‧Compensation device

141‧‧‧Brightness generating unit

142‧‧‧Scenario generating unit

143‧‧‧Compensation decision unit

144‧‧‧Compensated image generation unit

200‧‧‧Photometric compensation method

21‧‧‧Projected imagery

22‧‧‧Capture device light and scene light

23‧‧‧Use decoupling conversion for calibration

24‧‧‧Determining the response of the original image of the darkroom C(I _O )

25‧‧‧Determining the response of the scene light C(S)

26‧‧‧Determining the response of the compensated image C(I _C )

27‧‧‧ Generated compensation image I _C

300‧‧‧ device

31‧‧‧Wisdom glasses

32‧‧‧稜鏡

33‧‧‧Device light

34‧‧‧ Scene light

The first figure shows a system block diagram of a photometric compensation system of a see-through device of an embodiment of the present invention.

The second figure shows a flow chart of the photometric compensation method of the see-through device of the embodiment of the present invention.

The third diagram shows a schematic diagram of an apparatus for performing the photometric compensation system (first diagram) and the photometric compensation method (second diagram) of the present embodiment.

The fourth A diagram and the fourth B diagram illustrate the spectral sensitivity of the projector and the spectral sensitivity of the camera, respectively.

The first figure shows a system block diagram of a photometric compensation system 100 of a see-through device in accordance with an embodiment of the present invention, and the second figure shows a flow chart of a photometric compensation method 200 of a see-through device in accordance with an embodiment of the present invention. The blocks of the first figure and the steps of the second figure may be implemented by hardware, software or a combination thereof, and may be executed by a processor such as a digital image processor. The see-through device can be a wearable see-through device, such as smart glasses, but is not limited thereto.

The third figure shows a schematic diagram of an apparatus 300 for performing the photometric compensation system 100 (first diagram) and the photometric compensation method 200 (second diagram) of the present embodiment. The device 300 includes a projector 11 (e.g., a microprojector) that projects an image to the smart glasses 31 by a prism 32 (step 21). Image capture device 12 (e.g., a camera) captures device light 33 from smart glasses 31. Camera 12 also captures scene ray 34 from the scene (step 22). The purpose of this embodiment is to counteract the effects of the scene ray 34 so that the color of the projected image is preserved.

In this embodiment, a photometric model is first provided. The traditional photometric model assumes that the scene ray is fixed or negligible relative to the device ray. However, the photometric model of this embodiment considers both device light and scene light. The photometric model of this embodiment can be expressed as a vector form as follows: T(I, S) = C(I) + C(S) = MG(I) + C(S) (1) Where T(I,S) is the total camera response, C(I) is the camera response of the device light, C(S) is the camera response of the scene light, and M is the channel mismatch between the projector 11 and the camera 12. (mismatch), G(.) is the gamma function of the projector 11.

The fourth A and fourth B diagrams illustrate the spectral sensitivity of the projector 11 and the spectral sensitivity of the camera 12, respectively, showing the channel mismatch between the projector 11 and the camera 12.

In step 23, the calibration phase is performed in the darkroom using the calibration device 13 to block the scene light, thereby directly obtaining the camera response of the device light. Thereby, the formula (1) becomes: T(I, S) = C(I) = MG(I) (2)

其中

為解耦合轉換，其僅相關於投影機11與相機12的特性，且V(．)為尺度(scaled)伽碼函數。 In general, because M and G(.) are coupled unknowns, it is difficult to solve directly. According to one of the features of the present embodiment, the channel decoupling unit 131 is used to perform the calibration phase in the transformed domain, and thus the equation (2) can be expressed as follows:

among them

For decoupling conversion, it is only relevant to the characteristics of projector 11 and camera 12, and V(.) is a scaled gamma function.

與V(．)，因此無論場景或影像如何動態改變，僅須進行一次的計算即可。為了加速校準程序，可使用查表(look up table)來建構V(．)。 Since the solution problem of M and G(.) is converted to

With V(.), no matter how the scene or image changes dynamically, only one calculation is required. To speed up the calibration process, a look up table can be used to construct V(.).

(I,S)可寫為：

其中X

{R,G,B}，V_x(．)定義為尺度伽碼函數。 In detail, the decoupling camera response for each channel X

(I, S) can be written as:

Where X

{R, G, B}, V _x (.) is defined as a scale gamma function.

與V(．)相當於求解M與G(．)。

的求解細節可參考格羅斯伯格(M.D.Grossberg)等人於電氣與電子工程師協會會刊-電腦視覺與圖形識別(Proc.IEEE Computer Vision and Pattern Recognition(CVPR))2004年，卷1，頁452-459所發表的“使一物件看起來像另一物件：使用投影機-相機系統控制外貌(Making One Object Look Like Another：Controlling Appearance Using a Projector-Camera System)”。 Thereby solving

And V(.) is equivalent to solving M and G(.).

For details of the solution, see MDGrossberg et al., Proc. IEEE Computer Vision and Pattern Recognition (CVPR) 2004, Volume 1, page 452- 459, "Making One Object Look Like Another: Controlling Appearance Using a Projector-Camera System".

Next, the photometric compensation phase is performed using the compensation device 14. The total camera response of the original image is expressed as: T(I _O ,S)=C(I _O )+C(S) (5)

The total camera response of the compensated image is expressed as: T(I _C , S)=C(I _C )+C(S) (6)

For photometric compensation, the total camera response T(I _C ,S) of the compensated image must be equal to the camera response C(I _O ) of the original image of the darkroom, ie: T(I _C ,S)=C(I _C ) +C(S)=C(I _O ) (7)

In order to get C(I _C ), we must first obtain C(I _O ) and C(S). At step 24, a luminance generation unit 141 is used to obtain C(I _O ):

On the other hand, in step 25, the scene generation unit 142 is used and C(S) is obtained according to the equation (5) because T(I _O , S) and C(I _O ) are known. Thereby, in step 26, the compensation response unit 143 is used and the camera response C(I _C ) of the compensated image can be determined based on C(I _O ) and C(S).

(I_C)之後，於步驟27，使用補償影像產生單元144以得到I_C：

其中

When getting

After (I _C ), in step 27, the compensated image generating unit 144 is used to obtain I _C :

among them

According to the above, the present embodiment proposes a method for compensating for the luminosity distortion of the see-through smart glasses. Since the photometric compensation program uses only the distorted image, the method of the embodiment does not need to be recalibrated, so the user interaction is not interrupted. Thereby, the method of the embodiment can achieve the real-time performance of the augmented reality application of the smart glasses. The method of the present embodiment can be applied to the case where the intensity of the scene light is the same as the light of the device. When the scene light is smaller than the device light, the photometric distortion can be ignored. On the other hand, when the scene light is larger than the device light, it is difficult to recover the image by photometric compensation. In this case, the user can use the principle similar to sunglasses to reduce the scene light, or increase the power of the projector of the smart glasses.

The above description is only the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention; all other equivalent changes or modifications which are not departing from the spirit of the invention should be included in the following Within the scope of the patent application.

200‧‧‧Photometric compensation method

21‧‧‧Projected imagery

22‧‧‧Capture device light and scene light

23‧‧‧Use decoupling conversion for calibration

24‧‧‧Determining the response of the original image of the darkroom C(I _O )

25‧‧‧Determining the response of the scene light C(S)

26‧‧‧Determining the response of the compensated image C(I _C )

27‧‧‧ Generated compensation image I _C

Claims (20)

A photometric compensation method for a see-through device, comprising: providing a photometric model whose total response is equal to the sum of the response of the device light from the see-through device and the response of the scene light from the scene; performing a calibration phase in the conversion domain, which is only related to the projection The characteristics of the image capturing device of the machine and the see-through device; performing a compensation phase, obtaining a response of the original image in the darkroom, and determining a response of the compensated image according to the response of the original image and the response of the scene light; and according to the compensated image The response is to produce a compensated image. The photometric compensation method of the see-through device according to claim 1, wherein the response of the device light is equal to a product of a channel mismatch of the projector and the image capturing device and a gamma function of the projector. The photometric compensation method of the see-through device according to claim 2, wherein the calibration phase is performed in a dark room to block the scene light, so that the response of the device light is uniquely obtained. A photometric compensation method of a see-through device according to claim 3, wherein the response of the device light is equal to a product of the decoupling conversion and the scale gamma function. The photometric compensation method of the see-through device according to claim 1, wherein the response of the compensated image is equal to the response of the original image of the darkroom minus the response of the scene light. A photometric compensation system for a see-through device, comprising: a calibration device that performs a calibration phase in a conversion domain, which is only related to characteristics of the projector and the image capturing device of the see-through device, and provides a photometric model whose total response is equal to the perspective a sum of a response of the device light of the device and a response of the scene light from the scene; and a compensation device that performs a compensation phase to obtain a response of the original image in the darkroom, and then determines a compensated image according to the response of the original image and the response of the scene light The response image is generated based on the response of the compensated image. The photometric compensation system of the see-through device of claim 6, wherein the response of the device light is equal to a product of a channel mismatch of the projector and the image capturing device and a gamma function of the projector. The photometric compensation system of the see-through device of claim 7, wherein the calibration phase is performed in a darkroom to block scene light, thereby uniquely obtaining a response of the device light. A photometric compensation system for a see-through device according to claim 8 wherein the response of the device light is equal to the product of the decoupling conversion and the scale gamma function. The photometric compensation system of the see-through device of claim 9, wherein the calibration phase has to be performed only once, regardless of whether the image or scene projected to the see-through device is dynamically changed. The photometric compensation system of the see-through device according to claim 6, wherein the response of the compensated image is equal to the response of the original image of the darkroom minus the response of the scene light. The photometric compensation system of the fluoroscopy device of claim 6, wherein the compensation device comprises: a brightness generating unit that generates a response of the original image; and a scene generating unit that generates a response of the scene light after the calibration phase; The determining unit determines a response of the compensated image according to the response of the original image and the response of the scene light; and the compensated image generating unit generates the compensated image according to the response of the compensated image. The photometric compensation system of the see-through device of claim 6, wherein the see-through device comprises smart glasses. A see-through device comprising: at least one lens; a projector that projects an image to the lens; an image capturing device that extracts light from the lens and captures scene light from the scene; and the calibration device performs a calibration phase in the conversion domain, It is only related to the characteristics of the projector and the image capturing device, and provides a photometric model whose total response is equal to the sum of the response of the device light and the response of the scene light; and the compensation device performs the compensation phase and is obtained in the darkroom. The response of the original image is determined according to the response of the original image and the response of the scene light to compensate the response of the image; wherein the compensated image is generated according to the response of the compensated image. The see-through device of claim 14, wherein the response of the device light is equal to a product of a channel mismatch of the projector and the image capture device and a gamma function of the projector. The see-through device of claim 15 wherein the calibration phase is performed in a darkroom to block scene light to obtain a unique response to the light of the device. The see-through device of claim 16, wherein the response of the device light is equal to a product of the decoupling conversion and the scale gamma function. The see-through device of claim 17, wherein the calibration phase has to be performed only once, regardless of whether the image or scene is dynamically changed. The see-through device of claim 14, wherein the response of the compensated image is equal to the response of the original image of the darkroom minus the response of the scene light. The fluoroscopy device of claim 14, wherein the compensation device comprises: a brightness generating unit that generates a response of the original image; a scene generating unit that generates a response of the scene light after the calibration phase; and a compensation determining unit, And determining a response of the compensated image according to the response of the original image and the response of the scene light; and compensating the image generating unit to generate the compensated image according to the response of the compensated image.