TWI669682B - Image processing system and image processing method - Google Patents

Abstract

An image processing system includes: a camera, a positioning device, an attitude estimating device, and a processor. The camera is used to capture a real environment. The positioning device is used to detect an imaging position of the camera. The attitude estimating device is used to detect an imaging posture of the camera. The processor is configured to infer a light source information according to the time information and the latitude information, and according to the image capturing position, the image capturing posture, the real environment information corresponding to the real environment, the light source information, and the first virtual information of one of the first virtual objects and a light source tracking ( Ray tracing) algorithm to present the reflection of the real environment on the first virtual object.

Description

Image processing system and image processing method

The present invention relates to an image processing system and an image processing method, and more particularly to an image processing system and an image processing method applied to an augmented reality.

In general, in the augmented reality technology, it is easy to cause a phenomenon that a virtual object does not fuse with a real environment or a virtual object is not real enough. This type of defect is usually caused by not referring to the real environment when rendering a virtual object. For example, when a user views a scene in an augmented reality, the light and shadow of the virtual object are not adjusted following the direction or angle of the camera.

Therefore, how to make virtual objects closer to the real environment in augmented reality has become one of the problems to be solved.

According to an aspect of the present invention, an image processing system includes: a camera, a positioning device, a posture estimating device, and a processor. The camera is used to capture a real environment. The positioning device is used to detect an imaging position of the camera. The attitude estimating device is used to detect an imaging posture of the camera. The processor is configured to infer a light source information according to the time information and the latitude information, and according to the image capturing position, the camera posture, the real environment information corresponding to the real environment, the light source information, the first virtual information of the first virtual object, and a light source A ray tracing algorithm is used to present a reflection of the real environment on the first virtual object.

According to another aspect of the present invention, an image processing method includes: capturing a real environment by a camera; detecting an image capturing position of the camera by a positioning device; and detecting an image capturing posture of the camera by using a posture estimating device Deriving a light source information according to a time information and a latitude information by a processor; and by the processor according to the image capturing position, the image capturing posture, the real environment information corresponding to the real environment, the light source information, and one of the first virtual objects A virtual information and a ray tracing algorithm to present a reflection of the real environment on the first virtual object.

In summary, the image processing system and image processing method of the present application apply the imaging position, the imaging posture, the real environment information, the light source information, and the virtual information of the virtual object and the light source tracking algorithm to consider the position of the sun at the world coordinates, the color temperature of the light source, and the camera. The position and its orientation, the material and/or reflectivity of the real object, the material of the virtual object and/or the reflectivity are combined with the light source tracking algorithm to make the reflection of the real environment appear on the virtual object, so that the amplification is real. Virtual objects in the environment can appear closer to the real light and shadow.

Please refer to FIG. 1A. FIG. 1A is a block diagram of an image processing system 100a according to an embodiment of the present invention. In one embodiment, the image processing system 100a includes a camera 10, a positioning device 20, an attitude estimating device 30, and a processor 40. In one embodiment, the processor 40 is coupled to the camera 10, the positioning device 20, and the attitude estimating device 30, respectively.

In an embodiment, the camera 10 may be a Charge Coupled Device (CCD) or a Complementary Metal-Oxide Semiconductor (CMOS). The positioning device 20 may be a Global Positioning System (GPS) locator, and the global positioning system locator may be used to obtain position information of the camera 10. The attitude estimating device 30 can be implemented by an Inertial Measurement Unit (IMU), which can detect the orientation of the camera 10 (for example, toward the north or south, elevation or depression). The processor 40 can be implemented as a micro control unit, a microprocessor, a digital signal processor, an application specific integrated circuit (ASIC), or a logic circuit.

Please refer to FIG. 1B. FIG. 1B is a block diagram of an image processing system 100b according to another embodiment of the present invention. Compared with the image processing system 100a of FIG. 1A, the image processing system 100b of FIG. 1B further includes a display 50, and the processor 40 is coupled to the display 50. Display 50 can be a display device of a handheld electronic device (eg, a cell phone, a tablet) or a display device in a head mounted device. In one embodiment, the camera 10, the positioning device 20, the attitude estimating device 30, the processor 40, and the display 50 can be integrated into a single device (eg, a handheld electronic device).

Referring to FIG. 2, FIG. 2 is a flowchart of an image processing method 200 according to an embodiment of the present invention. The flow of the image processing method 200 of the present invention will be described in detail below. The elements mentioned in the image processing method 200 can be implemented by the elements described in FIG. 1A or FIG. 1B.

In step 210, camera 10 captures a real environment.

In an embodiment, a real environment information corresponding to the real environment is obtained by a precise map, and the real environment information includes three-dimensional information, a reflectivity, a color or a material information of each real object in the real environment. For example, when the camera 10 captures an office scene, the processor 40 can learn the three-dimensional information, reflectivity, color, or material of the real objects such as tables, chairs, and windows in the office scene according to the precise map of the corresponding office scene. .

In one embodiment, the material information and reflectivity in the real environment information can be applied as a reference for ray tracing when rendering, to confirm the direction of the real environment and or the virtual object from which direction.

In step 220, the positioning device 20 detects an imaging position of the camera 10.

In one embodiment, the positioning device 20 is a global positioning system locator that can be used to obtain an imaging position of the camera 10 (eg, GPS information of the camera 10).

In step 230, the attitude estimating device 30 detects an imaging posture of the camera 10.

In one embodiment, the attitude estimating device 30 is an inertial measuring unit, and the inertial measuring unit can detect the orientation of the camera 10 (for example, toward the north or south side, the elevation angle or the depression angle), thereby obtaining the imaging posture of the camera 10. .

The order of the above steps 210-230 can be adjusted according to actual implementation.

In step 240, the processor 40 infers a light source information based on time information (such as the current time and date) and latitude information (such as the latitude of the camera 10). In an embodiment, the time information and the latitude information can be obtained by the positioning device 20 or can be obtained by the network.

In an embodiment, when the processor 40 knows the time information and the latitude information from the positioning device 20 or the network, the light source information can be obtained or obtained by looking up the table. The light source information includes a light source position (such as the position of the sun at the coordinates of the world) or a color temperature information (such as the color of the light source).

In an embodiment, the processor 40 can establish a table corresponding to the color temperature information of the weather, for example, recording the weather conditions of the sunny day, the sunny day, the cloudy morning, the cloudy night, and the corresponding time period, so that when the processor 40 When you get time information and latitude information, you can know the weather of the place and check the table to get the color temperature information.

In step 250, the processor 40 renders the reflection of the real environment on the virtual object according to the imaging position, the imaging posture, the real environment information, the light source information, and a virtual information of a virtual object and a ray tracing algorithm. on.

In an embodiment, the processor 40 can obtain the imaging position, the imaging posture, the real environment information, the light source information, and the virtual information of the virtual object by using the foregoing steps 210-240. In step 250, the information and the light source tracking calculation can be performed. The law considers together to present the reflection of the real environment on a virtual object. In an embodiment, since the precise map contains the color of the real object, the color-reflecting reflection can be presented on the virtual object, so that the virtual object in the augmented reality can present a closer to the real light and shadow.

The application of the light source tracking algorithm is described below.

Please refer to FIG. 3 , which is a schematic diagram of an application light source tracking algorithm according to an embodiment of the present invention. In an embodiment, the virtual information of the virtual object OBJ1 is pre-preset information, and the virtual information includes a virtual position of the virtual object OBJ1 and a reflectivity of the virtual object OBJ1.

In FIG. 3, processor 40 applies a light source tracking algorithm to derive the brightness and color that should be presented for each pixel (eg, positions P1, P2) of display 50a. In this example, the human eye position P0 can be replaced with the position of the camera 10, and the light source tracking algorithm is calculated by the human eye position P0, and is projected by each pixel position (for example, positions P1, P2) on the display 50a. A ray, and the reflection, refraction, and/or shadow effects of the ray hitting the real environment EN and the virtual object OBJ1. Each pixel on display 50a corresponds to a piece of light information based on the reflected, refracted path of the ray in space.

For example, the processor 40 simulates emitting a ray from the position P1, and after the ray hits the real environment EN (for example, the mirror 60), generates a reflected ray from the real environment EN to hit the virtual object OBJ1, and then generates another from the virtual object OBJ1. A reflected ray strikes the light source SC1, whereby the processor 40 can estimate the brightness and color temperature that should be displayed at the position P1.

For another example, the processor 40 simulates emitting a ray from the position P2, the ray hits the ground shadow SD2 of the virtual object OBJ1, and generates a reflected ray from the ground shadow SD2 to hit the virtual object OBJ1 and the light source SC2, whereby the processor 40 can estimate the brightness and color temperature that should be displayed at position P2.

In the space depicted in Figure 3, there are actually more reflections, refractions, diffuses or shadows of the light sources (for example, shadows SD1, SD2). For convenience of explanation, only the parts involved in the above examples are shown. Rays are an example.

Next, the processor 40 according to the imaging position, the imaging posture, the real environment information, the light source information, the virtual object virtual information, and the light source tracking algorithm to present the reflection of the real environment on the virtual object and the virtual object The reflection is presented on another virtual object.

Please refer to FIGS. 4-6. FIG. 4-6 is a schematic diagram of an application image processing method according to an embodiment of the present invention. It should be particularly noted that the direction of the arrow in Fig. 4 is the calculation direction of the light source tracking algorithm, which is opposite to the actual illumination direction of the light source SC'. In Fig. 4, the virtual object OBJa is a virtual smooth metal ball which is suspended above the bottom plate FL. In addition, the light source SC' is illuminated to the real environment EN' and the virtual object OBJa, and the real environment EN' The light is then reflected onto the virtual object OBJa such that the virtual object OBJa presents a reflection of the real environment EN', and then the light on the virtual object OBJa is reflected back to the camera 10. Based on the above content and applying the light source tracking algorithm, the processor 40 can be based on the image capturing position (for example, the position of the camera 10), the image capturing posture (for example, the orientation of the camera 10), the real environment information of the real environment EN', and the light source SC'. The light source information and the virtual information of the virtual object OBJa are used to know the influence of the light source SC' and the real environment EN' on the virtual object OBJa to present the reflection of the real environment EN' on the virtual object OBJa.

In Fig. 5, it is similar to the principle of Fig. 4. In this example, the virtual object OBJa is a virtual smooth metal ball placed on the bottom plate FL, and the light source SC' is polished to the wall surface WL1. ~WL4 (ie, the real environment) and the virtual object OBJa, and the walls WL1 WL WL4 also reflect the light onto the virtual object OBJa, and then the light on the virtual object OBJa is reflected to the camera 10. Based on the above content and applying the light source tracking algorithm, the processor 40 can depend on the imaging position (for example, the position of the camera 10), the imaging posture (for example, the orientation of the camera 10), and the real environment (for example, the wall surface WL1 to WL4). The information, the light source information of the light source SC' and the virtual information of the virtual object OBJa, to know the influence of the light source SC' and the wall surface WL1~WL4 on the virtual object OBJa, thereby presenting the reflection of the wall surface WL1~WL4 The virtual object OBJa is on, and presents a virtual shadow SDa of the virtual object OBJa.

In an embodiment, after the reflection of the real environment (for example, the walls WL1 WL WL4) is presented on the virtual object (for example, the virtual object OBJa), the virtual object is referred to as a rendered object, and the display in FIG. 1B 50 is used to simultaneously present the real environment and the rendered object.

In Fig. 6, it is similar to the principle of Fig. 5. In this example, the virtual objects OBJa and OBJb are virtual smooth metal balls placed on the bottom plate FL, and the light source SC' is polished to the wall surface. WL1~WL4 (real environment) and virtual objects OBJa, OBJb, and wall WL1~WL4 also reflect light onto virtual objects OBJa, OBJb. In an embodiment, the light reflected to one of the virtual objects OBJa and OBJb may be reflected to the other, so that the reflection of the virtual object OBJb is presented on the virtual object OBJa or the reflection of the virtual object OBJa is presented. Virtual object OBJb.

In other words, for the virtual object OBJa, the processor 40 can depend on the actual position information of the imaging position (for example, the position of the camera 10), the imaging posture (for example, the orientation of the camera 10), and the real environment (for example, the wall surface WL1 WL WL4). The light source information of the light source SC' and the virtual information of the virtual objects OBJa, OBJb and the light source tracking algorithm to know the influence of the light source SC', the wall surfaces WL1 WL WL4 and the virtual object OBJb on the virtual object OBJa, The reflection of the wall surfaces WL1 WL WL4 and the virtual object OBJb is presented on the virtual object OBJa, and the virtual shadow SDa of the virtual object OBJa is presented.

On the other hand, for the virtual object OBJb, the processor 40 can depend on the imaging position (for example, the placement position of the camera 10), the imaging posture (for example, the orientation of the camera 10), and the real environment (for example, the wall surface WL1 to WL4). Environmental information, source information of the light source SC', and virtual information of the virtual objects OBJa, OBJb and the light source tracking algorithm to know the influence of the light source SC', the wall surface WL1~WL4 and the virtual object OBJa on the virtual object OBJb , the reflection of the wall surfaces WL1 WL WL4 and the virtual object OBJa is presented on the virtual object OBJb, and the virtual shadow SDb of the virtual object OBJb is presented.

In one embodiment, when a reflection of a real environment (eg, walls WL1 WL WL4) is presented on a virtual object (eg, virtual object OBJa), the virtual object is referred to as a rendered object. When the reflection of this virtual object is presented on another virtual object (eg, virtual object OBJb), this other virtual object is also a rendered object. The display 50 in Figure 1B can be used to simultaneously present the real environment and the rendered objects. Thereby, a plurality of virtual objects in the augmented reality can be rendered closer to the real light and shadow appearance.

In summary, the image processing system and image processing method of the present application apply the imaging position, the imaging posture, the real environment information, the light source information, and the virtual information of the virtual object and the light source tracking algorithm to consider the position of the sun at the world coordinates, the color temperature of the light source, and the camera. The position and its orientation, the material and/or reflectivity of the real object, the material of the virtual object and/or the reflectivity are combined with the light source tracking algorithm to make the reflection of the real environment appear on the virtual object, so that the amplification is real. Virtual objects in the environment can appear closer to the real light and shadow.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present case. Anyone skilled in the art can make various changes and refinements without departing from the spirit and scope of the present case. The scope defined in the patent application is subject to change.

100a, 100b‧‧‧ image processing system

10‧‧‧ camera

20‧‧‧ Positioning device

30‧‧‧ attitude estimation device

40‧‧‧ processor

200‧‧‧Image processing method

210~250‧‧‧Steps

OBJ1, OBJa, OBJb‧‧‧ virtual objects

P0‧‧‧ human eye position

P1, P2‧‧‧ position

50, 50a‧‧‧ display

60‧‧‧Mirror

EN, EN’‧‧‧Real environment

SC1, SC2, SC'‧‧‧ light source

SD1, SD2, SDa, SDb‧‧‧ shadow

FL‧‧‧Bottom plate

WL1~WL4‧‧‧ wall

The above and other objects, features, advantages and embodiments of the present disclosure will become more apparent and understood by the accompanying drawings. FIG. 1A is a block diagram of an image processing system according to an embodiment of the present invention. FIG. 1B is a block diagram of an image processing system according to another embodiment of the present invention; FIG. 2 is a flowchart of an image processing method according to an embodiment of the present invention; FIG. 3 is a flowchart according to an embodiment of the present invention; FIG. 4 is a schematic diagram of an application image processing method according to an embodiment of the present invention; FIG. 5 is a schematic diagram of an application image processing method according to an embodiment of the present invention; FIG. 6 is a schematic diagram of an application image processing method according to an embodiment of the present invention.

Claims (14)

An image processing system comprising: a camera for capturing a real environment; a positioning device for detecting an imaging position of the camera; and a posture estimating device for detecting an imaging posture of the camera; a processor for inferring a light source information according to a time information and a latitude information, and according to the image capturing position, the camera posture, a real environment information corresponding to the real environment, the light source information, and a first virtual object A first virtual information and a ray tracing algorithm to present the reflection of the real environment on the first virtual object. The image processing system of claim 1, wherein the real environment information is obtained by a precise map containing three-dimensional information, a reflectivity, a color or a material of each real object in the real environment. News. The image processing system of claim 1, wherein the light source information comprises a light source position or a color temperature information. The image processing system of claim 1, wherein the first virtual information comprises a virtual location of the first virtual object and a reflectivity of the first virtual object. The image processing system of claim 1, wherein the first virtual object becomes a first rendered object after the reflection of the real environment is presented on the first virtual object, the image processing system further comprising: a display, To simultaneously display the real environment and the first rendered object. The image processing system of claim 1, wherein the processor is further configured to: according to the image capturing position, the image capturing posture, the light source information, the first virtual information of the first virtual object, and a second virtual object The second virtual information and the light source tracking algorithm are configured to present the reflection of the first virtual object on the second virtual object, wherein the second virtual information includes a virtual location of the second virtual object and the second virtual The reflectivity of one of the objects. The image processing system of claim 6, wherein the first virtual object becomes a first rendered object after the reflection of the real environment is presented on the first virtual object, and the reflection of the first virtual object is presented in the first After the second virtual object becomes the second rendered object, the image processing system further includes: a display for simultaneously displaying the real environment, the first rendered object, and the second rendered object. An image processing method includes: capturing a real environment by a camera; detecting an image capturing position of the camera by a positioning device; detecting an image capturing posture of the camera by using a posture estimating device; The device infers a light source information according to the time information and the latitude information; and the processor according to the image capturing position, the camera posture, a real environment information corresponding to the real environment, the light source information, and one of the first virtual objects The first virtual information and a ray tracing algorithm are to present the reflection of the real environment on the first virtual object. The image processing method of claim 8, wherein the real environment information is obtained by a precise map, wherein the real environment information includes one of three-dimensional information, a reflectivity, a color or a material of each real object in the real environment. News. The image processing method of claim 8, wherein the light source information comprises a light source position or a color temperature information. The image processing method of claim 8, wherein the first virtual information comprises a virtual location of the first virtual object and a reflectivity of the first virtual object. The image processing method of claim 8, wherein the first virtual object becomes a first rendered object after the reflection of the real environment information is presented on the first virtual object, and the image processing method further comprises: The display simultaneously displays the real environment and the first rendered object. The image processing method of claim 8, further comprising: the processor according to the image capturing position, the image capturing posture, the light source information, the first virtual information of the first virtual object, and a second virtual object a second virtual information and the light source tracking algorithm to present a reflection of the first virtual object on the second virtual object, wherein the second virtual information includes a virtual location of the second virtual object and the second The reflectivity of one of the virtual objects. The image processing method of claim 13, wherein after the reflection of the real environment is presented on the first virtual object, the first virtual object becomes a first rendered object, and the reflection of the first virtual object is presented in the first After the second virtual object, the second virtual object becomes a second rendered object, and the image processing method further comprises: simultaneously displaying the real environment, the first rendered object and the second rendered object by using a display.