TWI843315B - Method of panoramic image fusion in augmented reality and computer program product implementing the same - Google Patents

Abstract

A method of panoramic image fusion in augmented reality and a computer program product implementing the same are disclosed. The method includes: locating a location of a mobile device on a pre-built spatial map according to a real-time image captured by the mobile device to search a panoramic image corresponding to the location of the mobile device from pre-acquired a plurality of special map based-panoramic images, further, retrieving an overlay image from the panoramic image, identifying a plane, and determining an overlay area on the plane of the overlay image according to the location of the mobile device, a special location of the panoramic image, and azimuth information and/or viewing angle information of the mobile device, thereby overlaying the overlay image on the plane according to the overlay area.

Description

Method for integrating ambient images in augmented reality and computer program product for executing the method

This case is about an augmented reality technology, more specifically, a method and a computer program product for integrating ambient images into augmented reality.

Augmented Reality (AR) can present information, images, objects, audio and video content in a virtual way in the real environment through electronic products such as mobile devices, head-mounted displays or head-up displays, achieving the effect of integrating virtual and real and presenting them simultaneously.

As the computing power of mobile electronic devices improves, the use of augmented reality is becoming more and more extensive. In addition to being used in games, augmented reality is also used in various exhibitions, route guidance or online shopping guides. For example, virtual route guidance arrows are displayed in the real environment of the exhibition hall on the screen of electronic products to guide or recommend users to visit routes.

However, the prior art was unable to integrate the image that was originally invisible behind the plane onto the plane, and dynamically change it in accordance with the position of the user's mobile device and the change of the viewing angle of the camera on the mobile device, so as to achieve a better perspective effect of the corresponding image that the user can see at different positions and angles.

Therefore, how to overcome the above shortcomings and/or develop more different augmented reality application scenarios is an issue that the industry urgently needs to study.

To solve the above problems and other problems, this case discloses a method for integrating ambient images in augmented reality and a computer program product for executing the method.

The method for integrating ambient images in augmented reality disclosed in this case includes the following steps: locating the location of a mobile device in a pre-established spatial map based on a real-time image obtained by the mobile device, and searching for an ambient image corresponding to the location of the mobile device from a plurality of ambient images pre-acquired based on the spatial map according to the location of the mobile device; The spatial position of the surrounding image, the azimuth information and/or the viewing angle information of the mobile device are used to extract a set of overlapping images from the surrounding image; and a plane is identified according to the location of the mobile device, the spatial position of the surrounding image, the azimuth information and/or the viewing angle information of the mobile device to determine the overlapping range of the overlapping images on the plane, so as to overlap the overlapping images on the plane according to the overlapping range.

In one embodiment, the step of extracting a set of overlapping images from the surrounding image according to the location of the mobile device, the spatial location of the surrounding image, the azimuth information and/or the viewing angle information of the mobile device includes: determining the size of the overlapping images according to the distance between the location of the mobile device and the spatial location of the surrounding image, the distance between the spatial location of the surrounding image and a given point in the surrounding image, and the distance between the location of the mobile device and the given point in the surrounding image.

In one embodiment, the step of extracting a set of overlapping images from the surrounding image according to the location of the mobile device, the spatial location of the surrounding image, the azimuth information and/or the viewing angle information of the mobile device includes: confirming the horizontal range and the vertical range of the image on the surrounding image according to the spatial location of the surrounding image, the spatial location of the surrounding image, the horizontal viewing angle range and the vertical viewing angle range of the mobile device, so as to extract the overlapping images from the surrounding image according to the horizontal range and the vertical range of the image on the surrounding image.

In one embodiment, the step of identifying a plane according to the location of the mobile device and the spatial location of the surrounding image to determine the overlapping range of the overlapping image on the plane includes: connecting the location of the mobile device and the spatial location of the surrounding image in a straight line, and identifying the plane between the location of the mobile device and the spatial location of the surrounding image along the direction of the straight line; and connecting the location of the mobile device with the end points of the horizontal range of the image and the end points of the vertical range of the image on the surrounding image to confirm the horizontal range of the overlapping and the vertical range of the overlapping on the plane.

In one embodiment, the step of overlaying the overlay image on the plane according to the overlay range includes: filtering the portion of the plane corresponding to the overlay range to overlay the overlay image on the portion of the plane. In another embodiment, the step of overlaying the overlay image on the plane according to the overlay range includes: determining the transparency of the portion of the plane corresponding to the overlay range according to the distance between the location of the mobile device and the plane, wherein the transparency is inversely proportional to the distance. In another embodiment, the step of overlaying the overlay image on the plane according to the overlay range includes: processing the overlay image to be transparent, so as to overlay the transparent overlay image on the plane.

In one embodiment, the real-time image of the mobile device includes the azimuth information and/or the viewing angle information of the mobile device. In another embodiment, the plurality of panoramic images have the spatial coordinate information, azimuth information, and/or depth information of the respective spatial maps.

The computer program product disclosed in this case executes the above-mentioned method of integrating ambient images in augmented reality after being loaded into a computer.

According to the method of integrating surrounding images in augmented reality and the computer program product implementing the method disclosed in this case, images that were originally invisible behind a plane can be integrated onto the plane, and can change dynamically with the position of the user's mobile device and the change of the viewing angle of the camera on the mobile device, so that the user can better see the perspective effect of the corresponding image at different positions and angles.

S201~S208: Steps

P1: Location of the mobile device

P2: Position of the plane

P2-1: The position of a point on a plane

P2-2: The position of another point on the plane

P3: Spatial location of the panoramic image

P4: Intersection point

P5: A given point in the panoramic image

E1, E2: Horizontal range endpoints on the panoramic image

E3, E4: vertical range endpoints on the panoramic image

R1: Image horizontal range

R2: Image vertical range

d1: Depth of field

d2: distance

d3: Depth of field

d4: distance

d5: distance

θ 1: Direction angle

Figure 1 is a schematic diagram of the process of integrating ambient images in augmented reality in this case.

FIG2 is a schematic diagram of extracting a nested image from a surrounding image in the method of fusing surrounding images in augmented reality in this case.

FIG3 is a schematic diagram of extracting a nested image from a surrounding image in the method of fusing surrounding images in augmented reality in this case.

FIG4 is a schematic diagram of determining the size of the overlapping images in the method of fusing ambient images in augmented reality in this case.

FIG5 is a schematic diagram of determining the size of the overlapping images in the method of fusing ambient images in augmented reality in this case.

FIG6 is a schematic diagram showing the transparency of the portion of the corresponding overlapping range of the plane in the method of fusing ambient images in augmented reality in this case.

The following specific examples are used to illustrate the implementation of this case. People familiar with this technology can easily understand the other advantages and effects of this case from the content disclosed in this article. The structures, ratios, sizes, etc. shown in the attached figures of this manual are only used to match the content disclosed in the manual for people familiar with this technology to understand and read, and are not used to limit the conditions under which this case can be implemented. Therefore, any modification, change or adjustment should still fall within the scope of the technical content disclosed in this case without affecting the effects and purposes that can be achieved by this case.

The terms "include", "comprising", "having", "containing" or any other variations thereof used herein are intended to cover a non-exclusive inclusion. For example, a method consisting of a series of steps is not necessarily limited to these steps, but may also include other steps not explicitly listed. In addition, unless otherwise specified, singular forms such as "a", "an", "the" also apply to plural forms, and "or", "and/or" and other forms can be used interchangeably.

Please refer to Figure 1, which is a schematic diagram of the process of the method of fusing ambient images in augmented reality, which mainly includes steps S201~S208.

In step S201, a spatial map is established. In one embodiment, a map coordinate system of a space is pre-established by capturing multiple feature points of the space. Then proceed to step S202.

In step S202, multiple panoramic images based on the spatial map are obtained. For example, a panoramic image is taken at a spatial position in the coordinate system of the spatial map, and the process is repeated to obtain more panoramic images based on the spatial map, and each panoramic image carries the reference point coordinates of the spatial map for subsequent search and positioning. In addition, each panoramic image has azimuth information and/or depth information, or information related to azimuth and/or depth can be derived. Then proceed to step S203.

In step S203, the location of the mobile device is located in the spatial map according to the real-time image obtained by the mobile device. In detail, the real-time image has the azimuth information and/or viewing angle information taken by the camera on the mobile device, and the spatial coordinates of the mobile device in the spatial map can be obtained during the positioning process. Then proceed to step S204.

In step S204, according to the location of the mobile device, a surrounding image corresponding to the location of the mobile device is searched in the plurality of surrounding images, wherein the spatial position of the surrounding image corresponding to the location of the mobile device must be within the viewing angle of the camera on the mobile device. In addition, if the location of the mobile device (i.e., the spatial coordinates of the mobile device) is too far from the spatial position of the surrounding image (i.e., the reference point coordinates of the surrounding image) (i.e., exceeding a preset value), the surrounding image will not be displayed. Then proceed to step S205.

In step S205, a nested image is captured from the surrounding image according to the location of the mobile device, the spatial location of the surrounding image, the azimuth information and/or the viewing angle information of the mobile device. In one embodiment, the horizontal range and the vertical range of the image on the surrounding image are first confirmed according to the spatial location of the surrounding image, the spatial location of the surrounding image, the horizontal viewing angle range and the vertical viewing angle range of the mobile device, and then the nested image is captured from the surrounding image according to the horizontal range and the vertical range of the image on the surrounding image. In another embodiment, the size of the stacked image is determined based on the distance between the location of the mobile device and the spatial location of the ambient image, the distance between the spatial location of the ambient image and a given point in the ambient image, and the distance between the location of the mobile device and the given point in the ambient image, so as to achieve the effect of making distant objects small and near objects large. Then proceed to step S206.

In step S206, a plane is identified according to the location of the mobile device and the spatial location of the ambient image. In one embodiment, a straight line is formed between the location of the mobile device and the spatial location of the ambient image, and the plane is identified between the location of the mobile device and the spatial location of the ambient image along the direction of the straight line. Then proceed to step S207.

In step S207, the overlapping range of the overlapping images on the plane is determined according to the location of the mobile device, the spatial location of the surrounding image, the azimuth information and/or the viewing angle information of the mobile device. In one embodiment, the location of the mobile device is connected with the end points of the horizontal range of the image and the end points of the vertical range of the image on the surrounding image to confirm the horizontal range and vertical range of the overlapping on the plane. In another embodiment, the overlapping range of the overlapping image on the plane can be circular, or any shape including the above-mentioned horizontal range and vertical range. Then proceed to step S208.

In step S208, the stacked image is stacked on the plane according to the stacking range. In one embodiment, the portion of the plane corresponding to the stacking range is filtered so that the stacked image is presented on the plane in an opaque manner. In another embodiment, the transparency of the portion of the plane corresponding to the stacking range is determined according to the distance between the location of the mobile device and the plane, wherein the transparency is inversely proportional to the distance. In yet another embodiment, the stacked image is processed to be transparent so that the transparent stacked image is stacked on the plane.

From the implementation example of the method of integrating ambient images in augmented reality shown in FIG1, it can be seen that the present invention can increase the perspective effect in the augmented reality environment, and because it adopts the ambient image of the actual position, the perspective range can be adjusted according to the user's position and viewing angle to achieve a realistic effect, and the blank plane space in the augmented reality can be effectively utilized, which is better used as an advertisement.

In one embodiment, the method of the present invention can be executed in a single or multiple devices such as a server, a computer or other devices with data processing, computing, storage, network communication and other functions, wherein the server, computer or device includes a central processing unit, a hard disk, a memory, etc.

In addition, the computer program product in this case is executed by loading the program into the computer. In addition, in addition to being stored in a recording medium, the computer program (product) can also be directly transmitted and provided on the Internet. The computer program (product) is a thing that contains a program that can be read by a computer and is not limited to an external form.

In addition, this case also provides a computer-readable recording medium, which is applied to a computing device or computer having a processor and/or memory, and the computer-readable recording medium stores instructions, and the computing device or computer can be used to execute the computer-readable recording medium through the processor and/or memory to execute the above method and/or content when executing the computer-readable recording medium. The computer-readable recording medium (such as a hard disk, a floppy disk, an optical disk, a USB flash drive) stores the computer program (product).

In one embodiment, steps S201 to S208 may be executed on a cloud computer, wherein the cloud computer includes a computer processor and a computer program (product) (or a non-transitory computer-readable storage medium), and the computer program (product) is executed by the computer processor to perform the above steps. In addition, in step S203, the client mobile device can use an application (such as an APP) loaded in an electronic product (such as a mobile device, a head-mounted display or a head-up display, etc.) to transmit the real-time image it has taken to the cloud computer, so that the cloud computer can locate the client mobile device. In step S206 or before, the cloud computer can transmit the spatial position of the surrounding image to the client mobile device, so that the client device can identify the plane.

Next, please refer to Figures 2 to 6, which illustrate a specific embodiment of the method for fusing ambient images in augmented reality of the present case, wherein Figures 2 and 3 are schematic diagrams showing the method for fusing ambient images in augmented reality of the present case of extracting an overlay image from an ambient image, Figure 4 is a schematic diagram showing the method for fusing ambient images in augmented reality of the present case of determining the size of an overlay image, Figure 5 is a schematic diagram showing the method for fusing ambient images in augmented reality of the present case of determining the size of an overlay image, and Figure 6 is a schematic diagram showing the method for fusing ambient images in augmented reality of the present case of determining the transparency of a portion of a plane corresponding to an overlay range.

As shown in Figures 2 and 3, the location P1 of the connected mobile device and the spatial location P3 of the panoramic image form a straight line, and then the straight line is extended to intersect the panoramic image at the intersection point P4. From the intersection point P4, along the horizontal plane, a total of about 120 degrees of image horizontal range R1 is taken clockwise and counterclockwise (i.e., the horizontal viewing angle range that conforms to the human binocular viewing angle can be taken), and from the intersection point P4 along the vertical plane, a total of about 55 degrees of image vertical range R2 is taken upward and downward (i.e., the vertical viewing angle range that conforms to the human binocular viewing angle can be taken). In this way, the horizontal range endpoints E1 and E2 on the panoramic image, and the vertical range endpoints E3 and E4 on the panoramic image are obtained.

In addition, as shown in FIG. 2 and FIG. 3, the position P1 of the mobile device is connected to the spatial position P3 of the surround image in a straight line. Through a camera lens that can provide depth of field and software that can identify planes in space (such as ARKit/ARCore), a plane can be identified along the straight line, and the position of the plane is P2. Connect P1 with the horizontal range endpoints E1 and E2 on the surround image, and connect P1 with the vertical range endpoints E3 and E4 on the surround image. These lines generate four intersections on the plane to delineate an overlapping range on the plane. The overlapping plane range can be any plane shape as long as it can contain the four point intersections on the plane. In one embodiment, the calculation of the overlapping range can be performed by the cloud computer or the client device, for example, the calculation of the overlapping range is performed on the cloud computer.

As shown in FIG. 4 , the distance from the location P1 of the mobile device to the location P2 of the plane is the depth of field d1, which can be obtained from the spatial recognition software (such as ARKit/ArCore), and the distance from the location P1 of the mobile device to the spatial location P3 of the surrounding image is d2, which can be calculated because both locations use the same positioning coordinate system. In one embodiment, the calculation in FIG. 4 can be performed by a cloud computer or a client device, for example, on a cloud computer.

In addition, the distance from the spatial position P3 of the ambient image to a given point P5 in the ambient image is the depth of field d3, which can be known from the ambient image, and the straight line from the location P1 of the mobile device to the spatial position P3 of the ambient image intersects at the ambient image as the intersection point P4. There is a distance d4 between the spatial position P3 of the ambient image and the intersection point P4, and there is a direction angle θ1 between the intersection point P4 and a given point P5 in the ambient image. Then, according to the trigonometric function calculation, as shown in Figure 5, the distance d5 from the location P1 of the mobile device to a given point P5 in the ambient image can be deduced as follows:

By using the distance d5 from the location P1 of the mobile device to a given point P5 in the surrounding image, the overlapping image to be overlapped on the plane can be appropriately scaled to achieve the visual effect of small distant objects and large near objects, avoiding the problem that the size of the surrounding image does not change regardless of the angle from which it is viewed, thereby increasing the realism of the perspective effect. In one embodiment, the size scaling of the overlapping image can be executed by the cloud computer or the client device, for example, the size scaling of the overlapping image is executed on the cloud computer.

In addition, there are many options for perspective effects. You can choose to use transparency to overlay the stacked images on a plane, or filter out part of the real plane image on the plane to directly show the stacked image behind the plane, like a TV wall effect. You can also more accurately use the depth of field information of the camera lens to identify different planes as a parameter of transparency. As shown in Figure 6, the stacked image of the camera lens closer to the mobile device has a higher transparency, while the stacked image farther away has a lower transparency, guiding the user to walk closer to the plane because he wants to see the fully transparent augmented reality effect, so that the depth of field from the camera lens of the mobile device to the plane is consistent, and because the closer to the plane, the stacked surrounding image will be enlarged as the user approaches, and it will be clearer. The transparency of the overlay image can be set using the following formula.

Transparency = smaller depth of field/larger depth of field = d1-1/d1-2.

For example, the distance between the location P1 of the mobile device and the location P2-1 of a point on the plane is d1-1, and the distance between the location P1 of the mobile device and the location P2-2 of another point on the plane is d1-2. If d1-1 is 1 meter and d1-2 is 2 meters, then the transparency of the image at d1-1 is 100%, while the transparency of the image at d1-2 is only 50%. In one embodiment, the calculation in FIG. 6 can be performed by a cloud computer or a client device, for example, on a cloud computer.

In summary, the method of integrating ambient images in augmented reality and the computer program product for executing the method can be applied to navigation and shopping or merchant advertising, so that during the guidance process, the content of augmented reality (AR) guidance can be increased by the see-through wall effect, so as to increase its convenience and practicality, and increase the effect that previous technologies cannot achieve. For example, when guiding indoors, the see-through product effect can be used to attract users to enter the merchant. For another example, in a department store space with many merchants, a transparent display cabinet effect of mixed reality (MR) can be provided for the store.

The above embodiments are only illustrative of the effects of this case, and are not intended to limit this case. Anyone familiar with this technology can modify and change the above embodiments without violating the spirit and scope of this case. Therefore, the scope of protection of the rights of this case should be as listed in the scope of the patent application described below.

S201~S208: Steps

Claims (10)

A method for integrating surrounding images in augmented reality comprises the following steps: establishing a spatial map to obtain a plurality of surrounding images based on the spatial map; locating a location of a mobile device in the spatial map according to a real-time image obtained by the mobile device, and searching for a surrounding image corresponding to the location of the mobile device from the plurality of surrounding images according to the location of the mobile device; and searching for a surrounding image corresponding to the location of the mobile device according to the location of the mobile device. At the location, the spatial location of the surrounding image, the azimuth information and/or the viewing angle information of the mobile device, a set of overlapping images is extracted from the surrounding image; and according to the location of the mobile device, the spatial location of the surrounding image, the azimuth information and/or the viewing angle information of the mobile device, a plane is identified to determine the overlapping range of the overlapping images on the plane, so as to overlap the overlapping images on the plane according to the overlapping range. As described in claim 1, the step of extracting a set of overlapping images from the surrounding image according to the location of the mobile device, the spatial location of the surrounding image, the azimuth information and/or the viewing angle information of the mobile device includes: determining the size of the overlapping images according to the distance between the location of the mobile device and the spatial location of the surrounding image, the distance between the spatial location of the surrounding image and a given point in the surrounding image, and the distance between the location of the mobile device and the given point in the surrounding image. The method as described in claim 1, wherein the step of extracting a set of overlapping images from the surrounding image according to the location of the mobile device, the spatial location of the surrounding image, the azimuth information and/or the viewing angle information of the mobile device comprises: confirming the horizontal range and the vertical range of the image on the surrounding image according to the spatial location of the surrounding image, the spatial location of the surrounding image, the horizontal viewing angle range and the vertical viewing angle range of the mobile device, to extract the overlapping images from the surrounding image according to the horizontal range and the vertical range of the image on the surrounding image. The method as described in claim 3, wherein the step of identifying a plane according to the location of the mobile device and the spatial location of the ambient image to determine the overlapping range of the overlapping image on the plane includes: connecting the location of the mobile device and the spatial location of the ambient image in a straight line, and identifying the plane between the location of the mobile device and the spatial location of the ambient image along the direction of the straight line; and connecting the location of the mobile device with the end points of the horizontal range of the image and the end points of the vertical range of the image on the ambient image to confirm the horizontal range of the overlapping and the vertical range of the overlapping on the plane. The method as described in claim 1, wherein the step of overlaying the overlay image on the plane according to the overlay range includes: filtering the portion of the plane corresponding to the overlay range to overlay the overlay image on the portion of the plane. The method as described in claim 1, wherein the step of overlaying the overlay image on the plane according to the overlay range includes: determining the transparency of the portion of the plane corresponding to the overlay range according to the distance between the location of the mobile device and the plane, wherein the transparency is inversely proportional to the distance. The method as described in claim 1, wherein the step of overlaying the overlay image on the plane according to the overlay range includes: processing the overlay image to be transparent, so as to overlay the transparent overlay image on the plane. The method as described in claim 1, wherein the real-time image of the mobile device includes azimuth information and/or viewing angle information of the mobile device. The method as described in claim 1, wherein the plurality of ambient images have spatial coordinate information, azimuth information, and/or depth information of the respective spatial maps. A computer program product, after being downloaded to a computer, executes the method of fusing ambient images in augmented reality as described in any one of claim items 1-9.

Images