TWI885862B - See-through display method and see-through display system - Google Patents

Abstract

A see-through display method and a see-through display system are disclosed. The method includes the following steps. A user image is captured toward the front side of a display through a first image sensor, and a scene image is captured toward the rear side of the display through a second image sensor. User location information associated with the three-dimensional reference coordinate system is obtained according to the user image. Scene position information associated with the three-dimensional reference coordinate system is obtained according to the scene image. A viewing frustum is determined based on the user position information and an actual size of the display. A projection matrix of the viewing frustum is used to generate a display frame projected on the display plane of the display according to the scene position information. The display frame is output through the display to show the scene behind the monitor.

Description

Transparent display method and transparent display system

The present invention relates to an image display technology, and in particular to a transmissive display method and a transmissive display system.

With the advancement of technology, augmented reality (AR) applications have become more and more popular. This technology has not only made breakthroughs in the field of entertainment, but has also been widely used in business, education, medicine and other fields. With the continuous maturity and popularization of AR technology, people can integrate virtual elements into the real world through AR glasses, smart phones, various handheld electronic devices or various wearable electronic devices, providing users with rich interactive experience. In general, AR technology will continue to change the lifestyle of modern people and bring more convenience and rich experience to modern people.

Generally speaking, the camera installed on the back of a handheld electronic device can shoot real scenes, and the AR screen displayed by the handheld electronic device includes real scene images and virtual elements superimposed on the real scene images. Traditionally, due to the mobility of handheld electronic devices and the unclear changes in relative position between the user and the user, it is unnecessary to determine the AR screen based on user tracking results. However, when attempting to apply AR technology to a large display at a fixed position, if the relative position relationship between the user and the display is not considered, the scene content in the AR screen will not meet the user's needs. For example, the user may not be able to view the scene objects of interest through the AR screen of the display.

The present invention provides a transmissive display method and a transmissive display system that can effectively solve the above problems.

An exemplary embodiment of the present invention provides a transmissive display method, which is applicable to a transmissive display system including a first image sensor, a second image sensor and a display. The transparent display method includes: capturing a user image toward the front side of the display through a first image sensor, and capturing a scene image toward the rear side of the display through a second image sensor; obtaining user position information related to a stereo reference coordinate system based on the user image; obtaining scene position information related to the stereo reference coordinate system based on the scene image; determining a viewing frustum based on the user position information and the actual size of the display; using a projection matrix of the viewing frustum to generate a display frame projected on a display plane of the display according to the scene position information; and outputting the display frame through the display to display the scene behind the display.

Another exemplary embodiment of the present invention provides a transmissive display system, the transmissive display system comprising: a first image sensor, a second image sensor, a display, and at least one processor. The processor is coupled to the first image sensor, the second image sensor and the display, and is used to: capture a user image toward the front side of the display through the first image sensor, and capture a scene image toward the rear side of the display through the second image sensor; obtain user position information related to a stereo reference coordinate system based on the user image; obtain scene position information related to a stereo reference coordinate system based on the scene image; determine a viewing frustum based on the user position information and the actual size of the display; generate a display frame projected on a display plane of the display based on the scene position information using a projection matrix of the viewing frustum; and output the display frame through the display to display the scene behind the display.

Based on the above, in the embodiment of the present invention, the user position information and the scene position information in the same stereo reference coordinate system can be obtained based on the user image and the scene image. The viewing cone used to determine the display content of the display frame can be based on the user position information and the actual size of the display. Therefore, the display scene content of the display frame output by the display can change in response to the user's movement and achieve good alignment with the actual scene around the display.

10,70: Transparent display system

110: First image sensor

120: Second image sensor

130: Display

140: Storage device

150: Processor

160: Depth sensor

F1: Display frame

Obj1: Scene object

U1: User

111,112,113:FOV

S1: Display plane

DP1~DP4: Vertex

VP1: User coordinates

51: Visual cone

S210~S260,S310~S330,S810~S870: Steps

Figures 1A to 1C are schematic diagrams of a transmissive display system according to an embodiment of the present invention.

Figure 2 is a flow chart of a penetration display method according to an embodiment of the present invention.

Figure 3 is a flow chart of obtaining scene location information according to an embodiment of the present invention.

Figure 4 is a schematic diagram of a three-dimensional reference coordinate system according to an embodiment of the present invention.

FIG5 is a schematic diagram of determining a viewing cone based on user location information according to an embodiment of the present invention.

Figures 6A and 6B are schematic diagrams of display scenes according to an embodiment of the present invention.

FIG7 is a schematic diagram of a transparent display system according to an embodiment of the present invention.

Figure 8 is a flow chart of a penetration display method according to an embodiment of the present invention.

Some exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. When the same element symbols appear in different drawings, they will be regarded as the same or similar elements. These exemplary embodiments are only part of the present disclosure and do not disclose all possible implementations of the present disclosure. More specifically, these exemplary embodiments are only examples of methods and systems within the scope of the patent application of the present disclosure.

Figures 1A to 1C are schematic diagrams of a transmissive display system according to an embodiment of the present invention.

Referring to FIG. 1A , in some embodiments, the transparent display system 10 can be implemented in, for example, the following electronic devices with image processing capabilities and data computing capabilities: laptops, tablet computers, personal computers, servers, game consoles, portable electronic devices, desktop computers, or other electronic devices. The transparent display system 10 includes a first image sensor 110, a second image sensor 120, a display 130, a storage device 140, and at least one processor 150.

The processor 150 is responsible for all or part of the operation of the transparent display system 10. For example, the processor 150 may include a central processing unit (CPU), a graphic processing unit (GPU) or other programmable general or dedicated microprocessor, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC), a programmable logic device (PLD) or other similar devices or a combination of these devices. The number of processors 150 may be one or more, and the present invention is not limited to this.

The storage device 140 is connected to the processor 150 and is used to temporarily or permanently store data, such as images, instructions, program codes, software modules, and the like. Specifically, the storage device 140 may include a volatile storage circuit. The volatile storage circuit is used to store data in a volatile manner. For example, the volatile storage circuit may include a random access memory (RAM) or a similar volatile storage medium. Alternatively, the storage device 140 may include a non-volatile storage circuit. The non-volatile storage circuit is used to store data in a non-volatile manner. For example, the non-volatile storage circuit may include a read only memory (ROM), a solid state drive (SSD) and/or a traditional hard disk drive (HDD) or similar non-volatile storage media. The number of storage devices 140 may be one or more, and the present invention is not limited thereto.

The display 130 is, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or other types of displays, and the present invention is not limited thereto. In some embodiments, the display 130 may be a stereoscopic display that provides different images to the left eye or the right eye of the user to present a stereoscopic visual effect, but the present invention is not limited thereto. For example, the display 130 may be a naked-eye 3D display or a glasses-type 3D display.

The first image sensor 110 is used to capture images and includes a camera lens having a lens and a photosensitive element. The first image sensor 110 can be implemented as a camera module composed of the lens, the photosensitive element and other components. The photosensitive element is, for example, a charge coupled device (CCD), a complementary metal-oxide semiconductor (CMOS) element or other elements, and the present invention is not limited thereto. From another point of view, the first image sensor 110 can be an RGB image sensor.

The second image sensor 120 is used to capture images and includes a camera lens having a lens and a photosensitive element. The second image sensor 120 can be implemented as a camera module composed of a lens, a photosensitive element and other components. The photosensitive element is, for example, a charge coupled device, a complementary metal oxide semiconductor element or other element, and the present invention is not limited thereto. From another point of view, the second image sensor 120 can be an RGB image sensor.

Referring to FIG. 1B , in the embodiment of the present invention, the user U1 is located in front of the display 130, and the first image sensor 110 is used to capture the user image toward the front of the display 130. According to the eye tracking technology or face tracking technology known to those skilled in the art, the user image can be used to track the eye position information or face position information of the user U1, etc. The second image sensor 120 is used to capture the scene image toward the back of the display 130. Based on this, the processor 150 can obtain the user position information of the user U1 based on the user image captured by the first image sensor 110, and the processor 150 can obtain the scene position information, such as the spatial position information of the scene object Obj1, based on the scene image captured by the second image sensor 120.

In the embodiment of the present invention, the processor 150 may determine the display frame F1 according to the user location information of the user U1. The display frame F1 is used to display the scene behind the display 130, and the display scene of the display frame F1 may be roughly aligned with the surrounding actual scene. As shown in FIG. 1B , although the user U1 cannot directly see the scene object Obj1 blocked by the display 130, the display frame F1 may include the image of the scene object Obj1 to realize the function of penetrating display. It is worth noting that when the user U1 moves, the scene content blocked by the display 130 also changes accordingly. The display scene of the display frame F1 can also change in response to the movement of the user U1, so that the display scene of the display frame F1 can remain roughly aligned with the surrounding actual scene.

Please refer to FIG. 1C . In an embodiment of the present invention, the transparent display system 10 may be a laptop. The first image sensor 110 may be a front camera module disposed above the display plane of the display 130 , and the second image sensor 120 may be a rear camera module disposed on the cover of the laptop. When the user U1 operates the laptop, the display 130 of the laptop may display the scene content obscured by the laptop according to the current position of the user U1 .

In some embodiments, the field of view (FOV) 111 of the first image sensor 110 is smaller than the FOV 112 of the second image sensor 120 to ensure that the second image sensor 120 captures sufficient scene content. In some embodiments, the lens of the second image sensor 120 can be implemented by a fisheye lens or a wide-angle lens with a large FOV.

In addition, in some embodiments, the scene range displayed by the display 130 can be determined according to the user position information of the user U1 and the actual size of the display 130. Further, under the condition that the user U1 is regarded as a virtual camera, the processor 150 can determine the FOV 113 and the viewing cone of the virtual camera according to the user position information of the user U1 and the actual size of the display 130.

FIG. 2 is a flow chart of a transparent display method according to an embodiment of the present invention, and the method flow of FIG. 2 can be implemented by the transparent display system 10 of FIG. 1A. Here, the user can view the scene content behind the display 130 through the display 130 of the transparent display system 10.

In step S210, the processor 150 captures a user image toward the front side of the display 130 through the first image sensor 110, and captures a scene image toward the back side of the display 130 through the second image sensor 120. Specifically, the first image sensor 110 is used to shoot toward the user viewing the display 130, and the second image sensor 120 is used to shoot toward the actual scene behind the display 130.

In step S220, the processor 150 obtains user position information associated with a stereo reference coordinate system based on the user image. It should be noted that the stereo reference coordinate system is defined based on the display plane of the display 130. In some embodiments, the user position information may include distance information between the user and the display 130. The processor 150 may estimate the distance information between the user and the display 130 based on the face size, pupil distance, or other facial features in the user image. Alternatively, in other embodiments, the user position information may include the three-dimensional user coordinates of the user in a three-dimensional coordinate system. The processor 150 can convert the image coordinates of the user in the user image into world coordinates in a world coordinate system (e.g., a stereo reference coordinate system established based on the display 130) according to the intrinsic parameters and external parameters of the first image sensor 110.

In some embodiments, the first coordinate axis and the second coordinate axis of the stereo reference coordinate system are parallel to the display plane of the display 130, and the origin of the stereo reference coordinate system is a reference point on the display plane. For example, the origin of the stereo reference coordinate system can be the center point on the display plane. The X axis and the Y axis of the stereo reference coordinate system are located on the display plane. That is, the display plane is the plane with Z=0 in the stereo reference coordinate system.

In some embodiments, the first image sensor 110 may also be used in conjunction with at least one depth sensor (not shown) or distance sensor (not shown) to perform image recognition and positioning of the user to obtain the three-dimensional user coordinates of the user in the stereo reference coordinate system.

In step S230, the processor 150 obtains scene position information associated with a stereo reference coordinate system based on the scene image. In some embodiments, the scene position information may include three-dimensional scene coordinates in a three-dimensional coordinate system. The processor 150 may convert the image coordinates in the scene image into world coordinates in a world coordinate system (e.g., a stereo reference coordinate system established based on the display 130) based on the intrinsic parameters and external parameters of the second image sensor 120. In some embodiments, the image coordinates in the scene image that are converted may be sampled from grid nodes of a three-dimensional grid.

In some embodiments, FIG. 3 is a flow chart of obtaining scene location information according to an embodiment of the present invention. Referring to FIG. 3 , in step S310, the processor 150 establishes a three-dimensional reference coordinate system based on the display plane of the display 130.

For example, FIG. 4 is a schematic diagram of a stereo reference coordinate system according to an embodiment of the present invention. Referring to FIG. 4 , the processor 150 may set the origin (0,0,0) of the stereo reference coordinate system as the center point on the display plane S1. The X axis of the stereo reference coordinate system is the display horizontal axis of the display 130, and the Y axis of the stereo reference coordinate system is the display vertical axis of the display 130. The Z axis of the stereo reference coordinate system passes through the origin (0,0,0) and is perpendicular to the display plane S1.

In step S320, the processor 150 determines the extrinsic parameters of the second image sensor 120 according to the spatial position relationship between the second image sensor 120 and the display 130. The extrinsic parameters of the second image sensor 120 describe the position and orientation of the second image sensor 120, and the conversion relationship between the second image sensor 120 and the world coordinate system. These extrinsic parameters are usually used to define the spatial posture (position and orientation) of the second image sensor 120, so as to map points in the camera coordinate system to the world coordinate system, or map points in the world coordinate system to the camera coordinate system.

In some embodiments, the processor 150 may define a stereo reference coordinate system based on the display plane of the display 130 and use the stereo reference coordinate system as the world coordinate system. Under this condition, the processor 150 may obtain the coordinate position of the second image sensor 120 in the stereo reference coordinate system according to the spatial position relationship between the second image sensor 120 and the display 130. In addition, other external parameters of the second image sensor 120, such as shooting direction, etc., may be obtained through a camera calibration procedure.

In step S330, the processor 150 performs coordinate conversion on the scene pixel coordinates in the scene image according to the internal parameters and external parameters of the second image sensor 120 to obtain scene position information associated with the stereo reference coordinate system. Specifically, the processor 150 may perform coordinate conversion based on the following formula (1) to convert the scene pixel coordinates in the scene image into stereo scene coordinates in the stereo reference coordinate system. In some embodiments, the processor 150 may convert the image coordinates of the grid nodes of the grid of the scene image into stereo scene coordinates.

其中，(u,v)代表影像座標，(X,Y,Z)代表世界座標，

為第二影像感測器120的內部參數矩陣，而

為第二影像感測器120的外部參數矩陣。外部參數矩陣包括旋轉矩陣(rotation matrix)R與平移向量(translation vector)T。第二影像感測器120的外部參數矩陣可用以表示第二影像感測器120於世界座標系(亦即立體參考座標系)中的位置與拍攝方向。如此一來，處理器150可透過座標轉換而將場景影像中的多個影像座標轉換為立體參考座標系中的立體場景座標。

Among them, (u,v) represents the image coordinates, (X,Y,Z) represents the world coordinates,

is the internal parameter matrix of the second image sensor 120, and

is the external parameter matrix of the second image sensor 120. The external parameter matrix includes a rotation matrix R and a translation vector T. The external parameter matrix of the second image sensor 120 can be used to represent the position and shooting direction of the second image sensor 120 in the world coordinate system (i.e., the stereo reference coordinate system). In this way, the processor 150 can convert multiple image coordinates in the scene image into stereo scene coordinates in the stereo reference coordinate system through coordinate conversion.

In some embodiments, the second image sensor 120 may include a fisheye lens or a wide-angle lens, and the fisheye lens or the wide-angle lens is used to capture a scene image. Before obtaining the scene position information related to the stereo reference coordinate system through coordinate conversion, the processor 150 may perform a deformation correction process on the scene image. In other words, the processor 150 may correct the image distortion of the fisheye image or the wide-angle image. Specifically, when the second image sensor 120 captures the scene image through the wide-angle lens, the processor 150 may perform the deformation correction process through formula (2). When the second image sensor 120 captures the scene image through the fisheye lens, the processor 150 may perform the deformation correction process through formula (3).

其中，

，θ=tan^-1 r，k_n為徑向畸變係數(radial distortion coefficients)，與p_n為切向畸變係數(tangential distortion coefficients)。

in,

, θ =tan ^-1 r , k _n is the radial distortion coefficient, and p _n is the tangential distortion coefficient.

Returning to FIG. 2 , in step S240 , the processor 150 determines the viewing frustum according to the user position information and the actual size of the display. The viewing frustum is a geometric body used to represent the visible area of the camera, and can also be called a projection viewing frustum. The viewing frustum is composed of six planes, namely the near plane, the far plane, the left plane, the right plane, the top plane, and the bottom plane. These planes define the area that the camera can see. In other words, this viewing frustum can be used to determine the local scene image to be captured from the scene image.

In an embodiment of the present invention, the processor 150 may set the user coordinates of the user in the stereo reference coordinate system as the coordinate position of the virtual camera, and determine the viewing cone based on the user coordinates. In some embodiments, the viewing cone changes in response to changes in the user position information. That is, when the user moves, the viewing cone will also change accordingly.

In some embodiments, the user position information may include a user coordinate associated with a stereo reference coordinate system, and the viewing cone is obtained by connecting the user coordinate with multiple vertices of the display plane of the display 130. That is, the left plane, right plane, top plane, and bottom plane of the viewing cone are determined according to the display range of the display 130. For example, FIG5 is a schematic diagram of determining the viewing cone according to the user position information according to an embodiment of the present invention. Referring to FIG5, the display plane S1 of the display 130 includes vertices DP1, DP2, DP3, and DP4. After obtaining the user coordinate VP1 in the stereo reference coordinate system according to the user image, the processor 150 can connect the user coordinate VP1 and multiple vertices DP1, DP2, DP3, and DP4 of the display plane S1 to obtain the viewing cone 51. The processor 150 can set the far plane and the near plane of the viewing cone 51 according to the preset distance. It can be known that the aspect ratio of the viewport is the aspect ratio of the display 130.

After determining the viewing cone according to the actual size of the display 130 and the user coordinates, the processor 150 can derive the parameters of the projection matrix. Specifically, each plane of the viewing cone defines the parameters in the projection matrix, such as the viewing angle, the viewport aspect ratio, the distance between the near plane and the far plane, etc. These parameters determine the values of the projection matrix. In some embodiments, the projection matrix can be an off-center perspective matrix. For example, the projection matrix P obtained by the processor 150 can be represented by formula (5).

其中，near代表近平面與使用者座標之間的距離，far代表遠平面與使用者座標之間的距離，right代表顯示器130的右顯示邊界的X座標，left代表顯示器130的左顯示邊界的X座標。top代表顯 示器130的上顯示邊界的Y座標，bottom代表顯示器130的下顯示邊界的Y座標。

Among them, near represents the distance between the near plane and the user coordinates, far represents the distance between the far plane and the user coordinates, right represents the X coordinate of the right display boundary of the display 130, and left represents the X coordinate of the left display boundary of the display 130. Top represents the Y coordinate of the upper display boundary of the display 130, and bottom represents the Y coordinate of the lower display boundary of the display 130.

In step S250, the processor 150 uses the projection matrix of the viewing cone to generate a display frame projected on the display plane of the display 130 according to the scene position information. Specifically, the processor 150 can multiply the four-dimensional coordinates (x, y, z, 1) of multiple three-dimensional scene coordinates in the stereo reference coordinate system with the projection matrix P to map these scene coordinates to the corresponding screen coordinates on the viewport (i.e., the display plane). The above scene coordinates can be the three-dimensional coordinates of multiple grid nodes in the stereo reference coordinate system. In other words, the local scene image in the scene image can be projected to the display plane via the projection matrix to generate a display frame.

In step S260, the processor 150 outputs a display frame through the display 130 to display the scene behind the display 130. Specifically, since the projection range of the scene image projected onto the display plane of the display 130 is determined according to the user's position information and the actual size of the display 130, the display frame output by the display 130 can not only present the scene behind the display 130, but also the scene image in the display frame can be aligned with the actual scene around the display 130. In addition, in response to the user's movement, the scene content of the display frame of the display 130 will also change accordingly.

For example, FIG. 6A and FIG. 6B are schematic diagrams of the display scene according to an embodiment of the present invention. Referring to FIG. 6A, when the user U1 is at the first position, the display frame of the display 130 may include the scene content behind the display 130. For example, the container blocked by the display 130 may be displayed in the display 130. Referring to FIG. 6B, when the user moves from the first position to the second position, the visual cone determined based on the user's position will change accordingly. Therefore, the scene content captured by the visual cone will also change, causing the scene content displayed by the display 130 to be adjusted accordingly. In some embodiments, the processor 150 may also use the above display frame as the background of the AR screen to provide an AR function or AR application.

FIG7 is a schematic diagram of a transmissive display system according to an embodiment of the present invention. Referring to FIG7 , the transmissive display system 70 may include a first image sensor 110, a second image sensor 120, a display 130, a storage device 140, at least one processor 150, and a depth sensor 160. Different from the embodiment of FIG1 , the transmissive display system 70 may also include a depth sensor 160 for sensing scene depth information. The depth sensor 160 may be implemented using active depth sensing technology and passive depth sensing technology. Active depth sensing technology may calculate depth information by actively emitting light sources, infrared rays, ultrasound waves, lasers, etc. as signals in combination with time difference ranging technology. Passive depth sensing technology can use two image sensors to capture two images in front of it at different viewing angles, and use the parallax of the two images to calculate depth information.

FIG8 is a flow chart of a transparent display method according to an embodiment of the present invention, and the method flow of FIG8 can be implemented by the transparent display system 70 of FIG7. Here, the user can view the scene content behind the display 130 through the display 130 of the transparent display system 70.

In this embodiment, the scene position information of the scene image includes three-dimensional grid scene information associated with a stereo reference coordinate system. The Z coordinate value of the grid node in the three-dimensional grid scene information in the stereo reference coordinate system can be generated according to the scene depth.

In step S810, the processor 150 captures a user image toward the front side of the display 130 through the first image sensor 110, and captures a scene image toward the back side of the display 130 through the second image sensor 120. In step S820, the processor 150 obtains user position information associated with a stereo reference coordinate system based on the user image. These steps can refer to the description of the aforementioned embodiment and will not be repeated here.

In step S830, the processor 150 obtains depth information corresponding to the scene image. For example, the value of each pixel (or position) in the depth map may indicate the depth value of the corresponding pixel (or position) in the scene image. The processor 11 may use the depth map as the depth information corresponding to the scene image.

In some embodiments, the processor 150 may use the depth sensor 160 to obtain depth information corresponding to the scene image. Alternatively, in other embodiments, the processor 150 may perform image preprocessing operations on the scene image to generate an adjusted scene image that meets the input requirements of the deep learning model. The processor 150 may analyze the adjusted scene image through the deep learning model to obtain depth information.

In some embodiments, the storage device 140 may store a deep learning model. The deep learning model is implemented based on a neural network structure such as a convolutional neural network (CNN) or a neural network-like network. The deep learning model is used to estimate (i.e., predict) the depth of each pixel (or position) in the scene image. In addition, the processor 150 may perform an image preprocessing operation on the scene image to generate an adjusted scene image that meets the input requirements of the deep learning model. For example, in the image preprocessing operation, the processor 150 may adjust the size of the scene image and/or convert the format of the scene image to generate an adjusted scene image. The processor 150 analyzes the adjusted scene image through the deep learning model to obtain depth information corresponding to the first image. For example, the processor 150 may input the adjusted scene image into the deep learning model, and then receive the output depth map of the deep learning model with respect to the adjusted scene image.

In some embodiments, the processor 150 may determine whether the depth sensor 160 for sensing scene depth information is available, or determine whether the depth learning model for estimating scene depth information is available. When the depth sensor 160 is available or the depth learning model is available, the processor 150 may obtain the depth information of the scene image.

When at least one of the depth sensor 160 and the deep learning model is available, in step S840, the processor 150 generates three-dimensional grid scene information associated with the stereo reference coordinate system based on the depth information and the scene image. In one embodiment, the height of the grid node of the three-dimensional grid in the Z-axis direction of the stereo reference coordinate system can be regarded as the depth of the three-dimensional grid node. Otherwise, when neither the depth sensor 160 nor the deep learning model is available, the processor 150 can generate grid information of a plane with the same height in the Z-axis direction.

In step S850a, the processor 150 determines the first viewing cone according to the right eye position information in the user position information and the actual size of the display 130. In step S850b, the processor 150 determines the second viewing cone according to the left eye position information in the user position information and the actual size of the display. The detailed operation of the processor 150 determining the viewing cone can refer to the description of the aforementioned embodiment. It should be particularly noted that in some embodiments, the processor 150 can locate the right eye position information of the user's right eye and the left eye position information of the left eye according to the user image. The right eye position information and the left eye position information can be the left eye coordinates and the right eye coordinates in the stereo reference coordinate system, respectively. Therefore, the processor 150 can determine the first viewing cone and the second viewing cone according to the left eye coordinates, the right eye coordinates and the actual size of the display 130. It can be seen that since the right eye coordinates are different from the left eye coordinates, the scene contents captured by the first viewing cone and the second viewing cone will also be different.

In step S860a, the processor 150 uses the projection matrix of the first viewing cone to generate a right-eye display frame projected on the display plane of the display 130 according to the scene position information. Specifically, the processor 150 may project the three-dimensional grid nodes in the scene image captured by the first viewing cone onto the display plane of the display 130 to render the right-eye display frame.

In step S860b, the processor 150 uses the projection matrix of the second viewing cone to generate a left-eye display frame projected on the display plane of the display 130 according to the scene position information. Specifically, the processor 150 may project the three-dimensional grid nodes in the scene image captured by the second viewing cone onto the display plane of the display 130 to render the left-eye display frame.

It should be noted that the processor 150 uses depth estimation technology or depth sensing technology to obtain the scene depth of the three-dimensional grid nodes, and then projects the three-dimensional grid nodes with depth information onto the display plane of the display 130. Therefore, the scene content presented by the display 130 can be more accurate, and the perceived position of the scene objects will not be improperly offset due to the lack of depth information.

In step S870, the processor 150 outputs the left eye display frame and the right eye display frame through the display 130 to display the scene behind the display 130. In some embodiments, when the display 130 is a naked eye 3D display, the processor 150 can perform image weaving processing on the left eye display frame and the right eye display frame to synchronously interlace and display the left eye display frame and the right eye display frame. When the display 130 is a glasses-type 3D display, the processor 150 can control the display 130 to alternately display the left eye display frame and the right eye display frame. In this way, the user can experience a stereoscopic visual effect.

Based on the above, in the embodiment of the present invention, the user position information and the scene position information in the same stereo reference coordinate system can be obtained based on the user image and the scene image. The viewing cone used to determine the display content of the display frame can be based on the user position information and the actual size of the display. Therefore, the display scene content of the display frame output by the display can change in response to the user's movement and achieve good alignment with the actual scene around the display. In addition, when the scene depth information is used to project the content of the scene image onto the display plane, the scene content presented by the display can be closer to the actual scene.

S210~S260: Steps

Claims (20)

A method for transparent display, applicable to a transparent display system including a first image sensor, a second image sensor and a display, comprising: Capturing a user image toward the front side of the display through the first image sensor, and capturing a scene image toward the back side of the display through the second image sensor; Obtaining user position information associated with a stereo reference coordinate system based on the user image; Obtaining scene position information associated with the stereo reference coordinate system based on the scene image; Determining a viewing frustum based on the user position information and the actual size of the display; Using the projection matrix of the viewing frustum, generating a display frame projected on the display plane of the display according to the scene position information; and The display frame is outputted through the display to display the scene behind the display. As described in claim 1, the second image sensor includes a fisheye lens or a wide-angle lens, and the fisheye lens or the wide-angle lens is used to capture the scene image. Before the step of obtaining the scene position information associated with the stereo reference coordinate system based on the scene image, the method further includes: performing a deformation correction process on the scene image. A transparent display method as described in claim 1, wherein the first coordinate axis and the second coordinate axis of the three-dimensional reference coordinate system are parallel to the display plane of the display, and the origin of the three-dimensional reference coordinate system is a reference point on the display plane. A transparent display method as described in claim 1, wherein the viewing cone changes in response to changes in the user's location information. The transparent display method as described in claim 1, wherein the step of obtaining the scene position information associated with the stereo reference coordinate system according to the scene image comprises: Establishing the stereo reference coordinate system based on the display plane of the display; Determining the external parameters of the second image sensor according to the spatial position relationship between the second image sensor and the display; and Performing coordinate conversion on the scene pixel coordinates in the scene image according to the internal parameters of the second image sensor and the external parameters to obtain the scene position information associated with the stereo reference coordinate system. A transparent display method as described in claim 1, wherein the user position information includes a user coordinate associated with the stereo reference coordinate system, and the viewing cone is obtained by connecting the user coordinate and multiple vertices of the display plane. As described in claim 1, the scene position information includes three-dimensional grid scene information associated with the stereo reference coordinate system, and the step of obtaining the scene position information associated with the stereo reference coordinate system based on the scene image includes: Obtaining depth information corresponding to the scene image; and Generating the three-dimensional grid scene information associated with the stereo reference coordinate system based on the depth information and the scene image. In the transparent display method as described in claim 7, the step of obtaining the depth information corresponding to the scene image includes: performing image preprocessing operations on the scene image to generate an adjusted scene image that meets the input requirements of the deep learning model; and analyzing the adjusted scene image through the deep learning model to obtain the depth information. In the transparent display method as described in claim 7, the step of obtaining the depth information corresponding to the scene image includes: Using a depth sensor to obtain the depth information corresponding to the scene image. The transparent display method as described in claim 1, wherein the display includes a stereoscopic display that provides right-eye display frames and left-eye display frames, and the step of determining the viewing cone according to the user position information and the actual size of the display includes: Determining a first viewing cone according to the right eye position information in the user position information and the actual size of the display; and Determining a second viewing cone according to the left eye position information in the user position information and the actual size of the display, wherein using the projection matrix of the viewing cone, the step of generating the display frame projected on the display plane of the display according to the scene position information includes: Using the projection matrix of the first viewing cone, the right eye display frame is generated according to the scene position information and projected on the display plane of the display; and Using the projection matrix of the second viewing cone, the left eye display frame is generated according to the scene position information and projected on the display plane of the display. A transparent display system comprises: a first image sensor; a second image sensor; a display; and at least one processor coupled to the first image sensor, the second image sensor and the display, wherein the at least one processor is used to: capture a user image toward the front side of the display through the first image sensor, and capture a scene image toward the back side of the display through the second image sensor; obtain user position information associated with a stereo reference coordinate system based on the user image; obtain scene position information associated with the stereo reference coordinate system based on the scene image; determine a viewing cone based on the user position information and the actual size of the display; Using the projection matrix of the viewing cone, a display frame projected on the display plane of the display is generated according to the scene position information; and outputting the display frame through the display to display the scene behind the display. A transparent display system as described in claim 11, wherein the second image sensor includes a fisheye lens or a wide-angle lens, the fisheye lens or the wide-angle lens is used to capture the scene image, and the at least one processor is used to: Perform a deformation correction process on the scene image. A transmissive display system as described in claim 11, wherein the first coordinate axis and the second coordinate axis of the three-dimensional reference coordinate system are parallel to the display plane of the display, and the origin of the three-dimensional reference coordinate system is a reference point on the display plane. A see-through display system as described in claim 11, wherein the viewing cone changes in response to changes in the user's location information. A transparent display system as described in claim 11, wherein the at least one processor is used to: establish the stereo reference coordinate system based on the display; determine the external parameters of the second image sensor according to the spatial position relationship between the second image sensor and the display; and perform coordinate conversion on the scene pixel coordinates in the scene image according to the internal parameters of the second image sensor and the external parameters to obtain the scene position information associated with the stereo reference coordinate system. A see-through display system as described in claim 11, wherein the user position information includes a user coordinate associated with the stereo reference coordinate system, and the viewing cone is obtained by connecting the user coordinate and multiple vertices of the display plane. A see-through display system as described in claim 11, wherein the scene position information includes a three-dimensional grid scene information associated with the stereo reference coordinate system, and the at least one processor is used to: obtain depth information corresponding to the scene image; and generate the three-dimensional grid scene information associated with the stereo reference coordinate system based on the depth information and the scene image. A see-through display system as described in claim 17, wherein the at least one processor is used to: perform image preprocessing operations on the scene image to generate an adjusted scene image that meets the input requirements of the deep learning model; and analyze the adjusted scene image through the deep learning model to obtain the depth information. The transparent display system as described in claim 17 further includes a depth sensor, and the at least one processor is connected to the depth sensor and is used to: Utilize the depth sensor to obtain the depth information corresponding to the scene image. A transparent display system as described in claim 11, wherein the display includes a stereoscopic display that provides right-eye display frames and left-eye display frames, and the at least one processor is used to: Determine a first viewing cone based on the right-eye position information in the user position information and the actual size of the display; and Determine a second viewing cone based on the left-eye position information in the user position information and the actual size of the display, wherein the at least one processor is used to: Generate the right-eye display frame projected on the display plane of the display based on the scene position information using the projection matrix of the first viewing cone; and Generate the left-eye display frame projected on the display plane of the display based on the scene position information using the projection matrix of the second viewing cone.