TWI907161B - Method for display virtual object, head mounted display system and head mounted display apparatus - Google Patents

Abstract

A method for display virtual object, a head mounted display system and a head mounted display apparatus are provided. The method includes the following steps. An image capturing device on the head mounted display apparatus is used to capture an environment image toward a display screen. A plurality of preset markers are presented on the display screen. Real-time pose information of the display screen is determined based on the preset markers in the environment image. Based on the real-time pose information of the display screen and device pose information of the head-mounted display apparatus, target pose information of the display screen in an augmented reality coordinate system is determined. According to the target pose information of the display screen, a virtual object anchored on the display screen is displayed through the head-mounted display device.

Description

Methods for displaying virtual objects, head-mounted display systems and head-mounted display devices

This invention relates to augmented reality display, and more particularly to a method for displaying virtual objects, a head-mounted display system, and a head-mounted display device.

With the advancement of technology, the application of Augmented Reality (AR) technology is becoming increasingly widespread. AR technology overlays virtual information onto the real world, creating a more interactive and immersive experience. Furthermore, head-mounted displays equipped with AR capabilities are increasingly integrating with other real-world display devices, enabling more diverse and functional display scenarios. However, for head-mounted displays to work smoothly with other display devices, the display position, size, and proportion of virtual objects need to be dynamically adjusted based on parameters such as the screen size, position, and resolution of the other display devices. This ensures optimal harmony between the virtual content and the real-world display. Currently, how to make virtual objects ideally match the display content areas of other display devices in real scenes remains a concern for technical personnel in this field.

This invention provides a method for displaying a virtual object, comprising the following steps: Capturing an environmental image towards a display screen using an image capturing device on a head-mounted display device. Presenting multiple preset markers on the display screen. Determining real-time pose information of the display screen based on the multiple preset markers in the environmental image. Determining target pose information of the display screen in an augmented reality coordinate system based on the real-time pose information of the display screen and the device pose information of the head-mounted display device. Displaying a virtual object anchored to the display screen via the head-mounted display device based on the target pose information of the display screen.

This invention provides a head-mounted display system, comprising a head-mounted display device and a computer device. The head-mounted display device includes an image capturing device. The image capturing device is used to capture an environmental image toward a display screen. A plurality of preset markers are displayed on the display screen. The computer device is connected to the head-mounted display device and includes a storage device and a processor. The processor is coupled to the storage device and configured to perform the following operations: determining real-time pose information of the display screen based on the plurality of preset markers in the environmental image; determining target pose information of the display screen in an augmented reality coordinate system based on the real-time pose information of the display screen and the device pose information of the head-mounted display device. Based on the target pose information on the display screen, a virtual object anchored to the display screen is displayed through a head-mounted display device.

This invention provides a head-mounted display device, comprising an image capturing device, a storage device, and a processor. The image capturing device captures an environmental image toward a display screen. Multiple preset markers are displayed on the display screen. The processor is coupled to the storage device and configured to perform the following operations: determining real-time pose information of the display screen based on the multiple preset markers in the environmental image; determining target pose information of the display screen in an augmented reality coordinate system based on the real-time pose information of the display screen and the device pose information of the head-mounted display device; and displaying a virtual object anchored to the display screen via the head-mounted display device based on the target pose information of the display screen.

Based on the above, in embodiments of the present invention, multiple preset markers are displayed on the screen. The real-time pose information of the display screen can be estimated based on the preset markers in the environmental image captured by the head-mounted display. The target pose information of the display screen in the augmented reality coordinate system can be determined based on the real-time pose information of the display screen and the device pose information of the head-mounted display. Therefore, the display position of the virtual object can be determined based on the target pose information of the display screen, allowing the user to view a virtual object stably anchored on the display screen via the head-mounted display. Based on this, the coordination accuracy and flexibility between virtual and real displays can be improved, achieving a smoother augmented reality experience.

Some embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Component symbols used in the following description, when appearing in different accompanying drawings, are considered to be the same or similar components. These embodiments are only a part of the present invention and do not disclose all possible embodiments of the present invention. More precisely, these embodiments are merely examples of methods and systems within the scope of the patent applications of the present invention.

Figures 1A and 1B are schematic diagrams of a head-mounted display system according to an embodiment of the present invention. Referring to Figure 1A, the head-mounted display system 10A includes a head-mounted display device 110 and a display screen 120. Referring to Figure 1B, the head-mounted display system 10B includes a head-mounted display device 110, a computer device 130, and a display screen 120.

In some embodiments, the display screen 120 may be, for example, a laptop screen, a desktop monitor, a tablet computer screen, a television, etc. From another perspective, the display screen 120 may be various types of displays, such as a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, or an Organic Light Emitting Diode (OLED), and the present invention is not limited thereto. Alternatively, in some embodiments, the display screen 120 may be a projection screen or other wall or other projection surface suitable for displaying projected images, etc.

In some embodiments, the head-mounted display device 110 may be, for example, augmented reality glasses or a mixed reality device. In some embodiments, the head-mounted display device 110 may be connected to an electronic device including a display screen 120 via a wired or wireless transmission interface. For example, the head-mounted display device 110 may be connected to a laptop computer including a display screen 120 via a wired or wireless transmission interface. Head-mounted display systems 10A and 10B can be used to provide augmented reality content to a user. The head-mounted display device 110 can be used to display virtual objects. In some embodiments, the virtual objects displayed by the head-mounted display device 110 are displayed as a display screen 120 anchored to a real-world scene.

In some embodiments, when a user wears a head-mounted display device 110 and faces a display screen 120 in a real-world setting, the virtual objects displayed by the head-mounted display device 110 may be virtual images or virtual windows, etc.

It should be noted that, in the embodiment of FIG1A, the head-mounted display device 110 may include an image capturing device 111, a storage device 112, and a processor 113, and the processor 113 of the head-mounted display device 110 can generate display content itself. On the other hand, in the embodiment of FIG1B, the head-mounted display device 110 may include an image capturing device 111 and is connected to a computer device 130 including a storage device 112 and a processor 113. That is, in the example of FIG1B, the head-mounted display device 110 displays display content determined by the processor 113 of the computer device 130.

Image capturing device 111 is used to capture environmental images and includes a camera lens with a lens and a photosensitive element. The photosensitive element is used to sense the intensity of light entering the lens, thereby generating an image. The photosensitive element can be, for example, a charge-coupled device (CCD), a complementary metal-oxide semiconductor (CMOS) element, or other elements, and the present invention is not limited thereto. In one embodiment, image capturing device 111 is fixedly mounted on head-mounted display device 110 and is used to capture the actual scene located in front of head-mounted display device 110. For example, when a user wears the head-mounted display device 110, the image capturing device 111 can be positioned between the user's eyes or in front of one of their eyes to capture images of the actual scene in front of the user. In this embodiment of the invention, the image capturing device 111 can capture images of the display screen 120.

It should also be noted that the content viewed by the user through the display of the head-mounted display device 110 is an augmented reality scene overlaid with virtual objects. The head-mounted display device 110 may include a display for displaying virtual objects (not shown in Figures 1A and 1B). In some embodiments, the display of the head-mounted display device 110 may be a light-transmitting display, so that the user can view the actual scene on the other side relative to the viewer. In other embodiments, the display of the head-mounted display device 110 may be a display that simultaneously displays the actual scene and virtual objects, that is, the head-mounted display device 110 can display the actual scene and virtual objects simultaneously.

The storage device 112 is used to store data and program code (such as operating system, application, driver) for access by the processor 113. The data can be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, or a combination thereof.

Processor 113 is coupled to storage device 112, such as a central processing unit (CPU), application processor (AP), or other programmable general-purpose or special-purpose microprocessor, digital signal processor (DSP), image signal processor (ISP), graphics processing unit (GPU), or other similar devices, integrated circuits, and combinations thereof. Processor 113 can access and execute program code and software elements recorded in storage device 112 to implement the method of displaying virtual objects in the embodiments of the present invention.

Figure 2 is an application scenario diagram of a head-mounted display system according to an embodiment of the present invention. Referring to Figure 2, when a user views the display screen 120 through the display of the head-mounted display device 110, the user can see an augmented reality scene where virtual objects V_obj1 and V_obj2 are superimposed on the actual scene. In detail, when the user views the virtual object while wearing the head-mounted display device 110, the image capturing device 111 captures a sequence of images, including multiple environmental images, onto the display screen 120. The processor 113 can determine the display parameters of the virtual objects V_obj1 and V_obj2 in real time based on these environmental images, such as display boundaries, display size, or display position, so that the virtual objects V_obj1 and V_obj2 can be displayed as anchored on the display screen 120. The virtual objects V_obj1 and V_obj2 displayed on the display of the head-mounted display device 110 can move relative to each other as the user moves or their head turns.

As illustrated in Figure 2, when a user views the display screen 120 through the display of the head-mounted display device 110, the user can see a virtual object V_obj1 extending outward from the right display border of the display screen 120, and a virtual object V_obj2 covering the display area of the display screen 120. The virtual objects V_obj1 and V_obj2 can be used to provide various information to the user, such as windows, documents, images, desktops, or visual output generated by running applications. However, Figure 2 is only an illustrative example, and the present invention does not limit the number of virtual objects or their display position.

It should be noted that the image capturing device 111 can capture multiple environmental images periodically and continuously (e.g., generating environmental images at a capture frame rate of 30Hz), and the processor 113 can repeatedly calculate the target pose information of the display screen 120 in the augmented reality coordinate system to continuously update the display position of the virtual object based on the target pose information of the display screen 120. Therefore, provided the conditions for displaying virtual objects V_obj1 and V_obj2 are met, even if the user's position changes or their head turns, virtual objects V_obj1 and V_obj2 can still be displayed as fixed positions anchored in the actual scene relative to the display screen 120. For example, even if the user's position changes or their head turns, the object boundary of virtual object V_obj2 can remain aligned with the screen boundary of display screen 112.

It should be noted that in this embodiment of the invention, the display screen 120 displays multiple preset markers, and the processor 113 can locate the real-time position of the display screen 120 and obtain the real-time posture of the display screen 120 based on the preset markers in the environmental image. Therefore, based on the real-time posture information of the display screen 120, the processor determines the display parameters of the virtual object, such as display boundaries, display size, or display position, so that the virtual object can be displayed as if anchored to the display screen 120. The following examples, in conjunction with the components of the head-mounted display systems 10A and 10B, illustrate the detailed steps of the method for displaying virtual objects.

Figure 3 is a flowchart of the anchoring display method according to an embodiment of the present invention. Referring to Figures 1A, 1B and 3, the method of this embodiment is applicable to the head-mounted display systems 10A and 10B in the above embodiments. The following describes the detailed steps of the method for displaying virtual objects in this embodiment with reference to the various components in the head-mounted display systems 10A and 10B.

In step S310, the processor 113 uses an image capturing device 111 on a head-mounted display 110 to capture an environmental image toward the display screen 120. Here, multiple preset markers are displayed on the display screen 120. That is, when the user wears the head-mounted display 110 and faces the display screen 120, the image capturing device 111 can capture an environmental image toward the display screen 120, including some or all of the preset markers.

In some embodiments, these default markers can be multiple QR codes of the same size. In some embodiments, multiple default markers can include multiple binary square markers (ArUco markers). Each binary square marker consists of a wide black border and an internal binary matrix, the internal matrix determining the identifier of the binary square marker. Alternatively, each binary square marker can be a square matrix composed of internal matrices of unit squares such as 4x4, 5x5, 6x6, 7x7, or 8x8. In other embodiments, these default markers can be QR codes.

In some embodiments, the display screen 120 may display multiple preset icons. Alternatively, in some embodiments, these preset icons may be projected onto the display screen 120. Alternatively, in some embodiments, these preset icons may be implemented as stickers or cards, etc., attached to the display screen 120.

In some embodiments, multiple default icons are displayed at multiple corners and the center of the display screen 120. The multiple default icons include multiple corner icons and one center icon. The multiple corner icons are located at multiple corners of the display screen 120, while the center icon is located at the center of the display screen 120. In some embodiments, the display screen 120 may display the four corner icons at the four corners of the screen respectively. The display screen 120 may display the center icon at the center of the screen respectively. These default icons may be binary squared icons corresponding to different icon identifiers.

For example, Figure 4 is a schematic diagram of a display screen and multiple preset marks according to an embodiment of the present invention. Referring to Figure 4, multiple corner marks M1 to M4 are displayed at the four corners of the display screen 120. The center mark M5 is located at the center of the display screen 120. The multiple corner marks M1 to M4 and the center mark M5 are all binary square marks of the same size and correspond to different mark identifiers.

In step S320, the processor 113 determines the real-time pose information of the display screen 120 based on multiple preset markers in the environmental image. Specifically, after acquiring the environmental image, the processor 113 can identify some or all of the multiple preset markers from the environmental image. The processor 113 can identify one or more preset markers from the environmental image through image processing such as edge detection and quadrilateral detection to obtain the displacement and rotation information of these preset markers in the camera coordinate system of the image capturing device 111. Subsequently, based on one or more preset displacement and rotation information marked on the camera coordinate system, the processor 113 can determine the real-time pose information of the display screen 120 in the camera coordinate system. The real-time pose information of the display screen 120 may include displacement and rotation information in the camera coordinate system.

In some embodiments, when the environmental image includes multiple corner markers M1 to M4 and a center marker M5, the processor 113 can use the Perspective-n-Point (PnP) problem to calculate the displacement and rotation information of the multiple corner markers M1 to M4 and the center marker M5 relative to the image capturing device 111.

In some embodiments, when multiple default markers are multiple binary square markers (ArUco markers), the processor 113 can estimate the rotation vector and displacement vector of each binary square marker based on the environmental image. In addition, in some embodiments, the processor 113 can convert the rotation vector of each binary square marker into a single quaternion.

In some embodiments, displacement information may be implemented as a translation vector. A translation vector may include three vector elements, representing the displacement along the X, Y, and Z axes, respectively. Rotation vectors may be implemented as rotation vectors or unit quaternions. For example, a rotation vector may include pitch, roll, and yaw angles.

In some embodiments, processor 113 may also check whether multiple identifiers in the environmental image match multiple preset identifiers on display screen 120. For example, processor 113 may determine whether the identifiers of multiple identifiers in the environmental image are the identifiers of multiple preset identifiers. When multiple identifiers in the environmental image do not match the multiple preset identifiers on display screen 120, processor 113 may disable the display function of virtual objects.

In step S330, the processor 113 determines the target pose information of the display screen 120 in an augmented reality coordinate system based on the real-time pose information of the display screen 120 and the device pose information of the head-mounted display device 110. Specifically, the processor 113 can perform coordinate conversion between a camera coordinate system and an augmented reality coordinate system (also referred to as a head-mounted display coordinate system) to obtain the real-time pose information of the display screen 120 in the augmented reality coordinate system. Here, the real-time pose information of the display screen 120 in the augmented reality coordinate system is the relative pose information of the display screen 120 relative to the head-mounted display device 110. The processor 113 can further determine the absolute pose information of the display screen 120 (i.e., the target pose information in the augmented reality coordinate system) based on the device pose information of the head-mounted display device 110 and the real-time pose information of the display screen 120 in the augmented reality coordinate system.

In step S340, the processor 113 displays a virtual object anchored to the display screen 120 via the head-mounted display device 110 based on the target pose information of the display screen 120. The processor 113 can determine the three-dimensional imaging position of the virtual object in the augmented reality coordinate system based on the target pose information of the display screen 120, and determine the display position of the virtual object in the display frame based on the three-dimensional imaging position of the virtual object in the augmented reality coordinate system. The virtual object is anchored to the display screen 120 and its position does not change as the head-mounted display device 110 moves, so that the virtual object can be integrated with the display screen 120 in the actual scene, thereby enhancing the visual experience and improving convenience.

Figure 5 is a flowchart of a method for displaying virtual objects according to an embodiment of the present invention. Referring to Figures 1A, 1B and 5, the method of this embodiment is applicable to the head-mounted display systems 10A and 10B in the above embodiments. The following describes the detailed steps of the method for displaying virtual objects according to this embodiment with reference to the various components in the head-mounted display systems 10A and 10B.

In step S510, processor 113 controls display screen 120 to display multiple preset markers. As shown in the example of Figure 4, display screen 120 can display four corner markers M1 to M4 and one center marker M5. In step S520, processor 113 uses image capturing device 111 on a head-mounted display device 110 to capture an environmental image towards display screen 120.

In step S530, the processor 113 determines the real-time pose information of the display screen 120 based on multiple preset markers in the environmental image. In this embodiment, step S530 can be implemented as steps S531 to S532.

In step S531, processor 113 calculates the rotation and displacement information of at least one of multiple preset markers in the environmental image. The rotation information can be a rotation vector or a quaternion. The displacement information can be a displacement vector. In step S532, processor 113 determines the first real-time pose information of the display screen 120 in the camera coordinate system of the image capturing device 111 based on the rotation and displacement information of at least one of the multiple preset markers.

Taking Figure 4 as an example, when the processor 113 identifies corner markers M1 to M4 and center marker M5 from the environmental image, the processor 113 can acquire rotation and displacement information of each corner marker M1 to M4 and the center marker M5. Subsequently, in some embodiments, the processor 113 can determine the first real-time pose information of the display screen 120 in the camera coordinate system based on the rotation and displacement information of each corner marker M1 to M4. Alternatively, in some embodiments, the processor 113 can determine the first real-time pose information of the display screen 120 in the camera coordinate system based on the rotation and displacement information of the center marker M5. Alternatively, in some embodiments, the processor 113 can determine the first real-time pose information of the display screen 120 in the camera coordinate system based on the rotation and displacement information of each corner marker M1 to M4 and the rotation and displacement information of the center marker M5.

In some embodiments, processor 113 can determine the real-time displacement and rotation information in the first real-time pose information of display screen 120 based on the rotation and displacement information of a center marker among multiple preset markers. Taking Figure 4 as an example, processor 113 can directly set the rotation and displacement information of center marker M5 as the first real-time pose information of display screen 120 in the camera coordinate system. The real-time displacement information in the first real-time pose information is the displacement information of center marker M5. The real-time rotation information in the first real-time pose information is the rotation information of center marker M5.

In some embodiments, processor 113 can perform a displacement averaging operation on the displacement information of multiple corner markers among multiple preset markers to obtain the real-time displacement information in the first real-time pose information. Taking Figure 4 as an example, processor 113 can calculate the displacement vectors of each corner marker M1 to M4, and calculate the three average values of these four displacement vectors on the three axes respectively, thereby obtaining the average displacement vector in the first real-time pose information. That is, the real-time displacement information in the first real-time pose information can be the average displacement vector.

In some embodiments, processor 113 can perform a spherical linear interpolation operation on the rotation information of multiple corner markers among multiple preset markers to obtain the real-time rotation information in the first real-time pose information. Taking Figure 4 as an example, processor 113 can calculate the quaternions of each corner marker M1 to M4. Then, processor 113 can perform a spherical linear interpolation operation on the quaternions of corner marker M1 and corner marker M4 to obtain a first spherical linear interpolation result. Processor 113 can perform a spherical linear interpolation operation on the quaternions of corner marker M2 and corner marker M3 to obtain a second spherical linear interpolation result. Next, the processor 113 can perform a spherical linear interpolation operation on the first spherical linear interpolation result and the second spherical linear interpolation result to obtain the real-time quaternion (i.e., real-time rotation information) in the first real-time pose information.

Furthermore, in some embodiments, the processor 113 can simultaneously determine the first real-time pose information of the display screen 120 based on the rotation and displacement information of the center marker and the rotation and displacement information of each corner marker. Figure 6 is a flowchart of determining the real-time pose information of the display screen according to an embodiment of the present invention. Please refer to Figure 6.

In step S611, processor 113 can obtain first rotation information and first displacement information based on rotation and displacement information from multiple corner markers. As mentioned above, processor 113 can obtain first rotation information based on rotation information from multiple corner markers through spherical linear interpolation. Processor 113 can obtain first displacement information based on displacement information from multiple corner markers through displacement averaging.

In step S612, processor 113 can obtain second rotation information and second displacement information based on the rotation information and displacement information of the center mark. The second rotation information can be the rotation information of the center mark. The second displacement information can be the displacement information of the center mark.

In step S613, the processor 113 may determine a weighting factor based on the distance between the head-mounted display device 110 and the display screen 120. That is, the weighting factor is determined based on the distance between the head-mounted display device 110 and the display screen 120. The processor 113 may determine the weighting factor based on the distance between the head-mounted display device 110 and the display screen 120 through a lookup table or function calculation. In some embodiments, the processor 113 may substitute the distance between the head-mounted display device 110 and the display screen 120 into a preset function to determine the weighting factor.

In step S614, processor 113 can perform a weighted sum operation on the first rotation information and the second rotation information according to the weight factor to obtain the real-time rotation information in the first real-time pose information. In step S615, processor 113 can perform a weighted sum operation on the first displacement information and the second displacement information according to the weight factor to obtain the real-time displacement information in the first real-time pose information.

For example, the first weighted weight corresponding to the first rotation information can be a weighting factor, and the second weighted weight corresponding to the second rotation information can be the difference between 1 and the weighting factor. For instance, if the first weighted weight is w1, then the second weighted weight can be 1-w1. When the head-mounted display 110 is closer to the display screen 120, the second weighted weight used for the center marker can be higher, and the first weighted weight used for the corner marker can be lower. Conversely, when the head-mounted display 110 is farther from the display screen 120, the first weighted weight used for the corner marker can be higher, and the second weighted weight used for the center marker can be lower. In other words, the second weighted weight of the center marker is inversely related to the distance between the head-mounted display 110 and the display screen 120. The first weighted weight of the corner marker is directly related to the distance between the head-mounted display device 110 and the display screen 120. This is because when the head-mounted display device 110 is closer to the display screen 120, the rotation and displacement information corresponding to the corner marker becomes unstable.

In some embodiments, processor 113 can determine the first real-time pose information of display screen 120 based on any three of a plurality of preset markers. Further, processor 113 can calculate a normal vector of display screen 120 based on the displacement information (i.e., camera coordinate position in the camera coordinate system) of any three of the plurality of preset markers, and convert this normal vector into rotation information of display screen 120. Wherein, processor 113 can convert the image coordinates of a plurality of preset markers in the environmental image into camera coordinate positions based on the camera intrinsic and extrinsic parameters of image capturing device 111.

In some embodiments, processor 113 may perform marker identification on the environmental image to obtain multiple preset markers in the environmental image. Based on the marker identification result, processor 113 may select a target pose estimation algorithm from multiple candidate pose estimation algorithms. This target pose estimation algorithm is used to determine the real-time pose information of the display screen 120. Specifically, as described in the foregoing embodiments, processor 113 may determine the real-time pose information of the display screen 120 based on one or more preset markers in the environmental image. Therefore, processor 113 may use marker identification results to determine a suitable target pose estimation algorithm.

In this way, even if it is impossible to identify all the preset markers presented on the display screen 120 from the environmental image, such as due to marker damage or other factors, the processor 113 can still switch to different target pose estimation algorithms to determine the real-time pose information of the display screen 120. For example, when the processor 113 only identifies the center marker from the environmental image, the processor 113 can determine the real-time pose information of the display screen 120 based on the rotation and displacement information of the center marker. For example, when the processor 113 identifies the center marker and all corner markers from the environmental image, the processor 113 can determine the use of each corner marker to calculate the real-time pose information of the display screen 120 based on the distance between the head-mounted display device 110 and the display screen 120, and ignore the center marker.

In step S540, the processor 113 determines the target pose information of the display screen 120 in an augmented reality coordinate system based on the real-time pose information of the display screen 120 and the device pose information of the head-mounted display device 110. In this embodiment, step S540 can be implemented as steps S541 to S542.

In step S541, the processor 113 can convert the first real-time pose information under the camera coordinate system into the second real-time pose information under the augmented reality coordinate system based on the coordinate transformation relationship between the augmented reality coordinate system and the camera coordinate system. This coordinate transformation relationship can be established in advance and recorded in the storage device 112. This coordinate transformation relationship depends on the placement position of the image capturing device 111 on the head-mounted display device 110 and the shooting direction, and can be generated by prior measurement.

For example, Figure 7 is a schematic diagram of coordinate transformation according to an embodiment of the present invention. The origin of the camera coordinate system Cam_c1 is the location of the image capturing device 111, and the Z-axis of the camera coordinate system points in front of the capturing device 111. Additionally, the coordinate origin Ori1 of the augmented reality coordinate system AR_c1 is located at the visual center or scene center and can be determined by the user (e.g., by pressing the initialization setting button on the head-mounted display 110). The Z-axis of the augmented reality coordinate system AR_c1 can be opposite to the Z-axis of the camera coordinate system Cam_c1. In the example of Figure 7, the device center point HPC1 of the head-mounted display 110 coincides with the coordinate origin Ori1 of the augmented reality coordinate system AR_c1. There is a fixed displacement between the image capturing device 111 and the device center point HPC1 of the head-mounted display device 110. The processor 113 can perform coordinate transformation based on this fixed displacement and the rotation relationship between the augmented reality coordinate system AR_c1 and the camera coordinate system Cam_c1, so as to convert the first real-time pose information under the camera coordinate system Cam_c1 into the second real-time pose information under the augmented reality coordinate system AR_c1.

In step S542, the processor 113 can obtain target pose information of the display screen 120 relative to a coordinate origin of the augmented reality coordinate system based on the device pose information of the head-mounted display device 110 and the second real-time pose information of the display screen 120. Specifically, the second real-time pose information of the display screen 120 is the relative pose information of the display screen 120 to the head-mounted display device 110 in the augmented reality coordinate system. Therefore, the processor 113 can obtain target pose information of the display screen 120 relative to a coordinate origin of the augmented reality coordinate system based on the device pose information of the head-mounted display device 110 in the augmented reality coordinate system and the second real-time pose information of the display screen 120.

For example, Figure 8 is a schematic diagram of coordinate transformation according to an embodiment of the present invention. Referring to Figure 8, after the coordinate origin Ori1 of the augmented reality coordinate system AR_c1 is calibrated, the head-mounted display device 110 can move arbitrarily in space. Referring to Figures 7 and 8, it can be seen that the device center point HPC1 of the head-mounted display device 110 can be moved from the coordinate origin Ori1 to other positions. Therefore, after performing coordinate transformation to obtain the second real-time pose information of the display screen 120 under the augmented reality coordinate system AR_c1, the processor 113 can obtain the target pose information of the display screen 120 relative to the coordinate origin Ori1 of the augmented reality coordinate system AR_c1 based on the displacement and rotation information of the head-mounted display device 110 relative to the coordinate origin Ori1 of the augmented reality coordinate system AR_c1 (i.e., device pose information).

In some embodiments, the processor 113 may convert the first real-time displacement information of the display screen 120 in the camera coordinate system into target displacement information relative to a coordinate origin of the augmented reality coordinate system according to the following formulas (1) to (3). Formula (1) Formula (2) Formula (3) Where, The first real-time displacement information of the display screen 120 relative to the image capturing device 111 in the camera coordinate system; The fixed displacement between the center point of the image capturing device 111 and the head-mounted display device 110; The rotational quaternion representing the head-mounted display device 110; This represents the device displacement information of the head-mounted display 110 in the augmented reality coordinate system; This represents the target displacement information displayed on screen 120 in the augmented reality coordinate system.

In some embodiments, the processor 113 may convert the first real-time rotation information of the display screen 120 in the camera coordinate system into target rotation information relative to a coordinate origin of the augmented reality coordinate system according to the following formulas (4) to (5). Formula (4) Formula (5) Where, This represents the first real-time position rotation information of the display screen 120 in the camera coordinate system; This represents a rotation matrix that rotates the X-axis by 180 degrees. This represents the device rotation information of the head-mounted display 110 in the augmented reality coordinate system; This represents the target rotation information displayed on screen 120 in the augmented reality coordinate system.

Finally, in step S550, the processor 113 displays a virtual object anchored to the display screen 120 via the head-mounted display device 110 based on the target pose information of the display screen 120.

In summary, in the embodiments of the present invention, the real-time pose information of the display screen can be estimated based on preset markers presented on the display screen in the environmental image. The target pose information of the display screen in the augmented reality coordinate system can be determined based on the real-time pose information of the display screen and the device pose information of the head-mounted display device. Therefore, the display position of the virtual object can be determined based on the target pose information of the display screen, allowing the user to view a virtual object stably anchored on the display screen via the head-mounted display device. Based on this, the coordination accuracy and flexibility between virtual and real displays can be improved, achieving a smoother augmented reality experience. This can enhance the user experience of viewing virtual objects using head-mounted displays.

Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone with ordinary skill in the art may make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended patent application.

10A, 10B: Head-mounted display system 110: Head-mounted display device 111: Image capturing device 112: Storage device 113: Processor 120: Display screen S310～S340, S510～550, S611～S615: Steps V_obj1, V_obj2: Virtual objects 130: Computer device AR_c1: Augmented reality coordinate system Cam_c1: Camera coordinate system HPC1: Device center point M1～M4: Corner markers M5: Center marker Ori1: Coordinate origin

Figure 1A is a schematic diagram of a head-mounted display system according to an embodiment of the present invention. Figure 1B is a schematic diagram of a head-mounted display system according to an embodiment of the present invention. Figure 2 is an application scenario diagram of displaying virtual objects according to an embodiment of the present invention. Figure 3 is a flowchart of a method for displaying virtual objects according to an embodiment of the present invention. Figure 4 is a schematic diagram of a display screen and multiple preset markers according to an embodiment of the present invention. Figure 5 is a flowchart of a method for displaying virtual objects according to an embodiment of the present invention. Figure 6 is a flowchart of determining the real-time pose information of the display screen according to an embodiment of the present invention. Figure 7 is a schematic diagram of coordinate transformation according to an embodiment of the present invention. Figure 8 is a schematic diagram of coordinate transformation according to an embodiment of the present invention.

S310~S340: Steps

Claims (12)

A method for displaying a virtual object includes: capturing an environmental image toward a display screen using an image capture device on a head-mounted display device, wherein a plurality of preset markers are displayed on the display screen; determining real-time pose information of the display screen based on the plurality of preset markers in the environmental image; determining target pose information of the display screen in an augmented reality coordinate system based on the real-time pose information of the display screen and device pose information of the head-mounted display device; and displaying a virtual object anchored to the display screen via the head-mounted display device based on the target pose information of the display screen. The method for displaying a virtual object as described in claim 1 further includes: controlling the display screen to display the plurality of preset markers, wherein the plurality of preset markers includes a plurality of corner markers and a center marker, the plurality of corner markers being located at a plurality of screen corners of the display screen, and the center marker being located at the center of the display screen. The method for displaying a virtual object as described in claim 1, wherein the step of determining the real-time pose information of the display screen based on the plurality of preset markers in the environmental image comprises: calculating rotational and displacement information of at least one of the plurality of preset markers in the environmental image; and determining first real-time pose information of the display screen in the camera coordinate system of the image capturing device based on the rotational and displacement information of at least one of the plurality of preset markers. The method for displaying a virtual object as described in claim 3, wherein the step of determining the target pose information of the display screen in the augmented reality coordinate system based on the real-time pose information of the display screen and the device pose information of the head-mounted display device includes: converting the first real-time pose information in the camera coordinate system to second real-time pose information in the augmented reality coordinate system based on a coordinate transformation relationship between the augmented reality coordinate system and the camera coordinate system; and obtaining the target pose information of the display screen relative to a coordinate origin of the augmented reality coordinate system based on the device pose information of the head-mounted display device and the second real-time pose information of the display screen. The method for displaying a virtual object as described in claim 3, wherein the step of determining the first real-time pose information of the display screen in the camera coordinate system of the image capturing device based on rotation and displacement information of at least one of the plurality of preset markers includes: Determining the real-time displacement information and real-time rotation information in the first real-time pose information of the display screen based on the rotation and displacement information of a center marker among the plurality of preset markers. The method for displaying a virtual object as described in claim 3, wherein the step of determining the first real-time pose information of the display screen in the camera coordinate system of the image capturing device based on rotation and displacement information of at least one of the plurality of preset markers includes: performing a displacement averaging operation on the displacement information of a plurality of corner markers among the plurality of preset markers to obtain the real-time displacement information in the first real-time pose information; and performing a spherical linear interpolation operation on the rotation information of the plurality of corner markers among the plurality of preset markers to obtain the real-time rotation information in the first real-time pose information. The method for displaying a virtual object as described in claim 6, wherein the plurality of preset markers includes a plurality of corner markers and a center marker, and the step of determining the first real-time pose information of the display screen in the camera coordinate system of the image capturing device based on rotation and displacement information of at least one of the plurality of preset markers includes: Acquiring first rotation information and first displacement information based on the rotation and displacement information of the plurality of corner markers; Acquiring second rotation information and second displacement information based on the rotation and displacement information of the center marker; Performing a weighted sum operation on the first rotation information and the second rotation information according to a weighting factor to obtain the real-time rotation information in the first real-time pose information; and The first displacement information and the second displacement information are weighted and summed according to the weighting factor to obtain the real-time displacement information in the first real-time pose information. The method for displaying a virtual object as described in claim 7, wherein the weighting factor is determined based on the distance between the head-mounted display and the display screen. The method for displaying a virtual object as described in claim 1 further includes: performing a marker identification on the environmental image to obtain the plurality of preset markers in the environmental image; and selecting a target pose estimation algorithm from a plurality of candidate pose estimation algorithms based on the marker identification result, wherein the target pose estimation algorithm is used to determine the real-time pose information of the display screen. The method for displaying virtual objects as described in claim 1, wherein the plurality of preset markers includes a plurality of binary square markers (ArUco markers). A head-mounted display system includes: A head-mounted display device, including: An image capturing device for capturing an environmental image toward a display screen, wherein a plurality of preset markers are displayed on the display screen; A computer device connected to the head-mounted display device and including: A storage device; and A processor coupled to the storage device, configured to: Determine real-time pose information of the display screen based on the plurality of preset markers in the environmental image; Determine target pose information of the display screen in an augmented reality coordinate system based on the real-time pose information of the display screen and device pose information of the head-mounted display device; and Based on the target pose information displayed on the display screen, a virtual object anchored to the display screen is displayed via the head-mounted display device. A head-mounted display device includes: an image capturing device for capturing an environmental image toward a display screen, wherein a plurality of preset markers are displayed on the display screen; a storage device; and a processor coupled to the storage device and configured to: determine real-time pose information of the display screen based on the plurality of preset markers in the environmental image; determine target pose information of the display screen in an augmented reality coordinate system based on the real-time pose information of the display screen and device pose information of the head-mounted display device; and display a virtual object anchored to the display screen via the head-mounted display device based on the target pose information of the display screen.