JP2022034640A - Method and system for drawing virtual environment - Google Patents

Abstract

To provide a method and a system for drawing a virtual environment for drawing a plurality of contents in a virtual environment separately.SOLUTION: A method of drawing a virtual environment includes: acquiring multiple different digital contents in the same three-dimensional virtual environment; and drawing the digital contents in the three-dimensional virtual environment in each of multiple image frames based on position information of the digital contents. The position information includes a position of one of the digital contents in the three-dimensional virtual environment with respect to others, and is used to display the image frames and a blended result of the image frames.SELECTED DRAWING: Figure 2

Description

The present invention relates to a virtual environment simulation, and more particularly to a method and a system for drawing a virtual environment.

Techniques for sensation, perception, and / or environment simulation, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and XR (Extended Reality), have become increasingly popular these days. The technique can be applied to a plurality of fields such as games, military training, medical treatment, remote control, and the like.

Note that there is a lot of digital content, such as avatars, media content, and other virtual objects in virtual environments. However, some content may be copyrighted or confidential. These contents may not be shared with others. For example, digital rights management (DRM) encrypted video can only be played on a display with permission. In the conventional approach, the 3D scene in which the DRM-encrypted video is played is drawn directly to the DRM-encrypted image frame, which causes the display to display the DRM-encrypted image frame. When trying, a blank frame will appear on the display without permission. It may not be convenient for live broadcasting or content sharing.

3D scenes with DRM-encrypted video cannot be shared. Therefore, the present invention is directed to a method and system for drawing a virtual environment for separately drawing a plurality of contents in the virtual environment.

In one of the exemplary embodiments, the method of drawing the virtual environment includes, but is not limited to, the following steps. Multiple digital contents are acquired. These digital contents are different contents in the same three-dimensional virtual environment. The digital content is drawn in each of a plurality of image frames according to the position information of the digital content. The location information includes the location of one of the digital contents in the three-dimensional virtual environment with respect to the other. The image frame and the blended result of the image frame are the image frames used to be displayed.

In one of the exemplary embodiments, the system that renders the virtual environment includes, but is not limited to, a host display, memory, and a processor. The memory stores the program code. The processor is connected to the host display and memory and loads the program code to perform the next step. The processor acquires a plurality of digital contents and draws the digital contents in a plurality of image frames according to the position information of the digital contents. These digital contents are different contents in the same three-dimensional virtual environment. The position information includes the position of one 3D content in the 3D virtual environment with respect to the other. The image frame and the blended result of the image frame are the image frames used to be displayed on the host display.

In view of the above, according to the method and system for drawing the virtual environment provided in one or more embodiments, a plurality of digital contents in one three-dimensional virtual environment are drawn separately. Therefore, the drawn frames or blend results can be displayed separately due to different requirements.

However, it should be understood that this overview does not include all aspects and embodiments of the invention and is not intended to be limiting or limiting in any way. The inventions disclosed herein are then understood by those of skill in the art to include their apparent improvements and modifications.

Attachments are included for further understanding of the invention, are incorporated herein and constitute part of this specification. The drawings represent embodiments of the invention and, together with the specification, serve to explain the principles of the invention.
It is a block diagram which shows the system by one of the exemplary embodiments of this invention. It is a block diagram which shows the software structure by one of the exemplary embodiments of this invention. It is a flow diagram which shows the method of drawing the virtual environment by one of the exemplary embodiments of this invention. It is a schematic diagram which shows the engine structure by one of the exemplary embodiments of this invention. It is a schematic diagram which shows the other engine structure by one of the exemplary embodiments of this invention. It is a flow diagram which shows the method of sharing the content by one of the exemplary embodiments of this invention.

Details of preferred embodiments of the present invention will be described. Examples are shown in the accompanying drawings. Wherever possible, the same reference numerals are used for the same or similar members in the drawings and the specification.

FIG. 1A is a block diagram showing a system 1 according to one of the exemplary embodiments of the present invention. With reference to FIG. 1A, system 1 includes, but is not limited to, one or more client devices 10, a server 30, and a host device 100. System 1 is adapted for VR, AR, MR, XR or other reality simulation related techniques.

The client device 10 is a mobile phone, screen, television, monitor, or other electronic device having a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or another display. Good, but not limited to this. In some embodiments, the client device 10 is used to process a multimedia data image, for example, an image in a virtual environment, a video loaded from the cloud, or a photo transmitted by a server 30.

The server 30 may be a cloud server, a streaming server, a personal computer, or the like. In some embodiments, the server 30 is used to provide a video stream, which allows the video to be played back on the client device 10.

The host device 100 includes, but is not limited to, a host display 110, a motion sensor 130, a memory 150, and a processor 170. The host device 100 can be, but is not limited to, a head mounted display (HMD) or digital eyeglasses compatible with VR, AR, MR, XR applications.

The host display 110 may be an LCD, an LED display, an OLED display, or another display. In one embodiment, the host display 110 is used to display data, for example, an image, a virtual reality image, a video, or a photograph transmitted from a processor 170. In one embodiment, the host display 110 may be incorporated into the main unit. In some embodiments, the host display 110 may be the display of an external device (such as a smartphone, tablet, etc.), which can be placed on the body of the HMD.

The motion sensor 130 may be an accelerometer, a gyroscope, a magnetic field meter, a laser sensor, an inertial measurement unit (IMU), an infrared (IR) sensor, an image sensor, a depth camera, or any combination of the above sensors. In one embodiment, the motion sensor 130 generates motion sensing data from the sensing results (eg, camera image, sensing intensity value, etc.) of the motion sensor 130, and thus senses the movement of the host device 100 or the system 1. Used. As an example, motion sensing data includes 3-DoF (Degree of Freedom) data, and 3-DoF data is the user's head in three-dimensional (3D) space such as yaw, roll, and pitch acceleration. It is related to the rotation data of the part. As another example, motion sensing data includes 6-DoF data. Compared to 3-DoF data, 6-DoF data is further related to the displacement of the user's head in three vertical axes: surge, heave, and sway acceleration. As another example, the motion sensing data includes the relative position and / or displacement of the user's head in 2D / 3D space. In some embodiments, the motion sensor 130 can be integrated into a handheld controller or wearable device connected to the host device 100, or can be integrated into the host device 100.

The memory 150 may be of any sort of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, similar devices, or a combination of the above devices. The memory 150 records program code, device configurations, buffer data, or persistent data (such as motion-sensing data, 3D content, or image frames), which will be introduced later.

The processor 170 is connected to the host display 110, the motion sensor 130, and the memory 150. The processor 170 is configured to load the program code stored in the memory 150 in order to perform the procedure of the exemplary embodiment of the invention.

In some embodiments, the processor 170 may be a central processing unit (CPU), a microprocessor, a microcontroller, an image processing unit (GPU), a digital signal processing (DSP) chip, a field programmable gate array (FPGA). .. The function of the processor 170 may be implemented by an independent electronic device or an integrated circuit (IC), and the operation of the processor 170 may be implemented by software.

FIG. 1B is a block diagram showing a software structure according to one of the exemplary embodiments of the present invention. The processor 170 loads the corresponding program code and executes one or more drawing engines 171 and the synthesis plug-in 173. The drawing engine 171 executes a three-dimensional drawing process, which is a three-dimensional computer graphics process for converting a three-dimensional model into a two-dimensional image. In some embodiments, the 2D image transformed by the drawing engine 171 includes location information in a 3D virtual environment. The compositing plug-in 173 blends or combines two or more image frames output from the drawing engine 171.

In one embodiment, the host device 100 includes a host display 110, a motion sensor 130, a memory 150, and a processor 170, via a wired connection, a wireless connection, or any type of communication capable of transmitting data. Communicates with the server 30 or the client device 10. In some embodiments, the processor 170 may not be provided in the same device along with the host display 110 and the motion sensor 130. However, since the devices equipped with the host display 110, the motion sensor 130, and the processor 170 each transmit and receive data to and from each other, they are communication transceivers or physical devices having compatible communication technologies such as Bluetooth, Wi-Fi, and infrared wireless communication. Transmission line may be further included. For example, the motion sensor 130 may be provided outside the host device 100, whereas the host display 110 and the processor 170 may be provided in the host device 100. As another example, the host display 110 and the motion sensor 130 may be provided outside the arithmetic unit, whereas the processor 170 may be provided inside the arithmetic unit.

In order to make it easier to understand the operation processing provided in one or more embodiments of the present invention, some embodiments are illustrated below in order to detail the operation processing of the system 1. The devices and modules in the system 1 are applied in the following embodiments to illustrate how to render the virtual environment provided herein. Each step of the method can be adjusted according to the actual mounting situation and should not be limited to those described herein.

FIG. 2 is a flow chart showing a method of drawing a virtual environment according to one of the exemplary embodiments of the present invention. With reference to FIG. 2, the processor 170 may acquire a plurality of digital contents (step S210). Specifically, the digital content may be a real or virtual 3D scene, avatar, video, photo, or other virtual object in a 3D virtual environment. The video or photographic file may be decoded by a decoder to generate the corresponding surface texture in a 3D virtual environment. The three-dimensional virtual environment may be a game environment, a virtual social environment, or a virtual conference. In one embodiment, the digital content includes a first content without access control protection and a second content with access control protection. Access control protection relates to access control of digital content, such as digital rights management (DRM) or other copyright related management techniques. Access control of digital content may be related to age classification, member classification, management classification, or other type of access control management. In some embodiments, the second content with access control protection is presented on the host display 110 but not on the client device 10. For example, if the second content is not allowed to be displayed on the client device 10, a blank or all black frame is presented on the client device 10. In one embodiment, the first content is a three-dimensional scene and the second content is a video with access control protection. In some embodiments, the first content and the second content may be any object or 3D model in the same 3D virtual environment.

The processor 170 may draw the digital content in each of a plurality of image frames (step S230). Specifically, in order to present the three-dimensional virtual environment on the client device 10 or the host display 110, the processor 170 may execute the three-dimensional drawing process. The 3D drawing process is a 3D computer graphics process that converts a 3D model into a 2D image. The 3D drawing process may include, for example, an application stage, a geometry processing stage, and a rasterization stage. At the application stage, the scene can be modified as needed, for example through user interaction through input devices or between animations. In the geometry transformation stage, processing of polygons and their vertices is performed. Geometry processing stages are, for example, vertex shading, projection, clipping, screen mapping. In the rasterization stage, the discrete fragments that are grid points are created from the continuous primitives output from the geometry processing stage.

In one embodiment, the processor 170 may generate an image frame according to the motion sensing data of the host device 100 or the system 1 acquired from the motion sensor 130. In the geometry processing stage, a viewer representing the user in the 3D virtual environment is defined. The processor 170 uses the motion sensing data of the host device 100 to determine the position and direction of the viewer's field of view from which the scene is drawn. For example, when the user's head moves or rotates, the three-dimensional scene of the field of view may change, and the image frame output from the drawing process also changes with the movement or rotation of the host device 100 or the system 1. good.

In one embodiment, one or more 3D drawing engines are provided by the processor 170 to draw digital content. In some embodiments, one master 3D drawing engine and one or more slave 3D drawing engines are provided. The master 3D drawing engine is used to draw the first content without access control protection, and the slave 3D drawing engine is used to draw the second content with access control protection. For example, the master 3D drawing engine draws a 3D scene in an image frame, and the slave 3D drawing engine draws an image having access control protection in the image frame.

FIG. 3 is a schematic diagram showing an engine structure 3 according to one of the exemplary embodiments of the present invention. Referring to FIG. 3, the engine structure 3 includes a master 3D drawing engine 31 and one or more slave 3D drawing engines 331 to 33n, where n is a positive number and the slave 3D drawing engines 331 to 33n. Is the quantity of. If the second content with access control protection does not exist in the 3D virtual environment, the master 3D drawing engine only draws the first content without access control protection in the image frame, and the output image frame is It is stored in the frame buffer 35. When the second content having access control protection exists in the three-dimensional virtual environment, the master three-dimensional drawing engine 31 may transmit the second content to the slave three-dimensional drawing engine 331. The master 3D drawing engine 31 and the plurality of slave 3D drawing engines 331 to 33n may draw digital contents, respectively. In some embodiments, different 3D drawing engines 331-33n may draw digital content in different formats, different protection protocols, or different resolutions. Based on the performance and functionality of the 3D drawing engines 331-33n, one or more second contents may be sequentially transmitted to the corresponding 3D drawing engine, whereby each 3D drawing engine 331-33n is 3D. The corresponding second content is drawn according to the performance and function of the drawing engines 331 to 33n. That is, the second content may be transmitted from the three-dimensional drawing engine 331 to the three-dimensional drawing engine 332. Next, the final 3D drawing engine 33n outputs the image frames of all the second contents to the frame buffer 35.

FIG. 4 is a schematic diagram showing another engine structure 4 according to one of the exemplary embodiments of the present invention. Referring to FIG. 4, the engine structure 4 includes a master 3D drawing engine 41, a plurality of slave 3D drawing engines 431 to 43m, and a plurality of frame buffers 45, where m is a positive number and the slave 3 It is a quantity of the dimension drawing engine 431 to 43 m. The difference between the engine structures 3 and 4 is that each slave 3D drawing engine 431 to 43m outputs an image frame to the corresponding frame buffer 45.

In one embodiment, the image frames corresponding to the different digital contents output from the frame buffer 45 may be displayed on the client device 10 or the host display 110. The processor 170 may upload an image frame to the server 30 to provide video streaming to the client device 10. For the HMD, the processor 170 may further generate image frames for both eyes of the user and warp the image frames for both eyes.

Since the plurality of digital contents in the same 3D virtual environment are drawn separately, the processor 170 can determine the image frame for the specific digital content shared or displayed on the specific display. In some embodiments, the processor 170 may determine whether to draw the plurality of digital contents separately based on the type of digital content, content sharing requirements, or other design requirements. For example, when one digital content is of the type encrypted by DRM, the processor 170 may determine that a plurality of digital contents in the same three-dimensional virtual environment are drawn separately. In another example, if one digital content is requested to be shared with the client device 10, the processor 170 may determine to draw a plurality of digital contents separately in the same 3D virtual environment. However, the processor 170 can also consider a plurality of digital contents in the same three-dimensional virtual environment as one content, and can draw all the digital contents together in the image frame.

In addition to presenting image frames corresponding to a plurality of digital contents separately, the processor 170 blends the image frames corresponding to the plurality of digital contents according to the position information of the digital contents in the three-dimensional virtual environment. Good (step S250). Specifically, the position information includes the absolute position of the digital content or the relative position of the digital content with respect to others in the three-dimensional virtual environment. In the projection shadow of the geometry processing stage, a plurality of 3D objects are projected onto a 2D plane. The depth information of these three-dimensional objects with respect to the viewer in the three-dimensional virtual environment included in the position information of the digital contents is necessary for projection. When two digital contents overlap in an image frame, the processor 170 uses the depth information of these digital contents via a synthesis plug-in 173 to determine which part of the digital content should be presented in the image frame. good. When the digital contents do not overlap in the image frame, the processor 170 supports all the digital contents based on the left-right relationship of these digital contents to the viewer contained in the position information of the digital contents via the synthesis plug-in 173. You may directly combine the image frames to be used. In some embodiments, meshes, shaders, transformations, and other data needed to draw the digital content may be used in the combination of image frames corresponding to the plurality of digital contents. Similarly, the blending result of the image frames corresponding to the plurality of digital contents may be displayed on the client device 10 and / or the host display 110. In addition, the processor 170 may further generate a blended image frame for both eyes of the user and warp the image frame for both eyes.

Below are scenarios for sharing digital content. FIG. 5 is a flow chart showing a method of sharing content according to one of the exemplary embodiments of the present invention. With reference to FIG. 5, the processor 170 determines whether or not to share digital content with the client device 10 (step S510). For example, sharing digital content requires presentation or live broadcast requirements. In addition, the digital content includes a first content without access control protection and a second content with access control protection. If the digital content is not shared, the processor 170 may consider all the digital content as one content and draw all the digital content directly into the image frame without the blending process in step S250 (step S520) and the drawing result will be. It can be displayed on the host display 110 (step S560). It is assumed that YouTuber holds the host display 110 and another client sees the client device 10. In some embodiments, in the absence of a second content with access control protection, the processor 170 simply displays only the first content on the host display 110 without blending image frames. It may be drawn in the image frame used.

On the other hand, if the digital content is to be shared, the processor 170 has access control protection with the first frame (ie, unprotected frame) of the image frame corresponding to the first content without access control protection. In order to generate the second frame (that is, the protection frame) of the image frame corresponding to the second content, the first and second contents may be drawn respectively (step S530). The processor 170 does not transmit the second frame of the image frame corresponding to the second content having access control protection, but rather the first frame of the image frame corresponding to the first content without access control protection (i.e., , Unprotected frame) may be transmitted to the client device 10. In this embodiment, the unprotected frame corresponding to the first content is uploaded to the server 30 (step S540), and then the unprotected frame can be presented on the client device 10 by streaming (step S550). On the other hand, the processor 170 blends the image frames corresponding to the first and second contents as in step S250 via the synthesis plug-in 173 (step S545). The blended result of the unprotected frame and the protected frame can then be displayed on the host display 110 (step S560).

In summary, the above exemplary embodiments depict methods and systems for drawing virtual environments. Multiple digital contents in the same virtual environment can be drawn separately. In addition, image frames corresponding to a plurality of digital contents can be combined based on the position information of the digital contents in the virtual environment. Therefore, drawing results of different digital contents can be presented on different displays. If one digital content has access control protection, the other unprotected digital content can be shared with other displays.

It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the invention without departing from the scope or spirit of the invention. In view of the above, the present invention is intended to cover modifications and variations of the present invention, provided that they are within the scope of the following claims and their equivalents.

The method and system for drawing a virtual environment of the present invention can be applied to digital content management.

1: System 10: Client device 30: Server 100: Host device 110: Host display 130: Motion sensor 150: Memory 170: Processor 171: Drawing engine 173: Synthetic plug-ins S210 to S250, S510 to S560: Steps 3, 4: Engine structure 31, 41: Master 3D drawing engine 311 to 31n, 431 to 43m: Slave 3D drawing engine 35, 45: Frame buffer

Claims (9)

Acquiring a plurality of digital contents, that the digital contents are different digital contents in the same three-dimensional virtual environment.
Based on the position information of the digital content, the digital content in the three-dimensional virtual environment is drawn in a plurality of image frames, and the position information is one of the digital contents in the three-dimensional virtual environment. Includes a position relative to one other, including that the image frame and the blended result of the image frame are used to be displayed.
How to draw a virtual environment. With the host display
Memory for storing program code and
Connected to the host display and the memory,
Acquiring a plurality of digital contents, that the digital contents are different digital contents in the same three-dimensional virtual environment.
Based on the position information of the digital content, the digital content in the three-dimensional virtual environment is drawn in a plurality of image frames, and the position information is one of the digital contents in the three-dimensional virtual environment. A processor that loads the program code to execute that the image frame and the blended result of the image frame are used to be displayed on the host display, including the position relative to the other.
including,
A system that draws a virtual environment. The digital content includes a first content without access control protection and a second content with access control protection.
The method for drawing a virtual environment according to claim 1, or a system for drawing a virtual environment according to claim 2. The step of drawing the digital content in the image frame is
In response to the existence of the second content having the access control protection, drawing the digital content in the image frame and drawing the digital content in the image frame, respectively.
After the step of drawing the digital content in the image frame, further comprising blending the image frame according to the position information of the digital content in the three-dimensional virtual environment.
The method for drawing the virtual environment according to claim 3, or
The processor
In response to the existence of the second content having the access control protection, drawing the digital content in the image frame and drawing the digital content in the image frame, respectively.
After the step of drawing the digital content in the image frame, further performing blending of the image frame according to the position information of the digital content in the three-dimensional virtual environment.
A system for drawing the virtual environment according to claim 3. The step of drawing the digital content in the image frame is
In response to the absence of the second content having access control protection, the first content is only drawn in the image frame used to be displayed without blending the image frames. Including being
The method for drawing the virtual environment according to claim 3, or
The processor
The image frame used to display the first content on the host display without blending the image frames in response to the absence of the second content having access control protection. Do more than just draw to,
A system for drawing the virtual environment according to claim 3. After the step of drawing the digital content in the image frame
By transmitting the first frame of the image frame without transmitting the second frame of the image frame, the first frame corresponds to the first content without the access control protection. The second frame further comprises corresponding to the second content having the access control protection.
The method for drawing the virtual environment according to claim 4, or
Including more client devices
The processor
By transmitting the first frame of the image frame to the client device without transmitting the second frame of the image frame, the first frame is the first frame without the access control protection. Corresponding to the content and further performing that the second frame corresponds to the second content having the access control protection.
A system for drawing the virtual environment according to claim 4. The step of drawing the digital content in the image frame is
To provide a master 3D drawing engine and at least one slave 3D drawing engine, the master 3D drawing engine is used to draw the first content without the access control protection. The inclusion that the at least one slave 3D drawing engine is used to draw the second content having the access control protection.
The method for drawing the virtual environment according to claim 4, or
The processor
To provide a master 3D drawing engine and at least one slave 3D drawing engine, the master 3D drawing engine is used to draw the first content without the access control protection. Further performing that the at least one slave 3D drawing engine is used to draw the second content having the access control protection.
A system for drawing the virtual environment according to claim 4. The master 3D drawing engine transmits the second content having the access control protection to the at least one slave 3D drawing engine.
Further comprising blending the image frame output from the master 3D drawing engine and the at least one slave 3D drawing engine.
The method for drawing the virtual environment according to claim 7, or
The processor
The master 3D drawing engine transmits the second content having the access control protection to the at least one slave 3D drawing engine.
Further performing blending of the image frame output from the master 3D drawing engine and the at least one slave 3D drawing engine.
A system for drawing the virtual environment according to claim 7. The step of drawing the digital content in the image frame is
Acquiring motion detection data of the user's head and
Including generating the image frame in response to the motion sensing data of the user's head.
The method of drawing the virtual environment according to claim 1, or
It also includes a motion sensor that works with the user's head and acquires motion sensing data for the user's head.
The processor
Further performing the generation of the image frame according to the motion sensing data of the user's head.
A system for drawing the virtual environment according to claim 2.

Images