TWI817875B - Electronic device and method of generating 3d image based on multi-thread - Google Patents

Abstract

An electronic device and a method of generating 3D image based on multi-thread are provided. The method includes: operating a first thread and a second thread by a processor; transmitting an image frame of 3D image to a first buffer in a storage medium by the first thread, wherein the image frame includes a 3D mesh and a texture corresponding to the 3D mesh; in response to the first buffer completing a reception of the image frame, loading the image frame in the first buffer to perform a depth estimation for the image frame by the second thread so as to generate depth information; updating the 3D mesh according to the depth information by the first thread; and generating a side-by-side image according to the updated 3D mesh and the texture by the first thread.

Description

Electronic device and method for generating three-dimensional images based on multi-threading

The present invention relates to an image processing technology, and in particular to an electronic device and method for generating three-dimensional images based on multiple threads.

With the development of machine learning technology, more and more displays are beginning to use machine learning technology to process images to provide users with a diverse viewing experience. For example, displays that support 3D image playback technology can use machine learning technology to instantly convert 2D images into 3D images. However, applying machine learning-based image processing technology may consume a lot of computing time. These calculation times may cause the output image during playback to be out of sync with the image processing results, resulting in a decrease in the quality of the output image.

The present invention provides an electronic device and method for generating three-dimensional images based on multi-threading, which can optimize the reading of graphic resources and depth prediction models.

The present invention is an electronic device that generates three-dimensional images based on multi-threading, including a processor and a storage medium. The storage medium includes a first buffer. The processor is coupled to the storage medium and runs a first execution thread and a second execution thread, wherein the first execution thread transmits an image frame of the three-dimensional image to the first buffer, wherein the image frame includes a three-dimensional grid and a frame corresponding to the three-dimensional grid. Texture; in response to the first buffer completing the reception of the image frame, the second execution thread reads the image frame in the first buffer to perform depth prediction on the image frame, thereby generating depth information; the first execution thread updates the three-dimensional image frame according to the depth information grid; and the first thread generates side-by-side images based on the updated three-dimensional grid and texture.

In an embodiment of the present invention, the above-mentioned storage medium further includes a second buffer, wherein the second thread transmits the depth information to the second buffer; and in response to the second buffer completing the reception of the depth information, the first The thread reads the depth information in the second buffer.

In an embodiment of the present invention, in response to the first buffer completing the reception of the image frame, the processor transmits an identification code corresponding to the image frame to the second execution thread; and in response to the identification code, the second execution thread reads Image frames in the first buffer.

In an embodiment of the present invention, in response to the second buffer completing the reception of the depth information, the processor transmits an identification code corresponding to the image frame to the first execution thread; and in response to the identification code, the first execution thread reads Depth information in the second buffer.

In an embodiment of the present invention, the above-mentioned electronic device further includes an image capturing device. The image capture device is coupled to the processor, wherein the first execution thread is configured to: perform eye tracking through the image capture device to obtain the field of view; project the field of view to the updated three-dimensional grid to obtain the region of interest; map the texture to region of interest to generate an output frame; and generate side-by-side images based on the output frame.

In an embodiment of the present invention, the above-mentioned first thread maps the texture to the updated three-dimensional mesh to generate side-by-side images.

In an embodiment of the present invention, the above-mentioned electronic device further includes a transceiver. The transceiver is coupled to the processor, wherein the first thread performs a knitting operation on the side-by-side images to generate a knitted image, and outputs the knitted image through the transceiver.

In an embodiment of the present invention, the above-mentioned second thread inputs image frames to the machine learning model to perform depth prediction.

The present invention provides a method for generating a three-dimensional image based on multiple threads, including: a processor running a first thread and a second thread; and the first thread transmitting image frames of the three-dimensional image to a first buffer in a storage medium. The image frame includes a three-dimensional grid and a texture corresponding to the three-dimensional grid; in response to the first buffer completing the reception of the image frame, the second execution thread reads the image frame in the first buffer to execute the image frame depth prediction, thereby generating depth information; the first execution thread updates the three-dimensional grid according to the depth information; and the first execution thread generates side-by-side images according to the updated three-dimensional grid and texture.

Based on the above, the electronic device of the present invention can trigger another thread to perform subsequent image processing steps based on the complete data in the buffer after one thread completes receiving the image frame or depth information through the buffer. Accordingly, the present invention can improve the buffer access speed and the synchronization between the image processing process and the three-dimensional image generation process.

In order to make the content of the present invention easier to understand, the following embodiments are given as examples according to which the present invention can be implemented. In addition, wherever possible, elements/components/steps with the same reference numbers in the drawings and embodiments represent the same or similar parts.

FIG. 1 is a schematic diagram of an electronic device 100 according to an embodiment of the invention. The electronic device 100 can generate three-dimensional images based on multiple threads. The electronic device 100 may include a processor 110, a storage medium 120, and a transceiver 130. In an embodiment, the electronic device 100 may further include an image capturing device 140 .

The processor 110 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose micro control unit (MCU), microprocessor, or digital signal processing unit. Digital signal processor (DSP), programmable controller, application specific integrated circuit (ASIC), graphics processing unit (GPU), image signal processor (ISP) ), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field programmable gate array (field programmable gate array) , FPGA) or other similar components or a combination of the above components. The processor 110 can be coupled to the storage medium 120, the transceiver 130 and the image capture device 140, and access and execute multiple modules and various applications stored in the storage medium 120. Processor 110 may have multi-threading capabilities. In this embodiment, the processor 110 can run the thread 111 and the thread 112.

The storage medium 120 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), or flash memory. , hard disk drive (HDD), solid state drive (SSD) or similar components or a combination of the above components, used to store multiple modules or various application programs that can be executed by the processor 110 . In this embodiment, the storage medium 120 can store multiple modules including the buffer 121, the buffer 122, the buffer manager 123, and the depth prediction model 124, the functions of which will be described later.

The transceiver 130 transmits and receives signals in a wireless or wired manner. Transceiver 130 may also perform, for example, low noise amplification, impedance matching, mixing, up or down frequency conversion, filtering, amplification, and similar operations.

The image capturing device 140 includes, for example, an image sensor such as a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor. The image capture device 140 can be used to capture images.

FIG. 2 is a schematic diagram of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention, where the method can be implemented by the electronic device 100 shown in FIG. 1 . The method may include task 200 performed by thread 111 and task 300 performed by thread 112 .

In step S210, the execution thread 111 may obtain the image frame of the three-dimensional image. Specifically, the three-dimensional image may include multiple image frames, and each image frame may include a three-dimensional mesh (3D mesh) and a texture corresponding to the three-dimensional mesh. The execution thread 111 may, for example, receive image frames of three-dimensional images from an external device through the transceiver 130 . The acquisition of the image frame may trigger the execution thread 111 to start executing the task 200 .

In step S220, if necessary, the thread 111 may perform resizing on the image frame. For example, assume that the resolution of the output image (or output frame) to be generated by the electronic device 100 is 1920x1080. The execution thread 111 may adjust the resolution of the image frame to be the same as the output image according to the resolution of the output image, that is, adjust the resolution to 1920x1080.

In step S230, the execution thread 111 may transmit the image frame to the buffer 121 for storage. The buffer 121 may send a confirmation message to the buffer manager 123 after the reception of the image frame is completed. The confirmation message may include but is not limited to an identification code corresponding to the image frame. For example, the confirmation message may include a texture identifier corresponding to the texture of the image frame received by the buffer 121 .

In step S310, the buffer manager 123 may send a confirmation message to the thread 112. The execution thread 112 can read the image frame stored in the buffer 121 according to the confirmation message. The confirmation message may include but is not limited to an identification code corresponding to the image frame. For example, the acknowledgment message may include a texture identification code corresponding to the texture of the image frame received by the buffer 121 . The execution thread 112 may read the image frame corresponding to the texture identification code from the buffer 121 according to the texture identification code.

In step S320, the thread 112 may perform preprocessing on the image frame. For example, the processor 110 may adjust the size of the image frame so that the image frame conforms to the input format of the depth prediction model 124 .

After completing the pre-processing of the image frame, in step S330, the execution thread 112 may perform depth estimation (depth estimation) on the image frame to generate depth information. Specifically, the execution thread 112 may input the image frame to the depth prediction model 124, so that the depth prediction model 124 outputs depth information corresponding to the image frame. The deep prediction model 124 may include, but is not limited to, a supervised machine learning model or an unsupervised machine learning model.

In step S340, the execution thread 112 may transmit the depth information to the buffer 122 for storage. The buffer 122 may send a confirmation message to the buffer manager 123 after completion of receiving the depth information. The confirmation message may include but is not limited to an identification code corresponding to the image frame. For example, the acknowledgment message may include a texture identification code corresponding to the texture of the image frame received by the buffer 121 .

In step S240, the buffer manager 123 may send a confirmation message to the thread 111. The confirmation message may include but is not limited to an identification code corresponding to the image frame (for example, a texture identification code). The acknowledgment message is used to notify the execution thread 111 that the data reception of the buffer 121 and the buffer 122 has been completed.

In step S250 , the execution thread 111 may update the three-dimensional grid of the image frame according to the depth information, and then generate a side-by-side (SBS) image based on the updated three-dimensional grid and texture. Specifically, step S250 may include steps S251 to S254. In step S251, the execution thread 111 may read the depth information stored in the buffer 122 according to the confirmation message, and update the three-dimensional grid according to the depth information.

In step S252, the execution thread 111 may generate a region of interest (ROI) according to the updated three-dimensional grid. The region of interest may be related to the field of view (FOV) of the viewer who is viewing the three-dimensional image. Specifically, in step S410 , the execution thread 111 may perform eye tracking through the image capture device 140 to obtain the viewer's field of view. In step S252, the thread 111 may project the view to the updated three-dimensional grid to obtain the region of interest. In other words, the region of interest of the image frame may change as the viewer's field of view changes.

In step S253, the thread 111 may map the texture corresponding to the three-dimensional grid to the region of interest corresponding to the three-dimensional grid to generate an output frame. The output frame is the result of the image frame being improved by depth information. In step S254, the execution thread 111 may generate side-by-side images according to the output frames.

In step S260 , the execution thread 111 may perform a weaving operation on the side-by-side images to generate a weaving image, and may output the weaving image through the transceiver 130 . For example, the execution thread 111 can transmit the knitting image to a display that supports three-dimensional images through the transceiver 130, so that the knitting image can be played through the display.

FIG. 3 illustrates a flowchart of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention, wherein the method can be implemented by the electronic device 100 shown in FIG. 1 . In step S510, the first execution thread and the second execution thread are executed by the processor. In step S520, the first thread transmits the image frame of the three-dimensional image to the first buffer in the storage medium, where the image frame includes the three-dimensional grid and the texture corresponding to the three-dimensional grid. In step S530, in response to the first buffer completing the reception of the image frame, the second thread reads the image frame in the first buffer to perform depth prediction on the image frame, thereby generating depth information. In step S540, the first execution thread updates the three-dimensional grid according to the depth information. In step S550, the first execution thread generates side-by-side images according to the updated three-dimensional mesh and texture.

In summary, the electronic device of the present invention can utilize multiple threads to read graphics resources, read image processing models, and generate three-dimensional images. After one thread completes receiving the image frame or depth information through the buffer, the buffer manager can trigger another thread to perform subsequent image processing steps based on the complete data in the buffer. Accordingly, the present invention can improve the buffer access speed and the synchronization between the image processing process and the three-dimensional image generation process. The present invention can also optimize the reading of graphics resources and image processing models.

100: Electronic devices

110: Processor

111, 112: Execution thread

120:Storage media

121, 122: Buffer

123:Buffer Manager

124: Depth prediction model

130:Transceiver

140:Image capture device

200, 300: Mission

S210, S220, S230, S240, S250, S251, S252, S253, S254, S260, S310, S320, S330, S340, S410, S510, S520, S530, S540, S550: Steps

FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the invention. FIG. 2 illustrates a schematic diagram of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention. FIG. 3 illustrates a flowchart of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention.

S510, S520, S530, S540, S550: Steps

Claims (8)

An electronic device that generates three-dimensional images based on multiple threads, including: a storage medium, including a first buffer; and a processor, coupled to the storage medium and running a first execution thread and a second execution thread, wherein the first execution thread The execution thread transmits an image frame of a three-dimensional image to the first buffer, wherein the image frame includes a three-dimensional grid and a texture corresponding to the three-dimensional grid; in response to the first buffer completing the image frame The second thread reads the image frame in the first buffer and inputs the image frame to the machine learning model to perform depth prediction on the image frame, thereby generating depth information; The first execution thread updates the three-dimensional mesh according to the depth information; and the first execution thread generates side-by-side images according to the updated three-dimensional mesh and the texture. The electronic device of claim 1, wherein the storage medium further includes a second buffer, wherein the second thread transmits the depth information to the second buffer; and in response to the second The buffer completes receiving the depth information, and the first thread reads the depth information in the second buffer. The electronic device of claim 1, wherein in response to the first buffer completing the reception of the image frame, the processor transmits an identification code corresponding to the image frame to the second execution thread. ;as well as In response to the identification code, the second thread reads the image frame in the first buffer. The electronic device of claim 2, wherein in response to the second buffer completing the reception of the depth information, the processor transmits an identification code corresponding to the image frame to the first execution thread. ; And in response to the identification code, the first thread reads the depth information in the second buffer. The electronic device of claim 1, further comprising: an image capture device coupled to the processor, wherein the first execution thread is configured to: perform eye tracking through the image capture device to obtain the field of view; Project the field of view to the updated three-dimensional grid to obtain a region of interest; map the texture to the region of interest to generate an output frame; and generate the side-by-side image based on the output frame. The electronic device of claim 1, wherein the first thread maps the texture to the updated three-dimensional grid to generate the side-by-side image. The electronic device of claim 1, further comprising: a transceiver coupled to the processor, wherein the first thread performs a knitting operation on the side-by-side images to generate a knitted image, and outputs it through the transceiver The weaving image. A method for generating three-dimensional images based on multi-threading, including: The processor runs a first thread and a second thread; the first thread transmits an image frame of the three-dimensional image to a first buffer in the storage medium, wherein the image frame includes a three-dimensional grid and a corresponding The texture of the three-dimensional grid; in response to the first buffer completing the reception of the image frame, the second thread reads the image frame in the first buffer and transfers the image Frames are input to a machine learning model to perform depth prediction on the image frames to generate depth information; the first execution thread updates the three-dimensional grid according to the depth information; and the first execution thread updates the three-dimensional grid according to the depth information. The updated three-dimensional mesh and texture produce side-by-side images.

Images