WO2025120714A1 - Content server, client terminal, image display system, display data transmission method, and display image generation method - Google Patents

Abstract

This content server 20 acquires neural networks 230a, 23b,... representing information about a plurality of types of 3D scenes by generating a learning image representing each scene in a display world and performing machine learning. The content server 20 divides the neural networks 230a, 23b,... to generate a plurality of neural networks 232a, 232b, etc., randomly switches the order of packets, and transmits the packets to a client terminal 10. The client terminal 10 draws a display image 238 corresponding to the most recent display viewpoint by, for example, returning the divided neural networks 232a, 232b, etc. to the original neural network 230a.

Description

Content server, client terminal, image display system, display data transmission method, and display image generation method

This invention relates to a content server, a client terminal, an image display system, a display data transmission method, and a display image generation method that display images of a three-dimensional display world.

　With the recent expansion of communication networks and developments in image processing technology, it has become possible to enjoy a wide variety of electronic content regardless of the viewing environment. For example, in the field of electronic games, a system has become widespread in which a server collects operation information entered into each client terminal and distributes game images that reflect this information as needed, allowing multiple players to participate in the same game regardless of location.

Not limited to electronic games, electronic content in which video images generated in real time in response to user operations are distributed from a server can utilize the server's abundant processing environment, making it easier to display high-quality images while minimizing the impact of the client terminal's processing performance. On the other hand, the constant transmission of operation information from the client terminal and the data transmission processing from the server that receives it can cause the display to be unable to keep up with viewpoint operations or images to be lost due to packet loss, resulting in a deterioration in the quality of the user experience.

The present invention was made in consideration of these problems, and its purpose is to provide a technology that reduces the impact of distribution on the quality of the user experience when processing images of content that is distributed from a server.

In order to solve the above problem, one aspect of the present invention relates to a content server. This content server is characterized by having a learning image generation unit that generates learning images that represent the state of each scene in a three-dimensional displayed world, where the situation changes in response to user operations, as viewed from multiple viewpoints; a type-specific 3D scene information acquisition unit that acquires multiple types of 3D scene information that represent the three-dimensional information of each scene through machine learning using the learning images as teacher data; and a 3D scene information transmission unit that transmits data on the multiple types of 3D scene information to a client terminal that uses the 3D scene information to draw a display image, in the order in which machine learning for each scene is completed.

Another aspect of the present invention relates to a client terminal. This client terminal is characterized by having an input information acquisition unit that acquires information on user operations and information on a display viewpoint for a three-dimensional display world, a 3D scene information acquisition unit that acquires from a server data on multiple types of 3D scene information that represent three-dimensional information acquired by machine learning for each scene in the display world whose situation changes according to user operations, and an image generation unit that draws at least a portion of a frame of a display image using the most recently acquired 3D scene information based on the latest display viewpoint.

Another aspect of the present invention relates to an image display system. This image display system includes a client terminal that displays an image of a three-dimensional display world in which the situation changes according to user operations, and a content server that transmits data used to generate the display image. The content server includes a learning image generation unit that generates images showing how each scene in the display world looks from multiple viewpoints as learning images, a type-specific 3D scene information acquisition unit that acquires multiple types of 3D scene information that represent the three-dimensional information of each scene by machine learning using the learning images as teacher data, and a 3D scene information transmission unit that transmits the multiple types of 3D scene information data to the client terminal in the order in which machine learning for each scene is completed. The client terminal includes an input information acquisition unit that acquires information on user operations and information on a display viewpoint for the display world, a 3D scene information acquisition unit that acquires data on the multiple types of 3D scene information from the content server, and an image generation unit that draws at least a part of the frame of the display image using the most recently acquired 3D scene information based on the latest display viewpoint.

Another aspect of the present invention relates to a display data transmission method. This display data transmission method is characterized by including the steps of: generating, as learning images, images that represent how each scene in a three-dimensional display world, in which the situation changes in response to user operations, is viewed from multiple viewpoints; acquiring multiple types of 3D scene information that represent the three-dimensional information of each scene through machine learning using the learning images as teacher data; and transmitting data on the multiple types of 3D scene information to a client terminal that uses the 3D scene information to draw a display image, in the order in which machine learning for each scene is completed.

Another aspect of the present invention relates to a display image generating method, which is characterized by including the steps of: acquiring information on user operations and information on a display viewpoint for a three-dimensional display world; acquiring from a server data on multiple types of 3D scene information representing three-dimensional information acquired by machine learning for each scene in the display world whose situation changes according to the user operations; and drawing at least a part of a frame of a display image using the most recently acquired 3D scene information based on the latest display viewpoint.

In addition, any combination of the above components, and any conversion of the present invention into a method, device, system, computer program, data structure, recording medium, etc., are also valid aspects of the present invention.

According to the present invention, when processing images of content that is distributed from a server, the impact of distribution on the quality of the user experience can be reduced.

1 is a diagram showing a configuration example of an image display system to which the present embodiment can be applied; FIG. 2 is a diagram illustrating an internal circuit configuration of a client terminal according to the present embodiment. FIG. 1 is a diagram showing a basic flow of image processing according to the present embodiment in comparison with the prior art. 2 is a diagram showing a configuration of functional blocks of a client terminal and a content server according to the present embodiment. FIG. 10 is a diagram illustrating a procedure in which a content server acquires 3D scene information in the present embodiment. FIG. 1A to 1C are diagrams for explaining how 3D scene information of different ranges is acquired in the present embodiment. 10 is a diagram showing a schematic diagram of data transition during transmission and reception of 3D scene information in the present embodiment. FIG. A diagram for explaining the temporal relationship between machine learning in a content server and image drawing in a client terminal in this embodiment. 11A and 11B are diagrams for explaining an image correction process performed by an image generating unit of a client terminal in the present embodiment.

FIG. 1 shows an example of the configuration of an image display system to which this embodiment can be applied. The image display system 1 includes client terminals 10a, 10b, 10c that display images in response to user operations, and a content server 20 that provides data used for display. Input devices 14a, 14b, 14c for user operations and display devices 16a, 16b, 16c that display images are connected to the client terminals 10a, 10b, 10c, respectively. The client terminals 10a, 10b, 10c and the content server 20 can establish communication via a network 8 such as a WAN (World Area Network) or a LAN (Local Area Network).

The client terminals 10a, 10b, 10c may be connected to the display devices 16a, 16b, 16c and the input devices 14a, 14b, 14c either wired or wirelessly. Alternatively, two or more of these devices may be formed integrally. For example, in the figure, the client terminal 10b is connected to a head-mounted display, which is the display device 16b. The head-mounted display can change the field of view of the displayed image according to the movement of the user wearing it on the head, so it also functions as the input device 14b.

The client terminal 10c is a portable terminal that is integrated with the display device 16c and the input device 14c, which is a touchpad that covers the screen of the display device 16c. In this way, there are no limitations on the external shape or connection form of the illustrated devices. There are also no limitations on the number of client terminals 10a, 10b, 10c and content servers 20 that are connected to the network 8. Hereinafter, the client terminals 10a, 10b, 10c will be collectively referred to as client terminals 10, the input devices 14a, 14b, 14c as input device 14, and the display devices 16a, 16b, 16c as display device 16.

The input device 14 may be any one or a combination of general input devices such as a controller, keyboard, mouse, touchpad, joystick, or various sensors such as a motion sensor or camera equipped in a head mounted display, and supplies the contents of user operations to the client terminal 10. The display device 16 may be any general display such as a liquid crystal display, plasma display, organic EL display, wearable display, or projector, and displays images output from the client terminal 10.

The content server 20 provides data of content accompanied by image display to the client terminal 10. The type of content is not particularly limited, and may be any of electronic games, decorative images, web pages, video chat using avatars, etc. In this embodiment, the content server 20 sequentially obtains information on user operations on the input device 14 from the client terminal 10, reflects this information in the world to be displayed, and transmits the necessary data so that an image representing this is displayed on the client terminal 10 side.

Figure 2 shows the internal circuit configuration of the client terminal 10. The client terminal 10 includes a CPU (Central Processing Unit) 122, a GPU (Graphics Processing Unit) 124, and a main memory 126. These components are interconnected via a bus 130. An input/output interface 128 is also connected to the bus 130. To the input/output interface 128, there are connected a communication unit 132 consisting of a peripheral device interface such as a USB or a network interface for a wired or wireless LAN, a storage unit 134 such as a hard disk drive or non-volatile memory, an output unit 136 that outputs data to the display device 16, an input unit 138 that inputs data from the input device 14, and a recording medium drive unit 140 that drives a removable recording medium such as a magnetic disk, optical disk, or semiconductor memory.

The CPU 122 executes an operating system stored in the storage unit 134 to control the entire client terminal 10. The CPU 122 also executes various programs that have been read from a removable recording medium and loaded into the main memory 126, or downloaded via the communication unit 132. The GPU 124 performs drawing processing in accordance with drawing commands from the CPU 122, and stores the display image in a frame buffer (not shown). The GPU 124 then converts the display image stored in the frame buffer into a video signal and outputs it to the output unit 136. The main memory 126 is composed of RAM (Random Access Memory), and stores programs and data necessary for processing. The content server 20 may also have a similar internal circuit configuration.

Figure 3 shows the basic flow of image processing in this embodiment in comparison with the prior art. In this embodiment, the main display target is a three-dimensional world in which various objects exist. The state of the world changes according to the provisions of the program or the user's operation. In the case of the general processing shown in (a), the content server constantly acquires information on the content of the user's operation, the position of the viewpoint relative to the displayed world, and the direction of the line of sight. Hereinafter, the entire three-dimensional space to be displayed is called the "display world", and the state of the displayed world within or near the display field of view is called the "scene". The position of the viewpoint and the direction of the line of sight relative to the scene may also be collectively referred to simply as the "viewpoint". The viewpoint may be manually operated by the user via the input device 14, or may be derived from the movement of the user's head using a motion sensor provided in the head-mounted display.

The content server draws the display image 200 in a field of view corresponding to the viewpoint information while changing the scene in response to user operations. The content server generates the display image 200 using well-known computer graphics drawing techniques such as ray tracing and rasterization. The content server transmits the generated display image 200 to the client terminal, which displays it on a display device. By repeating the illustrated process at a specified frame rate, a moving image showing the change in the scene in response to user operations, etc. is displayed on the client terminal side.

In this embodiment shown in (b), the content server 20 also acquires information on the content of user operations, the position of the viewpoint relative to the displayed world, and the direction of the gaze as needed, and similarly renders images while changing the scene accordingly. Meanwhile, in this embodiment, the content server 20 uses the images as training images 202 and as training data for machine learning. The content server 20 collects the training images 202 and performs machine learning to generate 3D scene information 204 that represents three-dimensional information about the scene.

NeRF (Neural Radiance Fields) is a method for acquiring information about three-dimensional space through machine learning (see, for example, Ben Mildenhall et al., "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis," Communications of the ACM, January 2022, Vol. 65, No. 1, pp. 99-106). When NeRF is introduced in this embodiment, first, the respective viewpoint information determined when generating the training images 202, i.e., the virtual viewpoint position and line of sight direction, are input, and the corresponding training images 202 are used as teacher data to obtain data representing the three-dimensional information of the scene by regression using a multilayer perceptron (MLP).

This data is a neural network that takes five-dimensional parameters consisting of position coordinates (x, y, z) and directional vector d (θ, φ) in three-dimensional space as input, and outputs volume density σ and color information c (RGB) of the three primary colors. In this embodiment, the data of this neural network is called "3D scene information." Note that various improvement methods have been proposed for NeRF, and the specific method is not particularly limited in this embodiment. Furthermore, it is not intended to limit the machine learning method to NeRF.

In this embodiment, the content represented by the learning image 202, and therefore the 3D scene information 204, can change from moment to moment. The figure shows the situation in which 3D scene information 204 of a scene at a certain time, or at an infinitesimal time that can be considered as a certain time, is generated. Hereinafter, a scene at a certain time, or at an infinitesimal time that can be considered as a certain time, may be expressed as "one scene". Note that if there is no movement in the displayed world, it can be treated as "one scene" regardless of time. Also, when 3D scene information 204 has been obtained for the previous scene, the content server 20 updates the 3D scene information 204 to correspond to the next scene. Hereinafter, the generation and updating of 3D scene information may be collectively referred to as "obtaining" 3D scene information.

In order to obtain accurate 3D scene information 204, it is desirable for the content server 20 to collect learning images 202 of one scene from as many viewpoints as possible. For this reason, the content server 20 may collect the learning images 202, for example, in the following manner.
(1) Generate viewpoints suitable for learning around the viewpoint that defines the field of view of the image that is actually displayed, and generate corresponding images. (2) Reuse images displayed on the devices of multiple users viewing the same scene.

Hereinafter, the viewpoint that defines the actual display field of view in the client terminal 10 is called the "display viewpoint" to be distinguished from the "learning viewpoint" that is set when generating a learning image. The content server 20 may implement only one of (1) and (2), or may implement both. For example, a viewpoint that is missing due to (2) may be supplemented by (1). The content server 20 transmits 3D scene information 204, which is the result of the learning, to the client terminal 10. The client terminal 10 generates a display image 206 using the transmitted 3D scene information 204.

By using the 3D scene information 204, the client terminal 10 can display the scene as it is viewed from any viewpoint with high quality, with a relatively low load. When NeRF is applied, the client terminal 10 generates a ray r that passes through each pixel on the view screen from the display viewpoint, and uses volume rendering to integrate the color along that direction to determine the pixel value C(r) of the display image as follows:

where t _n and t _f are the proximal and distal ends of the ray r, respectively, and T(t) is the cumulative transmittance in the direction of the ray, which can be expressed as follows:

The content server 20 repeats the process shown in the figure for the movement of the scene, updating the 3D scene information 204 at a predetermined rate and sequentially transmitting it to the client terminal 10. The client terminal 10 generates a display image 206 while updating the 3D scene information it uses, thereby making it possible to represent moving images from any viewpoint that have the same changes as the learning images 202 generated by the content server 20. For example, the client terminal 10 can display an image with little delay in response to changes in viewpoint by drawing a display image 206 that shows the scene as seen from the viewpoint immediately before display using the latest 3D scene information 204.

FIG. 4 shows the functional block configuration of the client terminal 10 and the content server 20 in this embodiment. The functions of the components in the illustrated functional blocks may be realized in circuitry or processing circuitry including general purpose processors, application specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and/or combinations thereof, configured or programmed to realize the functions described herein. A processor is considered to be a circuitry or processing circuitry including transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory.

In this specification, a circuit, unit, or means is hardware that is programmed to realize or executes a described function. The hardware may be any hardware disclosed in this specification or any hardware that is programmed to realize or known to execute the described function. If the hardware is a processor, which is considered to be a type of circuit, the circuit, means, or unit is a combination of hardware and software used to configure the hardware and/or processor.

The client terminal 10 includes an input information acquisition unit 50 that acquires input information such as user operations, a 3D scene information acquisition unit 52 that acquires 3D scene information data from the content server 20, a 3D scene information storage unit 54 that stores the acquired 3D scene information data, an image generation unit 56 that generates a display image, and an output unit 58 that outputs the display image data. The input information acquisition unit 50 acquires the contents of user operations from the input device 14 at any time. User operations include the selection and activation of content, and command input for content currently being executed.

The input information acquisition unit 50 also acquires information on the display viewpoint for the displayed world from the input device 14 or head-mounted display at any time or at a specified time interval. Technology for detecting the position and posture of the head of a user wearing a head-mounted display and acquiring viewpoint information based on this is well known, and this may also be applied in this embodiment. The input information acquisition unit 50 supplies the acquired information to the content server 20 and the image generation unit 56 as appropriate. The 3D scene information acquisition unit 52 sequentially acquires data of 3D scene information, which is continuously updated, from the content server 20.

As described below, the 3D scene information acquisition unit 52 acquires multiple types of 3D scene information representing one scene from the content server 20. The 3D scene information storage unit 54 stores the data of the multiple types of 3D scene information acquired by the 3D scene information acquisition unit 52. When the 3D scene information acquisition unit 52 acquires new 3D scene information, it updates the data of the same type of 3D scene information stored in the 3D scene information storage unit 54.

The image generation unit 56 uses the 3D scene information data most recently stored in the 3D scene information storage unit 54 to draw a display image at a predetermined frame rate. Here, the image generation unit 56 acquires the latest display viewpoint from the input information acquisition unit 50, and draws an image in the corresponding field of view using a technique such as the volume rendering described above. If 3D scene information corresponding to the transition of the scene can be prepared using machine learning, then even if the client terminal 10 generates the display image, it is possible to draw a high-quality image with a lighter load than with normal processing such as ray tracing.

The image generation unit 56 changes the display image representing one scene in time, space, or both by drawing using multiple types of 3D scene information. For example, in a mode in which the content server 20 transmits 3D scene information data in ascending order of information density, the image generation unit 56 updates the display image using the transmitted 3D scene information in sequence. This allows an image representing one scene to be displayed with low latency, and the visual image quality can be maintained by gradually increasing the resolution. The output unit 58 outputs the image drawn by the image correction unit 92 to the display device 16 at a predetermined rate for display.

The content server 20 includes an input information acquisition unit 70 that acquires input information from the client terminal 10, a learning viewpoint generation unit 72 that generates a learning viewpoint, a display world control unit 74 that controls the display world, a 3D model storage unit 76 that stores 3D models of objects, a learning image generation unit 78 that generates learning images, a type-specific 3D scene information acquisition unit 80 that acquires data on multiple types of 3D scene information, a 3D scene information storage unit 84 that stores the acquired data on the 3D scene information, and a 3D scene information transmission unit 86 that transmits the data on the 3D scene information to the client terminal 10.

The input information acquisition unit 70 acquires the contents of user operations and viewpoint information from the client terminal 10 at any time or at a specified time interval. The learning viewpoint generation unit 72 generates multiple learning viewpoints for generating learning images. The learning viewpoint generation unit 72 generates learning viewpoints around the latest display viewpoint acquired by the input information acquisition unit 70 according to a specified rule. For example, the learning viewpoint generation unit 72 places a specified number of learning viewpoints evenly inside a sphere of a specified radius centered on the latest display viewpoint. Here, the specified radius is a value obtained by multiplying the maximum speed expected for viewpoint movement by the longest time until display using the data is performed, for example.

The learning viewpoint generating unit 72 is not limited to distributing the learning viewpoints evenly, but may distribute more learning viewpoints in a range in which the viewpoint is expected to move, depending on the situation in the displayed world, etc. The learning viewpoint generating unit 72 may also set line of sight in a predetermined number of directions evenly for the viewpoint at each position, or may set more line of sight in directions in which the line of sight is expected to move. The learning viewpoint generating unit 72 may also use the latest display viewpoint itself as a learning viewpoint.

The display world control unit 74 controls the three-dimensional display world represented as content according to the content of the user operations acquired by the input information acquisition unit 70. For example, if the content is an electronic game, the display world control unit 74 places necessary objects such as user characters in the virtual space where the electronic game takes place, and gives them movement according to commands entered by the user and program specifications.

The three-dimensional model storage unit 76 stores three-dimensional models of objects that exist in the display world, and the display world control unit 74 reads them out as appropriate to use in constructing the display world. The learning image generation unit 78 generates images of the scene as viewed from multiple learning viewpoints generated by the learning viewpoint generation unit 72 as learning images. The learning image generation unit 78 preferably generates learning images using a technique capable of rendering high-quality images, such as ray tracing.

The type-specific 3D scene information acquisition unit 80 uses the learning images generated by the learning image generation unit 78 to generate 3D scene information through machine learning as described above, and updates the information to correspond to changes in the scene. Here, the type-specific 3D scene information acquisition unit 80 acquires multiple types of 3D scene information representing one scene. For example, the type-specific 3D scene information acquisition unit 80 acquires multiple pieces of 3D scene information in which the spatial density of the represented information differs from one another. Hereinafter, the spatial density of the information held by the 3D scene information is referred to as "information density." Information density may also be referred to as the resolution of the information held by the 3D scene information, or the spatial frequency of the information.

For example, according to the NeRF paper mentioned above, the vectors input to a neural network during training are converted into vectors in a high-dimensional space that includes high frequencies using Positional Encoding, which allows the high-frequency components of the output vector to be represented more accurately. The function γ used for the conversion is expressed as follows:

In the above formula, the larger the parameter L is, the more detailed 3D scene information including higher frequency components can be acquired. Using this, the type-specific 3D scene information acquisition unit 80 acquires multiple pieces of 3D scene information with different information densities in parallel by setting multiple parameters L and performing machine learning individually. However, the means for controlling the information density of the 3D scene information is not limited to this. For example, instead of Positional Encoding, it is possible to apply Multiresolution Hash Encoding, which sets a grid with multiple resolutions and expresses an input vector based on the positional relationship with its vertices (see, for example, Thomas Muller et al., "Instant Neural Graphics Primitives with a Multiresolution Hash Encoding," ACM Transactions on Graphics, July 2022, Vol. 41, No. 4, Article 102, pp. 1-15).

In this case, the type-specific 3D scene information acquisition unit 80 can acquire multiple pieces of 3D scene information with different information densities in parallel by setting multiple levels L corresponding to the number of grid resolutions and performing machine learning individually. In these cases, multiple values to be set as the parameter L are prepared in the internal memory of the type-specific 3D scene information acquisition unit 80. The type-specific 3D scene information acquisition unit 80 then sequentially uses multiple learning images generated for one scene by the learning image generation unit 78 to acquire multiple pieces of 3D scene information with different information densities in parallel, and stores them in the 3D scene information storage unit 84 or updates the same type of 3D scene information that has already been stored.

In this case, even for the same scene, the learning speed differs depending on the information density; the lower the information density, the faster learning is completed, and the faster the 3D scene information is updated. When setting a grid and performing machine learning, the type-specific 3D scene information acquisition unit 80 may set the level number L to only one and learn, and when transmitting to the client terminal 10, etc., select grids of different levels (e.g., grids of levels 0 to 1, grids of levels 0 to 2, ..., grids of levels 0 to L-1, etc.) individually to read out data. In this case, too, the learning is completed more quickly for lower level grids, i.e., grids with lower information density, and the effects described below are the same.

The types of 3D scene information acquired by the type-specific 3D scene information acquisition unit 80 are not limited to distinctions in information density. For example, the type-specific 3D scene information acquisition unit 80 may acquire multiple pieces of 3D scene information with different areas or objects to be learned in the display world. In this case, the smaller the range to be learned in the display world, the faster the learning is completed, and therefore the faster the 3D scene information is updated.

In this way, even if the multiple types of 3D scene information acquired by the type-specific 3D scene information acquisition unit 80 are for the same scene, the time required to complete the update may differ depending on the density of the information and the size of the range it represents. For this reason, the type-specific 3D scene information acquisition unit 80 records the time of the reflected scene in association with each of the multiple types of 3D scene information stored in the 3D scene information storage unit 84. Note that the type-specific 3D scene information acquisition unit 80 may acquire multiple types of 3D scene information with different combinations of information density and range of the learning target.

The 3D scene information transmission unit 86 transmits the latest 3D scene information data stored in the 3D scene information storage unit 84 to the client terminal 10. The 3D scene information transmission unit 86 transmits to the client terminal 10 3D scene information in order of completion of learning for one scene. For example, when data of 3D scene information with different information densities is to be transmitted, learning is completed in order from the 3D scene information with the lowest information density, as described above. Therefore, the 3D scene information transmission unit 86 transmits data of the 3D scene information with the lowest information density when learning of that information is completed, and transmits data of the 3D scene information with the next lowest information density when learning of that information is completed, and so on, gradually transmitting up to the 3D scene information with the highest information density.

The 3D scene information transmission unit 86 includes a division unit 88 inside. The division unit 88 divides each of the multiple types of 3D scene information representing one scene into multiple data. Each type of 3D scene information representing one scene is composed of an individual neural network. The division unit 88 divides each neural network into multiple neural networks. Here, division means removing some of the nodes that are different from each other while maintaining the structure of the nodes that are associated by a hash table or the like. The nodes to be removed in the neural network after division are determined randomly.

To alleviate overfitting, a technique called dropout is known, in which some of the nodes in a neural network are randomly deactivated (see, for example, Nitish Srivastava et al., "Dropout: A Simple Way to Prevent Neural Networks from Overfitting," Journal of Machine Learning Research, June 2014, Vol. 15, pp. 1919-1958). As is clear from dropout, the accuracy of the learning results of a neural network can be maintained at a certain level even if some of the nodes are deactivated.

By utilizing this characteristic, the client terminal 10 can generate a display image with relatively high accuracy even when using 3D scene information with some of the nodes removed. Therefore, the content server 20 can transmit multiple neural networks representing the same 3D scene information with some of the nodes removed, thereby suppressing an increase in data size and increasing robustness against packet loss. If there is no packet loss, the image generation unit 56 of the client terminal 10 can restore the neural network before division and draw the display image. If there is packet loss, the image generation unit 56 draws the display image using the neural network that it has acquired. As described above, it is possible to generate a display image in this case as well.

As mentioned above, multiple types of 3D scene information representing one scene may arrive from the content server 20 with a time lag. In this case, the image generation unit 56 of the client terminal 10 updates the display image using the newly acquired 3D scene information. Since the resolution of the display image corresponds to the information density of the 3D scene information, the resolution of the display image gradually increases by acquiring 3D scene information in order starting with the 3D scene information with the lowest information density. Here, 3D scene information with low information density has a smaller number of samples in volume rendering, and therefore the drawing speed of the display image increases.

As a result, by first acquiring low-information-density 3D scene information that requires a short learning time in the content server 20 and then drawing a low-resolution image in a short time, the time from the start of learning each scene to its display can be significantly shortened. Furthermore, ultimately, high-definition images can be displayed using high-information-density 3D scene information, making it possible to achieve low-latency display while minimizing the impact on appearance. As mentioned above, the types of 3D scene information are not limited to distinctions in resolution. For example, the content server 20 may learn only the partial area that the user is gazing at in a short time and transmit it first, and then transmit the 3D scene information for the entire scene later. In this case as well, a similar principle can be used to display an image in which the area being gazed at is displayed with low latency and updated to track the surrounding image.

The 3D scene information transmitting unit 86 may transmit 3D scene information representing one scene to the client terminal 10 using different communication protocols depending on the type of 3D scene information. For example, the 3D scene information transmitting unit 86 may transmit 3D scene information for an area where image detail should be prioritized using the highly reliable TCP/IP (Transmission Control Protocol/Internet Protocol), and transmit 3D scene information for an area where low latency in movement should be prioritized using UDP (User Datagram Protocol), which has a high transfer rate. In this way, the correspondence between the type of 3D scene information and the appropriate communication protocol is preset in the internal memory of the 3D scene information transmitting unit 86, etc.

FIG. 5 shows a schematic diagram of the procedure by which the content server 20 acquires 3D scene information. First, the display world control unit 74 constructs a display world 210 in which, for example, an enemy character 212 exists. As mentioned above, the display world 210 may be in motion, but here, the display world 210 corresponding to one scene is shown. In response to this, the learning viewpoint generation unit 72 generates multiple learning viewpoints based on the latest display viewpoint, etc., and the learning image generation unit 78 generates learning images 214a, 214b, 214c, etc. corresponding to each learning viewpoint. Note that there is no limit to the number of learning images to be generated.

The type-specific 3D scene information acquisition unit 80 performs machine learning using the learning images 214a, 214b, 214c, etc., and updates multiple types of 3D scene information. The substance of each piece of 3D scene information is a neural network 216a, 216b, 216c, .... The neural networks 216a, 216b, 216c, ... differ in at least one of the information density and the range they represent. When the information density is to be varied, the type-specific 3D scene information acquisition unit 80 performs learning, for example, by setting the above-mentioned parameter L for each. In this case, the lower the information density of the 3D scene information, the quicker learning is completed.

When the range to be represented is varied, the type-specific 3D scene information acquisition unit 80 performs learning by, for example, cutting out the corresponding area from the learning images 214a, 214b, 214c. In this case, the narrower the range of the 3D scene information represented, the sooner the learning is completed. When the information density and the range to be represented are changed in combination, the time until learning is completed varies depending on the balance. Qualitatively, the higher the information density and the wider the range represented, the slower the learning is completed, so the balance between information density and range is optimized to obtain an appropriate delay time. By repeating the illustrated process at a predetermined rate, the neural networks 216a, 216b, 216c, ... are each updated in accordance with the changes in the scene.

FIG. 6 is a diagram for explaining the manner in which 3D scene information of different ranges is acquired. Assume that a part of the scene of the display world 210 shown in FIG. 5 is displayed as the display image 222. The type-specific 3D scene information acquisition unit 80, for example, separately acquires 3D scene information representing a range 226 in the display world 210 corresponding to an important area 224 on the display image 222, and 3D scene information representing the entire scene including the outside of that range. The important area 224 here is, for example, an area within a predetermined range from the user's gaze point, an area within a predetermined range from the center of the display image, an area showing the battle situation or acquired items, an area where main objects such as enemy characters and user characters exist, etc., and the rules for selecting the area are set in advance according to the content of the content, etc.

Alternatively, the type-specific 3D scene information acquisition unit 80 may directly identify the main objects themselves that exist in the display world 210, or a range of a predetermined size that includes the objects, and acquire 3D scene information as the learning target. When determining the learning range based on the user's gaze point, a well-known gaze point detector is provided in the client terminal 10. The input information acquisition unit 70 of the content server 20 then acquires gaze point information from the client terminal 10 at a predetermined rate and determines the learning range individually.

The type-specific 3D scene information acquisition unit 80 may separately learn the range of the display world corresponding to the display image 222 being displayed on the client terminal 10 and a wider range including the outside of that range. In any case, the type-specific 3D scene information acquisition unit 80 performs machine learning by extracting corresponding partial areas from the multiple learning images generated by the learning viewpoint generation unit 72, and also performs machine learning of a wider range by using the entire learning images, for example.

The number of variations in ranges represented by the 3D scene information is not limited to two, and may be three or more. The inclusion relationship of the ranges is not limited, and 3D scene information may be obtained for each of multiple independent ranges. In any case, 3D scene information is obtained for the number of set ranges. The narrower the range, the shorter the learning time and drawing time, making it possible to display with low latency. By utilizing this characteristic, even if the information density of the 3D scene information corresponding to the important area 224 is increased to a certain extent, if the range is narrow, it is possible to complete drawing in the same amount of time as drawing a wide-area image using 3D scene information with low information density. As a result, it becomes possible to display the important area 224 with high definition and low latency.

Figure 7 shows a schematic diagram of data transitions during transmission and reception of 3D scene information. The diagram shows the passage of time from top to bottom, with (a) to (c) showing data state transitions in the content server 20 and (c) to (e) showing data state transitions in the client terminal 10. This diagram also shows 3D scene information representing one scene. First, the type-specific 3D scene information acquisition unit 80 of the content server 20 acquires multiple neural networks 230a, 230b, ... each corresponding to multiple types of 3D scene information, based on a newly generated learning image, as shown in (a).

Next, the 3D scene information transmission unit 86 of the content server 20 divides each of the neural networks 230a, 230b as shown in (b). That is, the 3D scene information transmission unit 86 generates a plurality of neural networks 232a, 232b from the neural network 230a in which some mutually different nodes have been removed. The 3D scene information transmission unit 86 also generates a plurality of neural networks 234a, 234b from the neural network 230b in which some mutually different nodes have been removed.

In the figure, the nodes that have been removed from the original neural networks 230a and 230b of the divided neural networks 232a, 232b, 234a, and 234b are shown with dotted lines. If, after dividing a neural network 230a, the neural networks 232a and 232b are obtained and the nodes that have been removed in one neural network 232a are left in the other neural network 232b, then by combining the two, the original neural network 230a can be completely restored.

However, as mentioned above, even if there are missing nodes due to packet loss or the like, it is possible to generate an image using the remaining neural networks. The number of divisions of the neural network is not limited to two. The division unit 88 of the 3D scene information transmission unit 86 preferably randomly determines nodes to be excluded using a method similar to dropout. The 3D scene information transmission unit 86 packetizes the divided neural networks 232a, 232b, 234a, and 234b and transmits them to the client terminal 10.

At this time, the 3D scene information transmission unit 86 randomly rearranges the transmission order of packets from multiple neural networks, as shown in (c). In the figure, the transmission order is shown as being arranged horizontally, with neural networks 232b, 234b, 232a, .... By rearranging the transmission order, it is possible to reduce the possibility that all 3D scene information of a certain type will be lost due to continuous packet loss. However, the rearrangement is performed between neural networks that have been updated in parallel within a certain allowable time. This prevents unnecessary delays in the rendering process due to the rearrangement.

When the 3D scene information acquisition unit 52 of the client terminal 10 acquires the packets in sequence, it reconstructs the neural network before division by returning the order of the extracted neural networks to their original order, as shown in (d). For this reason, the content server 20 adds metadata to each of the neural networks 232b, 234b, 232a, ... to be transmitted, indicating which of the original neural networks 230a, 230b, ... it has divided.

In the example shown in the figure, the 3D scene information acquisition unit 52 is able to acquire the neural networks 232a and 232b, and is therefore able to completely restore the original neural network 230a. On the other hand, as shown by the dotted frame 236, one of the split neural networks, neural network 234a, cannot be acquired due to packet loss for the original neural network 230b. In this case, the original neural network 230b cannot be completely restored. In any case, the image generation unit 56 of the client terminal 10 uses the acquired neural networks 232a, 232b, and 234b to draw the display image 238, as shown in (e).

Specifically, the image generation unit 56 draws an image of the scene seen from the latest display viewpoint by volume rendering using the neural networks 232a, 232b, and 234b. When multiple types of 3D scene information are transmitted with a time difference, the image generation unit 56 updates at least a portion of the display image 238 using the latest 3D scene information. This makes it possible to realize a display in which the image resolution gradually increases and important images move with particularly low latency.

Figure 8 is a diagram for explaining the temporal relationship between machine learning in the content server 20 and image drawing in the client terminal 10. Here, as an example, we consider a case in which multiple types of 3D scene information with different information densities are transmitted. The horizontal direction of the figure is the time axis, with the learning time in the content server 20 and the drawing time of each frame of the displayed image in the client terminal 10 each shown as a rectangle. The number attached to each drawing time rectangle indicates the frame order.

As shown in the upper part, the type-specific 3D scene information acquisition unit 80 of the content server 20 learns the first, second, ... nth types of 3D scene information individually. Here, the ordinal numbers correspond to the information density, with the first type being the lowest information density and the nth type being the highest information density. As shown in the figure, even if the type-specific 3D scene information acquisition unit 80 starts learning multiple types of 3D scene information simultaneously, the higher the information density, the longer it takes to complete the learning. First, when the content server 20 has completed learning the first type of 3D scene information, it transmits it to the client terminal 10, as shown by arrow A1.

The client terminal 10 uses the first type of 3D scene information to draw images of frames numbered 0, 1, and 2 at the lowest resolution from time t1. Meanwhile, when the content server 20 has completed learning the second type of 3D scene information, it transmits it to the client terminal 10, as indicated by arrow A2. The client terminal 10 begins drawing frames numbered 3 immediately after time t2 when the second type of 3D scene information is acquired, and draws images at a resolution corresponding to the second type of information density. By repeating the same process, the frame resolution gradually increases.

Once the content server 20 has completed learning the nth type of 3D scene information, it transmits it to the client terminal 10 as indicated by arrow An. The client terminal 10 begins drawing the image at the highest resolution corresponding to the nth type of information density, starting with frame number m+2, which begins drawing immediately after time tn when the 3D scene information is acquired. In this manner, in this embodiment, multiple types of 3D scene information representing the same scene are learned, and the 3D scene information that is completed learning the earliest is immediately transmitted to the client terminal 10. This allows the frame drawing start time to be significantly earlier than when only 3D scene information with a high information density, such as the nth type, is transmitted.

In other words, if the time when learning starts in the content server 20 is used as a reference, drawing can start with a delay of only the time Ta for data transmission and the time Tb required for learning the first type of 3D scene information. Note that in practice, before time t1, drawing of a frame may be performed using the 3D scene information of the previous scene transmitted immediately before. Furthermore, when drawing each frame, the image generating unit 56 performs volume rendering based on the most recent display viewpoint acquired immediately before. This not only increases the resolution from frame numbers 0 to m+2, but also makes it possible to display images that reflect the movement of the viewpoint with low delay. Here, the lower the information density of the first type or other 3D scene information, the fewer the number of samples in volume rendering and the higher the drawing speed, making it possible to display with less delay.

In this way, image generation unit 56 can be said to have the function of correcting an image that has already been generated so that it matches the latest display viewpoint, regardless of whether it has already been displayed. Figure 9 is a diagram for explaining the image correction process by image generation unit 56 of client terminal 10. Image 250a is a frame or part of a display image, and shows image 252a of a cylindrical object in the foreground. If the position of the cylindrical object changes relatively due to movement of the viewpoint, and object image 252b shifts in image 250b after the lapse of time Δt, it becomes necessary to newly draw background area 254 that was hidden in image 250a.

In conventional technology that displays images sent from the content server 20 without using 3D scene information, it is difficult to create a new area 254 that is not shown in the original image 250a. If 3D scene information containing information on the object and its surrounding area is used, all pixels of image 250b can be determined to correspond to the latest viewpoint, so it is naturally possible to draw area 254. Furthermore, if the display viewpoint moves and the positional relationship with the object and light source changes, the color of the image 252b of the object may also change. In this case, too, the change in color can be accurately expressed by using the 3D scene information to draw an image seen from a new display viewpoint. This makes it possible to generate image 250c after a time Δt has passed with high accuracy.

As described above, assuming that the client terminal 10 uses 3D scene information to draw each frame of the display image, images 250a and 250c are images that the client terminal 10 has drawn using the latest 3D scene information at that time. On the other hand, assuming a situation in which the content server 20 transmits the display image together with the 3D scene information, it is also possible for the client terminal 10 to correct image 250a transmitted from the content server 20 using the 3D scene information.

For example, the client terminal 10 generates an image 250c that has been corrected to reflect the movement of the viewpoint that occurs during the time difference Δt between when the image 250a is generated on the content server 20 side and when it is displayed on the client terminal 10. In this case as well, the image generating unit 56 of the client terminal 10 redraws the area 254 that is not shown in the image 250a generated by the content server 20 and the image 252b whose color has changed, using the latest 3D scene information.

In this case, since the high-quality image 250a generated by the content server 20 is to be combined with the area newly drawn by the client terminal 10, it is desirable for the image generation unit 56 to draw the necessary area using 3D scene information with high information density. Note that the image generation unit 56 can omit new drawing processing using 3D scene information in a situation where there is little sense of incongruity simply by moving or deforming the image in the image 250a transmitted from the content server 20. For example, for distant objects where the amount of image shift or color change is small in response to changes in the display viewpoint, the image generation unit 56 may perform correction by directly processing the image 250a.

For this reason, the image generation unit 56 stores in advance in an internal memory or the like rules for determining areas that need to be redrawn using 3D scene information. For example, when the speed of the display viewpoint exceeds a threshold value, or is predicted to exceed it, the image generation unit 56 may use 3D scene information to redraw objects whose distance from the viewpoint is less than or equal to the threshold value, and their surrounding areas. In this case, the content server 20 also transmits geometry information of the display world to the client terminal 10. This allows the image generation unit 56 to obtain changes in the distance between the display viewpoint and the object.

According to the present embodiment described above, the content server 20 generates learning images that represent the display world of the content, and acquires multiple types of 3D scene information for each scene. The content server 20 sequentially transmits the 3D scene information for which learning has been completed to the client terminal 10, and the client terminal 10 uses it to draw frames of the display image corresponding to the latest display viewpoint. By setting various types of 3D scene information, such as the density of information and the width of the range to be represented, the learning time can be changed, and the delay time until display can also be controlled. Furthermore, the level of detail can be increased in some areas, making it possible to display high-quality images with low latency that take into account the importance of the image, etc.

The content server 20 also divides the neural network that constitutes one piece of 3D scene information into multiple neural networks and transmits them to the client terminal 10. This makes it possible to improve robustness against packet loss while suppressing increases in data size. As a result, in image processing involving distribution from the content server 20, the impact of distribution can be reduced, improving the quality of the user experience.

The present invention has been described above based on an embodiment. The embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present invention.

As described above, the present invention can be used in various information processing devices such as content servers, game devices, head-mounted displays, display devices, mobile terminals, and personal computers, as well as image display systems that include any of these.

The present disclosure may include the following aspects.
[Item 1]
A content server, comprising:
The circuitry includes:
Generate learning images that represent scenes from multiple viewpoints of a three-dimensional display world in which the situation changes in response to user operations;
acquiring a plurality of types of 3D scene information representing three-dimensional information of each scene by machine learning using the learning images as training data;
Transmitting the plurality of types of 3D scene information data to a client terminal that draws a display image using the 3D scene information in the order in which machine learning for each scene is completed;
Content server.
[Item 2]
The circuitry includes:
2. The content server according to item 1, wherein a plurality of pieces of 3D scene information are acquired, each having a different spatial information density.
[Item 3]
The circuitry includes:
A content server according to item 1, which divides a neural network constituting one of the 3D scene information into multiple neural networks from which some nodes that are different from each other are excluded, and transmits the multiple neural networks to the client terminal.
[Item 4]
4. The content server according to claim 3, wherein the circuitry packetizes the neural network after division and randomly changes the transmission order of the packets.
[Item 5]
2. The content server of claim 1, wherein the circuitry obtains a plurality of pieces of 3D scene information representing different areas in the display world.
[Item 6]
6. The content server of claim 5, wherein the circuitry obtains the 3D scene information representing a range of the display world that corresponds to a defined area in an image displayed on the client terminal.
[Item 7]
6. The content server of claim 5, wherein the circuitry obtains 3D scene information for objects present in the display world.
[Item 8]
2. The content server according to claim 1, wherein the circuitry transmits data of the plurality of types of 3D scene information to the client terminal using different communication protocols.
[Item 9]
A client terminal, comprising:
The circuitry includes:
acquiring information on a user operation and information on a display viewpoint for a three-dimensional display world;
Acquire from a server a plurality of types of 3D scene information data representing three-dimensional information acquired by machine learning for each scene in the displayed world whose situation changes in response to the user operation;
Rendering at least a portion of a frame of a display image using the most recently acquired 3D scene information based on an updated viewing viewpoint.
Client terminal.
[Item 10]
The client terminal according to item 9, wherein the circuitry obtains a plurality of neural networks obtained by dividing a neural network that constitutes one of the 3D scene information, with some nodes that are different from each other being excluded, and reconstructs the neural network before division.
[Item 11]
The circuitry acquires a plurality of pieces of 3D scene information having different spatial information densities in ascending order of spatial information density;
10. A client terminal as described in item 9, which varies the resolution of the frames to correspond to the information density of the acquired 3D scene information.
[Item 12]
10. The client terminal according to item 9, wherein the circuitry uses the 3D scene information to draw an area of the display image transmitted from the server that is determined to require drawing due to a change in the display viewpoint.
[Item 13]
A client terminal that displays an image of a three-dimensional display world in which a situation changes in response to a user operation, and a content server that transmits data used to generate the display image, the client terminal and the content server having a circuitry configured to:
The circuitry of the content server includes:
generating images representing each scene of the display world as seen from a plurality of viewpoints as learning images;
By machine learning using the learning images as training data, a plurality of types of 3D scene information representing three-dimensional information of each scene is obtained;
Transmitting the plurality of types of 3D scene information data to the client terminal in the order in which machine learning for each scene is completed;
The circuitry of the client terminal includes:
acquiring information on the user operation and information on a display viewpoint for the display world;
acquiring the plurality of types of 3D scene information data from the content server;
Rendering at least a portion of a frame of a display image using the most recently acquired 3D scene information based on an updated viewing viewpoint.
Image display system.
[Item 14]
Generate learning images that represent scenes from multiple viewpoints of a three-dimensional display world in which the situation changes in response to user operations;
By machine learning using the learning images as training data, a plurality of types of 3D scene information representing three-dimensional information of each scene is obtained;
Transmitting the plurality of types of 3D scene information data to a client terminal that draws a display image using the 3D scene information in the order in which machine learning for each scene is completed;
A method for transmitting data for display.
[Item 15]
Acquiring information on a user operation and information on a display viewpoint for a three-dimensional display world;
Acquire from a server a plurality of types of 3D scene information data representing three-dimensional information acquired by machine learning for each scene in the displayed world whose situation changes in response to the user operation;
Rendering at least a portion of a frame of a display image using the most recently acquired 3D scene information based on an updated viewing viewpoint.
Display image generation method.
[Item 16]
A function of generating images for learning that represent each scene in a three-dimensional display world, the situation of which changes in response to user operations, as seen from multiple viewpoints;
A function of acquiring a plurality of types of 3D scene information representing three-dimensional information of each scene by machine learning using the learning images as training data;
A function of transmitting the data of the plurality of types of 3D scene information to a client terminal that draws a display image using the 3D scene information in the order in which machine learning for each scene is completed;
A recording medium on which a program for realizing the above on a computer is recorded.
[Item 17]
A function for acquiring information on user operations and information on a display viewpoint for a three-dimensional display world;
A function of acquiring from a server a plurality of types of 3D scene information data representing three-dimensional information acquired by machine learning for each scene in the displayed world whose situation changes in response to the user operation;
Rendering at least a portion of a frame of a display image using the most recently acquired 3D scene information based on an updated display viewpoint;
A recording medium on which a program for implementing the above on a computer is recorded.

1 Image display system, 10 Client terminal, 14 Input device, 16 Display device, 20 Content server, 50 Input information acquisition unit, 52 3D scene information acquisition unit, 54 3D scene information storage unit, 56 Image generation unit, 58 Output unit, 70 Input information acquisition unit, 72 Learning viewpoint generation unit, 74 Display world control unit, 76 3D model storage unit, 78 Learning image generation unit, 80 Type-specific 3D scene information acquisition unit, 84 Type-specific 3D scene information storage unit, 86 3D scene information transmission unit, 88 Splitting unit, 122 CPU, 124 GPU, 126 Main memory.

Claims (17)

a learning image generating unit that generates, as learning images, images that represent how each scene in a three-dimensional displayed world, whose situation changes in response to a user operation, is viewed from a plurality of viewpoints;
a type-specific 3D scene information acquisition unit that acquires multiple types of 3D scene information representing three-dimensional information of each scene by machine learning using the learning images as teacher data;
a 3D scene information transmission unit that transmits data of the plurality of types of 3D scene information to a client terminal that draws a display image using the 3D scene information in the order in which machine learning for each scene is completed;
A content server comprising: The content server according to claim 1, characterized in that the type-specific 3D scene information acquisition unit acquires multiple pieces of 3D scene information with different spatial information densities. The content server according to claim 1 or 2, characterized in that the 3D scene information transmission unit divides the neural network that constitutes one piece of 3D scene information into multiple neural networks from which some mutually different nodes are excluded, and transmits the divided neural networks to the client terminal. The content server according to claim 3, characterized in that the 3D scene information transmission unit packetizes the neural network after division and randomly changes the transmission order. The content server according to claim 1 or 2, characterized in that the type-specific 3D scene information acquisition unit acquires multiple pieces of 3D scene information representing different ranges in the display world. The content server according to claim 5, characterized in that the type-specific 3D scene information acquisition unit acquires the 3D scene information that represents the range of the display world that corresponds to an area defined in the image displayed on the client terminal. The content server according to claim 5, characterized in that the type-specific 3D scene information acquisition unit acquires 3D scene information targeting objects present in the display world. The content server according to claim 1, characterized in that the 3D scene information transmission unit transmits the data of the multiple types of 3D scene information to the client terminal using different communication protocols. an input information acquisition unit that acquires information on a user operation and information on a display viewpoint for a three-dimensional display world;
a 3D scene information acquisition unit that acquires from a server a plurality of types of 3D scene information data representing three-dimensional information acquired by machine learning for each scene in the displayed world whose situation changes in response to the user operation;
an image generator configured to render at least a portion of a frame of a display image using the most recently acquired 3D scene information based on a latest display viewpoint;
A client terminal comprising: The client terminal according to claim 9, characterized in that the 3D scene information acquisition unit acquires multiple neural networks obtained by dividing a neural network constituting one piece of 3D scene information, with some mutually different nodes removed, and reconstructs the neural network before division. The 3D scene information acquisition unit acquires a plurality of pieces of 3D scene information having different spatial information densities in order of decreasing density of the information,
11. The client terminal according to claim 9, wherein the image generating unit changes a resolution of the frames so as to correspond to the information density of the acquired 3D scene information. The client terminal according to claim 9 or 10, characterized in that the image generation unit uses the 3D scene information to render an area of the display image transmitted from the server that is determined to require rendering due to a change in the display viewpoint. The present invention includes a client terminal that displays an image of a three-dimensional display world in which a situation changes in response to a user operation, and a content server that transmits data used to generate the display image,
The content server,
a learning image generating unit that generates images representing how each scene in the display world is viewed from a plurality of viewpoints as learning images;
a type-specific 3D scene information acquisition unit that acquires multiple types of 3D scene information representing three-dimensional information of each scene by machine learning using the learning images as teacher data;
a 3D scene information transmission unit that transmits the plurality of types of 3D scene information data to the client terminal in the order in which machine learning for each scene is completed;
Equipped with
The client terminal includes:
an input information acquisition unit that acquires information on the user operation and information on a display viewpoint for the display world;
a 3D scene information acquisition unit that acquires the plurality of types of 3D scene information data from the content server;
an image generator configured to render at least a portion of a frame of a display image using the most recently acquired 3D scene information based on a latest display viewpoint;
An image display system comprising: generating, as learning images, images representing how each scene in a three-dimensional displayed world, in which a situation changes in response to a user operation, is viewed from a plurality of viewpoints;
acquiring a plurality of types of 3D scene information representing three-dimensional information of each scene by machine learning using the learning images as training data;
transmitting data of the plurality of types of 3D scene information to a client terminal that renders a display image using the 3D scene information in the order in which machine learning for each scene is completed;
A display data transmission method comprising: acquiring information on a user operation and information on a display viewpoint for a three-dimensional display world;
acquiring from a server a plurality of types of 3D scene information data representing three-dimensional information acquired by machine learning for each scene in the displayed world whose situation changes in response to the user operation;
Rendering at least a portion of a frame of a display image using the most recently acquired 3D scene information based on an updated viewing viewpoint;
A display image generating method comprising: A function of generating images for learning that represent each scene in a three-dimensional display world, the situation of which changes in response to user operations, as seen from multiple viewpoints;
A function of acquiring a plurality of types of 3D scene information representing three-dimensional information of each scene by machine learning using the learning images as training data;
A function of transmitting the data of the plurality of types of 3D scene information to a client terminal that draws a display image using the 3D scene information in the order in which machine learning for each scene is completed;
A computer program characterized by causing a computer to execute the above. A function for acquiring information on user operations and information on a display viewpoint for a three-dimensional display world;
A function of acquiring from a server a plurality of types of 3D scene information data representing three-dimensional information acquired by machine learning for each scene in the displayed world whose situation changes in response to the user operation;
Rendering at least a portion of a frame of a display image using the most recently acquired 3D scene information based on an updated display viewpoint;
A computer program characterized by causing a computer to execute the above.

Images