DE102017009121A1 - Prioritizing tile-based virtual reality video streaming using adaptive rate allocation - Google Patents

Abstract

The present disclosure includes methods and systems for streaming a high performance virtual reality video using adaptive rate allocation. In particular, an adaptive rate allocation system divides a panoramic video into segments or tiles and assigns priority to each tile or segment based on an input (eg, a field of view of a field of view) from a user-client device. Further, the adaptive rate allocation system streams each tile or segment to the user-client device according to the adaptive rate allocation, which maximizes bandwidth efficiency and video quality. In this way, the adaptive rate allocation system provides higher quality content to areas in the panoramic video that a user is currently most likely / most likely to see.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of U.S. Provisional Patent Application No. 62 / 430,313 filed on 5 December 2016, which is hereby incorporated by reference in its entirety.

BACKGROUND

Advances in computing hardware and networking technologies, including the availability of high bandwidth, have forced the development of network-based multimedia applications (online video games, streaming movies, etc.). Recently there is a significant interest in the use of three-dimensional (3D) media, such as e.g. 3D immersion and virtual reality video applications. Virtual reality videos are haunting videos that have been subjected to mapping to a 3D geometry that allows a user to look around while playing using a head-mounted virtual reality display (HMD), giving the viewer a sense of Gives depth in every direction.

Despite the recent advances in virtual reality video technology, traditional systems have many challenges in delivering virtual reality content at high quality levels. Virtual reality videos, the frequent 360 Including grade videos can be extremely bandwidth intensive and difficult to stream at acceptable quality levels. Accordingly, with virtual reality video technology, the main challenge is how to efficiently transmit the large virtual reality video streams requiring high bit rates to bandwidth constrained devices, such as a wireless virtual reality HMD. For example, 4K video resolution is increasingly seen as the minimum functional resolution for current virtual reality video systems, while future virtual reality video systems anticipate 8K, 12K, and higher resolutions. However, even at current resolutions, conventional systems have difficulty streaming at acceptable quality levels.

In response to this problem, many conventional systems limit streams for reducing resolutions as compared to their two-dimensional (2D) counterparts. Other conventional systems stream virtual reality video using a wide range of predetermined resolutions for different portions of each virtual reality video. However, this requires a large storage capacity because each section must be stored in multiple resolutions, which is a solution that is not useful as the resolutions for virtual reality videos increase. Further, some conventional systems ignore virtual reality video streaming standards, which hampers streaming across a plurality of types of virtual reality video devices (e.g., HMDs). These and other problems exist with respect to conventional virtual reality systems.

SHORT SUMMARY

One or more embodiments of the present disclosure provide or provide utility and / or solves or solves one or more of the above or other problems in the art with systems and methods that stream high performance virtual reality video through adaptive rate allocation. In particular, in one or more embodiments, the disclosed systems and methods divide a panoramic video into segments and / or tiles. These systems and methods then prioritize each segment and / or tile. Further, the systems and methods stream each segment and / or tile to the user-client device based on the assigned priorities. In this way, the systems and methods transmit higher quality content to areas in the video that a user is currently viewing / most likely to view a user, and transmit lower quality content to out-of-sight areas user. As a result, the systems and methods provide significant bandwidth savings.

Further, in one or more embodiments, the disclosed systems and methods receive a panoramic video for streaming to a user-client device. The disclosed systems and methods then divide the panoramic video into tiles that are geometrically aligned within the panoramic video. Next, the disclosed systems and methods receive an indication from a client device indicating a viewport (eg, view area information or field of view) shown within the panning video (ie, what the user views within the panning video). In addition, the disclosed systems and methods prioritize each of the tiles to the base of the view area information. In addition, for each tile, the systems or methods determine or assign a bit rate that corresponds to the priority assigned to the tile. The systems and methods then stream each tile to the client device using the bit rate assigned for the given tile.

Further, the disclosed systems and methods may utilize adaptive bit rate allocation to maximize bandwidth efficiency and video quality. For example, in one or more embodiments, the disclosed systems and methods determine a minimum video quality and bit rate necessary to stream all the tiles of the panoramic video to the user client device. Then, the disclosed systems and methods using the additional available bandwidth heuristically allocate rates based on field of view priorities until a bandwidth limit is reached and the bandwidth efficiency (e.g., throughput) with the client device is maximized.

Accordingly, the systems and methods disclosed herein achieve a balance between the need for high quality virtual reality video and limited bandwidth with minimal negative impact on a user's experience. As will be described in detail below, the disclosed systems and methods provide a semantic link between a user's field of view (eg, field-of-view field of view), virtual reality video spatial partitioning, and stream prioritization to facilitate bandwidth management a virtual reality high-performance video streaming ready.

Additional features and advantages of one or more embodiments of the present disclosure are set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of such example embodiments.

list of figures

• 1 zeigt ein schematisches Diagramm, das eine beispielhafte Umgebung zeigt, in der ein adaptives Ratenzuweisungssystem gemäß einer oder mehrerer Ausführungsform(en) implementiert werden kann.
• 2 zeigt ein beispielhaftes sphärisches bzw. räumliches Video (sowohl 3D als auch eben) mit einem Ansichtsbereich.
• 3A-3B zeigen ein beispielhaftes sphärisches bzw. räumliches Video, das in 3D-Netzsegmente oder Kacheln gemäß einer oder mehreren Ausführungsform(en) aufgeteilt ist.
• 4 zeigt ein beispielhaftes sphärisches bzw. räumliches Video, das in sechs Kacheln einer Kugel mit sechs Flächen gemäß einer oder mehreren Ausführungsform(en) aufgeteilt ist.
• 5A-5B zeigen ein beispielhaftes sphärisches bzw. räumliches Video, dem eine Priorität auf der Basis einer Nutzereingabe (z.B. eines Ansichtsbereichs) von einer Nutzer-Clientvorrichtung gemäß einer oder mehrerer Ausführungsform(en) zugeordnet ist.
• 6 zeigt ein beispielhaftes Diagramm von verschiedenen Auflösungen, die das adaptive Ratenzuweisungssystem zum Streamen von Kacheln eines räumlichen Videos zu einer Clientvorrichtung gemäß einer oder mehrerer Ausführungsform(en) nutzt.
• 7 zeigt ein Ablaufdiagamm des Streamens eines virtuelle Realität-Videos mittels des adaptiven Ratenzuweisungssystems gemäß einer oder mehrerer Ausführungsform(en).
• 8 zeigt eine Beispielausführungsform des adaptiven Ratenzuweisungssystems gemäß einer oder mehrerer Ausführungsform(en).
• 9 zeigt ein Flussdiagramm einer Reihe von Vorgängen in einem Verfahren zur Bereitstellung eines bandbreiteneffizienten virtuelle Realität-Videos gemäß einer oder mehrerer Ausführungsform(en).
• 10 zeigt ein Blockdiagramm einer beispielhaften Rechenvorrichtung gemäß einer oder mehrerer Ausführungsform(en).

The detailed description refers to the attached drawings given below.

• 1 FIG. 12 is a schematic diagram illustrating an exemplary environment in which an adaptive rate allocation system may be implemented according to one or more embodiments.
• 2 shows an exemplary spherical or spatial video (both 3D and even) with a viewing area.
• 3A - 3B show an exemplary spherical video that is divided into 3D mesh segments or tiles according to one or more embodiments.
• 4 FIG. 12 shows an exemplary spherical video divided into six tiles of a six-surface sphere according to one or more embodiments.
• 5A - 5B show an exemplary spherical video associated with a priority based on a user input (eg, a viewing area) of a user-client device according to one or more embodiments.
• 6 FIG. 12 shows an exemplary diagram of various resolutions that the adaptive rate allocation system uses to stream spatial video tiles to a client device according to one or more embodiments.
• 7 FIG. 10 shows a flowchart of streaming a virtual reality video by means of the adaptive rate allocation system according to one or more embodiments.
• 8th FIG. 12 illustrates an example embodiment of the adaptive rate allocation system according to one or more embodiments.
• 9 FIG. 12 is a flowchart of a series of operations in a method of providing bandwidth efficient virtual reality video according to one or more embodiments.
• 10 FIG. 12 is a block diagram of an example computing device according to one or more embodiments. FIG.

DETAILED DESCRIPTION

One or more embodiments of the present disclosure include or include an adaptive rate allocation system that includes bandwidth management for achieving high performance virtual reality video streaming. In particular, in one or more embodiments, the adaptive rate allocation system splits a virtual reality panaromavideo into a plurality of segments, prioritizes each segment based on a user's current field of view, and streams each segment based on the segment's priority and a corresponding one rate assignment. In this way, the adaptive rate allocation system enhances the streaming of the virtual reality panoramic video beyond that of conventional systems. Further, the adaptive rate allocation system transmits higher quality content to areas in the panoramic video that a user is currently viewing / most likely to view a user, as well as lower quality content to areas outside of the user's gaze.

In particular, the adaptive rate allocation system accesses or obtains a virtual reality panoramic video (or simply "panoramic video") that is transmitted by streaming to a client device (e.g., a virtual reality display device). The adaptive rate allocation system divides the panoramic video into segments, e.g. Tiles on, which are geometrically aligned within the panorama video. Next, the adaptive rate allocation system obtains view area information from the client device indicating a current position of the viewport (e.g., field of view) within the panoramic video. Based on the view area information, the adaptive rate allocation system prioritizes each of the segments. For example, the adaptive rate allocation system assigns higher priority to segments that overlap the view area or adjoin the view area than segments farther away from the view area. Further, the adaptive rate allocation system determines a rate (e.g., a bit rate) for each segment that corresponds to the assigned priority. The adaptive rate allocation system then streams each segment at the appropriate rate to the client device.

As mentioned above, the adaptive rate allocation system may determine a rate (e.g., a bit rate) for each panoramic video segment having an associated priority. The particular rate then dictates how the segment is streamed to the client device. In one or more embodiments, the adaptive rate allocation system utilizes a heuristic rate allocation algorithm to determine the rate for each segment. As mentioned above, the bandwidth resources are often limited, particularly when the client device wirelessly receives the panoramic video. As such, the adaptive rate allocation system utilizes adaptive rate allocation to maximize bandwidth efficiency and video quality.

In particular, the adaptive rate allocation system determines a minimum acceptable video quality for sending a panoramic video to the client device. The adaptive rate allocation system also determines a minimum bit rate that corresponds to the minimum acceptable video quality. The adaptive rate allocation system then identifies the amount of bandwidth required to stream each segment at the minimum bit rate (e.g., with the minimum acceptable video quality). Frequently, additional bandwidth is available beyond the minimum bit rate. Therefore, the adaptive rate allocation system identifies the amount of additional bandwidth between a bandwidth limit or a bandwidth budget and the minimum bit rate. Using the additional bandwidth, the adaptive rate allocation system may stream selected segments to the client device at higher video qualities.

In some example embodiments, based on the priority assigned to the segment, the adaptive rate allocation system determines which segments to stream with higher video quality (e.g., at a higher bit rate). As described below, the adaptive rate allocation system assigns priority to segments based on what a user is viewing (i.e., the viewing area) and the likelihood of what a user is considering next. For example, the adaptive rate allocation system streams segments with the highest assigned priority at the highest bitrate if sufficient additional bandwidth is available so that the current viewing of the user is provided to the user with the highest video quality. Accordingly, the adaptive rate allocation system may stream other segments with different assigned priorities at different bit rate levels (corresponding to video quality levels) based on the available additional bandwidth. In this way, the adaptive rate allocation system determines how to stream the video with the highest possible quality while maximizing bandwidth efficiency.

As mentioned above, in some example embodiments, the adaptive rate allocation system splits a panoramic video into tiles. For example, the adaptive rate allocation system splits a panoramic video into six-area rooms that include an upper tile, a lower tile, and four middle body tiles. For example, if the panning video is a "sperical video" (ie, a 360 degree video), then the adaptive rate allocation system divides the panning video into a six-area sphere, which will be described below with reference to FIGS 3B shown and described in more detail. Using the tiles in the six-area rooms, the adaptive rate allocation system allocates priorities based on the tile or tiles that the user is currently viewing, as indicated by the view area. The adaptive rate allocation system may then determine the bit rate at which each tile is to be streamed based on the combination of the assigned priority of a tile and the available bandwidth.

As has been mentioned, the adaptive rate allocation system may assign tiles a priority based on the viewport. The viewport provides an indication of the field of view of a user as well as the viewing direction of a user within a panoramic video. For example, in one or more embodiments, the adaptive rate allocation system allocates the highest priority to segments that overlap the view area. In addition, the adaptive rate allocation system assigns the second highest priority to segments adjacent to the view area and lower priority to segments farther from the view area. In some example embodiments, the adaptive rate allocation system assigns a priority based on what a user is likely to focus on next (e.g., an anticipated future position of the viewing area). For example, the adaptive rate allocation system assigns a lower priority to a segment that includes the sky (e.g., the top tile), which is unlikely to be viewed by the user.

In some example embodiments, the adaptive rate allocation system may add one or more digital enhancements to a segment before the segment is streamed to a client device. For example, the adaptive rate allocation system may insert an interactive 3D object into the spatial video. As another example, the adaptive rate allocation system enhances the spatial video with an advertisement that is intelligently located within a segment or within a tile.

As briefly described above, the media allocation system provides a number of advantages over conventional systems. For example, the adaptive rate allocation system provides enhanced bandwidth management while providing a user with high quality virtual reality video that enhances a user's virtual reality experience. In particular, the adaptive rate allocation system improves the efficiency of transmitting and streaming a spatial video to a client device. In some embodiments, the improved bandwidth efficiency compared to traditional standard cases resulted in bandwidth savings of up to 72%.

As the adaptive rate allocation system improves the spatial video streaming efficiencies, the adaptive rate allocation system results in less overall data processing and lower computational resources required on both the server transmitting the spatial video and the client device receiving the video. For example, the server device sends segments and the client device receives segments based on the user's current field of view within the spatial video. In other words, the adaptive rate allocation system allows the server device to more quickly search, locate, and send the segments that are currently most significant to the user. In addition, the client device receives and processes segments that are most significant to the user at the moment, reducing the burden and processing resources required on the client device (which are often limited). Furthermore, this benefit increases exponentially as the adaptive rate allocation system constantly repeats these actions throughout the user's streaming experience.

As another benefit, the adaptive rate allocation system provides more flexibility. In particular, the adaptive rate allocation system operates using known standards, such as Dynamic Adaptive Streaming over HTTP (DASH). As such, the adaptive rate allocation system works with most types of virtual reality display devices. In this way, the adaptive rate allocation system allows the systems and methods described below to operate on most client devices, unlike conventional systems that require the user to acquire a particular type of client device. Additional benefits, benefits, features and Features of the adaptive rate allocation system are described below with reference to the figures, which describe one or more embodiments of the adaptive rate allocation system.

With reference to the figures, FIG 1 a schematic diagram showing an example environment 100 shows in which an adaptive rate allocation system 104 can work. As shown, the example environment includes 100 one or more server device (s) 102 on which a multimedia streaming system 103 part of which is the adaptive rate allocation system 104 can be to aid in efficient streaming. The environment 100 may optionally include one or more content provider device (s) 106 and a client device (ie, a virtual reality display device 110 ). The one or more server device (s) 102 containing one or more content provider device (s) 106 and the virtual reality display device 110 may be through a network 112 communicate with each other, which stands for any suitable network over which the computing devices can communicate. Example networks will be described below with reference to FIGS 10 discussed in more detail.

As it has just been mentioned, the environment includes 100 the one or more server device (s) 102 , The one or more server device (s) 102 may or may generate, store, receive, and send any type of data, including virtual reality content and user input. For example, the one or more server devices send or send 102 Data about client devices, such as the virtual reality display device 110 , In an example embodiment, the one or more server devices include or include 102 a content server. In some example embodiments, the one or more server devices include or include 102 a communication server or a web hosting server. In addition, the one or more server devices may or may not 102 one or more of the following in relation to the 10 include computing devices described.

As it is in the 1 includes or includes the one or more server devices. 102 a multimedia streaming system 103 , The one or more server device (s) 102 can the entire multimedia streaming system 103 or part of it. The multimedia streaming system 103 can an application running on the one or more server device (s) 102 is running, or part of a software application that is powered by one or more server devices. 102 can be downloaded. For example, the multimedia streaming system includes 103 A web hosting application that enables client devices to interact with content that resides on one or more server devices. 102 is hosted. For example, in one or more embodiments, the environment is intervening 100 the virtual reality display device 110 to a web page providing a virtual reality experience (eg, comprising a spatial video) derived from the one or more server devices. 102 is supported. The virtual reality display device 110 operates an application that allows a user to access content stored on one or more server devices. 102 is hosted, accessed, viewed, and / or interacted with. Examples of multimedia streaming systems 103 include video streaming services, television streaming services, video game streaming services, etc.

In addition, the multimedia streaming system 103 the adaptive rate allocation system 104 include. The adaptive rate allocation system 104 Can it be the multimedia streaming system 103 allow streaming virtual reality content efficiently to client devices. The term "virtual reality content" as used herein refers to information that can be presented to a user by means of a virtual reality display device. In particular, the term "virtual reality content" includes digital content presented in a spatial video. Accordingly, in one or more embodiments, the virtual reality content refers to an image or video that includes a field of view that extends beyond the typical field of view of a pair of human eyes (eg, 180 °). In some embodiments, the virtual reality content includes 360 degree audiovisual content or, in other words, 360 degree content of a horizontal field of view. Virtual reality content items may include a digital image, a video, a website, a web page, a user interface, a menu item, a tool menu, a magazine, a slideshow, an animation, a social post, a comment, a blog, a data feed, audio data , Advertising, a vector graphic, a bitmap, a document, any combination of one or more of the foregoing, or other electronic content.

The terms "panoramic video" or "immersive video" generally refers to a video that has a larger field of view (eg, horizontal, vertical, or diagonal) than the one that can be displayed at once on a virtual reality display. As used herein, a panoramic video shows more than 180 degrees of a horizontal field of view and more than 90 degrees of a vertical field of view. For example, a spatial video or 360-degree video is an example of a panoramic video. About that In addition, a panoramic video includes a video captured by one or more digital cameras that displays a representation of content in front of, behind, and beside the digital cameras. Alternatively, in one or more embodiments, a panoramic video refers to a set of a plurality of videos (eg, captured by one or more of a plurality of digital cameras) combined or interconnected to provide an enlarged field of view.

The multimedia streaming system 103 and the adaptive rate allocation system 104 may include computer-executable instructions that, when executed by the one or more server devices, 102 , the client device 108 and / or the virtual reality display device 110 cause the corresponding computing device to perform a number of actions, as discussed in greater detail below.

It also includes, as in the 1 shown is the environment 100 optionally also one or more content provider device (s) 106 , The one or more content provider device (s) 106 may or may include various computing devices. In particular, the one or more content provider device (s) may or may not 106 Comprise computing devices operable to generate virtual reality content. For example, the one or more content provider devices include or include 106 one or more computing devices that provide or provide spatial video or other virtual reality content. The one or more content provider device (s) 106 may include one or more computing devices that are discussed below in connection with US Pat 10 is or are described. The content provider devices 106 can for the multimedia streaming system 103 provide content for streaming to the client devices. For example, the multimedia streaming system 103 in one or more embodiments, a third party service that assists a content provider in distributing a content.

As it is in the 1 shown includes the environment 100 a virtual reality display device 110 , The virtual reality display device 110 is a client device and may include various computing devices. For example, the virtual reality display device includes 110 in one or more embodiments, one or more computing devices described below in connection with U.S. Pat 10 is or are described.

In particular, the term "virtual reality display device" refers to a computing device that generates a digital 3D representation of a virtual environment. For purposes of illustration, the term "virtual environment" refers to a simulation of a physical presence in an environment or location (real or imaginary, regardless of size). For example, a virtual environment may include a simulation of a physical presence in a building, city, market, forest, planet, space, cell or particle.

A virtual reality display device includes a computing device (eg, a headset or head-mounted display, a mobile phone, a computer, a tablet, a laptop) that generates a digital 3D view of a virtual environment that senses the physical presence of a user simulates a generated sensory experience (eg a haunting visual display, speakers, etc.). For example, a virtual reality display device may include a head-mounted display that generates a haunting 3D representation of a virtual environment that modifies a view of the virtual environment based on user movement to simulate the physical presence in the virtual environment. In addition, a virtual reality display device may also include one or more sensors that detect user interaction. For example, a virtual reality display device may include optical sensors, clothing, gloves, or other sensors for detecting movements and / or hand gestures of a user. Accordingly, the virtual reality display device may include a user input device such as a user input device. a controller, a remote controller, a joystick, a keyboard, a voice command device.

In some example embodiments, the virtual reality display device includes 110 a computing device (computer, digital receiver, game console, etc.) and a separate headset, such as a wired or wireless head-mounted display. In other embodiments, the virtual reality display device includes 110 a device with an integrated display. The virtual reality display device 110 provides a view of a virtual environment via the head-mounted display and modifies the view of the virtual environment based on user movement to simulate the physical presence in the virtual environment.

Whether that 1 shows a particular arrangement of the devices, various additional arrangements are possible. For example, while the 1 a single client device 108 shows, includes the environment 100 in one or more embodiments, a plurality of client devices, each having a virtual reality display. In another example, the client device is 108 directly with the one or more server devices 102 connected, the network 112 is bypassed.

Accordingly, although the environment 100 from 1 is shown to have different components, the environment 100 have additional or alternative components. For example, the adaptive rate allocation system 104 be implemented on a single computing device. Alternatively, the adaptive rate allocation system 104 be implemented via a plurality of devices or components. Furthermore, the adaptive rate allocation system 104 on one of the multimedia streaming system 103 operated separate device.

For example, the multimedia streaming system identifies 103 in one or more embodiments, a virtual reality content (ie, a panoramic video) for streaming to the virtual reality display device 110 , For example, the adaptive rate allocation system takes effect 104 to a spatial video of the virtual reality content. The adaptive rate allocation system 104 Splits the spatial video into segments, such as a sphere with six faces that includes six tiles (eg, an upper tile, a lower tile, and four middle body tiles). For each tile generates the adaptive rate allocation system 104 different video quality levels that correspond to streaming the tiles at different bitrates. While some embodiments include dividing a spatial video into a six-surface sphere having six tiles, other embodiments may include other types of panoramic video into six or a different number of tiles. Further, one or more embodiments may include dividing a spatial video into more or less than six tiles. Accordingly, in one or more embodiments, the adaptive rate allocation system may associate each tile with any number of classes and / or bit rates. For example, the adaptive rate allocation system allocates each tile to one of four priority classes. Alternatively, the adaptive rate allocation system uses a set of twelve priority classes or a different number of priority classes.

Continuing the above example, the one or more server devices receive or receive 102 an input from the client device indicating the view area. As used herein, the term "viewing area" refers to a field of view that is displayed to a user within a virtual reality display. As such, a virtual reality display shows both a portion of the virtual reality content within a viewing area (eg, a displayed portion) and a virtual reality content outside the viewing area (eg, a peripheral portion). In other words, a virtual reality display includes both sections that are currently visible within the viewing area and in front of the user, as well as portions of the virtual reality content adjacent and behind the user that are outside of the field of view associated with the viewing area. A viewport may include information indicating the position of the viewport, such as the coordinates of the viewport within the spatial video. As such, the view area information indicates the displayed portion and the peripheral portion with respect to the spatial video.

Based on the viewing area, adaptive rate allocation system 104 assigns priorities to each tile in the six area sphere. In addition, the adaptive rate allocation system determines 104 the bitrate at which each tile should be streamed, based on the priority assigned to each tile and the available bandwidth. Then, the one or more server devices stream or stream 102 each of the tiles to the client device 108 with the bitrate coming through the adaptive rate allocation system 104 has been determined. In a number of embodiments, the adaptive rate allocation system splits a spatial video into segments or tiles before receiving the view area information. Further, in some example embodiments, the adaptive rate allocation system may generate and store different quality levels for each tile. Alternatively, the adaptive rate allocation system splits and generates 104 Segments for real-time spatial video based on the position of the viewport.

It will now be on the 2 Reference is made, which shows how a spatial video is subjected to a mapping from a 3D sphere to a planar 2D image. In particular, the shows 2 an example of a spatial video shown as both 3D sphere 202a and mapped planar 2D image 202b. In addition, the includes 2 a viewport 204a which is shown in the 3D sphere 202a and a corresponding viewing area 204b which is shown in the planar 2D image 202b.

As mentioned above, a spatial video (eg, a 360 degree video) is an embodiment of a panoramic video. For a simple explanation, the 2 to 7 described as spatial videos. It should be noted, however, that the techniques, principles, and actions described with respect to the spatial videos apply equally to other types of panoramic video. Accordingly, the describe 2 to 7 Balls with six surfaces. It should also be noted that the techniques, principles, and actions described with respect to six-surface spheres apply equally to other configurations of tiles or even six-surface shapes.

As a brief overview, spatial video provides a haunting experience for a user through the use of texturing. The adaptive rate allocation system 104 uses textures to provide a surface wrap for a 3D object mesh, where 3D textures are a logical extension of traditional 2D textures. Spatial videos are created by mapping a spatial video as a 3D texture onto a 3D geometry mesh, often a sphere, with the user in the center of that geometry. Accordingly, the shows 2 the 3D sphere 202a of a given spatial video frame, where a box 206 represents the position of the user. It is noted that while the 2 shows a given frame of a spatial video, the same concepts and principles described herein also apply to spatial videos having any number of frames.

As shown in the 3D sphere 202a, the viewing area represents 204a a 90 degree viewing angle in both the vertical and horizontal directions. For example, the box represents 206 in the center of the 3D sphere 202a is the orientation of a user and the viewing area 204a represents the direction in which the user looks. In particular, the box represents 206 the position of a virtual reality display through which the user views the spatial video. Because the orientation and / or the position of the box 206 changes or changes (for example, if the user moves the position of the virtual reality display), the position of the view area also changes 204a on the 3D sphere 202a, resulting in a change in what the user is viewing within the spatial video.

Accordingly, the view area 204b in the planar 2D image 202b, the same 90 degree viewing angle is displayed on the mapped flat 2D image 202b 204b capturing only a portion of the total available content of the spatial video (eg, 90 degrees of 360 degrees), the user can move freely within the spatial video to view additional content as the spatial video is played. As the user changes their viewing within the spatial video, the position of the view area moves 204b within the planar 2D image 202b. For example, the view area is moving 204b within the planar 2D image 202b in response to a change in the position of the virtual reality display by the user through which the user views the spatial video, left / right and up / down.

In some example embodiments, the viewing area is greater than a 90-degree viewing angle, such as a 96-degree viewing angle, or less than the 90-degree viewing angle. During the viewing area 204b In addition, in some example embodiments, the viewport is a rectangle, an oval, overlapping circles, or another shape. Further, it should be noted that while only one viewing area is described, showing the full field of view available to a user through a virtual reality display when the user views the virtual reality content in stereo (eg, in 3D), some embodiments include a separate viewport for each stereo stream of 3D virtual reality 3D video.

As mentioned above, the adaptive rate allocation system 104 may receive information about the position and orientation of the viewing area (eg, view area information) within the spatial video. As an overview of the viewing area tracking, the viewing area orientation is usually determined by means of a rotation with respect to the x-axis, y-axis and z-axis using Euler angles. Another method involves the use of a quaternion comprising a unitary rotation matrix. In particular, a quaternion consists of two sets of values: First, a set consisting of a [x, y, z] tuple, which represents the axis about which the device rotation takes place, and secondly a w component, which determines the extent of the Rotation that takes place around this axis. By means of these four values, the adaptive rate allocation system 104 accurately describe the device orientation while avoiding problems such as gimbal blockage.

In particular, the adaptive rate allocation system generates 104 unit area quaternion valid restrictions for tracking a view area set specifically for each of the six-area sphere 3D mesh segments by the following three steps. First, the adaptive transforms Rate allocation system 104 Euler angle into a unitary quaternion representation for virtual reality display alignment tracking. Next, calculate the adaptive rate allocation system 104 an array corresponding to a normalized direction vector of each quaternion. Finally, the adaptive rate allocation system combines 104 the values together to compute the constraints of segment-specific quaternion representations within the six-sphere sphere (ie, six-surface spaces). In this way, the adaptive rate allocation system 104 identify which 3D mesh segment or tile targets a user's viewport, and also identify movement of the viewport when the spatial video plays.

With reference to the 3A and 3B Additional details regarding the generation of spatial video segments are provided in accordance with one or more embodiments. In particular, the show 3A and 3B an example of a spatial video that is split into 3D mesh segments or tiles. For example, the shows 3A a 3D ball net 302a, which is divided into a plurality of mesh segments 304a-c is divided. Then the adaptive rate allocation system 104 use the segments to further divide the mesh into tiles. For example, the adaptive rate allocation system 104 create a sphere 302b with six faces based on the 3D mesh segments, as described below.

Before the procedure of the adaptive rate allocation system 104 describing a sphere having six surfaces comprising tiles, an outline of the generation of the 3D sphere mesh 302a will be given. In one or more embodiments, the adaptive rate allocation system generates 104 the 3D sphere mesh 302a based on a quality parameter. The quality parameter can contribute to a trade-off between smoothness and rendering speed. For example, if the quality parameter indicates increased smoothness, the spatial video renders more triangles, creating a higher quality spatial video. If the quality parameter indicates a higher rendering speed, the spatial video renders fewer triangles, making the spatial video look worse and less smooth.

In particular, the quality parameter is a combination of two subparameters that together control the smoothness of the sphere. The first subparameter is the number of slices representing the number of vertical slices (e.g., vertical grid lines) through the spatial video. The second sub-factor is the number of stacks representing the number of horizontal slices (e.g., horizontal grid lines) by the spatial video.

The following listing 1 represents a pseudo-code for generating the 3D sphere mesh 302a. It should be noted that other methods of generating the 3D sphere mesh 302a are available. Further, in embodiments where the panoramic video is not spatial video (eg, the panoramic video has a horizontal 180 degree or 270 degree field of view), the adaptive rate allocation system may be used 104 Apply appropriate techniques as stated below for a spatial video. For example, the adaptive rate allocation system generates 104 a complete 3D mesh network and then ignore or eliminate sections of the 3D mesh that have no content. In another case, the adaptive rate allocation system decreases 104 first the 3D mesh calculations to adjust the field of view of the panorama video. Listing 1 Create a 3D ball net M: number of stacks N: number of slices For all m: 0 ≤ m ≤ M and for all n: 0 ≤ n ≤ N-1, calculate and store a spatial point P (x, y, z) such that: P _X ← Sin (π × m / M) × Cos (2π × n / N) Py ← Sin (π × m / M) × Sin (2π × n / N), and P _Z ← Cos (π × m / M). Then, line segments are drawn between each vertex to create the 3D mesh.

As mentioned above, the quality parameter provides a trade-off between quality (ie, smoothness) and rendering speed. For example, using too many stacks and slices is wasting hardware resources, such as memory used for graphics processing unit (GPU) -supported video rendering, which can lead to a reduction in performance without greatly improving quality. On the other hand, using too few stacks and slices results in a blocky and granular quality. Accordingly, the adaptive rate allocation system chooses 104 originally a higher quality parameter, but can adaptively reduce the quality parameter when hardware resources are no longer available. In this way, the adaptive rate allocation system achieves 104 a balance between maximizing available computational resources to provide the highest possible quality of spatial video while not overloading the computing device.

Using the 3D sphere mesh 302a, the adaptive rate allocation system 104 Create 3D mesh segments 304a-c. As it is in the 3A shown shares the adaptive rate allocation system 104 3D sphere mesh 302a in a two-stage process into a plurality of different 3D meshes. In a first step, the adaptive rate allocation system shares 104 the ball in three main parts: an upper cap, a middle body and a lower cap. The top cap includes nets from the top of the ball (ie, the top pole) to the stack 306 the upper cap of the middle body (+ β °). The middle body includes nets that extend between the lower mesh cap 308 and the lower cap stack (-β °). The lower cap includes mesh triangles from the lower mesh cap 308 to the lower point of the sphere (ie, the lower pole). In this way, the adaptive rate allocation system generates 104 the upper network segment 304a , the middle body network segment 304b and the lower network segment 304c.

In the second step, the adaptive rate allocation system shares 104 the middle body further in a plurality of 3D networks. In particular, the adaptive rate allocation system shares 104 the middle body mesh segment 304b in additional network segments. For example, the adaptive rate allocation system shares 104 the middle body mesh segment 304b in two or more identical sizes. For example, if the middle body mesh segment 304b is a 360-degree segment, shares the adaptive rate allocation system 104 each additional segment in 120 degree, 90 degree, 60 degree or 30 degree segments. In other words, the adaptive rate allocation system shares 104 each additional mesh segment covering ɑ °, where ɑ ° is a factor of the entire central body mesh segment 304b is (eg 30 °, 60 °, 90 ° or 120 °). Alternatively, the adaptive rate allocation system shares 104 the additional network segments from the middle body network segment 304b asymmetric.

As mentioned above, the adaptive rate allocation system generates 104 in one or more embodiments, a sphere having six surfaces comprising tiles of the spatial video. To illustrate, the 3B a sphere with six surfaces 302b which includes six tiles of the six-sided sphere 310a-c. In particular, the shows 3B an upper tile 310a , four middle body tiles 310b and one bottom tile 310c ,

To form the ball with six surfaces 302b uses the adaptive rate allocation system 104 the following values for ɑ ° and β °, ɑ ° = π / 2 and β ° = π / 4. However, it should be noted that the adaptive rate allocation system 104 can use other values of ɑ ° and β °. For example, the adaptive rate allocation system chooses 104 ɑ ° and β ° based on the settings of the virtual reality display through which the user views the spatial video. Further, since the tiling method is a general method, the same method can be applied to other geometric projections besides a 3D sphere, such as a 3D cube, or if a panoramic video is less than 360 degrees as described above.

The 4 shows an example of a 2D mapping of a spatial video, which is divided into six tiles of a sphere with six surfaces. As shown, the split spatial video includes 402 the six tiles of a sphere with six faces 310a-c concerning the 3A - 3B have been described spatially tailored. In particular, the spatial video shows 402 in the 4 the upper tile 310a , the four middle body tiles 310b and the bottom tile 310c , While the split spatial video 402 only six tiles, it should be noted that the adaptive rate allocation system 104 may generate more or fewer tiles. For example, the adaptive rate allocation system generates 104 two bottom tiles or two middle body tiles.

In some example embodiments, the adaptive rate allocation system 104 utilizes a spatial relationship description such as MPEG-DASH SRD (described below) to split the Segments. In this way, the adaptive rate allocation system 104 to divide the tiles in a logical and organized way. In addition, the spatial relationship description allows the adaptive rate allocation system 104 to identify the position of each tile with respect to each other tile and the spatial video as a whole.

With reference to the 5A . 5B and 6 In the following, an additional description will be provided regarding the determination of an adaptive rate allocation for prioritizing a segment-based (eg, tile-based) spatial video streaming. In particular, the show 5A - 5B an example of a spatial video 502 which is assigned a priority on the basis of the user's viewing area. The 6 Figure 14 shows an example diagram of the use of different resolutions for each network segment (eg, tile) of the spatial video 502 ,

As an overview is in the 5A and 5B a spatial video 502 , which is shown as a planar 2D image, into a plurality (ie, twenty) tiles 506 divided up. While the spatial video 502 that in the 5A - 5B For simplicity of illustration, it should be noted that the concepts and principles described herein also apply to a corresponding 3D spatial video. For example, the adaptive rate allocation system shares 104 the 3D ball net in twenty tiles. In addition, while the flat 2D image of the spatial video 502 If more tiles are included than they are included in the six-sided sphere, it should be noted that the tiles in the 5 - 6 described principles apply to the tiles in a sphere with six surfaces or spaces with six surfaces having any number of 3D mesh segments.

As mentioned above, shows 5A the adaptive rate allocation system 104 , the tiles 506 within the spatial video 502 Assign priorities. In one or more embodiments, the adaptive rate allocation system orders 104 Priorities based on the view area 504 to. For example, the adaptive rate allocation system orders 104 a higher priority (eg P ₁ ) tiles within the immediate view area and a lower priority (eg P ₂ and P ₃ ) to the tiles in the area outside the view area in the spatial video 502.

For example, the adaptive rate allocation system receives 104 from a client device viewport information showing the position of the viewport 504 within the spatial video 502 specify. As it is shown, the view area is located 504 near the center of the spatial video 502 , Here assigns the adaptive rate allocation system 104 the tile that covers the view area 504 includes, the highest priority (eg P ₁ ). It also arranges the adaptive rate allocation system 104 Tiles attached to the viewing area 504 border, the second highest priority to (eg P ₂ ), due to the proximity of these tiles to the view area 504 and the likelihood that the user will look at these tiles next. Finally, as shown, assigns the adaptive rate allocation system 104 a low priority (eg P ₃ ) tiles too far away from the view area 504 thus, it is less likely for a user to view these tiles next. While three priority classes (eg, P ₁ , P _2, and P ₃ ) are shown, the adaptive rate allocation system may 104 use any number of priority classes if the tiles 506 Assigned priorities.

As described above, the adaptive rate allocation system 104 Tile a priority based on the proximity of the tile to the viewport 504 assign. In some embodiments, the adaptive rate allocation system orders 104 a priority based on the location of the tile within the spatial video 502 to. For example, as it says in the 5B shown is the adaptive rate allocation system 104 Tiles at the top of the spatial video 502 the lowest priority (eg P ₃ ), based on the low probability that the user is in the spatial video 502 directed upwards.

In some example embodiments, adaptive rate allocation system 104 prioritizes tiles based on virtual reality content within the tile. For example, the adaptive rate allocation system orders 104 a tile that contains an advertisement, a high priority, even if the tile away from the viewing area 504 is present. As another example, the adaptive rate allocation system orders 104 Tiles at the top of the spatial video 502 on the basis of a missing content within these tiles (eg only blue / gray sky) the lowest priority too. It should be noted that the adaptive rate allocation system 104 may associate a tile with a priority based on a combination of previously described embodiments.

As described above, the adaptive rate allocation system orders 104 Tile priorities based on where a user is most likely to look next (for example, an anticipated future position of the viewport). Mapping higher priorities as to where the user is most likely to look next may be particularly helpful if the future anticipated tiles are not within or adjacent to the user's current viewing area (ie, field of view). For example, the adaptive rate allocation system follows 104 in some embodiments, a text / story frame that provides an indication of where future events (eg, an action, a motion, or an anticipated user interaction) will take place within the panoramic video. This assigns the adaptive rate allocation system 104 Tiles give a higher priority to future events. The level to which the adaptive rate allocation system 104 Further, raising the priority of a tile may also be based on a significance value associated with each event.

Accordingly, in one or more embodiments, the adaptive rate allocation system orders 104 Tile priorities based on object recognition. For example, the adaptive rate allocation system identifies 104 the position of objects within a panorama video (in the current or future frames of the video). Based on object properties (eg, size, position, motion, relevance to a future event, or potential level of interest), the adaptive rate allocation system may 104 increase the priority of a tile to which the object belongs.

As mentioned above, the adaptive rate allocation system 104 performs a pre-allocation of the spatial video 502 in tiles 506 through before an input regarding the position of the viewing area 504 within the spatial video 502 Will be received. This will be the view area 504 often a plurality of tiles overlap. If the view area 504 a plurality of tiles 506 overlaps, the adaptive rate allocation system 104 assign the same priority to a plurality of tiles. For example, the shows 5B that the adaptive rate allocation system 104 the highest priority P ₁ assigns four tiles to the viewport 504 overlap.

In some example embodiments, the adaptive rate allocation system unites 104 the tiles that line the viewing area 504 overlap.

For example, as in the 5B shown unites the adaptive rate allocation system 104 the four tiles that make up the viewing area 504 overlap, into a single tile. For example, the adaptive rate allocation system treats 104 the four tiles as a single tile for the purpose of a rate allocation, which is described below. In this way, the adaptive rate allocation system 104 Edge formation problems due to a quality variation of the tiles avoid the viewing area 504 overlap, and allows a smooth viewing experience.

Using the assigned priorities for each tile, the adaptive rate allocation system may be used 104 then selectively select rates to stream each tile. In particular, the adaptive rate allocation system 104 assigns a bit rate for each tile based on the priority assigned to the tile so that bandwidth efficiency is maximized. In this way, the adaptive rate allocation system 104 stream high-quality spatial video to virtual reality display devices that have limited bandwidth availability.

The following is a detailed example of how the adaptive rate allocation system 104 Can allocate rates for streaming a spatial video. It should be noted that the adaptive rate allocation system 104 may employ other / additional methods, techniques, and methods for assigning spatial video streaming rates. In particular, while the following example is an embodiment of the adaptive rate allocation system 104 It should be noted that other combination optimization solutions may be used to solve this knapsack-type problem. For example, the adaptive rate allocation system 104 use other multiple choice knapsack problem algorithms.

To assign rates to each tile within a spatial video in an optimal way, uses the adaptive rate allocation system 104 a bandwidth budget (ie, the amount of available bandwidth) for a given virtual reality display device. In one or more embodiments, the virtual reality display device indicates the bandwidth limitations of the virtual reality display device. In other embodiments, the adaptive rate allocation system communicates 104 with the virtual reality display device for identifying bandwidth limitations. For example leads the adaptive rate allocation system 104 perform a series of download tests with the virtual reality display device to identify the amount of available bandwidth. For illustrative purposes, the available bandwidth budget for a given virtual reality display device is represented as W _B. Further, in some example embodiments, the adaptive rate allocation system updates 104 Bandwidth restrictions on a regular basis to account for any changes in the bandwidth budget.

After the adaptive rate allocation system 104 may have received the amount of available bandwidth for a given virtual reality display device (eg, W _B ) may be the adaptive rate allocation system 104 then allocate spatial video tiles to maximize bandwidth. First, the adaptive rate allocation system determines 104 the amount of bandwidth required to stream each tile of spatial video at the lowest quality level. In this way provides the adaptive rate allocation system 104 ensure that each tile of spatial video is transmitted to the virtual reality display device at a quality level that is minimally acceptable. Accordingly, the adaptive rate allocation system determines 104 the minimum bitrate required to stream all tiles at their lowest bit rates to the virtual reality display, shown as W _min .

It should be noted that the adaptive rate allocation system 104 generally quality associated with bitrate. Therefore, as the quality (eg, resolution, number of rendered triangles, pixel density) of a tile increases, the bit rate required to stream the tile also increases. However, in some example embodiments, the adaptive rate allocation system may be 104 increase the bitrate of a tile without changing the quality level.

After determining the minimum required bit rate (ie, W _min ), the adaptive rate allocation system identifies 104 the amount of excess bandwidth between the bandwidth budget (ie, W _B ) and the minimum required bit rate, represented as W _s . By way of illustration, the adaptive rate allocation system determines 104 W _S as: $$W_B-W_{Min}=W_S,$$

The adaptive rate allocation system 104 then allocates the excess bandwidth (ie, W _s ) in an optimal manner based on the priority assigned to the tiles. For example, the adaptive rate allocation system determines 104 in one or more embodiments, determining whether the excess bandwidth allows for sending higher priority tiles (eg, tiles that overlap the view area) at a higher quality level. In particular, the adaptive rate allocation system determines 104 whether the excess bandwidth allows for streaming one or more higher bit rate tiles to the virtual reality display device.

For example, the adaptive rate allocation system identifies 104 every tile that is assigned to the highest priority class in the 5B is indicated with a P ₁ . The adaptive rate allocation system 104 Also determines the amount of bandwidth required to stream the highest priority tiles at the highest quality level using the highest bit rate, referred to as W _P1 . This information determines the adaptive rate allocation system 104 whether the excess bandwidth allows you to stream the highest-priority tiles at the highest bitrate, or: $$\text{If}{W}_S-{\mathrm{<figure-callout\; id="1"\; label="W\; P"\; filenames="DE102017009121A1\_0018.png"\; state="\{\{state\}\}">W\; </figure-callout>}}_P\ge \mathrm{0th}$$

If the excess bandwidth can allow the highest priority tiles to be streamed at the highest bitrate, then the adaptive rate allocation system 104 assigns the highest bitrate to the highest priority tiles. In addition, the adaptive rate allocation system 104 and determine the extent of additional excess bandwidth remaining after streaming the highest priority tiles represented as W _S1 . For example, the adaptive rate allocation system determines 104 W _S1 as: $$W_S-W_{P1}=W_{S1},$$

Using the additional excess bandwidth (eg, W _S1 ) determines the adaptive rate allocation system 104 whether additional tiles from the spatial video can also be streamed at the highest bitrate to the client device (ie, with the higher quality). For example, the adaptive rate allocation system identifies 104 every tile that is assigned to the highest priority class in the 5B is designated by a P ₂ . As described above, the adaptive rate allocation system determines 104 the amount of bandwidth required to stream the highest priority tiles at the highest quality level using the highest bit rate, referred to as W _P2 . This information determines the adaptive rate allocation system 104 whether the extra excess bandwidth allows the second highest priority tiles to be streamed at the highest bitrate, or: $$\text{If}{W}_{S1}-{\mathrm{<figure-callout\; id="2"\; label="W\; P"\; filenames="DE102017009121A1\_0017.png"\; state="\{\{state\}\}">W\; </figure-callout>}}_P\ge \mathrm{0th}$$

If the excess bandwidth allows you to stream the tiles with the second highest priority at the highest bitrate, then the adaptive rate allocation system 104 the tiles with the second highest priority the highest bit rate. The adaptive rate allocation system 104 The method of determining whether the remaining bandwidth may allow for the lower priority tiles to be streamed at the highest quality level at the highest bit rate continues. If the adaptive rate allocation system 104 determines that each of the tiles in the spatial video can be streamed using the highest bitrate, then assigns the adaptive rate allocation system 104 as such, begins to stream the tiles to the virtual reality display device according to the bit rate assignments for each tile.

In many cases, however, the adaptive rate allocation system determines 104 due to the limited bandwidth budget of a virtual reality display device that the extent of the remaining bandwidth (eg, W _S and W _S1) is insufficient to stream one or more tile (s) that is assigned a specific priority class, or (for example P ₁ and P ₂ , which are streamed at the bit rates W _P1 and W _P2, respectively). In such cases, the adaptive rate allocation system identifies 104 whether the excess bandwidth can enable streaming of one or more tiles at the second highest quality level using the second highest bit rate.

For example, it is assumed that the adaptive rate allocation system 104 each tile has the highest priority (eg P ₁ ) assigned to stream at the highest bitrate. It is further believed that the adaptive rate allocation system 104 determined that tiles, where the two highest priority level has been assigned (for example, P ₂₎ can not be streamed with the highest bandwidth for a given bandwidth budget (that is, _S1 W - W _P2 <0). The adaptive rate allocation system determines this 104 whether the tiles assigned priority P ₂ can be streamed to the second highest quality virtual reality display device using the second highest bit rate. If so, the Adaptive Rate Allocation System 104 the tiles, to which the priority P _{2 is} assigned, the second highest bit rate. Further, the adaptive rate allocation system determines 104 correspondingly, if tiles with the assigned priority P ₃ can also be streamed at the second highest bit rate for the given remaining bandwidth.

However, if the adaptive rate allocation system 104 determines that the tiles associated with priority P ₂ can not be streamed to the virtual reality display device at the second highest quality level using the second highest bit rate determined by the adaptive rate allocation system 104 whether the tiles assigned the priority P ₂ can be streamed at the third highest quality level using the third highest bit rate given the remaining available bandwidth. The adaptive rate allocation system 104 repeats this procedure for each priority class and bit rate level until the amount of remaining available bandwidth is minimized or the adaptive rate allocation system 104 determines that one or more priority classes must or must be streamed at the minimum acceptable quality level using the lowest bit rate.

In this way provides the adaptive rate allocation system 104 First, make sure that all tiles in a spatial video can be streamed to a minimum acceptable level of quality. Then fill the adaptive rate allocation system 104 the remaining available bandwidth with higher quality levels for one or more spatial video tile (s) based on the tiles' assigned priorities until bandwidth efficiency is maximized. As such, the adaptive rate allocation system 104 adaptively prioritizes the tiles in a spatial video to provide higher quality levels for the tiles, which a user is currently looking at / most likely to have a user look at, and for decreasing the quality levels for the tiles outside of a user's viewport.

In some example embodiments, the adaptive rate allocation system 104 orders minimum or maximum quality levels for particular tiles in a spatial video, regardless of the priority assigned to the tile. For example, the adaptive rate allocation system determines 104 that the top row of tiles in the 5B streaming at no more than the second highest bitrate because the top row of tiles contains small amounts of virtual reality content, even if there is excess bandwidth available. Accordingly, the adaptive rate allocation system 104 also limit tiles that are at a given distance (eg, over two tiles) from the view area. Limiting the bandwidth of one or more tiles may be the adaptive rate allocation system 104 enable higher throughput levels as quality levels increase (eg, to 8K and 12K resolutions). For example, although a specific bit rate may be assigned to a particular tile, the Adaptive Ratings Allocation System assigns 104 the tile again has a limited bitrate, which allows other tiles to be streamed at higher bitrates than they could otherwise be streamed.

As mentioned above, the adaptive rate allocation system 104 may generate and stream tiles in a spatial video at various qualities using corresponding bit rates. A quality measure is the resolution of a tile. For example, the adaptive rate allocation system streams 104 the tiles at different resolutions using appropriate bit rates based on the priority assigned to the tile and the bandwidth budget.

To illustrate, the 6 an example diagram of different resolutions that the adaptive rate allocation system 104 uses to stream tiles of a spatial video to a client device as part of a resolution reduction process. In particular, the shows 6 a resolution hierarchy 600 that have a plurality of resolutions 602a-d includes. As shown, each resolution includes a resolution level (R _H ) and a resolution width (Rw). For example, R _H indicates the number of pixels / rows in a segment or tile, and R _W indicates the number of pixels / columns in a segment or tile.

As it is in the 6 shown, take the resolutions 602a-d from a lowest first resolution 602a (eg the minimum acceptable quality level) to a highest fourth resolution 602d (eg the highest quality level) too. For example, the first resolution 602a a part of the size of the second resolution 602b , in turn, part of the size of the third resolution 602c is. Accordingly, as the size of the resolutions increases significantly, the bit rate requirement required to stream the segment must also increase significantly, which is the number of tiles that the adaptive rate allocation system 104 at the given limited bandwidth budgets of client devices. While in the 6 four proportional levels of resolution are shown, it should be noted that the adaptive rate allocation system 104 can use any number of resolutions. Furthermore, the resolutions may vary in height and / or width.

In one or more embodiments, the adaptive rate allocation system 104 First, cast each tile in a spatial video at the lowest first resolution 602a to. Then assign the adaptive rate allocation system 104 re-streaming at higher resolutions, as described above, based on the priority of each tile and the amount of available bandwidth of one or more tiles (s). For example, after the adaptive rate allocation system 104 has divided a spatial video into a sphere with six faces, which has six tiles, has the adaptive rate allocation system 104 the middle body tile that overlaps the view area will stream at the highest fourth resolution 602d to, the two middle body tiles adjoining the viewing area, a stream at the third resolution 602c to, the last middle body tile a stream at the second resolution 602b to and the top tile and the bottom tile both stream at the lowest first resolution 602a (eg the minimum acceptable resolution) too. Based on the assignment, the adaptive rate allocation system streams 104 each of the tiles at a bit rate equal to the assigned resolution.

As in the illustrated embodiment in the 6 can be shown, the adaptive rate allocation system 104 select one of four resolutions for each tile. In this case, the adaptive rate allocation system 104 create and / or store each of the four resolutions for each tile. For example, the adaptive rate allocation system performs 104 a pre-processing for generating and Save each of the resolutions for each tile in a spatial video. After assigning a resolution to a tile based on the user interaction with the spatial video, the adaptive rate allocation system takes effect 104 then to the tile and streams it at the assigned resolution. Alternatively, the adaptive rate allocation system generates 104 in real time, the resolution required for each tile, based on tile allocation.

The adaptive rate allocation system 104 can use various methods of streaming the tiles to the client device. In some example embodiments, the adaptive rate allocation system streams 104 each of the tiles in parallel using multithreading techniques to ensure that simultaneous streaming of all tiles begins. In other words, the adaptive rate allocation system 104 regardless of the priority of a tile, each of the tiles simultaneously streams to the client device. Alternatively, the adaptive rate allocation system streams 104 the tiles according to the tile priority. In these embodiments, the adaptive rate allocation system streams 104 Tiles from the highest priority to the lowest priority. In this way, the adaptive rate allocation system streams 104 Tiling to the user the user is looking at / most likely to view the user before streaming tiles that are out of the user's view.

In other embodiments, the adaptive rate allocation system streams 104 Tiles on the basis of the resolution of the tiles. For example, the adaptive rate allocation system streams 104 the minimum acceptable resolution for all tiles. Then put the adaptive rate allocation system 104 provide the next lower resolution for the tiles assigned to the next lower bit rate. The adaptive rate allocation system 104 Repeat this procedure until all assigned resolutions have been sent. It should be noted that if the adaptive rate allocation system 104 after streaming a lower resolution, stream a higher resolution for the same tile, the higher resolution scales, builds, and adds to the lower resolution (see, eg, the use of SHVC (Scalable HEVC) of H.265 for stratified streaming). This way does not resend or replace the adaptive rate allocation system 104 not any data previously sent for a tile, but instead adds something to the data that was previously sent for the tile.

In some embodiments, the adaptive rate allocation system transmits 104 all tiles at the minimum acceptable resolution for all tiles. For any remaining tiles that have a higher resolution allocation, the adaptive rate allocation system streams 104 from the tile with the highest priority to the tile with the lowest priority. This way, then, after the adaptive rate allocation system streams 104 provides a base resolution for each tile, the adaptive rate allocation system 104 from the highest resolution tiles (eg the highest priority tiles) to the second lowest resolution tiles.

A detailed embodiment of assigning resolutions and / or bit rates to tiles in a spatial video is given below. As described above, the adaptive rate allocation system shares 104 a spatial video in n tiles, represented as T = {t ₁ , t ₂ , ..., t _n ). The highest possible quality level for each t _i ∈ T has a bit rate requirement s _ti and a priority coefficient of p _ti . As mentioned above, the adaptive rate allocation system orders 104 the highest priority tiles within the immediate viewport of the user (eg P ₁ ), and the lowest priority (eg P ₃ ) the tiles in the area outside the user's viewport in the spatial video.

Moreover, the quality contribution of the tile t _i , which is as q _ti, is a function of the bit rate request of the tile (ie, s _ti ) and the priority coefficient (ie, p _ti ). Accordingly, s _ti × p _ti = q _ti . In addition, as noted above, client devices often have a limited bandwidth budget (eg, W _B ) that limits the overall bit rate for all tiles that is the adaptive rate allocation system 104 to the client device.

In this example embodiment, let X = {x ₁ , x ₂ , ..., x _i } represent a set of tiles containing the adaptive rate allocation system 104 for the client device. Further, X _i ∈ X corresponds to the original tile t _i ∈ T. Accordingly, each x _{i has} a priority coefficient p _xi = p _ti , or, in other words, the priority coefficient between the sent tiles and the original tiles is the same.

angegeben ist. Ferner entspricht das niedrigste Qualitätsniveau der niedrigsten Bitrate, die durch R_L dargestellt ist, und wobei es sich um die minimale Bitrate handelt, die Nutzer tolerieren können (z.B. das minimal akzeptable Qualitätsniveau). Ferner ist der Qualitätsbeitrag jeder Kachel x_i gleich q_xi (z.B. s_xi × p_xi = q_th). Moreover, in the example embodiment L, the number of quality levels (eg resolutions) should be represented, with each quality level denoted by k, denoted R _k ( 0 ≤ k ≤ L) and the bit rate of the tile t _i with the representation R _k as $s_{t_i}^{R_k}$

is specified. Further, the lowest quality level corresponds to the lowest bit rate represented by R _L , and this is the minimum bit rate that users can tolerate (eg, the minimum acceptable quality level). Further, the quality contribution of each tile x _{i is} equal to q _xi (eg, s _xi x p _xi = q _th ).

dargestellt ist. Zusätzlich ist, da W_min ≤ W_B, das Ausmaß der verfügbaren Bandbreite, Ws, als Ws = W_B- W_min, dargestellt, wie es vorstehend beschrieben ist. Zusätzlich sortiert in der Beispielausführungsform das adaptive Ratenzuweisungssystem 104 zur Bestimmung der optimalen Bitrate für jede Kachel die priorisierte Liste von Kacheln durch eine globale Priorität von der größten zur kleinsten. Für eine einfache Schreibweise soll angenommen werden, dass die Kacheln neu indiziert werden, so dass die sortierte Liste von Kacheln t₁, t₂, ..., t_n ist. Wenn $s_{t1}-s_{t_1}^{R_L}\le W_s,$

dann ist die überschüssige Bandbreite (d.h., W_s) ausreichend, um ein Senden von t₁ bei der höchsten Bitrate (Ro) zu ermöglichen. Die Kachel x₁ hat die Bitrate s_x1 = s_t1 und trägt q₁ zu der durchschnittlichen Qualität bei. Darüber hinaus belässt dies die zusätzliche Bandbreite überschüssig (d.h., W_s1), da $W_{s1}=W_s-s_{t_1}^{R_0}-s_{t_1}^{R_L}$

für die restlichen Kacheln nach x₁. Der vorstehende Algorithmus wiederholt sich für t₂, t₃, ..., t_n, bis das adaptive Ratenzuweisungssystem 104 eine oder mehrere Kachel(n) t_l bei der höchsten Bitrate innerhalb des restlichen Bandbreitenbudgets W_l-1 nicht streamen kann.In addition, S should be the total bit rate of streaming spatial video tiles. Then, for each tile t _i in T, calculate the adaptive rate allocation system 104 q _i as described above, which is the contribution t _i would make to the average quality when using the adaptive rate allocation system 104 t _i streams at the highest possible bit rate. The adaptive rate allocation system 104 then calculates the minimum bitrate required to stream all tiles at their lowest bitrate, called the $W_{min}={\displaystyle \Sigma s_{t_i}^{R_L}}$

is shown. In addition, since W _min ≦ W _B , the amount of available bandwidth, Ws, is represented as Ws = W _B - W _min , as described above. Additionally, in the example embodiment, the adaptive rate allocation system sorts 104 To determine the optimal bitrate for each tile, prioritize the list of tiles by a global priority from largest to smallest. For a simple notation, let us assume that the tiles are reindexed so that the sorted list of tiles t ₁ , t ₂ , ..., t _n . If $s_{t1}-s_{t_1}^{R_L}\le W_s.$

then the excess bandwidth (ie, W _s ) is sufficient to allow transmission of t ₁ at the highest bit rate (Ro). The tile x ₁ has the bitrate s _x1 = s _t1 and contributes q ₁ to the average quality. In addition, this leaves the additional bandwidth excess (ie, W _s1 ) since $W_{s1}=W_s-s_{t_1}^{R_0}-s_{t_1}^{R_L}$

for the remaining tiles after x ₁ . The above algorithm repeats for t ₂ , t ₃ , ..., t _n until the adaptive rate allocation system 104 one or more tile (s) t _l can not stream at the highest bit rate within the remaining bandwidth budget W _l-1 .

so dass $s_{t_l}^{R_{L\text{'}}}\le W_{l-1}+s_{t_l}^{R_L}.$

Die Kachel x_l weist eine Bitrate $s_{x1}=s_{t_l}^{R_{L\text{'}}}$

auf und trägt q’_l zur durchschnittlichen Qualität bei. Dann wird das restliche Bandbreitenbudget nach dem Streamen x_l gleich $W_l=W_{l-1}-s_{t_l}^{R_{L\text{'}}}$

sein. Auch hier wiederholt der Beispielalgorithmus das vorstehende Verfahren zur Bestimmung der geeigneten Bitraten, der Ausmaße der überschüssigen Bandbreite und des Bandbreitenbudgets, und den Qualitätsbeitrag für jede der restlichen Kacheln x_l+1, x_l+2, ..., x_n. Auf diese Weise maximiert das adaptive Ratenzuweisungssystem 104 die Bandbreiteneffizienz, während Kacheln eines räumlichen Videos bei gegebenem Bandbreitenbudget bei dem höchstmöglichen Qualitätsniveau gestreamt werden.The adaptive rate allocation system 104 then determines the highest possible bit rate at which it can stream to the client device by computing the lowest quality level $L\text{':}L\text{'}\le L\text{.}$

so that $s_{t_l}^{R_{L\text{'}}}\le W_{l-1}+s_{t_l}^{R_L},$

The tile x _l has a bit rate $s_{x1}=s_{t_l}^{R_{L\text{'}}}$

and contributes q ' _l to the average quality. Then, the remaining bandwidth budget becomes equal to x ₁ after the streaming $W_l=W_{l-1}-s_{t_l}^{R_{L\text{'}}}$

be. Again, the example algorithm repeats the above procedure for determining the appropriate bit rates, the amounts of excess bandwidth and the bandwidth budget, and the quality contribution for each of the remaining tiles x _{1 + 1} , x _{1 + 2} , ..., x _n . This maximizes the adaptive rate allocation system 104 the bandwidth efficiency as spatial video tiles are streamed at the highest possible quality level given a bandwidth budget.

It should be noted that in one or more cases, such as above in the listing 2 shown is the adaptive rate allocation system 104 Run a one-time implementation at the beginning of the session for the main procedure. By performing the one-time implementation, the adaptive rate allocation system 104 run in real time and does not require any overhead ("overhead") during runtime. In other words, the adaptive rate allocation system 104 can operate efficiently in O (nlogn) time and O (n) space can produce near optimal solutions. Moreover, the approximation error depends on the difference between the bit rate chosen for the first tile, which can not be received at the highest bit rate (ie, t _l ), and the remaining excess bandwidth budget. It should be noted that the dissolution deterioration is optional. In alternative embodiments, the adaptive rate allocation system degrades 104 only the quality and the total bit rate instead of a change of resolutions.

In the 7 Fig. 10 is a flowchart of streaming a virtual reality video using the adaptive rate allocation system 104 shown. As shown, the 7 a server device 102 , which is the adaptive rate allocation system 104 includes, and a virtual reality display device 110 , The adaptive rate allocation system 104 receives a spatial video. For example, the adaptive rate allocation system receives 104 a spatial video at the highest resolution, which is the adaptive rate allocation system 104 later may possibly decrease. Alternatively, the adaptive rate allocation system receives 104 a plurality of spatial video resolutions of a content provider device. In other embodiments, the adaptive rate allocation system operates 104 to a stored spatial video on the server device (s) 102 to.

After receiving a spatial video, the adaptive rate allocation system shares 104 spatially 714 the spatial video into a sphere with six faces (ie, a panoramic video) that includes a 3D mesh segment. The adaptive rate allocation system 104 select a 3D template mesh and apply (eg, perform a mapping) the 3D template mesh to the received spatial video. As part of mapping the spatial video to the 3D template, the adaptive rate allocation system translates 104 Vertex positions and texture coordinates from the 3D template to the spatial video. In this way, the adaptive rate allocation system achieves 104 a rectangular mapping of the spatial video (ie, panoramic video).

In addition, the adaptive rate allocation system shares 104 the mapped spatial video vertically and horizontally (eg slicing and stacking) using the 3D mesh transferred from the 3D template. Then share the adaptive rate allocation system 104 using the splits, segments into six 3D mesh segments (or another number of 3D mesh segments) so that the sphere is formed with six faces (ie, six-face spaces).

As it is in the 7 716, the adaptive rate allocation system 104 generates 716 a plurality of quality levels for each 3D network segment. As mentioned above, the adaptive rate allocation system streams 104 the spatial video to a client device using the at most quality level and highest bit rates possible with the given bandwidth constraints (eg, the bandwidth budget). As a result, the adaptive rate allocation system streams 104 often different segments at different quality levels (eg resolutions).

To generate a plurality of quality levels for each 3D network segment, the adaptive rate allocation system generates 104 in one or more embodiments, a copy of each resolution level for each 3D mesh segment. If, for example, four resolutions are available (see, for example, the 6 ), then creates the adaptive rate allocation system 104 three lower-resolution copies of each 3D mesh segment on the highest-resolution copy received from content provider device 706.

In some example embodiments, as mentioned above, the adaptive rate allocation system receives 104 different quality levels for a spatial video from the content provider device 706 , In these embodiments, the adaptive rate allocation system shares 104 spatially copy each copy of the spatial video into corresponding 3D mesh segments (ie, the six-faced sphere) as described above. In this way, the adaptive rate allocation system has 104 Access to the various quality levels for each 3D spatial video segment (ie, panoramic video).

As shown, 718 identifies the adaptive rate allocation system 104 with the virtual reality display device 110 a bandwidth budget. For example, send the adaptive rate allocation system 104 and the virtual reality display device 110 Test files or other data between them to determine the amount of available bandwidth and / or a bandwidth budget. It also reviews and updates the adaptive rate allocation system 104 and the virtual reality display device 110 in one or more embodiments, the bandwidth budget periodically changes as network conditions change.

In some example embodiments, the virtual reality display device 110 provides for the adaptive rate allocation system 104 a given bandwidth budget. For example, the virtual reality display device 110 their maximum bit rate reception and / or processing capability to the adaptive rate allocation system 104 further. In various embodiments, the adaptive rate allocation system 104 identifies features of the virtual reality display device 110 or a network connection and then determines the bandwidth budget based on the properties. For example, the adaptive rate allocation system determines 104 the higher bandwidth budget for a wired virtual reality display device 110 and not for a wireless virtual reality display device 110 ,

By means of the bandwidth budget, the adaptive rate allocation system 104 also receives 720 view area information from the virtual reality display device 110 , As described above, viewport information gives the direction and orientation of the virtual reality display device 110 at. As such, the adaptive rate allocation system uses 104 the view area information for identifying the coordinates of the user's viewing area within the spatial videos, which is the adaptive rate allocation system 104 continues to use to determine which 3D mesh segments overlap and are adjacent to the viewport.

The adaptive rate allocation system 104 assigns 722 a priority to the 3D mesh segments based on the received view area information. For example, the adaptive rate allocation system provides 104 determine which 3D mesh segments overlap the viewport and assign the highest priority to these 3D mesh segments. It also arranges the adaptive rate allocation system 104 the highest or next higher priority to the 3D mesh segments adjacent to the viewport. It also arranges the adaptive rate allocation system 104 3D mesh segments further away from the viewing area (eg, outside the user's peripheral view within the spatial video) are given a lower priority. The adaptive rate allocation system 104 may alternatively assign priorities to 3D network segments as described above.

Based on the assigned priorities of each 3D network segment, 724 determines the adaptive rate allocation system 104 a bit rate allocation for each 3D network segment. For example, the adaptive rate allocation system identifies 104 a minimum bit rate that corresponds to the lowest acceptable quality of the spatial video. In other words, the adaptive rate allocation system 104 determines at which bit rate the adaptive rate allocation system 104 each 3D network segment to the client device 708 at the lowest quality level. In general, when each of the 3D network segments is streamed at the lowest quality level, the bit rate required to stream the 3D network segments is lower than the bandwidth budget.

In addition, there is often an additional bandwidth available that remains for single or multiple streaming at a higher quality level. For example, the adaptive rate allocation system identifies 104 in one or more embodiments, determine the highest priority 3D network segment (s) and determines whether the excess bandwidth is streaming the 3D network segment or the highest priority 3D network segments at a corresponding (higher) bit rate allows. If so, then assign the adaptive rate allocation system 104 the one or more highest priority 3D network segment (s) the corresponding bit rate. If not, the adaptive rate allocation system determines 104 whether the highest priority 3D mesh segment (s) can be streamed at the next highest quality level and bitrate. The adaptive rate allocation system 104 For example, as described in more detail above, the method repeats until each 3D network segment in the spatial video the highest possible quality level and bit rate that maximizes bandwidth utilization for a given bandwidth budget

As shown, 726 streams the adaptive rate allocation system 104 the 3D network segments by means of the allocated bit rates. The adaptive rate allocation system 104 Streams the 3D mesh segments in parallel, serially or in combination as described above. Furthermore, the adaptive rate allocation system 104 intelligently stream each of the 3D mesh segments based on the assigned priority and assigned bit rate of a 3D mesh segment, as described above.

In addition, the adaptive rate allocation system 104 the method of receiving 720 of viewport information, of mapping 722 a priority to the 3D mesh segments, determining 724 repeating a bit rate assignment and streaming the 3D mesh segments as spatial videos (eg as step 728 is shown) when the spatial video is played. For example, if the user interacts within the spatial video (eg, changing position, touching or touching objects, etc.), that view area information will be returned to the adaptive rate allocation system 104 continuously re-prioritizing the 3D mesh segments and reassigning bit rates for the 3D mesh segments to provide the user with an optimal virtual reality experience. For example, the adaptive rate allocation system receives 104 in some example embodiments, every one to two seconds of view area updates and provides the updated virtual reality content for the virtual reality display device 110 ready as described above.

In addition, the steps and actions described herein may be performed in a different order, may be performed repetitively or in parallel, or may be performed in parallel with different instances of the same or similar steps or actions. Although the 7 shows a step in which the adaptive rate allocation system 104 a spatial video (ie, panoramic video) in a sphere with six faces (ie, a six-face space) spatially partitioned 714 with a number of 3D mesh segments, shares the adaptive rate allocation system 104 in one or more embodiments, for example, the spatial video in a plurality of tiles, as described above.

With reference to the 8th In the following, additional details regarding components and performance of an embodiment of the adaptive rate allocation system will become apparent 104 provided. In particular, the shows 8th an example embodiment of the adaptive rate allocation system 104 , As shown, the adaptive rate allocation system works 104 in the 8th on one or more server device (s) 102 , such as the one or more server devices as described above in connection with FIG 1 has been described or are.

The adaptive rate allocation system 104 includes, but is not limited to, content management 802 , a content partitioning device 804 , a priority allocator 806 , a rate allocation facility 808 , a content provider 810 and a content database 812 , Each of these components will be described in more detail below.

As mentioned above and in the 8th is shown includes the adaptive rate allocation system 104 a content management 802 , The content management 802 can get panoramic videos and other virtual reality content. For example, content management receives 802 one or more virtual reality 3D panoramic video (s) by a content provider as a third party. In some example embodiments, content management 802 create, generate or modify a virtual reality content that is included in a panoramic video.

It also includes, as in the 8th shown is the adaptive rate allocation system 104 also a content partitioning device 804 , The content splitter 804 Split a panoramic video into a plurality of 3D mesh segments and / or tiles. For example, content divider 804 creates six-area spaces (eg, a six-space sphere) for a panoramic video (eg, a spatial video) as described above. The content splitter 804 can utilize various methods and techniques for splitting a panoramic video into segments or tiles, as described above.

It also includes, as in the 8th shown is the adaptive rate allocation system 104 also the priority assignment device 806 , The priority assignment device 806 generally assigns priority to each segment or tile in a panoramic video. The priority is often based on view area information received from a user client device and / or a virtual reality display. As described in detail herein, the priority allocator orders 806 Segments that a user views (such as within the view area) have a higher priority, and segments have a lower priority outside the immediate view of the user.

As it is in the 8th is shown includes the adaptive rate allocation system 104 also the rate allocation facility 808 , In general, the rate allocation device determines 808 at which quality level (eg resolution) and with which bitrates each segment should be streamed. In particular, the rate allocation facility weighs 808 the available bandwidth in a bandwidth budget with bitrate requests and constraints. In this way, the rate allocator maximizes 808 the video quality while maximizing bandwidth efficiency while providing the highest possible video quality. Additional details regarding the determination of rate assignments are given above.

As it is in the 8th is shown includes the adaptive rate allocation system 104 also the content provider 810 , The content provider 810 generally provides the assigned segments for one or more client devices with virtual reality display. For example, the content provider streams 810 parallel or serial each segment of a panorama video at the assigned bit rate. An additional description regarding the allocated allocated segments is given above.

In some example embodiments, the content provider streams 810 a panoramic video to a client device that has been encoded and packaged using MPEG-DASH SRD (Dynamic Adaptive Streaming over HTTP (DASH) Spatial Relationship Description (SRD)), as described in describing the spatial relationship of the 3D mesh segments and / or tiles in the virtual Reality space supported. DASH, also known as MPEG-DASH, is an ISO standard that provides adaptive bitrate streaming to a client device based on limited resources, such as the amount of bandwidth available to the client (eg, the bandwidth budget). The multimedia content is provided on an HTTP server and accompanied by a "Media Presentation Description (MPD)" as a directory of available segments, their different bit rate alternatives, their URL addresses, and other properties.

Further includes, as in the 8th shown is the adaptive rate allocation system 104 also the content database 812 , The content database 812 Saves the received panorama videos and / or copies of panorama videos at different quality levels. In addition, the content database can 812 storing split segments and / or tiles at different quality levels to be streamed to a user client device.

Each of the components 802 to 812 the adaptive rate allocation system 104 (as it is in the 8th shown) may include software, hardware or both. For example, the components 802 to 812 include one or more instructions stored on a computer-readable storage medium and executable on one or more computing devices, such as a client device or a server device. When executed by the one or more processors, the computer-executable instructions of the adaptive rate allocation system may 104 cause the computing device (s) to perform the feature learning procedures described herein. Alternatively, the components can 802 to 812 Hardware such as a special processing device for performing a particular function or group of functions. Alternatively, the components can 802 to 812 the adaptive rate allocation system 104 a combination of computer-executable instructions and hardware.

Furthermore, the components can 802 to 812 for example, as one or more operating systems, as one or more standalone application (s), as one or more modules of an application, as one or more plugins, as one or more library function (s) or functions that can be invoked by other applications and / or implemented as a cloud computing model. Consequently, the components can 802 to 812 as a stand-alone application, such as a desktop application or mobile application. Furthermore, the components can 802 to 812 be implemented as one or more web-based application (s) installed on a remote server. The components 802 to 812 can also be implemented in a bundle of mobile device applications or "apps". By way of illustration, the components may be 802 to 812 to be implemented in an application, including, but not limited to, ADOBE PRIMETIME software. "ADOBE" and "PRIMETIME" are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and / or other countries.

To evaluate the effectiveness of one or more embodiment (s), the evaluators used a Samsung Gear VR HMD to which the Samsung Galaxy S7 smartphone with 4GB RAM and Android Marshmallow 6.0.1 was installed as a virtual reality target display device. The evaluators used an Oculus Mobile SDK 1.0.3 connection with the Android SDK API 24 to develop a 360 VR video streaming application prototype based on MPEG-DASH SRD and use it to apply customization and experimentation.

The virtual reality target display device provides a total resolution of 2560 × 1440 ( 1280 x 1440 per eye) with a maximum frame rate of 60 FPS and a horizontal FoV (field of view) of 96 degrees. The evaluators set the vertical VRV of the 360 VR video prototype to 90 degrees. The evaluators created five different 360 quasi-rectangular formatted sample videos as test sequences for applying various embodiments publicly available. In particular, the five sample videos had the following specifications: Sample 1 : 3840 × 1920 resolution, 20.0 Mbps original bitrate, and 30 Hz FPS; sample 2 : 3840 × 1920 resolution, 13.1 Mbps original bitrate, and 30 Hz FPS; sample 3 : 1920 × 690 resolution, 4.4 Mbps original bitrate, and 30 Hz FPS; sample 4 : 3840 × 2160 resolution, 13.0 Mbps original bit rate, and 24 Hz FPS; and sample 5 : 4096 × 2048 resolution, 12.6 Mbps original bitrate, and 24 Hz FPS.

To capitalize on the features of the MPEG-DASH SRD-based streaming and texture the six-surface mesh, the reviewers spatially cut the video and created six different tiles based on a six-sphere sphere geometry. The evaluators encoded all video segments with a H.264 AVC coder in L = 4 different representations using hierarchical resolution reduction, the original resolutions being polynomial with every step has been diminished, as in the 6 is shown. Assuming R w (τREP i), the resolution width of a specific video (and correspondingly, the resolution height Rh (τREP i)) of a specific video tile is τ for representation i, then: $\text{R}\left(т\text{REP}i\right)=\text{R}\left(т\text{REP}1\right)*\frac{\left(4-\left(i-1\right)\right)}{4},$

The evaluators used a Dell XPS 8900 x64-based PC running Microsoft Windows 7 Professional version 6.1.7601 OS as an HTTP-based streaming server and using MPEG-DASH SRD to describe tiling. For quantitative quality measurement, the quality of a 360 VR video session is a function of bit rates for all tiles (where the maximum quality corresponds to the maximum resolution for all transferred tiles). This approach is general and independent of the quality measure. In this initial study, the evaluators used the average bit rate. Evaluators have used different sets of resolutions for different tiles to experimentally determine how prioritized adjustments affect the average bit rate and perceived quality.

The evaluators have created a statistic for the average bitrate and the results with current practices 360 VR video game machines that do not use a tile-based approach. Each trial was run for a total of 30 seconds, and during each attempt, each tile is within the user's viewport for 5 seconds. The evaluators measured the bandwidth usage as the average bit rate when assigning the maximum resolution for tiles within an immediate FoV, and for two different lower resolutions, one with the lowest resolution (REP 1 ) and another with the second highest resolution (REP3) assigned to the peripheral tiles.

As a base case, no adjustment is made, the 360 VR video is tiled, there is no awareness of the viewport, and all tiles are rendered with the highest representation (REP 1 ) streamed. For the sample 1 uses REP427.7% of the base case and REP 2 uses 74.1% of the base case. For the sample 2 uses REP 4 27.8% of the base case and REP 2 uses 62.9% of the base case. For the sample 3 uses REP 4 of 32.6% of the base case and REP 2 uses 72.0% of the base case. For the sample 4 REP4 uses 30.1% of the base case and REP 2 uses 74.4% of the base case. For the sample 5 uses REP445.3% of the base case and REP 2 uses 75.6% of the base case. Thus, compared to the base case, embodiments can significantly save bandwidth usage up to 72%.

Furthermore, even the lowest representation on the peripheral tiles, which were not directly in the viewing area, resulted in minor visual changes from the user perspective and sometimes were not even noticeable, while preserving the original quality for the main viewing area, thus providing a satisfactory user experience is sichergestelllt. The tiling approach can also increase the overall quality of 360 VR videos, allowing for nearly 360 degrees total video resolution of 8K and higher in the 360 degree space. This enables streaming of tiles within the viewing area even at even higher resolutions at or near the native resolution of HMD, which previously was not possible due to the limited hardware resources processing a single large video content.

The 1 to 8th , the related text and examples provide a number of different systems and apparatus for streaming virtual reality content. In addition to the foregoing, embodiments may also be described as flowcharts that include acts and steps in a method to achieve a particular result. For example, the shows 9 a flow chart of an example method according to one or more embodiments. That in terms of 9 described method may be performed with fewer or more actions or the actions may be performed in other orders. In addition, the actions described herein may be repeated or performed in parallel with each other, or in parallel with different instances of the same or similar actions.

The 9 FIG. 12 shows a flowchart of a series of operations in a method 900 for providing bandwidth-efficient virtual reality video according to one or more embodiments. In one or more embodiments, the method becomes 900 performed in a digital media environment using the adaptive rate allocation system 104 such as a digital media environment for providing an immersive virtual reality experience. The method 900 is intended to illustrate and illustrate one or more methods in accordance with the present disclosure do not limit potential embodiments. Alternative embodiments may include additional, fewer, or different processes than those described in the 9 are shown.

As it is in the 9 is shown, the method comprises 900 a process 910 splitting a panorama video into tiles. In particular, the process can 910 splitting a panoramic video into a plurality of tiles 310a-c include. For example, the process can 910 include using a spatial relationship description to divide the panoramic video into a plurality of tiles. In some example embodiments, the panoramic video includes a spatial video or a 360-degree video. In these embodiments, the tiles are combined to form a 360 degree field of view. In other embodiments, the panoramic video includes less than a 360 degree field of view. In several example embodiments, the panoramic video is a stereo video that includes two corresponding video streams. In one or more embodiments, the process includes 910 creating one or more top tiles, one or more bottom tiles, and a plurality of center body tiles.

As with the process 910 are used in some example embodiments 3D Network segments by performing a step of spatially dividing the panoramic video into three-dimensional ( 3D ) Creates spaces with six surfaces. In these embodiments, the step includes the operations of dividing the panoramic video into a plurality of vertical stacks and a plurality of horizontal slices and dividing the panoramic video based on one or more stacks and / or slices into a plurality of 3D meshes.

It also includes, as in the 9 shown is the procedure 900 a process 920 Receiving view area information. In particular, the process can 920 receiving view area information from a client device. The process 920 may specify a view area of the view area information, and may also include position data indicating the position of the view area (eg, the field of view) within the panorama video. Furthermore, the method can 900 a process of identifying one or more tiles of the plurality of tiles that overlap the view area and one or more tiles of the plurality of tiles that are adjacent to the viewing area.

In addition, the procedure includes 900 also a process 930 assigning priorities to each tile. In particular, the process can 930 assigning, on the basis of the view area information, priorities to each tile of the plurality of tiles 506 include. In some example embodiments, the process includes 930 also assigning a maximum priority level to the one or more tiles that overlap or overlap the view area, assigning a next higher priority level to the one or more tiles adjacent to or adjoining the view area, and assigning a lower priority level to a remainder of the one or more tiles of the plurality of tiles. In one or more embodiments, the process includes 930 assigning, based on the view area, priorities to each tile of the plurality of tiles by determining a correlation between the current location of each tile within the panoramic video and the current viewing area.

It also includes, as in the 9 shown is the procedure 900 a process 940 determining a rate for each tile. In particular, the process can 940 determining, for each tile, a bit rate that corresponds to the assigned priority of the tile. In one or more embodiments, the process may 940 include assigning bit rates to tile priority levels using a multiple-choice knapsack problem algorithm. For example, the process can 940 determining the minimum bit rate for streaming the plurality of tiles to the client device, wherein the minimum bit rate corresponds to a lowest acceptable quality of the panoramic video, and wherein streaming the plurality of tiles to the client device at the minimum bit rate does not exceed a bandwidth limit. In some example embodiments, the process includes 940 determining one or more higher bit rates for streaming the plurality of tiles to the client device, wherein the one or more higher bit rates correspond or correspond to one or more higher quality levels of the panoramic video.

In one or more embodiments, determining the bit rate corresponding to the assigned priority of each tile comprises determining a first remaining bandwidth amount by subtracting the minimum bit rate for the plurality of tiles from the bandwidth limit (ie, the bandwidth budget) Determining that the first remaining bandwidth extent is the streaming of one or more allows multiple highest priority tile (s) of the plurality of tiles at the highest bitrate of the higher bitrates. In additional embodiments, determining the bit rate corresponding to the assigned priority of each tile comprises determining a second remaining bandwidth amount by subtracting the highest bit rate for streaming the one or more highest priority tile (s) from the first remaining bandwidth extent, and determining that the second remaining bandwidth extent allows for streaming one or more next highest priority tile (s) of the plurality of tiles at the next higher bit rate of the higher bit rates, the next higher bit rate being below the highest bit rate and above the minimum bit rate.

In some embodiments, a bit rate is assigned by performing a step of determining bit rate allocation for each 3D network segment based on the assigned priority of each 3D network segment of 3D network segments. For example, the step includes determining a minimum bit rate for streaming the plurality of 3D networks to the client device, wherein the minimum acceptable bit rate corresponds to a lowest acceptable quality of the panoramic video, and wherein the streaming of the 3D network segments to the client device at the minimum bit rate is not exceeds a bandwidth limit; determining a remaining bandwidth amount by subtracting the minimum bit rate for the 3D network segments from the bandwidth limit; and determining that the remaining bandwidth extent enables streaming of one or more network segments that are or are assigned a highest priority at a highest bit rate.

In one or more embodiments, the process includes 940 interactively determining a highest available bit rate to stream each tile of the plurality of tiles based on a ranking of the assigned priority of each tile and a level of the remaining available bandwidth such that any remaining bandwidth is minimized, and wherein each tile of the plurality of tiles is at least is streamed at the minimum bitrate. In some example embodiments, tiles of the same priority are streamed at the same bit rate. Alternatively, tiles of the same priority are streamed at different bitrates. In a few embodiments, determining for each tile, a bit rate corresponding to the associated tile, that higher priority tiles are assigned a higher bit rate and lower bit rate tiles are assigned a lower bit rate.

As it is in the 9 is shown, the method comprises 900 also a process 950 streaming each tile to the client device. In particular, the process can 950 streaming each tile to the client device 108 using the appropriate particular bit rate. In some example embodiments, the process includes 950 the parallel streaming of the tiles by means of a multi-threading, so that with the streaming of all tiles and / or with the playing of all tiles on the client device 108 is started at the same time. Furthermore, the client device comprises 108 in one or more embodiments, a virtual reality display device 110 such as a wireless virtual reality display.

As used herein, the term "digital environment" refers generally to an environment that may be used, for example, as a standalone application (eg, a personal computer or a mobile application running on a computing device) as a module of an application, as a plug-in. in for an application, as a library function or functions that can be retrieved by a network application generation system, implemented as a server device and / or cloud computing system. A digital media environment enables the adaptive rate allocation system 104 given a limited bandwidth constraint, provides a haunting virtual reality experience to a user.

Embodiments of the present disclosure may include or utilize a special or generic computer that includes computer hardware, such as computer hardware. one or more processors and system memory, as discussed in greater detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for containing or storing computer-executable instructions and / or data structures. In particular, one or more of the acts described herein may be implemented, at least in part, as instructions residing in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (eg, a microprocessor) receives instructions from a non-transitory computer-readable medium (eg, a memory, etc.) and executes those instructions, thereby performing one or more operations, including one or more operations, here or there is or is, is or will be.

Computer readable media can be any available media that can be accessed by a general or specific computer system. Computer-readable media storing computer-executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media containing computer-executable instructions are transmission media. Thus, by way of example and not limitation, embodiments of the disclosure may include at least two distinctly different types of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, SSDs (eg, based on RAM), flash memory, phase change memory ("PCM"), other types of memory, other optical disk storage, Magnetic disk storage or other magnetic storage devices, or any other medium suitable for storing desired program code means in the form of computer-executable instructions or data structures that can be accessed by a general or special computer.

Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures may be automatically transferred from transmission media to non-transitory computer-readable storage media (devices) (or vice versa). For example, computer-executable instructions or data structures obtained via a network or data link may be buffered in RAM within a network interface module (eg, a "NIC") and then eventually to a computer system RAM and / or less volatile computer storage media (devices). be transmitted in a computer system. Thus, it should be noted that non-transitory computer-readable storage media (devices) can be included in computer system components that also use (or even predominantly) transmission media.

Computer-executable instructions include e.g. Instructions and data that, when executed in a processor, cause a general purpose computer, computer, or processing device to perform a particular function or set of functions. In some embodiments, computer-executable instructions are executed on a general computer so that the general computer becomes a particular computer that executes elements of the disclosure. The computer-executable instructions may be e.g. binary statements ("binaries"), intermediate format statements, such as Assembly language, or even a source code. Although the subject matter has been described in language specific to structural features and / or procedural operations, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the features and processes described are disclosed as example forms for implementing the claims.

One skilled in the art will appreciate that the disclosure can be made in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, cell phones , PDAs, tablets, pagers, routers, switches, and the like. The disclosure may also be made in distributed system environments where local and remote computer systems connected through a network (either through wired data links, wireless data links, or a combination of wired and wireless data links) perform both tasks. In a distributed system environment, program modules may reside in both local and remote storage devices.

Embodiments of the present disclosure may also be implemented in cloud computing environments. In this description, "cloud computing" is defined as a model that allows network access to a common pool of configurable computing resources as needed. For example, cloud computing may be deployed in the marketplace to provide anywhere, convenient on-demand access to the common pool of configurable computing resources. The shared pool of configurable computing resources can be quickly deployed through virtualization and released with a low overhead or service provider interaction, and then scaled accordingly.

A cloud computing model may be composed of various characteristics, such as on-demand self-service, broad network access, resource pooling, fast resilience, measured service, etc. A cloud computing model may also include various service models, such as software as a service ("SaaS"). ), Platform as service ("PaaS") and infrastructure as service ("IaaS"). A cloud computing model may also be distributed by various distribution models, such as a private cloud, a community cloud, a public cloud, a hybrid cloud, etc. In this specification and claims, a "cloud computing environment" is an environment in which cloud computing is employed ,

The 10 FIG. 3 illustrates in block diagram form an exemplary computing device 1000 that may be configured to perform one or more of the above described operations. It is clear that the adaptive rate allocation system 104 Implementations of the computing device 1000 may include. As it is in the 10 1, the computing device may be a processor 1002 , a store 1004 , a storage device 1006 , an I / O interface 1008, and a communication interface 1010. In certain embodiments, the computing device 1000 include fewer or more components than those used in the 10 are shown. Components of the computing device 1000 in the 10 are shown below in more detail.

In certain embodiments, the processor (s) comprise or include 1002 Hardware for executing instructions, such as those that make up a computer program. By way of example and not limitation, the processor (s) may or may not be 1002 the instructions from an internal register, an internal cache, a memory 1004 or a storage device 1006 retrieve (or fetch) and decode and execute.

The computing device 1000 includes a memory 1004 that is related to the processor (s) 1002 is coupled. The memory 1004 may be used to store data, metadata and programs for execution by the processor (s). The memory 1004 may include one or more of volatile and nonvolatile memories, such as random access memory ("RAM"), read only memory ("ROM"), semiconductor disk ("SSD"), flash memory, phase change memory ("PCM"), or other types of data storage. The memory 1004 can be an internal or a distributed or external memory.

The computing device 1000 comprises a storage device 1006 which includes a memory for storing data or instructions. As an example and not by way of limitation, the memory device 1006 comprise a non-volatile storage medium as described above. The storage device 1006 may include a hard disk drive (HDD), a flash memory, a serial universal bus (USB) drive, or a combination of these or other memory devices.

The computing device 1000 Also includes one or more input or output ("I / O") devices / interfaces 1008 provided to enable a user input to (such as user touches) the computing device 1000 perform an output from the computing device 1000 or otherwise receive data to and from the computing device 1000 transferred to. These I / O devices / interfaces 1008 may be a mouse, keypad or keyboard, touch screen, camera, optical scanner, network interface, modem or other known I / O devices, or a combination of such I / O devices. Devices / interfaces 1008 include. The touch screen can be activated with a pen or a finger.

The I / O devices / interfaces 1008 may include one or more devices for presenting an output to a user, including, but not limited to, a graphics module, a display (eg, a display screen), one or more output drivers (eg Display driver), one or more speakers, and one or more audio drivers. In certain embodiments, the devices / interfaces 1008 are configured to provide graphical data for a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and / or any other graphical content that may serve as a particular implementation.

The computing device 1000 may further include a communication interface 1010 include. The communication interface 1010 may include hardware, software, or both. The communication interface 1010 may include one or more interfaces for communicating (such as with packet-based communication) between the computing device and one or more other computing devices. 1000 or one or more network (s). By way of example and not limitation, the communication interface 1010 a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as WI-FI. The computing device 1000 can also have a bus 1012 include. The bus 1012 may include hardware, software, or both, the components of the computing device 1000 coupled with each other.

In the foregoing description, the invention has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the invention are described with reference to details discussed herein, and the accompanying drawings illustrate the various embodiments. The foregoing description and drawings are merely illustrative of the invention and should not be construed as limiting the invention. Numerous specific details are described so that various embodiments of the present invention will be readily understood.

The present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The described embodiments are to be considered in all respects only as illustrative and not restrictive. For example, the methods described herein may be performed with fewer or more steps / operations, or the steps / operations may be performed in a different order. In addition, the steps / operations described herein may be repeated or performed in parallel with different instances of the same or similar steps / operations. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. Any changes within the meaning and range of equivalency of the claims are intended to be within the scope of the invention.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 62430313 [0001]

Claims (20)

A computer-implemented method of streaming bandwidth-efficient virtual reality video in a digital media environment to provide a virtual reality experience, the method comprising: Partitioning a panoramic video into a plurality of tiles; Receiving view area information from a client device; Assigning, on the basis of the view area information, priorities to each tile of the plurality of tiles; Determining, for each tile, a bit rate that corresponds to the assigned priority of the tile; and Stream each tile to the client device using the appropriate specific bit rate. Method according to Claim 1 further comprising using a description of a spatial relationship for dividing the panoramic video into the plurality of tiles. Method according to Claim 2 in which the panoramic video comprises a 360 degree spherical video and in which the plurality of tiles are combined to form the 360 degree spherical video. Method according to Claim 3 wherein dividing the 360-degree spherical video into the plurality of tiles comprises creating one or more top tiles, one or more bottom tiles, and a plurality of center body tiles. A method according to any one of the preceding claims, wherein the panoramic video is a stereo video comprising two respective video streams. A method according to any one of the preceding claims, wherein the view area information comprises position data indicating a location of a viewing area within the panoramic video. Method according to Claim 6 further comprising identifying, based on the view area information, one or more tiles of the plurality of tiles that overlap or overlap the view area and one or more tiles of the plurality of tiles that are adjacent to the view area bordering or adjoining, and wherein assigning priorities to each tile of the plurality of tiles based on the view area comprises: assigning a highest priority to the one or more tiles that overlap or overlap the view area; Assigning a next higher priority to the one or more tiles that are adjacent to or adjoin the view area; and assigning a lower priority to a remainder of the one or more tiles of the plurality of tiles. Method according to Claim 7 further comprising assigning the same priority to each tile of the plurality of tiles that overlaps the view area. Method according to Claim 8 further comprising determining a minimum bit rate for streaming the plurality of tiles to the client device, wherein the minimum bit rate corresponds to a lowest acceptable quality of the panoramic video and wherein streaming the plurality of tiles to the client device at the minimum bit rate does not exceed a bandwidth limit. Method according to Claim 9 further comprising determining higher bit rates for streaming the plurality of tiles to the client device, wherein one or more higher bit rates of the higher bit rates correspond to one or more higher quality levels (s) of the panoramic video. Method according to Claim 10 wherein determining the bit rate corresponding to the assigned priority of each tile comprises assigning bit rates to tile priority levels using a multiple-choice knapsack problem algorithm. Method according to Claim 11 wherein assigning bit rates to tile priority levels using a multiple-choice knapsack problem algorithm is as follows: Determining a first remaining bandwidth amount by subtracting the minimum bit rate for the plurality of tiles from the bandwidth limit; and allocating the first remaining bandwidth to tiles having a highest priority level. Method according to Claim 12 wherein assigning the first remaining bandwidth to highest priority level tiles comprises streaming the highest priority level tiles at the highest bit rate. A system for streaming virtual reality video, the system comprising: one or more computer-readable storage (s) comprising or comprising a virtual reality content repository; and at least one computing device having stored thereon instructions that, when executed by the at least one computing device, cause the system to: partitioning a panoramic video into a plurality of tiles by partitioning the panoramic video into adjacent tiles; Receiving view area information from a client device indicating a current viewing area position within the panoramic video; assigning priorities to each tile of the plurality of tiles based on the current view area position by determining a correlation between the current view area position and each tile within the panoramic video; each tile assigns a bit rate corresponding to the tile's assigned priority by assigning a higher bit rate to tiles with a higher priority and a lower bit rate to tiles with a lower priority; and each tile to the client device using the corresponding allocated bitrate streams. System after Claim 14 and further comprising instructions thereon that, when executed by the at least one computing device, cause the system to: one or more tiles of the plurality of tiles that overlap or overlap the current view area location, and one or more tiles identify a plurality of tiles of the plurality of tiles adjacent to or adjacent to the current view area location; and prioritize each tile of the plurality of tiles based on the current view area position by: assigning a highest priority to the one or more tiles that overlap or overlap the current view area position; Assigning a next higher priority to the one or more tiles adjacent to or adjacent to the current view area position; and assigning a lower priority to a remainder of the one or more tiles of the plurality of tiles. System after Claim 14 or 15 and further comprising instructions thereon, when executed by the at least one computing device, causing the system to: determine a minimum bit rate for streaming the plurality of tiles to the client device, the minimum bit rate corresponding to a lowest acceptable quality of the panoramic video ; and streaming the plurality of tiles to the client device at the minimum bit rate that does not exceed a bandwidth limit. System according to one of Claims 14 to 16 and further comprising instructions for, when executed by the at least one computing device, causing the system to: iteratively determine an available bit rate for streaming each tile of the plurality of tiles based on a rating of the assigned priority of each tile; iteratively determines an amount of remaining available bandwidth so that the bandwidth is minimized; and each tile of the plurality of tiles streams at least at the minimum bit rate. A computer-implemented method of streaming bandwidth-efficient virtual reality video in a digital media environment to provide a virtual reality experience, the method comprising: Performing a step of spatially splitting a spatial video into a three-dimensional (3D) sphere of three-dimensional (3D) hexaface sphere comprising 3D mesh segments; Receiving view area information from a client device; Assigning, based on the view area information, priorities to each of the 3D network segments; Performing a step of determining a bit rate allocation for each of the 3D network segments based on the priority assigned to each of the 3D network segments; and streaming each of the 3D mesh segments to the client device using the bit rate determined for each of the 3D mesh segments. Method according to Claim 18 wherein performing the step of spatially dividing the spatial video into the 3D mesh segments comprises: dividing the spherical video into a plurality of vertical stacks and a plurality of horizontal slices; Splitting the spherical video, based on one or more horizontal slice (s), into a plurality of 3D networks; and further dividing the 3D mesh segment of the 3D mesh segments into additional 3D mesh segments based on the plurality of horizontal slices. Method according to Claim 19 wherein performing the step of determining bit rate allocation for each of the 3D mesh segments comprises assigning bit rates to tile priority levels of tiles with a multiple-choice knapsack problem algorithm.

Images