JP2018522430A - Method and apparatus for reducing spherical video bandwidth to a user headset - Google Patents

Abstract

The method determines at least one preferred view perspective associated with a three-dimensional (3D) video and encodes a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality. And encoding a second portion of the 3D video with a second quality, wherein the first quality is higher quality than the second quality.

Description

Cross-reference to related applications This application is a “Method and Apparatus to Reduce Spherical Video Bandwidth to User Headset” filed May 27, 2015. Claims the benefit of the entitled US Serial No. 62 / 167,121. This application is incorporated herein by reference in its entirety.

FIELD Embodiments relate to streaming spherical video.

Background Streaming spherical video (or other 3D video) can consume a significant amount of system resources. For example, encoded spherical video may include a large number of bits for transmission, which may consume a significant amount of bandwidth and processing and memory associated with the encoder and decoder.

Overview Exemplary embodiments describe systems and methods that optimize streaming video, streaming 3D video, and / or streaming spherical video.

  In one general aspect, a method determines at least one preferred view perspective associated with a three-dimensional (3D) video and a first portion of the 3D video corresponding to the at least one preferred view perspective. Encoding with a quality of 1 and encoding a second part of the 3D video with a second quality, the first quality being higher than the second quality.

  In another general aspect, the server and / or streaming server includes a controller configured to determine at least one preferred view perspective associated with a three-dimensional (3D) video, and an encoder, , Configured to encode a first portion of 3D video corresponding to at least one preferred view perspective with a first quality and to encode a second portion of 3D video with a second quality, The quality is higher than the second quality.

  In yet another general aspect, the method includes receiving a request for streaming video, the request includes displaying a user view perspective associated with a three-dimensional (3D) video, and the method further includes: Determining whether the perspective is stored in the view perspective data store; determining that the user view perspective is stored in the view perspective data store; incrementing a ranking value associated with the user view perspective; If it determines that the user view perspective is not stored in the view perspective data store, it adds the user view perspective to the view perspective data store and And a step of setting a ranking value associated with the perspective to 1.

  Implementations can include one or more of the following features. For example, the method (or server implementation) stores a first portion of 3D video in a data store, stores a second portion of 3D video in the data store, and for streaming video The method may further include receiving the request and streaming the first portion of the 3D video and the second portion of the 3D video as streaming video from the data store. The method (or server implementation) further includes receiving a request for streaming video, the request includes displaying a user view perspective, and the method further includes a 3D video corresponding to the user view perspective, Selecting as the encoded first portion of the 3D video, and streaming the selected first portion of the 3D video and the second portion of the 3D video as streaming video.

  The method (or server implementation) further includes receiving a request for streaming video, the request includes displaying a user view perspective associated with the 3D video, and the method further includes: Determining whether the user view perspective is stored in the view perspective data store, determining that the user view perspective is stored in the view perspective data store, incrementing a counter associated with the user view perspective, and the user view perspective Is not stored in the view perspective data store, the user view perspective is added to the view perspective data store and the user view The counter associated with Pekutibu may include a step of setting to 1. The method (or implementation at the server), wherein the step of encoding the second part of the 3D video uses at least one first quality of service (QoS) parameter in the first pass encoding operation. Encoding the first portion of the 3D video may include using at least one second quality of service (QoS) parameter in the second pass encoding operation. .

  For example, determining at least one preferred view perspective associated with a 3D video is based on at least one of historically viewed reference points and previously viewed view perspectives. . At least one preferred view perspective associated with the 3D video includes: 3D video viewer orientation, 3D video viewer position, 3D video viewer point, and 3D video viewer focus Based on at least one. The step of determining at least one preferred view perspective associated with the 3D video is based on a default view perspective, wherein the default view perspective includes characteristics of a user of the display device, characteristics of a group associated with the user of the display device, Based on at least one of director's cut and 3D video characteristics. For example, the method (or server implementation) repeatedly encodes at least a portion of a second portion of 3D video with a first quality and at least a portion of the second portion of 3D video. Streaming.

Exemplary embodiments will be more fully understood from the detailed description provided below and the accompanying drawings. In the figures, the same elements are represented by the same reference numerals, which are given by way of example only and are thus not a limitation of the exemplary embodiments.
FIG. 3 shows a two-dimensional (2D) representation of a sphere according to at least one exemplary embodiment. It is a figure which shows the expansion | deployment cylindrical representation of 2D representation of a sphere as 2D rectangular representation. FIG. 6 illustrates a method for encoding streaming spherical video in accordance with at least one exemplary embodiment. FIG. 6 illustrates a method for encoding streaming spherical video in accordance with at least one exemplary embodiment. FIG. 6 illustrates a method for encoding streaming spherical video in accordance with at least one exemplary embodiment. FIG. 6 illustrates a method for encoding streaming spherical video in accordance with at least one exemplary embodiment. FIG. 2 illustrates a video encoder system in accordance with at least one exemplary embodiment. FIG. 2 illustrates a video decoder system in accordance with at least one exemplary embodiment. FIG. 6 shows a flow diagram for a video encoder system in accordance with at least one exemplary embodiment. FIG. 4 shows a flow diagram for a video decoder system in accordance with at least one exemplary embodiment. FIG. 1 illustrates a system in accordance with at least one exemplary embodiment. FIG. 6 is a schematic block diagram of a computing device and a mobile computing device that can be used to implement the techniques described herein.

  It is noted that these figures are intended to illustrate the general features of the methods, structures and / or materials utilized in certain exemplary embodiments, and are intended to supplement the description provided below. Has been. However, these drawings are not to scale, and may not accurately reflect the structural or performance characteristics of any given embodiment, and may define or include characteristics encompassed by example embodiments. It should not be construed as limiting. For example, the positioning of the structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in the various drawings is intended to indicate the presence of similar or identical elements or features.

DETAILED DESCRIPTION OF EMBODIMENTS While the illustrative embodiments may include various modifications and alternative forms, those embodiments are shown by way of example in the drawings and will be described in detail herein. However, it is not intended that the exemplary embodiments be limited to the specific forms disclosed, but rather that the exemplary embodiments cover all modifications, equivalents, and alternatives falling within the scope of the claims. Should be understood. Like numbers refer to like elements throughout the description of the figures.

  An exemplary embodiment is a preferentially viewed portion of a spherical video (by a video viewer) that streams video, 3D video, spherical video (and / or other 3D video). Systems and methods configured to optimize based on, for example, directors cuts, historical viewings, etc. are described. For example, a director's cut may be a view perspective as selected by a video director (director) or producer. The director's cut may be based on the camera's (multi-camera) view that is selected or viewed when the video director or producer records the video.

  A spherical video, a frame of a spherical video, and / or a spherical image may have a perspective. For example, the spherical image may be an image of the earth. The interior perspective may be a view looking out from the center of the earth. Alternatively, the internal perspective may look at the universe on the earth. The external perspective may be a view looking down from space into the earth. As another example, the perspective may be based on a portion of the image that is visible. In other words, the visible perspective can be what a viewer can see. The visible perspective may be a part of a spherical image that is in front of the viewer. For example, when viewed from an internal perspective, the viewer may lie on the ground (eg, the Earth) and look at the universe. The viewer may see the moon, the sun, or a specific star in the image. However, although the ground on which the viewer lies is included in the spherical image, the ground is outside the current visible perspective. In this example, if the viewer rotates his head, the ground will be included in the surrounding visible perspective. When the viewer is face down, the ground will be in the visible perspective, but the moon, sun, or star will not be in the visible perspective.

  The visible perspective from the external perspective may be a part of a spherical image that is not obstructed (eg by another part of the image) and / or a part of a spherical image that is not curved until it disappears. Another portion of the spherical image may be brought from the external perspective to the visible perspective by moving (eg, rotating) the spherical image and / or by movement of the spherical image. Thus, the visible perspective is a portion of a spherical image that is within the visible range of the viewer of the spherical image.

  A spherical image is an image that does not change over time. For example, a spherical image from an internal perspective, such as for the Earth, may show the moon and stars in one position. On the other hand, spherical video (or a sequence of images) may change over time. For example, a spherical video from an internal perspective, such as for the Earth, may show a moving moon and stars (eg, due to Earth rotation) and / or a contrail across an image (eg, the sky).

  FIG. 1A is a two-dimensional (2D) representation of a sphere. As shown in FIG. 1A, a sphere 100 (eg, as a spherical image or frame of a sphere video) shows the orientation of the interior perspectives 105, 110, the exterior perspective 115, and the visible perspectives 120, 125, 130. The visible perspective 120 may be part of a spherical image as seen from the internal perspective 110. The visible perspective 120 may be a part of the sphere 100 as seen from the internal perspective 105. The visible perspective 125 may be a part of the sphere 100 as seen from the external perspective 115.

  FIG. 1B is a diagram illustrating a developed cylindrical representation 150 of a 2D representation of a sphere 100 as a 2D rectangular representation. An equirectangular projection of the image shown as the developed cylindrical representation 150 appears as a stretched image as the image moves vertically away from the center line between points A and B (up and down as shown in FIG. 1B). obtain. The 2D rectangular representation can be decomposed as an N × N block C × R matrix. For example, as shown in FIG. 1B, the illustrated expanded cylinder representation 150 is a 30 × 16 matrix of N × N blocks. However, other C × R dimensions are within the scope of this disclosure. The block may be a block (or a block of pixels) such as 2 × 2, 2 × 4, 4 × 4, 4 × 8, 8 × 8, 8 × 16, and 16 × 16.

  A spherical image is an image that is continuous in all directions. Therefore, if the spherical image is decomposed into a plurality of blocks, the plurality of blocks will be close together in the entire spherical image. In other words, there are no edges or boundaries as in a 2D image. In an exemplary implementation, adjacent end blocks may be adjacent to the boundary of the 2D representation. In addition, the adjacent block may be a block adjacent to the block on the boundary of the 2D representation. For example, adjacent end blocks are associated with two or more boundaries of a two-dimensional representation. In other words, because a spherical image is an image that is continuous in all directions, adjacent edges may be associated with upper and lower boundaries (eg, in a row of blocks) in the image or frame, and / or It may be associated with left and right boundaries (eg, in a row of blocks) in an image or frame.

  For example, if equirectangular projection is used, the adjacent end block may be the block at the other end of the column or row. For example, as shown in FIG. 1B, blocks 160 and 170 may be adjacent to each other (by column). Also, blocks 180 and 185 may be adjacent end blocks (for each column) of each other. Further, blocks 165 and 175 may be adjacent to each other (per row). View perspective 192 may include at least one block (and / or may overlap with at least one block). A block may be encoded as a region of an image, a region of a frame, a part or subset of an image or frame, a group of blocks, and the like. In the following, this group of blocks may be referred to as a tile or a group of tiles. For example, in FIG. 1B, tiles 190 and 195 are illustrated as a group of four blocks. The tile 195 is illustrated as being in the view perspective 192.

  In an exemplary embodiment, in addition to streaming encoded spherical video frames, as tiles (or groups of tiles) selected based on at least one reference point frequently viewed by the viewer. View perspective (eg, at least one reference point or view perspective seen so far) is encoded, eg, with higher quality (eg, higher resolution and / or less distortion) and encoded with spherical video It can be streamed with (or as part of) the frame. Thus, during playback, the viewer can see the decoded tiles (with higher quality) while the entire spherical video is playing, and the viewer's view perspective is the view frequently viewed by the viewer. Even when changing to a perspective, the entire spherical video is still available. The viewer can also change the viewing position or switch to another view perspective. If that other view perspective is included in at least one reference point that is frequently viewed by the viewer, then the played video is (e.g., of at least one reference point that is frequently viewed by the viewer). It can be of higher quality (eg, higher resolution) compared to some other view perspective (not one). One advantage of encoding and streaming only a selected portion or subset of an image or frame with higher quality is that it is not necessary to encode, stream and decode the entire spherical video with higher quality. Has the advantage that the selected image or frame of the spherical video can be decoded and played with higher quality, so that bandwidth usage and processing and memory resources associated with the encoder and decoder Increase efficiency.

  In a head mount display (HMD), viewers can visually perceive through the use of left (eg, left eye) and right (eg, right eye) displays that project perceived three-dimensional (3D) video or images. Experience virtual reality. According to an exemplary embodiment, a spherical (eg, 3D) video or image is stored on the server. The video or image can be encoded and streamed from the server to the HMD. Spherical video or images can be encoded as left and right images. The left and right images are packaged (eg, in a data packet) with metadata about the left and right images. The left and right images are then decoded and displayed by a left (eg left eye) display and a right (eg right eye) display.

  The systems and methods described herein are applicable to both left and right images, and throughout the present disclosure, depending on use cases, images, frames, parts of images, parts of frames, tiles, etc. It is called. In other words, the encoded data that is communicated from the server (eg, streaming server) to the user device (eg, HMD) and then decoded for display is the left image and / or right image associated with the 3D video or image. possible.

  2-5 are flowcharts of methods according to exemplary embodiments. The steps described with respect to FIGS. 2-5 include memory associated with the device (eg, as shown in FIGS. 6A, 6B, 7A, 7B, and 8 (described below)) (eg, at least May be performed by execution of software code stored in one memory 610) and executed by at least one processor (eg, at least one processor 605) associated with the device. However, alternative embodiments are possible, such as a system embodied as a special purpose processor. Although the steps described below are described as being performed by a processor, these steps are not necessarily performed by the same processor. In other words, at least one processor may perform the steps described below with respect to FIGS.

  FIG. 2 illustrates a method for storing a previous view perspective, where “historical” refers to a view perspective previously requested by a user. For example, FIG. 2 may illustrate the construction of a view perspective database that is often found in spherical video streams. As shown in FIG. 2, in step S205, a view of the view perspective is received. For example, tiles may be requested by a device that includes a decoder. The tile request may include information based on a perspective or view perspective regarding the orientation, position, point, or focus of the viewer on the spherical video. The perspective or view perspective may be a user view perspective, i.e., a view perspective of an HMD user. For example, a view perspective (eg, a user view perspective) may be a latitude and longitude location on a spherical video (eg, as an internal perspective or an external perspective). A view, perspective, or view perspective may be determined as a side of a cube based on spherical video. The display of the view perspective may also include spherical video information. In an exemplary implementation, the display of the view perspective may include information about frames associated with the view perspective (eg, a frame sequence). For example, views (eg, latitude and longitude positions, or edges) communicate to a streaming server from a user device (associated with) including an HMD using, for example, a Hypertext Transfer Protocol (HTTP). Can be done.

  In step S210, it is determined whether a view perspective (eg, a user view perspective) is stored in the view perspective data store. For example, a data store (eg, view perspective data store 815) may be queried or filtered based on information associated with the view perspective or the user view perspective. For example, the data store may be queried or filtered based on the latitude and longitude positions on the spherical video of the view perspective, and the timestamp in the spherical video where the view perspective was viewed. The time stamp may be a time and / or time range associated with the playback of the spherical video. The query or filter can be spatial proximity (eg, how close the current view perspective is to a given stored view perspective) and / or temporal proximity (eg, the current timestamp is stored for a given store). How close it is to the time stamp). If the query or filter returns results, the view perspective is stored in the data store. If no result is returned, the view perspective is not stored in the data store. If the view perspective is stored in the view perspective data store, in step S215, processing continues to step S220. If not, processing continues to step S225.

  In step S220, a counter or ranking (or ranking value) associated with the received view perspective is incremented. For example, the data store may include a data table that includes a previous view perspective (eg, the data store may be a database that includes multiple data tables). The data table may be a keyed (eg, unique to each) view perspective. The data table may include view perspective identification information, information associated with the view perspective, and a counter indicating how many times the view perspective has been requested. The counter may be incremented each time a view perspective is requested. The data stored in the data table may be anonymized. In other words, the data may be stored such that no user, device, session, etc. is mentioned (or there is no identification information for the user, device, session, etc.). Therefore, the data stored in the data table cannot be distinguished based on the video user or viewer. In an exemplary implementation, the data stored in the data table may be classified based on the user without identifying the user. For example, the data may include the user's age, age group, gender, type or role (eg, musician or audience), and the like.

  In step S225, the view perspective is added to the view perspective data store. For example, the identification information of the view perspective, the information associated with the view perspective, and the counter (or ranking value) set to 1 may be stored in the data table including the previous view perspective.

  In an exemplary embodiment, tiles associated with at least one preferred view perspective may be encoded with a higher QoS. QoS can be an implementation of the above-described quality (eg, encoder variable input that defines quality). For example, an encoder (eg, video encoder 625) can individually encode tiles associated with 3D video. Tiles associated with at least one preferred view perspective can be encoded with a higher QoS compared to tiles associated with the rest of the 3D video. In an exemplary implementation, 3D video uses a first QoS parameter (eg, in the first pass) or at least one first QoS parameter used in the first coding pass. Can be encoded. In addition, the tile associated with the at least one preferred view perspective is a second QoS parameter (eg, in the second pass) or at least one second QoS parameter used in the second coding pass. Can be encoded using. In this exemplary implementation, the second QoS is a higher QoS compared to the first QoS. In another example implementation, 3D video may be encoded as multiple tiles representing 3D video. Tiles associated with at least one preferred view perspective may be encoded using the second QoS parameter. The remaining tiles may be encoded using the first QoS parameter.

  In alternative implementations (and / or additional implementations), the encoder is used to generate the remaining 2D representation of the 3D video frame with tiles associated with at least one preferred view perspective. Projections can be made using different projection techniques or algorithms. Depending on the projection, there may be distortion in an area of the frame. Thus, projecting the tiles differently than the spherical frame improves the quality of the final image and / or uses the pixels more efficiently (e.g., requires fewer computer calculations or the user's Can reduce the burden on the eyes). In one exemplary implementation, the spherical image can be rotated before projecting the tile to direct the tile to a position with minimal distortion based on the projection algorithm. In another exemplary implementation, the tiles can use (and / or modify) a projection algorithm based on the position of the tiles. For example, projecting a spherical video frame into a 2D representation can use equirectangular projection, while projecting a spherical video frame into a representation that includes a portion to be selected as a tile can use a cubic projection.

  FIG. 3 shows a method for streaming 3D video. FIG. 3 illustrates a scenario where streaming 3D video is encoded on demand, such as during a live streaming event. As shown in FIG. 3, a request to stream 3D video is received in step S305. For example, 3D video, 3D video portions, or tiles that are available for streaming may be requested by a device that includes a decoder (eg, via user interaction with a media application). The request may include information based on a perspective or view perspective regarding the orientation, position, point, or focus of the viewer on the spherical video. Information based on a perspective or view perspective may be based on a current orientation or a default (eg, initialization) orientation. The default orientation can be, for example, a director's cut for 3D video.

  In step S310, at least one preferred view perspective is determined. For example, a data store (eg, view perspective data store 815) may be queried or filtered based on information associated with the view perspective. The data store may be queried or filtered based on latitude and longitude positions on the spherical video of the view perspective. In an exemplary implementation, the at least one preferred view perspective may be based on previous view perspectives. As such, the data store may include a data table that includes previous view perspectives. Depending on how many times the view perspective is requested, preferences may be displayed. Thus, the query or filter may include removing results that are less than the threshold counter value. In other words, the parameters set for the query of the data table containing the previous view perspective may include values for counters or rankings. Here, the result of the query must be above the threshold for the counter. The result of a query of the data table containing the previous view perspective can be set as at least one preferred view perspective.

  In addition, a default preferred view perspective (or multiple such view perspectives) may be associated with the 3D video. The default preferred view perspective may be a director's cut, a point of interest (eg, horizon, moving object, priority object), etc. For example, the purpose of a game may be to destroy an object (eg, a building or a vehicle). This object may be labeled as a priority object. A view perspective that includes a priority object may be displayed as a preferred view perspective. The default preferred view perspective may be included in addition to or instead of the previous view perspective. The default orientation can be, for example, the first set of preferred view perspectives based on, for example, an automated computer vision algorithm (eg, because there is no previous data if the video was first uploaded). Vision algorithms are preferred for videos that have motion or complex details, or features that were present in nearby stereo objects to infer what is interesting, and / or other previous preferred views of video. The view perspective portion may be determined.

  Other factors can be used in determining at least one preferred view perspective. For example, the at least one preferred view perspective may be a previous view perspective that is within range of the current view perspective (eg, close to the current view perspective). For example, at least one preferred view perspective is within the scope of the previous view perspective of the current user or of the group (type or category) to which the current user belongs (e.g., close to the view perspective) View perspective. In other words, the at least one preferred view perspective may include a view perspective (or tile) that is close in distance and / or close in time to the stored previous view perspective. The default preferred view perspective may be stored in the data store 815 containing the previous view perspective, or in a separate (eg, additional) data store not shown.

  In step S315, the 3D video is encoded using at least one encoding parameter based on at least one preferred view perspective. For example, the 3D video (or a portion thereof) may be encoded such that the portion including at least one preferred view perspective is encoded differently than the rest of the 3D video. Thus, the part containing at least one preferred view perspective can be encoded with a higher QoS compared to the rest of the 3D video. As a result, when rendered on the HMD, the portion containing at least one preferred view perspective may have a higher resolution than the rest of the 3D video.

  In step S320, the encoded 3D video is streamed. For example, a tile may be included in the transmission packet. The packet may include compressed video bits 10A. The packet may include an encoded 2D representation of a spherical video frame and an encoded tile (or tiles). The packet may include a transmission header. The header may include information indicating the mode or scheme usage in intra-frame coding by the encoder, among others. The header may include information indicating parameters used to convert the frame of the spherical video frame into a 2D rectangular representation. The header may include information indicating a parameter used to obtain the QoS of the encoded 2D rectangular representation and of the encoded tile. As described above, the QoS of tiles associated with at least one preferred view perspective may be different (eg, higher) than tiles that are not associated with at least one preferred view perspective.

  Streaming 3D video can be realized through the use of priority stages. For example, in the first priority stage, video data encoded with low (or minimum) QoS may be streamed. This allows HMD users to start a virtual reality experience. The higher QoS video can then be streamed to the HMD to replace the previously streamed low (or minimum criteria) QoS encoded video data (eg, data stored in buffer 830). . As an example, in the second stage, higher quality video or image data may be streamed based on the current view perspective. In the next stage, higher QoS video or image data may be streamed based on one or more preferred view perspectives. This may continue until the HMD buffer substantially contains only high QoS video or image data. In addition, this gradual streaming may loop with video or image data with progressively higher QoS. In other words, after the first iteration, the HMD includes video or image data encoded with a first QoS, and after the second iteration, the HMD is a video or video encoded with a second QoS. It contains image data, and after the third iteration, the HMD contains video or image data encoded with a third QoS. In an exemplary implementation, the second QoS is higher than the first QoS, and the third QoS is higher than the second QoS.

  Encoder 625 may operate offline as part of a setup procedure to make spherical video available for streaming. Each of the plurality of tiles may be stored in the view frame storage 795. Each of the plurality of tiles may be indexed such that each of the plurality of tiles can be stored with reference to a frame (eg, time-dependent) and with reference to a view (eg, view-dependent). Thus, each of the plurality of tiles depends on time and view, perspective, or view perspective, and can be recalled based on time dependency and view dependency.

  Thus, in an exemplary implementation, encoder 625 may be configured to perform a loop in which a frame is selected and a portion of the frame is selected as a tile based on a view perspective. The tile is then encoded and stored. The loop continues to circulate through multiple view perspectives. For example, if the desired number of view perspectives are saved as tiles, 5 degrees around the vertical line of the spherical image and 5 degrees around the horizontal line, a new frame is selected and the process is performed for all frames of the spherical video. Repeat until it has the desired number of tiles stored for them. In an exemplary embodiment, tiles associated with at least one preferred view perspective may be encoded with a higher QoS than tiles that are not tiles associated with at least one preferred view perspective. This is just one example implementation for encoding and storing tiles.

  FIG. 4 shows a method for storing encoded 3D video. FIG. 4 illustrates a scenario where streaming 3D video is pre-encoded and stored for future streaming. As shown in FIG. 4, at step S405, at least one preferred view perspective for the 3D video is determined. For example, a data store (eg, view perspective data store 815) may be queried or filtered based on information associated with the view perspective. The data store may be queried or filtered based on latitude and longitude positions on the spherical video of the view perspective. In an exemplary implementation, the at least one preferred view perspective may be based on previous view perspectives. Therefore, the data table includes the conventional view perspective. Depending on how many times the view perspective is requested, preferences may be displayed. Thus, the query or filter may include removing results that are less than the threshold counter value. In other words, the parameters set for the query of the data table containing the previous view perspective may include values for the counter. Here, the result of the query must be above the threshold for the counter. The result of a query of the data table containing the previous view perspective can be set as at least one preferred view perspective.

  In addition, a default preferred view perspective (or multiple such view perspectives) may be associated with the 3D video. The default preferred view perspective may be a director's cut, a point of interest (eg, horizon, moving object, priority object), etc. For example, the purpose of a game may be to destroy an object (eg, a building or a vehicle). This object may be labeled as a priority object. A view perspective that includes a priority object may be displayed as a preferred view perspective. The default preferred view perspective may be included in addition to or instead of the previous view perspective. Other factors can be used in determining at least one preferred view perspective. For example, the at least one preferred view perspective may be a previous view perspective that is within range of the current view perspective (eg, close to the current view perspective). For example, at least one preferred view perspective is within the scope of the previous view perspective of the current user or of the group (type or category) to which the current user belongs (e.g., close to the view perspective) View perspective. The default preferred view perspective may be stored in the data store containing the previous view perspective or in a separate (eg, additional) data table.

  In step S410, the 3D video is encoded using at least one encoding parameter based on at least one preferred view perspective. For example, a frame of 3D video may be selected and a portion of the frame may be selected as a tile based on the view perspective. The tile is then encoded. In an exemplary embodiment, tiles associated with at least one preferred view perspective may be encoded with a higher QoS. Tiles associated with at least one preferred view perspective can be encoded with a higher QoS compared to tiles associated with the rest of the 3D video.

  In alternative implementations (and / or additional implementations), the encoder is used to generate the remaining 2D representation of the 3D video frame with tiles associated with at least one preferred view perspective. Projections can be made using different projection techniques or algorithms. Depending on the projection, there may be distortion in an area of the frame. Thus, projecting tiles differently than spherical frames can improve the quality of the final image and / or use pixels more efficiently. In one exemplary implementation, the spherical image can be rotated before projecting the tile to direct the tile to a position with minimal distortion based on the projection algorithm. In another exemplary implementation, the tiles can use (and / or modify) a projection algorithm based on the position of the tiles. For example, projecting a spherical video frame into a 2D representation can use equirectangular projection, while projecting a spherical video frame into a representation that includes a portion to be selected as a tile can use a cubic projection.

  In step S415, the encoded 3D video is stored. For example, each of the plurality of tiles may be stored in the view frame storage 795. Each of the multiple tiles associated with the 3D video is indexed such that each of the multiple tiles can be stored with reference to a frame (eg, time-dependent) and with reference to a view (eg, view-dependent). It may be attached. Thus, each of the plurality of tiles depends on time and view, perspective, or view perspective, and can be recalled based on time dependency and view dependency.

  In an exemplary implementation, 3D video (eg, tiles associated with it) may be encoded and stored using variable encoding parameters. Accordingly, 3D video may be stored in different encoding states. These states may change based on QoS. For example, 3D video may be stored as multiple tiles, each encoded with the same QoS. For example, 3D video may be stored as multiple tiles, each encoded with different QoS. For example, the 3D video may be stored as multiple tiles partially encoded with QoS based on at least one preferred view perspective encoded.

  FIG. 5 illustrates a method for determining a preferred view perspective for 3D video. The preferred view perspective for 3D video may be in addition to the preferred view perspective based on previous viewing of 3D video. As shown in FIG. 6, at step S505, at least one default view perspective is determined. For example, a default preferred view perspective may be stored in a data table included in a data store (eg, view perspective data store 815). The data store may be queried or filtered based on the default display for 3D video. If the query or filter returns a result, the 3D video has an associated default view perspective. If it returns no results, the 3D video does not have an associated default view perspective. The default preferred view perspective may be a director's cut, a point of interest (eg, horizon, moving object, priority object), etc. For example, the purpose of a game may be to destroy an object (eg, a building or a vehicle). This object may be labeled as a priority object. A view perspective that includes a priority object may be displayed as a preferred view perspective.

  In step S510, at least one view perspective based on user characteristics / preferences / categories is determined. For example, HMD users may have characteristics based on previous use of the HMD. These characteristics may be based on statistical viewing preferences (e.g., preferring to see nearby objects rather than distant objects). For example, an HMD user may store user preferences associated with the HMD. These preferences may be selected by the user as part of the setup process. The preference may be general (e.g., attracted to movement) or video specific (e.g., tending to focus on a guitarist in a music performance). For example, an HMD user may belong to a group or category (eg, males 15 to 22 years old). For example, user characteristics / preferences / categories may be stored in a data table included in a data store (eg, view perspective data store 815). The data store may be queried or filtered based on the default display for 3D video. If the query or filter returns results, the 3D video has at least one associated preferred view perspective based on the associated characteristics / preference / category for the user. If it returns no results, the 3D video does not have an associated view perspective based on the user.

  In step S515, at least one view perspective based on the region of interest is determined. For example, the region of interest may be the current view perspective. For example, the at least one preferred view perspective may be a previous view perspective that is within range of the current view perspective (eg, close to the current view perspective). For example, at least one preferred view perspective is within the scope of the previous view perspective of the current user or of the group (type or category) to which the current user belongs (e.g., close to the view perspective) View perspective.

  In step S520, at least one view perspective based on at least one system characteristic is determined. For example, the HMD may have features that can enhance the user experience. One feature may be enhanced speech. Thus, in a virtual reality environment, a user may be attracted to a specific sound (eg, a game user may be attracted to an explosion sound). A preferred view perspective may be based on a view perspective that includes these audible cues. In step S525, at least one preferred view perspective for the 3D video is determined based on each of the aforementioned view perspective determinations and / or combinations / subcombinations thereof. For example, at least one preferred view perspective may be generated by merging or combining the results of the aforementioned queries.

  In the example of FIG. 6A, video encoder system 600 may be at least one computing device or may include at least one computing device and is configured to perform the methods described herein. It can represent virtually any computing device. As such, video encoder system 600 may include various components that can be utilized to implement the techniques described herein, or different or future versions thereof. By way of example, video encoder system 600 is illustrated as including at least one processor 605 and at least one memory 610 (eg, a non-transitory computer readable storage medium).

  FIG. 6A illustrates a video encoder system according to at least one exemplary embodiment. As shown in FIG. 6A, video encoder system 600 includes at least one processor 605, at least one memory 610, a controller 620, and a video encoder 625. At least one processor 605, at least one memory 610, controller 620, and video encoder 625 are communicatively coupled via bus 615.

  The at least one processor 605 may be utilized to execute instructions stored on the at least one memory 610, thereby various features and functions described herein, or additional or alternative Realize features and functions. At least one processor 605 and at least one memory 610 may be utilized for various other purposes. In particular, the at least one memory 610 may represent an example of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

  At least one memory 610 may be configured to store data and / or information associated with video encoder system 600. For example, the at least one memory 610 may be configured to store a codec associated with encoding the spherical video. For example, the at least one memory may be configured to store a code associated with selecting a portion of the spherical video frame as a tile to be encoded separately from the spherical video encoding. . At least one memory 610 may be a shared resource. As will be described in more detail below, a tile may be a plurality of pixels selected based on the viewer's view perspective during playback of a spherical viewer (eg, HMD). The plurality of pixels may be a block, a plurality of blocks, or a macroblock that may include a portion of a spherical image that may be viewed by a user. For example, video encoder system 600 may be an element of a larger system (eg, server, personal computer, mobile device, etc.). Accordingly, the at least one memory 610 is configured to store data and / or information associated with other elements in the larger system (eg, image / video supply, web browsing, or wired / wireless communication). May be.

  Controller 620 may be configured to generate various control signals and communicate the control signals to various blocks in video encoder system 600. Controller 620 may be configured to generate the control signal to implement the techniques described below. The controller 620 may be configured to control the video encoder 625 to encode an image, a sequence of images, a video frame, a video sequence, etc., according to an exemplary embodiment. For example, the controller 620 may generate a control signal corresponding to parameters for encoding the spherical video. Further details regarding the function and operation of video encoder 625 and controller 620 are described below in connection with at least FIGS. 7A, 4A, 5A, 5B, and 6-9.

  Video encoder 625 may be configured to receive video stream input 5 and output compressed (eg, encoded) video bits 10. Video encoder 625 may convert video stream input 5 into discrete video frames. The video stream input 5 may also be an image, so the compressed (eg, encoded) video bits 10 may also be compressed image bits. Video encoder 625 may further convert each discrete video frame (or image) into a matrix of blocks (hereinafter referred to as blocks). For example, video frames (or images) are organized into a 16 × 16, 16 × 8, 8 × 8, 8 × 4, 4 × 4, 4 × 2, 2 × 2, etc. matrix of blocks each having a number of pixels. It may be converted. Although these exemplary matrices are listed, exemplary embodiments are not limited thereto.

  The compressed video bit 10 may represent the output of the video encoder system 600. For example, the compressed video bit 10 may represent an encoded video frame (or encoded image). For example, the compressed video bit 10 may be ready for transmission to a receiving device (not shown). For example, the video bits may be transmitted to a system transceiver (not shown) for transmission to a receiving device.

  At least one processor 605 may be configured to execute computer instructions associated with controller 620 and / or video encoder 625. At least one processor 605 may be a shared resource. For example, video encoder system 600 may be an element of a larger system (eg, a mobile device). Accordingly, at least one processor 605 may be configured to execute computer instructions associated with other elements in a larger system (eg, image / video supply, web browsing, or wired / wireless communication). Good.

  In the example of FIG. 6B, video decoder system 650 may be at least one computing device and may represent virtually any computing device configured to perform the methods described herein. As such, video decoder system 650 may include various components that can be utilized to implement the techniques described herein, or different or future versions thereof. By way of example, video decoder system 650 is illustrated as including at least one processor 655 and at least one memory 660 (eg, a computer-readable storage medium).

  Thus, at least one processor 655 may be utilized to execute instructions stored on at least one memory 660, thereby providing various features and functions described herein, or additional or Implement alternative features and functions. At least one processor 655 and at least one memory 660 may be utilized for various other purposes. In particular, at least one memory 660 may represent an example of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein. According to an exemplary embodiment, video encoder system 600 and video decoder system 650 may be included in the same larger system (eg, a personal computer, mobile device, etc.). According to an exemplary embodiment, video decoder system 650 may be configured to implement a technique that is the opposite or opposite of that described with respect to video encoder system 600.

  At least one memory 660 may be configured to store data and / or information associated with video decoder system 650. For example, at least one memory 610 may be configured to store a codec associated with decoding encoded spherical video data. For example, at least one memory replaces encoded tiles and codes associated with decoding separately encoded spherical video frames, and pixels in the decoded spherical video frames with decoded tiles. May be configured to store a code for. The at least one memory 660 may be a shared resource. For example, video decoder system 650 may be an element of a larger system (eg, a personal computer, mobile device, etc.). Accordingly, the at least one memory 660 may be configured to store data and / or information associated with other elements in the larger system (eg, web browsing, or wireless communications).

  The controller 670 may be configured to generate various control signals and communicate the control signals to various blocks in the video decoder system 650. The controller 670 may be configured to generate the control signal to implement the video decoding technique described below. The controller 670 may be configured to control the video decoder 675 to decode video frames according to an exemplary embodiment. The controller 670 may be configured to generate a control signal corresponding to video decoding. Further details regarding the function and operation of video decoder 675 and controller 670 are described below.

  Video decoder 675 may be configured to receive a compressed (eg, encoded) video bit 10 input and output a video stream 5. Video decoder 675 may convert the compressed discrete video frame of video bits 10 into video stream 5. The compressed (eg, encoded) video bit 10 may also be a compressed image bit, and thus the video stream 5 may also be an image.

  At least one processor 655 may be configured to execute computer instructions associated with controller 670 and / or video decoder 675. At least one processor 655 may be a shared resource. For example, video decoder system 650 may be an element of a larger system (eg, a personal computer, mobile device, etc.). Accordingly, the at least one processor 655 may be configured to execute computer instructions associated with other elements in the larger system (eg, web browsing or wireless communication).

  7A and 7B show flow diagrams for the video encoder 625 shown in FIG. 6A and the video decoder 675 shown in FIG. 6B, respectively, in accordance with at least one exemplary embodiment. Video encoder 625 (described above) includes spherical-2D representation block 705, prediction block 710, transform block 715, quantization block 720, entropy coding block 725, inverse quantization block 730, and inverse transform block. 735, a reconstruction block 740, a loop filter block 745, a tile representation block 790, and a view frame storage 795. Other structural variations of video encoder 625 can be used to encode input video stream 5. As shown in FIG. 7A, the dashed line represents the reconstruction path between several blocks, and the solid line represents the forward path between several blocks.

  Each of the foregoing blocks is stored in a memory (eg, at least one memory 610) associated with the video encoder system (eg, as shown in FIG. 6A) and is associated with at least one processor (eg, It may be executed as software code executed by at least one processor 605). However, alternative embodiments are conceivable, such as a video encoder embodied as a special purpose processor. For example, each of the foregoing blocks (single and / or combined) may be an application specific integrated circuit or ASIC. For example, the ASIC may be configured as a transform block 715 and / or a quantization block 720.

  Spherical-2D representation block 705 may be configured to map a spherical frame or image to a 2D representation of the spherical frame or image. For example, a sphere may be projected onto a surface of another shape (eg, square, rectangle, cylinder, and / or cube). The projection may be, for example, an equirectangular cylinder or a semi equirectangular cylinder.

  Prediction block 710 may be configured to take advantage of video frame consistency (eg, pixels that have not changed compared to previously encoded pixels). The prediction may include two types. For example, the prediction may include intra-frame prediction and inter-frame prediction. Intraframe prediction relates to predicting pixel values in a block of images relative to reference samples in previously encoded neighboring blocks of the same image. For intra-frame prediction, the samples are generated in the same frame to reduce the residual encoded by the transform (eg, entropy encoding block 725) and entropy encoding (eg, entropy encoding block 725) portions of the predictive transform codec. Predicted from the reconstructed pixels. Inter-frame prediction relates to predicting pixel values in a block of an image relative to previously encoded image data.

  Transform block 715 may be configured to transform pixel values from the spatial domain to transform coefficients in the transform domain. The transform coefficients may correspond to a two-dimensional matrix of coefficients that are usually the same size as the original block. In other words, there may be as many transform coefficients as there are pixels in the original block. However, due to the conversion, some of the conversion coefficients may have a value equal to zero.

  Transform block 715 may be configured to transform the remainder (from prediction block 710), for example, into transform coefficients in the frequency domain. Typically, the transformations are Karhunen-Loeve Transform (KLT), Discrete Cosine Transform (DCT), Singular Value Decomposition Transform (SVD), and Asymmetric Discrete Sine Transform. (Asymmetric discrete sine transform: ADST).

  The quantization block 720 may be configured to reduce the data at each transform coefficient. Quantization may involve mapping values in a relatively large range to values in a relatively small range, thus reducing the amount of data needed to represent the quantized transform coefficients. . The quantization block 720 may convert the transform coefficients into discrete quantum values called quantized transform coefficients or quantization levels. For example, the quantization block 720 may be configured to add zero to data associated with the transform coefficient. For example, the encoding criteria may define 128 quantization levels in the scalar quantization process.

  The quantized transform coefficient is entropy encoded by an entropy encoding block 725. The entropy encoded coefficients are then output as compressed video bits 10 along with the information needed to decode the block, such as the type of prediction used, the motion vector, and the quantizer value. The The compressed video bit 10 may be formatted using various techniques such as run-length encoding (RLE) and zero-run coding.

  The reconstruction path in FIG. 7A is that both video encoder 625 and video decoder 675 (described below with respect to FIG. 7B) use the same reference frame to compress 10 compressed video bits (or compressed image bits). ) Exists to ensure that it is decrypted. The reconstruction path performs a function similar to that performed during the decoding process, which will be described in more detail below. The function includes dequantizing the quantized transform coefficient in inverse quantization block 730 to produce a derivative residual block (differential residual), and inverse transform block 735 in inverse. Inversely transforming the quantized transform coefficients. At reconstruction block 740, the prediction block predicted at prediction block 710 may be added to the differential residual to create a reconstruction block. Next, a loop filter 745 may be applied to the reconstructed block to reduce distortion, such as blocking artifacts.

  Tile representation block 790 may be configured to convert an image and / or frame into a plurality of tiles. A tile can be a grouping of pixels. A tile may be a plurality of pixels selected based on a view or a view perspective. The plurality of pixels may be a block, a plurality of blocks, or a macroblock that may include a portion of a spherical image that may be seen (or predicted to be seen) by a user. A part of the spherical image as a tile may have a length and a width. A portion of the spherical image may be two-dimensional or substantially two-dimensional. A tile may have a variable size (eg, what percentage of a sphere a tile covers). For example, the size of a tile is encoded based on, for example, how wide the viewer's field of view is, proximity to another tile, and / or how fast the user is turning his head, Can be streamed. For example, if the viewer is constantly looking around, a larger, lower quality tile may be selected. However, if the viewer is looking at one perspective, smaller and more detailed tiles may be selected.

  In one implementation, the tile representation block 790 invokes an instruction to the spherical-2D representation block 705 that causes the spherical-2D representation block 705 to generate a tile. In another implementation, tile representation block 790 generates tiles. In either implementation, each tile is then encoded individually. In yet another implementation, the tile representation block 790 invokes an instruction to the view frame storage 795 that causes the view frame storage 795 to store the encoded image and / or video frames as tiles. The tile representation block 790 can initiate an instruction to the view frame storage 795 that causes the view frame storage 795 to store the tile along with information or metadata about the tile. For example, information or metadata about tiles may include information related to the display of tile positions within an image or frame, encoding of tiles (eg, resolution, bandwidth, and / or 3D-2D projection algorithm), one The above association with the region of interest may be included.

  According to an exemplary implementation, encoder 625 may encode frames, portions of frames, and / or tiles with different qualities (or quality of service (QoS)). According to an exemplary embodiment, encoder 625 may encode a frame, a portion of a frame, and / or a tile multiple times, each with a different QoS. Thus, view frame storage 795 can store frames, portions of frames, and / or tiles that represent the same location within an image or frame, with different QoS. As such, the aforementioned information or metadata about the tile may include a frame, a portion of the frame, and / or a QoS indication when the tile is encoded.

  QoS may be based on compression algorithms, resolutions, transmission rates, and / or encoding schemes. Accordingly, encoder 625 may use a different compression algorithm and / or encoding scheme for each frame, part of a frame, and / or tile. For example, an encoded tile may have a higher QoS compared to a frame (associated with the tile) encoded by encoder 625. As described above, encoder 625 may be configured to encode a 2D representation of a spherical video frame. Thus, a tile (as a visible perspective that includes a portion of a spherical video frame) can be encoded with a higher QoS compared to a 2D representation of the spherical video frame. QoS may affect frame resolution when decoded. Thus, a tile (as a visible perspective that includes a portion of a spherical video frame) will have a higher resolution of the frame when the tile is decoded compared to a decoded 2D representation of the spherical video frame. Can be encoded. Tile representation block 790 may indicate the QoS with which the tile is to be encoded. The tile representation block 790 selects the QoS based on whether the frame, a portion of the frame, and / or the tile is a region of interest, is in the region of interest, is associated with a seed region, or the like. Also good. The region of interest and the seed region are described in more detail below.

  Video encoder 625 described above with respect to FIG. 7A includes the illustrated blocks. However, the exemplary embodiments are not limited thereto. Additional blocks may be added based on the different video coding configurations and / or techniques used. Also, each of the blocks shown in video encoder 625 described above with respect to FIG. 7A may be an optional block based on the different video encoding configurations and / or techniques used.

  FIG. 7B is a schematic block diagram of a decoder 675 configured to decode compressed video bits 10 (or compressed image bits). Similar to the reconstruction path of the encoder 625 described above, the decoder 675 includes an entropy decoding block 750, an inverse quantization block 755, an inverse transform block 760, a reconstruction block 765, a loop filter block 770, and a prediction block. 775, deblock filter block 780, and 2D representation-spherical block 785.

  The data elements in the compressed video bit 10 are used to generate a set of quantized transform coefficients (eg, using Context Adaptive Binary Arithmetic Decoding). It may be decoded by entropy decoding block 750. Inverse quantization block 755 inverse quantizes the quantized transform coefficients, and inverse transform block 760 inverse transforms the inverse quantized transform coefficients (using ADST) to reconstruct in encoder 625. Produces a differential residual that can be identical to that produced by.

  Using the header information decoded from the compressed video bit 10, the decoder 675 can use the prediction block 775 to produce the same prediction block that was produced in the encoder 675. The prediction block may be added to the differential residual to create a reconstructed block by the reconstruct block 765. A loop filter block 770 can be applied to the reconstruction block to reduce blocking artifacts. An unblock filter block 780 may be applied to the reconstructed block to reduce blocking distortion. The result is output as a video stream 5.

  The 2D representation-spherical block 785 may be configured to map a 2D representation of the spherical frame or image to the spherical frame or image. For example, mapping a 2D representation of a spherical frame or image to a spherical frame or image may be the reverse of the 3D-2D mapping performed by encoder 625.

  The video decoder 675 described above with respect to FIG. 7B includes the illustrated blocks. However, the exemplary embodiments are not limited thereto. Additional blocks may be added based on the different video coding configurations and / or techniques used. Also, each of the blocks shown in video decoder 675 described above with respect to FIG. 7B may be optional blocks based on the different video encoding configurations and / or techniques used.

  Encoder 625 and decoder 675 may each be configured to encode spherical video and / or images and to decode spherical video and / or images. A spherical image is an image including a plurality of pixels organized in a spherical shape. In other words, the spherical image is an image that is continuous in all directions. Thus, a viewer of a spherical image changes position or orientation (eg, moves his head or eyes) in any direction (eg, up, down, left, right, or any combination thereof) ) And part of the image can be viewed continuously.

  In an exemplary implementation, parameters used at and / or determined by encoder 625 may be used by other elements of encoder 405. For example, motion vectors used to encode 2D representations (eg, as used in prediction) may be used to encode tiles. Also used in prediction block 710, transform block 715, quantization block 720, entropy coding block 725, inverse quantization block 730, inverse transform block 735, reconstruction block 740, and loop filter block 745, and / or The parameters determined by the block may be shared between encoder 625 and encoder 405.

  A spherical video frame or part of an image may be processed as an image. Thus, a portion of a spherical video frame may be converted (or decomposed) into a C × R matrix of blocks (hereinafter referred to as blocks). For example, a portion of a spherical video frame may be a 16 × 16, 16 × 8, 8 × 8, 8 × 4, 4 × 4, 4 × 2, 2 × 2, etc. matrix of blocks each having a number of pixels. It may be converted to a C × R matrix.

  FIG. 8 illustrates a system 800 in accordance with at least one exemplary embodiment. As shown in FIG. 8, the system 700 includes a controller 620, a controller 670, a video encoder 625, a view frame storage 795, and an orientation sensor 835. The controller 620 further includes a view position control module 805, a tile control module 810, and a view perspective data store 815. The controller 670 further includes a view position determination module 820, a tile request module 825, and a buffer 830.

  According to an exemplary implementation, the orientation sensor 835 detects the orientation (or change in orientation) of the viewer's eyes (or head), and the view position determination module 820 is based on the detected orientation. Determining a view, perspective, or view perspective, tile request module 825 communicates the view, perspective, or view perspective (in addition to the spherical video) as part of the request for the tile or tiles. According to another exemplary implementation, the orientation sensor 835 detects orientation (or change in orientation) based on the image panning orientation as rendered on the HMD or display. For example, an HMD user may change the depth of focus. In other words, an HMD user may change his focus from a distant object to a nearby object with or without an orientation change (and vice versa). For example, a user can use a mouse, trackpad, or gesture (eg, on a touch-sensitive display) to select, move, drag, enlarge, etc., a portion of a spherical video or image as it is rendered on the display. May be used.

  The request for tiles may be communicated with the request for a frame of spherical video. Requests for tiles may be communicated together separately from requests for frames of spherical video. For example, a request for a tile may respond to a modified view, perspective, or view perspective, resulting in the need to replace a previously requested and / or queued tile.

  View position control module 805 receives and processes requests for tiles. For example, the view position control module 805 may determine a frame and the position of the tile or tiles in the frame based on the view. The view position control module 805 may then instruct the tile control module 810 to select the tile or tiles. Selecting a tile or multiple tiles may include passing parameters to video encoder 625. The parameters may be used by video encoder 625 during spherical video and / or tile encoding. Alternatively, selecting the tile or tiles may include selecting the tile or tiles from the view frame storage 795.

  Accordingly, the tile control module 810 may be configured to select a tile (or tiles) based on the view or perspective or view perspective of the user watching the spherical video. The tile may be a plurality of pixels selected based on the view. The plurality of pixels may be a block, a plurality of blocks, or a macroblock that may include a portion of a spherical image that may be viewed by a user. A part of the spherical image may have a length and a width. A portion of the spherical image may be two-dimensional or substantially two-dimensional. A tile may have a variable size (eg, what percentage of a sphere a tile covers). For example, the size of the tiles can be encoded and streamed based on, for example, how wide the viewer's field of view is and / or how fast the user is turning their head. For example, if the viewer is constantly looking around, a larger, lower quality tile may be selected. However, if the viewer is looking at one perspective, smaller and more detailed tiles may be selected.

  Accordingly, the orientation sensor 835 can be configured to detect the orientation (or change in orientation) of the viewer's eyes (or head). For example, the orientation sensor 835 may include an accelerometer to detect motion and a gyroscope to detect orientation. Alternatively or in addition, orientation sensor 835 may include a camera or infrared sensor focused on the viewer's eyes or head to determine the orientation of the viewer's eyes or head. Alternatively or additionally, the orientation sensor 835 may determine a portion of the spherical video or image as rendered on the display to detect the orientation of the spherical video or image. Orientation sensor 835 may be configured to communicate orientation and orientation change information to view position determination module 820.

  View position determination module 820 may be configured to determine a view or perspective view (eg, a portion of the spherical video that the viewer is currently viewing) for the spherical video. A view, perspective, or view perspective can be determined as a location, point, or focus on a spherical video. For example, the view may be a latitude and longitude position on a spherical video. A view, perspective, or view perspective may be determined as a side of a cube based on spherical video. The view (eg, latitude and longitude position, or edge) may be communicated to the view position control module 805 using, for example, hypertext transfer protocol (HTTP).

  View position control module 805 may be configured to determine the view position (eg, the frame and position within the frame) of the tile or tiles in the spherical video. For example, the view position control module 805 may select a rectangle centered at the view position, point, or focus (eg, latitude and longitude positions, or sides). The tile control module 810 may be configured to select the rectangle as a tile or multiple tiles. The tile control module 810 may be configured to instruct the video encoder 625 to encode the selected tile or tiles (eg, via parameters or configuration settings). And / or tile control module 810 may be configured to select a tile or tiles from view frame storage 795.

  As will be appreciated, the system 600 shown in FIG. 6A and the system 650 shown in FIG. 6B, and / or the system 800 shown in FIG. It may be implemented as 950 elements and / or extensions. Alternatively or in addition, the system 600 shown in FIG. 6A and the system 650 shown in FIG. 6B and / or the system 800 shown in FIG. General purpose computing device 900 and / or general purpose mobile computing device 950 may be implemented in a separate system having some or all of the features described in FIG.

  FIG. 9 is a schematic block diagram of a computing device and a mobile computing device that can be used to implement the techniques described herein. FIG. 9 is an example of a general purpose computing device 900 and a general purpose mobile computing device 950 that may be used with the techniques described herein. Computing device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be examples only and are intended to limit the implementation of the invention described in this document and / or in the claims. Not intended.

  The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high speed interface 908 connected to the memory 904 and a high speed expansion port 910, and a low speed interface connected to the low speed bus 914 and the storage device 906. 912. Each of the components 902, 904, 906, 908, 910, and 912 are interconnected using various buses and may be optionally mounted on a common motherboard or in other manners. The processor 902 is capable of processing instructions that are executed within the computing device 900, which instructions display graphical information for the GUI on an external input / output device such as a display 916 coupled to the high speed interface 908. Instructions stored in memory 904 or on storage device 906 are included for display. In other implementations, multiple processors and / or multiple buses may be used as appropriate with multiple memories and multiple types of memory. A plurality of computing devices 900 may also be connected, each device providing a portion of the required operation (eg, as a server bank, a group of blade servers, or a multiprocessor system).

  Memory 904 stores information within computing device 900. In one implementation, the memory 904 is one or more volatile memory units. In another implementation, the memory 904 is one or more non-volatile memory units. The memory 904 may also be another form of computer readable media such as a magnetic disk or optical disk.

  Storage device 906 can provide mass storage for computing device 900. In one implementation, the storage device 906 is a floppy disk device, hard disk device, optical disk device, or tape device, flash memory or other similar solid state memory device, or storage area network or other configuration. Or may be a computer readable medium, such as an array of devices including the device. A computer program product may be tangibly embodied on an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable or machine-readable medium, such as memory 904, storage device 906, or memory on processor 902.

  The high speed controller 908 manages bandwidth intensive operations for the computing device 900, while the low speed controller 912 manages lower bandwidth intensive operations. Such assignment of functions is exemplary only. In one implementation, high speed controller 908 couples to memory 904, display 916 (eg, via a graphics processor or accelerator), and a high speed expansion port 910 that can accept various expansion cards (not shown). Is done. In this implementation, low speed controller 912 is coupled to storage device 906 and low speed expansion port 914. A low-speed expansion port that can include various communication ports (eg, USB, Bluetooth, Ethernet, wireless Ethernet) to one or more input / output devices such as a keyboard, pointing device, scanner, or May be coupled to a networking device, such as a switch or router, via a network adapter, for example.

  The computing device 900 may be implemented in many different forms as shown in the figure. For example, it may be implemented multiple times as a standard server 920 or in a group of such servers. It may also be implemented as part of the rack server system 924. In addition, it may be implemented on a personal computer such as a laptop computer 922. Alternatively, components from computing device 900 may be combined with other components in a mobile device (not shown) such as device 950. Each such device may include one or more of the computing devices 900, 950, and the entire system may be comprised of multiple computing devices 900, 950 communicating with each other.

  Computing device 950 includes processor 952, memory 964, input / output devices such as display 954, communication interface 966, and transceiver 968, among other components. Device 950 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, some of which are optionally on a common motherboard or otherwise. It may be mounted.

  The processor 952 can execute instructions in the computing device 950, including instructions stored in the memory 964. The processor may be implemented as a chip set of chips that include separate analog and digital processors. The processor may provide cooperation between other components of the device 950, such as, for example, a user interface, applications executed by the device 950, and control of wireless communication by the device 950.

  The processor 952 may communicate with the user via a control interface 958 and a display interface 956 coupled to the display 954. The display 954 may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display), or an OLED (Organic Light Emitting Diode) display, or other suitable display technology. Display interface 956 may include appropriate circuitry for driving display 954 to present graphical information and other information to the user. The control interface 958 may receive commands from the user and convert them for delivery to the processor 952. In addition, an external interface 962 may be provided in communication with the processor 952 to allow near area communication between the device 950 and other devices. The external interface 962 may provide, for example, wired communication in some implementations, wireless communication in other implementations, and multiple interfaces may also be used.

  Memory 964 stores information within computing device 950. Memory 964 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 974 is also provided and may be connected to the device 950 via an expansion interface 972, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such extended memory 974 may provide extra storage space for device 950 or may store applications or other information for device 950. Specifically, extended memory 974 may include instructions for performing or supplementing the above-described process, and may also include secure information. Thus, for example, the extended memory 974 may be provided as a security module for the device 950 and may be programmed with instructions that allow the device 950 to be used safely. In addition, a secure application may be provided through the SIMM card with additional information, such as placing identification information on the SIMM card in a non-hackable manner.

  The memory may include, for example, flash memory and / or NVRAM memory as described below. In one implementation, the computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable or machine-readable medium, such as memory 964, expansion memory 974, or memory on processor 952, and may be received through transceiver 968 or external interface 962, for example.

  Device 950 may communicate wirelessly via communication interface 966, which may include digital signal processing circuitry as required. The communication interface 966, among other things, communicates under various modes or protocols such as GSM® voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA®, CDMA2000, or GPRS. May be provided. Such communication may occur, for example, via radio frequency transceiver 968. In addition, short range communications may occur using Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 970 may provide additional navigation-related and position-related wireless data to the device 950, which data is applied to the application running on the device 950. May be used as appropriate.

  Device 950 may also communicate in voice using an audio codec 960 that can receive verbal information from a user and convert it to usable digital information. The audio codec 960 may also generate sounds audible to the user, such as through a speaker, for example, in the handset of the device 950. Such sounds may include sounds from voice calls, may include recorded sounds (eg, voice messages, music files, etc.), and sound generated by applications running on device 950. May also be included.

  The computing device 950 may be implemented in many different forms as shown. For example, it may be implemented as a mobile phone 980. It may also be implemented as part of a smartphone 982, a personal digital assistant, or other similar mobile device.

  Some of the exemplary embodiments described above are described as processes or methods shown as flowcharts. Although these flowcharts describe the operation as a sequential process, many of the operations may be performed in parallel, simultaneously, or simultaneously. In addition, the order of operations may be rearranged. The process may be terminated when those operations are completed, but may have additional steps not included in the figure. These processes may correspond to methods, functions, procedures, subroutines, subprograms, and the like.

  The methods described above, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform necessary tasks may be stored on machine-readable or computer-readable media such as storage media. The processor may perform the necessary tasks.

  The specific structural and functional details disclosed herein are merely representative for describing exemplary embodiments. However, the exemplary embodiments may be embodied in many alternative forms and should not be construed as limited to only the embodiments set forth herein.

  It will be understood that terms such as first, second, etc. may be used herein to describe various elements, but that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element may be referred to as a second element, and, similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

  When an element is referred to as being connected or coupled to another element, it will be understood that an element may be directly connected or coupled to another element or that there may be intervening elements . In contrast, when an element is referred to as being directly connected or directly coupled to another element, there are no intervening elements. Other terms used to describe the relationship between elements should be interpreted in a similar manner (eg, “between” and “directly between”, “adjacent” and “directly adjacent”, etc.). is there.

  The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms “comprises, comprising” and / or “includes, including”, as used herein, identify the presence of the mentioned feature, integer, step, action, element and / or component. However, it will be further understood that it does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

  Also, in some alternative implementations, the functions / acts mentioned may occur out of the order shown in the figures. For example, two figures shown in succession may actually be performed simultaneously, or sometimes in reverse order, depending on the functionality / action involved.

  Unless defined otherwise, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which an exemplary embodiment belongs. Have In addition, terms such as defined in commonly used dictionaries, for example, should be construed as having a meaning consistent with their meaning in the context of the related art, and are not clearly defined herein It will be understood that as long as it is not interpreted in an idealized or overly formal sense.

  With respect to algorithms or symbolic representations of operations on data bits in software or computer memory, the exemplary embodiments described above and portions of corresponding detailed descriptions are presented. These descriptions and representations are intended to enable those skilled in the art to effectively convey the substance of their work to others skilled in the art. An algorithm is considered to be a consistent sequence of steps leading to a desired result when the term is used herein and generally. These steps are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

  In the exemplary embodiments described above, references to acts and symbolic representations of actions (eg, in the form of flowcharts) that can be implemented as program modules or functional processes perform specific tasks or specific abstract data. Includes routines, programs, objects, components, data structures, etc. that implement types and can be described and / or implemented using existing hardware with existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate array (FPGA) computers, or the like.

  However, it should be borne in mind that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise stated or apparent from the description, terms such as display processing, computing, calculation, or judgment manipulate data represented as physical electronic quantities in computer system registers and memory; A computer system or similar electronic computing device that converts the data into computer system memory, registers, or other data that is also represented as physical quantities in such information storage devices, transmitting devices, or display devices Refers to actions and processes.

  Also, aspects implemented by the software of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. . The program storage medium may be magnetic (eg, floppy disk or hard drive) or optical (eg, compact disk read only memory, ie CD ROM), read only or random access. Also good. Similarly, the transmission medium may be a twisted pair wire, coaxial cable, optical fiber, or some other suitable transmission medium known in the art. Exemplary embodiments are not limited by these aspects of a given implementation.

  Finally, while the appended claims describe specific combinations of the features described herein, the scope of the disclosure is not limited to those specific combinations claimed, but instead Regardless of whether the combination is specifically recited in the appended claims at this time, it extends to encompass any combination of the features or embodiments disclosed herein.

Claims (20)

Determining at least one preferred view perspective associated with a three-dimensional (3D) video;
Encoding a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality;
Encoding a second portion of the 3D video with a second quality, wherein the first quality is a higher quality compared to the second quality. Storing the first portion of the 3D video in a data store;
Storing the second portion of the 3D video in the data store;
Receiving a request for streaming video;
The method of claim 1, further comprising: streaming the first portion of the 3D video and the second portion of the 3D video as the streaming video from the data store. The method further includes receiving a request for streaming video, wherein the request includes displaying a user view perspective, and the method further includes:
Selecting a 3D video corresponding to the user view perspective as the encoded first portion of the 3D video;
The method of claim 1, comprising streaming the selected first portion of the 3D video and the second portion of the 3D video as the streaming video. Receiving a request for streaming video, wherein the request includes a display of a user view perspective associated with the 3D video, the method further comprising:
Determining whether the user view perspective is stored in a view perspective data store;
Determining that the user view perspective is stored in the view perspective data store, incrementing a counter associated with the user view perspective;
Determining that the user view perspective is not stored in the view perspective data store, adding the user view perspective to the view perspective data store and setting the counter associated with the user view perspective to 1; The method of claim 1 comprising: Encoding the second portion of the 3D video comprises using at least one first quality of service (QoS) parameter in a first pass encoding operation;
The encoding of the first portion of the 3D video comprises using at least one second quality of service (QoS) parameter in a second pass encoding operation. Method.   The step of determining the at least one preferred view perspective associated with the 3D video is based on at least one of a reference point seen so far and a view perspective seen so far. The method according to 1.   The at least one preferred view perspective associated with the 3D video includes the orientation of the viewer of the 3D video, the location of the viewer of the 3D video, the point of viewer of the 3D video, and the viewer of the 3D video The method of claim 1, wherein the method is based on at least one of: Determining the at least one preferred view perspective associated with the 3D video is based on a default view perspective;
The default view perspective is
Display device user characteristics,
Characteristics of the group associated with the user of the display device;
Director's cut and
The method of claim 1, wherein the method is based on at least one of the characteristics of the 3D video. Repetitively encoding at least a portion of the second portion of the 3D video with the first quality;
The method of claim 1, further comprising streaming the at least part of the second portion of the 3D video. A streaming server,
A controller configured to determine at least one preferred view perspective associated with a three-dimensional (3D) video;
An encoder, wherein the encoder is
Encoding a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality;
A streaming server configured to encode a second portion of the 3D video with a second quality, wherein the first quality is a higher quality than the second quality. The controller further includes:
Storing the first portion of the 3D video in a data store;
Storing the second portion of the 3D video in the data store;
Receiving a request for streaming video;
The streaming server of claim 10, configured to cause streaming the first portion of the 3D video and the second portion of the 3D video as the streaming video from the data store. The controller further includes:
Configured to cause a request for streaming video to be received, the request including a display of a user view perspective, the controller further comprising:
Selecting a 3D video corresponding to the user view perspective as the encoded first portion of the 3D video;
The streaming server of claim 10, configured to cause the selected first portion of the 3D video and the second portion of the 3D video to be streamed as the streaming video. The controller further includes:
Configured to cause a request for streaming video to be received, the request including a display of a user view perspective associated with the 3D video, the controller further comprising:
Determining whether the user view perspective is stored in a view perspective data store;
Determining that the user view perspective is stored in the view perspective data store, incrementing a counter associated with the user view perspective;
Determining that the user view perspective is not stored in the view perspective data store, adding the user view perspective to the view perspective data store and setting the counter associated with the user view perspective to 1; The streaming server of claim 10, wherein the streaming server is configured to cause Encoding the second portion of the 3D video includes using at least one first quality of service (QoS) parameter in a first pass encoding operation;
The encoding of the first portion of the 3D video comprises using at least one second quality of service (QoS) parameter in a second pass encoding operation. Streaming server.   The determining of the at least one preferred view perspective associated with the 3D video is based on at least one of a reference point seen so far and a view perspective seen so far. 10. The streaming server according to 10.   The at least one preferred view perspective associated with the 3D video includes an orientation of the viewer of the 3D video, a location of the viewer of the 3D video, a viewer point of the 3D video, and a viewer of the 3D video The streaming server of claim 10, wherein the streaming server is based on at least one of the following: Determining the at least one preferred view perspective associated with the 3D video is based on a default view perspective;
The default view perspective is
Display device user characteristics,
Characteristics of the group associated with the user of the display device;
Director's cut and
The streaming server according to claim 10, wherein the streaming server is based on at least one of the characteristics of the 3D video. The controller further includes:
Repetitively encoding at least a portion of the second portion of the 3D video with the first quality;
The streaming server of claim 10, configured to cause the at least a portion of the second portion of the 3D video to be streamed. A method,
Receiving a request for streaming video, wherein the request includes a display of a user view perspective associated with a three-dimensional (3D) video, the method further comprising:
Determining whether the user view perspective is stored in a view perspective data store;
Determining that the user view perspective is stored in the view perspective data store, incrementing a ranking value associated with the user view perspective;
If it is determined that the user view perspective is not stored in the view perspective data store, the user view perspective is added to the view perspective data store, and the ranking value associated with the user view perspective is set to 1. Including a method. Determining at least one preferred view perspective associated with the 3D video based on the ranking value associated with the stored user view perspective;
Encoding a first portion of the 3D video corresponding to the at least one preferred view perspective with a first quality;
20. The method of claim 19, further comprising: encoding a second portion of the 3D video with a second quality, wherein the first quality is a higher quality compared to the second quality. .

Images