[go: up one dir, main page]

US20180192063A1 - Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions - Google Patents

Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions Download PDF

Info

Publication number
US20180192063A1
US20180192063A1 US15/860,358 US201815860358A US2018192063A1 US 20180192063 A1 US20180192063 A1 US 20180192063A1 US 201815860358 A US201815860358 A US 201815860358A US 2018192063 A1 US2018192063 A1 US 2018192063A1
Authority
US
United States
Prior art keywords
data set
video
frame data
set type
type
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US15/860,358
Inventor
Zhuo Wang
Yongtao Tang
Ruoxi Zhao
Haoyan Zu
Chia-Chi Chang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Black Sails Technology Inc
Original Assignee
Black Sails Technology Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Black Sails Technology Inc filed Critical Black Sails Technology Inc
Priority to US15/860,358 priority Critical patent/US20180192063A1/en
Assigned to Black Sails Technology Inc. reassignment Black Sails Technology Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CHANG, CHIA-CHI, TANG, YONGTAO, WANG, ZHUO, ZHOU, RUOXI, ZU, HAOYAN
Publication of US20180192063A1 publication Critical patent/US20180192063A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179—Video signal processing therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/398—Synchronisation thereof; Control thereof
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/001—Model-based coding, e.g. wire frame
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0876—Network utilisation, e.g. volume of load or congestion level
    • H04L43/0888—Throughput
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/75—Media network packet handling
    • H04L65/752—Media network packet handling adapting media to network capabilities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/139—Format conversion, e.g. of frame-rate or size
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/189—Recording image signals; Reproducing recorded image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
    • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/366—Image reproducers using viewer tracking
    • H04N13/378—Image reproducers using viewer tracking for tracking rotational head movements around an axis perpendicular to the screen
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/366—Image reproducers using viewer tracking
    • H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/40—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/233—Processing of audio elementary streams
    • H04N21/2335—Processing of audio elementary streams involving reformatting operations of audio signals, e.g. by converting from one coding standard to another
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234345—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234381—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
    • H04N21/2387—Stream processing in response to a playback request from an end-user, e.g. for trick-play
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/239—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests
    • H04N21/2393—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests involving handling client requests
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
    • H04N21/2405—Monitoring of the internal components or processes of the server, e.g. server load
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440236—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179—Video signal processing therefor
    • H04N9/3185—Geometric adjustment, e.g. keystone or convergence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/79—Processing of colour television signals in connection with recording
    • H04N9/87—Regeneration of colour television signals
    • H04N9/8715—Regeneration of colour television signals involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2215/00—Indexing scheme for image rendering
    • G06T2215/16—Using real world measurements to influence rendering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting

Definitions

  • the present disclosure relates to video processing technology, and more particularly, to a method and a system for transcoding a VR video.
  • Virtual Reality is a computer simulation technology for creating and experiencing a virtual world. For example, a three-dimensional real-time image can be presented based on a technology which tracks a user's head, eyes or hands.
  • full-view video data can be pre-stored on a server, and then transmitted to a display device.
  • a display device can be glasses, a head-mounted display, etc.
  • a video is displayed on the display device in accordance with a viewport of the user.
  • the present disclosure relates to a method and a system for video transcoding to solve the above problems.
  • a provided method for transcoding a VR video which comprises: obtaining an audio data set and a frame data set type one by decoding source VR video data; obtaining a frame data set type two from the frame data set type one; obtaining a frame data set type three from the frame data set type one; obtaining an enhancement data set by subtracting the frame data set type two from the frame data set type three; obtaining a base video set by combining and segmenting the frame data set type two and the audio data set; and obtaining an enhancement video set by encoding and segmenting the enhancement data set.
  • the source VR video data and the frame data set type one have a source resolution
  • the frame data set type two and the frame data set type three have the same target resolution and are obtained by different manners
  • the base video set and the enhancement video set are used for VR video playbacks.
  • the source resolution is greater than or equal to the target resolution.
  • the step of obtaining the frame data set type two from the frame data set type one comprises: obtaining the frame data set type two by scaling down the frame data set type one to the target resolution losslessly.
  • the step of obtaining the frame data set type three from the frame data set type one comprises: compressing the frame data set type one by a predetermined video encoding method; obtaining a base video having a basic resolution from the compressed frame data set type one by decreasing resolution; obtaining base video data by decoding the base video; and obtaining the frame data set type three by scaling up the base video data with a interpolation algorithm type one.
  • the basic resolution is less than the target resolution.
  • the interpolation algorithm type one is a bilinear interpolation algorithm.
  • the step of obtaining the base video set by combining and segmenting the frame data set type two and the audio data set comprises: combining the frame data set type two and the audio data set into at least one video with sound track; and obtaining the base video set by segmenting the at least one video with sound track in accordance with a timeline.
  • the step of obtaining the enhancement video set by encoding and segmenting the enhancement data set comprises: compressing the enhancement data set by a predetermined video encoding method; and obtaining the enhancement video set by a segmenting process operated on the enhancement data set.
  • the segmenting process is performed in accordance with a timeline and/or in a spatial dimension.
  • the spatial dimension is related to a user's viewport.
  • a provided system for transcoding a VR video which comprises: a segmentation module, configured to decode source VR video data to obtain an audio data set and a frame data set type one; a first generating module, configured to obtain a frame data set type two from the frame data set type one; a second generating module, configured to obtain a frame data set type three from the frame data set type one; a difference calculation module, configured to subtract the frame data set type three from the frame data set type two to obtain an enhancement data set; a combining and segmenting module, configured to combine and segment the frame data set type two and the audio data set to obtain a base video set; an encoding and segmenting module, configured to encode and segment the enhancement data set to obtain an enhancement video set; and a storage module, configured to store the base video set and the enhancement video set.
  • the source VR video data and the frame data set type one have a source resolution
  • the frame data set type two and the frame data set type three have the same target resolution and are obtained by different manners
  • the base video set and the enhancement video set are used for VR video playbacks.
  • the source resolution is greater than or equal to the target resolution.
  • the present disclosure provides high quality display with better efficiency by transcoding a VR video to the base video and the enhancement video, by storing videos as the base video set and the enhancement video set, by transmitting the base video and the enhancement video separately, and by retrieving a high quality video from the base video and the enhancement video during video playbacks.
  • FIG. 1 is a diagram illustrating an example of the network-based virtual reality playback system
  • FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1 ;
  • FIG. 3 is a flowchart showing a method for transcoding a VR video according to an embodiment of the present disclosure
  • FIG. 4 is a flowchart showing detailed steps of obtaining a frame data set type three in FIG. 3 ;
  • FIG. 5 is a flowchart showing detailed steps of obtaining a base video set in FIG. 3 ;
  • FIG. 6 is a flowchart showing detailed steps of obtaining an enhancement video set in FIG. 3 ;
  • FIG. 7 is a block diagram illustrating a system for transcoding a VR video according to an embodiment of the present disclosure.
  • FIG. 1 is a diagram illustrating an example network of a VR playback system.
  • the VR playback system 10 includes a server 100 and a display device 120 which are coupled with each other through a network 110 , and a VR device.
  • the server 100 may be a stand-alone computer server or a server cluster.
  • the server 100 is used to store various video data and to store various applications that process these video data.
  • various daemons run on the server 100 in real time, so as to process various video data in the server 100 and to respond various requests from VR devices and the display device 120 .
  • the network 110 may be a selected one or selected ones from the group consisting of an internet, a local area network, an internet of things, and the like.
  • the display device 120 may be any of the computing devices, including a computer device having an independent display screen and a processing capability.
  • the display device 120 may be a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a palmtop computer, a personal digital assistant, a smart phone, an intelligent electrical apparatus, a game console, an iPad/iPhone, a video player, a DVD recorder/player, a television, or a home entertainment system.
  • the display device 120 may store VR player software as a VR player. When the VR player is started, it requests and downloads various video data from the server 100 , and renders and plays the video data in the display device.
  • the VR device 130 is a stand-alone head-mounted device that can interact with the display device 120 and the server 100 , to communicate the user's current information to the display device 120 and/or the server 100 through signaling.
  • the user's current information is, for example, a viewing angle of the user.
  • the display device 120 can flexibly process the currently played video data.
  • the display device 120 determines that a core viewing region for the user has been changed and starts to play video data with high resolution in the changed core viewing region.
  • the VR device 130 is a stand-alone head-mounted device.
  • the VR device 130 is not limited thereto, and the VR device 130 may also be an all-in-one head-mounted device.
  • the all-in-one head-mounted device itself has a display screen, so that it is not necessary to connect the all-in-one head-mounted device with the external display device.
  • the display device 120 may be eliminated.
  • the all-in-one head-mounted device is configured to obtain video data from the server 100 and to perform playback operation, and the all-in-one head-mounted device is also configured to detect user's current position modification information and to adjust the playback operation according to the position modification information.
  • FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1 . The method includes the following steps.
  • step S 10 a video data processing procedure is operated on server side.
  • step S 20 display device side obtains the position modification information by interacting with the VR device.
  • step S 30 according to the user's position modification information, the display device side requests the server side to provide the video data and receives the video data.
  • step S 40 the display device side renders the received video data.
  • the step S 10 is used to process the video data stored on the server side.
  • the source VR video data is further processed during the video data processing procedure according to the embodiments of the present disclosure.
  • the source VR video data with one video coding format is converted to the video data with another video coding format by a video encoding method.
  • the source VR video data with a low resolution is converted to the video data with a high resolution, in order to meet high demand required by the display device.
  • the video encoding method is a method for converting a video data file organized in one video coding format to a video data file organized in another video coding format by a specific compressing technique.
  • the most important coding standards for video streaming transmission includes H.261, H.263 and H.264 which are set by the International Telecommunication Union.
  • FIG. 3 is a flowchart showing a method for transcoding a VR video according to an embodiment of the present disclosure.
  • the method shown in FIG. 3 may be used in the above-described video data processing procedure as a preferred embodiment.
  • the method for transcoding the VR video includes following steps specifically.
  • step S 100 the source VR video data is decoded into an audio data set type one and a frame data set type one.
  • the source VR video data contains audio data and full-view video data.
  • the source VR video data contains video data in a horizontal 360-degree and vertical 120-degree viewing angle range.
  • the source VR video data may have an original resolution of 12,600 ⁇ 6,000 pixels.
  • the video data includes image data distributed in a plurality of consecutive frames. The image data of each frame is required to have the original resolution.
  • the source VR video data is decoded according to the video coding format of the source VR video data, and the audio data of the plurality of frames and the image data of the plurality of frames are extracted.
  • the audio data of the plurality of frames constitutes an audio data set
  • the image data of the plurality of frames constitutes a frame data set.
  • step S 200 a frame data set type two is obtained by scaling down the frame data set type one losslessly to a target resolution.
  • the source VR video data has the original resolution of 12,600 ⁇ 6,000 pixels and the target resolution is 6,300 ⁇ 3,000 pixels.
  • the video data with the target resolution can be obtained from the source VR video data with the original resolution. Since the frame data set type one contains video data distributed in the plurality of frames, the obtained frame data set type two also contains data distributed in the plurality of frames.
  • step S 300 the frame data set type one is scaled down during encoding and scaled up during decoding, for being converted into a frame data set type three with the target resolution.
  • the original resolution of the source VR video data is 12,600 ⁇ 6,000 pixels, so that data of each frame in the frame data set type one has a resolution of 12,600 ⁇ 6,000 pixels.
  • a video data having a basic resolution is obtained firstly by scaling down the frame data set during encoding, and then the frame data set type three is obtained by scaling up the video data to the target resolution during decoding.
  • Both of the frame data set type three and the frame data set type two contain video data distributed in the plurality of frames, and have a same resolution equal to the target resolution, however, the frame data set type three and the frame data set type two are obtained by different manners.
  • a following relationship is satisfied:
  • the video data with low resolution generates low quality images
  • the video data with high resolution generates high quality images
  • step S 400 an enhancement data set is obtained by a subtraction between the frame data set type two and the frame data set type three.
  • the enhancement data set is obtained by the subtraction between the frame data set type two and the frame data set type three, following the formulas described below.
  • value of each pixel in the enhancement data set is obtained by a subtraction operated on r, g, b components of a same pixel in the frame data set type two and the frame data set type three.
  • step S 500 a base video set is obtained by combining and segmenting the frame data set type two and the audio data set type one.
  • Video-audio data is obtained by combining data in each frame of the frame data set type two and data in each frame of the audio data set type one, and then the video-audio data is segmented in accordance with a timeline.
  • a time segmenting unit is defined, and the video-audio data with a certain duration is segmented into a plurality of videos with sound track according to the time segmenting unit.
  • a 2-minute time segmenting unit is defined, and the video-audio data with a 10-minute duration is segmented into five 2-minute videos with sound track according to the time segmenting unit.
  • step S 600 an enhancement video set is obtained by encoding, compressing and segmenting the enhancement data set.
  • the enhancement data set is segmented after being encoded by a predetermined coding method, wherein the predetermined coding method includes H.264, H.264 and so on.
  • the enhancement video set is obtained by a segmenting process operated on the encoded video data.
  • the segmenting process is performed in accordance with a timeline and/or in a spatial dimension.
  • the segmenting process is generally based on a time segmenting unit same with that defined in the step S 500 .
  • the segmenting process in the spatial dimension may be implemented in a variety of ways. For example, after a space segmenting unit is defined, the video data is segmented into a collection of sub data uniformly according to the space segmenting unit.
  • FIG. 4 is a flowchart showing detailed steps of obtaining a frame data set type three in FIG. 3 .
  • step S 301 the frame data set type one is compressed and encoded by a predetermined video encoding method.
  • each frame of data in the frame data set type one is compressed by a specific video encoding method, which is, for example, based on H.264 or H.265 format.
  • step S 302 a base video having a basic resolution is obtained from the compressed frame data set type one by decreasing resolution.
  • the base video is obtained according to the frame data set type one, wherein the base video is the video data having the basic resolution.
  • step S 303 a base video data is obtained by decoding the base video having the basic resolution.
  • the base video is decoded by a decoding method corresponding to the step S 301 , in order to generate a frame data set having the basic resolution.
  • step S 304 a frame data set type three is obtained by interpolation algorithm from the base video data.
  • video data having the target resolution is obtained based on the base video having the basic resolution, that is, the frame data set type three is obtained.
  • the target resolution is equal to or higher than the basic resolution.
  • FIG. 5 is a flowchart showing detailed steps of obtaining a base video set in FIG. 3 .
  • step S 501 at least one video with sound track is obtained by combining the frame data set type two and the audio data set type one.
  • the at least one completed video with sound track is obtained by recombining the frame data set type two and the audio data set type one.
  • the at least one video with sound track has the target resolution. Different videos with sound track of the at least one video with sound track may have a same duration or different durations.
  • step S 502 the at least one video with sound track is segmented into a base video set.
  • the at least one video with sound track is segmented in accordance with a timeline.
  • the at least one video with sound track is segmented into a plurality of sub pieces with a same duration or different durations, and the plurality of sub pieces constitute the base video set.
  • FIG. 6 is a flowchart showing detailed steps of obtaining an enhancement video set in FIG. 3 .
  • step S 601 the enhancement data set is compressed and encoded by a predetermined video encoding method.
  • each frame of data in the frame data set type one is compressed by a specific video encoding method, which is, for example, based on H.264 or H.265 format.
  • step S 602 the encoded enhancement data set is segmented to generate an enhancement video set.
  • the enhancement video set is obtained.
  • the segmenting process is performed in accordance with a timeline and/or in a spatial dimension.
  • the segmenting process is generally based on a time segmenting unit same with that defined in the step S 500 .
  • the segmenting process in the spatial dimension may be implemented in a variety of ways. For example, a space segmenting unit is defined, and the video data is segmented into a collection of sub data uniformly according to the space segmenting unit.
  • the method for transcoding the VR video processes VR video data into the base video set and the enhancement video set.
  • the enhancement video set is superposed with the base video set, in order to implement high quality display.
  • the base video set and the enhancement video set are transmitted separately to improve transmission efficiency.
  • FIG. 7 is a block diagram illustrating a system for transcoding a VR video according to an embodiment of the present disclosure.
  • the system for transcoding the VR video comprises a segmentation module 701 , a first generating module 702 , a second generating module 703 , a difference calculation module 704 , a combining and segmenting module 705 , an encoding and segmenting module 706 and a storage module 707 .
  • the segmentation module 701 is configured to decode source VR video data to obtain an audio data set and a frame data set type one, wherein the source VR video data and the frame data set type one both have a source resolution.
  • the first generating module 702 is configured to obtain a frame data set type two from the frame data set type one.
  • the second generating module 703 is configured to obtain a frame data set type three from the frame data set type one.
  • the frame data set type two and the frame data set type three have a same target resolution and are obtained by different manners.
  • the frame data set type two is obtained by decreasing a resolution of the frame data set type one to the target resolution
  • the frame data set type three is obtained by decreasing the resolution of the frame data set type one to the target resolution after the frame data set type one is encoded and compressed.
  • the first, the second and the frame data set type threes each contain data distributed in a plurality of frames.
  • the difference calculation module 704 is configured to subtract the frame data set type three from the frame data set type two to obtain an enhancement data set.
  • the enhancement data set is obtained by subtracting the frame data set type three generated by the second generating module 703 from the frame data set type two generated by the first generating module 702 .
  • the combining and segmenting module 705 is configured to combine and segment the frame data set type two and the audio data set to obtain a base video set.
  • the encoding and segmenting module 706 is configured to encode and segment the enhancement data set to obtain an enhancement video set.
  • the storage module 707 is configured to store the base video set and the enhancement video set.
  • the base video set and the enhancement video set are used for VR video playbacks.
  • the enhancement video set is superposed with the base video set, in order to implement high quality display.
  • the source resolution is greater than or equal to the target resolution.
  • the system for transcoding the VR video is configured to process VR video data into the base video set and the enhancement video set, and to transmit the base video set and the enhancement video set in order to implement high quality transmission, at the same time, effect of the playback is not influenced or even improved.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Graphics (AREA)
  • Human Computer Interaction (AREA)
  • Geometry (AREA)
  • Environmental & Geological Engineering (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Image Generation (AREA)
  • Processing Or Creating Images (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Image Processing (AREA)

Abstract

A VR video transcoding method is disclosed. In the method, source VR video data is decoded to obtain an audio data set and a frame data set type one. The source VR video data and the frame data set type one have a source resolution. A frame data set type two and a third frame data are obtained from the frame data set type one. The frame data set type two and the frame data set type three have the same target resolution and are obtained by different manners. An enhancement data set is obtained by subtracting the frame data set type three from the frame data set type two. A base video set is obtained by combining and segmenting the frame data set type two and the audio data set. An enhancement video set is obtained by encoding and segmenting the enhancement data set. The base video set and the enhancement video set are used for VR video playbacks. The base video set and the enhancement video set are transmitted separately and combined with each other into video content to improve transmission efficiency.

Description

    CROSS-REFERENCE TO RELATED APPLICATION
  • This application claims the priority and benefit of U.S. provisional application 62/441,936, filed on Jan. 3, 2017, which is incorporated herein by reference in its entirety.
    • BACKGROUND OF THE DISCLOSURE Field of the Disclosure
  • The present disclosure relates to video processing technology, and more particularly, to a method and a system for transcoding a VR video.
    • Background of the Disclosure
  • Virtual Reality (VR) is a computer simulation technology for creating and experiencing a virtual world. For example, a three-dimensional real-time image can be presented based on a technology which tracks a user's head, eyes or hands. In the network-based virtual reality technology, full-view video data can be pre-stored on a server, and then transmitted to a display device. A display device can be glasses, a head-mounted display, etc. A video is displayed on the display device in accordance with a viewport of the user.
  • However, a high-resolution video data occupies large transmission of bandwidth and requires high computing power from the display devices. Presenting high-resolution VR video on internet is difficult, Precisely, the existing video streaming technology cannot fulfill the virtual reality.
  • Therefore, in order to present VR video smoothly in real-time, it is desirable to further improve the existing video streaming technology to save bandwidth and reduce performance requirements for display devices, by a new way to encode and store the VR video data on the server.
    • SUMMARY OF THE DISCLOSURE
  • In view of this, the present disclosure relates to a method and a system for video transcoding to solve the above problems.
  • According to the first aspect of the present disclosure, there is a provided method for transcoding a VR video, which comprises: obtaining an audio data set and a frame data set type one by decoding source VR video data; obtaining a frame data set type two from the frame data set type one; obtaining a frame data set type three from the frame data set type one; obtaining an enhancement data set by subtracting the frame data set type two from the frame data set type three; obtaining a base video set by combining and segmenting the frame data set type two and the audio data set; and obtaining an enhancement video set by encoding and segmenting the enhancement data set. Wherein, the source VR video data and the frame data set type one have a source resolution, the frame data set type two and the frame data set type three have the same target resolution and are obtained by different manners, and the base video set and the enhancement video set are used for VR video playbacks.
  • Preferably, the source resolution is greater than or equal to the target resolution.
  • Preferably, the step of obtaining the frame data set type two from the frame data set type one comprises: obtaining the frame data set type two by scaling down the frame data set type one to the target resolution losslessly.
  • Preferably, the step of obtaining the frame data set type three from the frame data set type one comprises: compressing the frame data set type one by a predetermined video encoding method; obtaining a base video having a basic resolution from the compressed frame data set type one by decreasing resolution; obtaining base video data by decoding the base video; and obtaining the frame data set type three by scaling up the base video data with a interpolation algorithm type one. Wherein, the basic resolution is less than the target resolution.
  • Preferably, the interpolation algorithm type one is a bilinear interpolation algorithm.
  • Preferably, the step of obtaining the base video set by combining and segmenting the frame data set type two and the audio data set comprises: combining the frame data set type two and the audio data set into at least one video with sound track; and obtaining the base video set by segmenting the at least one video with sound track in accordance with a timeline.
  • Preferably, the step of obtaining the enhancement video set by encoding and segmenting the enhancement data set comprises: compressing the enhancement data set by a predetermined video encoding method; and obtaining the enhancement video set by a segmenting process operated on the enhancement data set.
  • Preferably, the segmenting process is performed in accordance with a timeline and/or in a spatial dimension.
  • Preferably, the spatial dimension is related to a user's viewport.
  • According to the second aspect of the present disclosure, there is a provided system for transcoding a VR video, which comprises: a segmentation module, configured to decode source VR video data to obtain an audio data set and a frame data set type one; a first generating module, configured to obtain a frame data set type two from the frame data set type one; a second generating module, configured to obtain a frame data set type three from the frame data set type one; a difference calculation module, configured to subtract the frame data set type three from the frame data set type two to obtain an enhancement data set; a combining and segmenting module, configured to combine and segment the frame data set type two and the audio data set to obtain a base video set; an encoding and segmenting module, configured to encode and segment the enhancement data set to obtain an enhancement video set; and a storage module, configured to store the base video set and the enhancement video set. Wherein, the source VR video data and the frame data set type one have a source resolution, the frame data set type two and the frame data set type three have the same target resolution and are obtained by different manners, and the base video set and the enhancement video set are used for VR video playbacks.
  • Preferably, the source resolution is greater than or equal to the target resolution.
  • The present disclosure provides high quality display with better efficiency by transcoding a VR video to the base video and the enhancement video, by storing videos as the base video set and the enhancement video set, by transmitting the base video and the enhancement video separately, and by retrieving a high quality video from the base video and the enhancement video during video playbacks.
    • BRIEF DESCRIPTION OF THE DRAWINGS
  • Features and benefits relates to the present disclosure will be emphasized by attached figures and descriptions. The attached figures includes:
  • FIG. 1 is a diagram illustrating an example of the network-based virtual reality playback system;
  • FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1;
  • FIG. 3 is a flowchart showing a method for transcoding a VR video according to an embodiment of the present disclosure;
  • FIG. 4 is a flowchart showing detailed steps of obtaining a frame data set type three in FIG. 3;
  • FIG. 5 is a flowchart showing detailed steps of obtaining a base video set in FIG. 3;
  • FIG. 6 is a flowchart showing detailed steps of obtaining an enhancement video set in FIG. 3; and
  • FIG. 7 is a block diagram illustrating a system for transcoding a VR video according to an embodiment of the present disclosure.
    •  DETAILED DESCRIPTION OF THE DISCLOSURE
  • Exemplary embodiments of the present disclosure will be described in more details below with reference to the accompanying drawings. In the drawings, like reference numerals denote like members. The figures are not drawn to scale, for the sake of clarity. Moreover, some well-known parts may not be shown.
  • FIG. 1 is a diagram illustrating an example network of a VR playback system. The VR playback system 10 includes a server 100 and a display device 120 which are coupled with each other through a network 110, and a VR device. For example, the server 100 may be a stand-alone computer server or a server cluster. The server 100 is used to store various video data and to store various applications that process these video data. For example, various daemons run on the server 100 in real time, so as to process various video data in the server 100 and to respond various requests from VR devices and the display device 120. The network 110 may be a selected one or selected ones from the group consisting of an internet, a local area network, an internet of things, and the like. For example, the display device 120 may be any of the computing devices, including a computer device having an independent display screen and a processing capability. The display device 120 may be a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a palmtop computer, a personal digital assistant, a smart phone, an intelligent electrical apparatus, a game console, an iPad/iPhone, a video player, a DVD recorder/player, a television, or a home entertainment system. The display device 120 may store VR player software as a VR player. When the VR player is started, it requests and downloads various video data from the server 100, and renders and plays the video data in the display device. In this example, the VR device 130 is a stand-alone head-mounted device that can interact with the display device 120 and the server 100, to communicate the user's current information to the display device 120 and/or the server 100 through signaling. The user's current information is, for example, a viewing angle of the user. According to the information, the display device 120 can flexibly process the currently played video data. In some embodiments, when a position of user's helmet changes, the display device 120 determines that a core viewing region for the user has been changed and starts to play video data with high resolution in the changed core viewing region.
  • In the above embodiment, the VR device 130 is a stand-alone head-mounted device. However, those skilled in the art should understand that the VR device 130 is not limited thereto, and the VR device 130 may also be an all-in-one head-mounted device. The all-in-one head-mounted device itself has a display screen, so that it is not necessary to connect the all-in-one head-mounted device with the external display device. For example, in this example, if the all-in-one head-mounted device is used as the VR device, the display device 120 may be eliminated. At this point, the all-in-one head-mounted device is configured to obtain video data from the server 100 and to perform playback operation, and the all-in-one head-mounted device is also configured to detect user's current position modification information and to adjust the playback operation according to the position modification information.
  • FIG. 2 is a flowchart showing a method used in the VR playback system of FIG. 1. The method includes the following steps.
  • In step S10, a video data processing procedure is operated on server side.
  • In step S20, display device side obtains the position modification information by interacting with the VR device.
  • In step S30, according to the user's position modification information, the display device side requests the server side to provide the video data and receives the video data.
  • In step S40, the display device side renders the received video data.
  • Wherein, the step S10 is used to process the video data stored on the server side. Unlike conventional processing method in which source VR video data is directly stored as display data and provided to the display device, the source VR video data is further processed during the video data processing procedure according to the embodiments of the present disclosure. For example, the source VR video data with one video coding format is converted to the video data with another video coding format by a video encoding method. Or, the source VR video data with a low resolution is converted to the video data with a high resolution, in order to meet high demand required by the display device. Wherein, the video encoding method is a method for converting a video data file organized in one video coding format to a video data file organized in another video coding format by a specific compressing technique. Currently, the most important coding standards for video streaming transmission includes H.261, H.263 and H.264 which are set by the International Telecommunication Union.
  • FIG. 3 is a flowchart showing a method for transcoding a VR video according to an embodiment of the present disclosure. The method shown in FIG. 3 may be used in the above-described video data processing procedure as a preferred embodiment. The method for transcoding the VR video includes following steps specifically.
  • In step S100, the source VR video data is decoded into an audio data set type one and a frame data set type one.
  • The source VR video data contains audio data and full-view video data. For example, the source VR video data contains video data in a horizontal 360-degree and vertical 120-degree viewing angle range. The source VR video data may have an original resolution of 12,600×6,000 pixels. The video data includes image data distributed in a plurality of consecutive frames. The image data of each frame is required to have the original resolution. In this step, the source VR video data is decoded according to the video coding format of the source VR video data, and the audio data of the plurality of frames and the image data of the plurality of frames are extracted. The audio data of the plurality of frames constitutes an audio data set, and the image data of the plurality of frames constitutes a frame data set.
  • In step S200, a frame data set type two is obtained by scaling down the frame data set type one losslessly to a target resolution.
  • For example, the source VR video data has the original resolution of 12,600×6,000 pixels and the target resolution is 6,300×3,000 pixels. By decreasing the resolution of the source VR video data, the video data with the target resolution can be obtained from the source VR video data with the original resolution. Since the frame data set type one contains video data distributed in the plurality of frames, the obtained frame data set type two also contains data distributed in the plurality of frames.
  • In step S300, the frame data set type one is scaled down during encoding and scaled up during decoding, for being converted into a frame data set type three with the target resolution.
  • For example, the original resolution of the source VR video data is 12,600×6,000 pixels, so that data of each frame in the frame data set type one has a resolution of 12,600×6,000 pixels. In this step, a video data having a basic resolution is obtained firstly by scaling down the frame data set during encoding, and then the frame data set type three is obtained by scaling up the video data to the target resolution during decoding. Both of the frame data set type three and the frame data set type two contain video data distributed in the plurality of frames, and have a same resolution equal to the target resolution, however, the frame data set type three and the frame data set type two are obtained by different manners. In addition, in the embodiment, a following relationship is satisfied:

  • the original resolution≥the target resolution≥the basic resolution.
  • For the same display device, the video data with low resolution generates low quality images, and the video data with high resolution generates high quality images.
  • In step S400, an enhancement data set is obtained by a subtraction between the frame data set type two and the frame data set type three.
  • In this step, for example, the enhancement data set is obtained by the subtraction between the frame data set type two and the frame data set type three, following the formulas described below.
  • It is assumed that Px, yOriginal=(r, g, b)T is a pixel with a coordinate (x, y) in the frame data set type two, wherein r, g, b∈[L,H], and Px, yScaledBase=(r′, g′, b′)T is a pixel a pixel in the frame data set type two with a coordinate (x, y) in the frame data set type three, wherein r′, g′, b′∈[L, H], then all x and y satisfies the following formula (1):
  • Px , y NormalizedResidual = Px , y Original - Px , y ScaledBase + H - L 2 ( 1 )
  • That is to say, value of each pixel in the enhancement data set is obtained by a subtraction operated on r, g, b components of a same pixel in the frame data set type two and the frame data set type three.
  • However, those skilled in the art should understand that the present disclosure is not limited thereto, it can also be extended into other embodiments using other color space models such as YUV, CMY, HSV, HSI.
  • In step S500, a base video set is obtained by combining and segmenting the frame data set type two and the audio data set type one.
  • Video-audio data is obtained by combining data in each frame of the frame data set type two and data in each frame of the audio data set type one, and then the video-audio data is segmented in accordance with a timeline. For example, a time segmenting unit is defined, and the video-audio data with a certain duration is segmented into a plurality of videos with sound track according to the time segmenting unit. For example, a 2-minute time segmenting unit is defined, and the video-audio data with a 10-minute duration is segmented into five 2-minute videos with sound track according to the time segmenting unit.
  • In step S600, an enhancement video set is obtained by encoding, compressing and segmenting the enhancement data set.
  • The enhancement data set is segmented after being encoded by a predetermined coding method, wherein the predetermined coding method includes H.264, H.264 and so on. The enhancement video set is obtained by a segmenting process operated on the encoded video data. The segmenting process is performed in accordance with a timeline and/or in a spatial dimension. For convenience, the segmenting process is generally based on a time segmenting unit same with that defined in the step S500. While the segmenting process in the spatial dimension may be implemented in a variety of ways. For example, after a space segmenting unit is defined, the video data is segmented into a collection of sub data uniformly according to the space segmenting unit.
  • FIG. 4 is a flowchart showing detailed steps of obtaining a frame data set type three in FIG. 3.
  • In step S301, the frame data set type one is compressed and encoded by a predetermined video encoding method.
  • In this step, each frame of data in the frame data set type one is compressed by a specific video encoding method, which is, for example, based on H.264 or H.265 format.
  • In step S302, a base video having a basic resolution is obtained from the compressed frame data set type one by decreasing resolution.
  • By using one of the algorithms including nearest neighbor interpolation algorithm, bilinear interpolation algorithm, cubic convolution algorithm and other algorithms, the base video is obtained according to the frame data set type one, wherein the base video is the video data having the basic resolution.
  • In step S303, a base video data is obtained by decoding the base video having the basic resolution.
  • The base video is decoded by a decoding method corresponding to the step S301, in order to generate a frame data set having the basic resolution.
  • In step S304, a frame data set type three is obtained by interpolation algorithm from the base video data.
  • By using one or more of the algorithms including nearest neighbor interpolation algorithm, bilinear interpolation algorithm, cubic convolution algorithm and other algorithms, video data having the target resolution is obtained based on the base video having the basic resolution, that is, the frame data set type three is obtained. The target resolution is equal to or higher than the basic resolution.
  • FIG. 5 is a flowchart showing detailed steps of obtaining a base video set in FIG. 3.
  • In step S501, at least one video with sound track is obtained by combining the frame data set type two and the audio data set type one.
  • In this step, the at least one completed video with sound track is obtained by recombining the frame data set type two and the audio data set type one. The at least one video with sound track has the target resolution. Different videos with sound track of the at least one video with sound track may have a same duration or different durations.
  • In step S502, the at least one video with sound track is segmented into a base video set.
  • The at least one video with sound track is segmented in accordance with a timeline. The at least one video with sound track is segmented into a plurality of sub pieces with a same duration or different durations, and the plurality of sub pieces constitute the base video set.
  • FIG. 6 is a flowchart showing detailed steps of obtaining an enhancement video set in FIG. 3.
  • In step S601, the enhancement data set is compressed and encoded by a predetermined video encoding method.
  • In this step, each frame of data in the frame data set type one is compressed by a specific video encoding method, which is, for example, based on H.264 or H.265 format.
  • In step S602, the encoded enhancement data set is segmented to generate an enhancement video set.
  • In this step, by a segmenting process which is same with the segmenting process in the step S500, the enhancement video set is obtained.
  • The segmenting process is performed in accordance with a timeline and/or in a spatial dimension. For convenience, the segmenting process is generally based on a time segmenting unit same with that defined in the step S500. While the segmenting process in the spatial dimension may be implemented in a variety of ways. For example, a space segmenting unit is defined, and the video data is segmented into a collection of sub data uniformly according to the space segmenting unit.
  • The method for transcoding the VR video according to the embodiments of the present disclosure processes VR video data into the base video set and the enhancement video set. During playback, the enhancement video set is superposed with the base video set, in order to implement high quality display. The base video set and the enhancement video set are transmitted separately to improve transmission efficiency.
  • FIG. 7 is a block diagram illustrating a system for transcoding a VR video according to an embodiment of the present disclosure. The system for transcoding the VR video comprises a segmentation module 701, a first generating module 702, a second generating module 703, a difference calculation module 704, a combining and segmenting module 705, an encoding and segmenting module 706 and a storage module 707.
  • The segmentation module 701 is configured to decode source VR video data to obtain an audio data set and a frame data set type one, wherein the source VR video data and the frame data set type one both have a source resolution.
  • The first generating module 702 is configured to obtain a frame data set type two from the frame data set type one.
  • The second generating module 703 is configured to obtain a frame data set type three from the frame data set type one. The frame data set type two and the frame data set type three have a same target resolution and are obtained by different manners. For example, the frame data set type two is obtained by decreasing a resolution of the frame data set type one to the target resolution, and the frame data set type three is obtained by decreasing the resolution of the frame data set type one to the target resolution after the frame data set type one is encoded and compressed. The first, the second and the frame data set type threes each contain data distributed in a plurality of frames.
  • The difference calculation module 704 is configured to subtract the frame data set type three from the frame data set type two to obtain an enhancement data set. The enhancement data set is obtained by subtracting the frame data set type three generated by the second generating module 703 from the frame data set type two generated by the first generating module 702.
  • The combining and segmenting module 705 is configured to combine and segment the frame data set type two and the audio data set to obtain a base video set.
  • The encoding and segmenting module 706 is configured to encode and segment the enhancement data set to obtain an enhancement video set.
  • The storage module 707 is configured to store the base video set and the enhancement video set. The base video set and the enhancement video set are used for VR video playbacks. During the playback, the enhancement video set is superposed with the base video set, in order to implement high quality display. Preferably, the source resolution is greater than or equal to the target resolution.
  • The system for transcoding the VR video according to the embodiments of the present disclosure is configured to process VR video data into the base video set and the enhancement video set, and to transmit the base video set and the enhancement video set in order to implement high quality transmission, at the same time, effect of the playback is not influenced or even improved.
  • The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical applications, to thereby enable others skilled in the art to best utilize the disclosure. The disclosure is intended to cover alternatives, modifications and equivalents that may be included within the spirit and scope of the disclosure as defined by the appended claims.
  • The foregoing descriptions of specific embodiments of the present disclosure have been presented, but are not intended to limit the disclosure to the precise forms disclosed. It will be readily apparent to one skilled in the art that many modifications and changes may be made in the present disclosure. Any modifications, equivalence, variations of the preferred embodiments can be made without departing from the doctrine and spirit of the present disclosure.

Claims (10)

1. A method for transcoding a VR video, comprising:
obtaining an audio data set and a frame data set type one by decoding source VR video data, wherein the source VR video data and the frame data set type one have a source resolution;
obtaining a frame data set type two from the frame data set type one;
obtaining a frame data set type three from the frame data set type one, wherein the frame data set type two and the frame data set type three have the same target resolution and are obtained by different manners;
obtaining an enhancement data set by subtracting the frame data set type two from the frame data set type three;
obtaining a base video set by combining and segmenting the frame data set type two and the audio data set; and
obtaining an enhancement video set by encoding and segmenting the enhancement data set, wherein the base video set and the enhancement video set are used for VR video playbacks.
2. The method according to claim 1, wherein the source resolution is equal to or higher than the target resolution.
3. The method according to claim 2, wherein the step of obtaining the frame data set type two from the frame data set type one comprises:
obtaining the frame data set type two by scaling down the frame data set type one losslessly to the target resolution.
4. The method according to claim 2, wherein the step of obtaining the frame data set type three from the frame data set type one comprises:
compressing the frame data set type one by a predetermined video encoding method;
obtaining a base video having a basic resolution from the compressed frame data set type one by decreasing resolution, wherein the basic resolution is less than the target resolution;
obtaining base video data by decoding the base video; and
obtaining the frame data set type three by scaling up the base video data with a interpolation algorithm type one.
5. The method according to claim 4, wherein the interpolation algorithm type one is a bilinear interpolation algorithm.
6. The method according to claim 1, wherein the step of obtaining the base video set by combining and segmenting the frame data set type two and the audio data set comprises:
combining the frame data set type two and the audio data set into at least one video with sound track; and
obtaining the base video set by segmenting the at least one video with sound track in accordance with a timeline.
7. The method according to claim 1, wherein the step of obtaining the enhancement video set by encoding and segmenting the enhancement data set comprises:
compressing the enhancement data set by a predetermined video encoding method; and
obtaining the enhancement video set by a segmenting process operated on the enhancement data set.
8. The method according to claim 7, wherein the segmenting process is performed in accordance with a timeline and/or in a spatial dimension.
9. The method according to claim 8, wherein the spatial dimension is related to a user's viewport.
10. A system for transcoding a VR video, comprising:
a segmentation module, configured to decode source VR video data to obtain an audio data set and a frame data set type one, wherein the source VR video data and the frame data set type one have a source resolution;
a first generating module, configured to obtain a frame data set type two from the frame data set type one;
a second generating module, configured to obtain a frame data set type three from the frame data set type one, wherein the frame data set type two and the frame data set type three have the same target resolution and are obtained by different manners;
a difference calculation module, configured to subtract the frame data set type three from the frame data set type two to obtain an enhancement data set;
a combining and segmenting module, configured to combine and segment the frame data set type two and the audio data set to obtain a base video set;
an encoding and segmenting module, configured to encode and segment the enhancement data set to obtain an enhancement video set, wherein the base video set and the enhancement video set are used for VR video playbacks; and
a storage module, configured to store the base video set and the enhancement video set.
US15/860,358 2017-01-03 2018-01-02 Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions Abandoned US20180192063A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/860,358 US20180192063A1 (en) 2017-01-03 2018-01-02 Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201762441936P 2017-01-03 2017-01-03
US15/860,358 US20180192063A1 (en) 2017-01-03 2018-01-02 Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions

Publications (1)

Publication Number Publication Date
US20180192063A1 true US20180192063A1 (en) 2018-07-05

Family

ID=62711388

Family Applications (6)

Application Number Title Priority Date Filing Date
US15/860,430 Abandoned US20180191868A1 (en) 2017-01-03 2018-01-02 Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization
US15/860,471 Abandoned US20180192022A1 (en) 2017-01-03 2018-01-02 Method and System for Real-time Rendering Displaying Virtual Reality (VR) On Mobile Using Head-Up Display Devices
US15/860,494 Abandoned US20180189980A1 (en) 2017-01-03 2018-01-02 Method and System for Providing Virtual Reality (VR) Video Transcoding and Broadcasting
US15/860,392 Abandoned US20180192044A1 (en) 2017-01-03 2018-01-02 Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming
US15/860,358 Abandoned US20180192063A1 (en) 2017-01-03 2018-01-02 Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions
US15/860,449 Expired - Fee Related US10334238B2 (en) 2017-01-03 2018-01-02 Method and system for real-time rendering displaying high resolution virtual reality (VR) video

Family Applications Before (4)

Application Number Title Priority Date Filing Date
US15/860,430 Abandoned US20180191868A1 (en) 2017-01-03 2018-01-02 Method and System for Downloading Multiple Resolutions Bitrate for Virtual Reality (VR) Video Streaming Optimization
US15/860,471 Abandoned US20180192022A1 (en) 2017-01-03 2018-01-02 Method and System for Real-time Rendering Displaying Virtual Reality (VR) On Mobile Using Head-Up Display Devices
US15/860,494 Abandoned US20180189980A1 (en) 2017-01-03 2018-01-02 Method and System for Providing Virtual Reality (VR) Video Transcoding and Broadcasting
US15/860,392 Abandoned US20180192044A1 (en) 2017-01-03 2018-01-02 Method and System for Providing A Viewport Division Scheme for Virtual Reality (VR) Video Streaming

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/860,449 Expired - Fee Related US10334238B2 (en) 2017-01-03 2018-01-02 Method and system for real-time rendering displaying high resolution virtual reality (VR) video

Country Status (2)

Country Link
US (6) US20180191868A1 (en)
CN (6) CN108366272A (en)

Families Citing this family (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10291910B2 (en) * 2016-02-12 2019-05-14 Gopro, Inc. Systems and methods for spatially adaptive video encoding
US10331862B2 (en) * 2017-04-20 2019-06-25 Cisco Technology, Inc. Viewport decryption
CN110324601A (en) * 2018-03-27 2019-10-11 京东方科技集团股份有限公司 Rendering method, computer product and display device
US11232532B2 (en) * 2018-05-30 2022-01-25 Sony Interactive Entertainment LLC Multi-server cloud virtual reality (VR) streaming
US10623791B2 (en) 2018-06-01 2020-04-14 At&T Intellectual Property I, L.P. Field of view prediction in live panoramic video streaming
US10812774B2 (en) 2018-06-06 2020-10-20 At&T Intellectual Property I, L.P. Methods and devices for adapting the rate of video content streaming
US10616621B2 (en) * 2018-06-29 2020-04-07 At&T Intellectual Property I, L.P. Methods and devices for determining multipath routing for panoramic video content
US11019361B2 (en) 2018-08-13 2021-05-25 At&T Intellectual Property I, L.P. Methods, systems and devices for adjusting panoramic view of a camera for capturing video content
CN109343518B (en) * 2018-09-03 2021-07-02 浙江大丰实业股份有限公司 Universal car platform on-site drive platform
US11128869B1 (en) * 2018-10-22 2021-09-21 Bitmovin, Inc. Video encoding based on customized bitrate table
CN109375369B (en) * 2018-11-23 2021-05-18 国网天津市电力公司 Distortion preprocessing method in VR (virtual reality) large-screen cinema mode
CN111510777B (en) 2019-01-30 2021-11-23 上海哔哩哔哩科技有限公司 Method and device for measuring network speed, computer equipment and readable storage medium
CN111669666A (en) * 2019-03-08 2020-09-15 北京京东尚科信息技术有限公司 Method, device and system for simulating reality
CN111866485A (en) * 2019-04-25 2020-10-30 中国移动通信有限公司研究院 Stereoscopic picture projection and transmission method, device and computer-readable storage medium
CN110336994B (en) * 2019-07-04 2021-06-22 上海索倍信息科技有限公司 Naked eye 3D display system
CN110381331A (en) * 2019-07-23 2019-10-25 深圳市道通智能航空技术有限公司 A kind of image processing method, device, equipment of taking photo by plane and storage medium
CN110490962B (en) * 2019-08-20 2023-09-15 武汉邦拓信息科技有限公司 A remote rendering method based on video streaming
CN110544425A (en) * 2019-09-13 2019-12-06 广州城市职业学院 ancient building VR display system
CN114450939B (en) 2019-09-19 2024-07-30 交互数字Ce专利控股公司 Apparatus and method for generating and rendering immersive video
CN111489428B (en) * 2020-04-20 2023-06-30 北京字节跳动网络技术有限公司 Image generation method, device, electronic equipment and computer readable storage medium
US11245911B1 (en) * 2020-05-12 2022-02-08 Whirlwind 3D, LLC Video encoder/decoder (codec) for real-time applications and size/b and width reduction
CN111754614B (en) * 2020-06-30 2024-07-02 平安国际智慧城市科技股份有限公司 VR-based video rendering method and device, electronic equipment and storage medium
CN112468806B (en) * 2020-11-12 2022-07-26 中山大学 Panoramic video transmission optimization method for cloud VR platform
CN114286142B (en) * 2021-01-18 2023-03-28 海信视像科技股份有限公司 Virtual reality equipment and VR scene screen capturing method
CN113347402A (en) * 2021-06-28 2021-09-03 筑友建筑装饰装修工程有限公司 Improved method, device and storage medium for rendering immersive content based on Unity
WO2023021617A1 (en) * 2021-08-18 2023-02-23 日本電信電話株式会社 Web meeting video quality estimating device, web meeting video quality estimating method, and program
US11812146B2 (en) * 2021-12-02 2023-11-07 Fotonation Limited Method and system for camera motion blur reduction
CN114466220B (en) * 2022-01-29 2025-01-03 维沃移动通信有限公司 Video downloading method and electronic device
CN114979799B (en) * 2022-05-20 2024-07-26 北京字节跳动网络技术有限公司 Panoramic video processing method, device, equipment and storage medium
CN115002519A (en) * 2022-05-31 2022-09-02 北京势也网络技术有限公司 A method of playing 8K panoramic video files under low bandwidth network
CN115396731A (en) * 2022-08-10 2022-11-25 北京势也网络技术有限公司 Panoramic video playback method, device, electronic equipment and readable storage medium
CN115567723A (en) * 2022-09-30 2023-01-03 遥在(山东)数字科技有限公司 Video transmission display method based on multilayer image buffering and partition image coding
CN116880723B (en) * 2023-09-08 2023-11-17 江西格如灵科技股份有限公司 3D scene display method and system
CN119479531B (en) * 2025-01-15 2025-03-21 长春大学 A display control method and system for HMD device

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170302719A1 (en) * 2016-04-18 2017-10-19 Qualcomm Incorporated Methods and systems for auto-zoom based adaptive video streaming
US20180295375A1 (en) * 2017-04-05 2018-10-11 Lyrical Labs Video Compression Technology, LLC Video processing and encoding

Family Cites Families (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3196889B2 (en) * 1996-09-05 2001-08-06 株式会社アルファ Three-dimensional image processing method and computer-readable recording medium storing a program for causing a computer to execute the three-dimensional image processing method
TWI262725B (en) * 2005-06-30 2006-09-21 Cheertek Inc Video decoding apparatus and digital audio and video display system capable of controlling presentation of subtitles and method thereof
US8897370B1 (en) * 2009-11-30 2014-11-25 Google Inc. Bitrate video transcoding based on video coding complexity estimation
US8862763B2 (en) * 2011-03-30 2014-10-14 Verizon Patent And Licensing Inc. Downloading video using excess bandwidth
US8907968B2 (en) * 2011-03-31 2014-12-09 Panasonic Corporation Image rendering device, image rendering method, and image rendering program for rendering stereoscopic panoramic images
US8810598B2 (en) * 2011-04-08 2014-08-19 Nant Holdings Ip, Llc Interference based augmented reality hosting platforms
GB2501929B (en) * 2012-05-11 2015-06-24 Sony Comp Entertainment Europe Apparatus and method for augmented reality
US20150208103A1 (en) * 2012-08-08 2015-07-23 National University Of Singapore System and Method for Enabling User Control of Live Video Stream(s)
US9129429B2 (en) * 2012-10-24 2015-09-08 Exelis, Inc. Augmented reality on wireless mobile devices
GB2509953B (en) * 2013-01-18 2015-05-20 Canon Kk Method of displaying a region of interest in a video stream
US9196199B2 (en) * 2013-02-12 2015-11-24 Pixtronix, Inc. Display having staggered display element arrangement
CN103702139B (en) * 2013-12-13 2017-02-01 华中科技大学 Video-on-demand system based on scalable coding under mobile environment
US9398250B2 (en) * 2014-01-06 2016-07-19 Arun Sobti & Associates, Llc System and apparatus for smart devices based conferencing
CN105025351B (en) * 2014-04-30 2018-06-29 深圳Tcl新技术有限公司 The method and device of DST PLAYER buffering
JP6337614B2 (en) * 2014-05-23 2018-06-06 セイコーエプソン株式会社 Control device, robot, and control method
US20150346812A1 (en) * 2014-05-29 2015-12-03 Nextvr Inc. Methods and apparatus for receiving content and/or playing back content
CN104240281A (en) * 2014-08-28 2014-12-24 东华大学 Virtual reality head-mounted device based on Unity3D engine
CN104268922B (en) * 2014-09-03 2017-06-06 广州博冠信息科技有限公司 A kind of image rendering method and image rendering device
US10812546B2 (en) * 2014-12-24 2020-10-20 Intel IP Corporation Link-aware streaming adaptation
CN104616243B (en) * 2015-01-20 2018-02-27 北京道和汇通科技发展有限公司 A kind of efficient GPU 3 D videos fusion method for drafting
US20160261908A1 (en) * 2015-03-05 2016-09-08 Htc Corporation Media streaming system and control method thereof
CN104717507A (en) * 2015-03-31 2015-06-17 北京奇艺世纪科技有限公司 Video transcoding method and device
CN104735464A (en) * 2015-03-31 2015-06-24 华为技术有限公司 Panorama video interactive transmission method, server and client end
US10083363B2 (en) * 2015-05-26 2018-09-25 Nbcuniversal Media, Llc System and method for customizing content for a user
WO2016201423A1 (en) * 2015-06-12 2016-12-15 Google Inc. Electronic display stabilization for head mounted display
US9956054B2 (en) * 2015-06-25 2018-05-01 EchoPixel, Inc. Dynamic minimally invasive surgical-aware assistant
WO2017060423A1 (en) * 2015-10-08 2017-04-13 Koninklijke Kpn N.V. Enhancing a region of interest in video frames of a video stream
CN106919248A (en) * 2015-12-26 2017-07-04 华为技术有限公司 It is applied to the content transmission method and equipment of virtual reality
CN105916022A (en) * 2015-12-28 2016-08-31 乐视致新电子科技（天津）有限公司 Video image processing method and apparatus based on virtual reality technology
CN105455285B (en) * 2015-12-31 2019-02-12 北京小鸟看看科技有限公司 A kind of virtual implementing helmet adaptation method
US10412130B2 (en) * 2016-04-04 2019-09-10 Hanwha Techwin Co., Ltd. Method and apparatus for playing media stream on web browser
CN105898565A (en) * 2016-04-28 2016-08-24 乐视控股（北京）有限公司 Video processing method and device
US9721393B1 (en) * 2016-04-29 2017-08-01 Immersive Enterprises, LLC Method for processing and delivering virtual reality content to a user
CN106060570B (en) * 2016-06-30 2019-06-14 北京奇艺世纪科技有限公司 A kind of full-view video image plays, coding method and device
CN106060515B (en) * 2016-07-14 2018-11-06 腾讯科技（深圳）有限公司 Panorama pushing method for media files and device
CN106231317A (en) * 2016-09-29 2016-12-14 三星电子（中国）研发中心 Video processing, coding/decoding method and device, VR terminal, audio/video player system
US10595069B2 (en) * 2016-12-05 2020-03-17 Adobe Inc. Prioritizing tile-based virtual reality video streaming using adaptive rate allocation
CN107087212B (en) * 2017-05-09 2019-10-29 杭州码全信息科技有限公司 Interactive panoramic video transcoding and playback method and system based on spatial scalable coding

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170302719A1 (en) * 2016-04-18 2017-10-19 Qualcomm Incorporated Methods and systems for auto-zoom based adaptive video streaming
US20180295375A1 (en) * 2017-04-05 2018-10-11 Lyrical Labs Video Compression Technology, LLC Video processing and encoding

Also Published As

Publication number Publication date
US20180192026A1 (en) 2018-07-05
US20180192022A1 (en) 2018-07-05
CN108377381A (en) 2018-08-07
CN108391103A (en) 2018-08-10
CN108366272A (en) 2018-08-03
CN108419093A (en) 2018-08-17
US10334238B2 (en) 2019-06-25
US20180189980A1 (en) 2018-07-05
CN108366293A (en) 2018-08-03
CN108419142A (en) 2018-08-17
US20180192044A1 (en) 2018-07-05
US20180191868A1 (en) 2018-07-05

Similar Documents

Publication Publication Date Title
US20180192063A1 (en) Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions
US11653065B2 (en) Content based stream splitting of video data
US20190273929A1 (en) De-Blocking Filtering Method and Terminal
JP7679014B2 (en) Supporting Multiview Video Motion with Disocclusion Atlases
US20180152663A1 (en) View-dependent operations during playback of panoramic video
EP3920131A1 (en) Re-projecting flat projections of pictures of panoramic video for rendering by application
US10735826B2 (en) Free dimension format and codec
JP6151355B2 (en) Panorama picture processing
CN104735464A (en) Panorama video interactive transmission method, server and client end
KR100513056B1 (en) Apparatus And Method for Adapting Graphics Contents and System therefor
EP4165877B1 (en) Representing volumetric video in saliency video streams
US10284810B1 (en) Using low-resolution frames to increase frame rate of high-resolution frames
US20220014447A1 (en) Method for enhancing quality of media
US12063389B2 (en) 3D prediction method for video coding
US20190141332A1 (en) Use of synthetic frames in video coding
WO2025152573A1 (en) Information display method based on dynamic digital human avatar, and electronic device
US20240163476A1 (en) 3d prediction method for video coding
JP2013128260A (en) Image encoder, image decoder, image encoding method and image decoding method
EP1719342B1 (en) Method of visualizing a large still picture on a small-size display.
Chen et al. Simplified carriage of MPEG immersive video in HEVC bitstream
JP4633595B2 (en) Movie generation device, movie generation method, and program
US20250133224A1 (en) Supporting multiple target display types
CN118283298A (en) Video transmission method, processing method, apparatus, device, medium, and program product
Sormain Offline H. 264 encoding method for omnidirectional videos with empirical region-of-interest
CN117641010A (en) Front-end plug-in-free streaming media player and working method thereof

Legal Events

Date Code Title Description
AS Assignment

Owner name: BLACK SAILS TECHNOLOGY INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WANG, ZHUO;TANG, YONGTAO;ZHOU, RUOXI;AND OTHERS;REEL/FRAME:044518/0180

Effective date: 20180102

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION