[go: up one dir, main page]

CN114900508A - Method for transmitting VR application data based on webrtc - Google Patents

Method for transmitting VR application data based on webrtc Download PDF

Info

Publication number
CN114900508A
CN114900508A CN202210530055.8A CN202210530055A CN114900508A CN 114900508 A CN114900508 A CN 114900508A CN 202210530055 A CN202210530055 A CN 202210530055A CN 114900508 A CN114900508 A CN 114900508A
Authority
CN
China
Prior art keywords
data
rtp
client
server
header
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210530055.8A
Other languages
Chinese (zh)
Other versions
CN114900508B (en
Inventor
罗俊华
邹琼
周双全
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Ruiyun Technology Co ltd
Original Assignee
Shenzhen Rayvision Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Rayvision Technology Co ltd filed Critical Shenzhen Rayvision Technology Co ltd
Priority to CN202210530055.8A priority Critical patent/CN114900508B/en
Publication of CN114900508A publication Critical patent/CN114900508A/en
Application granted granted Critical
Publication of CN114900508B publication Critical patent/CN114900508B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/80—Responding to QoS
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
    • H04L69/163—In-band adaptation of TCP data exchange; In-band control procedures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22—Parsing or analysis of headers
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00—Reducing energy consumption in communication networks
    • Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Multimedia (AREA)
  • Information Transfer Between Computers (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention discloses a method for transmitting VR application data based on webrtc, which pre-expands RTP head, wherein the expanded content comprises the following steps: add URI to rtp _ parameters.cc; add extension type in rtp _ rtcp _ definitions.h; defining extension data and a data structure in rtp _ header _ extensions.h; rtp _ header _ extensions.cc. The invention transmits the needed VR control information in the media packet by expanding the RTP head, the control information and the media data are achieved at the same time, and the processing logic of the client is reduced; control information of a data channel does not need to be waited, so that delay is reduced; the IP head and the UDP head added to a transmission layer are wasted without independently sending a control information block, so that the bandwidth is reduced; the image smoothness is obviously improved in the use of the project.

Description

Method for transmitting VR application data based on webrtc
Technical Field
The invention relates to the technical field of VR (virtual reality), in particular to a method for transmitting VR application data based on webrtc.
Background
In a traditional VR data transmission mode, UDP or TCP transmission is usually selected to be directly used, and some users also use webrtc to transmit video through a media channel and transmit VR control information, gesture data, keys and the like through a data channel. In the prior art, if UDP or TCP is directly used, the screen is lost or the packet is blocked, so that the user experience is influenced; and the webrtc is directly used, the media channel and the data channel are different in transmission strategy, VR control information does not arrive in time, or the subsequent processing logic is complex, so that the jamming and the time delay are increased to a certain extent, and the user experience is also influenced.
Accordingly, the prior art is deficient and needs improvement.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: the method for transmitting VR application data based on webrtc is provided, the processing difficulty of the client is reduced, the blocking and time delay of the client are reduced, and the user experience is improved.
The technical scheme of the invention is as follows: a method for transmitting VR application data based on webrtc is provided, which comprises the following steps.
S1: an extended RTP header; the expanded content includes: URI is added to rtp _ parameters.cc; adding extension types into rtp _ rtcp _ definitions.h; defining extension data and a data structure in rtp _ header _ extensions.h; rtp _ header _ extensions.cc.
S2: the VR client is connected to the signaling server via TCP.
S3: the VR server is connected to the signaling server via TCP.
S4: the VR client calls the createOffer of webrtc to create an Offer SDP; SDP, Session Description Protocol, is a control Protocol for describing multimedia sessions.
S5: SendOffer 1: and the VR client transmits the offer SDP to the signaling server through the TCP.
S6: SendOffer 2: the signaling server forwards the offer SDP to the VR media server.
S7: createnanswer: the VR media server receives offer and calls webrtc's createnaswer method to create answer SDP.
S8: SendAnswer 1: the VR media server transmits the answer SDP to the signaling server through TCP.
S9: SendAnswer 2: the signaling server forwards the answer SDP to the VR client.
S10: P2P media communication establishment procedure.
S11: and the VR client sends the attitude index and the attitude data of the hardware provided with the VR client to the VR server.
S12: the VR server sends VR video data to a VR client, wherein an RTP extension header contains newly defined extension data: the position index and the frame send time.
The scheme is based on webrtc transmission, can fully utilize the existing transmission technology of webrtc, such as packet loss retransmission, FEC, transmission bandwidth evaluation and the like, and is superior to the direct use of UDP or TCP in the aspect of real-time transmission; meanwhile, by expanding the RTP header, information in VR control is added into the RTP expansion header and transmitted together with the media data in the media channel, so that the problems of jamming and time delay caused by inconsistent transmission strategies of the media channel and the data channel are solved, and the user experience is improved.
Further, the pose data includes: location information and direction information of hardware.
Further, the URI is: the ratio of rayvision to rayxr to vr-control-info.
Further, the extension type is: RTPExtensionType kRtpextensionVRConrtolInfo.
Further, the data structure includes: ID; len: the length of the data; poseId: attitude index, client generation; frameSentTime: frame send time, server generation.
Further, the hardware is a head-mounted VR display device.
Further, the VR server receives the posture data of the hardware, renders the image according to the posture data, and then returns the rendered image and the rendered posture data to the VR client.
By adopting the scheme, the invention provides a method for transmitting VR application data based on webrtc, the needed VR control information is transmitted in the media packet by expanding the RTP head, the control information and the media data are simultaneously achieved, and the processing logic of the client is reduced; control information of a data channel does not need to be waited, so that delay is reduced; the additional IP header and UDP header of a transmission layer are wasted without independently sending a control information block, so that the bandwidth is reduced; the image smoothness is obviously improved in the use of the project.
Drawings
FIG. 1 is a flow chart illustrating steps among a schematic diagram of a VR client, a signaling server, and a VR server in accordance with the present invention;
fig. 2 is a schematic diagram of a data structure.
Detailed Description
The invention is described in detail below with reference to the figures and the specific embodiments.
Referring to fig. 1 and 2, the present invention provides a method for transmitting VR application data based on webrtc, which includes the following steps.
S1: an extended RTP header; the expanded content includes: URI is added to rtp _ parameters.cc; add extension type in rtp _ rtcp _ definitions.h; defining extension data and a data structure in rtp _ header _ extensions.h; and (4) packing and unpacking data in rtp _ header _ extensions. The URI is: the ratio of rayvision to rayxr to vr-control-info. The extension types are: RTPExtensionType kRtpextensionVRConrtolInfo. The data structure includes: ID; len: data length (len); poseId: attitude index, client generation; frameSentTime: frame send time, server generation.
S2: the VR client is connected to the signaling server via TCP.
S3: the VR server is connected to the signaling server via TCP.
S4: the VR client calls the createOffer of webrtc to create an Offer SDP; SDP, Session Description Protocol, is a control Protocol for describing multimedia sessions.
S5: SendOffer 1: and the VR client transmits the offer SDP to the signaling server through the TCP.
S6: SendOffer 2: the signaling server forwards the offer SDP to the VR media server.
S7: createnanswer: the VR media server receives offer and calls webrtc's createnaswer method to create answer SDP.
S8: SendAnswer 1: the VR media server transmits the answer SDP to the signaling server through TCP.
S9: SendAnswer 2: the signaling server forwards the answer SDP to the VR client.
S10: P2P media communication establishment procedure.
S11: and the VR client sends the attitude index and the attitude data of the hardware provided with the VR client to the VR server. The hardware is a head-mounted VR display device. The pose data includes: location information and direction information of hardware.
S12: the VR server sends VR video data to a VR client, wherein an RTP extension header comprises newly defined extension data: the position index and the frame send time. And the VR server receives the posture data of the hardware, renders the image according to the posture data, and then returns the rendered posture data to the VR client. The VR server renders the VR image according to the posture data of the hardware, the rendered VR image and the posture data are returned to the VR client side, the hardware and the VR client side do not need to perform excessive calculation processing, and the VR image can be directly played according to the posture data, so that time delay can be effectively achieved, and smoothness is improved; meanwhile, the requirements on hardware can be effectively reduced, and the vehicle cost is saved.
The scheme is based on webrtc transmission, can fully utilize the existing transmission technology of webrtc, such as packet loss retransmission, FEC, transmission bandwidth evaluation and the like, and is superior to the direct use of UDP or TCP in the aspect of real-time transmission; meanwhile, by expanding the RTP header, information in VR control is added into the RTP expansion header and transmitted together with the media data in the media channel, so that the problems of jamming and time delay caused by inconsistent transmission strategies of the media channel and the data channel are solved, and the user experience is improved.
In summary, the present invention provides a method for transmitting VR application data based on webrtc, where the RTP header is extended to transmit the required VR control information in the media packet, and the control information and the media data are simultaneously achieved, so as to reduce the processing logic of the client; control information of a data channel does not need to be waited, so that delay is reduced; the additional IP header and UDP header of a transmission layer are wasted without independently sending a control information block, so that the bandwidth is reduced; the image smoothness is obviously improved in the use of the project.
The present invention is not limited to the above preferred embodiments, and any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (7)

1. A method for transmitting VR application data based on webrtc is characterized by comprising the following steps:
s1: an extended RTP header; the expanded content includes: URI is added to rtp _ parameters.cc; add extension type in rtp _ rtcp _ definitions.h; defining extension data and a data structure in rtp _ header _ extensions.h; packing and unpacking data in rtp _ header _ extensions.cc;
s2: the VR client is connected to the signaling server through the TCP;
s3: the VR server is connected to the signaling server through TCP;
s4: the VR client calls the createOffer of webrtc to create an Offer SDP; SDP, Session Description Protocol, is a control Protocol for describing multimedia sessions;
s5: SendOffer 1: the VR client transmits the offer SDP to the signaling server through TCP;
s6: SendOffer 2: the signaling server forwards the offer SDP to the VR media server;
s7: createnanswer: the VR media server receives the offer, and calls a createAnswer method of webrtc to create an answer SDP;
s8: SendAnswer 1: the VR media server transmits the answer SDP to the signaling server through the TCP;
s9: SendAnswer 2: the signaling server forwards an answer SDP to the VR client;
s10: P2P media communication establishment procedure;
s11: the VR client sends the attitude index and the attitude data of the hardware provided with the VR client to the VR server;
s12: the VR server sends VR video data to a VR client, wherein an RTP extension header contains newly defined extension data: the position index and the frame send time.
2. The method of claim 1, wherein the gesture data comprises: location information and direction information of hardware.
3. The method of claim 1, wherein the URI is: the ratio of rayvision to rayxr to vr-control-info.
4. The method of claim 1, wherein the extension type is: RTPExtensionType kRtpextensionVRConrtolInfo.
5. The method of claim 1, wherein the data structure comprises: ID; len: the length of the data; poseId: attitude index, client generation; frameSentTime: frame send time, server generation.
6. The method of claim 1, wherein the hardware is a head-mounted VR display device.
7. The method of claim 1, wherein the VR server receives pose data of hardware, renders an image according to the pose data, and returns the rendered pose data to the VR client.
CN202210530055.8A 2022-05-16 2022-05-16 Method for transmitting VR application data based on webrtc Active CN114900508B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210530055.8A CN114900508B (en) 2022-05-16 2022-05-16 Method for transmitting VR application data based on webrtc

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210530055.8A CN114900508B (en) 2022-05-16 2022-05-16 Method for transmitting VR application data based on webrtc

Publications (2)

Publication Number Publication Date
CN114900508A true CN114900508A (en) 2022-08-12
CN114900508B CN114900508B (en) 2023-08-29

Family

ID=82724802

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210530055.8A Active CN114900508B (en) 2022-05-16 2022-05-16 Method for transmitting VR application data based on webrtc

Country Status (1)

Country Link
CN (1) CN114900508B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116192821A (en) * 2022-12-29 2023-05-30 深圳市瑞云科技股份有限公司 A method for transmitting video stream control information based on webrtc
WO2024075092A1 (en) * 2022-10-31 2024-04-11 Lenovo (Singapore) Pte. Ltd. User interaction data transportation using real-time transport protocol header extension

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101008753B1 (en) * 2010-01-14 2011-01-14 주식회사 씨앤드디큐브 Multimedia data streaming system
GB201709619D0 (en) * 2017-06-16 2017-08-02 Canon Kk Methods, devices, and computer programs for improving streaming of portions of media data
WO2018041780A1 (en) * 2016-08-29 2018-03-08 Koninklijke Kpn N.V. Communicating in a virtual reality environment
US20180183849A1 (en) * 2016-12-22 2018-06-28 Hanwha Techwin Co., Ltd. Method and device for media streaming between server and client using rtp/rtsp standard protocol
WO2018126702A1 (en) * 2017-01-06 2018-07-12 华为技术有限公司 Streaming media transmission method applied to virtual reality technology and client
WO2019157803A1 (en) * 2018-02-13 2019-08-22 华为技术有限公司 Transmission control method
CN111479121A (en) * 2020-04-08 2020-07-31 北京智能工场科技有限公司 Live broadcasting method and system based on streaming media server
CN113315823A (en) * 2021-05-21 2021-08-27 广州赞赏信息科技有限公司 Low-delay audio and video transmission method
CN114036044A (en) * 2021-10-28 2022-02-11 深圳市瑞云科技有限公司 Openvr development drive-based debugging method

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101008753B1 (en) * 2010-01-14 2011-01-14 주식회사 씨앤드디큐브 Multimedia data streaming system
WO2018041780A1 (en) * 2016-08-29 2018-03-08 Koninklijke Kpn N.V. Communicating in a virtual reality environment
US20180183849A1 (en) * 2016-12-22 2018-06-28 Hanwha Techwin Co., Ltd. Method and device for media streaming between server and client using rtp/rtsp standard protocol
WO2018126702A1 (en) * 2017-01-06 2018-07-12 华为技术有限公司 Streaming media transmission method applied to virtual reality technology and client
GB201709619D0 (en) * 2017-06-16 2017-08-02 Canon Kk Methods, devices, and computer programs for improving streaming of portions of media data
WO2019157803A1 (en) * 2018-02-13 2019-08-22 华为技术有限公司 Transmission control method
CN111479121A (en) * 2020-04-08 2020-07-31 北京智能工场科技有限公司 Live broadcasting method and system based on streaming media server
CN113315823A (en) * 2021-05-21 2021-08-27 广州赞赏信息科技有限公司 Low-delay audio and video transmission method
CN114036044A (en) * 2021-10-28 2022-02-11 深圳市瑞云科技有限公司 Openvr development drive-based debugging method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
JEFF: "WebRTC研究：RTP报头扩展", Retrieved from the Internet <URL:https://blog.jianchihu.net/webrtc-research-rtp-header-extension.html> *
JINBAO GAO;YUANZHANG LI;HONGWEI JIANG;LU LIU;XIAOSONG ZHANG: "An RTP extension for reliable user-data transmission over voip traffic", 《SOCIALSEC 2019: SECURITY AND PRIVACY IN SOCIAL NETWORKS AND BIG DATA》, pages 74 *
耿超: "面向Web AR的微服务化可定制业务系统的设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》, pages 138 - 1242 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024075092A1 (en) * 2022-10-31 2024-04-11 Lenovo (Singapore) Pte. Ltd. User interaction data transportation using real-time transport protocol header extension
GB2637647A (en) * 2022-10-31 2025-07-30 Lenovo Singapore Pte Ltd User interaction data transportation using real-time transport protocol header extension
CN116192821A (en) * 2022-12-29 2023-05-30 深圳市瑞云科技股份有限公司 A method for transmitting video stream control information based on webrtc

Also Published As

Publication number Publication date
CN114900508B (en) 2023-08-29

Similar Documents

Publication Publication Date Title
US9742820B2 (en) Latency differential mitigation for real time data streams
US7447775B1 (en) Methods and apparatus for supporting transmission of streaming data
JP5545295B2 (en) How to establish a thin client session
CN103188300B (en) The methods, devices and systems of VOIP phone are realized in cloud computing environment
CN114900508B (en) Method for transmitting VR application data based on webrtc
JP2009527155A (en) Bi-directional media session establishment system, method and apparatus based on IP multimedia subsystem
CN101656747A (en) Method and system for transmitting streaming media data
US7543072B1 (en) Method and system capable of performing a data stream over multiple TCP connections or concurrent interleave of multiple data streams over multiple TCP connections
WO2023071656A1 (en) Information transmission method and apparatus
CN101146100A (en) A Realization Method of SIP VoIP Based on Transmission Protocol SCTP and DCCP
CN114793225B (en) Data communication method and system for cloud mobile phone and true mobile phone
US20070115945A1 (en) Communication protocol and method for operating a volp communication with additional data transmission
US8831018B2 (en) Media conversion device for interconnecting communication terminal devices with media converted and a method therefor
EP2211507A1 (en) Method, device and system for processing real time streaming protocol
CN103684970B (en) The transmission method of media data flow and thin terminal
CN114710568B (en) Audio and video data communication methods, equipment and storage media
CN109361671A (en) A streaming media transmission architecture method based on SIP protocol
US7770211B2 (en) Unauthorized access prevention method, unauthorized access prevention apparatus and unauthorized access prevention program
US8582559B2 (en) System and method for handling media streams
WO2004081798A1 (en) Transmitter apparatus and transmitting method
CN114125508A (en) Reliability Guarantee Method of Video Multicast in Wireless Domain
KR100624854B1 (en) Media retransmission device and method
CN114710821A (en) Method, system and storage medium for VoLTE relay access to SIP contact center
CN100499586C (en) Conversion of crossing medium-flow over network address
JP2002152256A (en) Address translation apparatus and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CP01 Change in the name or title of a patent holder

Address after: 518000 17th floor, block B, Sunshine Technology Innovation Center, No.2 Shanghua Road, Nanshan street, Nanshan District, Shenzhen City, Guangdong Province

Patentee after: Shenzhen Ruiyun Technology Co.,Ltd.

Address before: 518000 17th floor, block B, Sunshine Technology Innovation Center, No.2 Shanghua Road, Nanshan street, Nanshan District, Shenzhen City, Guangdong Province

Patentee before: SHENZHEN RAYVISION TECHNOLOGY CO.,LTD.

CP01 Change in the name or title of a patent holder