Detailed Description
In the related art, the real-time communication based on WebRTC has the problems of a large number of times of connection between the terminal device and the server, long single connection time, and poor user experience. Taking a network transmission scheme of a cloud game as an example, P2P (peer-to-peer connection) establishment is generally required based on WebRTC to transmit network transmission game pictures, and DATA CHANNEL (data channel) establishment is performed to transmit control instructions; it can be seen that the network transmission scheme in the related art needs to correspondingly use different communication connections in different interaction processes, which has the problems of multiple interaction between the terminal device and the server and longer total connection time caused by longer single connection establishment time and more connection establishment times.
Cloud gaming is a game based on cloud computing. In the running mode of the cloud game, a running main body of the game program and a game picture presentation main body are separated; for example, the running subject of the game program is typically a server, and the game screen presentation subject is typically a terminal device. The running main body of the game program is used for rendering the game picture, and then the data related to the game picture after the rendering is finished is sent to the game picture presentation main body through a network; in addition to the data related to the game picture transmitted through the network, the control command related to the game is often transmitted. In a cloud game scene, a corresponding network communication connection needs to be established before instruction transmission is carried out, and a corresponding network communication connection needs to be established before data stream transmission is carried out, particularly in a scene that network is recovered after network interruption when the cloud game is started and when the cloud game is operated, the cloud game is started and operated under great influence caused by a network transmission scheme based on WebRTC in the related technology, and the problems of poor cloud game user experience, low satisfaction degree of a user to the cloud game and the like are caused. In view of this, the present disclosure provides a data transmission method and apparatus, an electronic device, and a storage medium, and embodiments of the present disclosure will be described in detail below.
As shown in fig. 1, one or more embodiments of the present disclosure provide a data transmission method, which is applied to a server, and includes, but is not limited to, steps S101 to S103.
Step S101, establishing network communication connection with the terminal equipment according to the history connection establishment information, wherein the history connection establishment information is the history information of the connection between the server and the terminal equipment.
The history linking information specifically related to this embodiment may include, but is not limited to, a terminal device ID (Identity document, identity number), a terminal device IP (Internet Protocol ) address, and terminal device configuration information, for example.
The terminal devices involved in the present embodiment may include, but are not limited to, one or more of a smart phone, a tablet computer, a PC (Personal Computer ), a television, a wearable device, and the like.
In one or more embodiments of the present disclosure, establishing a network communication connection with a terminal device according to historical connection establishment information includes: the server can inquire the historical connection establishment information by using the connection establishment parameters sent by the terminal equipment, and if the historical connection establishment information matched with the connection establishment parameters is inquired, the connection establishment between the terminal equipment and the server is determined, the connection is successful, and the connection establishment parameters do not need to be replied; if, however, no history connection establishment information matching the connection establishment parameters is queried, it is determined that the terminal device has not been connected with the server, and the new connection establishment parameters including handshake information are restored.
The network communication connection between the terminal device and the server in this embodiment is specifically Web Transport, so that the time required for establishing connection between the application program starting process on the terminal device and the server is reduced.
As shown in connection with fig. 4, the specific process of establishing the network communication connection between the terminal device and the server in this embodiment may include, but is not limited to, the following steps.
S401, the terminal equipment inquires history information connected with the server.
S402, the terminal equipment sends QUIC (Quick UDP Internet Connection, fast UDP network connection) Transport (transmission) connection establishment parameters to the server according to the detected historical information of the connection with the server. The connection establishment parameters include, for example, config_id, transport_ PARAMETERS, IDLE _timeout, max_stream_number, etc., where config_id (configuration identifier) represents public key information used by handshake between the terminal device and the server, and transport_parameters (transmission parameters) represent transmission parameters of buffered historical connection, such as information of initial window size, etc.; idle_timeout represents idle timeout, and after idle time of connection without data interaction exceeds idle_timeout, connection is automatically disconnected; max_stream_number represents the maximum number of multiplexed data streams beyond which no successful creation can be made in one connection.
S403, the server inquires the history linking information of the terminal equipment. If historical connection establishment information matched with the connection establishment parameters is queried, determining that the terminal equipment is connected with the server, and indicating that the connection is successful; if the historical connection establishment information matched with the connection establishment parameters is not queried, determining that the terminal equipment is not connected with the server.
S404, the server performs QUIC Transport connection establishment reply, which may include: if no history connection establishment information exists, the new connection establishment parameters are restored; if there is history connection establishment information, the connection is successful.
S411, the terminal equipment sends a QUIC Transport connection establishment request to the server.
S412, the QUIC Transport reply connection is successful.
Step S102, receiving a control instruction from the terminal device based on the network communication connection, wherein the control instruction is used for controlling the process of a first application program on the server, and the first application program is a version of the application program on the server.
The process related to the disclosure specifically refers to a primary operation activity of a program in a terminal device on a certain data set, and is a basic unit of resource allocation performed by an operating system on the terminal device.
In this embodiment, the content transmitted by the network includes a control instruction, which may be a control instruction for live broadcast, video conference, or cloud game, for example, a control instruction in this embodiment may include, but is not limited to, a game start (GAME START) instruction, a game operation instruction, and the like.
Taking game operation instructions as an example, the manner in which the control instructions are triggered may include, but is not limited to, screen clicking, rotation, sharpness selection, etc. operations by the user.
Taking a game start instruction as an example, the time consumption of the cloud game first frame image loading process can be greatly shortened, so that the starting speed of the cloud game can be greatly improved.
As shown in connection with fig. 4, step S102 in embodiments of the present disclosure may include, but is not limited to, steps S421 to S424.
S421, the terminal device creates a bidirectional instruction stream based on the operation of the user, and sends an instruction to the server, wherein the operation of the user can be, for example, screen clicking. The process of creating a bi-directional instruction stream includes: bidirection Stream (bi-directional reliable flow) objects are created on the basis of the established network communication connection.
Wherein Bidirection Stream (bi-directional reliable flow) refers to a data flow that can be transmitted and received by both the terminal device and the server.
In an alternative embodiment of the present disclosure, step S421 can be performed simultaneously with step S411, which helps to further shorten the instruction interaction time.
S422, the server creates a bidirectional instruction stream object, and the instruction result returns the user operations such as screen clicking and the like.
S423, the server replies the instruction result. When an instruction is required to be transmitted, the terminal device and the server transmit and receive data through Bidirection Stream (bidirectional reliable stream).
S424, a subsequent instruction interaction process may then take place. The method and the device can be used for live broadcasting, video conference or cloud game and other scenes, taking a starting instruction as an example in the cloud game scene, and starting to start the local game after the server receives the starting instruction; and executing a game progress collection process of the audio and video information. Taking a Game Stop instruction as an example, the server closes the audio and video push stream and closes the Game process after receiving the end instruction sent by the terminal equipment.
Step S103, pushing a data stream corresponding to the control instruction to the terminal device based on the network communication connection, where the data stream is used for loading and/or running a second application program on the terminal device, and the second application program is a version of the application program on the terminal device.
In order to better realize the functions of the application programs, the embodiment of the disclosure runs a first application program on a server and a second application program on a terminal device.
The data stream in this embodiment specifically includes an audio/video data stream, and the content transmitted by the network includes the audio/video data stream.
The terminal equipment is provided with a separate first encoding and decoding module, and the first encoding and decoding module is used for decoding the received data stream so as to decode the audio and video information. Wherein the first codec module may include an encoder and a decoder.
The terminal device may be provided with a second application program, and the application program according to this embodiment may be, for example, a cloud game program, a video conference program, or a live broadcast program.
The method and the device can be used for transmitting the control instruction and the audio/video data stream based on one network communication connection, multiplexing channels created by the same network communication connection, and the problem that the network communication connection is established for transmitting different contents for many times in the related technology is avoided.
As shown in fig. 2, in one or more alternative embodiments of the present disclosure, a data stream corresponding to a control instruction is pushed to a terminal device based on a network communication connection, including but not limited to steps S201 to S203.
Step S201, determining audio and video information, where the audio and video information is audio and video information in a process of the first application program running according to the control instruction, where the audio and video information includes audio and video.
In this embodiment, the audio and video information is determined by collecting the audio and video information; the audio and video information is the audio and video information in the process of the first application program running according to the control instruction.
The audio and video information in this embodiment is specifically audio and video information of a first application program running on a server, and the collection mode of the audio and video information is real-time collection.
Step S202, based on the audio and video information, a transport stream is created, and the transport stream is included in the data stream.
In the embodiment, a multi-frame transmission stream is created by utilizing audio and video information; the data stream is specifically a multi-frame unidirectional transmission stream.
The unidirectional transport stream may include, for example, a video transport stream and an audio transport stream, and/or an audio-video mixed stream.
Step S203, pushing the transport stream to the terminal device based on the network communication connection.
In this embodiment, the server pushes the multi-frame unidirectional transport stream to the terminal device, based on the network communication connection in particular.
The above-described embodiments of the present disclosure create a unidirectional transport stream for transmission based on collected audio and video, the unidirectional transport stream being more suitable for network transmission, which helps to improve the efficiency of audio and video information transmission.
In one or more alternative embodiments of the present disclosure, as shown in connection with fig. 4, creating a transport stream based on audio-visual information includes: encoding the audio-video information into a first transport stream and a second transport stream; the first transport stream is for being transmitted to the terminal device based on a reliable transmission, and the second transport stream is for being transmitted to the terminal device based on an unreliable transmission, the transport streams including the first transport stream and the second transport stream.
The first transport stream includes a plurality of data packets for transmission, and the second transport stream includes a plurality of data packets for transmission.
In this embodiment, the server side is configured with an independent second codec module, where the second codec module is used to encode the collected audio and video information.
Specifically, the second codec module of the present embodiment can be used to encode the audio and video information into a first transport stream and a second transport stream, where the encoding process includes a data compression process of the audio and video screen information, and the data formats of the first transport stream and the second transport stream may be, for example, AVI, H265, and the like, and the second codec module includes an encoder and a decoder.
Based on the transmission scheme of Web Transport, the embodiment can effectively avoid the coupling problem between the transmission protocol and the coding/decoding protocol of the WebRTC, thereby being capable of well enabling the optimization of the transmission layer and the optimization of audio/video coding/decoding encapsulation to be independently carried out.
The transmission strategy combining reliable transmission and unreliable transmission is synthesized, the first transmission stream can be reliably transmitted from the server to the terminal equipment, the second transmission stream is transmitted from the server to the terminal equipment in a best effort manner, the smoothness of the video is preferentially ensured, and the data packets with low importance degree are discarded, and the mode has the following maximum advantages: even if the network environment is unstable, for example, the network quality suddenly deteriorates, the present embodiment can preferentially ensure that the first transport stream is transmitted from the server to the terminal device, thereby maximally ensuring the normal presentation of the game screen of the terminal device.
As shown in connection with fig. 4, in one or more alternative embodiments of the present disclosure, the first transport stream is an I-frame (Intra coded frames, intra-coded frame) unidirectional transport stream, the I-frame transport stream being for transmission to the terminal device based on a first transport protocol; the second transport stream is a P-frame (PREDICTIVE CODED FRAMES, forward predictive coded frame) unidirectional transport stream, the P-frame transport stream being for transmission to the terminal device based on a second transport protocol. The priority of the first transmission protocol is higher than the priority of the second transmission protocol, which indicates that the priority of the unidirectional transmission stream transmission of the I frame is higher than the priority of the unidirectional transmission stream transmission of the P frame. The audio and video information is encoded in a manner of setting a plurality of P frames after one I frame, for example, an encoding output sequence similar to IPPPIPPP.
For the server, the I-frame unidirectional transport stream and the P-frame unidirectional transport stream in this embodiment are unidirectional transport streams from the server to the terminal device.
More specifically, the present embodiment may perform transmission of data streams, control instructions, and the like through a qic (Quick UDP Internet Connection, fast UDP network connection) Transport, where UDP refers to User Datagram Protocol, i.e., user datagram protocol.
The present embodiment transmits the I-frame output by the second codec module through QUIC Unidirection Stream (unidirectional reliable stream) of the highest priority, thereby ensuring that the I-frame, which is the key frame, is transmitted in the user limited network with the highest priority and reliability. For the data stream (P frame) with the non-highest priority, the embodiment adopts a best effort mode to perform low-priority transmission (i.e. low-priority transmission) or discard in the user limited network, and the embodiment transmits the P frame output by the second codec module through QUIC Datagrams (unreliable data message) with the secondary priority.
The embodiment can still meet the playing requirement of the terminal equipment under the special condition that the terminal equipment only receives the I frame, so as to improve the experience of the application program under the condition of poor network environment, such as the cloud game, video conference or live scene experience.
For the terminal device, the received P frames are jitterbuffer (jitter buffer) processed and sequenced, and then sent to the decoder in the first codec module, and the decoder decodes with reference to the I frame after receiving the P frames, and then performs rendering and displaying.
In addition, the smoothness of the cloud game picture of the terminal equipment can be ensured based on the mode of preferentially transmitting the I frame, so that the problem that the cloud game picture is blocked easily in the related technology is avoided.
As shown in fig. 4, step S103 in the present embodiment may include, but is not limited to, steps S431 to S472.
S431, the server creates and sends the I-frame unidirectional transport stream.
S432, the terminal equipment creates a data channel of the I frame.
The server writes the video I frame data output by the encoder into the created unidirectional stream, and the web transport protocol stack is responsible for sending; the terminal device passively creates a unidirectional stream data channel after receiving the data received by the stream.
S441, the server creates a P-frame unidirectional transport stream and transmits it.
S442, the terminal equipment creates a data channel of the P frame.
S451, the server collects audio and video information and encodes the collected audio and video information.
S461, the server encodes the audio and video information into I frames and then sends the I frames to the terminal equipment.
S462, after receiving the I frame, the terminal device sends the I frame to the decoder.
S471, the server encodes the audio and video information into P frames and then sends the P frames to the terminal equipment.
S472, after receiving the P frame, the terminal equipment sends the P frame to jitterbuffer (jitter buffer), and sends the P frame to a decoder according to sep sequence (bucket sequence).
As shown in fig. 3, in one or more alternative embodiments of the present disclosure, the data transmission method further includes, but is not limited to, steps S301 to S302.
In step S301, the control instruction is converted into a native instruction, which is an instruction executed in the process of the first application program.
The native instruction in this embodiment refers to an instruction that can be directly executed in the process of the first application program without parsing.
In step S302, the process of the first application program is controlled by using the local instruction.
In particular, the embodiment may send the local instruction to a process of the first application running on the server, where the local instruction is executed in the process of the first application.
Based on the mode of converting the control instruction into the local instruction of the server, the method and the device can also directly send the local instruction into the process of the first application program, so that the effectiveness and reliability of the execution of the control instruction are obviously improved.
In one or more optional embodiments of the disclosure, the data transmission method further includes: and sending a specified instruction to the terminal equipment, wherein the specified instruction is used for controlling the state of the second application program.
The server sending a specified instruction to the terminal device, which may include, for example, the server letting the terminal device change decoding parameters, the server informing the terminal device information in real time, such as multi-person control transfer, full-field mute voice; the server transmits the positioning information of the cloud game instance to the terminal device or the like.
In an alternative embodiment of the present disclosure, receiving a control instruction from a terminal device based on a network communication connection includes: receiving a control instruction from the terminal device based on a third transmission protocol; the data transmission method further comprises the following steps: receiving a file packet from the terminal equipment based on a fourth transmission protocol, wherein a plurality of file packets form a file uploaded by the terminal equipment; wherein the priority of the second transmission protocol is higher than the priority of the third transmission protocol, and the priority of the third transmission protocol is higher than the priority of the fourth transmission protocol.
Preferably, the first transport protocol is a unidirectional reliable streaming protocol; and/or the second transmission protocol is an unreliable data message transmission protocol; and/or the third transport protocol is a bi-directional reliable streaming protocol; and/or, the fourth transmission protocol is a network transmission sending stream transmission protocol. The unidirectional reliable streaming protocol of the present embodiment is QUIC Unidirection Stream, the unreliable data packet transmission protocol is QUIC Datagrams, the bidirectional reliable streaming protocol is QUIC Bidirection Stream, and the network transmission and transmission streaming protocol is WebTransportSendStream.
Based on the above-mentioned multiple different transmission protocols to transmit multiple different data, the present disclosure can reliably ensure that the I-frame transmission stream, the P-frame transmission stream, the control command and the file are sequentially transmitted in order from high to low in priority, and further optimize loading and/or running of the application program on the terminal device.
Specifically, the process of uploading the file by the terminal device may include: s481, the terminal device creates a file transport stream as required. S482, the terminal equipment uploads the file.
Taking a cloud game scene as an example, terminal equipment sends an archive file of a game, a login-free file of a game account number and the like to a server side; the server sends game highlights, game archive files and the like to the terminal device.
The present embodiment uploads the file from the terminal device to the server through WebTransportSendStream (network transport send stream) of the lowest priority.
The file package can be transmitted through the additionally newly added transmission channel, so that the function of transmitting the file to the server by the terminal equipment is realized; of course, on the basis of the present disclosure, a function of pushing content to the terminal device by the server can also be realized.
The method and the device can also realize the file transmission function in the cloud game scene, and enrich the function application in the cloud game scene.
In combination with the above embodiments, the embodiments of the present disclosure may specifically transmit the control instruction and the specified instruction through QUIC Bidirection Stream (bidirectional reliable stream), so as to implement the interaction function of the instruction between the terminal device and the server.
In some embodiments, for example, in an extreme network situation where the network environment of the terminal device is very poor, the transmission process of the instruction may be delayed to ensure that the key frames of the video are played at the terminal device.
The embodiment of the disclosure divides the priorities of the audio and video data streams, the control instructions and the files to meet the continuous and stable running of the cloud game pictures of the terminal equipment based on the data streams with higher priorities, and realizes the interactive function of the control instructions based on the secondary priorities, and can also meet the file transmission function of users in the cloud game scene based on the functions.
As shown in fig. 5, one or more embodiments of the present disclosure can also provide a data transmission method applied to a terminal device, based on the same inventive concept as the data transmission method applied to a server of the present disclosure.
The data transmission method applied to the terminal device may include, but is not limited to, steps S501 to S503.
In step S501, a network communication connection is established with the server, where the network communication connection is established according to the history connection establishment information in the server, and the history connection establishment information is the history information of the connection between the server and the terminal device.
In step S502, a control instruction is sent to the server based on the network communication connection, where the control instruction is used to control a process of a first application program on the server, and the first application program is a version of the application program on the server.
In an alternative embodiment of the present disclosure, the first application may be a first cloud game, and the control instruction is used to control a process of the first cloud game on the server.
Step S503, receiving, from the server, a data stream corresponding to the control instruction based on the network communication connection, the data stream being used for loading and/or running of a second application on the terminal device, the second application being a version of the application on the terminal device.
In an alternative embodiment of the disclosure, the second application may be a second cloud game, and the data stream may be used for loading and/or running the second cloud game on the terminal device.
Optionally, the server is configured to determine audio and video information, where the audio and video information is audio and video information in a process of the first application program running according to the control instruction, and the audio and video information includes audio and video; the server is used for creating a transport stream based on the audio and video information, wherein the transport stream comprises the data stream; the server is used for pushing the transmission stream to the terminal equipment based on the network communication connection. The server can be used for determining the audio and video information in a mode of collecting the audio and video information.
Optionally, the server is further operable to encode the audio-visual information into a first transport stream and a second transport stream; the first transport stream is for being transmitted to the terminal device based on a reliable transmission, and the second transport stream is for being transmitted to the terminal device based on an unreliable transmission, the transport streams including the first transport stream and the second transport stream.
Optionally, the first transport stream is an I-frame transport stream, the I-frame transport stream being for being transmitted to the terminal device based on the first transport protocol; the second transport stream is a P-frame transport stream, which is used for being transmitted to the terminal device based on a second transport protocol; wherein the priority of the first transport protocol is higher than the priority of the second transport protocol.
Optionally, the server is configured to convert the control instruction into a local instruction, where the local instruction is an instruction executed in a process of the first application program; the server is used for controlling the progress of the first application program by utilizing the local instruction.
Optionally, the data transmission method further includes: a designation instruction is received from the server, the designation instruction for controlling a state of the second application.
Optionally, the server is configured to receive a control instruction from the terminal device based on a third transmission protocol; the data transmission method further comprises the following steps: and sending file packages to the server based on a fourth transmission protocol, wherein a plurality of file packages form files uploaded by the terminal equipment.
Wherein the priority of the second transmission protocol is higher than the priority of the third transmission protocol, and the priority of the third transmission protocol is higher than the priority of the fourth transmission protocol.
Preferably, the first transport protocol is a unidirectional reliable streaming protocol; and/or the second transmission protocol is an unreliable data message transmission protocol; and/or the third transport protocol is a bi-directional reliable streaming protocol; and/or, the fourth transmission protocol is a network transmission sending stream transmission protocol. The unidirectional reliable streaming protocol of the present embodiment is QUIC Unidirection Stream, the unreliable data packet transmission protocol is QUIC Datagrams, the bidirectional reliable streaming protocol is QUIC Bidirection Stream, and the network transmission and transmission streaming protocol is WebTransportSendStream.
As shown in fig. 6, one or more embodiments of the present disclosure can also provide a data transmission apparatus applied to a server based on the same inventive concept as a data transmission method applied to a server of the present disclosure.
The data transmission device applied to the server includes, but is not limited to, a first connection module 601, a first receiving module 602, and a data pushing module 603.
The first connection module 601 is configured to establish a network communication connection with a terminal device according to history connection establishment information, where the history connection establishment information is history information of connection between a server and the terminal device.
The first receiving module 602 is configured to receive, from the terminal device, a control instruction based on the network communication connection, where the control instruction is used to control a process of a first application program on the server, and the first application program is a version of the application program on the server.
The data pushing module 603 is configured to push, to the terminal device, a data stream corresponding to the control instruction based on the network communication connection, where the data stream is used for loading and/or running a second application program on the terminal device, and the second application program is a version of the application program on the terminal device.
Optionally, the data pushing module includes an acquisition module, a creation module, and a transmission module.
The acquisition module is used for determining audio and video information, wherein the audio and video information is the audio and video information in the process of the first application program running according to the control instruction, and the audio and video information comprises audio and video.
And the creation module creates a transport stream based on the audio and video information, wherein the transport stream is included in the data stream.
And the transmission module is used for pushing the transmission stream to the terminal equipment based on network communication connection.
Optionally, the creating module is used for encoding the audio-video information into a first transport stream and a second transport stream; the first transport stream is for being transmitted to the terminal device based on a reliable transmission, and the second transport stream is for being transmitted to the terminal device based on an unreliable transmission, the transport streams including the first transport stream and the second transport stream.
Optionally, the first transport stream is an I-frame transport stream, the I-frame transport stream being for being transmitted to the terminal device based on the first transport protocol; the second transport stream is a P-frame transport stream, which is used for being transmitted to the terminal device based on a second transport protocol; wherein the priority of the first transport protocol is higher than the priority of the second transport protocol.
Optionally, the data transmission device further comprises an instruction conversion module and a process control module.
The instruction conversion module is used for converting the control instruction into a local instruction, wherein the local instruction is an instruction executed in the process of the first application program.
And the process control module is used for controlling the process of the first application program by utilizing the local instruction.
Optionally, the data transmission device further comprises an instruction sending module.
The instruction sending module is used for sending a specified instruction to the terminal equipment, wherein the specified instruction is used for controlling the state of the second application program.
Optionally, the first receiving module 602 may be configured to receive a control instruction from the terminal device based on the third transmission protocol.
The data transmission device also comprises a file receiving module.
And the file receiving module is used for receiving file packages from the terminal equipment based on a fourth transmission protocol, and a plurality of file packages form files uploaded by the terminal equipment.
Wherein the priority of the second transmission protocol is higher than the priority of the third transmission protocol, and the priority of the third transmission protocol is higher than the priority of the fourth transmission protocol.
Preferably, the first transport protocol is a unidirectional reliable streaming protocol; and/or the second transmission protocol is an unreliable data message transmission protocol; and/or the third transport protocol is a bi-directional reliable streaming protocol; and/or, the fourth transmission protocol is a network transmission sending stream transmission protocol. The unidirectional reliable streaming protocol of the present embodiment is QUIC Unidirection Stream, the unreliable data packet transmission protocol is QUIC Datagrams, the bidirectional reliable streaming protocol is QUIC Bidirection Stream, and the network transmission and transmission streaming protocol is WebTransportSendStream.
As shown in fig. 7, at least one embodiment of the present disclosure can also provide a data transmission apparatus applied to a terminal device based on the same inventive concept as the data transmission method of the present disclosure applied to a server.
The data transmission device applied to the terminal device includes, but is not limited to, a second connection module 701, an instruction sending module 702, and a data receiving module 703.
The second connection module 701 is configured to establish a network communication connection with the server, where the network communication connection is established according to historical connection establishment information in the server, and the historical connection establishment information is historical information of connection between the server and the terminal device.
The instruction sending module 702 is configured to send a control instruction to the server based on the network communication connection, where the control instruction is used to control a process of a first application program on the server, and the first application program is a version of the application program on the server.
The data receiving module 703 is configured to receive, from the server, a data stream corresponding to the control instruction based on the network communication connection, where the data stream is used for loading and/or running a second application program on the terminal device, and the second application program is a version of the application program on the terminal device.
Optionally, the server is configured to determine audio and video information, where the audio and video information is audio and video information in a process of the first application program running according to the control instruction, and the audio and video information includes audio and video; the server is used for creating a transport stream based on the audio and video information, wherein the transport stream comprises the data stream; the server is used for pushing the transmission stream to the terminal equipment based on the network communication connection. The server can be used for determining the audio and video information in a mode of collecting the audio and video information.
Optionally, the server is further operable to encode the audio-visual information into a first transport stream and a second transport stream; the first transport stream is for being transmitted to the terminal device based on a reliable transmission, and the second transport stream is for being transmitted to the terminal device based on an unreliable transmission, the transport streams including the first transport stream and the second transport stream.
Optionally, the first transport stream is an I-frame transport stream, the I-frame transport stream being for being transmitted to the terminal device based on the first transport protocol; the second transport stream is a P-frame transport stream, which is used for being transmitted to the terminal device based on a second transport protocol; wherein the priority of the first transport protocol is higher than the priority of the second transport protocol.
Optionally, the server is configured to convert the control instruction into a local instruction, where the local instruction is an instruction executed in a process of the first application program; the server is used for controlling the progress of the first application program by utilizing the local instruction.
Optionally, the data transmission device further comprises a second receiving module.
And the second receiving module is used for receiving a specified instruction from the server, wherein the specified instruction is used for controlling the state of the second application program.
Optionally, the server is configured to receive a control instruction from the terminal device based on the third transmission protocol.
The data transmission device also comprises a file sending module.
And the file sending module is used for sending file packages to the server based on a fourth transmission protocol, and a plurality of file packages form files uploaded by the terminal equipment.
Wherein the priority of the second transmission protocol is higher than the priority of the third transmission protocol, and the priority of the third transmission protocol is higher than the priority of the fourth transmission protocol.
Preferably, the first transport protocol is a unidirectional reliable streaming protocol; and/or the second transmission protocol is an unreliable data message transmission protocol; and/or the third transport protocol is a bi-directional reliable streaming protocol; and/or, the fourth transmission protocol is a network transmission sending stream transmission protocol. The unidirectional reliable streaming protocol of the present embodiment is QUIC Unidirection Stream, the unreliable data packet transmission protocol is QUIC Datagrams, the bidirectional reliable streaming protocol is QUIC Bidirection Stream, and the network transmission and transmission streaming protocol is WebTransportSendStream.
As shown in fig. 8, at least one embodiment of the present disclosure can also provide an electronic device, based on the same inventive concept as a data transmission method applied to a server, including a memory and a processor, in which computer-readable instructions are stored, which when executed by the processor, cause the processor to perform the data transmission method in any of the embodiments of the present disclosure. The specific implementation process of the data transmission method is described in detail in the present specification, and will not be described here again.
The electronic device includes a processor, a non-volatile storage medium, a memory, and a network interface connected by a system bus. The nonvolatile storage medium of the electronic device stores an operating system, a database and computer readable instructions, the database can store a control information sequence, and the computer readable instructions can enable the processor to realize a data transmission method when the computer readable instructions are executed by the processor. The processor of the electronic device is configured to provide computing and control capabilities to support the operation of the entire electronic device. The memory of the electronic device may have stored therein computer readable instructions that, when executed by the processor, cause the processor to perform a data transmission method. The network interface of the electronic device is used for communicating with the terminal connection. It will be appreciated by those skilled in the art that the structure shown in fig. 8 is merely a block diagram of a portion of the structure associated with the disclosed aspects and is not limiting of the electronic device to which the disclosed aspects apply, and that a particular electronic device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
As shown in fig. 8, at least one embodiment of the present disclosure can also provide a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the data transmission method in any of the embodiments of the present disclosure, based on the same inventive concept as the data transmission method applied to the server. The specific implementation process of the data transmission method is described in detail in the present specification, and will not be described here again.
All or part of the above-described embodiment methods may be implemented by controlling related hardware by a computer program, which may be stored in a computer readable storage medium, and the program may include the above-described embodiment methods when executed. Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection (electronic device) with one or more wires, a portable computer cartridge (magnetic device), a random access Memory (RAM, random Access Memory), a Read-Only Memory (ROM), an erasable programmable Read-Only Memory (EPROM, erasable Programmable Read-Only Memory, or flash Memory), an optical fiber device, and a portable compact disc Read-Only Memory (CDROM, compact Disc Read-Only Memory). In addition, the computer-readable storage medium may even be paper or other suitable medium upon which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable gate arrays (PGA, programmable GATE ARRAY), field Programmable gate arrays (FPGA, field Programmable GATE ARRAY), and the like.
In the description of the present specification, a description referring to the terms "present embodiment," "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present disclosure. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present disclosure, the meaning of "a plurality" is at least two, such as two, three, etc., unless explicitly specified otherwise.
The above description is of the preferred embodiments of the present disclosure, but is not intended to limit the present disclosure, and any modifications, equivalents, and simple improvements made within the spirit of the present disclosure should be included in the scope of the present disclosure.