CN120455735A

CN120455735A - Method, device, equipment and medium for transmitting multimedia resources

Info

Publication number: CN120455735A
Application number: CN202410173912.2A
Authority: CN
Inventors: 吴波
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2024-02-07
Filing date: 2024-02-07
Publication date: 2025-08-08

Abstract

The present application discloses a method, apparatus, device and medium for transmitting multimedia resources, which detects network quality information between the client and the user, and receives first experience quality information fed back by the client at the current time node; based on the network quality information, a first resource set can be determined from a plurality of resource backups with different bit rates corresponding to the multimedia resource, and then, based on the change in the first experience quality information relative to the experience quality information fed back by the client before the current time node, a target resource backup can be determined from the first resource set, and then target multimedia resource data sent to the client is generated. The present application can comprehensively consider the network quality and the client's service experience, and coordinately adjust the transmission strategy of the multimedia resource, which is conducive to optimizing the playback effect of the client. The technical solution of the present application can be widely used in various scenarios such as cloud technology and content distribution.

Description

Method, device, equipment and medium for transmitting multimedia resources

Technical Field

The present application relates to the field of information technologies, and in particular, to a method, an apparatus, a device, and a medium for transmitting multimedia resources.

Background

Currently, with the rapid development of information technology, related applications have gradually been integrated into the life of people, and various services are provided for people. For example, in some scenarios, there are multimedia applications. Multimedia applications refer to the creation, transmission, and presentation of multimedia content using related media technologies (e.g., audio, video, images, etc.). In multimedia application related applications, a client and a server are generally included, the server is responsible for storing, encoding and transmitting multimedia resources, and the client is responsible for receiving and playing multimedia resources.

In the related art, when multimedia resources are transmitted between a server and a client, some resource specifications, such as a definition specification of video resources, are generally set fixedly, so as to adapt to playing conditions of the client conveniently. However, in practical application, it is found that, because of higher complexity and easy fluctuation of the environments of the network and the client hardware, the server often has difficulty in determining the resource code rate required to be transmitted, so that the network bandwidth cannot be better utilized, and the playing effect of the client is affected.

In summary, the technical problems in the related art are to be improved.

Disclosure of Invention

The embodiment of the application provides a transmission method, a device, equipment and a medium for multimedia resources, which can cooperatively adjust the transmission strategy of the multimedia resources and optimize the playing effect of a client.

An aspect of an embodiment of the present application provides a method for transmitting a multimedia resource, where the method includes:

detecting network quality information between the client and the network;

Receiving first quality of experience information fed back by the client at the current time node;

Determining a first resource set from a plurality of resource backups corresponding to the multimedia resources according to the network quality information, wherein the code rates of the resource backups are different, and the first resource set comprises a preset number of resource backups;

Determining a target resource backup from the first resource set according to the change condition of the first quality of experience information relative to the historical quality of experience information, and generating target multimedia resource data based on the target resource backup, wherein the historical quality of experience information is the quality of experience information fed back by the client before the current time node;

And sending the target multimedia resource data to the client.

In another aspect, an embodiment of the present application provides a device for transmitting a multimedia resource, where the device includes:

the detection unit is used for detecting network quality information between the detection unit and the client;

the receiving unit is used for receiving the first quality of experience information fed back by the client side at the current time node;

The processing unit is used for determining a first resource set from a plurality of resource backups corresponding to the multimedia resources according to the network quality information, wherein the code rates of the resource backups are different, and the first resource set comprises a preset number of resource backups;

The generation unit is used for determining a target resource backup from the first resource set according to the change condition of the first quality of experience information relative to the historical quality of experience information and generating target multimedia resource data based on the target resource backup, wherein the historical quality of experience information is the quality of experience information fed back by the client before the current time node;

And the sending unit is used for sending the target multimedia resource data to the client.

Optionally, the network quality information includes available bandwidth of the network, and the processing unit is specifically configured to:

Sequencing the resource backups according to the code rate of the resource backups;

And comparing the available bandwidth of the network with the code rates, and determining a preset number of resource backups with continuous sequences as a first resource set according to the comparison result, wherein the first resource set comprises resource backups with corresponding code rates smaller than the available bandwidth of the network, and the first resource set comprises resource backups with corresponding code rates larger than the available bandwidth of the network.

Optionally, the processing unit is specifically configured to:

recording the sequencing numbers corresponding to the resource backups according to the sequencing order;

Comparing the available bandwidth of the network with the code rates, and determining a first code rate from the code rates backed up by the resources, wherein the first code rate is the code rate closest to the available bandwidth of the network among the code rates smaller than or equal to the available bandwidth of the network;

inquiring a first sequencing number corresponding to a first resource backup, wherein the first resource backup is a resource backup corresponding to the first code rate;

dividing the difference value obtained by subtracting 1 from the preset number by 2, and rounding the quotient to obtain a first numerical value;

determining a second sorting number according to the difference value between the first sorting number and the first numerical value;

and sequentially selecting the resource backups of the preset number from the second sequencing number towards the sequencing direction of corresponding code rate increase to obtain a first resource set.

Optionally, the first quality of experience information includes first katon index data, and the generating unit is specifically configured to:

Determining whether the stuck condition of the client is deteriorated according to the first stuck index data and the historical experience quality information;

and if the blocking condition of the client side is worsened, determining a target resource backup from the resource backups of which the corresponding code rates in the first resource set are smaller than the available bandwidth of the network.

Optionally, the generating unit is specifically configured to:

Determining second experience quality information from the historical experience quality information, wherein the second experience quality information is experience quality information fed back by the client at a time node which is the last time of the current time node, and the second experience quality information comprises second clamping index data;

calculating a first difference value of the first katon index data relative to the second katon index data;

Comparing the first difference value with a first threshold value;

And if the first difference value is smaller than or equal to the first threshold value, determining that the jam condition of the client is not deteriorated, or if the first difference value is larger than the first threshold value, determining that the jam condition of the client is deteriorated.

Optionally, the generating unit is specifically configured to:

determining a first accumulated value corresponding to the click-through index data before the current time node according to the historical experience quality information;

carrying out weighted summation on the first katon index data and the first accumulated value to obtain a second accumulated value;

calculating a second difference value of the second accumulated value relative to the first accumulated value;

Comparing the second difference value with a second threshold value;

and if the second difference value is smaller than or equal to the second threshold value, determining that the jam condition of the client is not deteriorated, or if the second difference value is larger than the second threshold value, determining that the jam condition of the client is deteriorated.

Optionally, the generating unit is specifically configured to:

if the jam condition of the client is deteriorated, determining a jam deterioration level corresponding to the client according to the first difference value and the first threshold;

And determining a target resource backup from the resource backups of which the corresponding code rates in the first resource set are smaller than the available bandwidth of the network according to the cartoon deterioration level.

Optionally, the first quality of experience information includes first katon index data and first buffer index data, and the generating unit is specifically configured to:

Determining whether the cache condition of the client is deteriorated according to the first cache index data and the historical experience quality information;

If the blocking condition of the client is not deteriorated and the caching condition of the client is deteriorated, determining that a second resource backup in the first resource set is a target resource backup, wherein the second resource backup is a resource backup corresponding to a second code rate, and the second code rate is greater than or equal to the code rate used by the client at the last time node of the current time node.

Optionally, the generating unit is specifically configured to:

And if the blocking condition of the client is not deteriorated and the caching condition of the client is deteriorated, carrying out frame loss processing on the target resource backup to generate target multimedia resource data.

Optionally, the generating unit is specifically configured to:

detecting the frame type of each resource frame in the target resource backup;

determining a target frame to be discarded in the target resource backup according to the frame type;

detecting play time intervals of two resource frames adjacent to the target frame;

And discarding the target frame if the playing time interval is smaller than or equal to a third threshold value, or reserving the target frame if the playing time interval is larger than the third threshold value.

Optionally, the generating unit is specifically configured to:

detecting the frame type of each resource frame in the target resource backup;

Determining a first number of consecutive target frames as a frame set;

Discarding one target frame in the frame set, and reserving other target frames except the discarded target frame in the frame set.

And if the blocking condition of the client is not deteriorated and the caching condition of the client is not deteriorated, determining a target resource backup from the resource backups of which the corresponding code rates in the first resource set are larger than the available bandwidth of the network.

In another aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;

The memory is used for storing a computer program;

the processor executes the computer program to realize the transmission method of the multimedia resource.

In another aspect, an embodiment of the present application provides a computer readable storage medium storing a computer program, where the computer program is executed by a processor to implement a method for transmitting multimedia resources as described above.

The embodiment of the application at least has the beneficial effects that the method, the device, the equipment and the medium for transmitting the multimedia resources are provided, the network quality information between the clients is detected, the first quality of experience information fed back by the clients at the current time node is received, the network quality information can reflect the network condition between the server and the clients, the first quality of experience information can reflect the service condition of the clients for actually playing the multimedia resources, the first resource set can be determined from a plurality of resource backups corresponding to the multimedia resources and with different code rates according to the network quality information, and then the target resource backup can be determined from the first resource set according to the change condition of the first quality of experience information relative to the quality of experience information fed back by the clients before the current time node, so as to generate target multimedia resource data sent to the clients. The application can integrate network quality and service experience of the client, cooperatively adjust the transmission strategy of the multimedia resource, is beneficial to improving the response efficiency of the multimedia application server and optimizing the playing effect of the client.

Drawings

The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate and do not limit the application.

Fig. 1 is a schematic diagram of transmission of a multimedia resource according to an embodiment of the present application;

Fig. 2 is a schematic diagram of an implementation environment of a method for transmitting multimedia resources according to an embodiment of the present application;

Fig. 3 is a flow chart of a method for transmitting multimedia resources according to an embodiment of the present application;

Fig. 4 is a schematic diagram illustrating transmission of another multimedia resource according to an embodiment of the present application;

Fig. 5 is a schematic flowchart of a multimedia resource transmission provided in an embodiment of the present application;

Fig. 6 is a schematic diagram of acquiring and transmitting quality of experience information according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a selection of alternative sets provided in an embodiment of the present application;

FIG. 8 is a schematic diagram of determining a play time interval according to an embodiment of the present application;

fig. 9 is a schematic diagram of a frame loss processing provided in an embodiment of the present application;

Fig. 10 is a schematic structural diagram of a transmission device for multimedia resources according to an embodiment of the present application;

Fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the application, but are merely examples of apparatuses and methods consistent with aspects of embodiments of the application as detailed in the accompanying claims.

It is to be understood that the terms "first," "second," and the like, as used herein, may be used to describe various concepts, but are not limited by these terms unless otherwise specified. These terms are only used to distinguish one concept from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of embodiments of the present application. The words "if", as used herein, may be interpreted as "when" or "in response to a determination", depending on the context.

The terms "at least one", "a plurality", "each", "any" and the like as used herein, at least one includes one, two or more, a plurality includes two or more, each means each of the corresponding plurality, and any one means any of the plurality.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the application only and is not intended to be limiting of the application.

Before describing embodiments of the present application in detail, some of the terms and expressions that are referred to in the embodiments of the present application will be described first, and the terms and expressions that are referred to in the embodiments of the present application are applicable to the following explanation.

1) TCP (Transport Control Protocol), transmission control protocol, a connection-oriented, reliable transport layer protocol, for transmitting data in a computer network.

2) QUIC (Quick UDP Internet Connection), a UDP (User Datagram Protocol) based transport protocol, aims to provide faster network connection speeds and lower delays.

3) Blockchain Blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and the like. The blockchain is essentially a decentralised database, which is a series of data blocks generated by cryptographic methods, each data block containing a batch of information of network transactions for verifying the validity (anti-counterfeiting) of the information and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, and an application services layer. The blockchains can comprise public chains, alliance chains and private chains, wherein the public chains refer to blockchains in which any person can enter a blockchain network at any time to read data, send data or contend for accounting, the alliance chains refer to blockchains which are jointly participated in management by a plurality of organizations or institutions, the private chains refer to blockchains with certain centralized control, the writing right of an account book of the private chains is controlled by a certain organization or institution, and the access and the use of data are strictly rights managed.

In the related art, when multimedia resources are transmitted between a server and a client, some resource specifications, such as a definition specification of video resources, are generally set fixedly, so as to adapt to playing conditions of the client conveniently. For example, referring to fig. 1, fig. 1 is a schematic diagram illustrating transmission of a multimedia resource according to an embodiment of the present application. In fig. 1, taking audio and video data as an example of a multimedia resource, a server may include an application layer and a transmission layer as a transmitting end of the audio and video data, where an encoder may be included in the application layer and is responsible for encoding the audio and video data into a specified data format so as to facilitate transmission to a client. The client is used as a receiving end of the audio and video data, and can comprise a player for playing the received audio and video data.

Because the network environment between the sending end and different receiving ends and the hardware environment of different receiving ends may have differences, in order to improve the transmission performance of the audio and video data as much as possible, so that the audio and video data can be reliably and stably transmitted to the receiving end, the occurrence of the conditions of blocking and the like is reduced, and for the same piece of audio and video data, the sending end often sets different resource specifications, for example, a plurality of definition specifications can be set. Each definition specification corresponds to different transmission code rates and frame rates, so that various resource files are formed. Specifically, referring to fig. 1, for example, for audio/video data, four definition specifications may be set, which are 1080p, 720p, 360p and 144p, respectively, and their respective corresponding code rates are 5Mbps, 2.5Mbps, 1Mbps and 0.5Mbps, respectively. When the sending end transmits the audio and video data to the receiving end, the network condition between the sending end and the receiving end can be detected, so that a proper resource file is adapted to the receiving end. For example, if the sender detects that the available bandwidth of the network between the sender and the receiver is reduced from 5Mbps to 3Mbps, the definition specification of the transmitted audio and video data can be adaptively replaced by 720p from 1080p, so that the probability of affecting the transmission of the audio and video data is reduced, and the playing stability of the multimedia resource is improved.

However, in practical application, it is found that the environments of the network and the client hardware have higher complexity and are easy to fluctuate, the code rate of transmission is regulated and controlled only depending on the network condition between the server and the client, the problem that the available bandwidth of the network is difficult to fully utilize may occur, and the service experience of the client in actual playing is easily affected. In order to improve the above problems, the server needs to set as many different resource specifications as possible to adapt to the change of the network situation. However, setting more resource specifications increases coding and storage overhead of the server, and increases implementation cost of multimedia resource transmission. Therefore, in the current implementation, it is difficult to adapt to the environmental complexity of the network and the client hardware in the transmission of the multimedia resource, and the playing effect of the client is easily affected.

In view of this, in the embodiment of the present application, a method, an apparatus, a device, and a medium for transmitting multimedia resources are provided, where network quality information between clients is detected, and first quality of experience information fed back by the clients at a current time node is received, where the network quality information can reflect network conditions between a server and the clients, and the first quality of experience information can reflect service conditions of the clients actually playing the multimedia resources, and according to the network quality information, a first resource set can be determined from a plurality of resource backups corresponding to the multimedia resources and having different code rates, and then according to a change condition of the first quality of experience information relative to experience quality information fed back by the clients before the current time node, a target resource backup can be determined from the first resource set, so as to generate target multimedia resource data sent to the clients. The application can integrate network quality and service experience of the client, cooperatively adjust the transmission strategy of the multimedia resource, is beneficial to improving the response efficiency of the multimedia application server and optimizing the playing effect of the client.

The transmission method of the multimedia resource provided by the embodiment of the application mainly relates to the technical field of information, and can comprise, but is not limited to, various application scenes such as cloud technology, content distribution and the like. It can be appreciated by those skilled in the art that the method for transmitting multimedia resources provided in the embodiment of the present application may be performed in various application scenarios. Specifically, the following is an exemplary description:

For example, in some embodiments, the method for transmitting multimedia resources in the embodiments of the present application may be applied in a live network scenario. In the network live broadcast, real-time video and audio contents are transmitted to terminal equipment of a spectator through a network, and in a scene of the network live broadcast, multimedia resources are transmitted in a real-time mode, so that the spectator can watch and listen to the contents in real time. Common applications for webcasting include online live platforms, sports event live, music and concert live, etc., without limitation.

For example, in some embodiments, the method for transmitting multimedia resources according to the embodiments of the present application may be applied in a video playing scene. Video playback involves transmitting recorded or stored video content to a viewer's device for viewing. In a video playing scenario, multimedia assets may be accessed by way of streaming media service transmissions or download transmissions, and viewers may view video content via an online video platform or local video player.

For example, in some embodiments, the transmission method of the multimedia resource in the embodiments of the present application may be applied in the context of an online conference. For example, online conferences are real-time audio and video communications and collaboration by connecting multiple participants over a network. In online conferences, multimedia assets are transmitted in real-time, including video, audio, and shared screens between participants.

Of course, it will be understood that the above application scenario is merely exemplary, and is not meant to limit the practical application of the transmission method of the multimedia resource provided in the embodiment of the present application. Those skilled in the art will appreciate that, in different application scenarios, the specified task may be performed by using the transmission method of the multimedia resource provided in the embodiment of the present application.

It should be noted that, in each embodiment of the present application, when related processing is required according to data related to the identity or characteristics of the object, such as information of the object, behavior data of the object, history data of the object, and position information of the object, permission or consent of the object is obtained first, and the collection, use, processing, etc. of the data complies with related laws and regulations and standards of related countries and regions. In addition, when the embodiment of the application needs to acquire the sensitive information of the object, the independent permission or independent agreement of the object is acquired through a popup window or a jump to a confirmation page and the like, and after the independent permission or independent agreement of the object is explicitly acquired, the necessary object related data for enabling the embodiment of the application to normally operate is acquired.

The following describes in detail the implementation of the embodiments of the present application with reference to the drawings. First, a method for transmitting multimedia resources according to an embodiment of the present application is described with reference to the accompanying drawings.

Referring to fig. 2, fig. 2 is a schematic diagram illustrating an implementation environment of a method for transmitting multimedia resources according to an embodiment of the present application. In this implementation environment, the main hardware and software body includes a terminal device 210 and a server 220. The terminal device 210 and the server 220 are in communication connection.

Specifically, the transmission method of the multimedia resource provided in the embodiment of the present application may be executed on the server 220 side.

Illustratively, in some embodiments, a software client for playing the multimedia resource may be installed on the terminal device 210, the server 220 may be a background server of the software client, the terminal device 210 may send a request for acquiring the multimedia resource to the server 220, and the server 220 may transmit the multimedia resource to the terminal device 210. Specifically, in the embodiment of the present application, the server 220 may detect network quality information between the terminal device 210 and receive quality of experience information fed back by the terminal device 210, where the quality of experience information includes first quality of experience information corresponding to a current time node and historical quality of experience information fed back before the current time node. The server 220 then determines a first resource set from the plurality of resource backups corresponding to the multimedia resources according to the network quality information, where the first resource set includes a predetermined number of resource backups, and each resource backup has a different code rate, and the server 220 determines a target resource backup from the first resource set according to a change condition of the first quality of experience information relative to the historical quality of experience information, and generates target multimedia resource data based on the target resource backups, and sends the target multimedia resource data to the terminal device 210. The terminal device 210 receives the target multimedia resource data and plays the target multimedia resource data, and then can further collect feedback experience quality information to the server 220, and the feedback experience quality information is circularly executed in this way, so that the stability of the transmission of the multimedia resource is improved, and the playing effect of the multimedia resource of the client is improved.

The terminal device 210 of the above embodiment may include, but is not limited to, a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle-mounted terminal, an aircraft, and the like.

The server 220 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), basic cloud computing services such as big data and artificial intelligent platforms, and the like.

In addition, server 220 may also be a node server in a blockchain network.

A communication connection may be established between the terminal device 210 and the server 220 through a wireless network or a wired network. The wireless network or wired network may be configured as the internet, or any other network, such as any combination including, but not limited to, a local area network (Local Area Network, LAN), metropolitan area network (Metropolitan AreaNetwork, MAN), wide area network (Wide AreaNetwork, WAN), mobile, wired or wireless network, private network, or virtual private network, using standard communication techniques and/or protocols. The software and hardware main bodies can adopt the same communication connection mode or different communication connection modes, and the application is not particularly limited.

Of course, it can be understood that the implementation environment in fig. 2 is only some optional application scenarios of the transmission method of the multimedia resource provided in the embodiment of the present application, and the actual application is not fixed to the software and hardware environment shown in fig. 2. The method provided by the embodiment of the application can be applied to various technical fields, such as cloud technology, content distribution and the like, and the application is not particularly limited to the fields.

In the following, a detailed description and an explanation of a method for transmitting a multimedia resource provided in the embodiment of the present application are described in conjunction with the above description of an implementation environment.

As shown in fig. 3, in an embodiment of the present application, a method for transmitting a multimedia resource is provided, and the method for transmitting a multimedia resource may be applied to the server 220 shown in fig. 2. Referring to fig. 3, the transmission method of the multimedia resource provided in the embodiment of the present application specifically includes, but is not limited to, steps 310 to 350:

step 310, detecting network quality information between the client and the network;

In the embodiment of the application, the provided transmission method of the multimedia resource can be executed by a server, the server can be in communication connection with the client, receive the related request or feedback information of the client, and transmit the multimedia resource to the client. In particular, in the embodiment of the present application, the data type of the multimedia resource transmitted between the server and the client is not limited. Illustratively, in some embodiments, the multimedia resources transmitted between the server and the client may include audio-video data, which may include a series of consecutive images and their corresponding audio data, which may be real-time (e.g., live) or recorded (e.g., movie television programs), in some embodiments, the multimedia resources transmitted between the server and the client may include audio data, and in other embodiments, the multimedia resources transmitted between the server and the client may include image data, which may be static or dynamic, which is not a limitation of the present application.

It should be noted that, in the embodiment of the present application, the server may be simultaneously connected to a plurality of different clients in communication, and the present application is not limited to a specific number of client connections. The multimedia resources transmitted between the server and the different clients may be the same data type or different data types. For example, the server may transmit audio and video data to one part of clients and simultaneously transmit audio data to another part of clients, and the transmission method of the multimedia resource provided in the embodiment of the present application may be used in the process.

In this step, the server may detect network quality information with the client when transmitting the multimedia resource to the client. Specifically, in the embodiment of the present application, the network quality information detected by the server may include one or more data indexes capable of measuring the network quality between the server and the client. For example, the network quality information may include, but is not limited to, a time delay (delay), an available bandwidth of the network, a packet loss rate of the network, and a jitter of the network, where the time delay is a time consumed between sending data and receiving data, and the lower time delay means that the data can reach the client faster, so that the network communication performance is better, the waiting time of playing the multimedia resource can be reduced, and the real-time performance is improved. The available bandwidth of a network refers to the rate at which the network transmits data, representing the amount of data that can be transmitted per unit time. Higher network available bandwidth means that large amounts of data can be transferred faster, facilitating the provision of higher quality multimedia resources. The network packet loss rate refers to the proportion of lost data packets in the original transmitted data packets in the data transmission process, and a lower packet loss rate indicates that the network is stable, the data transmission is well ensured, and a higher packet loss rate indicates that the network stability is lower and the reliability of the data transmission is insufficient. Network jitter refers to time delay variation of data packets in the transmission process, and similarly, lower network jitter means that intervals among the data packets are more stable, reliability of data transmission is better, and smooth multimedia resource playing effect is provided.

In this step, the specific implementation manner of the server to detect the network quality information is not limited, and it can be implemented according to actual requirements and in combination with technologies in the related field. For example, in some embodiments, the server may send a small test data packet to the client, measure the round trip delay of the test data packet, and obtain more accurate index data such as delay and network jitter through multiple tests, in some embodiments, the server may send a test data packet with a certain size to the client, and then detect the duration required for transmission, thereby calculating the index data of the available bandwidth of the network, and in some embodiments, the server may further be equipped with a related network diagnostic tool, and perform tracking detection on the test data packet sent to the client through the network diagnostic tool, so as to determine the index data such as delay, packet loss rate, and network jitter.

Step 320, receiving first quality of experience information fed back by the current time node client;

In the embodiment of the application, for the client, the client can periodically detect the playing condition of the multimedia resource and collect the experience quality information while playing the multimedia resource. The quality of experience information may be used to characterize the playing performance of the client side multimedia resource, which may also include at least one index data, and in the embodiment of the present application, the type of the index data included in the quality of experience information is not limited. Illustratively, in some embodiments, the quality of experience information may include, but is not limited to, metric data relating to a clip, buffer, play frame rate, etc. The blocking index data may further include content such as blocking times, blocking durations, etc., the buffer index data may include content such as buffered resource frame durations, frame numbers, and specific buffered resource sizes, etc., and the play frame rate may be used to record an average value of frames rendered per second in the play process of the video type multimedia resource, so as to represent the picture smoothness of video play. Of course, it can be understood that the quality of experience information in the embodiment of the present application is not limited to the type of index data described above, and the specific content included in the embodiment of the present application can be set according to actual requirements.

The client may send the aggregated quality of experience information to the server. In this step, the server may receive the quality of experience information fed back by the current time node client, and record the quality of experience information as the first quality of experience information.

It may be appreciated that in the embodiment of the present application, the client may periodically feed back quality of experience information to the server. Thus, in the case of a multimedia resource that has been played by the client for a period of time, the server will also receive one or more feedback of quality of experience information from the client before the current time node, and in the embodiment of the present application, and recording the experience quality information fed back to the server by the client before the current time node as historical experience quality information, wherein the historical experience quality information can specifically comprise at least one experience quality information fed back by the client.

In the embodiment of the application, the method for detecting the playing condition of the multimedia resource by the client to obtain the experience quality information is not limited, and in some embodiments, the client side can be configured with a related plug-in, the plug-in can be a performance monitoring tool, and the number of frames rendered per second, time stamps of starting and ending each time of rendering, storage data in a buffer zone and the like when the client plays the multimedia resource are obtained by calling a related interface, so that index data related to the click-through, the playing frame rate and the like are obtained by calculation, or pause or interrupt events in the process of playing the multimedia resource can be monitored, and the index data related to the click-through are obtained by analysis. In addition, the application does not limit the period of the feedback of the experience quality information from the client to the server, and can be flexibly set according to actual requirements.

And 330, determining a first resource set from a plurality of resource backups corresponding to the multimedia resources according to the network quality information, wherein the code rates of the resource backups are different, and the first resource set comprises a preset number of resource backups.

In the embodiment of the application, for the server, resource backups with different specifications can be set for each multimedia resource. Specifically, for the application of the application, the set code rate of each resource backup is different, so that the application can be conveniently adapted to various network transmission environments. Of course, in the embodiment of the present application, for each multimedia resource, the number of corresponding resource backups may be two or more, and the specific amount of the present application is not limited.

In addition, it should be noted that in the embodiment of the present application, the number of resource backups corresponding to different multimedia resources may be the same, or may be different, for example, for a multimedia resource with a larger transmission requirement, the number of resource backups corresponding to the multimedia resource may be set more, and for a multimedia resource with a smaller transmission requirement, the number of resource backups corresponding to the multimedia resource with a smaller transmission requirement may be set less.

In this step, after the server determines the network quality information between the server and the client, a resource set may be determined from a plurality of resource backups corresponding to the multimedia resources required by the client according to the network quality information, where the resource set may include part or all of the plurality of resource backups corresponding to the multimedia resources. The number of resource backups included in the first resource set may be determined in advance, and in the embodiment of the present application, the number is recorded as a predetermined number.

Specifically, in this step, according to the network quality information, the first resource set is determined from a plurality of resource backups corresponding to the multimedia resources, which may be combined with the current network quality information, to determine some resource backups that are more suitable for transmission. For example, assuming that the current available bandwidth of the network is 2.7Mbps, the code rates corresponding to the plurality of resource backups include several of 0.5Mbps, 1Mbps, 2.5Mbps, 3Mbps and 5Mbps, and the predetermined number is 2, then the resource backups with two code rates closest to the current available bandwidth of the network may be selected as the first resource set, that is, it may be determined that the resource backups with corresponding code rates of 2.5Mbps and 3Mbps form the first resource set. Of course, in other embodiments, other data indexes reflecting network quality in the network quality information may be integrated to determine the first resource set, which is not limited by the present application.

It should be added that in the embodiment of the present application, before determining the first resource set, the multimedia resource required by the client needs to be determined first, which may be determined by analyzing the request packet of the client, and the related technology has more applications, which are not described herein in detail.

It can be understood that, in the embodiment of the present application, in order to successfully determine the first resource set from the plurality of resource backups corresponding to the multimedia resource, it should be ensured that the total number of resource backups corresponding to the multimedia resource is greater than or equal to the set predetermined number. Therefore, in the embodiment of the present application, when the predetermined number is set, all the multimedia resources transmitted by using the method in the embodiment of the present application may be queried first, the minimum value is determined from the corresponding number of resource backups, and then the predetermined number is set to be less than or equal to the minimum value, so that the first resource set may be determined for each multimedia resource conveniently later.

Step 340, determining a target resource backup from the first resource set according to the change condition of the first quality of experience information relative to the historical quality of experience information, and generating target multimedia resource data based on the target resource backup, wherein the historical quality of experience information is the quality of experience information fed back by the client before the current time node;

in this step, for the obtained first resource set, the target resource backup may be determined from the first quality of experience information according to a change condition of the first quality of experience information relative to the historical quality of experience information. The target multimedia asset data is then generated based on the target asset backup.

Specifically, in the embodiment of the present application, the first resource set includes a predetermined number of resource backups, where the predetermined number of resource backups may include resource backups having a corresponding code rate greater than the available bandwidth of the network and resource backups having a corresponding code rate less than the available bandwidth of the network. In the embodiment of the application, according to the first quality of experience information and the historical quality of experience information, whether the playing effect of the current client is better or worse can be determined. It can be understood that if the playing effect of the current client is continuous and stable, the resource backup with larger code rate can be properly selected for transmission so as to further play effect of the client as far as possible, and if the playing effect of the current client is deteriorated, the resource backup with smaller code rate can be properly selected for transmission so as to improve the stability of resource transmission and improve the playing effect of the client.

In this step, after determining the target resource backup from the first resource set, the target multimedia resource data may be generated according to the target resource backup. In some embodiments, the target resource backup may be directly determined as the target multimedia resource data, and in some embodiments, the target resource backup may be further subjected to related optimization processing to obtain more applicable target multimedia resource data.

And step 350, transmitting the target multimedia resource data to the client.

In this step, after the target multimedia resource data is obtained, it may be sent to the client. After receiving the target multimedia resource data, the client can play the target multimedia resource data by using the related play assembly, so that the quality of experience information can be fed back to the server continuously, and the cyclic resource transmission control is realized.

It can be understood that in the method for transmitting multimedia resources provided in the embodiment of the present application, network quality information between clients is detected, and first quality of experience information fed back by the clients at the current time node is received, where the network quality information can reflect network conditions between the server and the clients, and the first quality of experience information can reflect service conditions of the clients actually playing the multimedia resources. The method provided by the embodiment of the application can integrate network quality and service experience of the client, and cooperatively adjust the transmission strategy of the multimedia resource, thereby being beneficial to improving the response efficiency of the multimedia application server and optimizing the playing effect of the client.

Specifically, in one possible implementation, the network quality information includes a network available bandwidth, and determining, according to the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource includes:

And comparing the available bandwidth of the network with the magnitudes of the code rates, and determining a predetermined number of resource backups with continuous sequences as a first resource set according to the comparison result, wherein the first resource set comprises the resource backups with the corresponding code rates smaller than the available bandwidth of the network, and the first resource set comprises the resource backups with the corresponding code rates larger than the available bandwidth of the network.

In the embodiment of the present application, as described above, the network quality information may include index data of available bandwidth of the network, where the available bandwidth of the network can better reflect the code rate of the multimedia resource suitable for transmission by the current network. Therefore, when the first resource set is determined from the plurality of resource backups corresponding to the multimedia resources according to the network quality information, the resource backups contained in the first resource set can be determined according to the corresponding situations of the network available bandwidth and the code rate of the resource backups.

Specifically, in the embodiment of the application, for each resource backup corresponding to the multimedia resource, the resource backups may be ordered according to the size relation of the code rates, for example, the resource backups may be ordered according to the order of the corresponding code rates from small to large, or the resource backups may be ordered according to the order of the corresponding code rates from large to small. The available bandwidth of the network and the size of each code rate can then be compared, and a predetermined number of sequential resource backups are determined as a first set of resources according to the comparison result.

Illustratively, for example, there are 6 resource backups currently, and the corresponding code rates are 1.5Mbps, 5Mbps, 2.4Mbps, 3.6Mbps, 8Mbps, and 1Mbps, respectively, and the available bandwidth of the network is 2.7Mbps, and the predetermined number is 3. The resource backups are ranked according to the relation of the code rate, the corresponding code rate is used for representing each resource backup, the ranking is from small to large, the obtained ranking results are 1Mbps, 1.5Mbps, 2.4Mbps, 3.6Mbps, 5Mbps and 8Mbps, and then the available bandwidth of the network can be compared with the size of each code rate to find that the available bandwidth is between 2.4Mbps and 3.6 Mbps. It can be understood that, because 2.4Mbps and 3.6Mbps are closer to the available bandwidth of the network by 2.7Mbps, the use of the resource backups corresponding to the two code rates can more easily realize stable and reliable transmission, and have higher utilization rate of network transmission resources, so that the two can be added into the first resource set. Whereas 1.5Mbps adjacent to a 2.4Mbps ordering location and 5Mbps adjacent to a 3.6Mbps ordering location are closer to the network available bandwidth 2.7Mbps than 1Mbps, 8Mbps not consecutive to a 2.4Mbps or 3.6Mbps ordering location, it is more reasonable to add 1.5Mbps or 5Mbps to the first set of resources. The resulting first set of resources may be backups of resources corresponding to 1.5Mbps, 2.4Mbps, 3.6Mbps in sequential order, or backups of resources corresponding to 2.4Mbps, 3.6Mbps, 5Mbps in sequential order.

It may be understood that in the embodiment of the present application, in determining the first resource set, a predetermined number of resource backups with consecutive ranks may be taken, and the resource backups are constrained to include resource backups with a code rate smaller than the available bandwidth of the network and resource backups with a code rate greater than the available bandwidth of the network, so that the first resource set may be ensured to include resource backups closest to the available bandwidth of the network, which is beneficial to improving the transmission stability of multimedia resources. In addition, in the embodiment of the application, the first resource set comprises the resource backup corresponding to the code rate smaller than the available bandwidth of the network and the resource backup corresponding to the code rate larger than the available bandwidth of the network, so that the flexibility of the target resource backup can be conveniently and subsequently determined, and the method is suitable for complex client hardware environments.

In the embodiment of the application, it can be understood that, when the first resource set is determined, in order to make the selection of the code rate more balanced, the code rate corresponding to the resource backup of the network available bandwidth in the first resource set can be located at the middle position as far as possible. The following incorporates a way of determining a first set of resources to achieve this effect.

Specifically, in one possible implementation manner, comparing the available bandwidth of the network with the size of each code rate, determining a predetermined number of resource backups with sequential ranks as a first resource set according to the comparison result, including:

comparing the available bandwidth of the network with the code rate of each code rate, and determining a first code rate from the code rates backed up by each resource, wherein the first code rate is the code rate which is the closest to the available bandwidth of the network in the code rates smaller than or equal to the available bandwidth of the network;

Inquiring a first sequencing number corresponding to a first resource backup, wherein the first resource backup is a resource backup corresponding to a first code rate;

dividing the difference value obtained by subtracting 1 from the preset number by 2, and rounding the quotient value to obtain a first numerical value;

And sequentially selecting a preset number of resource backups from the second sequencing number to the sequencing direction of corresponding code rate increase to obtain a first resource set.

In the embodiment of the application, after the resource backups are ordered according to the corresponding code rates, the ordering numbers of the resource backups can be recorded according to the ordering order, the ordering numbers can be increased from the small to the large according to the order of numbers, for example, the ordering numbers can be increased from 1 (or any number such as 0), the number corresponding to the resource backup with the code rate ordering being in the first position is 1, the number corresponding to the resource backup with the next position is 2, and so on until the number corresponding to the resource backup with the last ordering position is determined. And then, the available bandwidth of the network and the size of each code rate can be compared, and the code rate which is smaller than or equal to the available bandwidth of the network and is closest to the available bandwidth of the network is determined from the code rates of the resource backup. By way of example, for each of the resource backups described above, the corresponding code rates are 1Mbps, 1.5Mbps, 2.4Mbps, 3.6Mbps, 5Mbps, 8Mbps, and if the available bandwidth of the network is 2.7Mbps, then the first code rate may be determined to be 2.4Mbps. Then, the ranking number corresponding to the first resource backup may be queried and recorded as the first ranking number. Assuming that the numbers corresponding to the code rates 1Mbps, 1.5Mbps, 2.4Mbps, 3.6Mbps, 5Mbps, 8Mbps are sequentially 1 to 6, the first ranking number is determined to be 3.

In the embodiment of the present application, the code rate corresponding to the first ordering number is smaller than or equal to the available bandwidth of the network and is closest to the available bandwidth of the network, if the number of the code rates smaller than the available bandwidth of the network and larger than the available bandwidth of the network in the determined first resource set is the same as possible, the difference obtained by subtracting 1 according to the preset number is divided by 2, and then the quotient is rounded, and the obtained value is recorded as the first value. And taking the resource backup of the first numerical value from the position of the first sequencing number to the sequencing direction of the corresponding code rate reduction, and taking the resource backup of the preset number minus the first numerical value from the position of the first sequencing number to the sequencing direction of the corresponding code rate increase, so as to obtain the first resource set. Specifically, in the embodiment of the present application, a difference between a first ranking number and a first numerical value may be calculated, the difference is determined as a second ranking number, and then, from the second ranking number, a predetermined number of resource backups are selected in a ranking direction in which a corresponding code rate increases, so as to obtain a first resource set.

For example, assuming that there are 20 resource backups in a sequence from small to large according to the corresponding code rate, the corresponding numbers are sequentially 1 to 20, the predetermined number is 6, the available bandwidth of the network is greater than the code rate corresponding to the resource backups with the number of 7 and less than the code rate corresponding to the resource backups with the number of 8, ideally, it is desirable to select the resource backups with the number of 5 to 10 as the first resource set. According to the method provided by the embodiment of the application, the first sequence number is determined to be 7, the first numerical value is determined to be 2, and therefore, the obtained second sequence number is 5, from the resource backup with the number of 5, 5 resource backups are sequentially selected in the sequence direction of the corresponding code rate increase, and the resource backups with the numbers of 5 to 10 can be just taken as the first resource set.

It can be understood that in the embodiment of the present application, by determining the first resource set in the above manner, the code rate distribution of the resource backups in the first resource set can be more balanced as much as possible, and the applicable target resource backups can be more efficiently determined later, which is beneficial to improving the response efficiency of the multimedia application server and the stability of multimedia resource transmission.

Specifically, in one possible implementation, the first quality of experience information includes first katon index data, and determining, from the first resource set, a target resource backup according to a change condition of the first quality of experience information relative to the historical quality of experience information includes:

determining whether the jam condition of the client is deteriorated according to the first jam index data and the historical experience quality information;

If the client side is deteriorated in the stuck condition, determining a target resource backup from the resource backups of which the corresponding code rates in the first resource set are smaller than the available bandwidth of the network.

In the embodiment of the present application, as described above, the quality of experience information may include katon index data. In the embodiment of the application, the jamming index data in the first quality of experience information is recorded as first jamming index data. When determining the change condition of the first quality of experience information relative to the historical quality of experience information, determining whether the jam condition of the client is deteriorated according to the first jam index data and the historical quality of experience information.

Specifically, in the embodiment of the present application, the blocking condition of the client is worsened, which can be divided into two cases. For example, in some embodiments, the first click index data received by the current time node may not have a click phenomenon before, where the click condition of the client may be considered to be worse, in some embodiments, the first click index data received by the current time node may have a click phenomenon before, and the relevant index data may represent that the click condition of the client is more serious, for example, the number of times of the click is increased or the duration of the click is prolonged, or the click condition of the client may be considered to be worse.

In contrast, if the first jamming index data received by the current time node reflects that the client does not have jamming, for example, the jamming number is 0 or the jamming duration is 0, it is indicated that the client does not have jamming, and it is also possible to consider that the jamming of the client is not deteriorated. Or in some embodiments, the first jamming index data reflects improvement of the jamming condition of the client, for example, the jamming times are reduced, the jamming duration is shortened, and the like, and the jamming condition of the client can be considered to be not worsened.

In the embodiment of the application, if the jam condition of the client is deteriorated, the communication environment between the current server and the client is poor, or the hardware environment where the client is located is poor. In order to reduce the probability of the occurrence of the stuck condition as much as possible and optimize the playing effect of the client, a resource backup with the corresponding code rate smaller than the available bandwidth of the network can be selected from the first resource set to serve as a target resource backup. Therefore, the method and the device can better adapt to the transmission and playing environment of the current multimedia resource, improve the probability of stable transmission of the target multimedia resource data to the client, and reduce the probability of subsequent occurrence of the jamming condition of the client.

Specifically, in one possible implementation manner, determining whether the stuck condition of the client is worse according to the first stuck index data and the historical experience quality information includes:

determining second body quality of experience information from the historical quality of experience information, wherein the second body quality of experience information is the quality of experience information fed back by a time node of the client at the last time node of the current time node, and the second body quality of experience information comprises second katon index data;

Calculating a first difference value of the first jamming index data relative to the second jamming index data;

comparing the first difference value with a first threshold value;

If the first difference value is smaller than or equal to the first threshold value, determining that the jam condition of the client is not deteriorated, or if the first difference value is larger than the first threshold value, determining that the jam condition of the client is deteriorated.

In the embodiment of the present application, when determining whether the katon condition of the client is worse according to the first katon index data and the historical quality of experience information, in some embodiments, the quality of experience information of the last time node may be used to make a determination separately. Specifically, in the embodiment of the present application, the quality of experience information corresponding to the previous time node of the current time node is recorded as second body quality of experience information, and the katon index data included in the second body quality of experience information is recorded as second katon index data. It can be understood that the second katon index data can represent the katon condition fed back by the client at the previous time node, if the relative value of the first katon index data exceeds the second katon index data, the katon condition of the client can be considered to be worsened, and if the relative value of the first katon index data is basically close to the second katon index data, or the relative value of the first katon index data is smaller than the second katon index data, the katon condition of the client can be considered to be not worsened.

In the embodiment of the application, the difference value of the first katon index data relative to the second katon index data can be calculated and recorded as the first difference value. The specific calculation mode of the difference value can be flexibly set according to the requirement. For example, in some embodiments, a difference between the first and second click-through indicator data may be calculated and then determined as a first difference value, and in some embodiments, a ratio of the difference to the first click-through indicator data (or the second click-through indicator data) may also be calculated and determined as the first difference value.

It can be appreciated that in the embodiment of the present application, if the first click-through index data is less than or equal to the second click-through index data, it may be considered that the click-through condition of the client is not degraded. If the first jamming index data is larger than the second jamming index data, and the difference value between the first jamming index data and the second jamming index data is larger or the proportion of the difference value to the first jamming index data (or the second jamming index data) is larger, the jamming condition of the client can be considered to be worsened. Therefore, in the embodiment of the present application, a threshold may be set and recorded as a first threshold, where the first threshold is a positive number, and the size of the threshold is not limited by the present application. If the calculated first difference value is a negative number, it can be determined that the jam condition of the client is not deteriorated, if the calculated first difference value is a positive number and the value is smaller and is within a first threshold, it can be determined that the relevant value of the first jam index data is basically close to the second jam index data, the jam condition of the client is not deteriorated, and if the calculated first difference value is larger than the first threshold, it can be considered that the jam condition of the client is deteriorated.

It can be understood that in the embodiment of the application, based on comparison between the second jamming index data and the first jamming index data, whether the jamming condition of the client is deteriorated or not can be rapidly determined, which is beneficial to improving the transmission efficiency of the multimedia resource.

Comparing the second difference value with a second threshold value;

In the embodiment of the present application, when determining whether the katon condition of the client is worse according to the first katon index data and the historical quality of experience information, in some embodiments, the quality of experience information of the historical period may be used to make a determination comprehensively. Specifically, in the embodiment of the present application, according to the historical quality of experience information, an accumulated value corresponding to the katon index data before the current time node may be determined and recorded as a first accumulated value. Then, the first click-on index data and the first accumulated value can be weighted and summed to obtain a new accumulated value, and the new accumulated value is recorded as a second accumulated value.

When the second cumulative value is obtained, a difference value of the second cumulative value relative to the first cumulative value may be calculated, and in the embodiment of the present application, the difference value is recorded as the second difference value. The specific meaning and calculation manner of the second difference value may be implemented by referring to the foregoing first difference value, which is not described herein in detail. It can be understood that if the correlation value of the first click on index data is larger, that is, if the click on condition of the client is worsened, the obtained second cumulative value will be increased relatively to the first cumulative value, so that the value of the second difference value will be larger, whereas if the correlation value of the first click on index data is smaller, that is, if the click on condition of the client is not worsened, the obtained second cumulative value will not be increased or only slightly increased relative to the first cumulative value, and the value of the second difference value will be smaller. Therefore, in the embodiment of the present application, a threshold may be set and recorded as a second threshold, where the second threshold is a positive number, and similarly, the size of the threshold is not limited by the present application. If the calculated second difference value is a negative number, it can be determined that the jam condition of the client is not deteriorated, if the calculated second difference value is a positive number and the value is smaller and is within a second threshold, it can be considered that the jam condition of the client is not deteriorated, and if the calculated value of the first difference value is larger and is larger than the second threshold, it can be considered that the jam condition of the client is deteriorated.

It should be noted that, in the embodiment of the present application, when the first cumulative value is determined, it may also be updated in a weighted manner step by step. For example, the first accumulated value may be weighted by the second katon metric data and the accumulated value obtained by the last time node of the current time node, which is not limited by the present application.

It can be understood that in the embodiment of the application, the judgment is comprehensively made based on the experience quality information of the historical period, so that the interference caused by temporary performance fluctuation can be reduced, the blocking condition of the client can be reflected more accurately, and the transmission operation of the multimedia resource can be conveniently and adaptively carried out.

Specifically, in one possible implementation manner, if the katon condition of the client side is worsened, determining the target resource backup from the resource backups with the corresponding code rate smaller than the available bandwidth of the network in the first resource set includes:

if the jam condition of the client side is deteriorated, determining a jam deterioration level corresponding to the client side according to the first difference value and the first threshold value;

and determining a target resource backup from the resource backups with the corresponding code rates smaller than the available bandwidth of the network in the first resource set according to the katon degradation level.

In the embodiment of the application, if the client is determined to have deterioration in the stuck condition, the target resource backup can be determined by combining the severity of the deterioration when the target resource backup is determined from the resource backups in the first resource set. Specifically, in the embodiment of the present application, the katon degradation level corresponding to the client may be determined according to the first difference value and the first threshold. For example, when the first difference value exceeds the first threshold, it is determined that the client has a degradation, and then a multiple between the first difference value and the first threshold may be further calculated, where if the multiple between the first difference value and the first threshold is greater, it indicates that the client has a more serious degradation of the client, i.e. the client has a higher degradation level of the client, and conversely, if the multiple between the first difference value and the first threshold is smaller, it indicates that the client has a less serious degradation of the client, i.e. the client has a lower degradation level of the client.

In the embodiment of the application, after the katon degradation level is determined, the target resource backup can be determined according to the katon degradation level. Specifically, it can be understood that, if the katon degradation level of the client is higher, it indicates that the transmission and playing of the multimedia resource are relatively unstable, so as to improve the katon condition of the client as much as possible, the code rate of the transmission can be reduced, that is, the code rate in the first resource set can be selected to be smaller than the available bandwidth of the network, and the relatively smaller resource backup is used as the target resource backup. In contrast, if the katon degradation level of the client is low, it means that the transmission and playing of the multimedia resource are stable, and in order to improve the playing effect of the client as much as possible, a higher code rate transmission can be used, that is, a resource backup with a code rate smaller than the available bandwidth code rate of the network in the first resource set can be selected as the target resource backup.

Of course, it should be noted that, in the embodiment of the present application, the katon degradation level corresponding to the client may also be determined according to the second difference value and the second threshold, so as to adaptively determine the target resource backup from the first resource set, which is not described herein in detail.

Specifically, in one possible implementation manner, the first quality of experience information includes first katon index data and first buffer index data, and determining, according to a change condition of the first quality of experience information relative to the historical quality of experience information, a target resource backup from the first resource set includes:

In the embodiment of the present application, as described above, the quality of experience information may further include buffer index data. Specifically, the buffer index data in the first quality of experience information is recorded as first buffer index data, and in the embodiment of the present application, whether the buffer condition of the client is deteriorated may be determined according to the first buffer index data and the historical quality of experience information, and the specific determination mode may be implemented by referring to the katon index data, which is not described herein in detail.

In the embodiment of the application, if the client is determined to have no deterioration in the stuck condition and the client is determined to have deterioration in the cache condition, the communication environment between the server and the client in the previous stage is good, and the playing effect is affected to a certain extent by the hardware performance of the client. In the embodiment of the application, the code rate used by the node in the last time of the current time node can be firstly inquired and recorded as the third code rate. If the communication environments of the current server and the client are still good, the transmission code rate can be properly improved, namely, one code rate which is larger than or equal to the third code rate can be determined in the first resource set and is recorded as a second code rate, the resource backup corresponding to the second code rate is recorded as a second resource backup, and then the second resource backup can be determined as a target resource backup.

In the embodiment of the present application, if the communication environments of the current server and the client are worse, and the second code rate greater than or equal to the third code rate does not exist in the first resource set, the server may keep to transmit by using the resource backup of the third code rate, so as to avoid the situation that the playing effect is worse due to the replacement of the code rate.

Specifically, in one possible implementation, generating the target multimedia resource data based on the target resource backup includes:

If the blocking condition of the client is not deteriorated and the caching condition of the client is deteriorated, carrying out frame loss processing on the target resource backup to generate target multimedia resource data.

In the embodiment of the application, as described above, if the katon condition of the client is not deteriorated and the buffer condition is not deteriorated, the server can properly raise the transmission code rate of the multimedia resource under the condition that the communication environment is good, and maintain the transmission code rate used before under the condition that the communication environment is poor. In order to alleviate the problem of cache deterioration at the client side, in the embodiment of the application, frame loss processing can be performed on the target resource backup to obtain target multimedia resource data, so that the client can stably realize normal play of the multimedia resource as much as possible.

Specifically, in one possible implementation manner, frame loss processing is performed on the target resource backup, so as to generate target multimedia resource data, which includes:

Detecting the frame type of each resource frame in the target resource backup;

If the playing time interval is smaller than or equal to the third threshold value, discarding the target frame, or if the playing time interval is larger than the third threshold value, reserving the target frame.

In the embodiment of the application, when frame loss processing is performed on the target resource backup, in some embodiments, the frame type of each resource frame in the target resource backup can be detected first. Taking a video frame as an example, the video frame includes a frame that needs to be used when encoding and decoding, and also includes a frame that is not relied upon by other frames when encoding and decoding, and for the second video frame, the video frame can be regarded as a type that can be discarded. Therefore, the frames to be discarded in the target resource backup can be determined according to the frame types of the resource frames, and in the embodiment of the application, the frames are marked as target frames.

After determining the target frames, in the embodiment of the application, in order to reduce the influence on the playing performance of the client, the target frames are not directly discarded, but the playing time interval of two adjacent resource frames of the discarded target frames is constrained to be within a certain range, so that the smoothness of the picture is better as much as possible. Specifically, in the embodiment of the present application, the play time interval of two resource frames adjacent to each target frame may be detected. In the embodiment of the present application, two adjacent resource frames of the target frame refer to one resource frame before the target frame and one resource frame after the target frame. If the playing time interval of the two resource frames is shorter, it is indicated that the target frames in the middle of the two resource frames are discarded, the playing fluency is less affected, and the target frames can be selected to be discarded, and conversely, if the playing time interval of the two resource frames is longer, it is indicated that the target frames in the middle of the two resource frames are discarded, the playing fluency is more affected, and at the moment, the target frames can be selected to be reserved.

In the embodiment of the application, a time threshold may be set and recorded as a third threshold. The target frame may be discarded if the play time interval of two resource frames adjacent to the target frame is less than or equal to the third threshold, and the target frame may be reserved if the play time interval of two resource frames adjacent to the target frame is greater than the third threshold.

Detecting the frame type of each resource frame in the target resource backup;

Determining a first number of consecutive target frames as a frame set;

In the embodiment of the application, when frame loss processing is performed on the target resource backup, in some embodiments, after determining the target frames to be discarded, there may be a situation that a plurality of target frames are continuous, and if all the target frames are discarded, the problem of image splitting or blocking is easily caused. Therefore, in the embodiment of the present application, the continuity condition of the target frames may be detected, a value may be preset and recorded as the first number, and the continuous first number of target frames may be determined as the frame set, where the continuity refers to that each target frame is adjacent. For the whole frame set, one target frame is discarded, and other target frames except the discarded target frame are reserved. Therefore, the probability of picture splitting or blocking caused by frame loss processing can be reduced, and the playing effect of the client can be improved.

In the embodiment of the application, when the frame set is divided, the target frame in the frame set does not participate in the division of other frame sets. For example, if the first set number is 5, and there are 20 target frames in a certain piece of data backed up by the target resource, the 20 target frames may be divided into 4 frame sets, each frame set discards 1 target frame, and retains 4 target frames. After 20 target frames are subjected to frame loss processing, 16 target frames remain.

It should be noted that in the embodiment of the present application, the two strategies for frame dropping processing may be combined together, for example, the target frames to be dropped are determined according to the playing time intervals of two resource frames adjacent to each target frame, then the target frames to be dropped are screened according to whether there is a continuous condition, the target frames to be dropped are updated to be the target frames to be reserved, and finally the dropping processing is performed.

If the jam condition of the client is not deteriorated and the cache condition of the client is not deteriorated, determining a target resource backup from the resource backups with the code rate larger than the available bandwidth of the network corresponding to the first resource set.

In the embodiment of the application, if the blocking condition and the caching condition of the client are not deteriorated, a higher code rate can be selected for transmitting the multimedia resources, namely, in the first resource set, the corresponding resource backup with the code rate larger than the available bandwidth of the network is determined as the target resource backup. Therefore, the playing performance of the client can be improved, and network communication resources and hardware resources of the client are well utilized.

The method for transmitting the multimedia resource provided by the application is described and illustrated in detail below with reference to a specific application implementation flow.

Referring to fig. 4, fig. 4 is a schematic diagram illustrating transmission of another multimedia resource according to an embodiment of the present application. In fig. 4, taking an example that the multimedia resource is audio/video data, a server structure using the transmission method of the multimedia resource provided in the embodiment of the present application is described. Specifically, in the embodiment of the application, the enabling server detects the network quality information and perceives the client experience quality information, and the corresponding coding and decoding strategies can be quickly adjusted through the cooperation of the transmission layer and the application layer, so that the problems that the coding strategies in the traditional transmission mode are limited and the response cannot be quickly made are solved, and the playing performance and the service experience of the client can be effectively optimized.

Referring to fig. 4, in an embodiment of the present application, a server includes an application layer and a transport layer, where the application layer may store multiple resource backups corresponding to multimedia resources, where each resource backup has a different code rate. The transmission layer may include a control module, a transmitting module, a receiving module, and a monitoring module. The transmitting module and the receiving module may be connected to the receiving end (i.e., client) through the internet.

In the embodiment of the application, in the transmission process of the multimedia resource, the following steps are mainly executed:

① After receiving the audio and video traffic, the client replies a message confirmation message to the server, and periodically feeds back buffer index data (such as buffer duration, buffer audio and video frame number, buffer data amount, etc.) or katon index data (such as katon number, katon duration, etc.) of the own player to the server, after receiving the information, the receiving module of the ② server transmits the information to the monitoring module, the monitoring module evaluates network quality information according to the information and extracts client buffer index data or katon index data, the ③ monitoring module transmits the evaluated network quality information and experience quality information (including buffer index data or katon index data) of the client to the control module of the server, the control module decides a control instruction to the application layer according to the information, the ④ control module transmits the control instruction to the application layer, the application layer selects audio and video data with proper definition, code rate and frame rate according to the control instruction, the selected audio and video data (namely, target backup) to the monitoring module, the control module evaluates the network quality information and extracts client buffer index data or katon index data, the ③ monitoring module transmits the evaluated network quality information and experience quality information (including buffer index data or katon index data) of the client to the control module, the control module determines a control instruction to the application layer, the control instruction is transmitted to the application layer, the application layer selects audio and video data with proper definition, the audio and video data with frame rate and audio and video data of the frame rate according to the control instruction, the control module is ⑧ is processed by the control resource, and the audio and video data is processed by the audio and video data of the audio and video data.

Specifically, the above-described flow is described and illustrated in detail below.

Referring to fig. 5, fig. 5 is a schematic diagram illustrating a specific flow of multimedia resource transmission according to an embodiment of the present application.

In the embodiment of the application, after receiving the audio and video data from the server, the client replies a message confirmation message (pkt_ack message) and a message (pkt_qoe_ack message) containing the quality of experience information QoE of the client to the server:

In the embodiment of the application, the QoE of the client comprises, but is not limited to, the index data (Player_ freezing) of the click index encountered by the client Player in the past period, specifically comprising the click times (freezing _times) and the click time length (freezing _duration), the index data (Player_buffering) of the current client Player, specifically comprising the buffered audio/video frame duration (buffer_time_len), the frame number (buffer_frame_frame_count), the buffered data size (buffer_frame_size) and the like.

In the embodiment of the present application, in order to achieve the acquisition of the quality of experience information QoE of the client, a QoE acquisition module and a QoE feedback module may be configured on the client, and referring to fig. 6, fig. 6 shows a schematic diagram of the acquisition and transmission of quality of experience information provided in the embodiment of the present application. In the embodiment of the application, after receiving the audio and video data from the server, the client transmits the audio and video data to the player cache of the application layer through the transmission layer, and the player acquires the audio and video frames from the cache and displays the audio and video frames in the pictures and the sound of the player. In the embodiment of the present application, the QoE acquisition module of the client periodically acquires the index data of the QoE of the experience quality from the player cache and the player, specifically, the time interval for the QoE acquisition module of the client to acquire the QoE of the experience quality may be denoted as t_qoe, where the size of t_qoe may be configured by an administrator, and for example, t_qoe=100 ms.

In the embodiment of the application, a QoE acquisition module of a client can transmit the information to a QoE feedback module after periodically acquiring the katon index data and the cache index data, and the QoE feedback module constructs a QoE feedback message Pkt_QoE_ack containing the information after acquiring the katon index data and the cache index data transmitted by the QoE acquisition module and transmits the message back to a server, wherein the content carried by the message Pkt_QoE_ack can be represented by a formula (1):

Pkt_QoE_ack={Player_freezing, Player_bufferin} (1)

In the embodiment of the application, after receiving a reply message acknowledgement message (pkt_ack message) from a client and the pkt_qoe_ack message containing the quality of experience information QoE, a receiving module of the server transmits the messages to a monitoring module of the server, and after receiving the ACK message or the pkt_qoe_ack message sent by the receiving module, the monitoring module detects the real-time network quality information QoS and extracts the quality of experience information QoE of the client, which is as follows:

In the embodiment of the application, the monitoring module of the server can detect (or measure) the network quality information, namely the available bandwidth BtlBW of the network, the round trip time RTT and the network packet Loss rate loss_rate. The monitoring module periodically transmits a pkt_qos message to the control module, where the message includes the detected network quality information, as shown in formula (2):

Pkt_QoS={BtlBW, RTT, Loss_rate} (2)

In the embodiment of the application, after receiving the pkt_qoe_ack message transmitted from the receiving module, the monitoring module of the server extracts the carried clip index data player_ freezing and buffer index data player_buffering of the client Player from the message, and transmits the information to the control module of the server through the message pkt_qoe, wherein the format of the pkt_qoe message transmitted to the control module is shown in formula (3):

Pkt_QoE={Play_freezing, Player_buffering} (3)

In the embodiment of the present application, a control module of a server maintains a table for network quality information QoS and quality of experience information QoS of each connected client, and updates the table after receiving a pkt_qos message and a pkt_qoe message from a monitoring module, and decides a control instruction sent to an application layer according to parameter information in the table, specifically as follows:

referring to Table 1, table 1 shows a maintenance Table table_qos corresponding to network quality information QoS provided in the embodiment of the present application.

TABLE 1

Index (I)	Previously accumulated value	Latest value	Latest accumulated value
				BtlBW	Cum_bw_last	BW_new	Cum_bw_new
RTT	Cum_rtt_last	RTT_new	Cum_rtt_new
				Loss_rate	Cum_loss_last	Loss_rate_new	Cum_loss_new

In the embodiment of the application, when the control module receives the pkt_qos message from the monitoring module, extracts the information of the latest network available bandwidth BtlBW, round trip time RTT, network packet Loss rate loss_rate and the like carried by the message, and updates the latest value (value_new) of the three indexes in the Table 1, wherein the latest accumulated value (value_new) corresponding to the table_qos is calculated by the formula (4):

Cum_new=alpha*Cum_last+(1-alpha)*value_new (4)

Wherein, cure_last refers to the index accumulated value before the latest index values (i.e. BW_new, RTT_new and Low_rate_new) are obtained, and parameter alpha is a weight value of calculating the latest accumulated value, which is 0≤alpha≤1, and the parameter can be configured by an administrator, and under the default, alpha=0.8. In table 1, the previous accumulated values of three indexes of the network available bandwidth, the round trip delay and the network packet Loss rate are respectively represented by a cut_bw_last, a cut_rtt_last and a cut_loss_last, the latest values of the three indexes are respectively represented by a bw_new, an rtt_new and a loss_rate_new, and the latest accumulated values of the three indexes are respectively represented by a cut_bw_new, a cut_rtt_new and a cut_loss_new.

Referring to Table 2, table 2 shows a maintenance Table table_qoe corresponding to quality of experience information QoS provided in the embodiment of the present application.

TABLE 2

Index (I)	Previously accumulated value	Latest value	Latest accumulated value
				freezing_times	Cum_ft_last	Ft_new	Cum_ft_new
freezing_duration	Cum_fd_last	Fd_new	Cum_fd_new
				buffer_time_len	Cum_btl_last	Bt_new	Cum_bt_new
buffer_frame_amount	Cum_bfa_last	Bfa_new	Cum_bfa_new
				buffer_frame_size	Cum_bfs_last	Bfs_new	Cum_bfs_new

In the embodiment of the application, when the control module receives the pkt_qoe message from the monitoring module, extracts the latest information such as the katon index data (i.e. the first katon index data) and the cache index data (i.e. the first cache index data) carried by the pkt_qoe message, and updates the latest value (marked as value_ QoE _new) of the indexes in the above Table 2, and the latest accumulated value (Cur_ QoE _new) of the maintenance Table table_qoe is calculated by the formula (5):

Cum_qoe_new=beta*Cum_qoe_last+(1-beta)*value_qoe_new (5)

Wherein, curie_ qoe _last refers to the accumulated value before the latest index value is obtained (i.e. the first accumulated value), the parameter beta is a weight value for calculating the latest accumulated value, 0 is less than or equal to beta is less than or equal to 1, the parameter is configured by an administrator, beta=0.8 under default, in Table 2, the "previous accumulated value" of the number of clamping times, the clamping time, the audio-video frame time, the frame number and the data size of the buffer is respectively represented by Curie_ft_last, curie_fd_last, curie_ btl _last, curie_ Bfa _last and Curie_ bfs _last, the latest values of the five indexes are respectively represented by Ft_new, fd_new, bt_new, bfa_new and Bfs _new, and the latest accumulated values of the five indexes (i.e. the second accumulated value) are respectively represented by Curie_ft_new, curie_d_new, curie_24_new and Curie_ bfs _new.

In the embodiment of the application, a control module of a server decides a control instruction sent to an application layer according to the latest table_QoS Table and the latest table_QoE Table, and the control instruction is specifically as follows:

In the embodiment of the application, the resource backup of the same audio and video data resource (uniquely identified by URI) is represented by the identifier BKP_ID, for example, when n resource backups exist, BKP_0 is the audio and video data resource backup with the lowest audio and video resource definition, the minimum code rate and the minimum frame rate, and BKP_n is the audio and video data resource backup with the highest audio and video resource definition, the maximum code rate and the maximum frame rate.

At this time, the ins_ctl message (labeled as pkt_ins_ctl message) sent by the control module may be represented by the following formula (6):

Pkt_ins_ctl={conn_id, URI, BKP_ID} (6)

Wherein conn_id represents the identity of the connection, which can be uniquely identified by five tuples (i.e. source address, source port number, destination address, destination port number and protocol number) in the TCP protocol, and by the connection identity of the protocol itself in the QUIC protocol,

In the embodiment of the application, the resource backup selected in the previous period is represented by BKP_s, and the BKP_ID in the control instruction ins_ctl sent to the application layer by the control module is determined by the detected network quality information QoS and the detected experience quality information QoS. Specifically, the control module first decides an alternative Set bkp_set (i.e. a first resource Set) of bkp_ids to be selected according to the network available bandwidth BtlBW in the pkt_qos packet, and the specific method is as follows:

The control module selects a backup of resources in the alternative Set bkp_set from the vicinity of the network available bandwidth BtlBW. Referring to fig. 7, fig. 7 is a schematic diagram illustrating a selection of an alternative set according to an embodiment of the present application. As shown in fig. 7, the control module selects the bkp_id closest to the detected network available bandwidth BtlBW according to the code rate of the audio/video data. In the embodiment of the present application, the number m (i.e., the predetermined number) of resource backups BKP in bkp_set may be configured in advance by an administrator, and in an exemplary embodiment, m may be 4, so that the alternative Set bkp_set may be represented by the following formula (7):

BKP_Set={BKP_j, BKP_j+1, BKP_j+2, ..., BKP_j+m-1} (7)

Wherein the code rate of BKP_ { (2 x j+m-1)/2 } is not less than BtlBW and not more than BKP_ { (2 x j+m-1)/2+1 }, in the embodiment of the application,

The result of (2 x j+m-1)/2 is an integer fraction.

In the embodiment of the present application, for convenience of description, bkp_id adjacent to the network available bandwidth BtlBW in the code rate corresponding to the resource backup BKP is respectively denoted as bkp_p and bkp_q.

The control module decides according to the katon index data and the buffer index data in the packet pkt_qoe, and selects a process of backing up the target resource from the alternative Set bkp_set, specifically as follows:

If the katon index data fed back by the client is poor, selecting a resource backup BKP with a code rate smaller than the network available bandwidth BtlBW from the alternative Set BKP_set, wherein in the embodiment of the application, the judging basis of the degradation of the Play_ freezing is any one of the following conditions:

i. The client's latest stuck time ft_new (or stuck time fd_new) increases by a proportion (noted as ratio_ freezing _new) of more than 1 monitoring period over a set threshold ratio_ freezing _threshold, where the threshold ratio_ freezing _threshold is configured by the administrator, illustratively ratio_ freezing _threshold = 10%;

ii. the last accumulated number of jams of the client (or the duration of time, com fd new) increases by a proportion (noted as ratio_com freezing) exceeding a threshold ratio_com_ freezing _threshold compared to the previous accumulated number of times value, com ft last (or the accumulated duration value, com fd last); wherein the threshold atio _cum_ freezing _threshold is configured by an administrator, illustratively atio _cum_ freezing _threshold = 10%;

in the embodiment of the present application, the ratio_cum_ freezing may be calculated by the formula (8) (taking the accumulated stuck number as an example):

Ratio_cum_freezing=(cum_fd_new–cum_fd_last)/cum_fd_last (8)

It should be noted that, in the embodiment of the present application, if the ratio_ freezing _new (or the ratio_cut_ freezing) exceeds 1 (but is less than 2) times the ratio_ freezing _threshold (or the ratio_cut_ freezing), then the resource backup BKP (i.e., bkp_p) with the smallest gap from the network available bandwidth BtlBW is selected from the resource backups BKP with the code rate smaller than the network available bandwidth BtlBW, and if the ratio_ freezing _new (or the ratio_cut_ freezing) exceeds 2 (but is less than 3) times the ratio_ freezing _threshold (or the ratio_cut_ freezing), the resource backup BKP with the second smallest gap from the network available bandwidth BtlBW is selected from the resource backups BKP with the code rate smaller than the network available bandwidth BtlBW, and other cases may be omitted here.

In the embodiment of the application, if the cache index data fed back by the client is poor but the katon index data is not poor, a resource backup with the code rate larger than the network available bandwidth BtlBW can be selected from the alternative set, or the previous BKP_s is continuously selected, and the corresponding differentiated frame loss operation is executed.

In the embodiment of the application, the decision basis of the cache index data degradation is one of the following arbitrary conditions:

i. The latest buffer time bt_new (or buffer frame number bfa _new, buffer frame size bfs _new) of the client exceeds the ratio of the reduction of the previous 1 monitoring period (recorded as ratio_buffering_new) by more than a preset threshold ratio_buffering_threshold, wherein the threshold size is configured by an administrator, and the ratio_buffering_threshold=10%;

The newly accumulated buffer duration of the client (or accumulated buffered frame number of times, cum bfa new, accumulated buffered frame size of cum bfs new) is reduced by a proportion (denoted ratio_buffering_cum) exceeding a preset threshold ratio_cum_buffering_threshold, wherein the threshold size is configured by an administrator, and the ratio_cum_buffering_threshold=10%;

In the embodiment of the application, if the cache index data fed back by the client is not degraded and the cartoon index data is not degraded, the resource backup with the code rate larger than the available bandwidth BtlBW of the network is selected from the BKP_set, the specific selection of the resource backup with the code rate larger than the available bandwidth BtlBW of the network can be Set according to the actual requirement, for example, the current server determines that the cache index data is not degraded and the cartoon index data is not degraded, the available bandwidth BtlBW =1 Mbps of the network is obtained, the code rate of the resource backup BKP exceeding the available bandwidth BtlBW of the network in the alternative Set BKP_set is respectively 1.5Mbps, 2Mbps and 5Mbps, and if the constraint that the code rate exceeds the available bandwidth BtlBW of the network is 0.5Mbps to 2Mbps, the resource backup with the code rate of 2Mbps can be selected.

In the embodiment of the application, the control module of the server can send the selected BKP_ID to the application layer in the form of a Pkt_ins_ctl message, and the application layer pulls corresponding audio and video data according to the BKP_ID and transmits the corresponding audio and video data to the control module:

If the control module monitors that the current connection does not need to perform frame loss operation, transmitting the audio and video data received from the application layer to the transmitting module; if the control module monitors that the current connection needs to perform frame dropping operation, the control module needs to perform dropping operation on non-key video frames in the audio and video stream. Specifically, the audio and video frame types include an audio frame, a video I frame, a video P frame, a discardable video B frame, and a non-discardable video B frame, where a discardable B frame indicates that other frames do not depend on the frame to perform encoding and decoding, and a non-discardable B frame indicates that other frames may depend on the frame to perform encoding and decoding, and in the embodiment of the present application, when the control module performs a discarding operation on a currently connected B frame, the following frame discarding rule is complied with:

i. The above 4 audio and video frames such as audio frames, video I frames, P frames and non-discardable B frames can not be discarded; the control module performs frame dropping operation on the B frames which can be dropped;

The play time interval delta_pts between two adjacent video frames after the frame loss cannot exceed its threshold delta_threshold (i.e. the third threshold). Referring to fig. 8, fig. 8 shows a schematic diagram for determining a play time interval provided in an embodiment of the present application, in fig. 8, there are consecutive multi-frame video frames, where a fourth video frame 4 belongs to a discardable video B frame, and when deciding whether to discard the fourth video frame 4, the play time interval between two adjacent video frames (a third video frame 3 and a fifth video frame 5) is calculated, which can be obtained by using the formula (9):

Delta_pts=Pts_5–Pts_3 (9)

Wherein Pts_3 and Pts_5 represent the playing time points of the third video frame 3 and the fifth video frame 5 at the client player respectively, then compare with the Delta_threshold, if the playing time interval Delta_pts is larger than the threshold Delta_threshold, the fourth video frame 4 is reserved, otherwise, the fourth video frame 4 is discarded. In the embodiment of the application, the threshold delta_threshold is configured by an administrator and declared in a configuration file, and delta_threshold=200 ms is exemplified.

In the embodiment of the present application, if the continuous multiple video frames are all discardable video B frames, the control module does not discard all the B frames, but performs frame discarding according to the parameter amountcan drop B, specifically, at most 1 discardable B frames in the continuous amountcan drop B discardable B frames.

For example, referring to fig. 9, fig. 9 is a schematic diagram illustrating a frame loss processing according to an embodiment of the present application. In fig. 9, one includes six video frames, which are all video frames that can be discarded. If all the six video frames are discarded, the number of times or duration of the blocking of the client player may be increased, in this embodiment of the present application, the parameter amountcan_drop_b=3 may be set, that is, if 3 video frames that can be discarded continuously occur, the control module may discard only one of the video frames, for example, in fig. 9, the first video frame 1, the second video frame 2, and the third video frame 3 may be determined as one frame set 910, and then one video frame (say, the first video frame 1) is discarded from the frame set 910, and the remaining other video frames are reserved, so as to complete the frame discarding process.

In the embodiment of the application, the control module transmits the audio and video data after frame loss to the transmitting module after executing the frame loss operation according to the rule, and the transmitting module transmits the audio and video data from the control module to the client in the form of a message after receiving the data, so that the playing of the audio and video data can be completed at the client.

Referring to fig. 10, the embodiment of the present application further provides a device for transmitting multimedia resources, where the device includes:

A detecting unit 1010, configured to detect network quality information between the network quality information and the client;

a receiving unit 1020, configured to receive first quality of experience information fed back by a current time node client;

the processing unit 1030 is configured to determine, according to the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource, where each resource backup has a different code rate, and the first resource set includes a predetermined number of resource backups;

The generating unit 1040 is configured to determine a target resource backup from the first resource set according to a change condition of the first quality of experience information relative to the historical quality of experience information, and generate target multimedia resource data based on the target resource backup, where the historical quality of experience information is quality of experience information fed back by the client before the current time node;

a transmitting unit 1050, configured to transmit the target multimedia resource data to the client.

Optionally, the network quality information comprises a network available bandwidth, and the processing unit is specifically configured to:

Optionally, the processing unit is specifically configured to:

Optionally, the generating unit is specifically configured to:

comparing the first difference value with a first threshold value;

Optionally, the generating unit is specifically configured to:

Comparing the second difference value with a second threshold value;

Optionally, the generating unit is specifically configured to:

Detecting the frame type of each resource frame in the target resource backup;

Optionally, the generating unit is specifically configured to:

Detecting the frame type of each resource frame in the target resource backup;

Determining a first number of consecutive target frames as a frame set;

It can be understood that the content in the embodiment of the method for transmitting a multimedia resource shown in fig. 3 is applicable to the embodiment of the apparatus for transmitting a multimedia resource, and the functions of the embodiment of the apparatus for transmitting a multimedia resource are the same as those of the embodiment of the method for transmitting a multimedia resource shown in fig. 3, and the advantages achieved are the same as those achieved by the embodiment of the method for transmitting a multimedia resource shown in fig. 3.

The embodiment of the application also discloses an electronic device, which comprises:

at least one processor;

At least one memory for storing at least one program;

The at least one program, when executed by the at least one processor, causes the at least one processor to implement an embodiment of a method of transmitting a multimedia asset as shown in fig. 3.

It can be understood that the content in the embodiment of the method for transmitting a multimedia resource shown in fig. 3 is applicable to the embodiment of the electronic device, and the functions specifically implemented by the embodiment of the electronic device are the same as those in the embodiment of the method for transmitting a multimedia resource shown in fig. 3, and the beneficial effects achieved by the embodiment of the method for transmitting a multimedia resource shown in fig. 3 are the same as those achieved by the embodiment of the method for transmitting a multimedia resource shown in fig. 3.

The electronic device of the embodiment of the application can be a terminal device, a computer device or a server device.

Referring to fig. 11, fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Taking an example in which the electronic device is a terminal device, in fig. 11, the electronic device 1100 may include an RF (Radio Frequency) circuit 1110, a memory 1120 including one or more computer-readable storage media, an input unit 1130, a display unit 1140, a sensor 1150, an audio circuit 1160, a short-range wireless transmission module 1170, a processor 1180 including one or more processing cores, and a power supply 1190. It will be appreciated by those skilled in the art that the device structure shown in fig. 11 is not limiting of the terminal device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.

The RF circuit 1110 is used to receive and transmit signals during a message or call, and in particular, to receive downlink information from a base station, process it by one or more processors 1180, and transmit data related to uplink to the base station. Typically, RF circuitry 1110 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, an LNA (LowNoise Amplifier ), a duplexer, and the like. In addition, RF circuitry 1110 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol including, but not limited to, GSM (Global System ofMobile communication, global system for mobile communications), GPRS (GENERAL PACKET Radio Service), CDMA (Code Division Multiple Access ), WCDMA (WidebandCode DivisionMultiple Access, wideband code division multiple access), LTE (Long Term Evolution ), email, SMS (ShortMessaging Service, short message Service), and the like.

Memory 1120 may be used to store software programs and modules (or units). The processor 1180 executes various functional applications and data processing by running software programs and modules (or units) stored in the memory 1120. The memory 1120 may mainly include a storage program area that may store an operating system, application programs required for at least one function (such as a sound playing function, an image playing function), etc., and a storage data area that may store data (such as audio data, a phonebook), etc., created according to the use of the electronic device 1100. In addition, memory 1120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1120 may also include a memory controller to provide access to the memory 1120 by the processor 1180 and the input unit 1130. While fig. 11 shows RF circuit 1110, it is to be understood that it is not an essential component of electronic device 1100 and may be omitted entirely as desired within the scope of not changing the essence of the invention.

The input unit 1130 may be used to receive input numerical or character information and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to object settings and function control. In particular, the input unit 1130 may include a touch-sensitive surface 1131 and other input devices 1132. The touch-sensitive surface 1131, also referred to as a touch display screen or touch pad, may collect touch operations on or near an object (e.g., the object's operation on the touch-sensitive surface 1131 using any suitable object or accessory such as a finger, stylus, etc.) and actuate the corresponding connection device according to a pre-set program. Alternatively, the touch sensitive surface 1131 may include two portions, a touch detection device and a touch controller. The touch controller receives touch information from the touch detection device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 1180, and can receive and execute instructions sent by the processor 1180. In addition, the touch-sensitive surface 1131 may be implemented using various types of resistive, capacitive, infrared, surface acoustic waves, and the like. In addition to the touch-sensitive surface 1131, the input unit 1130 may also include other input devices 1132. In particular, other input devices 1132 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, mouse, joystick, etc.

The display unit 1140 may be used to display information input by or provided to an object and various graphical object interfaces of the control electronic device 1100, which may be composed of graphics, text, icons, video, and any combination thereof. The display unit 1140 may include a display panel 1141, and optionally, the display panel 1141 may be configured in the form of an LCD (Liquid CRYSTAL DISPLAY), an OLED (Organic Light-emitting diode), or the like. Further, the touch sensitive surface 1131 may be overlaid on the display panel 1141, and upon detection of a touch operation thereon or thereabout by the touch sensitive surface 1131, the touch sensitive surface is passed to the processor 1180 to determine the type of touch event, and the processor 1180 then provides a corresponding visual output on the display panel 1141 in accordance with the type of touch event. Although in FIG. 11 the touch-sensitive surface 1131 and the display panel 1141 are implemented as two separate components for input and output functions, in some embodiments the touch-sensitive surface 1131 may be integrated with the display panel 1141 to implement input and output functions.

The electronic device 1100 may also include at least one sensor 1150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor that may adjust the brightness of the display panel 1141 according to the brightness of ambient light, and a proximity sensor that may turn off the display panel 1141 or the backlight when the electronic device 1100 moves to the ear. As one of the motion sensors, the gravitational acceleration sensor can detect the acceleration in all directions (generally three axes), and can detect the gravity and direction when stationary, and can be used for applications of recognizing the gesture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer gesture calibration), vibration recognition related functions (such as pedometer and knocking), and other sensors such as gyroscopes, barometers, hygrometers, thermometers and infrared sensors, which are also configured in the electronic device 1100, will not be described herein.

Audio circuitry 1160, speaker 1161, and microphone 1162 may provide an audio interface between an object and electronic device 1100. The audio circuit 1160 may transmit the received electrical signals converted from audio data to the speaker 1161 for conversion into sound signals for output by the speaker 1161, and on the other hand, the microphone 1162 may convert the collected sound signals into electrical signals for reception by the audio circuit 1160 for conversion into audio data, and then output the audio data to the processor 1180 for processing, and then to be transmitted to another electronic device via the RF circuit 1110, or output the audio data to the memory 1120 for further processing. Audio circuit 1160 may also include an ear bud jack to provide communication of a peripheral ear bud with electronic device 1100.

The short-range wireless transmission module 1170 may be a WIFI (WIRELESS FIDELITY ) module, a bluetooth module, an infrared module, or the like. The electronic device 1100 can transmit information with wireless transmission modules provided on other devices through the short-range wireless transmission module 1170.

The processor 1180 is a control center of the electronic device 1100, connecting various parts of the overall device using various interfaces and lines, performing various functions of the electronic device 1100 and processing data by running or executing software programs or modules stored in the memory 1120, and invoking data stored in the memory 1120. Optionally, the processor 1180 may include one or more processing cores, and optionally, the processor 1180 may integrate an application processor and a modem processor, wherein the application processor primarily processes operating systems, object interfaces, application programs, and the like, and the modem processor primarily processes wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 1180.

The electronic device 1100 also includes a power supply 1190 (e.g., a battery) for powering the various components, and optionally, the power supply 1190 can be logically connected to the processor 1180 via a power management system that can perform functions such as managing charge, discharge, and power consumption. The power supply 1190 may also include one or more of any components, such as a direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

Although not shown, the electronic device 1100 may further include a camera, a bluetooth module, etc., which will not be described herein.

The embodiment of the application also discloses a computer readable storage medium, in which a processor executable program is stored, which when executed by a processor is used to implement the transmission method embodiment of the multimedia resource as shown in fig. 3.

It can be understood that the content in the embodiment of the method for transmitting a multimedia resource shown in fig. 3 is applicable to the embodiment of the computer readable storage medium, and the functions specifically implemented by the embodiment of the computer readable storage medium are the same as those in the embodiment of the method for transmitting a multimedia resource shown in fig. 3, and the beneficial effects achieved by the embodiment of the method for transmitting a multimedia resource shown in fig. 3 are the same as those achieved by the embodiment of the method for transmitting a multimedia resource shown in fig. 3.

The embodiment of the application also discloses a computer program product or a computer program, which comprises computer instructions, wherein the computer instructions are stored in the computer readable storage medium, and the processor of the electronic device shown in fig. 11 can read the computer instructions from the computer readable storage medium, and execute the computer instructions by the processor, so that the computer device executes the embodiment of the method for transmitting the multimedia resource shown in fig. 3.

It can be understood that the content in the embodiment of the method for transmitting a multimedia resource shown in fig. 3 is applicable to the present computer program product or the embodiment of the computer program, and the functions implemented by the present computer program product or the embodiment of the computer program are the same as those of the embodiment of the method for transmitting a multimedia resource shown in fig. 3, and the advantages achieved are the same as those achieved by the embodiment of the method for transmitting a multimedia resource shown in fig. 3.

In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present application are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.

Furthermore, while the application is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the functions and/or features may be integrated in a single physical device and/or software module or may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present application. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be apparent to those skilled in the art from consideration of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the application as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the application, which is to be defined in the appended claims and their full scope of equivalents.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method of the embodiments of the present application. The storage medium includes a U disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, an optical disk, or other various media capable of storing program codes.

Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of techniques known in the art, discrete logic circuits with logic gates for implementing logic functions on data signals, application specific integrated circuits with appropriate combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

In the foregoing description of the present specification, reference has been made to the terms "one embodiment/example", "another embodiment/example", "certain embodiments/examples", and the like, means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the spirit and scope of the application as defined by the appended claims and their equivalents.

While the preferred embodiment of the present application has been described in detail, the present application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and the equivalent modifications or substitutions are intended to be included in the scope of the present application as defined in the appended claims.

Claims

1. A method for transmitting multimedia resources, characterized in that the method comprises:

Detect network quality information between the client and the client;

receiving first experience quality information fed back by the client at a current time point;

Determining, based on the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource; wherein each of the resource backups has a different bit rate, and the first resource set includes a predetermined number of resource backups;

Determining a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information, and generating target multimedia resource data based on the target resource backup; wherein the historical experience quality information is experience quality information fed back by the client before the current time node;

The target multimedia resource data is sent to the client.

2. The method for transmitting multimedia resources according to claim 1, wherein the network quality information includes available network bandwidth; and determining the first resource set from a plurality of resource backups corresponding to the multimedia resource based on the network quality information comprises:

sorting the resource backups according to their bit rates;

Compare the available network bandwidth and the sizes of each of the bit rates, and determine a predetermined number of consecutively ordered resource backups as a first resource set based on the comparison result; wherein, the first resource set includes resource backups whose corresponding bit rates are less than the available network bandwidth, and the first resource set includes resource backups whose corresponding bit rates are greater than the available network bandwidth.

3. The multimedia resource transmission method according to claim 2, wherein the comparing the available network bandwidth and the respective bit rates and determining a predetermined number of consecutively ordered resource backups as a first resource set based on the comparison result comprises:

Record the sorting numbers corresponding to each resource backup in sorted order;

Comparing the available network bandwidth with the respective bit rates, and determining a first bit rate from the bit rates of the resource backups; wherein the first bit rate is a bit rate that is closest to the available network bandwidth among the bit rates that are less than or equal to the available network bandwidth;

Querying a first sorting number corresponding to a first resource backup; wherein the first resource backup is a resource backup corresponding to the first bit rate;

Dividing the difference obtained by subtracting 1 from the predetermined number by 2, and rounding the quotient to obtain a first value;

Determining a second sort number according to a difference between the first sort number and the first value;

Starting from the second sorting number, in the sorting direction of increasing bit rate, the predetermined number of resource backups are selected in sequence to obtain a first resource set.

4. The method for transmitting multimedia resources according to claim 1, wherein the first quality of experience information includes first jamming indicator data; and determining a target resource backup from the first resource set based on a change in the first quality of experience information relative to historical quality of experience information comprises:

Determining whether the jamming condition of the client has deteriorated according to the first jamming indicator data and the historical quality of experience information;

If the jamming situation of the client worsens, a target resource backup is determined from the resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth.

5. The method for transmitting multimedia resources according to claim 4, wherein determining whether the jamming condition of the client has deteriorated based on the first jamming indicator data and the historical quality of experience information comprises:

Determining second experience quality information from the historical experience quality information; the second experience quality information is experience quality information fed back by the client at a time node previous to the current time node, and the second experience quality information includes second jamming indicator data;

Calculating a first difference value between the first jamming indicator data and the second jamming indicator data;

comparing the first difference value with a first threshold;

If the first difference value is less than or equal to the first threshold, it is determined that the client's jamming situation has not worsened; or, if the first difference value is greater than the first threshold, it is determined that the client's jamming situation has worsened.

6. The method for transmitting multimedia resources according to claim 4, wherein determining whether the jamming condition of the client has deteriorated based on the first jamming indicator data and the historical quality of experience information comprises:

Determine, based on the historical quality of experience information, a first cumulative value corresponding to the jamming indicator data before the current time node;

Performing a weighted summation on the first jamming indicator data and the first cumulative value to obtain a second cumulative value;

calculating a second difference value of the second cumulative value relative to the first cumulative value;

Comparing the second difference value with the second threshold;

If the second difference value is less than or equal to the second threshold, it is determined that the client's jamming situation has not worsened; or, if the second difference value is greater than the second threshold, it is determined that the client's jamming situation has worsened.

7. The method for transmitting multimedia resources according to claim 5, wherein if the client's lag worsens, determining a target resource backup from resource backups in the first resource set whose corresponding bitrate is less than the network's available bandwidth comprises:

If the jamming condition of the client worsens, determining a jamming deterioration level corresponding to the client according to the first difference value and the first threshold;

According to the jamming deterioration level, a target resource backup is determined from resource backups in the first resource set whose corresponding bit rates are less than the available network bandwidth.

8. The method for transmitting multimedia resources according to claim 1, wherein the first quality of experience information includes first jamming indicator data and first cache indicator data; and determining a target resource backup from the first resource set based on a change in the first quality of experience information relative to historical quality of experience information comprises:

determining, based on the first cache indicator data and the historical quality of experience information, whether a cache situation of the client has deteriorated;

If the client's lag situation does not worsen and the client's cache situation worsens, determine that the second resource backup in the first resource set is the target resource backup; wherein, the second resource backup is a resource backup corresponding to a second bit rate, and the second bit rate is greater than or equal to the bit rate used by the client at the previous time node of the current time node.

9. The method for transmitting multimedia resources according to claim 8, wherein generating target multimedia resource data based on the target resource backup comprises:

If the jamming situation of the client does not worsen, and the cache situation of the client worsens, frame loss processing is performed on the target resource backup to generate target multimedia resource data.

10. The method for transmitting multimedia resources according to claim 9, wherein the step of performing frame loss processing on the target resource backup to generate target multimedia resource data comprises:

Detecting the frame type of each resource frame in the target resource backup;

Determining, according to the frame type, a target frame to be discarded in the target resource backup;

Detecting a playback time interval between two resource frames adjacent to the target frame;

If the playback time interval is less than or equal to a third threshold, the target frame is discarded; or if the playback time interval is greater than the third threshold, the target frame is retained.

11. The method for transmitting multimedia resources according to claim 9, wherein the step of performing frame loss processing on the target resource backup to generate target multimedia resource data comprises:

Detecting the frame type of each resource frame in the target resource backup;

Determine the first consecutive target frames as a frame set;

A target frame in the frame set is discarded, and other target frames in the frame set except the discarded target frame are retained.

12. The method for transmitting multimedia resources according to claim 1, wherein the first quality of experience information includes first jamming indicator data and first cache indicator data; and determining a target resource backup from the first resource set based on a change in the first quality of experience information relative to historical quality of experience information comprises:

If the client's freezing condition does not worsen and the client's cache condition does not worsen, a target resource backup is determined from resource backups in the first resource set whose corresponding bit rates are greater than the available network bandwidth.

13. A multimedia resource transmission device, characterized in that the device comprises:

A detection unit, used to detect network quality information between the client and the network;

A receiving unit, configured to receive first experience quality information fed back by the client at a current time node;

a processing unit, configured to determine, based on the network quality information, a first resource set from a plurality of resource backups corresponding to the multimedia resource; wherein each of the resource backups has a different bit rate, and the first resource set includes a predetermined number of resource backups;

a generating unit, configured to determine a target resource backup from the first resource set based on a change in the first experience quality information relative to historical experience quality information, and generate target multimedia resource data based on the target resource backup; wherein the historical experience quality information is experience quality information fed back by the client before the current time node;

A sending unit is used to send the target multimedia resource data to the client.

14. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the multimedia resource transmission method according to any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method for transmitting multimedia resources according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, it implements the multimedia resource transmission method according to any one of claims 1 to 12.