CN105898384B

CN105898384B - A kind of method and apparatus of streaming media video mixing frame per second control

Info

Publication number: CN105898384B
Application number: CN201610270343.9A
Authority: CN
Inventors: 叶荣华; 刘志聪; 孙石平; 张冲
Original assignee: Guangzhou Ncast Electronic Science & Technology Co Ltd
Current assignee: Guangzhou Ncast Electronic Science & Technology Co Ltd
Priority date: 2016-04-26
Filing date: 2016-04-26
Publication date: 2019-03-22
Anticipated expiration: 2036-04-26
Also published as: CN105898384A

Abstract

The embodiment of the invention provides a kind of method and devices of streaming media video mixing frame per second control, one or more streaming media video is loaded into respectively in corresponding one or more buffer areas, the streaming media video includes video frame, and the method comprises determining that the corresponding weight of one or more described streaming media video；Respectively from one or more of buffer areas, one or more corresponding streaming media video is extracted frame by frame and is mixed；When there are buffer empty, the weights sum that statistics is stored with all streaming media videos in the buffer area of streaming media video is the first total weight value；Judge whether first total weight value is less than preset threshold value；If so, carrying out dilatation for the buffer area for being stored with streaming media video；If it is not, then carrying out interleave in the buffer empty.The present invention makes user experience compared to any method is used alone more preferably to find equalization point in fluency and real-time both demands.

Description

Method and device for controlling streaming media video mixing frame rate

Technical Field

The present invention relates to the field of video processing technologies, and in particular, to a method and a device for controlling a mixed frame rate of a streaming media and a video.

Background

With the increasingly compact convergence of the media market and the network service, the streaming media is spread and used more and more widely, wherein in the systems such as an MCU (Multipoint Control Unit) of a video conference system, a splitter of a security monitoring system, and the like, the streaming media video mixing technology is beginning to replace the original analog video mixer to be applied in a large amount.

Taking the MCU of the video conference system as an example, a video mixer (video mixer) in the MCU obtains streaming media videos of a plurality of terminals to mix into the same video, and outputs the mixed video at a fixed frame rate to perform subsequent operations such as encoding and transmission, so that bandwidth and terminal performance are saved through a secondary compression mode of decoding, mixing and encoding in practice. The prior decollator in the security monitoring system mainly acts on the processing of analog signals, and along with the networking of the security monitoring system, the processing mode of the decollator is the same as that of a video conference system, and the streaming media video is subjected to a secondary compression mode.

A video mixer based on a network data source in a video conference system and a security monitoring system is also called as a streaming media video mixer. In the streaming video mixer, due to network jitter, frame rate asymmetry (frame rate of each input streaming video is not consistent with that of the mixed output video), and the like, the streaming video mixer must take frame rate control measures so that the mixer outputs the mixed video with a stable frame rate for subsequent encoding.

There are two main frame rate control methods, one of which is fluency-oriented and takes precedence, and this method ensures fluency of video by setting a larger buffer, but causes a larger delay. Another method is to optimize for real-time performance, and the method is to reduce the buffer area as much as possible to improve the real-time performance, but the fluency of the video is poor.

In a streaming video mixer, fluency and real-time performance are mutually restricted and contradictory, so a technical problem to be solved urgently by those skilled in the art is: a streaming media video mixing frame rate control mechanism is provided, which finds a balance point between the requirements of smoothness and real-time performance and enables the user experience to be better than that of either method alone.

Disclosure of Invention

The technical problem to be solved by the embodiments of the present invention is to provide a method for controlling a frame rate of a mixed streaming media and video, so as to find a balance point between two requirements of fluency and real-time performance, and make a user experience better than that of using either method alone.

Correspondingly, the embodiment of the invention also provides a device for controlling the frame rate of the streaming media and video mixture, which is used for ensuring the realization and the application of the method.

In order to solve the above problems, the present invention discloses a method for controlling a frame rate of a mixed streaming media and video, wherein one or more paths of streaming media and video are loaded into one or more corresponding buffer areas respectively, the streaming media and video comprise video frames, and the method comprises:

determining the weight corresponding to the one or more paths of streaming media videos;

respectively extracting one or more corresponding streaming media videos from the one or more buffer areas frame by frame for mixing;

when an empty buffer area exists, counting the sum of the weights of all the streaming media videos in the buffer area in which the streaming media videos are stored as a first total weight;

judging whether the first total weight value is smaller than a preset threshold value or not; if yes, expanding the capacity of the buffer area in which the streaming media video is stored; and if not, performing frame interpolation in the empty buffer area.

Preferably, the step of determining the weight corresponding to the one or more paths of streaming media videos includes:

reading layout parameters in a preset configuration file; the layout parameters comprise the ratio of the pixels of the one or more paths of streaming media videos in the preset total pixels;

and taking the ratio as the weight of the one-way or multi-way streaming media video respectively.

receiving additional parameters; the additional parameters comprise new weight values of the one or more paths of streaming media videos;

and taking the new weight as the weight of the one-way or multi-way streaming media video.

Preferably, after the step of determining the weight corresponding to the one or more paths of streaming media videos, the method further includes:

calculating the sum of the weights of the one or more paths of streaming media videos to obtain a second total weight;

judging whether the second total weight is not equal to a preset value or not;

if yes, calculating the product of the weight of the one or more paths of streaming media videos and the ratio of a preset numerical value to a second total weight, and taking the product as the weight of the one or more paths of streaming media videos.

Preferably, the step of inserting frames in the empty buffer includes:

copying a video frame which is mixed before the streaming media video corresponding to the empty buffer area;

inserting the video frame into the empty buffer.

Preferably, the preset threshold is 0.7.

The embodiment of the invention also discloses a device for controlling the mixed frame rate of the streaming media and the video, wherein one or more paths of streaming media videos are respectively loaded into one or more corresponding buffer areas, the streaming media videos comprise video frames, and the device comprises:

the determining module is used for determining the weight corresponding to the one-way or multi-way streaming media video;

the mixing module is used for extracting one or more corresponding streaming media videos from the one or more buffer areas frame by frame to mix;

the counting module is used for counting the sum of the weights of all the streaming media videos in the buffer area in which the streaming media videos are stored as a first total weight when the empty buffer area exists;

the judging module is used for judging whether the first total weight value is smaller than a preset threshold value or not; if yes, calling a capacity expansion module, and if not, calling an interpolation module;

the capacity expansion module is used for expanding the capacity of the buffer area in which the streaming media video is stored;

and the frame interpolation module is used for performing frame interpolation in the empty buffer area.

Preferably, the determining module comprises:

the reading submodule is used for reading the layout parameters in the preset configuration file; the layout parameters comprise the ratio of the pixels of the one or more paths of streaming media videos in the preset total pixels;

and the first adjusting submodule is used for respectively taking the ratio as the weight of the one-way or multi-way streaming media video.

Preferably, the determining module comprises:

a receiving submodule for receiving additional parameters; the additional parameters comprise new weight values of the one or more paths of streaming media videos;

and the second adjusting submodule is used for taking the new weight value as the weight value of the one-way or multi-way streaming media video.

Preferably, the apparatus further comprises:

the calculation module is used for calculating the sum of the weights of the one or more paths of streaming media videos to obtain a second total weight;

the judging module is used for judging whether the second total weight is not equal to a preset value or not; if yes, calling a third adjusting module;

and the third adjusting module is used for calculating the product of the weight of the one or more paths of streaming media videos and the ratio of a preset numerical value to the second total weight as the weight of the one or more paths of streaming media videos.

Compared with the prior art, the embodiment of the invention has the following advantages:

in the embodiment of the invention, the weight is calculated according to the conditions of video content, layout parameters, user requirements and the like, when the buffer is empty due to network jitter, the calculation result of the weight is used as the basis for frame skipping or buffer increase, so that the size of the buffer is dynamically controlled, and the adaptive process enables the buffer to be set to be a reasonable size finally. Because the smoothness of all videos is not required to be guaranteed, the smoothness and the real-time performance of the videos can be guaranteed under the conditions of less required buffer areas and shorter delay time, and the user experience effect is improved.

Drawings

FIG. 1 is a basic workflow of a streaming video mixer;

FIG. 2 is a schematic diagram of a video blending layout a;

FIG. 3 is a schematic diagram of a video mixing layout b;

fig. 4 is a flowchart illustrating steps of an embodiment 1 of a method for controlling a frame rate of a streaming video mixture according to the present invention;

fig. 5 is a flowchart illustrating the steps of an embodiment 2 of a method for controlling a frame rate of a streaming video mixture according to the present invention;

FIG. 6 is a schematic diagram of a video blending layout c;

FIG. 7 is a schematic illustration of a queuing theory model of the present invention;

FIG. 8 is a schematic diagram of a buffer and a probability of meeting a requirement of a corresponding frame rate control method according to the present invention;

FIG. 9 is a general diagram of a buffer and a probability of satisfying a requirement of a corresponding frame rate control method according to the present invention.

Fig. 10 is a block diagram of an embodiment of a frame rate control apparatus for mixing streaming media and video according to the present invention.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.

Taking the MCU of the video conference system as an example, refer to a basic workflow of a streaming media video mixer shown in fig. 1, and the specific flow is as follows:

in a video conference system, streaming media videos of terminals arrive at a Multipoint Control Unit (MCU) from different network links, pass through a buffer area, and are controlled at a Frame rate (Frame rate control)₎Under the action, the streaming media videos are respectively decoded by a decoder according to a certain time interval, the decoded images are flushed into a streaming media video mixer to be mixed to generate new images, and the new images are encoded and packaged to form new video streams which are distributed to terminals.

In a specific implementation, when the input frame rate and the output frame rate of the streaming video mixer are not equal, or the input arrival is unstable due to network jitter caused by noise existing in the network or performance problems of the switching device, frame rate control needs to be adopted. For example, when the input frame rate is 20 (frames/second) and the output frame rate is 30 (frames/second), theoretically, 1 frame data needs to be inserted (copied) and mixed every 2 frames, but considering that the arrival of the streaming video is unstable due to network jitter, the frame insertion may be performed randomly according to the situation of the streaming video in the buffer, and even frame skipping (dropping) operation may be performed to ensure synchronization of the streaming videos due to serious delay of the arrival of the streaming video due to excessive network jitter.

If the frame rate control method with the optimal fluency is adopted, when the network jitter is required, any frame is not skipped as far as possible so as to guarantee the fluency of the video, so that the method needs a larger buffer area, and the delay is longer. If a frame rate control method with optimal real-time performance is adopted, the buffer area is required to be reduced as much as possible, the real-time performance is improved, and because the smaller buffer area cannot cope with interference caused by network jitter, the method can cause a large number of frame skipping, so that the video smoothness is poor.

In a commonly adopted frame rate control method, each path of streaming media video is treated equally, and a system needs to ensure the fluency of all videos, so that the system needs a buffer area which is much larger than that for processing a single video. On the contrary, if a smaller buffer is set, due to the equal treatment, the chances of frame insertion or frame loss of each path of streaming media video due to network jitter are also the same.

When the streaming media video mixer performs video mixing, the streaming media videos are scaled and cut, and are arranged according to a certain layout, and in a specific implementation, a video area most concerned by a user must exist in a finally output video, that is, a certain path of input video of the streaming media mixer is most concerned by the user. Referring to a schematic diagram of a video mixing layout a shown in fig. 2 and a schematic diagram of a video mixing layout b shown in fig. 3, in the two video mixing layout diagrams of fig. 2 and 3, a video numbered 1 is the content most focused by the user, and the rest 234 is the next. If the fluency and the real-time performance of the video of the label 1 are preferentially ensured, the system can reduce the delay and ensure the fluency as much as possible under the condition of ensuring the user experience.

It is based on the above circumstances that the present inventor has creatively proposed one of the core concepts of the embodiments of the present invention that the proposed frame rate control mechanism with priority to the attention content weights each path of streaming video, and uses the sum of the weighted values as the basis for frame skipping or buffer increase, so as to dynamically control the size of the buffer, and this adaptive process will make the buffer finally set to a reasonable size. Compared to "fluency optimized", a smaller buffer is required since there is no requirement to guarantee fluency for the entire video. Compared with the optimal real-time performance, the fluency of the main body is ensured. The method achieves the goal of optimal user experience, and finds an optimal balance point between instantaneity and fluency.

Referring to fig. 4, a flowchart illustrating steps of embodiment 1 of a method for controlling a frame rate of a streaming video mixture according to the present invention is shown, where one or more streaming videos may be loaded into one or more corresponding buffers, respectively, where the streaming videos may include video frames, and the method specifically includes the following steps:

step 101, determining a weight corresponding to the one or more paths of streaming media videos;

in specific implementation, a streaming media video is accessed to a system through a Real Time Streaming Protocol (RTSP) protocol and the like, and is unpacked to obtain compressed video data. The format of the compressed video data may be h264, h263, wmv, mpeg2, vp8, jpeg, and so on.

When the streaming media video mixer mixes the videos, the videos are zoomed, cut and arranged according to a certain layout, and a video area which is most concerned by a user exists in the finally output video, namely, a certain path of input video of the mixer is most concerned by the user. In order to ensure the user experience effect, different weights can be set for each path of streaming media video according to the attention degree of the user.

102, respectively extracting one or more corresponding streaming media videos from the one or more buffer areas frame by frame for mixing;

in a specific implementation, the streaming media mixer obtains each path of streaming media video from the buffer area by taking a frame as a unit, and performs scaling, arrangement and mixing to obtain video data according to a specified layout. The mixed video data may be YUV (Luma and chroma, a color coding method) data, among others. At present, the main video is based on YUV, and YUV can be further subdivided into multiple formats such as I420, YUY2, NV12 and the like. Of course, the hybrid video may also be data based on other formats such as RGB or RGBA.

It should be noted that when receiving an instruction from a user or another device (e.g., an intelligent tracking system), making a layout switch, notifying the streaming video mixer to change the scaling and arrangement, and recalculating the weights, i.e., re-executing step 101.

103, when an empty buffer area exists, counting the sum of the weights of all the streaming media videos in the buffer area in which the streaming media videos are stored as a first total weight;

in practical application, when a buffer area is empty due to factors such as network jitter, that is, when a streaming media video mixer cannot currently obtain a certain path or some paths of streaming media videos, policy control is started at this time, a calculation result is obtained through calculation by combining weights of the paths of streaming media videos, and a processing mode of the streaming media video mixer is determined by using the calculation result, for example, whether the size of the buffer area is adjusted or not is determined.

Step 104, judging whether the first total weight value is smaller than a preset threshold value; if yes, executing step 105, otherwise, executing step 106;

105, performing capacity expansion on the buffer area in which the streaming media video is stored;

and 106, performing frame interpolation in the empty buffer area.

In the embodiment of the invention, if the empty buffer area exists, the sum of the weight values of all the streaming media videos in the buffer area in which the streaming media videos are stored is counted to calculate the total weight value, and whether the total weight value is smaller than a preset threshold value is judged. If the current video fluency is smaller than the preset threshold value, the fluency of the current video does not meet the expected value, and the buffer area for storing the streaming media video is increased; if the current video smoothness is greater than the preset threshold value, the current video smoothness meets the expected value, and in order to ensure that each path of streaming media video has at least one frame of data when the streaming media video mixer works, frame insertion operation is performed in an empty buffer area.

In order to further understand the embodiments of the present invention, the following description uses specific examples. Specifically, in a preferred example of a specific application of the present invention, the frame rate method based on attention content priority may include the following modules:

the input source management module: and the device is responsible for connecting an input source through various real-time streaming media protocols, and decoding after acquiring streaming media video data.

A buffer module: for buffering the images of the decoded streaming video.

A frame rate control module: the frame rate in the mixing process is controlled.

A weight calculation module: and setting the weight of each path of streaming media video according to a preset rule.

A policy control module: and judging the conditions such as network jitter and the like, and determining the processing mode of the streaming media mixer for each path of streaming media video.

A mixer module: and mixing the images of the multi-path decoded streaming media video into one image according to a certain layout.

The directing module: and receiving a switching instruction generated by a user or other equipment, and controlling the streaming media video mixer to change the layout.

The coding module: and compressing the video coding output by the streaming media video mixer.

Live broadcast recording module: and broadcasting the compressed video live or recording the compressed video to a hard disk.

It should be noted that, since the live broadcast and the recording may not be performed at all, the display may be performed only by the display device, and such an application that is only used for the display is called a frame divider. The buffer may be provided before decoding or after decoding.

Referring to fig. 5, a flowchart illustrating steps of embodiment 2 of a method for controlling a frame rate of a streaming video mixture according to the present invention is shown, where one or more streaming videos may be loaded into one or more corresponding buffers, respectively, where the streaming videos may include video frames, and the method may specifically include the following steps:

step 201, reading layout parameters in a preset configuration file; the layout parameters comprise the ratio of the pixels of the one or more paths of streaming media videos in the preset total pixels;

in a specific implementation, the layout parameters may be pre-stored in a configuration file of the streaming video mixer. When the configuration file is loaded, zooming and arranging are carried out according to the layout parameters, and video data are mixed.

Step 202, the ratio is respectively used as the weight of the one-way or multi-way streaming media video;

in the embodiment of the present invention, the weight of each path of streaming media video may be according to the proportion of each path of video to the pixels. For example, when the streaming media video mixer processes the mixing task of N streaming media videos this time, the total pixels output after mixing is S (resolution is long × resolution is high), and if a certain streaming media video occupies pixels in the layout as S_nThen, the weight C of the streaming media video in frame rate control_n＝S_n/S。

Taking the video mixing layout of fig. 1 as an example, assuming that the pixels of the input source are 1280 × 720, the output resolution of the streaming video mixer can be set to 1706 × 720. Video 1 does not need scaling, its occupied pixel is 1280 × 720, and videos 2, 3, 4 are scaled 426 × 240, and the weights of videos 1, 2, 3, 4 are 0.75, 0.083, respectively.

In a preferred embodiment of the present invention, the weight value may also be obtained by:

step S11, receiving additional parameters; the additional parameters comprise new weight values of the one or more paths of streaming media videos;

and step S12, using the new weight as the weight of the one or more paths of streaming media videos.

In practical applications, for example, in some specific situations, a user may want to manually specify weights, for example, in the video mixing layout of fig. 1, the pixels occupied by the videos 1, 2, and 3 are equal, but the user pays more attention to the video 1, and in order to facilitate the user configuration, the user only needs to set the video 1 as a "main picture", set a higher weight (for example, 0.6) for the video 1, and allocate the remaining weights according to the pixel proportion.

It should be noted that the weight is calculated according to conditions such as video content, layout parameters, user requirements, and the like, or calculated according to a comprehensive situation of some of the above conditions, so as to obtain the weight. For example, in a specific implementation, the calculation may be performed according to a dynamic degree in the video (a higher weight is given to a dynamic state of the video content), or according to a position of the video in the layout (different positions are given different weights), which is not limited in this embodiment of the present invention.

In a preferred embodiment of the present invention, the method may further include:

step S21, judging whether the second total weight is not equal to a preset value; if yes, go to step S22;

step S22, calculating a product of the weight of the one or more paths of streaming media videos and a ratio of a preset value to a second total weight as the weight of the one or more paths of streaming media videos.

In the embodiment of the invention, after the weight of each path of streaming media video is distributed once, verification is required to be carried out.

Specifically, the process of checking the weights may be to determine whether the sum of the weights is equal to a preset value, and if not, to adjust the weights, so that the sum of the weights is finally equal to the preset value, and if necessary, to perform scaling.

The pseudo code for the verification process is as follows:

the specific process of the pseudo code is as follows:

1. and calculating the sum of the weights of all paths of streaming media videos, and setting a preset numerical value m to be 1.

It should be noted that the preset value can be any reasonable value. For example, in the video mixing layout of fig. 2, if all the output images are allocated by the streaming media videos, the preset value m may be set to be 1, and in the video mixing layout of fig. 3, if the output images are overlapped by the videos 1 and 2, the preset value m may be set to be a value greater than 1, which is not limited in the embodiment of the present invention.

2. And judging whether the sum of the pixels occupied by each path of streaming media video in the layout is not equal to the total pixels output by the streaming media video mixer (namely, the sum of the weights sum is not equal to a preset value m), for example, the sum is smaller than or larger than the total pixels output by the streaming media video mixer. Wherein reference is made in particular to a schematic diagram of a video blending layout c shown in fig. 6 for the case of less than total pixels being output.

3. If the sum of the weights sum is not equal to the preset value m, each weight is multiplied by m/sum, for example, if m is 1, each weight may be multiplied by 1/sum, so that the sum of the weights sum of the final streaming media videos sum is m.

Of course, other ways may also be adopted to check the weights, so that the sum of the weights is finally equal to the preset value, which is not limited in the embodiment of the present invention.

Step 203, extracting one or more corresponding streaming media videos from the one or more buffer areas frame by frame respectively and mixing the videos;

step 204, when an empty buffer area exists, counting the sum of the weights of all the streaming media videos in the buffer area where the streaming media videos are stored as a first total weight;

step 205, judging whether the first total weight value is smaller than a preset threshold value; if yes, go to step 206, otherwise go to step 207;

step 206, performing capacity expansion on the buffer area in which the streaming media video is stored;

and step 207, performing frame interpolation in the empty buffer area.

In a preferred embodiment of the present invention, the step 207 may include the following sub-steps:

substep S31, copying a video frame mixed before the streaming media video corresponding to the empty buffer area;

sub-step S32 inserts the video frame into the empty buffer.

In a particular implementation, the buffer in a particular way or ways may be empty due to network jitter. At this time, it is determined whether the buffer is empty due to single network jitter or multiple network jitter. If the current path of streaming media video data is caused by single network jitter, temporarily considering that the path of streaming media video data is offline, and deleting the streaming media video from the streaming media video mixer; if the network jitter is caused by multiple network jitter, the network jitter of the streaming media video is considered to be large, and whether the system buffer needs to be increased or not needs to be determined.

The decision process pseudo-code is as follows:

the specific process of the pseudo code is as follows:

1. accumulating the weight of each path of streaming media video with non-empty buffer area to obtain a total weight C_sumWherein

2. If the total weight is less than K, i.e. C_sumIf the current video mixing effect is less than K, the current video mixing effect is considered to not meet the expected value after weighted calculation, the buffer area should be increased, and the fluency is improved.

Where K (a value range (0, m), for example, 0.7, where the higher the value of K, the higher the fluency, and when K is 0, the higher the fluency is, the same as the real-time priority effect, and when K is m, the same as the fluency priority effect).

3. If the total weight is greater than K, then C_sumAnd if the current video mixing effect is not less than K, considering the expected value met by the current video mixing effect after weighted calculation, and performing frame insertion processing on the streaming media video with an empty buffer area in order to ensure that each path of streaming media video has at least one frame of data when the streaming media video mixer works, for example, video frame data of the previous frame for streaming media video mixing can be copied and put into the buffer area.

In order to further understand the advantages of the embodiments of the present invention, the following describes in detail a frame rate control method optimized for the content of interest in a comparative manner.

In the embodiment of the present invention, a frame rate control method with priority on the content is proposed on the basis of make internal disorder or usurp research on two frame rate control methods mainly used at present. And modeling the three methods by using a relevant theory of the queuing theory, and comparing the advantages and disadvantages of various models under different requirements.

Establishing a queuing theory model:

since real-time streaming media data are output at a constant rate at a source end and do not arrive at random, a queuing theory model cannot be directly applied, but a process that a data packet caused by network jitter due to noise or interference signals on a network occupies a channel can be considered to be randomly arriving and obey Poisson distribution. While the input content stream within the same hypothetical channel can be considered to have a linear dependence on the arrival of the interfering signal. The frame rate control portion within the dashed line in fig. 1 is converted into a schematic diagram of a queuing theory model of the present invention as shown in fig. 7.

If the input content stream is used as the discussion object, the probability that the queue length of each queue (i.e., buffer) is 0 and the influence thereof on the system need to be calculated. However, the model needs to be converted into a model using noise flow as a discussion object, and the converted form is unchanged from that shown in fig. 7, but the input object is noise flow, and the probability that each queue becomes full of noise and the influence thereof on the system are calculated.

If the link bandwidth is w and the input content bit rate is b, the service rate for noise is μ ═ w-b, and noise exceeding this rate may cause the content stream to fail to reach the buffer, which may be considered as noise occupying the buffer. To simplify the demonstration process, it is assumed in this example that the bit rate of each input content stream is the same and the jitter strength and bandwidth of each link are the same. Since the jitter intensity is the same, namely the arrival rate of each path of noise lambda₁＝λ₂...＝λ_NIf λ is equal, and if the bit rate and bandwidth of each input content are the same, then μ₁＝μ₂...＝μ_Nμ. The maximum queue length is set to be N, and the number of queues is set to be K. Since the arrival conforms to the Poisson distribution, the service is fixed-length distribution, and the queue (buffer) is limited, the model is actually K M/D/1/N/∞ mixed models which are connected in parallel and then in series. StringThe rules of the combined services are different according to different requirements of the three frame rate control methods.

1. Frame rate control method for fluency optimization

The frame rate control method with the optimal fluency requires that each video picture is as smooth as possible without skipping any data. I.e. it is required that the content stream of each buffer cannot be empty, which is equivalent to each buffer not being filled with noise streams. That is, if any queue in the system is occupied by noise, the fluency optimization requirement is violated.

As can be seen from the M/D/1/N/∞ model calculation formula, the probability of the queue reaching the longest limit is as follows:

P_N＝ρ^NP₀

wherein

Then

When the optimal demand of fluency is met, the probability that none of the K queues reaches the upper limit is as follows:

assuming that the bandwidth w is 10(100kb/s), the input content bit rate b is 8(100kb/s), and the noise arrival rate λ is 1(100kb/s), the service rate μ is 2(100kb/s), and if the queue upper limit N is 5(100kb), K is 9 queues, the calculated probability of satisfying the requirement is:

P＝0.865

that is, when the buffer is 500kb, there is a 86.5% probability that the fluency requirement is met during the operation of the system.

2. Frame rate control method for optimal real-time performance

The frame rate control method with optimal real-time performance requires high application, and reduces buffering or even no buffering as much as possible within an acceptable effect range to meet the requirement of minimum delay. Similarly, the probability of the queue reaching the upper limit is known as:

if the frame skipping is required to be not more than 30%, that is, the queue reaching the upper limit at the same time is required to be not more than K × 30%.

When the real-time optimal requirement is met, the probability that at least K-K queues are not fully occupied by noise is as follows:

under the same conditions as above, the following were calculated:

P＝0.88

that is, when the buffer is 500kb, there is 88% probability that the real-time requirement will be satisfied during the operation of the system.

3. Frame rate control method for attention content optimization

In a specific implementation, the video is generally scaled and arranged according to the user's requirement when being mixed, such as the mixed form shown in fig. 2 and 3. In fig. 2 and fig. 3, the frame marked by the video with the sequence number 1 is the content most concerned by the user, so that the delay time can be reduced as much as possible while ensuring that the effect of the frame is optimal, and the frame skipping of other non-concerned frames can be ignored within an acceptable range. Such a method is an optimal frame rate control method for the content of interest.

Because only the optimal effect of a certain video is considered, the system only needs to ensure that the probability of being empty of a certain queue in a specified time is lower than a limit value. Then the calculation process can view the system as a single team of system A of M/D/1/N/∞ models, plus a system B of K-1M/D/1/N/∞ models connected in parallel and then in series, and the total buffer value of the system is A, B the maximum value of the two systems.

The a system is first calculated.

For a single team system, the probability that it meets the demand:

under the same conditions as above, the following were calculated:

P_A＝0.984

and calculating the system B. Because the B system weakens the requirement that the fluency emphasizes the real-time performance, the calculation method of the B system accords with the calculation method facing the real-time performance optimization, namely:

under the same conditions as above, the following were calculated:

P_B＝0.894

the probability that the system will meet the demand at the same time is A, B:

P＝P_A*P_B＝0.879

i.e., when the buffer is 500kb, the system runs with a probability of 87.9% to meet the content-of-interest priority requirement.

And comparing and analyzing three frame rate control modes:

the example calculates the probability that each method will satisfy the requirement under the same condition under a specific condition by assuming a method of substituting a condition into a parameter. In order to further analyze the condition of each method under different limiting conditions, the relation between the cache N and the probability P that the cache meets the requirement in each method is drawn through matlab, and specifically, a schematic diagram of the probability that a cache region meets the requirement of a corresponding frame rate control method shown in fig. 8 may be specifically referred to, where a is a fluency priority method, b is a real-time priority method, c is a content-of-interest priority method, and d is a composite image of the three methods. In order to further facilitate comparison of differences of the methods under the same condition, the three methods are drawn to the same image, and specifically, a general schematic diagram of a cache and a probability meeting requirements of a corresponding frame rate control method according to the present invention shown in fig. 9 may be referred to. As can be easily seen from fig. 9(a), under any same condition, the probability that the content-of-interest priority method satisfies the requirement is between the other two frame rate control methods, and it can be inferred that if the satisfying probability is the same, the buffer size required by the method is also between the other two methods, as shown by the dotted line in fig. 9 (b). In summary, the method for prioritizing the content of interest can reduce the required buffering duration to a certain extent without reducing the user experience.

Two common frame rate control methods in the streaming media video mixing technology discussed in this example, and an optimal frame rate control method for the content of interest is proposed on the basis of the two common frame rate control methods. Through modeling, several methods are analyzed and compared by using the relevant theory of queuing theory. Compared with the prior art, the frame rate control method for the attention content optimization can find a proper balance point in two requirements of fluency and instantaneity, and the user experience effect is good.

It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.

Referring to fig. 10, a block diagram of an embodiment of an apparatus for controlling a frame rate of a streaming video mixture according to the present invention is shown, where one or more streaming videos may be loaded into one or more corresponding buffers, respectively, where the streaming video may include video frames, and specifically includes the following modules:

a determining module 301, configured to determine a weight corresponding to the one or more paths of streaming media videos;

in a preferred embodiment of the present invention, the determining module 301 may include:

A mixing module 302, configured to extract one or more corresponding streaming media videos from the one or more buffers frame by frame for mixing;

the counting module 303 is configured to count that the sum of the weights of all the streaming media videos in the buffer where the streaming media videos are stored is a first total weight when there is an empty buffer;

a judging module 304, configured to judge whether the first total weight is smaller than a preset threshold; if yes, calling a capacity expansion module 305, and if no, calling an interpolation module 306;

a capacity expansion module 305, configured to perform capacity expansion on the buffer area in which the streaming media video is stored;

and a frame interpolation module 306, configured to perform frame interpolation in the empty buffer.

In a preferred embodiment of the present invention, the apparatus may further include:

For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.

The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present invention have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the invention.

Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.

The above detailed description is provided for a method and a device for controlling a streaming media video mixing frame rate according to the present invention, and a specific example is applied in the present document to explain the principle and the implementation of the present invention, and the description of the above embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A method for frame rate control of a streaming video mixture, wherein one or more streaming videos are loaded into one or more corresponding buffers, respectively, and the streaming videos include video frames, the method comprising:

judging whether the first total weight value is smaller than a preset threshold value or not; if yes, expanding the capacity of the buffer area in which the streaming media video is stored; if not, performing frame interpolation in the empty buffer area;

the method further comprises the following steps:

judging whether the second total weight is not equal to a preset value or not;

if yes, calculating the product of the weight of the one path of streaming media video and the ratio of a preset numerical value to a second total weight, and taking the product as a new weight of the one path of streaming media video;

or,

and calculating the product of the weight of the multi-path streaming media video and the ratio of a preset numerical value to a second total weight as a new weight of the multi-path streaming media video.

2. The method according to claim 1, wherein the step of determining the weight corresponding to the one or more streaming media videos comprises:

3. The method according to claim 1 or 2, wherein the step of determining the weight corresponding to the one or more streaming media videos comprises:

4. The method of claim 1, wherein the step of inserting frames in the empty buffer comprises:

inserting the video frame into the empty buffer.

5. The method of claim 1, wherein the preset threshold is 0.7.

6. An apparatus for frame rate control of a streaming video mixture, wherein one or more streams of streaming video are loaded into one or more corresponding buffers, respectively, the streams of streaming video comprising video frames, the apparatus comprising:

the frame interpolation module is used for performing frame interpolation in the empty buffer area;

the device further comprises:

a third adjusting module, configured to calculate a product of a weight of the one path of streaming media video and a ratio of a preset value to a second total weight, where the product is used as a new weight of the one path of streaming media video;

or,

and the second total weight value is used for calculating the product of the weight value of the multi-path streaming media video and the ratio of a preset numerical value to the second total weight value, and the product is used as a new weight value of the multi-path streaming media video.

7. The apparatus of claim 6, wherein the determining module comprises:

8. The apparatus of claim 6 or 7, wherein the determining module comprises: