CN105429983B - Acquire method, media termination and the music lesson system of media data - Google Patents

Abstract

The invention discloses a method for collecting media data, a media terminal and a music teaching system. The media terminal includes a video acquisition unit, a video buffer, an audio acquisition unit, an audio buffer, a sending buffer, a transmission unit and a control unit. The video capture unit captures images and encodes them into video frames. Video buffers are suitable for holding video frames. The audio collection unit collects sound and encodes it into audio frames. Audio buffers are suitable for storing audio frames. The sending buffer stores the data frames to be sent. Each data frame is a video frame or an audio frame. The transmission unit is adapted to transmit data frames to be sent to the media server. The control unit is adapted to detect the audio buffer and push the audio frames therein into the sending buffer. When the audio buffer is empty, if the number of data frames to be sent does not exceed the threshold and the video buffer is not empty, the control unit extracts video frames from the video buffer and pushes them to the sending buffer.

Description

Method for collecting media data, media terminal and music teaching system

technical field

The invention relates to the communication field, in particular to a method for collecting media data, a media terminal and a music teaching system.

Background technique

At present, in real-time communication solutions such as video conferencing or webcasting, a terminal collecting media data may collect media data such as video frames and audio frames, and transmit the media data to the playback terminal. For example, video frames and audio frames may be encapsulated together and transmitted through the network. Alternatively, the terminal encapsulates the video frame and the audio frame respectively and transmits them.

However, the existing network conditions are complex and changeable, for example, there are problems such as network jitter and intermittent interruption. When the collection terminal sends audio and video data through the network, there are problems such as network delay and network congestion. Therefore, when the media player acquires audio and video data from the acquisition terminal, it faces the problem of being stuck and not smooth.

Contents of the invention

For this reason, the present invention provides a new solution for collecting media data, which effectively solves at least one of the above problems.

According to one aspect of the present invention, a media terminal is provided, including a video acquisition unit, a video buffer, an audio acquisition unit, an audio buffer, a sending buffer, a transmission unit, and a control unit. The video capture unit is adapted to capture images and encode them into video frames. The video buffer is adapted to store video frames from the video acquisition unit. The audio capture unit is adapted to capture sound and encode it into audio frames. The audio buffer is adapted to store audio frames from the audio capture unit. The send buffer is suitable for storing one or more data frames to be sent. Each data frame to be sent is a video frame from the video buffer or an audio frame from the audio buffer. The transmission unit is adapted to transmit one or more data frames to be transmitted to the media server. The control unit is adapted to detect the audio buffer and push the audio frames therein to the sending buffer, and when the audio buffer is empty, judge whether the number of data frames to be sent in the sending buffer does not exceed a threshold and the video buffer does not Is empty. If the number of data frames to be sent does not exceed the threshold and the video buffer is not empty, the control unit extracts a video frame from the video buffer and pushes it into the sending buffer.

According to still another aspect of the present invention, a method for collecting media data is provided. The method is suitable for execution in a media terminal. The media terminal includes a send buffer. The send buffer is suitable for storing one or more data frames to be sent. Each data frame to be sent is a video frame or an audio frame. The method includes the following steps. Capture images and encode them into video frames, and store the video frames in the video buffer. Collect sound and encode it into an audio frame, and store the audio frame in the audio buffer. Detects an audio buffer and pushes audio frames from it into the send buffer. If the audio buffer is empty, it is judged whether the number of data frames to be sent in the sending buffer does not exceed the threshold and the video buffer is not empty. If the number of data frames to be sent does not exceed the threshold and the video buffer is not empty, a video frame is extracted from the video buffer and pushed to the sending buffer. Transmitting the one or more data frames to be sent to a media server.

According to yet another aspect of the present invention, a music teaching system is provided, including a media terminal, a media server and a media player according to the present invention. The media server is adapted to receive audio frames and video frames sent by the media terminal. The media player is adapted to acquire and play audio frames and video frames from the media server.

According to the media data collection scheme of the present invention, the audio frame in the audio buffer can be pushed to the sending buffer first, and when the audio buffer is empty and the number of data frames in the sending buffer does not exceed the threshold, the video will be buffered The video frames in the region are pushed to the send buffer. In this way, the media data collection solution according to the present invention can always give priority to transmission of audio frames, and transmit video frames in the interval of transmission of audio frames. Especially, when the network bandwidth is low (that is, the number of data frames in the sending buffer exceeds the threshold), the media data acquisition scheme according to the present invention can stop pushing video frames to the sending buffer, and normally send audio frames in the audio buffer. Frames are pushed into the send buffer. Such a working method can effectively reduce the amount of data that the transmission unit needs to send when the network bandwidth is low, thereby effectively ensuring the real-time transmission of audio frames. In other words, according to the media data collection solution of the present invention, the priority transmission of audio frames is realized, thereby avoiding the problem of interruption of the sound played by the media player. It should be noted that in the occasions where the importance of sound is high, such as music teaching, the media data acquisition scheme according to the present invention ensures the real-time transmission of sound data, and the media player can play continuous, undistorted audio data, thereby greatly improving user experience. experience.

Description of drawings

To the accomplishment of the foregoing and related ends, certain illustrative aspects are herein described, taken in conjunction with the following description and drawings, which are indicative of the various ways in which the principles disclosed herein may be practiced, and all aspects and their equivalents are intended to fall within the scope of within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent by reading the following detailed description in conjunction with the accompanying drawings. Like reference numerals generally refer to like parts or elements throughout this disclosure.

Fig. 1 shows a block diagram of an exemplary music teaching system 100 according to the present invention;

FIG. 2 shows a block diagram of a media terminal 200 according to some embodiments of the invention; and

Fig. 3 shows a flowchart of a method 300 for collecting media data according to some embodiments of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

FIG. 1 shows a block diagram of an exemplary music teaching system 100 according to the present invention. As shown in FIG. 1 , the music teaching system 100 may include multiple student clients 110 , a server 120 and a teacher client 130 . In the music teaching system 100, the student client 110 and the teacher client 130 communicate in real time through the server 120 for online music teaching. For example, when a student performs a performance, the student client 110 can be implemented as a media terminal, which collects media data such as video and audio related to the student's performance, and transmits the media data to the teacher client 130 through the server 120 . The teacher client 130 can be implemented as a media player, receiving and playing media data, so that the teacher can understand the performance of the students in real time. At the same time, the teacher client 130 can also be implemented as a media terminal, which collects media data such as the teacher's feedback on the student's performance or teaching demonstration, and transmits it to the student client through the server 120 . The student client 110 can be implemented as a media player, receiving and playing media data from the teacher client 130, so that the teacher can give real-time feedback on the performance of the students, or give real-time teaching demonstrations to the students. In short, both the student client 110 and the teacher client 130 can be implemented as media terminals and media playback terminals. Here, the media data includes, for example, teaching content such as fingering, breath, sound of musical instruments, and instructional texts for playing a musical instrument, but is not limited thereto.

Usually, the music teaching system 100 faces the problem of unstable transmission bandwidth, such as network jitter, intermittent network interruption, and the like. However, high-quality music teaching has high requirements for real-time, synchronization and fluency of media data. The invention proposes a new media terminal aiming at the link of collecting media data in the music teaching system. The media terminal in the music teaching system will be further exemplified below in conjunction with FIG. 2 . Certainly, the media terminal may be a student client or a teacher client. To simplify the description, the specific types of media terminals will not be distinguished below. Likewise, both the student client 110 and the teacher client 130 can be implemented as media playback terminals. It should be noted that the media terminal according to the present invention can be applied in a music teaching system, but is not limited thereto. For example, the media terminal according to the present invention can also be applied in real-time streaming media solutions such as video conferencing and game live broadcast.

FIG. 2 shows a block diagram of a media terminal 200 according to some embodiments of the invention. Here, the computing device may be configured as the media terminal 200 . Computing devices can be implemented as part of small-sized portable (or mobile) electronic devices such as cellular phones, personal digital assistants (PDAs), personal media player devices, wireless Internet browsing devices, personal headsets, Application-specific devices, or hybrid devices that can include any of the above functions. The computing device can also be implemented as a personal computer including, but not limited to, desktop and notebook computer configurations.

As shown in FIG. 2 , the media terminal 200 includes a video acquisition unit 210 , a video buffer 220 , an audio acquisition unit 230 , an audio buffer 240 , a sending buffer 250 , a transmission unit 260 and a control unit 270 .

The video capture unit 210 is adapted to capture images and encode them into video frames. For example, the video capture unit 210 shoots a scene where a student plays a musical instrument, so as to acquire a sequence of video frames. According to an embodiment of the present invention, the video acquisition unit 210 includes a camera 211 and an encoding unit 212 . The camera 211 is adapted to capture raw image frames. The acquisition parameters of the original image frame are, for example, a size of 640*480 and 25 frames per second, but are not limited thereto. In addition, the video acquisition unit 210 may also record the current time value as the first time stamp of capturing the original image frame each time an original image frame is captured. According to an embodiment of the present invention, an example of the format of the original image frame is:

{dwstamp videodata}

Among them, dwstamp is the first timestamp, and videodata is the image frame in YUV420 format.

The encoding unit 212 is adapted to encode raw image frames. For example, the encoding unit 212 can perform H.264 (Joint Video Team (JVT, Joint Video Team) jointly formed by ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Motion Picture Experts Group (MPEG)) on the original image frame. Proposed Highly Compressed Digital Video Codec Standard) format for encoding. The encoding parameters of the encoding unit 212 include a complete group of pictures (GOP). The length of the GOP is, for example, 100 frames, that is, one picture group has 100 video frames. Here, a GOP starts with an I frame followed by several P frames. There are also B frames between adjacent P frames. For example, the partial sequence of a GOP is I P B P B P P P P B P. Wherein, the I frame is an intra-frame coded frame, which contains complete image information, and does not refer to any additional information to reconstruct the image. A P frame is a forward predictive coded frame, which is predicted from the previous P frame or I frame. B-frames are bi-directionally predicted compressed frames. When compressing an image frame into a B frame, the encoding unit 212 compresses the current frame according to the differences in data of the adjacent previous frame, current frame and subsequent frame. The encoding parameters of the encoding unit 212 may also include encoding frame rate and encoding size, but are not limited thereto. According to an embodiment of the present invention, a code example of encoding unit 212 performing an encoding operation is as follows:

Ret = CLDC_Open(width, height, bitrate, mode);

width: the width of video compression

height: the height of video compression

bitrate: Target bitrate for video compression

mode: video compression mode control

//Image compression engine initialization

Ret = CLDC_Encode(pBuf, dwbase, m_pVideoBuffer, nEncoderLen);

pBuf: the collected image data buffer

dwbase: the length of the original image data

m_pVideoBuffer: buffer for image compression

nEncoderLen: the length of the compressed image

// Generate H264 image stream containing SPS and PPS after image encoding

Ret = CLDC_Close();

//The image encoding engine is closed

The video buffer 220 is suitable for storing video frames generated by the video capture unit 210 . Here, the video buffer 220 is, for example, a ring buffer. For example, the video buffer 220 may always store the latest 20 frames of data generated.

The audio collection unit 230 is adapted to collect sound and encode it into audio frames. In an embodiment according to the present invention, the audio collection unit 230 may encode the collected sound in ACC (Advanced Audio Coding, Advanced Audio Coding) format, and the generated code rate is 192 kbps. Here, the audio collection unit 230 may collect music and breath when the musical instrument is played, but is not limited thereto. In one embodiment of the present invention, the relevant code examples of the audio collection unit 230 performing audio collection and encoding operations are as follows:

int nSamples=AUDIO_SAMPLERATE; //Audio sampling rate

int nChannels=1; //Number of channels

int nBits=16; //The number of audio samples

int nAudioBitrate=192000; //The bit rate of the audio output

int nRet = 0;

nRet＝m_pFaacCodec->Open(nSamples, nChannels, nBits, nAudioBitrate);

//Set the audio compression parameters and open the audio compressor

nCodecRet=m_pFaacCodec->Encode(pBuf,nLen,streamabuffer,nEncodeLen);

pBuf: raw audio data

nLen: the length of the audio data

streamabuffer: buffer for encoding output

nEncodeLen: encoded data length

//Audio encoding, the encoded data is [ADTS header]+[audio compression data]

nRet=m_pFaacCodec->Close();

//End of audio encoding

In addition, the audio collection unit 230 may also capture the time value of the collected audio frame and record it as a second time stamp. The second time stamp corresponding to each audio frame is, for example, the collection time of the first audio sampling point of the frame.

The audio buffer 240 is suitable for storing audio frames generated by the audio acquisition unit 230 . Here, the audio buffer 240 is, for example, a ring buffer. Each audio frame may also include a second time stamp corresponding thereto. In this way, when the media player acquires the audio frame and the video frame transmitted by the media terminal 200, it can perform synchronous processing on the audio frame and the video frame according to the first timestamp and the second timestamp.

The transmit buffer 250 is adapted to store one or more data frames to be transmitted. Each data frame to be sent is a video frame from the video buffer 220 or an audio frame from the audio buffer 240 .

The transmission unit 260 is adapted to sequentially transmit the data frames to be transmitted in the transmission buffer 250 to the media server. In this way, the media server can transmit the data frame to be sent to the media player. The media player receives and plays video data and audio data.

As mentioned above, the sending buffer 250 stores data frames waiting to be transmitted by the transmission unit 260 . The control unit 270 is adapted to extract data frames from the audio buffer 220 and the video buffer 240 and push them to the sending buffer 250 . Generally speaking, in streaming media systems such as music teaching, sound is more important than data such as video information. The control unit 270 performs priority detection on the audio buffer 220 each time a push operation is performed. In one case, the audio buffer 240 has audio frames, and the control unit 270 extracts one audio frame and pushes it to the sending buffer 250 . Another situation is that the control unit 270 detects that the audio buffer 240 is empty. In other words, when the control unit 270 performs this push operation, the audio buffer 240 has no audio frames to be sent. The control unit 270 continues to detect whether the video frame buffer 220 is empty and whether the number of data frames to be transmitted in the transmission buffer 250 exceeds a threshold. The threshold value is 5, for example. Here, the reason for detecting whether the data frames in the sending buffer 250 exceed the threshold is to judge whether the current transmission unit 260 is in a blocked state (that is, the network is abnormal) according to the number of data frames in the sending buffer 250 . In other words, when the transmission unit 260 is not in the blocking state, it can transmit the data frames in the sending buffer 250 to the network in time. Then, the transmit buffer 250 will not accumulate data frames exceeding the threshold. On the contrary, when the transmission unit 260 is in the blocked state, it cannot transmit the data frames in the sending buffer 250 to the network in time. The number of data frames accumulated in the transmit buffer 250 may exceed a threshold. It should be noted that the order of detecting the video buffer 220 and detecting the sending buffer 250 is not too limited in the present invention. In one embodiment, the control unit 270 first detects whether the video buffer 220 is empty, and no longer detects whether the data frame data in the sending buffer 250 exceeds a threshold when the video buffer 220 is empty. In other words, if the video buffer 220 is empty, the push operation ends. Here, the time interval between two adjacent push operations is, for example, 10 milliseconds. In addition, if the video buffer 220 is not empty, the control unit 270 detects whether the amount of data frames in the sending buffer 250 exceeds a threshold. In yet another embodiment, the control unit 270 first detects the sending buffer 250, and when the number of data frames does not exceed the threshold, detects whether the video buffer 220 is empty. In an implementation according to the present invention, the code example of the working process of the control unit 270 is as follows:

As mentioned above, when the control unit 270 detects that the video buffer 220 is not empty and the number of data frames to be sent does not exceed the threshold, it extracts a video frame from the video buffer 220 and pushes it to the sending buffer 250 . In summary, the control unit 270 according to the present invention can preferentially push the audio frames in the audio buffer 240 to the sending buffer 250, and only when the audio buffer 240 is empty and the number of data frames in the sending buffer does not exceed the threshold The video frames in the video buffer 220 will be pushed to the sending buffer 250 . In this way, the media terminal 200 according to the present invention can always give priority to transmission of audio frames, and transmit video frames in the interval of transmission of audio frames. Especially, when the network bandwidth is low (that is, the number of data frames in the sending buffer 250 exceeds a threshold), the media terminal 200 according to the present invention can stop pushing video frames to the sending buffer 250, and normally send the audio buffer 240 The middle audio frame is pushed into the sending buffer 250 . Such a working method can effectively reduce the amount of data to be sent by the transmission unit 260 when the network bandwidth is low, thereby effectively ensuring real-time transmission of audio frames. In other words, the media terminal 200 according to the present invention implements the priority transmission of audio frames, thereby avoiding the problem of the sound being played by the media player terminal being interrupted. It should be noted that in music teaching and other occasions where the importance of sound is high, the media terminal according to the present invention ensures real-time transmission of sound data, and the media player can play continuous, undistorted audio data.

In addition, when the number of data frames in the sending buffer 250 is greater than the threshold, the control unit 270 confirms that the current network is abnormal. The control unit 270 is also adapted to calculate the transmission bit rate of the video frames in the transmission buffer by the transmission unit 260, so that the video acquisition unit 210 can adjust the generation bit rate of the video frames according to the transmission bit rate. According to an embodiment of the present invention, the control unit 270 can count the number of video frames sent by the transmission unit to the sending buffer within a predetermined time (for example, 2 seconds), and then calculate the sending code rate. Subsequently, the video acquisition unit 210 may adjust the image acquisition parameters and encoding parameters according to the sending bit rate, so as to adjust the bit rate for generating video frames. For example, the video capture unit 210 may adjust the size (resolution) of the captured original image or the capture frame rate. For another example, the video acquisition unit 220 may adjust encoding parameters when performing an encoding operation on the original image frame. The coding parameters include, for example, a coding frame rate, a coding size of a generated video frame, and the like. In this way, after the video acquisition unit 210 adjusts the generated bit rate, the generated bit rate matches the sending bit rate of the video frame by the transmission unit 260 . Further, the transmission unit 260 can send the video frames in real time, effectively reducing the situation that the number of data frames in the sending buffer 250 exceeds the threshold, and avoiding the fact that the video frame bit rate in the video buffer 220 is too high and cannot be processed. Timely transmission issues. In addition, since the video acquisition unit can adjust the encoding frame rate, the sequence of video frames transmitted by the transmission unit 260 is uniform in acquisition time. In this way, the media player avoids the problems of excessive frame delay and picture jumping of the played video.

Fig. 3 shows a flowchart of a method 300 for collecting media data according to some embodiments of the present invention. The method 300 is suitable for execution in a media terminal according to the present invention.

As shown in FIG. 3 , the method 300 starts at step S310. In step S310, the sound is collected and encoded into an audio frame, and the audio frame is stored in an audio buffer. The method 300 may also include step S320 of collecting images and encoding them into video frames, and storing the video frames in a video buffer. Here, the audio buffer and the video buffer are, for example, ring buffers. In addition, in step S320, the current time at which the original image corresponding to the video frame is captured may also be recorded as the first time stamp. Each video frame may also include a corresponding first timestamp. In step S310, the acquisition time of the audio frame may also be recorded as the second time stamp. Here, the second time stamp is, for example, the time stamp of the first sampling point of the audio frame. The audio frame may also include a second timestamp. In this way, when the media player acquires the audio frame and the video frame transmitted by the media terminal 200, it can perform synchronous processing on the audio frame and the video frame according to the first timestamp and the second timestamp.

A media terminal according to the invention comprises a send buffer. The send buffer is suitable for storing one or more data frames to be sent. Each data frame to be sent is a video frame from a video buffer or an audio frame from an audio buffer. For the audio frames stored in the audio buffer and the video frames stored in the video buffer, the method 300 controls the network transmission of the audio frames and video frames by performing steps S330, S340 and S350. In step S330, the audio buffer is detected and the audio frames therein are pushed to the sending buffer. In addition, if it is detected in step S330 that the audio buffer is empty, the method 300 executes step S340. In step S340, it is judged whether the number of data frames to be sent in the sending buffer does not exceed the threshold and the video buffer is not empty. Specifically, according to an embodiment of the present invention, in step S340, firstly, it is detected whether the video buffer is empty. If the video buffer is empty, there is currently no video frame to be sent, and the method 300 proceeds to step S330. If the video frame is not empty, continue to detect whether the number of data frames to be sent in the sending buffer exceeds the threshold. If the number of data frames to be sent exceeds the threshold, it indicates that the current transmission network is blocked. In order to transmit the audio frame first, the video frame will not be pushed to the sending buffer, but continue to execute step S330. According to yet another embodiment of the present invention, in step S340, it is first detected whether the number of data frames to be sent in the current sending buffer exceeds a threshold. If the threshold is exceeded, the video buffer is no longer detected, but step S330 is continued.

In addition, if in step S340, it is detected that the number of data frames to be sent does not exceed the threshold and the video buffer is not empty, then step S350 is performed. In step S350, a video frame is extracted from the video buffer and pushed to the sending buffer.

As mentioned above, the method 300 according to the present invention is suitable for generating audio frames and video frames by executing steps S310 and S320, and selecting the audio frames and video frames to be sent by executing steps S330, S340, and S350, and then the The data frame sent is stored in the sending buffer. For the data frames in the sending buffer, the method 300 performs network transmission by executing step S360. In step S360, the data frame to be sent in the sending buffer is transmitted to the media server. A more specific implementation manner of the method 300 according to the present invention is consistent with the working manner of the media terminal 200 in FIG. 2 , and will not be repeated here.

A10. The method described in A8 or A9, wherein the video buffer, the audio buffer and the sending buffer are ring buffers. A11. The method according to any one of A8-A10, wherein the step of judging whether the number of data frames to be sent in the sending buffer does not exceed a threshold and the video buffer is not empty includes: detecting whether the video buffer is empty, if not empty, continue to detect whether the number of data frames to be sent in the sending buffer exceeds the threshold. A12. The method according to any one of A8-A11, wherein the step of judging whether the number of data frames to be sent in the sending buffer does not exceed a threshold and the video buffer is not empty includes: detecting the sending buffer Whether the number of data frames to be sent in the zone exceeds the threshold, and if the threshold is not exceeded, check whether the video buffer is empty. A13. The method according to any one of A8-A12, wherein the video frame includes a first timestamp, which is the capture time value of the image corresponding to the video frame; the audio frame includes the first timestamp Two timestamps, where the second timestamp is the capture time value of the sound corresponding to the audio frame. A14. The method according to any one of A8-A13, wherein the threshold is 5.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline this disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will understand that the modules or units or components of the devices in the examples disclosed herein may be arranged in the device as described in this embodiment, or alternatively may be located in a different location than the device in this example. in one or more devices. The modules in the preceding examples may be combined into one module or furthermore may be divided into a plurality of sub-modules.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

Furthermore, some of the described embodiments are described herein as a method or combination of method elements that may be implemented by a processor of a computer system or by other means for performing the described function. Thus, a processor with the necessary instructions for carrying out the described method or element of a method forms a means for carrying out the method or element of a method. Furthermore, elements described herein of an apparatus embodiment are examples of means for carrying out the function performed by the element for the purpose of carrying out the invention.

As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe generic objects merely means referring to different instances of similar objects and is not intended to imply such The described objects must have a given order temporally, spatially, sequentially or in any other way.

While the invention has been described in terms of a limited number of embodiments, it will be apparent to a person skilled in the art having the benefit of the above description that other embodiments are conceivable within the scope of the invention thus described. In addition, it should be noted that the language used in the specification has been chosen primarily for the purpose of readability and instruction rather than to explain or define the inventive subject matter. Accordingly, many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. With respect to the scope of the present invention, the disclosure of the present invention is intended to be illustrative rather than restrictive, and the scope of the present invention is defined by the appended claims.

Claims (13)

1. a kind of media termination, including：

Video acquisition unit suitable for acquisition image and is encoded to video frame；

Screen buffer is suitable for video frame of the storage from the video acquisition unit；

Audio collection unit suitable for acquisition sound and is encoded to audio frame；

Audio buffer is suitable for audio frame of the storage from the audio collection unit；

Buffering area is sent, is suitable for storing one or more data frames to be sent, wherein each data frame to be sent is from institute State the video frame of screen buffer or the audio frame from the audio buffer；

Transmission unit is suitable for one or more of data frame transfers to be sent to media server；And

Control unit is adapted to detect for audio buffer and is pushed to audio frame therein to send in buffering area, and slow in audio When to rush area be empty, judge whether that the quantity for sending data frame to be sent in buffering area is less than threshold value and screen buffer is not Sky,

If the quantity of data frame to be sent is less than threshold value and screen buffer is not sky, extracted from the screen buffer One frame video frame is simultaneously pushed in transmission buffering area；

Wherein, described control unit is further adapted for when the quantity of data frame to be sent in detecting transmission buffering area is more than threshold value, Transmission code check of the transmission unit to video frame is detected, and code check adjusting parameter is generated according to the transmission code check；And

The video acquisition unit is further adapted for adjusting the generation code check of video frame according to the code check adjusting parameter.

2. media termination as described in claim 1, wherein the screen buffer, the audio buffer and the transmission Buffering area is loop buffer.

3. media termination as claimed in claim 1 or 2, wherein described control unit is suitable for being judged whether according to following manner The quantity for sending data frame to be sent in buffering area is less than threshold value and screen buffer is not sky：

Whether detection screen buffer is empty, if being not sky, continues to detect data frame to be sent in the transmission buffering area Whether quantity is more than threshold value.

4. media termination as claimed in claim 1 or 2, wherein described control unit is suitable for being judged whether according to following manner The quantity for sending data frame to be sent in buffering area is less than threshold value and screen buffer is not sky：

Whether the quantity for detecting data frame to be sent in the transmission buffering area is more than threshold value, and when being less than threshold value, detection Whether screen buffer is empty.

5. media termination as claimed in claim 1 or 2, wherein

The video frame includes stabbing at the first time, which is the capture time value of the corresponding image of the video frame；

The audio frame includes the second timestamp, which is the capture time value of the corresponding sound of the audio frame.

6. media termination as claimed in claim 1 or 2, wherein the threshold value is 5.

7. a kind of method of acquisition media data, suitable for being executed in media termination, which includes sending buffering area, is fitted In the one or more data frames to be sent of storage, wherein each data frame to be sent is video frame or audio frame, this method Including：

Acquisition image is simultaneously encoded to video frame, and the video frame is stored in screen buffer；

Acquisition sound is simultaneously encoded to audio frame, and the audio frame is stored in audio buffer；

Audio frame therein is simultaneously pushed in transmission buffering area by detection audio buffer；

If audio buffer is sky, judge whether that the quantity for sending data frame to be sent in buffering area is less than threshold value and regards Frequency buffering area is not sky,

If the quantity of data frame to be sent is less than threshold value and screen buffer is not sky, carried from the screen buffer It takes a frame video frame and is pushed to and send in buffering area；

When the quantity of data frame to be sent in detecting transmission buffering area is more than threshold value, detection transmission unit is to video frame Code check is sent, and code check adjusting parameter is generated according to the transmission code check；

The generation code check of video frame is adjusted according to the code check adjusting parameter；And

By one or more of data frame transfers to be sent to media server.

8. the method for claim 7, wherein the screen buffer, the audio buffer and transmission buffering Area is loop buffer.

9. method as claimed in claim 7 or 8, wherein the number for judging whether to send data frame to be sent in buffering area Amount is less than threshold value and screen buffer is not that empty step includes：

Whether detection screen buffer is empty, if being not sky, continues to detect data frame to be sent in the transmission buffering area Whether quantity is more than threshold value.

10. method as claimed in claim 7 or 8, wherein the number for judging whether to send data frame to be sent in buffering area Amount is less than threshold value and screen buffer is not that empty step includes：

Whether the quantity for detecting data frame to be sent in the transmission buffering area is more than threshold value, and when being less than threshold value, detection Whether screen buffer is empty.

11. method as claimed in claim 7 or 8, wherein

The video frame includes stabbing at the first time, which is the capture time value of the corresponding image of the video frame；

The audio frame includes the second timestamp, which is the capture time value of the corresponding sound of the audio frame.

12. method as claimed in claim 7 or 8, wherein the threshold value is 5.

13. a kind of music lesson system, including：

Media termination as described in any one of claim 1-6；

Media server is suitable for receiving audio frame and video frame that the media termination is sent；And

Media play end, suitable for being obtained from the media server and playing the audio frame and video frame.