CN101198008A - A method and system for realizing multi-screen and multi-picture - Google Patents

Abstract

The present invention provides a method and system for realizing multiple screens and multiple pictures in a video conferencing system. The video conferencing system includes a multi-point control unit and several terminals, including the following steps: (a) establishing a connection between the terminal and the multi-point control unit, The terminal sends the local video stream to the multipoint control unit; (b) the multipoint control unit synthesizes the input video stream of the terminal into multiple output video streams; (c) the multipoint control unit sends the multiple output video streams to the terminal, The content of each output video stream is a multi-screen synthesized by one or more input video streams; (d) the terminal displays the multiple output video streams on multiple displays after receiving the multiple output video streams. By adopting the method of the present invention, terminals participating in communication can use multiple displays to display video images of multiple remote conference sites, and users can see larger or more video images of remote conference sites, which improves the user's video experience, It enhances the user communication effect and improves the use value of the system.

Description

A method and system for realizing multi-screen and multi-picture

technical field

The invention relates to the field of video conferencing, in particular to a method and system for realizing multi-screen and multi-picture in a video conferencing system.

Background technique

A video conferencing system is a multimedia communication system that supports sound, video and data communication, and it includes a video conference terminal, MCU (Multipoint Control Unit, multipoint control unit) and some other equipment.

The video conferencing systems currently on the market generally conform to the ITU-T H.320 (International Telecommunication Union telecommunication standard H.320) or ITU-T H.323 standards, and there are also systems that conform to the IETF SIP (Internet Engineering Task Force Session Initiation Protocol) standard .

The terminal is the device used by the user. The terminal contains a codec. The codec is responsible for completing the compression encoding and decoding of sound, video and other media. The terminal is also connected to a microphone, a camera, a display, and a sound playback subsystem to complete sound and video. The terminal also includes a user input interface, and the user inputs instructions and information to the terminal through the input interface. The terminal is responsible for establishing a connection with the other party's terminal, negotiating the codec capabilities of both parties, and using the codec algorithm supported by both parties to communicate.

MCU is used to complete multi-party communication. The terminal participating in the multi-party communication establishes a call with the MCU, and sends and receives media, and the MCU is responsible for completing the exchange and mixing of media. The MCU can synthesize the video obtained from multiple terminals into a multi-screen image, and then send it to different terminals.

Some terminals also support multi-party communication capabilities, that is, they can establish connections with multiple terminals at the same time. In this document, this type of terminal is referred to as a terminal with a built-in MCU function.

In a multi-point conference, the multi-screen function allows users to see the videos of multiple calling terminals at the same time, which helps to improve the call effect, and is a very popular feature.

At present, multi-screens are usually synthesized by an MCU or a terminal with a built-in MCU, and finally sent to a receiving terminal as a video stream.

In many cases, the user's terminal equipment is equipped with more than two displays. At present, the video conferencing system supports two video streams, but the existing multi-screen display only allows one video stream to be synthesized and transmitted, and the other video stream is only transmitted. An image from a video source, such as a VGA display signal from a computer. If multiple pictures are synthesized into one video stream, they can only be displayed on one display, so the display area allocated to each sub-picture is relatively small.

Contents of the invention

The technical problem to be solved by the present invention is to provide a method and system for realizing multi-screen and multi-picture, which solves the problem that in existing video conferencing, multi-picture can only be displayed on one screen, resulting in a small sub-picture display area, and improves user experience .

In order to solve the above technical problems, the present invention provides a method for realizing multi-screen and multi-picture in a video conferencing system. The video conferencing system includes several terminals, including the following steps:

(a) The terminal establishes a connection with the multipoint control unit, and the terminal sends the local video stream to the multipoint control unit;

(b) The multi-point control unit synthesizes the input video stream sent by the terminal into a plurality of output video streams, and each output video stream content is a multi-picture synthesized by one or more of the input video streams;

(c) the multipoint control unit sends the plurality of output video streams to the terminal;

(d) After receiving the multiple output video streams, the terminal displays them on multiple displays.

Furthermore, the above method may also have the following features, in step (a), the multi-point control unit is an independent multi-point control unit or built in any terminal of the video conferencing system.

Further, the above method can also have the following characteristics, in step (a), the terminal and the multipoint control unit pass ITU-T H.323 or ITU-T H.320 or Internet Engineering Task Force Session Initiation Protocol IETF The SIP protocol establishes the connection.

Further, the above method can also have the following features, in step (a), the multipoint control unit and the terminal establish a two-way audio channel, a video channel from the terminal to the multipoint control unit, one or more multipoint A video channel from the control unit to the terminal, and transmits the input video stream and the output video stream through the audio channel and the video channel.

Furthermore, the above method may also have the following features, the number of video channels from the multi-point control unit to the terminal depends on the number of monitors of the terminal, and the multi-point control unit obtains the number of monitors of the terminal through negotiation or manual configuration.

Further, the above method may also have the following features, in the step (b), the user sends a signaling through the terminal to request the multi-point control unit to synthesize the effect of the local multi-picture of the terminal, and the multi-point control unit Synthesis of the input video stream is performed according to the signaling request.

Further, the above method can also have the following features, the signaling request includes one or more of the following parameters: the number of output video streams, the number of multi-pictures of each output video stream, the sub-picture content contained in the multi-picture, the multi-picture layout.

Further, the above method may also have the following feature, in the step (b), the output video stream synthesized by the multipoint control unit for each terminal does not include the local input video stream of the terminal.

Further, the above method may also have the following feature, in the step (c), the multi-point control unit sends a signaling to inform the terminal of the multi-screen layout of each output video stream.

The present invention also proposes a multi-screen multi-picture video conferencing system, including a terminal and a multi-point control unit. The terminal is used to send local video streams to the multi-point control unit, and is also used to receive the multi-point multiple output video streams sent from the point control unit and displayed on multiple displays;

The multi-point control unit is configured to synthesize the input video stream of the terminal into multiple output video streams and send them to the terminal, wherein the content of each output video stream is a multi-point video stream synthesized from one or more input video streams. screen.

By adopting the method of the present invention, terminals participating in communication can use multiple displays to display video images of multiple remote conference sites, and users can see larger or more video images of remote conference sites, which improves the user's video experience, It enhances the user communication effect and improves the use value of the system.

Description of drawings

Fig. 1 is a flow chart of the method of the present invention;

Fig. 2 is a device networking diagram of an embodiment of the present invention;

Fig. 3 is a method flowchart of an embodiment of the present invention;

Fig. 4 is a schematic diagram of MCU synthesizing multiple pictures in the embodiment of the present invention;

FIG. 5 is a schematic diagram of some typical layouts of multiple screens;

FIG. 6 is a schematic diagram of a signaling interaction sequence for establishing a connection between a terminal and an MCU.

Detailed ways

The present invention synthesizes multiple pictures into more than two video streams, each video stream is respectively a multi-picture video (including a special case of a single picture), and each video stream is displayed on a different display. In this way, under the same total number of sub-pictures, each sub-picture has a larger display area available, and the user can watch a larger sub-picture, thereby improving the quality of video experience.

As shown in Figure 1, the method of the present invention comprises the following steps:

Step 110, the terminal participating in the communication establishes a connection with the MCU providing multipoint communication service or the built-in MCU terminal.

Among them, the connection between the terminal and the MCU or the built-in MCU terminal can be established using but not limited to one of ITU-TH.323, ITU-T H.320, and IETF SIP protocols.

In the connection establishment process, the participation of third-party equipment is not excluded. However, these devices are not the core of the present invention, so the related description is omitted.

The MCU or the built-in MCU terminal can obtain the number of monitors of the terminal through negotiation, and the MCU or the built-in MCU terminal can also obtain the number of monitors of the terminal through other means (such as manual configuration).

The MCU and the terminal establish a two-way audio channel, a two-way video channel (equivalent to a video channel from the terminal to the MCU, and a video channel from the MCU to the terminal), and multiple one-way video channels from the MCU to the terminal. The number of one-way video channels Depends on the number of monitors of the terminal. When the terminal contains n displays, n-1 unidirectional video channels need to be established. The above-mentioned method for establishing a channel between the MCU and the terminal is just an example, and the channel may also be established in other ways, and the number of channels is not limited to the above description. The corresponding number of channels can be established according to the needs of the terminal. For example, the terminal includes multiple displays, but only some of them are used for display, and only video channels are established for these displays.

Step 120, the terminal sends the local video stream to the MCU or the built-in MCU terminal.

In step 130, the MCU or a terminal with a built-in MCU synthesizes video streams input by multiple terminals into multiple output video streams, each output video stream may contain one or more sub-pictures, and each sub-picture is an image of one input video stream.

Among them, the user can send a signaling through the terminal device to request the MCU or the built-in MCU terminal local terminal multi-picture composite effect, the parameters in the signaling request include but not limited to: the number of output video streams, the multi-picture of each output video stream number, sub-picture content, multi-picture layout, etc.

Step 140, the MCU or the terminal with built-in MCU sends the multiple output video streams mentioned in step 130 to the terminal.

The MCU or a terminal with a built-in MCU can notify the terminal of the multi-screen layout of each output video stream through signaling.

Step 150, the terminal receives the multiple output video streams, and displays the multiple output video streams on different displays.

The present invention also proposes a multi-screen multi-picture video conferencing system, including a terminal and a multi-point control unit. The terminal is used to send local video streams to the multi-point control unit, and is also used to receive the multi-point The multiple output video streams sent by the point control unit are displayed on multiple displays; the multi-point control unit is configured to synthesize the input video streams of the terminal into multiple output video streams and send them to the terminal, wherein The content of each output video stream is a multi-picture composed of one or more input video streams. FIG. 2 describes the device connection relationship of a specific embodiment of the present invention. In this example, both the terminal and the MCU are connected to the IP network and communicate through the IP network. The communication between the terminal and the MCU complies with the ITU-T H.323 standard. In this embodiment, each terminal has two displays and supports receiving two video streams. The MCU has the capability of synthesizing multiple images.

Fig. 3 describes the method flow of the embodiment of the present invention.

Step 310, terminal 1 establishes an H.323 connection with the MCU.

The terminal 1 may actively call the MCU, or the MCU may call the terminal 1.

The H.323 connection establishment process includes the call signaling process and the H.245 signaling process. The established connection includes a signaling channel and multiple media channels, including a bidirectional audio channel, a bidirectional video channel, and a unidirectional video channel from the MCU to the terminal. The specific signaling interaction process is shown in FIG. 6 .

Terminal 2, Terminal 3, and Terminal 4 also establish connections with the MCU in the same manner. Since all connections have similar characteristics, without departing from the generality, the connection between terminal 1 and MCU is taken as an example for description.

Step 320, terminal 1 sends the video stream of the conference site to the MCU through a video channel.

In step 330, the MCU synthesizes the video streams sent by the four terminals into a plurality of output video streams.

In this embodiment, two output video streams are synthesized, and each output video stream contains video images of some terminals. For example, one output video stream contains video images of Terminal 1 and Terminal 2, and the other output video stream contains Terminal 3 and Terminal 2. Video image of Terminal 4.

The MCU can also synthesize different multi-screen effects for each terminal. For example, the output video stream sent to each terminal does not include the terminal's own input video stream.

In step 340, the MCU sends the synthesized two output video streams to Terminal 1 through two video channels. At the same time, the MCU may notify the terminal of the multi-picture layout style of each output video stream through signaling.

Step 350, after the terminal receives the output video streams, the two output video streams are displayed on different displays.

FIG. 4 is a schematic diagram of MCU processing multi-picture synthesis. The video streams from the 4 terminals are finally merged into two output video streams, and each output video stream contains video images of 2 terminals.

FIG. 5 is a schematic diagram of several multi-picture display layouts. The figure shows layout examples of 1 screen, 2 sub-pictures (left and right), 3 sub-pictures (1 up and 2 down), 4 sub-pictures (one up, down, left and right), 6 sub-pictures, and 9 sub-pictures.

FIG. 6 is a schematic diagram of a signaling interaction sequence for establishing a connection between a terminal and an MCU.

Assume that the terminal actively calls the MCU.

1. The terminal first establishes a channel for transmitting call signaling with the MCU.

2. The terminal performs call signaling interaction with the MCU on the call signaling channel. The terminal sends the Setup message, the MCU responds to the Alerting and Connect messages, and the calling process is completed.

3. A channel for transmitting H.245 control signaling is established between the terminal and the MCU.

4. The terminal performs control signaling interaction with the MCU on the control signaling channel.

The terminal sends a TerminalCapabilitySet message to the MCU, and the MCU responds with a TerminalCapabilitySetAck confirmation message. The MCU sends a TerminalCapabilitySet message to the terminal, and the terminal responds with a TerminalCapabilitySetAck confirmation message. The ability exchange between the two parties is completed. The capability set of the terminal and MCU indicates that it supports the receiving and sending of one audio stream and two video streams.

The terminal sends a MasterSlaveDetermination message to the MCU, the MCU responds to the MasterSlaveDeterminationAck confirmation message, the MCU sends the MasterSlaveDetermination message to the terminal, and the terminal responds to the MasterSlaveDeterminationAck confirmation message. Both master and slave decided to complete the negotiation.

The terminal sends an OpenLogicalChannel message to the MCU, requesting to establish a sound stream channel, and the MCU sends an OpenLogicalChannel message to the terminal, requesting to establish a sound stream channel. The terminal responds to the OpenLogicalChannelAck confirmation message, and the MCU responds to the OpenLogicalChannelAck confirmation message. The two sides established a two-way sound channel.

Similarly, a first bidirectional video stream channel is established between the terminal and the MCU, and a second unidirectional video stream channel is also established between the terminal and the MCU. Finally, the MCU and the terminal send and receive media data on the audio and video stream channels.

By adopting the method of the present invention, terminals participating in communication can use multiple displays to display video images of multiple remote conference sites, and users can see larger or more video images of remote conference sites, which improves the user's video experience, It enhances the user communication effect and improves the use value of the system.

Claims (10)

1. realize the method for multi-screen and multi-picture in the video conferencing system, described video conferencing system comprises multipoint control unit and several terminals, comprises following steps:

(a) terminal and multipoint control unit connect, and terminal sends to described multipoint control unit to local video stream;

(b) described multipoint control unit synthesizes a plurality of outputting video streams to the input video stream that described terminal sends, and each outputting video streams content is by the synthetic many pictures of one or more input video streams;

(c) described multipoint control unit sends described a plurality of outputting video streams to described terminal;

(d) described terminal is presented on a plurality of displays after receiving described a plurality of outputting video streams.

2. the method for claim 1 is characterized in that, in the step (a), described multipoint control unit is an independent multipoint control unit or is built on the terminal of arbitrary described video conferencing system.

3. the method for claim 1 is characterized in that, in the step (a), described terminal and described multipoint control unit by ITU-T H.323 or ITU-T H.320 or the Internet engineering duty group session initializtion protocol IETF Session Initiation Protocol connect.

4. the method for claim 1, it is characterized in that, in the step (a), described multipoint control unit and described terminal are set up a two way audio passage, a terminal arrives the video channel of multipoint control unit, one or more multipoint control units transmit described input video stream and outputting video streams to the video channel of terminal by described voice-grade channel and video channel.

5. as claim 1 or 4 described methods, it is characterized in that described multipoint control unit depends on the display number of terminal to the video channel number of terminal, described multipoint control unit through consultation or manual configuration obtain the display number of described terminal.

6. the method for claim 1, it is characterized in that, in the described step (b), the user is by the synthetic effect of the local many pictures of the described described terminal of the described multipoint control unit of terminal transmission signaling request, and described multipoint control unit carries out the synthetic of input video stream according to described signaling request.

7. method as claimed in claim 6 is characterized in that, comprises following one or more parameter in the described signaling request: output video flow amount, many frame numbers of every road outputting video streams, the sprite content that many pictures comprise, many picture layout.

8. as claim 1 or 6 or 7 described methods, it is characterized in that in the described step (b), described multipoint control unit is not for comprising the input video stream of described terminal this locality in the synthetic outputting video streams of each described terminal.

9. as claim 1 or 6 or 7 described methods, it is characterized in that in the described step (c), described multipoint control unit sends many picture layout of described each outputting video streams of terminal of signaling.

10. video conferencing system that realizes multi-screen and multi-picture, comprise terminal, multipoint control unit, it is characterized in that, described terminal, be used for local video stream is sent to described multipoint control unit, also be used to receive a plurality of outputting video streams of described multipoint control unit transmission and be presented at a plurality of displays;

Described multipoint control unit is used for the input video stream of described terminal is synthesized a plurality of outputting video streams and sends to described terminal, and wherein each outputting video streams content is by the synthetic many pictures of one or more input video streams.

Images