TWI432027B - Method applied to endpoint of video conference system and associated endpoint - Google Patents

Description

Method and related endpoints applied to video conferencing system endpoints

The present invention relates to a method and related endpoints for application to an endpoint of a video conferencing system, and more particularly to a method and associated end for instantly capturing a picture of a transmitting endpoint at an receiving endpoint during an video conference. point.

The video conferencing system enables users in remote locations to exchange views, information, intelligence and knowledge through video, and has been widely used by the public. The video conferencing system has a plurality of endpoints, and the user exchanges videoconferencing packets with other users of the remote endpoints through the local endpoints for video conferencing.

When conducting a video conference, the local user who provides the data content such as materials, images, newsletters, and dynamic video will transfer the data content to other participants. Therefore, the endpoint used by the local end user becomes a sending endpoint, and the data content is encapsulated into a video conference packet and sent to the endpoint of the remote participant. The far-end participant becomes the terminal user, and the endpoint used by the terminal becomes the receiving endpoint to receive the video conference packet transmitted from the sending endpoint.

During the video conference, in order to grasp the focus of the video conference and manage the content of the video conference, the end user will want to extract the content of the information provided by the local user. However, in the current conventional technology, the terminal user cannot perform instant capture during the video conference, and it is necessary to wait for the video conference to lag behind to obtain the data content through the complicated post-production process. Please refer to FIG. 1 , which is a flow 100 of the prior art in which the terminal user retrieves the content of the user data of the local device. During the video conference (step 102), the terminal user at the receiving endpoint needs to activate the video conference recording function (step 104) to record the video conference process. After the video conference ends and the video is completed (step 106), the terminal user needs to replay the previously recorded video conference process with the playback software (step 108). During the playing period, the terminal user must search for the data content screen to be retrieved again; when the terminal user finds the data content screen to be retrieved, the image capturing command is sent to the playing software (step 110), and the software is The picture code of the content to be retrieved is compressed into an image (step 112) so that the image can be backed up to the storage device (step 114).

In other words, in the prior art, even if the end user has found the content of the data to be retrieved during the video conference, the terminal user cannot perform the retrieval immediately, and must pass the post-production (steps 108 to 114). The process can be taken. This is both inconvenient and time consuming for the end user.

In order to overcome the shortcomings of the prior art, the present invention proposes a technology capable of actively extracting data content at the receiving end point, so that the terminal user of the receiving end point can take the desired data content screen as an image.

One of the objectives of the present invention is to provide a method for a receiving endpoint in a video conferencing system, comprising: receiving a video conference packet, obtaining data content from the video conference packet, and decoding and decompressing the data content to obtain Picture; if a command is received, the picture is captured (encoded and compressed) into an image. The picture can be a YUV (luminance-chrominance) format picture that can be converted to a picture in RGB format for playback. The captured image may be a still image that may be stored in a storage device and/or transmitted to a user sharing the other endpoints.

In one embodiment, the video conferencing packet is a packet of the H.323 protocol format, including video & data, wherein the content of the data, or the content of the briefing, is the content of the video material in the H.239 protocol format. That is, the video conferencing packet follows the H.323 format and communication protocol. H.323 is an audio-visual communication protocol recommended by ITU-T (international telecommunications union-telecommunication standardization sector), and H.239 is the data sharing recommended by ITU-T. The standard, which can be applied to the H.323 protocol, carries the video content of the video in the H.323 format packet; in addition, the video conference protocol format followed by the video conference packet can also be the Session Initiation Protocol (SIP). The packet containing the video & data is transmitted, and the Binary Floor Control Protocol (BFCP) is used to transmit the content (ie, the content of the briefing).

Another object of the present invention is to provide a method for receiving an endpoint in a video conferencing system, comprising: receiving a video conference packet, obtaining data content from the video conference packet, and obtaining a first picture from the data content; After receiving an automatic capture command, scene switching detection is performed on the first screen; if scene switching is detected, the first screen or the second screen after the image is captured as an image. In an embodiment, the scene switching detection determines whether the first picture is an inter-frame coding picture; if yes, it can determine that the scene is switched, and automatically encodes the first picture coded in the frame. Compressed into an image. If no scene change is detected, the first picture is not captured as an image.

It is still another object of the present invention to provide an endpoint for a video conferencing system, including a network interface, a processing module, and a capture module. The network interface receives a video conference packet; the processing module obtains the data content from the video conference packet, and obtains the image from the data content; when the capture module receives a capture command, the image is captured as an image. .

Another object of the present invention is to provide an endpoint for a video conferencing system, including a network interface, a processing module, a scene switching detection module, and a capture module. After receiving an automatic capture command, the scene switching detection module starts scene switching detection on the screen in the data content; when the scene switching is detected, the capturing module can make the corresponding image captured as an image. .

In order to better understand the above and other aspects of the present invention, the preferred embodiments are described below, and in conjunction with the drawings, the detailed description is as follows:

Please refer to FIG. 2, which illustrates a process 200 according to an embodiment of the present invention. When a video conference is performed in a video conference system, the process 200 can be applied to a terminal user's receiving endpoint to let the terminal user The data content screen provided by the host user can be captured as an image immediately during the video conference. The steps of the process 200 can be described as follows.

Step 202: During the video conference, the process 200 can be implemented at the receiving end of the terminal user.

Step 204: The receiving endpoint receives the video conference packet sent by the host end user by the sending endpoint, for example, a video conference packet in the H.323 format.

Step 206: Obtain the content of the data provided by the host user from the video conference packet, for example, the video content in the H.239 format.

Step 208: Perform video decoding and decompression on the data content. For example, the data content may include one or more pictures, each picture being a frame of dynamic video; the transmitting end point may be H.263, H.263+ (H.263v2) or H.264, etc. The video codec specification compresses the picture code into the data content. In contrast, when step 208 is performed, the receiving endpoint can perform video decoding and decompression on the data content by using the corresponding video codec specification. H.263, H.263+ and H.264 are video coding specifications developed by the ITU-T VCEG (Video Coding Experts Group).

Step 210: After performing step 208, the screen may be obtained from the data content; each screen may be a screen in the YUV format. The YUV format describes the color of a dynamic or static image as a luminance parameter and a two-chrominance parameter. After performing step 210, proceed to step 212 and/or step 216.

Step 212: Convert the YUV format picture into an RGB format (red-green-blue color) format picture.

Step 214: Display the RGB format picture of step 212 on the screen of the receiving end point, so that the terminal user of the receiving end point can receive the content of the data provided by the host end user.

Step 216: During the continuation of step 208, the receiving endpoint may repeatedly check whether the terminal user triggers a retrieval command. If yes, proceed to step 218.

Step 218: After receiving the capture command, compress the picture code of step 208 into an image. For example, the receiving endpoint may compress the picture encoding into a still image file using an encoding format such as JPEG or JPEG 2000. JPEG and JPEG 2000 are image coding standards for JPEG (joint photographic experts group). After the image capture is complete, proceed to step 220 and/or step 222.

Step 220: The image captured in step 218 can be backed up to a storage device, such as a non-volatile storage device, such as a memory card, a flash memory, a compact disc, and/or a hard disk.

Step 222: The image captured in step 218 can also be shared to other users, for example, participants of a video conference. In one embodiment, the image retrieved by the end user can be encoded and compressed into data content in the receiving endpoint (eg, encoded in H.263, H.263+, or H.264 format to H.239). The content of the format), and this content can be encapsulated in a video conferencing packet (such as a video conferencing packet in H.323 format) and sent to other participants in the same video conference.

It can be seen from the process 200 that during the video conference, when the terminal user sees the data content screen to be captured, the capture command can be triggered immediately, so that the image can be encoded and compressed into an image, and can be further backed up and stored. And / or share.

Please refer to FIG. 3, which is a schematic diagram of an endpoint EB applied to a video conferencing system 50 according to an embodiment of the present invention. The video conferencing system 50 includes a plurality of endpoints, and in FIG. 3, the endpoints EA, EB, and EC are represented; these endpoints exchange packets for the video conference via the network 52, and the technique of the present process 200 can be implemented. Each endpoint, for example, is the endpoint EB. The host EB is provided with a host 60, a source 76 and an output periphery 74. The host 60 includes a command interface 64, a capture module 66, a network interface 68, a processing module 69 and an input interface. 62.

Network 52 can be a wired network, a wireless network, and/or a combination of both. In endpoint EB, network interface 68 is coupled to network 52 to receive and/or transmit packets, such as video conferencing packets, via network 52. The input and output interface 62 is coupled to the source 76 and the output periphery 74. For example, the input/output interface 62 can include various video, audio, and/or data input ports, connectors, and/or terminals, such as an AV terminal, an S terminal, a digital video interface (DVI), and a high definition video. The input interface of the HDMI (High Definition Multimedia Interface) specification and/or the USB (Universal Serial Bus) connector, etc., to receive video signals, audio signals, and/or data from the source 76. Source 76 can include computers (desktops, laptops, and/or tablets), cameras, digital cameras, physical cameras, and/or microphones, to name a few.

The output interface 62 can also include various video, audio, and/or data output ports for transmitting video signals, audio signals, and/or data to the output periphery 74. For example, the output perimeter 74 can include a playback device such as a screen, television, display, projector, speaker, etc., and/or storage device for backing up data. The input/output interface 62 may also include a wireless network interface to contact the source 76 and/or the output periphery 74 having wireless communication functions by radio waves.

The processing module 69 is coupled between the network interface 68 and the input/output interface 62 to control the operation of the host 60 and the video conference. For example, when the network interface 68 receives the video conference packet from the network 52, the processing module 69 can obtain the data content from the video conference packet, and decode and decompress the data content to obtain the picture, audio, and/or data. The video, audio and/or data may be output to the video signal, audio signal and/or data by the input/output interface 62 for playback and/or storage. In contrast, the video signal, audio signal and/or data provided by the source 76 are received by the output interface 62, compressed by the processing module 69 and carried in the data content, and then packaged into the video conference packet. The encapsulated video conferencing packet can be sent to the network 52 via the network interface 68.

The command interface 64 is coupled to the processing module 69 and the capture module 66 to implement a human interface and receive a user-triggered command 78 to enable the user to manipulate the host 60 via the human interface. For example, the command interface 64 can include buttons, a keyboard, a touchpad, and/or a touch screen to allow the user to trigger various commands, such as the capture command of step 216. In an embodiment, the host 60 can configure a dedicated button, a shortcut button, and/or a button combination for the capture command, so that the user can conveniently and quickly capture the received content image during the video conference. image.

In one embodiment, the command interface 64 can also include a remote control transmitter and a remote control receiver (not shown). A keyboard, a mouse, a touchpad, and/or a touch screen are disposed on the remote control transmitter to accept user manipulation; and the remote control receiver is disposed on the host 60. The user's manipulation on the remote control transmitter is translated into a corresponding command, and the remote control signal is sent to the remote control receiver, so that the host 60 can receive the command via the remote control receiver. In one embodiment, the remote control is sent The camera can be configured with a dedicated button for the capture command, allowing the user to perform a one-click capture of the data content screen during the video conference. And/or the command interface 64 may also include a camera that captures the user's image, a sensor that senses the user's position and/or motion, and the like, allowing the user to trigger various commands with gestures and/or actions. And/or, the command interface 64 may also include a voice microphone and a voice recognition mechanism, etc. When the user speaks the command keyword, the command interface 64 can trigger the corresponding command.

The capture module 66 is used to implement the techniques of the process 200 of the present invention (step 216). When the endpoint EB is the receiving endpoint and the video conferencing packet is received by the other endpoint (such as the endpoint EA), the capturing module 66 is received by the command interface 64 for a certain picture obtained by the data content in the video conferencing packet. By capturing the command, the capture module 66 can cause the picture to be captured as an image. For example, when the capture module 66 receives the capture command, the processing module 69 can cause the image module to compress the image into an image. The processing module 69 and/or the capture module 66 can be implemented in hardware, software, firmware, or a combination thereof. For example, the processing module 69 can be a processor, and the host 60 can include a non-volatile or volatile memory (not shown) to store a capture code; when the processing module 69 executes the UI By taking the code, the function of the capture module 66 can be implemented, so that the processing module 69 can compress the image code that the user wants to capture into an image when receiving the capture command.

Please refer to FIG. 4, which illustrates a process 300 according to an embodiment of the present invention. When a video conference is performed in a video conference system, the process 300 can be applied to a terminal user's receiving endpoint to enable the terminal user. The receiving endpoint can automatically capture the data content screen provided by the host user during the video conference as an image. The steps of process 300 can be described as under.

Step 302: During the video conference, the process 300 can be implemented at the receiving end of the terminal user.

Step 304: The receiving endpoint receives the video conference packet sent by the host end user by the sending endpoint, for example, a video conference packet in the H.323 format.

Step 306: Obtain the content of the data provided by the host user from the video conference packet, for example, the video content in the H.239 format.

Step 308: Perform video decoding and decompression on the data content. For example, the content of the material may include one or more pictures.

Step 310: During the continuation of step 308, the receiving endpoint may repeatedly check whether the terminal user triggers a retrieval command. If yes, proceed to step 312.

Step 312: The receiving end performs scene switching detection on the screen of the data content; if the scene switching is detected, the process proceeds to step 314. In an embodiment, when performing the scene switching detection, determining whether each picture is an intra-frame coding picture; if a picture is a frame-encoded picture, determining that the picture is The screen for scene switching.

Step 314: After performing step 308, the picture in the YUV format can be obtained from the data content. After the scene switch is detected in step 312, step 314 may continue to step 316.

Step 316: When the scene switching scene is detected in step 312, the picture coding can be compressed into an image in this step. For example, the receiving endpoint may compress the picture encoding into an image file using an encoding format such as JPEG or JPEG 2000. After the image capture is complete, proceed to step 318.

Step 318: The image captured in step 316 can be backed up in storage. For example, it is a non-volatile storage device.

It can be seen from the process 300 that during the video conference, when the terminal user of the receiving endpoint triggers the automatic capturing command, the receiving endpoint performs scene switching detection and automatically captures the scene switching screen as a map. Image (step 316); in contrast, if scene switching is not detected, it is not necessary to capture a picture in which scene switching has not occurred as an image.

When the host side compresses the picture coding of the video frame to the data content, compression techniques such as intra-frame coding, inter-frame coding (or predictive coding), and/or bidirectional coding are alternately applied. . In the video stream, if the difference between a certain picture F(i) and the previous picture F(i-1) is not large, the picture F(i) is compressed by the inter-frame coding; in the picture F(i) When performing inter-frame coding compression, the compression is performed on the difference between the pictures F(i) and F(i-1). When the picture F(i) is to be decoded and decompressed, not only the difference between the pictures F(i) and F(i-1) but also the picture F(i-1) must be referred to in order to obtain the picture F. (i). Conversely, if a picture F(j) differs greatly from the previous picture F(j-1), the picture F(j) is independently compressed by the intraframe coding without reference to other pictures.

In other words, if the picture F(i) is an inter-frame coded picture, the representative picture F(i) and the previous picture (such as the picture F(i-1)) present a small difference in the same scene, the picture F(i) There is no scene switching. In contrast, if the picture F(j) is a frame-encoded picture, the representative picture F(j) and the previous picture (such as the picture F(j-1)) present different scenes, and the picture F(j) is the scene switching picture. . Therefore, in the process 300, when the automatic capture command is triggered, the scene switching screen F(j) is automatically captured as an image without scene switching. The picture F(i) will not be captured as an image.

For example, the host user will explain the same slide for a period of time during the video conference; during this period, since the scenes of the video conference are all in the same slide, the video frame during this period is displayed. Will be compressed with inter-frame coding. In contrast, if the host user switches to the next slide, the scene will switch, and the process 300 can automatically capture the image of the new slide as an image. That is to say, when the terminal user triggers the automatic capture command at the receiving end point, different slides can be automatically captured as a series of images.

There are other embodiments for scene switching detection. For example, features of each picture (such as image edges, etc.) can be identified, and feature differences of different pictures can be compared; if the feature difference exceeds a certain critical value, it is determined that there is scene switching.

During the steps 312 to 316, the YUV format picture obtained from the material content can also be converted to an RGB format picture and played back, similar to steps 212 and 214 in the process 200. The automatically captured images in step 316 can also be shared, similar to step 222. When the process 300 is performed, the terminal user may also terminate the automatic capture after triggering the automatic capture command; after the automatic capture is terminated, the receiving terminal no longer performs scene switch detection, and does not automatically capture the image as an image.

Please refer to FIG. 5, which illustrates a process 410 according to another embodiment of the present invention. When a video conference is performed in a video conference system, the process 410 can be applied to a terminal end user's receiving endpoint for the terminal to use. The receiving endpoint of the user can automatically capture the data content screen provided by the host user during the video conference as an image. The process 410 follows steps 302 to 312 of the process 300. After the step 312, the following steps are continued.

Step 412: After determining the screen of the scene change, the Nth screen after the screen is taken as the object of the image capture, and N is a preset number; for example, the Nth frame after the scene change screen The encoded picture is used as an object for image capture. For example, if the screen F(i) is a screen in which the scene changes, the screen F(i+N) is the object of the image capture.

Step 414: Obtain a YUV format picture, that is, a YUV format picture of the picture F(i+N), for the Nth picture after the scene change.

Step 416: Compress the YUV format picture of the Nth picture after the scene change into an image; that is, capture the picture F(i+N) as an image.

When the available bandwidth of the network is low, the picture quality of the in-frame coded picture reflecting the scene change is limited. Therefore, the preset number (ie, N, for example, 30) of the frame after the scene change can be automatically captured by the image to improve the image quality of the image capture.

Please refer to FIG. 6, which is a flow chart 420 according to still another embodiment of the present invention. When a video conference is performed in a video conference system, the process 420 can be applied to the terminal end user's receiving endpoint for the terminal to use. The receiving endpoint of the user can automatically capture the data content screen provided by the host user during the video conference as an image. The process 420 follows steps 302 to 312 of the process 300. After the step 312, the following steps are continued.

Step 422: After determining the screen of the scene change, find a screen that meets a preset condition after the screen as an object of image capture. In an embodiment, the image is captured by the screen after the scene change screen is equal to a preset value; the preset value may depend on the network bandwidth, for example, the network bandwidth. a certain preset ratio. For example, if the network bandwidth is 256MB, the preset ratio is 1/30, and the default value can be 9MB (ie 256*1/30). For example, if the picture F(i) is an intra-frame coded picture of a scene change, the file size of each picture is checked one by one for its subsequent inter-frame coded pictures F(i+1), F(i+2), and the like. Whether it meets (equal to or greater than) the preset value; if a certain picture F(i+N) meets the preset value, the picture F(i+N) is taken as the object of the image capture.

Step 424: Acquire a picture of the YUV format, that is, a YUV format picture of the picture F (i+N), for the picture that meets the preset condition after the scene change.

Step 426: Compress the YUV format picture that meets the preset condition after the scene change into an image; that is, capture the picture F(i+N) as an image.

Since the file size of the screen is related to the image quality, the process 420 can also ensure the image quality captured by the image.

In an embodiment, the process 200 and the 300/410/420 can also be performed synchronously, so that the terminal user receiving the endpoint can determine the picture to be captured (flow 200), and also allows the receiving end point to automatically capture the scene switching. Picture (flow 300/410/420).

Please refer to FIG. 7, which is a schematic diagram of an endpoint EB2 applied to a video conferencing system 50b according to an embodiment of the present invention. The video conferencing system 50b includes a plurality of endpoints, and the endpoints EA, EB2, and EC are represented in FIG. 7; the endpoints exchange packets for the video conference via the network 52, and the technique of the flow 300 of the present invention can be implemented. Each endpoint, for example, is the endpoint EB2. The host EB2 is provided with a host 80, a source 96 and an output periphery 94. The host 80 includes a command interface 84, a capture module 86, a scene switch detection module 82, a network interface 88, and a The processing module 89 and an input/output interface 83 are provided. The functions and embodiments of the network interface 88, the processing module 89, the input/output interface 83, the source 96, and the output periphery 94 can be respectively from the The network interface 68, the processing module 69, the input/output interface 62, the source 76, and the output periphery 74 are analogous.

The command interface 84 receives the user-triggered command 98, such as the automatic capture command of step 310; various embodiments of the command interface 84 may refer to the command interface 64 of FIG. For example, the command interface 84 can configure a dedicated button for the automatic capture command, allowing the user of the endpoint EB2 to trigger an automatic capture command with a single button.

The scene switch detection module 82 and the capture module 86 are used to implement the techniques of the process 300/410/420 of the present invention (steps 311/212/422 and 316). When the endpoint EB2 is the receiving endpoint and the other endpoint (such as the endpoint EA) receives the video conference packet, the processing module 89 obtains each screen of the data content by the video conference packet; if the user triggers the automatic capture command, the scenario The switch detection module 82 will start scene change detection for each picture. When the scene switching detection module 82 detects the scene switching screen (flow 300), the preset number of pictures after the scene switching (flow 410), and/or the screen that meets the preset condition after the scene switching (flow 420), The capture module 86 causes the image to be captured as an image; for example, the capture module 86 can cause the processing module 89 to compress the image encoding into an image. The processing module 89, the capture module 86, and/or the scene switching detection module 82 can be implemented by hardware, software, firmware, or a combination thereof. For example, the processing module 89 can be a processor, and the host 80 can include a memory (not shown) for storing a scene change detection code and a capture code; when the processing module 89 executes The scene switching detection code and the capture code can respectively implement the functions of the scene switching detection module 82 and the capturing module 86, so that the processing module 89 can start the scene switching detection after receiving the automatic capturing command. Measure, automatically switch the scenes of each painting The faces are sequentially captured as a series of images.

In summary, in the prior art, the end user of the video conference cannot actively take the video content screen of the video conference as an image during the video conference. In contrast, the technology of the present invention allows the terminal user to actively take the image as an image during the time conference, not only avoiding complicated and time-consuming post-production, but also focusing on the content of the video conference and knowledge management. It is also more convenient.

In conclusion, the present invention has been disclosed in the above preferred embodiments, and is not intended to limit the present invention. A person skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, the scope of the invention is defined by the scope of the appended claims.

50, 50b‧‧‧ video conferencing system

52‧‧‧Network

64, 84‧‧‧ command interface

66, 86‧‧‧ capture module

68, 88‧‧‧ network interface

69, 89‧‧‧ processing module

62, 83‧‧‧ input and output interface

74, 94‧‧‧ Output periphery

76, 96‧‧‧ source

78, 98‧‧‧ Order

60, 80‧‧‧ host

82‧‧‧ Scene Switch Detection Module

EA, EB, EC, EB2‧‧‧ endpoints

100, 200, 300, 410, 420‧‧‧ processes

Steps 102-114, 202-222, 302-318, 412-416, 422-426‧‧

The first figure is a process in which the terminal user performs video conference data content acquisition according to the prior art.

Figure 2 is a flow diagram of an embodiment of the present invention.

FIG. 3 is a schematic diagram of an endpoint of an embodiment of the present invention applied to a video conferencing system.

4 to 6 are flowcharts according to another embodiment of the present invention.

FIG. 7 is a schematic diagram of an endpoint of an embodiment of the present invention applied to a video conference system.

200. . . Process

202-222. . . step

Claims (11)

A method for applying an endpoint of a video conferencing system, comprising: receiving a video conference packet; obtaining a data content from the video conference packet; obtaining a first picture from the data content; performing a first picture for the first picture Scene switching detection; and if the scene switching is detected, the first picture or a second picture after the first picture is captured as an image. The method of claim 1, wherein when the scene switching detection is performed, determining whether the first picture is an intra-frame.coding picture. The method of claim 1, further comprising: if the scene switching is not detected, the first picture is not captured as an image. The method of claim 1, further comprising: starting the scene switching detection after receiving an automatic capture command. The method of claim 1, wherein the second picture is a preset number of pictures after the first picture. The method of claim 1, wherein the The file size of the two screens conforms to a preset value. An endpoint for a video conferencing system includes: a network interface for receiving a video conference packet; a processing module for obtaining a data content from the video conference packet, and obtaining a first content from the data content a scene switching detection module, performing a scene switching detection on the first screen; and a capture module; when the scene switching detection module detects a scene switching, the capturing module makes the The first picture or a second picture after the first picture is captured as an image. The endpoint of claim 7, wherein the scene switching detection module starts the scene switching detection after receiving an automatic capturing command. The endpoint of claim 7, wherein the scene switching module determines whether the first picture is a frame-encoded picture, and performs the scene switching detection accordingly. The endpoint of claim 7, wherein the second picture is a preset number of pictures after the first picture. The endpoint of claim 7, wherein the file size of the second screen conforms to a preset value.