CN119814964A

CN119814964A - Server, terminal equipment and video conference remote control method

Info

Publication number: CN119814964A
Application number: CN202411931055.4A
Authority: CN
Inventors: 王晓辉; 徐延霞; 王云刚; 国廷峰
Original assignee: Juhaokan Technology Co Ltd
Current assignee: Juhaokan Technology Co Ltd
Priority date: 2024-12-25
Filing date: 2024-12-25
Publication date: 2025-04-11

Abstract

Some embodiments of the present application illustrate a server, a terminal device, and a video conference remote control method, which relates to the field of remote control. The method includes: receiving a remote control request sent by a first terminal device, and sending the remote control request to a second terminal device to establish remote control of the first terminal device over the second terminal device; if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location; if a keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message is only sent to the second terminal device, so that the second terminal device performs the operation corresponding to the keyboard and mouse control message. When multiple parties participate in a video conference, the server can store the audio and video data sent by the controlled end for other devices to pull the stream, and can also send the keyboard and mouse control message sent by the control end only to the terminal device with which the remote control connection is established, so as to achieve the purpose of remote control, without the need to install other software, simplifying user operations.

Description

Server, terminal equipment and video conference remote control method

Technical Field

The application relates to the technical field of remote control, in particular to a server, terminal equipment and a video conference remote control method.

Background

With the development of information technology, the appearance of online video conferences breaks through the space obstruction, improves the working efficiency of users, and is closely related to the work and life of the users. The video conference office software supports the function of sharing a computer desktop in a conference, so that the conference material can be watched together conveniently, the conference efficiency is improved, the meaning cannot be accurately expressed by language communication under partial scenes, a lot of time is spent on determining small problems such as text positions, the conference progress is dragged slowly, and the time of a user is wasted.

In order to save the time of a user, special remote assistance software can be selected to control the computers of the sharing persons to perform actual operation, but the scheme needs to install the same remote assistance software on a control end and a controlled end which participate in a conference to realize that other meeting persons control the computers of the sharing persons, and the user operation is relatively complicated.

Disclosure of Invention

The application provides a terminal device, a server and a video conference remote control method, wherein when a plurality of parties participate in a video conference, the server can directly establish remote control connection between at least two terminal devices, can store audio and video data sent by a controlled terminal for streaming of other devices, and can also send a mouse control message sent by a control terminal to only the terminal device which establishes remote control connection with the terminal device, so that the purpose of remote control is achieved, other software is not required to be installed, and user operation is simplified.

In a first aspect, some embodiments of the present application provide a server, including:

a communication module;

a processing module configured to:

Receiving conference requests respectively sent by first terminal equipment, second terminal equipment and third terminal equipment, so that the first terminal equipment, the second terminal equipment and the third terminal equipment join the same video conference;

Receiving a remote control request sent by the first terminal equipment, wherein the remote control request comprises a second equipment identifier of the second terminal equipment, and sending the remote control request to the second terminal equipment corresponding to the second equipment identifier so as to establish remote control of the second terminal equipment by the first terminal equipment;

Under the condition that remote control of the first terminal equipment to the second terminal equipment is established, if audio and video data uploaded by the second terminal equipment are received, the audio and video data are stored to a preset position so as to be acquired and played by the first terminal equipment and the third terminal equipment;

If the mouse control message uploaded by the first terminal equipment is received, the mouse control message is only sent to the second terminal equipment, so that the second terminal equipment executes the operation corresponding to the mouse control message, and the mouse control message is not sent to the third terminal equipment.

The technical scheme has the advantages that when a plurality of parties participate in a video conference, the server can directly establish remote control connection between at least two terminal devices, can store audio and video data sent by the controlled end for streaming of other devices, and can also send a mouse control message sent by the control end to the terminal device which establishes remote control connection with the control end, so that the purpose of remote control is achieved, other software is not required to be installed, and user operation is simplified.

In some embodiments, the processing module performs sending a mouse control message to the second terminal device, and is further configured to:

reading a key mouse control message at a key mouse message queue head corresponding to the second equipment identifier, wherein the key mouse control message in the key mouse message queue is uploaded by the first terminal equipment;

when the receiving time difference value of the key mouse control message positioned at the tail of the key mouse message queue and the key mouse control message positioned at the head of the key mouse message queue is smaller than or equal to a first preset difference value, the key mouse control message positioned at the head of the key mouse message queue is sent to second terminal equipment corresponding to a second equipment identifier;

And deleting the key mouse control message positioned at the head of the key mouse message queue under the condition that the receiving time difference between the key mouse control message positioned at the tail of the key mouse message queue and the key mouse control message positioned at the head of the key mouse message queue is larger than a first preset difference, and continuously reading the next key mouse control message positioned at the head of the key mouse message queue until the receiving time difference between the key mouse control message positioned at the tail of the key mouse message queue and the key mouse control message positioned at the head of the key mouse message queue is smaller than or equal to the first preset difference, and transmitting the key mouse control message positioned at the head of the key mouse message queue to second terminal equipment corresponding to the second equipment identifier.

The technical scheme has the advantages that under the condition that network is blocked or a plurality of first terminal devices send the mouse control messages at the same time to cause excessive mouse control messages in a mouse message queue, the mouse control messages at the head of the queue can be deleted and not sent to the second terminal device, so that the phenomenon that the second terminal device is overloaded in a short time when recovering the network is avoided.

Determining a first mouse control message as a mouse control message with a receiving time difference between the first mouse control message and a mouse control message positioned at the head of a mouse queue smaller than a second preset difference;

Acquiring a first equipment identifier and a priority corresponding to the first equipment identifier in a first mouse control message, determining the first mouse control message with the highest priority as a second mouse control message, determining the first mouse control message except the second mouse control message as a third mouse control message, and determining the first equipment identifier as the equipment identifier of the first terminal equipment;

deleting the third mouse control message, and sending the second mouse control message to a second terminal device corresponding to the second device identifier.

The technical scheme has the advantages that when a plurality of first terminal devices send the mouse control messages simultaneously or successively, the mouse control message with relatively lower priority corresponding to the first terminal device is deleted, and the non-deleted mouse control message is sent to the second terminal device, so that the second terminal device preferentially executes the mouse control message sent by the first terminal device with higher priority.

In some embodiments, the processing module, after the first terminal device, the second terminal device, and the third terminal device join the same video conference, is further configured to:

Receiving a request for sending shared clipboard content by a first terminal device, a second terminal device and/or a third terminal device, and adding a first device identifier of the first terminal device, a second device identifier of the second terminal device and/or a third device identifier of the third terminal device into a shared clipboard list;

And under the condition that the clipboard content sent by the first terminal equipment, the second terminal equipment or the third terminal equipment is received, sending the clipboard content to the target terminal equipment corresponding to the equipment identifier in the shared clipboard list so as to change the clipboard content of the target terminal equipment.

The technical scheme has the advantages that the server can send the clipboard content uploaded by the reference terminal equipment to the terminal equipment in the shared clipboard list, so that the clipboard content sharing function is realized.

In a second aspect, some embodiments of the present application provide a first terminal device, including:

A display configured to display a user interface;

a communication device configured to establish a communication connection with a server;

A controller configured to:

In response to an instruction to create or join a conference, sending a conference request to a server, so that the server joins a first terminal device to the video conference based on the conference request, wherein the first terminal device, the second terminal device and the third terminal device join the same video conference;

Responding to an instruction for controlling the second terminal equipment, sending a remote control request to the server, so that the server sends the remote control request to the second terminal equipment corresponding to the second equipment identifier, and establishing remote control of the first terminal equipment on the second terminal equipment, wherein the remote control request comprises the second equipment identifier of the second terminal equipment;

acquiring audio and video data from a preset position of a server, and playing the audio and video data in a rendering window;

And acquiring the mouse control message sent to the rendering window, and sending the mouse control message to the server so that the server sends the mouse control message to the second terminal equipment, and the second terminal equipment executes the operation corresponding to the mouse control message.

The technical scheme has the advantages that when a plurality of parties participate in a video conference, remote control connection can be directly established with the second terminal equipment, screen images of the second terminal equipment can be displayed on a rendering window in a pulling mode from a server, and a mouse control message can be sent to the second terminal equipment through the server, so that the purpose of remote control is achieved, other software is not required to be installed, and user operation is simplified.

In some embodiments, the mouse control message comprises a keyboard control message and a mouse control message, the controller performing retrieving the mouse control message sent to the rendering window and sending the mouse control message to the server, further configured to:

Under the condition that the rendering window is a focus window, keyboard control information is obtained by utilizing keyboard-mouse hook programming, the keyboard control information is sent to a server, and the focus window is used for representing a window which is being interacted with a user;

And under the condition that the rendering window is a focus window and the position indicated by the mouse is in the rendering window, obtaining a mouse control message by utilizing the programming of a keyboard-mouse hook, and sending the mouse control message to the server.

The technical scheme has the advantages that the keyboard control information is obtained by utilizing the keyboard-mouse hook programming, the implementation mode of a user interface of a rendering window is not relied on, when the keyboard control information is more, the processing quantity of the keyboard control information can be reduced, the execution efficiency is improved, and the portability of the hook programming is strong.

In some embodiments, after the first terminal device, the second terminal device, and the third terminal device join the same video conference, the controller is further configured to:

Responding to an instruction of the shared clipboard, and sending a shared clipboard content request to a server so that the server adds a first device identifier of first terminal equipment to a shared clipboard list;

And under the condition that the clipboard content is detected to be changed and the video conference application does not change the clipboard content, sending the clipboard content to a server, so that the server sends the clipboard content to target terminal equipment corresponding to the equipment identifier in the shared clipboard list, and the target terminal equipment changes the clipboard content.

The technical scheme has the advantages that when the clipboard content changes and is not changed due to video conference application, the clipboard content can be sent to the server and shared to the terminal equipment in the shared clipboard list by the server, so that the shared clipboard function is realized, and the user operation is facilitated.

In a third aspect, some embodiments of the present application provide a second terminal device, including:

A display configured to display a user interface;

A controller configured to:

In response to an instruction to create or join a conference, sending a conference request to a server, so that the server joins a second terminal device to the video conference based on the conference request, wherein the first terminal device, the second terminal device and the third terminal device join the same video conference;

Responding to an instruction for agreeing to the remote control request, sending an agreeing control message and audio-video data to a server, so that the server establishes remote control of the first terminal device to the second terminal device based on the agreeing control message, and storing the audio-video data to a preset position for the first terminal device and the third terminal device to acquire and play;

And receiving a mouse control message sent by the server, and executing an operation corresponding to the mouse control message, wherein the mouse control message is uploaded to the server by the first terminal equipment.

The technical scheme has the advantages that when a plurality of parties participate in a video conference, remote control connection can be directly established with the first terminal equipment, screen images can be uploaded to the server for streaming and playing of other participant equipment, the corresponding operation can be executed by receiving the keyboard and mouse control message sent by the first terminal equipment through the server, the purpose of remote control is achieved, other software is not required to be installed, and user operation is simplified.

Responding to the instruction of the shared clipboard, and sending a shared clipboard content request to a server so that the server adds a second device identifier of a second terminal device to a shared clipboard list;

In a fourth aspect, some embodiments of the present application provide a video conference remote control method, applied to a server, including:

In the technical scheme provided by the embodiment of the application, after the first terminal equipment, the second terminal equipment and the third terminal equipment join the same video conference, the first terminal equipment is received to send a remote control request, and the remote control request is sent to the second terminal equipment so as to establish the remote control of the first terminal equipment to the second terminal equipment. And if the mouse control message uploaded by the first terminal equipment is received, only the mouse control message is sent to the second terminal equipment, so that the second terminal equipment executes the operation corresponding to the mouse control message and does not send the mouse control message to the third terminal equipment. When multiple parties participate in a video conference, the server can directly establish remote control connection between at least two terminal devices, can store audio and video data sent by the controlled end for streaming of other devices, and can only send a mouse control message sent by the control end to the terminal device which establishes remote control connection with the mouse control message, so that the purpose of remote control is achieved, other software is not required to be installed, and user operation is simplified.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a timing diagram of a video conference remote control method according to some embodiments of the present application;

Fig. 2 is a flowchart of a method for remotely controlling a video conference according to some embodiments of the present application;

fig. 3 is a schematic diagram of a video conference page of a first terminal device according to some embodiments of the present application;

FIG. 4 is a schematic diagram of a screen sharing data transmission according to some embodiments of the present application;

fig. 5 is a schematic diagram of a video conference page of a first terminal device according to some embodiments of the present application;

Fig. 6 is a schematic diagram of a prompt page of a second terminal device according to some embodiments of the present application;

Fig. 7 is a schematic diagram of a video conference page of a second first terminal device according to some embodiments of the present application;

FIG. 8 is a flowchart of a method for processing keyboard and mouse events according to some embodiments of the present application;

FIG. 9 is a flowchart of a method for processing a hardware message according to some embodiments of the present application;

Fig. 10 is a schematic diagram of a video conference page of a third first terminal device according to some embodiments of the present application;

FIG. 11 is a diagram illustrating another screen sharing data transmission according to some embodiments of the present application;

FIG. 12 is a flow chart of a method for sharing clipboards according to some embodiments of the application;

Fig. 13 is a timing diagram of another video conference remote control method according to some embodiments of the present application.

Detailed Description

Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The embodiments described in the examples below do not represent all embodiments consistent with the application. Merely exemplary of systems and methods consistent with aspects of the application as set forth in the claims.

It should be noted that the brief description of the terminology in the present application is for the purpose of facilitating understanding of the embodiments described below only and is not intended to limit the embodiments of the present application. Unless otherwise indicated, these terms should be construed in their ordinary and customary meaning.

The terms first, second, third and the like in the description and in the claims and in the above-described figures are used for distinguishing between similar or similar objects or entities and not necessarily for describing a particular sequential or chronological order, unless otherwise indicated. It is to be understood that the terms so used are interchangeable under appropriate circumstances.

The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or apparatus that comprises a list of elements is not necessarily limited to all elements explicitly listed, but may include other elements not expressly listed or inherent to such product or apparatus.

The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware or/and software code that is capable of performing the function associated with that element.

In the embodiment of the present application, the terminal device is a device that inputs a program and data to a computer or receives a computer output processing result via a communication facility. The terminal equipment is usually installed in a convenient place where it can be connected to a remote computer by means of a communication facility, and is mainly composed of a communication interface control device and a dedicated or selected input-output device.

Terminal devices include, but are not limited to, smart televisions, mobile terminals, computers, wearable devices, virtual reality devices, augmented reality devices, and the like. Mobile terminals include, but are not limited to, notebook computers, tablet computers, smart phones, and the like.

The terminal equipment comprises control terminal equipment and controlled terminal equipment. The control terminal device generally refers to a terminal device that can control other terminal devices, i.e., a terminal device that sends control instructions to other terminal devices. The controlled terminal device generally refers to a terminal device that can be controlled by other terminal devices, i.e., a terminal device that receives control instructions sent by other terminal devices. The terminal device may be either a controlled terminal device or a control terminal device.

The terminal device may include a display, a communication means, a controller and a user input interface.

In some embodiments, the display includes a display function component for presenting a picture, and a drive component that drives the display of an image. The display is used for receiving and displaying the image signals output by the controller. For example, the display may be used to display video content, image content, and components of menu manipulation interfaces, user Interface (UI), and the like.

In some embodiments, the communication apparatus is a component for communicating with an external device or server according to various communication protocol types. The terminal device may be provided with a plurality of communication apparatuses according to different supported communication modes. For example, when the terminal device supports wireless network communication, the terminal device may be provided with a communication means including a WiFi function. When the terminal equipment supports bluetooth connection communication, the terminal equipment needs to be provided with a communication device comprising a bluetooth function.

The communication means may make the terminal device in communication connection with an external device or a server by means of a wireless or wired connection. The wired connection can connect the terminal device with the external device through components such as a data line, an interface and the like. The wireless connection may then connect the terminal device with an external device via a wireless signal or a wireless network. The terminal device may directly establish a connection with the external device, or may indirectly establish a connection through a gateway, a route, a connection device, and the like.

In some embodiments, the controller may include at least one of a central processor, a video processor, an audio processor, a graphic processor, a power supply processor, first to nth interfaces for input/output, and controls the operation of the terminal device and responds to the operation of the user through various software control programs stored on the memory. The controller controls the overall operation of the terminal device.

In some embodiments, the user input interface may be used to receive instructions from user input.

With the development of information technology, the appearance of online video conferences breaks through the space obstruction, improves the working efficiency of users, and is closely related to the work and life of the users. The video conference office software supports the function of sharing a computer desktop in a conference, so that the conference material can be watched together conveniently, the conference efficiency is improved, the meaning cannot be accurately expressed by language communication under partial scenes, a lot of time is spent on determining small problems such as text positions, the conference progress is dragged slowly, and the time of a user is wasted. To save user time, special remote assistance software may be selected to control the sharing person's computer for actual operation, such as adding remote desktop control functionality in a video conference.

In some embodiments, as shown in fig. 1, in the case of joining a video conference, the first control terminal device sends a first control request to the controlled terminal device after receiving an instruction of controlling the controlled terminal device input by a user. And the controlled terminal equipment establishes communication connection with the first control terminal equipment after receiving the first control request. After receiving a first control instruction input by a user to the controlled terminal equipment, the first control terminal equipment sends the first control instruction to the controlled terminal equipment so that the controlled terminal equipment executes an operation corresponding to the first control instruction. And the second control terminal equipment sends a second control request to the controlled terminal equipment after receiving an instruction for controlling the controlled terminal equipment by the user input. And the controlled terminal equipment disconnects the communication connection with the first control terminal equipment after receiving the second control request, and establishes the communication connection with the second control terminal equipment. After receiving a second control instruction input by a user to the controlled terminal equipment, the second control terminal equipment sends the second control instruction to the controlled terminal equipment so that the controlled terminal equipment executes an operation corresponding to the second control instruction.

In the above embodiment, only a single control terminal device is supported to remotely control a shared personal computer (controlled terminal device) at the same time, if other users have control requirements, the existing remote control must be ended first, then the other users are invited to carry out remote control again, the user operation is relatively complicated, and the controlled terminal devices cannot be controlled by multiple persons at the same time.

In order to realize remote control in video conference and simplify user operation, the embodiment of the application provides a server. The server comprises a communication module and a processing module. As shown in fig. 2, the processing module performs the following steps:

step S201, receiving conference requests sent by the first terminal device, the second terminal device and the third terminal device respectively, so that the first terminal device, the second terminal device and the third terminal device join the same video conference.

In some embodiments, the first, second, and third terminal devices respectively respond to an instruction to launch a video conferencing application, displaying a video conferencing home page comprising a create conference control and a join conference control.

After receiving the confirmation operation of the user to the creation of the conference control, namely, the instruction of creating the conference, the first terminal device, the second terminal device or the third terminal device sends a conference creation request to the server. The server creates a video conference according to the conference creation request, and issues page data to the first terminal device, the second terminal device or the third terminal device so that the video conference page corresponding to the page data is displayed.

In some embodiments, the audio call data of each participant terminal and/or the video call data collected by the camera component are uploaded to a room corresponding to the video conference, and cached, and each terminal pulls the audio and video call data of the opposite terminal according to the identification of the opposite terminal in the video conference, so as to obtain the audio data and/or the video data of the opposite terminal for playing.

After receiving confirmation operation of the user to join the conference control, namely, a conference joining instruction, the first terminal device, the second terminal device or the third terminal device displays a conference joining page, wherein the conference joining page comprises a conference number input box. After receiving the conference number input by the user in the conference box, sending a conference joining request to the server, wherein the conference request comprises the conference number. The server enables the first terminal equipment, the second terminal equipment or the third terminal equipment to join in the video conference corresponding to the conference number, and issues page data to the first terminal equipment, the second terminal equipment or the third terminal equipment so that the video conference page corresponding to the page data can be displayed.

In other embodiments, after receiving the confirmation operation of the user on the video conference link in a certain page, the first terminal device, the second terminal device, and the third terminal device may start the video conference application and jump to the video conference page corresponding to the video conference link.

The first terminal device, the second terminal device and the third terminal device may join the same video conference via the same conference number or the same video conference link.

Step S202, a remote control request sent by the first terminal equipment is received, and the remote control request is sent to the second terminal equipment corresponding to the second equipment identifier, so that remote control of the first terminal equipment to the second terminal equipment is built.

Step S203, under the condition that the remote control of the first terminal equipment to the second terminal equipment is established, if the audio and video data uploaded by the second terminal equipment are received, the audio and video data are stored to a preset position for the first terminal equipment and the third terminal equipment to acquire and play. The audio and video data are screen images intercepted by the second terminal equipment at intervals of preset time length.

In some embodiments, after the second terminal device initiates the screen sharing, the first terminal device may send a remote control request, and send the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the first terminal device on the second terminal device.

The second terminal equipment initiates the screen sharing step, which comprises responding to the instruction of screen sharing input by the user, intercepting the screen images at intervals of preset time length and uploading the screen images to the server in a video stream mode. The server stores the video stream to a preset position for the first terminal device and the third terminal device to acquire and play. The video conference page comprises a screen sharing control, and can receive a screen sharing instruction input by a user for confirming operation of the screen sharing control.

Illustratively, as shown in FIG. 3, the videoconference page includes a attendee control 31 and a toolbar 32. The participant control 31 may display the image data collected by the participant's corresponding account name, account header, and/or the terminal device image collection device used by the participant. Toolbar 32 includes mute control 321, open video control 322, and screen share control 323. When a confirmation operation of the mute control 321 is received, so that the mute control 321 is changed into a non-opened state, the sound collection device is started, and the audio call data collected by the sound collection device is sent to the server. The terminal equipment of other participants acquires the audio call data from the server and plays the audio call data. When a confirmation operation of the mute control 321 is received, so that the mute control 321 is changed to an on state, the sound collecting device is turned off.

When a confirmation operation of the opening video control 322 is received, so that the opening video control 322 is changed into an opening state, the image acquisition device is started, and video call data acquired by the image acquisition device are sent to the server. The terminal devices of other participants acquire the video call data from the server, and display images corresponding to the video call data in the image display areas on the participant controls 31 corresponding to the video call data transmitting ends. Upon receiving a confirmation operation to open the video control 322, the image capture device is turned off when the open video control 322 is changed to the off state.

It should be noted that, the terminal devices of the participants may package the image data collected by the image collecting device and/or the audio data collected by the sound collecting device into a call video stream and/or a call audio stream, and transmit the call video stream and/or the call audio stream to the server through a Real-time transmission protocol (Real-time Transport Protocol, RTP). The terminal devices of other participants acquire call video streams and/or call audio streams from preset positions of the server through a real-time transmission protocol, then decode, render the call video streams in windows corresponding to the identifiers of the call video stream sending ends in the video conference pages, and/or play audio so as to know images and sounds shared by the terminal devices of other participants in real time.

The screen sharing control 323 is used for triggering whether to share the screen to the participants, and can receive clicking or selecting operation of the screen sharing control 323 by the user.

As shown in fig. 4, after receiving the confirmation operation of the screen sharing control, the second terminal device intercepts the screen image at intervals of a preset duration, packages the screen image into a shared screen video stream, and transmits the shared screen video stream to the server through a real-time transmission protocol. The first terminal device and the third terminal device acquire the shared screen video stream from the server through a real-time transmission protocol, then decode and render the shared screen video stream in a rendering window shared by the screen in the video conference page, and watch the screen image shared by the second terminal device in real time.

In some embodiments, after receiving the confirmation operation of the user on the screen sharing control 323, the second terminal device may directly display the terminal screen display page, package the terminal screen display page into a shared screen video stream, upload the shared screen video stream to the server, and pull and play the shared screen video stream by the server for the first terminal device and the third terminal device.

In other embodiments, after receiving the confirmation operation of the user on the screen sharing control 323, the second terminal device may acquire a background page corresponding to the application running in the background, and display a page control corresponding to the background page on the video conference page. After receiving the page control selected by the user, displaying a background page corresponding to the page control on a foreground, packaging a terminal screen display page at the moment into a shared screen video stream, uploading the shared screen video stream to a server, and pulling and playing the shared screen video stream by the server for the first terminal equipment and the third terminal equipment.

In some embodiments, when the second terminal device does not share the screen, the first terminal device and the third terminal device only need to pull the call video stream and/or the call audio stream from the server, and display an image corresponding to the call video stream and/or play an audio corresponding to the call audio stream in a display area corresponding to the call video stream transmitting end in the video conference page.

In some embodiments, after the second terminal device shares the screen, the first terminal device and the third terminal device need to pull the shared screen video stream in addition to the shared screen talk video stream and/or talk audio stream from the server. The shared screen video stream is used to display a screen image of the second terminal device in a screen-shared rendering window in the video conference page.

After the first terminal device displays the video conference page in a video conference number or video conference link mode, the shared screen video stream uploaded by the second terminal device is pulled from the server, a rendering window of screen sharing is displayed in the video conference page, and an image corresponding to the shared screen video stream, namely, a screen image of the second terminal device is displayed in the rendering window.

Rendering the window includes requesting a control. After receiving the confirmation operation of the user on the request control, the first terminal device sends a remote control request to the server, wherein the remote control request comprises a second device identifier of the second terminal device. And the server sends the remote control request to the second terminal equipment corresponding to the second equipment identifier. The remote control request may further include a name of a participant corresponding to the first terminal device, and may be used to display a participant who wants to control the device at the same time when the control prompt information is displayed on the second terminal device.

In some embodiments, the second terminal device may directly send the consent connection request to the server after receiving the remote control request sent by the first terminal device. The server may directly establish a remote control of the first terminal device to the second terminal device.

In other embodiments, the second terminal device displays the consent control in the videoconference page after receiving the remote control request sent by the first terminal device. And after receiving the confirmation operation of the consent control, sending a consent connection request to the server. The server may directly establish a remote control of the first terminal device to the second terminal device.

Illustratively, as shown in fig. 5, the video conference page of the first terminal device includes a rendering window 51 and a request control 52. Upon receiving a confirmation operation from the user to the request control 52, a remote control request is sent to the server and sent by the server to the second terminal device. As shown in fig. 6, the prompt page of the second terminal device includes prompt information 61, consent control 62, and rejection control 63. A confirmation operation of the consent control 62 by the user may be received and a consent connection request sent to the server. The server may directly establish a remote control of the first terminal device to the second terminal device.

In still other embodiments, after sharing the screen image or assigning the background page to the first terminal device and the third terminal device of the participant, the second terminal device may receive an instruction from the user to switch to the video conference page on which the participant information is displayed, and display the participant control on the video conference page. After receiving the confirmation operation of the meeting personnel control, displaying a personnel setting frame, wherein the personnel setting frame comprises a controlled control. After receiving the confirmation operation of the controlled control, the controlled control request is sent to the first terminal equipment corresponding to the participant control. The first terminal device may send an approval message to the server upon receiving the instruction to approve the controlled request, so that the server establishes remote control of the second terminal device by the first terminal device.

In other embodiments, after joining the video conference, the first terminal device may directly send the remote control request without the second terminal device initiating screen sharing, and send the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the first terminal device on the second terminal device.

The first terminal device can display a personnel setting frame after receiving confirmation operation of the meeting personnel control in the video conference page, wherein the personnel setting frame comprises a controlled request control and a control request control. And after receiving the confirmation operation of the request control, sending a remote control request to the server. The remote control request includes a second device identifier of the participant control corresponding to the second terminal device. And the server sends the remote control request to the second terminal equipment corresponding to the second equipment identifier. The second terminal equipment receives the instruction of agreeing to the controlled request, sends an agreeing message to the server, and judges whether to share the screen. If the screen is not shared, intercepting the screen image at intervals of preset duration and sending the screen image to the server in a video stream mode. The server establishes remote control of the first terminal device to the second terminal device after receiving the consent message. The server stores the received video stream to a preset position for the first terminal device and the third terminal device to pull and play.

After receiving the confirmation operation of the controlled control, the first terminal device sends a controlled request to the server, wherein the controlled request comprises a second device identifier of the second terminal device corresponding to the participant control. And the server sends the application controlled request to the second terminal equipment corresponding to the second equipment identifier. At this time, the first terminal device is a controlled device, and the second terminal device is a control device. After receiving the instruction of agreeing to the control request, the second terminal equipment sends an agreeing message to the server, the server establishes remote control of the second terminal equipment on the first terminal equipment and informs the first terminal equipment, the first terminal equipment judges whether to share the screen, if not, screen images are intercepted at intervals of preset duration and sent to the server in a video stream mode, and therefore the server sends video streams to the second terminal equipment and the third terminal equipment.

Illustratively, as shown in FIG. 7, the personnel settings box 71 of the first terminal device includes a request controlled control 711, a request control 712, a share clipboard control 713, and a move out conference control 714. After receiving the confirmation operation of the request controlled control 711, the server sends the request controlled to the server, and the server sends the request controlled to the second terminal device corresponding to the participant control, so as to establish remote control of the second terminal device on the first terminal device. After receiving the confirmation operation to the request control 712, a remote control request is sent to the server, and the server sends the remote control request to the second terminal device corresponding to the participant control, so as to establish the remote control of the first terminal device to the second terminal device.

In some embodiments, the server establishes remote control of the second terminal device by the first terminal device via Websocket protocol. The server can utilize the Websocket protocol to carry out remote control, the first terminal device establishes a Websocket long connection with the server, the user name of the second terminal device is required to be added into the sent message, and the server carries out directional forwarding according to the user name contained in the message. However, the overhead of the method is high, the efficiency is low, the priority of long connection transmission is far higher than that of real-time transmission protocol, and under the condition of poor network conditions, namely video conference blocking, the control instruction transmitted by the first terminal equipment through the long connection is not affected by the network and is normally transmitted to the second terminal equipment, and the latest screen picture of the second terminal equipment can not be transmitted to the first terminal equipment, so that the control instruction is invalid or is not matched with the intention of a user.

In some embodiments, to address the problems of high overhead, inefficiency, and mismatch of control instructions and user intent, the server establishes remote control of the second terminal device by the first terminal device through a stream control transmission protocol (Stream Control Transmission Protocol, SCTP) tunnel. The server may establish a plurality of first terminal devices and stream control transmission protocol tunnels of the virtual video conference room in the server, and a second terminal device and stream control transmission protocol tunnels of the virtual video conference room according to the remote control request. The first terminal equipment needs to add the equipment identifier of the second terminal equipment in the transmitted message, and uploads the message to a message queue corresponding to the second equipment identifier of the second terminal equipment in the virtual video conference room through the stream control transmission protocol tunnel, and the server reads the message in the message queue corresponding to the second equipment identifier and transmits the message to the second terminal equipment, so that the directional forwarding to the second terminal equipment is realized and the directional forwarding to the third terminal equipment is not realized. The stream control transmission protocol tunnel may transmit a key mouse control message of the first terminal device to the second terminal device.

A stream control transmission protocol is a protocol that transmits multiple data streams simultaneously between two ends of a network connection. SCTP, as a transport layer protocol, can be understood as a protocol similar to TCP (Transport Control Protocol, transmission control protocol) and UDP (User Datagram Protocol ), which provides services like TCP while combining some of the advantages of UDP. SCTP is a data transfer protocol that provides reliability, efficiency, and ordering.

It should be noted that, in the embodiment of the present application, the second terminal device may simultaneously receive remote control requests sent by the plurality of first terminal devices, that is, simultaneously establish remote control of the plurality of first terminal devices on the second terminal device, and may establish remote control with the new terminal device without disconnecting the current remote control.

Step S204, if the mouse control message uploaded by the first terminal equipment is received, the mouse control message is only sent to the second terminal equipment, so that the second terminal equipment executes the operation corresponding to the mouse control message, and the mouse control message is not sent to the third terminal equipment.

The keyboard and mouse control messages include a keyboard control message (keyboard event) and a mouse control message (mouse event). The keyboard and mouse event processing flow of the terminal device may be as shown in fig. 8. After receiving the operation of the user on the mouse and the keyboard, generating a hardware event, storing the hardware event into a system message queue, further distributing the system event to the message queues of all application programs, and sequentially processing all the messages by the application programs to trigger response events such as keyboard input, mouse movement, mouse clicking and the like.

In some embodiments, after the first terminal device receives the hardware message sent to the rendering window, the window callback function of the rendering window judges whether the hardware message is a mouse control message, if the hardware message is a mouse control message, the mouse control message is analyzed and converted, then the analyzed and converted message is sent to the remote control module, and the remote control module sends the server through the stream control transmission protocol tunnel. The server sends the mouse control message only to the second terminal device.

However, the analysis of keyboard and mouse events depends on the UI implementation of the rendering window, such as Microsoft base class library (Microsoft Foundation Classes, MFC), and the application developed by the cross-platform application development framework (Qt), which are different. In addition, the processing manner of the UI implementation of the rendering window is executed serially, that is, each piece of hardware information needs to be processed in turn, resulting in low execution efficiency.

In some embodiments, in order to not depend on the UI implementation manner of the rendering window and improve the execution efficiency, the method for acquiring the keyboard and mouse control message sent to the rendering window by using the hook programming technology may include that the first terminal device acquires the keyboard control message by using the keyboard and mouse hook programming and sends the keyboard control message to the server when the rendering window is a focus window, wherein the focus window is used for representing a window which is interacting with a user;

Hook programming (hooking), also known as "hooking" technology, is a computer programming term that refers to various techniques for modifying or extending the behavior of an operating system, application, or other software component by intercepting function calls, messaging, event transfers between software modules, and code that handles the intercepted function calls, events, messages, known as hooks (hooks).

As shown in fig. 9, an embodiment of the present application adds a key mouse hook process to the rendering window of a video conferencing application. The keymouse hook takes over all hardware messages that the system sends to the rendering window and determines if the hardware message is a mouse or keyboard event. And if the hardware message is a mouse or keyboard event, analyzing and processing the mouse or keyboard event or transmitting the mouse or keyboard event to a remote control module for analyzing and processing. If the hardware information is not a mouse or keyboard event, the hardware information is still transferred to the rendering window.

A keyboard event typically contains information about the virtual key code, which indicates which key is activated (e.g., key of a keyboard such as X, enter, esc), key status, which indicates whether the current key is in a pressed or sprung state, etc. By analyzing the keyboard event, two pieces of information including a virtual key code and a key state are extracted and sent to the second terminal equipment, and the second terminal equipment converts the information into the keyboard event for execution, so that the effect of remote keyboard control is realized.

A mouse event typically includes information such as a cursor position, which represents the coordinates of the mouse within the display, a virtual key code, which represents the mouse key (e.g., left, right, center key) that triggered the event, a key state indicating whether the current mouse key is in a pressed or sprung state, and a key state, which is null in the event if a user operation is received to move only the mouse. Rendering is typically performed on the premise of ensuring the aspect ratio of the shared picture in order to ensure the viewing experience, and at this time, the picture of the controlled terminal device does not fill the entire rendering window, as shown in fig. 5. According to the position coordinates of the rendering window, the position of the mouse relative to the rendering window can be calculated, further according to the coordinates of the actual rendering area, the position of the mouse relative to the rendering picture can be calculated, and finally according to the proportion of the rendering picture to the original picture (such as rendering of the original 1920x1080 picture in the 640x360 area), the position of the mouse relative to the picture of the controlled terminal equipment can be calculated. And the converted cursor position, virtual key code and key state information are sent to the second terminal equipment, and the second terminal equipment converts the information into mouse event execution according to the information, so that the effect of remote mouse control is realized.

In some embodiments, after the server establishes remote control of the first terminal device to the second terminal device, the upper layer of the terminal page of the second terminal device may display a control column, and display, on the control column, a control of a participant corresponding to at least one login user account of the first terminal device, so as to prompt the user about which participants are allowed to control the terminal.

After the second terminal device receives the mouse control message, the first terminal device corresponding to the mouse control message can be determined to send the first terminal device, and the first terminal device is displayed on a meeting participant control corresponding to the first terminal device, so that a user is prompted which meeting participant currently controls the second terminal device.

Illustratively, as shown in fig. 10, when the second terminal device shares the screen, a control bar 101 is displayed at the upper layer of the terminal page, and the control bar 101 includes a first participant control 1011 and a second participant control 1012. The participant user a corresponding to the first participant control 1011 is currently indicated to be controlling the second terminal device.

It should be noted that, when the hook programming receives a plurality of mouse control messages within a preset time period, the hook deletes a plurality of mouse control messages before the last mouse control message, thereby reducing the number of processing the mouse control messages and improving the execution efficiency. For example, during the movement of the mouse, the calculation of the mouse position can be reduced and only the last mouse position within the preset time period is reserved. Hook programming is highly portable.

The first terminal equipment sends the mouse control message to the server, the server stores the mouse control message to a mouse message queue, and simultaneously, the server sequentially reads the mouse control message in the mouse message queue and sends the mouse control message to the second terminal equipment.

After the first terminal device obtains the key mouse control message sent to the rendering window, the data type, the sender identifier, namely the first device identifier of the first terminal device and the appointed receiver identifier, namely the second device identifier of the second terminal device, are added into the key mouse control message, and are sent to the virtual video conference room of the server through the established stream control transmission protocol tunnel. The data type is a mouse control message.

After receiving the data sent by the first terminal device, the server first identifies the data type. If the data type is the mouse control message, the appointed receiving end identification is identified, whether the appointed receiving end identification is the equipment identification of the virtual video conference room or not is judged, if the appointed receiving end identification is not the equipment identification of the virtual video conference room, the mouse control message is abandoned, and if the appointed receiving end identification is the equipment identification of the virtual video conference room, the mouse control message is placed into a mouse message queue corresponding to the appointed receiving end identification, namely the second equipment identification of the second terminal equipment.

After each terminal device joins the virtual video conference room, the device identifier of each terminal device is stored in a device list of the virtual video conference room, and whether the designated receiving end identifier is the device identifier of the virtual video conference room can be determined by judging whether the designated receiving end identifier is in the device list.

Once the server detects that the key mouse control message is stored in the key mouse message queue, the server reads the key mouse control message in the key mouse message queue and sends the read key mouse control message to the terminal equipment corresponding to the equipment identifier of the key mouse message queue through the established stream control transmission protocol tunnel. For example, the second device-identified keymouse control message in the keymouse message queue is sent to the second terminal device.

In some embodiments, in a case where a network condition is poor or a plurality of first terminal devices send a mouse control message at the same time, a situation that a plurality of mouse control messages cannot be sent out in time in a mouse message queue is backlogged, and a specific implementation manner of sending the mouse control message to the second terminal device by the server may include:

Illustratively, the first preset difference is 500ms, there are 10 keymouse control messages in the keymouse message queue, keymouse control message 1, keymouse control message 2, &. If the difference between the receiving time of the mouse control message 1 and the receiving time of the mouse control message 10 are greater than 500ms, deleting the mouse control message 1, and then judging whether the difference between the receiving time of the mouse control message 2 and the receiving time of the mouse control message 10 are greater than 500ms, wherein no new mouse control message is stored in the mouse message queue. If the difference in reception time between the keypad control message 2 and the keypad control message 10 is less than 500ms, the keypad control message 2 may be transmitted to the second terminal device.

According to the embodiment of the application, under the condition that network is blocked or a plurality of first terminal devices send the mouse control messages at the same time to cause excessive mouse control messages in the mouse message queue, the mouse control messages positioned at the head of the queue can be deleted and not sent to the second terminal device, so that the phenomenon that the second terminal device is overloaded in a short time when recovering the network is avoided.

In some embodiments, in a case where a plurality of first terminal devices send the mouse control messages simultaneously, the mouse message queue may include a plurality of mouse control messages, and one embodiment of the server sending the mouse control messages to the second terminal device may include:

And determining the first mouse control message as the mouse control message with the receiving time difference of the mouse control message at the head of the mouse queue smaller than the second preset difference.

The first mouse control message may or may not include a mouse control message located at the head of the mouse queue. And under the condition that the first mouse control message does not comprise the mouse control message positioned at the head of the mouse queue, the mouse control message positioned at the head of the queue is sent to the second terminal equipment, and then the second mouse control message is sent to the second terminal equipment.

Illustratively, the second preset difference is 100ms, there are 10 keymouse control messages in the keymouse message queue, keymouse control message 1, keymouse control message 2, &. The priority of the first terminal equipment is respectively two-level, one-level and three-level, the priority of the mouse control message 1 is three-level, the mouse control message 1, the mouse control message 2 and the mouse control message 4 are deleted, and the mouse control message 3 is sent to the second terminal equipment. And then continuing to acquire the mouse control message with a reception time difference of less than 100ms from the mouse control message at the head of the team.

The priority of the first mouse control message is determined by a first terminal device of a transmitting end included in the first mouse control message, and the priority of the first terminal device can be selected in a video conference page by a second terminal device.

For example, as shown in fig. 10, priority levels are displayed on the first participant control 1011 and the second participant control 1012 in the control column 101, and after receiving the confirmation operation of the priority setting control 1013, the priorities of the first participant and the second participant may be changed, that is, the priorities of the first terminal devices corresponding to the first participant and the second participant may be changed.

And reading the mouse control message at the head of the queue of the mouse message corresponding to the second equipment identifier, and then judging whether a fourth mouse control message with the receiving time difference of the mouse control message at the head smaller than a third preset difference exists. And if the fourth mouse control message does not exist, transmitting the fourth mouse control message to the second terminal equipment. And deleting the fifth mouse control message if the fourth mouse control message exists, and sending the fourth mouse control message with the latest receiving time to the second terminal equipment, wherein the fifth mouse control message is a mouse control message except for the fourth mouse control message with the latest receiving time.

For example, the third preset difference is 50ms, there are 10 keymouse control messages in the keymouse message queue, keymouse control message 1, keymouse control message 2, &. If there is no mouse control message having a reception time difference of less than 50ms from the mouse control message 1, the mouse control message 1 is transmitted to the second terminal device. If the receiving time difference between the mouse control message 1 and the receiving time of the mouse control message 1 is less than 50ms and the receiving time is sequentially the mouse control message 2, the mouse control message 3 and the mouse control message 4, deleting the mouse control message 1, the mouse control message 2 and the mouse control message 3, and sending the mouse control message 4 to the second terminal device. And then continuing to judge whether there is a mouse control message with a time difference of less than 50ms from the reception time of the mouse control message at the head of the team.

In some embodiments, as shown in fig. 11, the image data and the audio data are transmitted in the same manner as in fig. 4, and will not be described here again. The first terminal device sends the key mouse control message to the server through the stream control transmission protocol, and sends the key mouse control message to the terminal device sharing the screen, namely the second terminal device through the server.

In some embodiments, in the process of sharing the screen by the second terminal device, the first terminal device may need to display the specified text or picture on the screen of the second terminal device, and may select to input the specified text or picture to the chat area of the instant messaging software or the public chat area in the video conference, so that the specified text or picture is pasted to the specified area from the chat area of the second terminal device, which is tedious for the user to operate.

In some embodiments, to simplify user operation, clipboards may be shared. And the first terminal equipment, the second terminal equipment and/or the third terminal equipment send a shared clipboard content request to the server after receiving the confirmation operation of the shared clipboard control in the video conference page. The shared clipboard content request includes a device identification. The server adds the first device identifier of the first terminal device, the second device identifier of the second terminal device and/or the third device identifier of the third terminal device to the shared clipboard list.

If the third terminal device does not establish a stream control transmission protocol tunnel with the virtual video conference room, the stream control transmission protocol tunnel between the third terminal device and the virtual video conference room needs to be established after the third device identifier is added to the shared clipboard list, so as to transmit clipboard contents.

Illustratively, in fig. 7, the first terminal device, upon receiving a confirmation operation for the shared clipboard control 713, sends a shared clipboard content request to the server. The server adds the first device identification of the first terminal device to the shared clipboard list.

As shown in FIG. 12, one embodiment of a shared clipboard may include a first terminal device, a second terminal device, or a third terminal device that may determine whether the clipboard content has changed. If the video conference application does not change the clipboard content, adding the data type to the clipboard content, and then sending the data type to a server through a stream control transmission protocol tunnel, wherein the data type is the clipboard content. After the server identifies that the uploaded data type is the clipboard content, the clipboard content is sent to target terminal equipment corresponding to the equipment identifier in the shared clipboard list, and the target terminal equipment changes the clipboard content. If the video conference application changes the clipboard content, then continuing to determine if the clipboard content has changed.

The clipboard is a continuous memory space which can be changed according to the size of stored information in the system and is used for temporarily storing and exchanging information, and different application programs running on the system share the same clipboard. A hidden window may be created at each of the first, second and third terminal devices to monitor changes in clipboard content, and the window may receive a message when the clipboard content changes. If the content of the clipboard is not changed by the video conference application, which means that the content of the clipboard is completed by the user operation, the content of the clipboard needs to be extracted and sent to a server through a stream control transmission protocol tunnel, and the server forwards the content to other terminal devices. If the video conference application changes the content of the clipboard, the content of the clipboard transmitted by other terminal equipment is changed, so that the content of the clipboard is not required to be transmitted to other terminal equipment, and monitoring can be continued. And after the first terminal equipment, the second terminal equipment or the third terminal equipment receives the clipboard content forwarded by the server, writing the clipboard content into the memory space of the system clipboard, thereby realizing the function of sharing the clipboard.

Illustratively, in fig. 5, upon receiving a confirmation operation of the shared clipboard control 53 by the user, a clipboard content sharing function of the first terminal device, the second terminal device, and the third terminal device participating in the same video conference may be implemented.

When the content of the clipboard is a file, the file is sliced and divided, slices are sequentially transmitted through a stream control transmission protocol tunnel, the integrity check is carried out after all the slices are received by a receiving end, and then the final file is synthesized, wherein the file comprises pictures, documents, audio, videos and the like.

In some embodiments, after receiving the confirmation operation of the shared clipboard control in the video conference page, the second terminal device may share the clipboard content with the first terminal device that has been remotely controlled, that is, the first device identifier of the first terminal device is automatically added to the shared clipboard list, or may select the participant sharing the clipboard, and if the terminal device of the participant selected by the user does not establish the flow control transmission protocol tunnel with the virtual video conference room, the flow control transmission protocol tunnel needs to be established first.

In other embodiments, after receiving the confirmation operation of the shared clipboard control in the video conference page, the second terminal device may send a request for sharing the clipboard to the first terminal device that has established remote control, and after the first terminal device agrees to the request for sharing the clipboard, automatically add the first device identifier of the first terminal device to the list of sharing the clipboard, so that the clipboard content may be shared with the first terminal device.

In still other embodiments, after receiving the confirmation operation of the shared clipboard control in the video conference page, the first terminal device and the third terminal device may default to share clipboard content with the second terminal device, and may set the device identifiers of the first terminal device and the third terminal device to the shared clipboard list of the second terminal device after the second terminal device agrees to the sharing request. If the owner of the shared clipboard list is the second terminal device, the clipboard content uploaded by the device identifier in the shared clipboard list may only be shared to the second terminal device, and the clipboard content uploaded by the second terminal device may be shared to the terminal devices corresponding to the device identifiers in all the shared clipboard list.

In some embodiments, after receiving the confirmation operation of the shared clipboard control in the video conference page, the upper layer of the terminal page of the second terminal device may display a clipboard, and display, on the clipboard, a participant control corresponding to at least one first terminal device login user account number, which participants may be prompted to control terminal clipboard sharing with the user.

After receiving the clipboard content sent by the server, the second terminal device can determine the sending first terminal device corresponding to the clipboard content and display the first terminal device on the meeting personnel control corresponding to the sending first terminal device so as to prompt the user which terminal device of the meeting personnel the current clipboard content is shared.

Illustratively, as shown in fig. 10, when the second terminal device shares the screen, a clipboard 102 is displayed at the upper layer of the terminal page, and the clipboard 102 includes a first participant control 1021 and a second participant control 1022. The control terminal device currently instructing the participant user a to share clipboard content.

In some embodiments, a timing diagram of a remote control method in a video conference may be as shown in fig. 13. The first terminal device, the second terminal device and the third terminal device respectively send conference requests, and the server enables the first terminal device, the second terminal device and the third terminal device to join the same video conference based on the conference requests. The first terminal equipment receives a control instruction of the first user to the second terminal equipment, sends a remote control request to the server, and sends the remote control request to the second terminal equipment by the server, and the second terminal equipment displays a prompt page of the first user request control, wherein the prompt page comprises an agreement control. And the second terminal equipment receives the confirmation operation of the second user on the consent control, namely the consent control instruction, and sends a consent message to the server. The server establishes remote control of the first terminal equipment to the second terminal equipment through the SCTP tunnel. The second terminal equipment intercepts screen images at intervals of preset time length and forms video streams to be sent to the server, the server stores the video streams to preset positions, and informs the first terminal equipment and the third terminal equipment to pull the video streams from the preset positions and display images corresponding to the video streams in rendering windows of the video conference interface.

The first terminal equipment obtains the mouse control message by using hook programming, and sends the mouse control message to the server through the stream control transmission protocol tunnel, and the server processes the mouse control message and sends the mouse control message to the second terminal equipment, wherein the processing comprises deleting part of the mouse control message when the mouse control message is more in the mouse message queue. And the second terminal equipment executes the operation corresponding to the keyboard and mouse control message.

After the first terminal equipment and the second terminal equipment respectively receive the shearing plate sharing instruction input by the first user and the second user, a sharing request is sent to a server, and the server adds the first equipment identifier of the first terminal equipment and the second equipment identifier of the second terminal equipment in the sharing request to a shearing plate sharing list. And the second terminal equipment judges whether the clipboard content is changed, if so, judges whether the video conference application changes the clipboard content, and if not, the second terminal equipment sends the clipboard content to the server through the stream control transmission protocol tunnel, and the server sends the clipboard content to the first terminal equipment. The first terminal device alters the clipboard content. After the first terminal device executes the operation corresponding to the sharing instruction, the first terminal device also executes the steps of judging whether the clipboard content is changed or not and the follow-up steps, so that the clipboard content of the first terminal device is shared to the second terminal device.

The embodiment of the application provides a multi-user remote desktop control function based on a video conference, and simultaneously comprises the functions of clipboard sharing and file transmission, and the collaborative operation picture is synchronized to other users in the conference in real time, so that the multi-user remote desktop control function is suitable for application scenes such as online office work and teaching.

Some embodiments of the present application also provide a computer-readable storage medium, which may store a program. When the computer storage medium is configured in a display device or a server, the program may include program steps involved in the video conference remote control method in the above embodiment when executed. The computer storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), or the like.

An embodiment of the application provides an electronic device, which comprises a processor and a memory for storing instructions executable by the processor. The processor is configured to read the executable instructions from the memory and execute the instructions to implement the video conference remote control method in the above embodiment.

It should be noted that the above embodiments are merely for illustrating the technical solution of the present application and not for limiting the same, and although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solution described in the above embodiments may be modified or some or all of the technical features may be equivalently replaced, and these modifications or substitutions do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiments of the present application.

The foregoing description, for purposes of explanation, has been presented in conjunction with specific embodiments. The illustrative discussions above are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed above. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles and the practical application, to thereby enable others skilled in the art to best utilize the embodiments and various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. A server, comprising:

Communication module;

The processing module is configured as follows:

Receiving conference requests respectively sent by a first terminal device, a second terminal device, and a third terminal device, and enabling the first terminal device, the second terminal device, and the third terminal device to join the same video conference;

receiving a remote control request sent by the first terminal device, the remote control request including a second device identifier of the second terminal device, and sending the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by the first terminal device;

In the case of establishing remote control of the second terminal device by the first terminal device, if audio and video data uploaded by the second terminal device is received, the audio and video data is stored in a preset location for the first terminal device and the third terminal device to obtain and play;

If a keyboard and mouse control message uploaded by the first terminal device is received, the keyboard and mouse control message is only sent to the second terminal device so that the second terminal device performs the operation corresponding to the keyboard and mouse control message, and the keyboard and mouse control message is not sent to the third terminal device.

2. The server according to claim 1, wherein the processing module executes sending the keyboard and mouse control message to the second terminal device, and is further configured to:

Reading a keyboard and mouse control message at the head of a keyboard and mouse message queue corresponding to the second device identifier, where the keyboard and mouse control message in the keyboard and mouse message queue is uploaded by the first terminal device;

When the difference between the reception time of the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, sending the keyboard and mouse control message at the head of the keyboard and mouse message queue to the second terminal device corresponding to the second device identifier;

When the difference in reception time between the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is greater than a first preset difference, delete the keyboard and mouse control message at the head of the keyboard and mouse message queue, and continue to read the next keyboard and mouse control message at the head of the keyboard and mouse message queue, until the difference in reception time between the keyboard and mouse control message at the end of the keyboard and mouse message queue and the keyboard and mouse control message at the head of the keyboard and mouse message queue is less than or equal to the first preset difference, and then send the keyboard and mouse control message at the head of the keyboard and mouse message queue to the second terminal device corresponding to the second device identifier.

3. The server according to claim 1, wherein the processing module executes sending the keyboard and mouse control message to the second terminal device, and is further configured to:

Determine a keyboard and mouse control message whose reception time difference with the keyboard and mouse control message at the head of the keyboard and mouse queue is less than a second preset difference as a first keyboard and mouse control message;

Obtaining a first device identifier in the first keyboard and mouse control message and a priority corresponding to the first device identifier, and determining the first keyboard and mouse control message with the highest priority as the second keyboard and mouse control message, and determining the first keyboard and mouse control messages other than the second keyboard and mouse control message as the third keyboard and mouse control message, wherein the first device identifier is the device identifier of the first terminal device;

The third keyboard and mouse control message is deleted, and the second keyboard and mouse control message is sent to the second terminal device corresponding to the second device identifier.

4. The server according to claim 1, wherein after the first terminal device, the second terminal device and the third terminal device join the same video conference, the processing module is further configured to:

receiving a request for sharing clipboard content sent by the first terminal device, the second terminal device and/or the third terminal device, and adding the first device identification of the first terminal device, the second device identification of the second terminal device and/or the third device identification of the third terminal device to a shared clipboard list;

When receiving the clipboard content sent by the first terminal device, the second terminal device or the third terminal device, the clipboard content is sent to the target terminal device corresponding to the device identifier in the shared clipboard list to change the clipboard content of the target terminal device.

5. A first terminal device, comprising:

A display; configured to display a user interface;

The controller is configured as:

In response to an instruction to create or join a conference, sending a conference request to the server, so that the server enables the first terminal device to join a video conference based on the conference request, wherein the first terminal device, the second terminal device, and the third terminal device join the same video conference;

In response to an instruction to control the second terminal device, sending a remote control request to the server, so that the server sends the remote control request to the second terminal device corresponding to the second device identifier, so as to establish remote control of the second terminal device by the first terminal device, wherein the remote control request includes the second device identifier of the second terminal device;

Acquire audio and video data from a preset position of the server, and play the audio and video data in a rendering window;

Acquire a keyboard and mouse control message sent to the rendering window, and send the keyboard and mouse control message to the server, so that the server sends the keyboard and mouse control message to the second terminal device, and the second terminal device performs an operation corresponding to the keyboard and mouse control message.

6. The first terminal device according to claim 5, wherein the keyboard and mouse control message includes a keyboard control message and a mouse control message, and the controller executes acquiring the keyboard and mouse control message sent to the rendering window and sending the keyboard and mouse control message to the server, and is further configured to:

In the case where the rendering window is a focus window, the keyboard control message is acquired by using keyboard and mouse hook programming, and the keyboard control message is sent to the server, wherein the focus window is used to represent the window that is interacting with the user;

When the rendering window is the focus window and the position indicated by the mouse is within the rendering window, the mouse control message is acquired by using keyboard and mouse hook programming, and the mouse control message is sent to the server.

7. The first terminal device according to claim 5, characterized in that after the first terminal device, the second terminal device and the third terminal device join the same video conference, the controller is further configured to:

In response to the instruction to share the clipboard, sending a request to share the clipboard content to the server, so that the server adds the first device identifier of the first terminal device to the shared clipboard list;

When it is detected that the clipboard content has changed and it is not the video conferencing application that changes the clipboard content, the clipboard content is sent to the server, so that the server sends the clipboard content to the target terminal device corresponding to the device identifier in the shared clipboard list, and the target terminal device changes the clipboard content.

8. A second terminal device, comprising:

A display; configured to display a user interface;

The controller is configured as:

In response to an instruction to create or join a conference, sending a conference request to the server, so that the server enables the second terminal device to join the video conference based on the conference request, wherein the first terminal device, the second terminal device and the third terminal device join the same video conference;

In response to an instruction to approve the remote control request, a control approval message and audio and video data are sent to the server, so that the server establishes remote control of the second terminal device by the first terminal device based on the control approval message, and stores the audio and video data in a preset location for the first terminal device and the third terminal device to obtain and play;

Receive a keyboard and mouse control message sent by the server, and execute an operation corresponding to the keyboard and mouse control message, wherein the keyboard and mouse control message is uploaded to the server by the first terminal device.

9. The second terminal device according to claim 8, characterized in that after the first terminal device, the second terminal device and the third terminal device join the same video conference, the controller is further configured to:

In response to the instruction to share the clipboard, sending a request to share the clipboard content to the server, so that the server adds the second device identifier of the second terminal device to the shared clipboard list;

10. A video conference remote control method, characterized in that it is applied to a server and comprises: