CN102164168A - A transmission and display method and system for dragging images - Google Patents

Abstract

The invention discloses a transmission display method of a dragging image, which is applied to a remote control process and comprises the following steps: detecting the position of an existing window body at a local side to change; and replacing and displaying the existing frame to the new position of the existing frame on the network side. The invention can effectively reduce the occupancy rate of processing resources and transmission resources and improve the processing speed and the transmission speed.

Description

A transmission and display method and system for dragging images

technical field

The invention belongs to the field of remote control, and in particular relates to a transmission and display method and system for dragging images.

Background technique

The so-called remote control means that the management staff connects to the controlled computer in a different place through the Internet, displays the desktop environment of the controlled computer on their own computer, and configures, installs software, Modify and so on.

How remote control works: Remote control must be done through the network. The local computer is the sending end of the manipulation command, called the master control end, and the non-local controlled computer is called the controlled end. The master control end and the controlled end can be located in the same local area network, or two or more computers connected to the Internet at any location. Usually, the server-side program is installed in the computer of the master control end, and the client-side program is installed in the computer of the controlled end; when in use, the server-side program in the computer of the master control end sends a signal to the client program in the computer of the controlled end to establish a special Then through this remote service, use various remote control functions to send remote control commands to control the operation of various applications in the computer at the controlled end.

VNC (Virtual Network Computing virtual network computer) is a remote control technology, which is essentially a cross-platform display system, which transmits the complete window interface of the computer at the controlled end to the computer screen at the main control end through the network for display. When the operator uses the host computer to control the controlled computer, as if sitting in front of the screen of the controlled computer, the operator can start the application program of the controlled computer, use or steal the files and materials of the controlled computer, It is even possible to use the external printing device (printer) and communication device (modem or dedicated line, etc.) of the controlled computer to print and access the external network and intranet.

Because VNC is an abstract tool, it has no direct connection with the graphics system of the client machine. VNC technology captures the entire screen of the controlled terminal, and transmits the screen as a bitmap to the host terminal for display. Therefore, When the user of the client computer drags a window to move on the screen with the mouse, the VNC technology also needs to continuously capture and transmit the entire desktop to the host control end for reproduction, and the amount of transmitted data will inevitably be large. Reappearing the screen will also consume more resources, and the transmission efficiency and processing efficiency are very low, which will directly lead to the slow update speed of the picture displayed on the main control terminal, the picture lags seriously, and the refresh effect is not obvious, thus affecting the main control. The remote control of the controlled terminal by the control terminal gives the user of the controlled terminal the feeling that the response of the remote control main control terminal is very insensitive, and there is a very large delay.

Contents of the invention

In view of this, the technical problem to be solved by the present invention is to provide a transmission and display method and system for dragging images, which can effectively reduce the occupancy rate of processing resources and transmission resources, and improve processing speed and transmission speed.

In order to solve the above-mentioned technical problems, the present invention discloses a transmission and display method for dragging images, including: during the remote control process, detecting that the position of an existing window has changed on the local side; Have the form replace the display to its new location.

Further, the method also includes: transmitting the original position and the new position of the existing window to the network side at the local side.

Further, detecting that the position of an existing form changes includes: detecting on the local side that there is a mouse click and drag operation and that the existing form that has been dragged to a new position has been changed in the previous remote control process is transmitted to the network side.

Further, the existing form has been transmitted to the network side in the previous remote control process, including: at the local side, according to the image of the existing form sent during the saved previous remote control process If hit, it is determined that the existing form has been transmitted to the network side in the previous remote control process.

Further, replacing and displaying the existing form on the network side includes: obtaining the image of the existing form from the saved image received in the previous remote control process on the network side, and replacing and displaying the image to its new form. Location.

Further, the method also includes: filling the display background at the original position of the existing window on the network side.

Further, it also includes: capturing the image at the original position of the existing form on the local side and sending it to the network side; filling and displaying the image at the original position of the existing form on the network side image.

Further, it includes: dividing the image at the original position of the existing window into several blocks on the local side; for each block, identifying the elements contained therein, if there is an element with the largest area and occupying the area When the proportion of the area of the block exceeds a threshold, the block is compressed using a compression method corresponding to the type of the element and transmitted to the network side along with the location information of each block; each block is received and decompressed on the network side, Recombining each block according to the position information of each block to restore the image, filling and displaying the image at the original position.

In order to solve the above technical problems, the present invention also discloses a transmission and display system for dragging images, including: a transmission device and a display device; the transmission device is used to detect an existing window at the local side during the remote control The position of the form changes; the display device is used to replace and display the existing form to its new position on the network side.

Further, the transmission device is also used to transmit the original position and the new position of the existing window to the display device.

Further, the transmission device is also used to detect whether there is a mouse click and drag operation and whether the existing window dragged to a new position has been transmitted to the display device during the previous remote control process.

Further, the transmission device is also used to perform a hit in the storage device according to the image of the existing form, and if it is hit, it is determined that the existing form has been transmitted to the The display device; the storage device is a cache located on the side of the transmission device or a cache located at a third party and shared with the display device, and is used to cache the image of the existing window sent by the transmission device.

Further, the display device is also used to obtain the image of the existing window from the storage device; the storage device is a buffer that is synchronized with the transmission device and located on the side of the display device or located at the first The three-party location and the buffer shared with the transmission device are used for buffering and receiving the image of the existing form sent by the transmission device.

Further, the display device is also used to fill the display background at the original position of the existing window.

Further, the transmission device is also used to capture the image at the original position of the existing window and send it to the display device; the display device is also used to The in-place fill shows the image.

Further, the transmission device divides the image at the original position of the existing window into several blocks, and for each block, identifies the elements contained in it, if there is an element with the largest area and occupies the area When the proportion of the area of the block exceeds a threshold, the block is compressed using a compression method corresponding to the type of the element and transmitted to the display device together with the position information of the block; the display device also It is used for receiving and decompressing each block, recombining each block according to the position information of each block to restore the image, and filling and displaying the image at the original position.

Compared with existing solutions, the technical effects obtained by the present invention are:

1) The present invention is particularly aimed at the situation of window dragging. By judging the occurrence of window dragging, the network side directly reads and displays the moving window from the cache, avoiding repeated data transmission and greatly reducing The total amount of data that needs to be transmitted saves the occupation of transmission resources;

2) At the same time, for the original position after the window is moved, the background or new image is also filled in through classification processing, which ensures the accuracy of restoring the desktop on the network side;

3) At the same time, when filling the new image, the transmission size of the image is further reduced by classifying and compressing the image, which reduces the amount of transmitted data.

Description of drawings

Fig. 1 is the overall flowchart of the embodiment of the present invention;

Fig. 2 is the flowchart of the detection form dragging of the embodiment of the present invention;

Fig. 3 is the detailed flowchart of the classification compression processing of the application example of the present invention;

Fig. 4 is a schematic diagram of the system of the embodiment of the present invention.

Detailed ways

The implementation of the present invention will be described in detail below in conjunction with the drawings and examples, so that the realization process of how to use technical means to solve technical problems and achieve technical effects in the present invention can be fully understood and implemented accordingly.

The core idea of the present invention is: during the remote control process, the local side detects that the position of an existing window changes; and the network side replaces and displays the existing window to its new position. The window may be a window in the system desktop, or another window displayed in the window, and the present invention is not limited thereto.

The partition transmission method for remote control of the present invention will be described below with an embodiment. The user accepts the server's request and takes over the user's terminal for remote control. In the process of remote control, the server needs to capture the desktop of the user terminal in real time or periodically, and save it as a picture and send it to the server for display, so that it can be displayed on the server. restore the desktop of the user terminal on the server, so that maintenance personnel can control the user terminal on the server.

The transmission and display method of the dragging image of the present invention is applied to the process that the service side takes over the user side during remote control, and captures and transmits the desktop of the user terminal. As shown in Figure 1, it specifically includes the following operations:

Step S1, during the remote control process, it is detected on the local side that the position of an existing window has changed;

At the same time, the original position and the new position of the existing form are also transmitted to the network side at the local side;

Step S2, scan the original position on the local side, and execute step S3 or S4 depending on the situation;

When the local scan finds that the existing form is superimposed on another form or image, step S3 is performed, and when the existing form is directly displayed on the desktop background, step S4 is performed;

Step S3, capture the image at the original position of the existing form on the local side and send it to the network side, and execute step S5;

Step S4, fill the display background at the original position of the existing form on the network side, and execute step S5;

Step S5, replacing and displaying the existing window to its new position on the network side;

Because it is an existing form, the existing form has been transmitted to the network side in the previous remote control process, then the image of the existing form is directly obtained from the cache on the network side, and the image is stored Replace the display to its new position, so as to reproduce the effect of dragging and displaying the window on the local side at this moment on the network side;

The cache is a network-side cache synchronized with the local side or a cache located at a third-party location and shared with the local side.

For step S1 , the judging method of detecting whether there is a change in the position of an existing window can be realized through the following steps. As shown in Figure 2, it includes the following steps:

Step SS1, detect whether there is a mouse click and drag operation on the local side, if yes, it means that the window may be moved, but the image of the window itself does not change, go to step SS2; if it does not happen, it can generally be regarded as not The window moves and the process ends;

Step SS2, monitoring whether the existing form dragged to the new position has been transmitted to the network side in the previous remote control process, if yes, it can be determined that the form has been moved, and step S5 can be guaranteed In the network side, the image of the existing window can be obtained by directly reading the cache, avoiding network transmission, and then jump to step S2; otherwise, go to step SS3;

Specifically, on the local side, a hit can be performed in the cache according to the image of the existing form, and if it is hit, it is determined that the existing form has been transmitted to the network side in the previous remote control process; if it is not hit , it is determined that the existing window has not been transmitted to the network side in the previous remote control process, so in step S5 the image of the existing window cannot be obtained by directly reading the cache on the network side, and step SS3 is performed;

The cache is a cache on the local side or a cache located at a third-party location and shared with the network side;

Step SS3, capturing and transmitting the image of the existing window on the local side, and receiving and displaying the image on the network side.

For step S3 of the present invention, the image on the original position of the existing window on the local side is transmitted to the network side, or for step SS3, the image of the existing window on the local side is transmitted to the network side, both can be passed through the following steps: It is transmitted to the network side in the manner of classified transmission. As shown in Figure 3, the following steps are included:

Step SSS1, acquiring the image at the original position of the existing form or the image of the existing window at the local side;

Step SSS2, dividing the image into several blocks;

Dividing the image into several blocks in any manner, for example, the image can be divided into nine blocks according to the nine-square grid, or it can be divided into sixteen grids, etc., and the present invention is not limited thereto;

Step SSS3, identifying elements contained therein for each block;

The elements include at least: background, text, and pictures;

Identify the background in the elements in the block according to the graphics device interface GDI inside the system; for the background, it can be a picture with rich colors, or a picture with a single color;

Identify the color types contained in the block, and identify the text and/or pictures in the elements in the block according to the number of color types; generally speaking, the color types contained in the pictures will be more abundant , and compared to pictures, the color types contained in text are relatively single. Therefore, a threshold can be set for color types. When the number of color types is less than the threshold, the element is identified as text; the color type When the number is greater than the threshold, identifying the element as a picture;

Of course, the picture can also be divided more carefully, and the first threshold and the second threshold can be set for the color type, the first threshold is greater than the second threshold; when the number of color types is smaller than the second threshold, the identified The element is text; when the number of the color types is greater than the first threshold, identify the element as a picture with rich colors; when the color type is between the first threshold and the second threshold, identify the element as not rich in color picture of;

Step SSS4, for each block, find the element with the largest area and determine the proportion of the element in the area of the block, if the proportion of the element in the area of the block exceeds a threshold, execute Step SSS6, if the proportion of the element in the area of the block does not exceed a threshold, execute step SSS5;

Generally speaking, if the proportion of the element with the largest area in the block exceeds a preset threshold, it can be considered that the block is mainly composed of this element; if the proportion of the element in the block If the threshold is not exceeded, it means that the elements in the block are of various types and filled on average, and the block is not mainly composed of certain elements;

Step SSS5, divide the block into several second-level blocks, execute step SSS3 again for each second-level block, and identify the elements contained therein;

Because the block is not mainly composed of certain elements, and different types of elements are suitable for different compression methods, so the appropriate compression method cannot be selected for the block at this time; the block can be further divided into A plurality of secondary blocks can be divided into secondary blocks in any manner, for example, the blocks can be divided into nine secondary blocks according to a nine-square grid, or can be divided into sixteen grids, etc., and the present invention is not limited thereto. If the main constituent elements cannot be determined in some secondary blocks at this time, the secondary blocks can also be divided again and repeated multiple times, the present invention is not limited thereto;

Step SSS6, using a compression method corresponding to the element type to compress the block and transmit it to the network side, and at the same time transmit the location information of the block to the network side;

Since the block is mainly composed of such elements, it is most effective and suitable to use the compression method corresponding to this element for this block;

For the selection of compression methods, the following principles can be followed:

For a block mainly composed of text, text is the main object, and text often carries data that needs to be recognized by the naked eye, so it is necessary to ensure the quality of data restoration, and the size of the text itself is also small, so lossless compression can be used;

For a block mainly composed of pictures with less color, the picture with less color is the main object. Since the size of the picture itself is not large, lossless encoding and compression can be used;

For a block mainly composed of colorful pictures, the colorful picture is the main object. For remote control, this kind of picture is often not the key data, and certain distortion is allowed. Therefore, a lossy encoding compression method can be used, such as JPG , can obtain a higher compression ratio, greatly reduce the size of the picture, and obtain a higher rate during transmission.

Compared with directly capturing the entire area for transmission, after such classification compression, the size of the area can be reduced by 30%-50%, which greatly reduces the amount of data and greatly improves the transmission rate;

Step SSS7, receiving and decompressing the image of each block on the network side, reorganizing and restoring the image of each block on the network side according to the position information of each block, filling and displaying the image at the original position, so as to reproduce it on the network side Show the effect of dragging and displaying the local side window at this moment;

The location information may be the horizontal and vertical coordinates of each block on the area where the image changes.

The partition transmission system for remote control of the present invention will be described below with an embodiment.

As shown in FIG. 4 , a transmission and display system for dragging images includes: a transmission device 1 on the local side and a display device 2 on the network side;

The transmission device 1 is used to detect a change in the position of an existing window at the local side during the remote control process, and transmit the original position and the new position of the existing window to the display device together; If the existing form is superimposed and displayed on another form or image, it is also used to capture the image at the original position of the existing form and send it to the display device;

The display device 2 is configured to replace and display the existing form to its new position on the network side, fill the display background in the original position of the existing form or fill in the original position of the existing form display said image;

Specifically, the transmission device 1 is also used to detect whether there is a mouse click and drag operation and whether the existing window dragged to a new position has been transmitted to the display during the previous remote control process. device; hit in the cache 3 according to the image of the existing form, if hit, it is determined that the existing form has been transmitted to the display device in the previous remote control process; the cache 3 is The local cache located on the side of the transmission device 1 or the cache located at a third party and shared with the display device 2 is used to cache the image of the existing window sent by the transmission device 1 .

The display device 2 is also used to obtain the image of the existing form from the cache 4; the cache 4 is a cache located on the side of the display device 2 that is synchronized with the data of the transmission device 1 or located at The third-party cache shared with the transmission device 1 is used to cache and receive the image of the existing form sent by the transmission device 1 .

The transmission device 1 divides the image at the original position of the existing window into several blocks, and for each block, identifies the elements contained therein. If there is an element with the largest area and occupying If the proportion of the area exceeds a threshold, the block is compressed using a compression method corresponding to the type of the element and transmitted to the display device together with the position information of the block; the display device 2 also uses After receiving and decompressing each block, recombining each block according to the position information of each block to restore the image, filling and displaying the image at the original position.

The above-mentioned system and device correspond to the features of the above-mentioned partition transmission method for remote control, and for the deficiencies, please refer to the above-mentioned partition transmission method for remote control.

The above description shows and describes several preferred embodiments of the present invention, but as mentioned above, it should be understood that the present invention is not limited to the forms disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various Various other combinations, modifications, and environments can be made within the scope of the inventive concept described herein, by the above teachings or by skill or knowledge in the relevant field. However, changes and changes made by those skilled in the art do not depart from the spirit and scope of the present invention, and should all be within the protection scope of the appended claims of the present invention.

Claims (16)

1. A transmission and display method for dragging an image, characterized in that it is applied to remote control, including: A change in the position of an existing window is detected on the local side; The existing form is replaced and displayed to its new position on the network side. 2. The transmission and display method according to claim 1, further comprising: The original position and the new position of the existing window are also transmitted to the network side at the local side. 3. The transmission and display method according to claim 2, wherein the detection of a change in the position of an existing window further comprises: The existing window that is detected to have a mouse click and drag operation on the local side and is dragged to a new position has been transmitted to the network side during the previous remote control process. 4. The transmission and display method according to claim 3, wherein the existing form has been transmitted to the network side during the previous remote control process, further comprising: On the local side, according to the image of the existing form, a hit is performed in the saved image sent during the previous remote control process. If it is hit, it is determined that the existing form has been transmitted to the previous remote control process. network side. 5. The transmission and display method according to claim 1, wherein, at the network side, the existing form is replaced and displayed to its new position, further comprising: On the network side, the image of the existing window is acquired from the saved image received in the previous remote control process, and replaced and displayed to its new position. 6. The transmission and display method according to claim 2, further comprising: Fill and display the background at the original position of the existing form on the network side. 7. The transmission and display method according to claim 2, further comprising: Grab the image at the original position of the existing form on the local side and send it to the network side; Filling and displaying the image at the original position of the existing form on the network side. 8. The transmission and display method according to claim 7, further comprising: On the local side, the image at the original position of the existing form is divided into several blocks; For each block, identify the elements contained therein, if there is an element with the largest area and the proportion of the area of the block exceeds a threshold, compress the block using a compression method corresponding to the category of the element And transmit to the network side together with the location information of each block; Receiving and decompressing each block at the network side, recombining each block according to the position information of each block to restore the image, filling and displaying the image at the original position. 9. A transmission and display system for dragging images, characterized in that it is applied to remote control, comprising: a transmission device and a display device; The transmission device is configured to detect a change in the position of an existing window at the local side; The display device is used to replace and display the existing window to its new position on the network side. 10. The transmission display system according to claim 9, characterized in that, The transmission device is further configured to transmit the original position and the new position of the existing window to the display device. 11. The transmission display system as claimed in claim 10, characterized in that, The transmission device is further configured to detect whether there is a mouse click and drag operation and whether the existing window dragged to a new position has been transmitted to the display device during the previous remote control process. 12. The transmission display system as claimed in claim 11, characterized in that, The transmission device is further configured to perform a hit in the storage device according to the image of the existing form, and if it is a hit, determine that the existing form has been transmitted to the display device during the previous remote control process; The storage device is a cache located on the side of the transmission device or a cache located at a third party and shared with the display device, and is used to cache the image of the existing window sent by the transmission device. 13. The transmission display system as claimed in claim 9, characterized in that, The display device is also used to obtain the image of the existing window from the storage device; The storage device is a buffer located at the side of the display device or a buffer located at a third party and shared with the transmission device, and is used for buffering and receiving the image of the existing form sent by the transmission device. 14. The transmission display system as claimed in claim 10, characterized in that, The display device is further configured to fill the display background at the original position of the existing window. 15. The transmission display system of claim 10, wherein: The transmission device is further configured to capture the image at the original position of the existing window and send it to the display device; The display device is further configured to fill and display the image at the original position of the existing window. 16. The transmission display system of claim 15, wherein: The transmission device divides the image at the original position of the existing window into several blocks, and for each block, identifies the elements contained therein, and if there is an element with the largest area and occupying the area of the block The proportion exceeds a threshold, using a compression method corresponding to the type of the element to compress the block and transmitting it to the display device together with the position information of the block; The display device is further configured to receive and decompress each block, recombine each block to restore the image according to the position information of each block, and fill and display the image at the original position.

Images