Image optimization transmission system for remote terminal management and control
Technical Field
The invention relates to a technology in the field of computer remote control, in particular to an image optimization transmission system for remote terminal management and control based on window detection block cache.
Background
The remote terminal management and control system usually needs to adopt a remote desktop to perform more intuitive monitoring and control on the device, and in the implementation of the remote desktop, bitmap information transmission of images occupies a large amount of network bandwidth. One of the commonly used optimization methods is to reduce image quality by using image compression, and network delay can be reduced by data transmission after compression. The other method is that by utilizing image cache, a sender and a receiver need to maintain the same cache structure at the same time, and only instruction information needs to be transmitted when repeated data is found by accessing the cache during transmission. The existing solution usually adopts static cache to store preset image bitmap information, and then performs differential compression transmission based on image block comparison between the previous frame and the next frame. Because the static cache is adopted, and the cache object is generally a desktop image, the cache hit effect is not ideal after the screen is used for a long time.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides an image optimization transmission system for remote terminal management and control, which reduces image data transmission redundancy based on window detection and block cache.
The invention is realized by the following technical scheme:
the invention relates to an image optimization transmission system facing remote terminal management and control, comprising: the system comprises a window detection module, an image real-time processing module and a data sending module which are positioned at a server side and a data receiving module which is positioned at a client side, wherein: the window detection module identifies and positions a rectangular frame applied by a window based on an openCV image processing method, calculates a hash value in a blocking mode and inserts local blocking cache contents; the image real-time processing module carries out difference blocking processing on the image and cache access according to the blocked hash value, namely, the hash value of the image is calculated according to the difference blocking of two frames of real-time images before and after screen capture of the equipment, and whether the image is hit or not is inquired in the local blocking cache content; the data sending module selects a compression mode of corresponding image data according to a hit result, constructs a control head to be transmitted and a data segment and then sends the control head and the data segment to the client, so that synchronization of the current image data and synchronization of cache information are realized; and the data receiving module is used for constructing cache information in combination with local cache contents after decompression, synchronously caching and calculating the current screen image content.
The identification is that the mouse button monitors the click event to trigger the window scanning, namely the difference change area is calculated by the front and the back frame image information.
The updating of the cache refers to: when the background window detection module successfully detects, two parts of cache contents are added at the same time, the previous frame is used for caching the window area, namely a covered background image before the window is created, and when the window is closed, cache query hits to quickly roll back (bak cache); the caching of the window area by the later frame is to cache a newly-added window, and the subsequent operation of re-opening the window can trigger cache hit so as to reduce data transmission (cur cache).
The graphic blocking processing means: after dividing the screen image into 256 blocks of 16 × 16, the image blocks are ID-labeled.
The step of calculating the hash of the image blocks is as follows: mapping the original information of the image to a shorter data string to realize the quick comparison and retrieval of the image, which specifically comprises the following steps: interpolation, graying, DCT (discrete cosine transformation) change and quantization are carried out on a unit image block by a perceptual hash generation algorithm, a one-dimensional vector which is 64 bits in length and consists of 0 and 1 is generated and recorded as a content hash value of the block, and then the image block similarity is calculated by calculating the Hamming distance.
Preferably, the data sending module performs retransmission through secondary verification to make up for error checking with small probability in the perceptual hash generation algorithm.
The block cache is as follows: and positioning the rectangular window area obtained by the cache window detection method to the changed image block, and then inserting the update data into the chain structure maintained by the image block.
Preferably, the block cache adopts a structure of a doubly linked list plus hash cache of LinkedList and hashmap, and the structure includes: the hash value used for searching the contrast cache image block, and the cache insertion time meta information and the data pointer used for executing the replacement strategy.
The block cache adopts a Least Recently Used (LRU) or FIFO algorithm.
The cache access comprises the following steps: when the foreground image transmission is preprocessed, calculating the difference value of two frames before and after the current moment to find a changed image block, and performing cache access according to the block ID. And if the cache is hit, transmitting the image block ID + the image block hash, and if the cache is not hit, not performing cache insertion updating, and directly transmitting the complete data of the image change block. Because a change block cache miss indicates a dynamic content change that is likely a window, with no recurring cache behavior. When cache insertion is carried out, cache replacement frequently occurs, and cache contents which should be resident and reproduced are replaced. And secondly, when the background window detection module processes the window, detecting a rectangular window and performing cache access. And if the cache is hit, skipping to continue, and if the cache is not hit, performing image block cache insertion.
The determining the image compression mode according to the hit result comprises the following steps: lossy compression and lossless reversible compression, wherein: analyzing the position of a window area by a window detection module, wherein the window area is stored in a lossy compression mode as a whole and corresponds to a data transmission path after cache hit; and for other areas, local dynamic content updating of a desktop background layer or a window can be judged, the quality of the image is ensured by using a lossless compression algorithm RLE, and a direct transmission path of a block difference value corresponds to the block difference value.
The lossless compression is realized by adopting but not limited to Run Length Encoding (RLE), and the lossy compression is realized by adopting but not limited to jpeg compression algorithm.
Technical effects
Compared with the prior art, the data transmission of the invention occupies smaller network bandwidth due to the adoption of a specific image caching and image compression scheme. Under a real scene use test, the cache hit rate under the LRU-based cache replacement strategy is 37%, and compared with an open source scheme, the real-time image reduces the network delay by 58%.
Drawings
FIG. 1 is a flowchart illustrating the operation of a window detection module and a data synchronization module in an embodiment;
FIG. 2 is a flow chart of the real-time image processing module and the data synchronization module in the embodiment;
FIG. 3 is a block diagram of a system according to the present invention.
Detailed Description
The embodiment relates to an image optimized transmission method facing remote terminal management and control according to the system, which comprises a window detection and cache data synchronization stage and a real-time image processing and image data synchronization stage.
As shown in fig. 1, the window detection and cache data synchronization stage includes:
the method comprises the following steps: the server-side remote control module monitors mouse keys, when the click operation of the mouse keys is monitored, the window cache detection module is triggered within the duration (configurable, default continuous 20 frames), difference calculation is carried out on two continuous frames of images, and similar non-change points are removed to obtain a difference image.
Step two: and (3) performing edge detection on canny or using multiple threshold values for binarization by using an image processing opencv library function in the change point.
Step three: and finding the outline by using a findContours function, obtaining a polygonal outline by using an approxColyDP function, and finding a quadrilateral with a convex shape.
Step IV: and judging whether the cosine of the included angle (the angle is 90 degrees) of every two adjacent straight lines in the contour conforms to the rectangular window, and judging whether the length, the width and the area are within the given threshold range, and judging whether the application window exists in the continuous area.
Step five: if the detection of the rectangular window is not successful, the current process is ended. If the detection is successful, image compression is respectively carried out on the detection areas (corresponding to the bakcache and the cur cache) of the front frame and the rear frame, and a jpeg method is adopted for lossy compression.
Step (c): and calculating the hash value of the image block, inquiring a cache structure in the memory, searching an existing cache chain through the block ID, and finishing the current process if the hash value of the cache block is matched and hit.
Step (c): and cache insertion is carried out when cache is not hit, and a cache chain node records the hash value of the current image block, updates time element information and a data block pointer. And when the cache chain data is full, triggering a cache elimination algorithm to carry out strategy replacement. And meanwhile, after the operation is executed by the server, the server sends a message, and the synchronous client establishes a cache structure. And (3) sending a message header cache hit bit flag of 01, wherein the message data is compressed block data.
As shown in fig. 2, the real-time image processing and image data synchronization stage includes:
the method comprises the following steps: and the remote desktop receives the real-time image transmitted from the equipment end in real time, and the image is subjected to standardization processing and then is subjected to difference calculation in blocks.
Step two: and calculating hash of the image block by the difference block, accessing the cache structure through the block ID, respectively retrieving cache data chains corresponding to the block ID, and matching the hash value of the stored data to detect cache hit.
Step three: and (4) cache is not hit, lossless compression is carried out on the original image data, and an RLE method is adopted in a compression algorithm. And the message header buffer hit bit flag is 00, and the compressed data is filled in the message data part.
Step four: if the cache hits, the control command information is transmitted, the message header cache hit bit flag is 10, and the control information (the image difference change block ID + the difference block hash value) is compressed and then filled into the message data part.
Step five: and carrying out data transmission on the optimized image information, and directly decompressing the message by the user client side through reading the message, wherein the flag is 00 to obtain image change data. And when the flag is 10, accessing and reading the cache value of the client according to the block ID + hash value, and decompressing to obtain image change data.
Step six: when an abnormal condition occurs (the server side and the client side are abnormally restarted), the client side cache and the server side cache maintain inconsistent data, and the client side cannot find the cache content of the host machine memory when receiving flag is 10, so that the fifth step fails. The client carries the original request header to re-request the server, the server analyzes the data of the block ID + the block hash, the buffer hit bit flag of the message header is re-sent to be 01, and the message data is compressed block data.
As shown in fig. 3, a system for implementing the method according to this embodiment includes: the system comprises a window detection module, an image real-time processing module and a data sending module which are positioned at a server side and a data receiving module which is positioned at a client side. Through specific practical experiments, a maximum limit 512M is set for cache occupation of a server under an 8-core 64G linux machine, an LRU (least recently used) strategy is selected for updating a storage strategy, real-time images of a device end are transmitted to a server end (30FPS) according to an RDP (remote desktop protocol), the resolution of the images of the device end is in a resolution scene from 800 x 600 to 1920 x 108010, a jpeg algorithm is selected for lossy compression of a real-time image processing module, an RLE (recursive least squares) algorithm is selected for lossless compression, and on the premise that image quality (measured by PSNR (Peak Signal to noise ratio) is guaranteed to be certain, image network delay is reduced by about 58%.
The foregoing embodiments may be modified in many different ways by those skilled in the art without departing from the spirit and scope of the invention, which is defined by the appended claims and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.