WO2011036715A1 - Server and system - Google Patents

Abstract

Provided is a server terminal (1) that shares an image to be displayed on a screen thereof with a client terminal (2), and that is provided with: a display unit (116) that displays, on the screen, an image stored in a frame buffer (111); an update information generating unit (101) that generates update information including an update image that updates a portion of the image; a sequence number adding unit (102) that adds sequence numbers to each of the update information; a simple information generating unit (106) that generates simple information that has an amount of information not more than the amount of information of the update information; a drawing command creating unit (104) that creates a drawing command that includes the update information, the sequence number added to this update information, simple information associated with a sequence number that is older than the sequence number added to the aforementioned update information, and the older sequence number; and a drawing command transmitting unit (105) that creates a drawing command packet from the drawing command and transmits the drawing command packet.

Description

Server, system

The present invention relates to a server and a system.

There is a “screen sharing system” in which a desktop image of a server terminal is packetized and distributed in real time from a server terminal connected via a network to a client terminal. According to this, one desktop image can be commonly referred to by both users at remote locations, and an efficient collaborative work environment can be realized between the two bases via the network.
When constructing a screen sharing system, there is a method of distributing only the updated part together with its drawing coordinates every time an update occurs on a desktop image (for example, Patent Document 1).

In the screen sharing system, if a defect occurs in the transmission of the update image packet from the server terminal to the client terminal, the area corresponding to the defect cannot be drawn on the client terminal. This can cause discomfort for the user.

For example, if a pop-up window is moved in the desktop image and there is a loss in the update image packet transmission that draws the area before the pop-up window is moved with the background image, the area before the pop-up window is moved, the area after the move In some cases, a pop-up window may be displayed, which causes the user to feel uncomfortable.

In the case of Patent Document 1, in the screen sharing system, when there is a defect in the transmission of the update image packet from the server terminal to the client terminal, the loss notification is received from the client terminal to the server terminal, and the loss notification is received. The server terminal transmits update image data for restoration to the client terminal. As a result, drawing defects can be repaired.

In addition, in order to solve the lack of drawing, it is also possible to use a transmission protocol having a retransmission function of a lost packet represented by TCP (Transmission Control Protocol).

However, with these retransmission methods, the client terminal cannot draw the updated image packet until the lost retransmission packet arrives.

In the method described in Patent Document 1, a lack of drawing may appear in a form that is visually recognized by the user until the loss notification and the updated update image data for restoration are reciprocated between the client terminal and the server terminal. Such lack of drawing during the round trip time is particularly problematic in situations where the round trip time between terminals is large, such as via a WAN (Wide Area Network) or wireless network.

JP 2008-109269 A

An object of the present invention is to suppress drawing defects even when packet loss occurs in a screen sharing system including a client terminal and a server.

In order to achieve the above object, a server according to an embodiment of the present invention is a server that shares an image to be displayed on a screen with a client terminal, and stores a first image to be displayed on the screen. A storage unit; a display unit that displays an image of the first storage unit on the screen; an update image that updates a part of an image stored in the first storage unit; and the screen on which the update image is displayed. An update information generation unit that generates update information including area information representing the position of the information, a sequence number addition unit that adds a sequence number for each update information, and an information amount that is equal to or less than the information amount of the update information, A simple information generating unit that generates simple information corresponding to the update information, a second storage unit that stores the simple information and a sequence number added to the update information corresponding to the simple information in association with each other; update information A drawing command creating unit for creating a drawing command including the sequence number added to the update information, simple information associated with a sequence number older than the sequence number added to the update information, and the old sequence number; A transmission unit that generates and transmits a drawing command packet from the drawing command.

According to the present invention, in a screen sharing system composed of a client terminal and a server, it is possible to suppress drawing defects even if packet loss occurs.

1 is a block diagram showing a communication system. The block diagram which shows a server terminal and a client terminal. The flowchart which shows operation | movement of a server terminal. The figure which shows the example of area | region information. The figure which shows the example of a conversion table with whether a packet loss rate and simple information are produced. The figure which shows the example of the corresponding | compatible table with whether a CPU load factor and simple information are produced. The figure which shows the example of update information. The figure which shows the 1st example of simple information. The figure drawn based on the simple information of the 1st example. The figure which shows the 2nd example of simple information. The figure drawn based on the simple information of the 2nd example. The figure which shows the correspondence of an image quality and an image quality parameter. The figure which shows the example of a conversion table with a band usage rate and whether simple information is added. The figure which shows the example of a conversion table with whether a packet loss rate and simple information are added. The figure which shows the example of the conversion table of the input state of a pointing device, and whether to add simple information. The flowchart which shows operation | movement of a server terminal. The figure which shows the area | region information containing the area | region which overlaps among area | region information corresponding to a loss notification. The figure which reconfigure | reconstructed area | region information. The flowchart which shows operation | movement of a client terminal. The figure which shows the example of a drawing command.

Hereinafter, embodiments of the present invention will be described.

FIG. 1 shows the configuration of the screen sharing system of this embodiment. In the screen sharing system of this embodiment, a server terminal 1 and a client terminal 2 are connected via a network 3. The server terminal 1 and the client terminal 2 share an image to be displayed on the screen of each terminal via the network 3.

That is, when the server terminal 1 intends to update the contents of its own desktop screen, the server terminal 1 generates image information (update information) of a portion (area) to be updated and displays it on the desktop screen, and also displays the same image information (update). Information) is packetized and transmitted to the client terminal 2. The client terminal 2 reflects the received image information (update information) on its own screen, so that the server terminal 1 and the client terminal 2 can display the same updated screen.

In this case, in preparation for a case where a part of the packetized image information (update information) is lost, the server terminal 1 also transmits simple information when transmitting the image information (update information).

Simple information is information corresponding to update information, and is information having an information amount equal to or less than the information amount of the update information.

As a result, the client terminal 2 can draw with simple information, so that even if a part of the packet is lost, it can be drawn using the simple information, and apparently Packet loss can be suppressed.

FIG. 2 shows the configuration of the server terminal 1 and the client terminal 2.

The server terminal 1 includes an update information generation unit 101, a sequence number addition unit 102, an addition determination unit 103, a drawing command generation unit 104, a drawing command transmission unit 105, a simple information generation unit 106, and a simple information storage unit. 107, input status acquisition unit 108, communication status acquisition unit 109, encoding unit 110, frame buffer 111, loss notification analysis unit 112, loss notification reception unit 113, generation determination unit 114, update information history A storage unit 115, a display unit 116, and a CPU state acquisition unit 117 are included.

The update information generation unit 101 is an image (updated image) after updating the area to be updated on the display screen (desktop) according to the operating state of the OS (Operating System) and various applications executed on the server terminal 1. And information (region information) indicating the position of the region to be updated. The updated image is rectangular screen data corresponding to a part of the desktop. The area information is a numerical value indicating the display position of the update image on the desktop (a numerical value indicating the storage position when the update image is stored in the frame buffer 111). A pair of update image and area information is called update information. The update information generation unit 101 outputs the generated update information to both the frame buffer 111 and the sequence number addition unit 102.

The sequence number adding unit 102 has a storage area for storing the sequence number count value. Every time update information is received from the update information generating unit 101, the sequence number count value is stored in the storage area. It is added to the update information. For example, the sequence number adding unit 102 resets the sequence number count value to zero when the operation of the server terminal 1 is started, and each time the sequence number count value is added to the update information, the sequence number count value in the storage area is incremented by 1 Increment. The sequence number adding unit 102 outputs the update information with the sequence number count value added to both the generation determining unit 114 and the update information history storage unit 115.

The update information history storage unit 115 has a storage area for storing a pair of region information and sequence number of update information, and receives a pair of region information and sequence number of update information from the sequence number adding unit 102. This is stored in the storage area. In addition, when the update information history storage unit 115 receives the update information region information and sequence number pair acquisition request from the loss notification analysis unit 112, the update information history storage unit 115 notifies the loss information region information and sequence number pair corresponding to the request. The data is output to the analysis unit 112.

Upon receiving the update information and sequence number pair from the sequence number adding unit 102, the generation determination unit 114 receives the CPU load status and input acquired from the CPU status acquisition unit 117, the input status acquisition unit 108, and the communication status acquisition unit 109, respectively. It is determined whether or not it is necessary to generate simple information according to the state of the device {for example, a pointing device or a keyboard device (not shown)} and the communication state. If the generation determination unit 114 determines that it is necessary, the generation determination unit 114 inputs the update information and sequence number pair to both the addition determination unit 103 and the simple information generation unit 106. When it is determined that it is not necessary, the update information and sequence number pair is output only to the addition determination unit 103.

Upon receiving the update information and sequence number pair from the generation determination unit 114, the addition determination unit 103 receives the CPU load state, the input device, and the CPU load state acquired from the CPU state acquisition unit 117, the input state acquisition unit 108, and the communication state acquisition unit 109. It is determined whether or not to add a pair of simple information and sequence number to the pair of update information and sequence number according to the state or communication state. The addition determination unit 103 outputs a determination result as to whether or not to generate a drawing command in which the pair of the simple information and the sequence number is added to the pair of the update information and the sequence number, to the drawing command creation unit 104.

When receiving the update information / sequence number pair and the determination result from the addition determination unit 103, the drawing command generation unit 104 generates a drawing command. If the determination result of the addition determination unit 103 determines that the pair of simple information and sequence number is to be added, a drawing command is generated by adding the pair of simple information and sequence number to the pair of update information and sequence number. If it is determined, a drawing command is generated that does not add the pair of simple information and sequence number to the pair of update information and sequence number. The drawing command creation unit 104 acquires a pair of simple information and sequence number from the simple information storage unit 107. For example, the drawing command creation unit 104 acquires from the simple information storage unit 107 a pair of simple information and sequence number corresponding to a sequence number that is one older than the sequence number of the update information. The drawing command creation unit 104 performs an encoding process by the encoding unit 110 on the update information / sequence number pair and the simple information / sequence number pair as necessary, and then compares the update information and the sequence number. A drawing command including a pair and a pair of the simple information and the sequence number is created. When the drawing command creation unit 104 creates the drawing command, the drawing command creation unit 104 outputs the drawing command to the drawing command transmission unit 105.

When the drawing command transmission unit 105 receives the drawing command from the drawing command creation unit 104, the drawing command transmission unit 105 creates a drawing command packet from the drawing command and transmits it to the client terminal 2 via the network 3. The drawing command packet is obtained by converting the drawing command into a format that can be transmitted on the network 3.

When the simple information generation unit 106 receives a pair of update information and a sequence number from the generation determination unit 114, the simple information generation unit 106 generates simple information corresponding to the update information. When the simple information generation unit 106 generates the simple information, the simple information and the sequence number are associated with each other and output to the simple information storage unit 107. Here, the simple information is information corresponding to the update image, and is information having an information amount equal to or less than the information amount of the update information. The reason why information having an information amount equal to or smaller than the information amount of the update information is used as the simple information is to suppress a tight communication band.

The input state acquisition unit 108 acquires the input state of an input device such as a pointing device (not shown) or a keyboard device (not shown) connected to the server terminal 1. When an input state acquisition request is received from the generation determination unit 114 or the addition determination unit 103, the input state is output.

The communication status acquisition unit 109 acquires the communication status of a network interface (not shown) in the server terminal 1. When a communication status acquisition request is received from the generation determination unit 114 or the addition determination unit 103, the communication state is output.

The CPU state acquisition unit 117 acquires a load state of a CPU (Central Processing Unit) (not shown) in the server terminal 1. In addition, when a communication state acquisition request is received from the generation determination unit 114 or the addition determination unit 103, the load state is output.

Upon receiving the update image and sequence number pair or the simple information and sequence number pair, the encoding unit 110 performs an appropriate encoding process and outputs the result to the drawing command creation unit 105.

The frame buffer 111 has a storage area for storing the entire image of the display screen (screen data corresponding to the entire desktop). When the frame buffer 111 receives a pair of update information and sequence number from the update information generation unit 101, the frame buffer 111 stores the update image included in the update information in the area on the storage area indicated by the area information included in the update information. (Overwrite. Further, when the frame buffer 111 receives the screen data acquisition request together with the area information from the loss notification analysis unit 112, the frame buffer 111 outputs an image of the area of the screen data stored in itself.

The display unit 116 displays the screen data stored in the frame buffer 111.

When the loss notification receiving unit 113 receives the loss notification packet from the client terminal 2 via the network 3, the loss notification receiving unit 113 extracts the loss notification included in the loss notification packet. The loss notification receiving unit 113 outputs the loss notification to the loss notification analyzing unit 112.

When the loss notification analysis unit 112 receives a loss notification from the loss notification reception unit 113, the loss notification analysis unit 112 extracts a sequence number from the loss notification. The loss notification analysis unit 112 outputs the sequence number to the update information history storage unit 115 and requests acquisition of update information corresponding to the sequence number. When the loss notification analysis unit 112 receives the update information, the loss notification analysis unit 112 extracts area information from the update information, outputs the area information to the frame buffer 111, and requests acquisition of the latest screen data regarding the area indicated by the area information on the display screen. To do. The loss notification analysis unit 112 configures update information from the pair of the latest screen data and the region information, and outputs the update information to the sequence number adding unit 102.

The client terminal 2 includes, as its constituent elements, a drawing command reception unit 201, a loss detection unit 202, a drawing determination unit 203, a drawing execution unit 204, a frame buffer 205, a display unit 206, a decoding unit 207, A loss notification transmission unit 208 and an update information history storage unit 209 are included.

When the drawing command receiving unit 201 receives a drawing command packet from the server terminal 1 via the network 3, the drawing command receiving unit 201 extracts a drawing command from the received packet. The drawing command receiving unit 201 outputs the drawing command to the loss detecting unit 202.

The loss detection unit 202 extracts the sequence number added to the update information included in the drawing command, and detects whether or not a loss has occurred in the arrival of the drawing command packet at the client terminal. The loss detection unit 202 has a storage area for storing a sequence number count value. For example, the sequence number count value is reset to 0 at the start of operation, and a drawing command is received every time a drawing command is received. Then, the sequence number count value in the storage area is incremented by one. The loss detection unit determines that no loss has occurred when the sequence number added to the update information matches the sequence number count value in the storage area, and the sequence number added to the update information is stored in the storage area. If it is newer (larger) than the sequence number count value, it is detected that a loss has occurred. Here, the sequence number of the lost update information is, for example, smaller than the sequence number added to the update information and a value equal to or greater than the sequence number count value in the storage area. When the loss detection unit 202 detects that a loss has occurred, the loss detection unit 202 outputs the lost sequence number to the loss notification transmission unit 208. Further, the loss detection unit 202 outputs a drawing command together with the lost sequence number to the drawing determination unit 203. If no loss is detected, the loss detection unit 202 outputs only the drawing command to the drawing determination unit 203.

When the drawing determination unit 203 receives the drawing command, the drawing determination unit 203 checks whether the sequence number lost along with the drawing command is input. When there is a sequence number lost with the drawing command, the drawing determining unit 203 determines whether there is simple information corresponding to the lost sequence number from the pair of the simple information and the sequence number included in the drawing command. Confirm. If there is simple information corresponding to the sequence number lost in the drawing command, the simple information is output to the drawing execution unit 204 and the update information is output to the drawing execution unit 204. If there is no simple information corresponding to the lost sequence number in the drawing command, or if the lost sequence number is not input to the drawing determining unit 203, the drawing determining unit 203 sends only the updated image to the drawing executing unit 204. Output.

Upon receiving the update information or simple information, the drawing execution unit 204 performs a decoding process on the update information and simple information by the decoding unit 207 as necessary, and then displays the display screen stored in the frame buffer 205 Overwrite and update a part of the contents of.

The frame buffer 205 has a storage area for storing the entire image of the display screen, like the frame buffer 111 in the server terminal 1.

The display unit 206 displays the screen data stored in the frame buffer 205.

Upon receiving the update information or simple information, the decoding unit 207 performs a decoding process corresponding to the encoding process of the encoding unit 110 in the server terminal 1 and outputs the result to the drawing execution unit 204.

When the loss notification transmission unit 208 receives the lost sequence number from the loss detection unit 202, the loss notification transmission unit 208 creates a loss notification packet from this and transmits it to the server terminal 1 via the network 3. The loss notification packet is obtained by converting the loss notification into a format that can be transmitted in the network 3.

The update information history storage unit 209 has a storage area for storing a pair of update information and a sequence number, and stores this as a history when receiving a pair of update information and a sequence number. When a history read request is received together with a sequence number, the history is searched and update information corresponding to the sequence number is returned.

Next, processing performed by the screen sharing system according to the present embodiment will be described.

FIG. 3 illustrates processing performed by the server terminal 1. FIG. 3 is a flowchart performed when the screen update of the server terminal 1 is generated.

When the server terminal 1 and the client terminal 2 are in a state where screen sharing is being performed, the updated image generation unit 101 changes the operating states of the OS and various applications executed on the server terminal 1 and presents them to the user. It is checked whether there is an update in the visual information. If there is an update (YES in S101), update information is generated (S102). As described above, the update information includes an update image and region information. The update image is rectangular screen data corresponding to a part of the desktop, and is, for example, the values of three colors of red, green, and blue in each pixel of the rectangle. The area information is a numerical value indicating the display position of the updated image on the desktop. Here, as shown in FIG. 4, the area information is the coordinate value of the upper left vertex of the rectangle of the updated image when the upper left corner of the desktop is the origin of the xy coordinate plane, the horizontal width of the desktop is Width, and the vertical width is Height. Left, Top) and the coordinate value (Right, Bottom) of the lower right vertex. The update information generation unit 101 inputs the generated update information to the frame buffer 111. The frame buffer 111 overwrites and updates the area indicated by the area information in the storage area of the frame buffer 111 with the updated image. The display unit 116 displays screen data in the frame buffer 111. Further, the update image generation unit 101 requests the addition of the sequence number to the update information by outputting the generated update information to the sequence number addition unit 102.

Next, in S103, the sequence number adding unit 102 that has received the sequence number addition request refers to the sequence number count value stored in its own sequence number storage area, and uses this sequence number count value as a sequence number. A sequence number is added to the update information. The sequence number adding unit 102 resets the sequence number count value to zero at the start of screen sharing, and thereafter increments by 1 each time a sequence number is added to the update information in step S103.

Next, in S109, the sequence number adding unit 102 outputs the pair of region information and sequence number included in the update information to the update information history storage unit 115. The update information history storage unit 115 stores region information and sequence number pairs in its own storage region. When the sequence number adding unit 102 receives the notification that the storage is completed, the sequence number adding unit 102 outputs a pair of the sequence number and the update information to the generation determining unit 114.

Next, in S104, upon receiving a pair of sequence number and update information, the generation determination unit 114 determines whether to generate simple information for the update information. The generation determination unit 114 determines whether or not to create simple information corresponding to the update information, based on various types of information obtained from the input state acquisition unit 108, the communication state acquisition unit 109, and the CPU state acquisition unit 117. be able to. Further, not only a simple determination of whether to create or not, but also the details at the time of simple information creation, such as what kind of simple image information to create, can be judged. A specific example of simple information or an example of a method for creating simple information will be described later. If it is determined that simple information is to be created, the process flow proceeds to S105. If it is determined not to create, the flow of processing moves to S106.

A first example for determining whether to create simple information is shown. FIG. 5 is an example in which the determination is made based on the packet loss rate in the network acquired from the communication status acquisition unit 109. Here, the loss rate is, for example, a value obtained by dividing the number of losses known from the loss notification received by the loss notification receiving unit 113 by the number of drawing command packets transmitted by the drawing command transmitting unit 105. The communication state acquisition unit 109 calculates a packet loss rate in the network by receiving a loss notification from the loss notification analysis unit 112 and stores the packet loss rate in a storage area in the communication state acquisition unit 109. Upon receiving a packet loss rate acquisition request from the generation determination unit 114, the communication status acquisition unit 109 replies with the stored loss rate. Based on this, the generation determination unit 114 checks whether or not the loss rate exceeds a specified value (for example, 0.1%). Make a decision.

A second example for determining whether to create simple information is shown. FIG. 6 is an example of determination based on the CPU load factor acquired from the CPU state acquisition unit 117. When the CPU status acquisition unit 117 receives the CPU load factor acquisition request from the generation determination unit 114, the CPU state acquisition unit 117 answers the CPU usage rate occupied by the application that performs screen sharing. Based on this, the generation determination unit 114 checks whether or not the CPU usage rate exceeds a specified value (for example, 80%). do. FIG. 6 also shows an example of determination of simple information to be generated as well as simple determination of whether or not to create. According to this, for example, if the CPU usage rate exceeds a specified value (for example, 20%), it is determined that simple information with low image quality is created, and if it does not exceed, simple information with high image quality is created.

As described above, when the packet loss rate of the network is small, it is considered that the frequency at which the simple information is used in the client terminal 2 is low. Therefore, by using the determination method of the first example, the CPU resource consumption is reduced by creating the simple information. Can be achieved. Further, when the packet loss rate is high, the frequency of using simple information can be increased, and the lack of drawing of the client terminal can be effectively suppressed. Further, according to the second example, the CPU of the server terminal 1 is created by not creating simple information when the CPU usage rate of the server terminal 1 is high, or by creating simple information with low image quality that consumes less CPU resources. Reduction of resource consumption can be achieved. The determination may be made by combining the information of the first example and the second example. Further, the determination may not be made using the information of the first example and the second example. For example, the generation determination unit 114 may always determine that simple information is generated, and the simple information generation unit 106 may generate simple information every time update information is generated. Alternatively, information such as the number of colors, the size, and the position of the update image of the update information may be used. Alternatively, an application or GUI type including the update information may be used.

In S105, upon receiving the sequence number / update information pair from the generation determination unit 114, the simple information generation unit 106 creates simple information corresponding to the update information. The simple information storage unit 107 has a storage area for storing a pair of simple information and sequence number, and stores the pair of simple information and sequence number input from the simple information generation unit 106 in the storage area.

The 1st example of simple information and its preparation method is shown. FIG. 7 shows update information, FIG. 8 shows simple information, and FIG. 9 shows a drawing drawn based on the simple information. As shown in FIG. 7, the update image is a pop-up window image that notifies the user of a warning message. The updated image is shown as a bitmap. As shown in FIG. 8, the simple information includes a single pair of color information and area information. The information representing the area uses the same coordinate axis as the area information included in the update information of the creation source. The color information is, for example, each value of red, green, and blue (0 to 255). An alpha value may be added to this. As the value of the color information included in the simple information, for example, the color used most frequently among the colors constituting the window image is calculated, and the value is used. Alternatively, an average value of color values of each pixel constituting the window image may be calculated and used. Alternatively, color information fixedly defined through screen sharing may be used. Alternatively, color information fixedly defined according to the application or GUI type in which the update information is included may be used. Next, a second example of simple information and a method thereof will be described. The update information is the information shown in FIG. 7 as in the first example. FIG. 10 shows simple information, and FIG. 11 shows a diagram drawn based on the simple information. As shown in FIG. 10, the simple information is composed of a plurality of pairs of color information and area information. In the simple information, the area information is associated with the color information drawn in the area of the area information. The color information of the simple information is, for example, identified by the GUI (Graphical User Interface) type (in the example, the title bar portion and the dialog portion) constituting the window image of the updated image and divided for each portion, This is information obtained by calculating color information for each division by the same method. Further, the area information of the simple information indicates the area after division. As in the second example, according to the method of dividing the update image according to the GUI type and calculating the color information, simple information with higher reproducibility can be obtained. In dividing the update image, it is not always necessary to follow the GUI type, and an arbitrary method may be used. For example, the image may be divided according to the type of application indicated by the updated image, or a similar color block may be detected from the updated image using an arbitrary algorithm, and the division may be configured for each block. Further, it is not necessary to calculate color information every time simple information is created. It is also possible to use a method in which the calculated color information is stored together with information such as the GUI type and the application type, and thereafter, the stored color information is reused for an update image of the same type. In this case, the color information corresponding to the type may be recalculated at an arbitrary timing. With such a method, for example, the burden of color information calculation can be reduced while ensuring the reproducibility of drawing particularly effectively in a part where the change in drawing tendency is small through screen sharing, such as a title bar part.

Next, a third example of simple information and its method is shown. In the third example, the simple information is an image obtained by performing an irreversible compression process on the updated image and area information of the image. As a lossy compression method, any method such as JPEG (Joint Photographic Experts Group) can be used. In step S104, when the generation determination unit 114 determines what image quality is to be created as the simple information, the image quality can be changed by changing the compression method of the simple information. For example, in FIG. 6, when the generation determination unit 114 determines the image quality (high or low) of the simple information according to the CPU usage rate, as shown in FIG. This shows how to create simple information with various image quality parameters. The image quality parameter is a value that takes a value between 0 and 100, and indicates the reproducibility of drawing in the above-described lossy compression. In the case of 100, the same reproducibility as the original image is obtained. Each time the simple information generation unit 106 receives a request to create high-quality or low-quality simple information from the generation determination unit 114, the simple information generation unit 106 sets an image quality parameter according to the above determination and performs compression processing. The simple information generation unit 106 defines a pair of compressed image data and area information as simple information. By using the irreversible image compression algorithm in this way, it is possible to create simple information with high reproducibility of drawing.

Next, a fourth example of simple information and its method is shown. A fourth example is shown. In the fourth example, the simple information is composed of a pair of a sequence number value and area information used in the past. That is, the simple information causes the client terminal 2 to reuse the update information that was previously transmitted from the server terminal 1 to the client terminal 2 as simple information corresponding to the update information. In order to enable such processing, the sequence number adding unit 102 of the server terminal 1 stores the update image in the update information history storage unit 115 as a history in addition to the region information and the sequence number in step S109 described above. There is a need. In step S <b> 105, the simple information generation unit 106 acquires the update image / region information pair history stored in the update information history storage unit 115, and then uses the arbitrary algorithm to update the simple information creation source update image. The update image on the history closest to is selected, and the value of the sequence number and the region information of the update image on the selected history are determined as simple information. Further, the client terminal 2 has an update information history storage unit 209 for storing an update image, and stores a sequence number and update information included therein every time a drawing command packet is received. Thus, by reusing update information transmitted in the past as simple information, simple information with high reproducibility can be created with a small amount of information.

Next, in step S106, when the addition determination unit 103 receives a pair of the sequence number and the update information from the generation determination unit 114, in the next processing of the drawing command creation unit 104, the addition number is updated to the sequence number and the update information as a drawing command. It is determined whether or not to add simple information and a sequence number of the simple information. Specifically, the addition determination unit 103 adds simple information to the update information and a sequence number of the simple information based on various information obtained from the input state acquisition unit 108, the communication state acquisition unit 109, and the CPU state acquisition unit 117. It is determined whether or not to add. In addition to simple determination of whether to add or not, it is also possible to determine whether to add simple information corresponding to which sequence number and the sequence number of the simple information. If it is determined to be added, the addition determining unit 103 outputs the sequence number / update information pair and the sequence number of the simplified information to be added to the drawing command creating unit 104. The flow of processing moves to step S107. On the other hand, if it is determined not to be added, the addition determining unit 103 outputs only the sequence number / update information pair to the drawing command creating unit 104, and the flow of processing proceeds to step S108.

A first example of a method for determining whether to add simple information and a sequence number of the simple information will be shown. FIG. 13 is an example in which a determination is made based on the usage amount of the communication bandwidth acquired from the communication state acquisition unit 109. The communication status acquisition unit 109 calculates a currently used bandwidth out of available bandwidths and stores it in its own storage area. When the communication state acquisition unit 109 receives the bandwidth usage amount acquisition request from the addition determination unit 103, the communication state acquisition unit 109 notifies the stored bandwidth usage amount. Based on this, the addition determination unit 103 checks whether or not the bandwidth usage exceeds a specified value (for example, 5%), and if it exceeds, determines that the simple information and the sequence number of the simple information are not added. To do. If it does not exceed, for example, if it is 1% or more, it is determined that the simple information corresponding to the update information of one sequence number immediately before the sequence number of the update information and the sequence number of the simple information are added, and less than 1% If there is, it is determined that the simple information corresponding to the update information of the two immediately preceding sequence numbers and the sequence number of the simple information are added. For example, if the sequence number included in the update information received by the generation determination unit 114 is 100, the update information of the immediately preceding one sequence number means the update information corresponding to the sequence number 99. Further, the update information of the last two sequence numbers means update information corresponding to the sequence numbers 98 and 99.

A second example of a method for determining whether to add simple information and a sequence number of the simple information will be shown. FIG. 14 is an example of determination based on the packet loss rate in the network acquired from the communication status acquisition unit 109. The communication state acquisition unit 109 calculates a packet loss rate in the network by receiving a loss notification from the loss notification analysis unit 112 and stores the packet loss rate in its own storage area. Upon receiving a packet loss rate acquisition request from the addition determination unit 103, the communication state acquisition unit 109 replies with the stored loss rate. Based on this, the addition determination unit 103 checks whether or not the loss rate exceeds a specified value (for example, 0.1%), and if it exceeds, determines that the simple information and the sequence number of the simple information are added. To do. In the case of exceeding, it is determined based on the correspondence of FIG. 14 whether the simple information corresponding to the immediately preceding update information and the sequence number of the simple information are added.

A third example of a method for determining whether to add simple information and a sequence number of the simple information will be shown. FIG. 15 is an example in which the determination is made based on the input state of the pointing device connected to the server terminal 1 acquired from the input state acquisition unit 108. When receiving an input state acquisition request for the pointing device from the addition determination unit 103, the input state acquisition unit 108 checks and answers the input state. Based on this, the addition determination unit 103 determines whether or not a drag operation is performed on an arbitrary window from the layout state of the window on the desktop acquired by an arbitrary method. If it is determined that a drag operation is being performed, it is determined that simple information corresponding to the immediately preceding three update information and the sequence number of the simple information are added. If it is determined that it has not been performed, it is determined that the simple information corresponding to the immediately preceding update information and the sequence number of the simple information are added.

As described above, the specific example in which the addition determination unit 103 determines whether or not to add the simple information and the sequence number of the simple information based on the information such as the communication state and the input state has been described. According to the first example, it is possible to suppress the tightness of the communication band by reducing the number of simple information added when the ratio of the communication bandwidth currently in use is high. Further, according to the second example, by adding a lot of simple information when the packet loss rate of the network is large, even if a plurality of drawing command packets are lost at the same time, drawing by the simple information can be performed. It becomes possible. Further, according to the third example, a lot of simple information is added when it is expected that the screen is continuously updated over a wide range of the desktop, such as a drag operation of the window, so that drawing by packet loss can be performed. Chipping can be suppressed more effectively.

The addition determination unit 103 may determine whether or not to add the simple information and the sequence number of the simple information by combining the first example, the second example, and the third example. . Further, the addition determination unit 103 may not make a determination using the information of the third example of the first example and the second example. For example, the addition determination unit 103 may always determine that a fixed number of simple information and a sequence number of simple information are added. Alternatively, information such as the number of colors, the size, and the position of the update image of the update information may be used. Alternatively, an application or GUI type including the update information may be used.

Next, in step S107, the drawing command creation unit 104 receives update information, a sequence number, and information related to the addition of simple information (whether or not to add simple information or a sequence number to be added) from the addition determination unit 103. First, the sequence number of the simple information to be added is output to the simple information storage unit 107, and the acquisition of the simple information corresponding to the sequence number is requested. In response to this, the simple information storage unit 107 searches its own storage area and returns corresponding simple information.

Next, in S108, the drawing command creating unit 104 creates a drawing command. The drawing command is composed of a pair of update information and sequence number input from the addition determination unit 103 and a pair of simple information and sequence number acquired from the simple information storage unit 107. The drawing command creating unit 104 configures a drawing command from the information after the encoding unit 110 compresses the update image included in the update information. Here, the encoding unit 110 performs arbitrary image compression processing such as JPEG. The drawing command creation unit 104 outputs the created drawing command to the drawing command transmission unit 105 and requests transmission of the drawing command. When the drawing command transmission unit 105 receives the drawing command from the drawing command creation unit 104, the drawing command transmission unit 105 adds information necessary for transmission on the network 3, such as a UDP header or an IP (Internet Protocol) header, and then sends the client command to the client terminal 2. Send to.

On the other hand, when it is determined in S106 that the simple drawing command is not added, in Step S108, when the drawing command creating unit 104 receives the update information and the sequence number from the adding determining unit 103, the same processing as Step S107 is performed. However, unlike the step S107, simple information is not acquired, so that the update command included in the update information is subjected to compression processing by the encoding unit 110, and then a rendering command is formed from the update information after compression processing and the sequence number. . The drawing command creation unit 104 outputs the drawing command to the drawing command transmission unit 105 and requests transmission. When the drawing command transmission unit 105 receives the drawing command from the drawing command creation unit 104, the drawing command transmission unit 105 adds information necessary for transmission on the network 3, such as a UDP header or an IP (Internet Protocol) header, and then sends the client command to the client terminal 2. Send to.

Next, an operation when the loss notification packet is received from the client terminal 2 among the processes performed by the server terminal 1 by the screen sharing system according to the present embodiment will be described.

FIG. 16 is a flowchart showing an operation when the server terminal 1 receives a loss notification packet.

When the loss notification reception unit 113 receives the loss notification packet from the client terminal 2, the loss notification reception unit 113 extracts the loss notification from the packet and notifies the loss notification analysis unit 112 in step S201. The loss notification is composed of one or more sequence numbers.

In S202, when the loss notification analysis unit 112 extracts the sequence number from the loss notification, the loss notification analysis unit 112 outputs the sequence number to the update information history storage unit 115 and requests acquisition of area information corresponding to the sequence number. As described above, the update information history storage unit 115 stores the sequence number and the history of the area information in its own storage area, searches the history corresponding to the requested sequence number, and outputs the corresponding area information. When the loss notification analysis unit 112 receives the region information, the loss notification analysis unit 112 requests the frame buffer 111 to obtain the latest screen data relating to the region indicated by the region information. Note that the loss notification analysis unit 112 does not necessarily request acquisition of screen data relating to the same area as the acquired area information. An example is shown in FIG. FIG. 17 illustrates a case where sequence numbers 100, 101, and 102 are included in the loss notification received by the loss notification analysis unit 112. Further, at this time, the value of the region information acquired from the update information history storage unit 115 is also written for each of the sequence numbers. At this time, the loss notification analysis unit 112 does not acquire screen data from the frame buffer 111 for each of the sequence numbers 100, 101, and 102, but can reconfigure the region information so that there is no overlapping region. . FIG. 18 shows information on the area after reconstruction. The loss notification analysis unit 112 acquires the latest screen data from the frame buffer 111 for the reconfigured area. In this way, by reconfiguring the region information so that there is no overlapping portion, it is possible to suppress consumption of CPU resources and compression of communication bandwidth due to unnecessary screen data acquisition. When the loss notification analysis unit 112 acquires the screen data, the loss notification analysis unit 112 forms a pair with the corresponding region information, and outputs this to the sequence number adding unit 102.

In step S203, the process flow proceeds to step S102 in FIG. The sequence number adding unit 102 receives update information from the update information generation unit 101 for each pair of input image data and region information (if a plurality of pairs are input). Process in the same way as After that, when update information is input from the update information generation unit 101, the processing of S103 to S109 is performed, and the drawing command transmission unit 105 sends a drawing command including a pair of image data, region information, and sequence number to the client terminal 2. Send.

Next, processing performed by the client terminal 2 will be described with reference to FIG. FIG. 19 is a flowchart showing the operation of the client terminal 2.

In S301, when receiving the drawing command packet via the network 3, the drawing command receiving unit 201 extracts the drawing command from the drawing command packet. The drawing command includes a sequence number / update information pair and zero or more simple information / sequence number pairs. The drawing command receiving unit 201 outputs the drawing command to the loss detection unit 202 and requests loss detection.

In S302, when receiving the drawing command, the loss detection unit 202 extracts a sequence number corresponding to the update information from the drawing command, and determines whether or not a loss has occurred in receiving the drawing command packet. The loss detection unit 202 refers to the sequence number count value stored in its own sequence number count value storage area, and checks whether or not the extracted sequence number matches this sequence number count value. If they match (indicated by = in S302 of FIG. 19), it is determined that no loss has occurred, and the loss detection unit 202 outputs the drawing command to the drawing determination unit 203. The process flow proceeds to step S305. If they do not match and the extracted sequence number is newer than the sequence number count value (>), it is determined that a packet loss has occurred, and the flow of processing proceeds to step S303. If they do not match and the extracted sequence number is older than the sequence number (<), it is determined that a previously arrived drawing command packet has arrived in duplicate, the drawing command is discarded, and this processing ends. (Step S304). For example, if the sequence number count value is 100 and the received sequence number is 103, the loss detection unit 202 determines that a loss has occurred in the rendering command packet corresponding to the sequence numbers 100, 101, and 102. On the other hand, when the sequence number is older than the sequence number count value (for example, 98), the loss detection unit 202 determines that the received packet that has arrived previously has arrived in duplicate. In any of the three cases, after determination, the sequence number count value is overwritten and updated with the value of the sequence number.

In step S303, the loss detection unit 202 outputs the sequence number (100, 101, 102 in the above example) of the rendering command packet in which the loss has occurred to the loss notification transmission unit 208, and requests transmission of the loss notification. In response to this, the loss notification transmission unit 208 adds information necessary for transmission in the network 3, such as a UDP header or an IP (Internet Protocol) header, to these sequence numbers, and then sends them to the server terminal 1. To send. When the transmission of the loss notification is completed, the loss detection unit 202 outputs a combination of the drawing command input from the drawing command receiving unit 201 and the sequence number of the drawing command in which the loss has occurred to the drawing determination unit 203. The process flow proceeds to S305.

In S305, when the drawing determination unit 203 receives the drawing command, it first checks whether or not simple information is included in the drawing command. If not included, the drawing determination unit 203 outputs update information included in the drawing command to the drawing execution unit 204 and requests execution of drawing using the update information. The process flow proceeds to step S308. If simple information is included, the drawing determination unit 203 next determines whether or not to perform drawing using simple information. The process flow proceeds to S306.

In S306, when a drawing command is input from the loss detection unit 202, the drawing determination unit 203 determines whether or not to execute drawing with the added simple information according to the following procedure. First, the drawing determination unit 203 confirms whether or not a drawing command and information on a sequence number in which a loss has occurred are input together. If it has been input, it is further confirmed whether or not the simple information corresponding to the value of the sequence number where the loss has occurred is included in the drawing command. When the simple information corresponding to the value of the sequence number in which the loss has occurred is included in the drawing command, the drawing determining unit 203 determines to perform drawing using the simple information. In this case, the process flow proceeds to step S307. In other cases, that is, when the information of the sequence number where the loss has occurred is not input together, or when the simplified information corresponding to the value of the sequence number where the loss occurs is not input together, the drawing determination The unit 203 determines not to perform drawing with simple information. In this case, the drawing determination unit 203 outputs update information included in the drawing command to the drawing execution unit 204 and requests execution of drawing using the update information. The process flow proceeds to step S308.

In S307, when the drawing determination unit 203 determines to execute drawing with simple information, the drawing determination unit 203 outputs the simple information to the drawing execution unit 204 and requests execution of drawing. When the drawing execution unit 204 receives the simple information, the drawing execution unit 204 overwrites and updates the area indicated by the area information of the simple information in the frame buffer 205. Upon receiving a drawing completion notification, the drawing determination unit 203 outputs update information included in the drawing command received from the loss detection unit 202 to the drawing execution unit 204 and requests execution of drawing based on the update information. The process flow proceeds to step S308.

In step S308, upon receiving input of update information, the drawing execution unit 204 extracts area information and an update image from the update information, and overwrites and updates the area indicated by the area information in the frame buffer 205 with the update image. If necessary, decompression processing by the decoding unit 207 is performed. The display unit 206 displays screen data in the frame buffer 205.

Specific examples of processing from step S301 to step S308 will be given. When the sequence number count value of the loss detection unit 202 is 100, it is assumed that a drawing command packet including update information of the sequence number 101 is received. As shown in FIG. 20, it is assumed that simple information of sequence number 100 is added to this drawing command in addition to update information. At this time, since the current sequence number count value is 100, the loss detection unit 202 expects that the sequence number added to the update information of the received drawing command packet is 100. However, actually, since the drawing command with the sequence number 101 added to the update information has been received, the loss detection unit 202 determines that the drawing command with the sequence number 100 has been lost in the network 3 (step S302). The loss detection unit 202 notifies the server terminal 1 through the loss notification transmission unit 208 that a loss has occurred in the sequence number 100 (step S303), and then draws update information of the sequence number 101 and simple information of the sequence number 100. The data is output to the determination unit 203. At this time, the fact that the sequence number 100 is lost is also notified. In response to this, the drawing determination unit 203 adds simple information to the input (step S305), and this simple information corresponds to the sequence number (100) in which the loss has occurred. It is determined to perform drawing with this simple information (step S306). Therefore, the drawing execution unit 204 first executes drawing with the simple information of the sequence number 100 (fills the specified area with the specified color information) (step S307), and then executes (updates) the drawing with the update information of the sequence number 101. An image is drawn) (step S308).

Examples of results of drawing using simple information are shown in FIGS.

As described above, according to the present embodiment, the client terminal 2 receives the update information packet including the simple information generated from the past update information, and only receives the simple information corresponding to the loss when the packet loss is detected. Use to draw. Conventionally, when a packet loss occurs, the drawing is missing, but when the present invention is applied, it is detected that a loss has occurred, and an emergency drawing process is performed using simple information as an alternative. . As a result, it is possible to avoid a sense of incongruity of drawing disruption recognized by the user at the time of packet loss. The present invention can be expected to be particularly effective in a network where the round trip time between the server terminal and the client terminal is large, such as via a WAN.

The server terminal 1 according to the present embodiment can also be realized by using a general-purpose computer device as basic hardware. That is, the update information generation unit 101, the sequence number addition unit 102, the addition determination unit 103, the drawing command creation unit 104, the drawing command transmission unit 105, the simple information generation unit 106, the simple information storage unit 107, Input state acquisition unit 108, communication state acquisition unit 109, encoding unit 110, frame buffer 111, loss notification analysis unit 112, loss notification reception unit 113, generation determination unit 114, and update information history storage unit 115 The display unit 116 and the CPU state acquisition unit 117 can be realized by executing a program on a processor mounted on the computer device. At this time, the server terminal 1 can be realized by executing the above program. At this time, the server terminal 1 may be realized by installing the above program in a computer device in advance, or may be stored in a storage medium such as a CD-ROM or distributed through the network. Thus, this program may be realized by appropriately installing it in a computer device. The update information history storage unit 115, the simple information storage unit 107, and the frame buffer 111 are a memory, a hard disk or a CD-R, a CD-RW, a DVD-RAM, a DVD-R incorporated in or attached to the computer device. It can be realized by appropriately using a storage medium such as

The client terminal 2 according to the present embodiment can also be realized by using a general-purpose computer device as basic hardware. That is, the drawing command receiving unit 201, the loss detecting unit 202, the drawing determining unit 203, the drawing executing unit 204, the frame buffer 205, the display unit 206, the decoding unit 207, the loss notification transmitting unit 208, the update The information history storage unit 209 can be realized by executing a program on a processor mounted on the computer device. At this time, the client terminal 2 can be realized by executing the above program. At this time, the client terminal 2 may be realized by installing the above program in a computer device in advance, or may be stored in a storage medium such as a CD-ROM or distributed through the network. Thus, this program may be realized by appropriately installing it in a computer device. The update information history storage unit 209 and the frame buffer 205 include a memory, a hard disk or a storage medium such as a CD-R, a CD-RW, a DVD-RAM, a DVD-R, etc. incorporated in or externally attached to the computer device. It can be realized by appropriately using.

Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Server terminal, 2 ... Client terminal, 3 ... Network, 101 ... Update information generation part, 102 ... Sequence number addition part, 103 ... Addition judgment part, 104 ... Drawing command creation unit, 105 ... Drawing command transmission unit, 106 ... Simple information generation unit, 107 ... Simple information storage unit, 108 ... Input state acquisition unit, 109 ... Communication state acquisition unit, DESCRIPTION OF SYMBOLS 110 ... Encoding part, 111 ... Frame buffer, 112 ... Loss notification analysis part, 113 ... Loss notification receiving part, 114 ... Generation judgment part, 115 ... Update information log | history storage part, 116: Display unit, 117: CPU state acquisition unit, 201: Drawing command reception unit, 202: Loss detection unit, 203: Drawing determination unit, 204: Drawing execution unit, 205 ... Frame buffer, 06 ... display unit, 207 ... decoding unit, 208 ... loss notification transmission unit, 209 ... update information history storage unit

Claims (7)

A server that shares an image to be displayed on the screen with a client terminal,
A first storage unit for storing an image to be displayed on the screen;
A display unit for displaying an image of the first storage unit on the screen;
An update information generation unit that generates update information including an update image that updates a part of the image stored in the first storage unit and area information that represents a position of the screen on which the update image is displayed;
A sequence number adding unit for adding a sequence number for each update information;
An information amount equal to or less than the information amount of the update information, and a simple information generation unit that generates simple information corresponding to the update information;
A second storage unit that stores the simplified information and the sequence number added to the update information corresponding to the simplified information in association with each other;
Drawing command creation for creating a drawing command including the update information, a sequence number added to the update information, simple information associated with a sequence number older than the sequence number added to the update information, and the old sequence number And
A transmission unit that generates and transmits a drawing command packet from the drawing command. A system comprising: the server according to claim 1; and a client terminal that shares an image to be displayed on a screen with the server according to claim 1.
The client terminal is
A third storage unit for storing an image to be displayed on the screen;
A second display unit for displaying an image of the third storage unit on the screen;
A drawing command receiving unit that receives the drawing command packet from the server and extracts the drawing command from the drawing command packet;
A drawing unit for drawing the updated image included in the drawing command in a third storage unit;
A detection unit that detects whether or not there is a drawing command packet that has not been received by the drawing command receiving unit among drawing command packets transmitted by the server from the sequence number added to the update information;
The drawing unit has the same sequence number as the sequence number added to the update information included in the unreceived drawing command packet when the detecting unit detects that there is the unreceived drawing command packet. A client terminal characterized in that it is simplified information associated with a number and is rendered in the third storage unit based on the simplified information included in the rendering command. The system according to claim 2, wherein the simple information is information including area information and color information representing a color drawn in an area indicated by the area information. The system according to claim 2, wherein the simple information is information including a lossy compressed image of the updated image and region information. The simple information is information including a sequence number and area information older than the sequence number added to the update information included in the drawing command,
Each time the client terminal receives a drawing command packet from the server, the client terminal further includes a fourth storage unit that stores the update information included in the drawing command packet in association with the sequence number added to the update information. Prepared,
The drawing unit of the client terminal draws in the third storage unit based on update information corresponding to the same sequence number as the old sequence number included in the simplified information and stored in the fourth storage unit The system according to claim 2. When the packet loss rate in the network is larger than a first threshold, the drawing command creation unit of the server uses the update information, the sequence number added to the update information, and the sequence number added to the update information. Create a simple instruction associated with the old sequence number and a drawing command that includes the old sequence number.
When the packet loss rate in the network is smaller than a predetermined value, the simplified information includes the update information and the sequence number added to the update information, and is associated with a sequence number older than the sequence number added to the update information. 6. The system according to claim 5, wherein a drawing command not including the information and the old sequence number is created. When the packet loss rate in the network is greater than a second threshold value that is greater than a first threshold value, the drawing command creation unit is configured to update the update information, a sequence number added to the update information, a plurality of simplified information, The system according to claim 6, wherein a drawing command including a sequence number associated with the simple information is created.

Images