JP2016018260A - Client server system, control method, and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a client server system for controlling a timing at which execution screen information generated on the basis of a predicted operation is received at a client side in accordance with the communication processing capability of a network, and for displaying the execution screen at an appropriate timing.SOLUTION: A client server system includes: input means for detecting the approach of an indication object; operation prediction means for measuring communication processing capability between a client terminal and a server device, and for calculating a prediction contact position and a prediction contact time on the basis of the change of the position information of the indication object acquired by the input means, and for transmitting operation prediction information on the basis of the communication processing capability; execution screen information generation means for generating execution screen information including the execution result of processing corresponding to the prediction contact position; and display means for controlling the display of an execution screen included in the execution screen information generated by the execution screen information generation means such that the execution screen is displayed on a display installed in the client terminal after the prediction contact time.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a client server system in which a client terminal and a server device are connected via a network, a control method, and a control program. In particular, the present invention relates to a client server system, a control method, and a control program in which a client terminal and a server device constitute a thin client system.

  In a general thin client system, an OS or application software that originally operates on a client terminal is operated on a server device as a virtual client terminal (OS: Operating System). Then, the OS and application software are operated on the server device, and an execution screen of a virtual terminal operating in the server device is displayed on the user client terminal. When the user performs an operation on the display on the screen of the client terminal, the client terminal transmits the user's operation as operation information to the server device. The server device that has received the operation information performs the operation of the contents described in the operation information on the virtual client terminal operating in the server device, and changes the operation of the OS or application software. At this time, the server device sequentially acquires the screen display of the virtual terminal and transmits it to the client terminal as execution screen information. The client terminal sequentially displays the received execution screen information on the screen of the client terminal.

  In a general thin client system, a client terminal and a server device are connected via a network. Therefore, a delay time called a transmission delay occurs in data transfer in communication between the client terminal and the server device. When other communication traffic increases on a line used for communication between the client terminal and the server apparatus (referred to as cross traffic), the communication throughput that can be transmitted and received between the client terminal and the server apparatus decreases. When the cross traffic increases, it takes extra time to transmit and receive the same data, and the transmission delay increases. Also, if the wireless quality deteriorates due to the movement of the client terminal connected to the mobile network, the communication throughput decreases as the cross traffic increases, and the transmission delay increases.

  In a general thin client system, a transmission delay occurs in transmission / reception of operation information and execution screen information on a network between a client terminal and a server device, as compared with a case where an application is executed in a client terminal. As a result, there is a problem that the response time from when the user operates the client terminal to when the screen reflecting the user operation is displayed is increased by the transmission delay.

  Patent Document 1 discloses a thin client system in which the local environment is configured as a network boot type and the remote environment is configured as a screen transfer type. In the thin client system of Patent Document 1, since the user data file server for storing user data is shared so as to be accessible from both the remote environment and the local environment, the user data can be unified. As a result, user data management becomes easy, and it becomes easy to select the local environment and the remote environment according to the use environment, so that the transmission delay can be reduced depending on the use environment.

  Japanese Patent Application Laid-Open No. 2004-228688 allows a job setting operation to be performed before the apparatus is started by displaying a simple operation screen between when the apparatus is turned on and when the normal operation screen is displayed. A job processing apparatus is disclosed. In the job processing apparatus of Patent Literature 2, immediately after the apparatus is started, a simple screen prepared in advance is displayed to accept input from the user. When the loading and initialization of the program are completed, a normal screen that inherits the setting information input on the simplified screen is displayed. As a result, the response time from the start of the apparatus to the display of the normal operation screen can be shortened, and the job can be set on the simplified screen. Therefore, an effect that leads to reducing the transmission delay of the network can be obtained.

  In recent years, information terminals are often provided with a touch panel (PC: Personal Computer), such as a smartphone or a tablet PC. The touch panel has a method such as a capacitance method or an electromagnetic induction method, and can detect the approach / contact of a pointing object such as a finger or a pen by using a change in capacitance or electromagnetic energy. In an information terminal equipped with such a touch panel, when the pointing object approaches, the contact position of the pointing object on the touch panel can be predicted. Therefore, before the pointing object comes into contact with the touch panel, it is possible to predict an operation that occurs due to the touch on the touch panel based on the approach position of the pointing object and the display position of the content.

  Patent Document 3 discloses a server-based computing system that predicts a contact position of an electromagnetic induction pen on a display panel of a client terminal and performs processing on the server side in advance based on the predicted contact position. Yes. In the server-based computing system of Patent Document 3, first, the client terminal predicts the tap position from the coordinates of the pen in a non-contact state, selects a program selection item based on the predicted tap position, and selects the selected selection. The item is transmitted to the server device. Then, the server device executes a program related to the selected selection item and stores an execution screen. When the server device receives the tap position information from the client terminal and the actual tap position information matches the predicted position information, the server device transmits the stored execution screen to the client terminal.

JP 2007-310508 A JP2013-52682A JP 2009-238087 A

  According to the thin client system of Patent Document 1, since the remote environment and the local environment can be switched, the transmission delay can be reduced depending on the use environment. However, in a general thin client system, there are many cases where the local environment cannot be used, and thus there is a problem that the transmission delay of the remote environment factor cannot be solved. This is because when the local environment cannot be used, the remote environment and the local environment cannot be switched.

  According to the job processing apparatus of Patent Document 2, the response time of the simple screen immediately after starting the apparatus is earlier than the response time of the normal screen. However, there is a problem in that the response time from the start of the device operation to the display of the simple screen is not shortened. In addition, since the job processing apparatus of Patent Document 2 only supports the screen immediately after the apparatus is activated, the response time for operations on other screens after activation is not shortened. Furthermore, there is a problem that it is necessary to prepare a simple screen in advance and the versatility is low.

  According to the server-based computing system of Patent Document 3, a tap position is predicted before the pen performs a tap operation, and a program corresponding to the predicted tap position is executed to prepare an execution screen. Response speed can be improved. However, when there is a delay factor in the network, there is a problem in that a delay may occur in the transfer time of the execution screen from the server device to the client terminal. This is because the network communication processing capability is not taken into consideration when the execution screen is transmitted from the server device to the client terminal.

  An object of the present invention is to control the timing at which execution screen information generated based on a predicted operation is received on the client side according to the communication processing capability of the network, and display the execution screen at an appropriate timing Is to provide a system.

The client server system of the present invention is a thin client system in which processing corresponding to an operation on the client terminal is executed by the server device, and by detecting the approach and contact of the pointing object with respect to the input surface provided in the client terminal, the input The input unit that acquires the position information of the pointing object with respect to the surface in association with the detection time, and the communication processing capability between the client terminal and the server device are measured, and the change in the position information of the pointing object acquired by the input unit Based on the predicted contact position, which is the position where the pointing object is predicted to contact the input surface, and the predicted contact time, which is the time when the pointing object is predicted to contact the input surface, and the identifier of the client terminal, Operation prediction information including predicted contact position and predicted contact time is sent based on communication processing capability. Predicting contact by executing in the virtual environment on the server device the operation predicting means that receives the operation prediction information and executes the process executed corresponding to the predicted contact position on the input surface of the client terminal to which the identifier is attached. Execution screen information generation means for generating execution screen information including screen information that is an execution result of the process corresponding to the position;
Control is performed so that the execution screen information generated by the execution screen information generation means is received and the execution screen corresponding to the screen information included in the received execution screen information is displayed on the display provided in the client terminal after the predicted contact time. Display means.

  The control method of the present invention is a thin client system in which processing corresponding to an operation on a client terminal is executed by a server device, by detecting the approach and contact of a pointing object with respect to the input surface provided in the client terminal. The position information of the pointing object is acquired in association with the detection time, the communication processing capability between the client terminal and the server device is measured, and the pointing object is input based on the change in the acquired position information of the pointing object The predicted contact position that is predicted to contact the surface and the predicted contact time that is the time when the pointing object is predicted to contact the input surface are calculated, and the identifier of the client terminal, the predicted contact position, and the predicted contact time are calculated. The operation prediction information including and is transmitted based on the communication processing capability, the operation prediction information is received, and the identifier is attached. Execution screen information including screen information that is the execution result of the process corresponding to the predicted contact position is executed in the virtual environment on the server device by executing the process executed corresponding to the predicted contact position on the input surface of the client terminal. Control is performed so that an execution screen corresponding to the screen information included in the generated execution screen information is displayed on the display provided in the client terminal after the predicted contact time.

  A control program according to the present invention is a thin client system that executes processing corresponding to an operation on a client terminal on a server device, and detects an approach and contact of a pointing object with respect to the input surface provided on the client terminal. Processing for acquiring the position information of the pointing object with respect to the detection time, measuring the communication processing capability between the client terminal and the server device, and based on the change in the positional information of the acquired pointing object Calculates a predicted contact position that is a position predicted to contact the input surface and a predicted contact time that is a time when the pointing object is predicted to contact the input surface, and the identifier of the client terminal, the predicted contact position, and the prediction Processing to transmit operation prediction information including contact time based on communication processing capability, and reception of operation prediction information Screen information that is the execution result of the process corresponding to the predicted contact position by executing, in the virtual environment on the server device, the process executed corresponding to the predicted contact position on the input surface of the client terminal to which the identifier is attached And a process for controlling the execution screen corresponding to the screen information included in the generated execution screen information to be displayed on the display provided in the client terminal after the predicted contact time. Let the computer run.

  According to the present invention, the client server which controls the timing at which the execution screen information generated based on the predicted operation is received on the client side according to the communication processing capability of the network and displays the execution screen at an appropriate timing A system can be provided.

It is a block diagram which shows the function structure of the client server system which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the function structure of the thin client system which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the flow of a process of the thin client system which concerns on the 2nd Embodiment of this invention. It is a flowchart for demonstrating the connection process in the thin client system which concerns on the 2nd Embodiment of this invention. It is a flowchart for demonstrating the measurement process in the thin client system which concerns on the 2nd Embodiment of this invention. It is a conceptual diagram for demonstrating the change of the positional relationship of the input panel of a client terminal, and a pointing object in the thin client system which concerns on the 2nd Embodiment of this invention. It is a flowchart for demonstrating the operation prediction process in the thin client system which concerns on the 2nd Embodiment of this invention. It is a flowchart for demonstrating the execution screen production | generation process in the thin client system which concerns on the 2nd Embodiment of this invention. It is a flowchart for demonstrating the execution screen display process in the thin client system which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the function structure of the thin client system which concerns on the 3rd Embodiment of this invention. It is a block diagram which shows an example of the hardware constitutions of the client terminal which concerns on embodiment of this invention. It is a block diagram which shows an example of the hardware constitutions of the server apparatus which concerns on embodiment of this invention.

  EMBODIMENT OF THE INVENTION Below, the form for implementing this invention is demonstrated using drawing. However, the preferred embodiments described below are technically preferable for carrying out the present invention, but the scope of the invention is not limited to the following.

(First embodiment)
The client server system 1 according to the first embodiment of the present invention will be described with reference to FIG. The client server system 1 according to the first embodiment shows a schematic configuration of the client server system according to the embodiment of the present invention, and the detailed configuration and operation are the second and third. This will be described in the embodiment.

  The client server system 1 according to the first embodiment is realized as a thin client system. For example, the client server system 1 can be realized as a screen transfer type thin client system. As the screen transfer method, a server blade method, a blade PC method, a VDI method, or the like can be applied (PC: Personal Computer, VDI: Virtual Desktop Infrastructure). The client server system 1 according to the present embodiment may be realized as a network boot type thin client system.

  As shown in FIG. 1, the client server system 1 according to the first embodiment has a configuration in which a client terminal 10 and a server device 20 are communicably connected via a network 30. In FIG. 1, only one client terminal 10 and one server device 20 are illustrated, but actually, at least one client terminal 10 and at least one server device 20 are connected to each other via a network 30. It can be set as the structure connected. Although not shown in FIG. 1, the client terminal 10 and the server device 20 are provided with communication means for transmitting and receiving data to and from each other.

(Client terminal)
First, an outline of the client terminal 10 according to the present embodiment will be described.

  The client terminal 10 is an information terminal connected to the server device 20 via the network 30. For example, the client terminal 10 can be realized as an information terminal including a touch panel, such as a smartphone or a tablet PC. When the client terminal 10 is realized by a smartphone or a tablet PC, it is preferable to apply a touch panel having both an input function and a display function. The client terminal 10 may be realized as an information terminal having a touch panel as an input function and a display as a display function separately.

  For example, the client terminal 10 may be realized by a desktop PC, a notebook PC, a mobile phone, a PDA, or the like (PDA: Personal Data Assistance). Further, for example, the client terminal 10 may be realized as a stationary terminal device such as an ATM, a cash dispenser, or a public terminal (ATM: Automated Teller Machine). Further, for example, the client terminal 10 may be realized as a portable data terminal used for ticket / ticket issuance, order reception, article management, payment terminal, and the like. Note that the above-described example of the client terminal 10 is an example, and does not limit the scope of the present invention.

  As shown in FIG. 1, the client terminal 10 includes an input unit 11, an operation prediction unit 12, and a display unit 13.

  The input unit 11 has a function of detecting a pointing object that is in close proximity to or in contact with the client terminal 10, and is a unit that receives an input operation by a user. The input means 11 has an input surface that is in contact with the input operation, and outputs different output values depending on the distance between the pointing object adjacent to the input surface and the input surface. Therefore, the distance between the pointing object and the input surface can be measured by analyzing the output value of the input unit 11. Note that the distance between the pointing object and the input surface is output as the location of the pointing object associated with the detection time. If the change of the location of the pointing object is verified, the moving speed of the pointing object and the moving locus of the pointing object can be grasped.

  The input unit 11 is realized by a touch panel, for example. As the touch panel, those having a function as a proximity sensor such as an electromagnetic induction type, a capacitance type, an ultrasonic type, an electromagnetic type, and an infrared type are suitable. Note that the touch panel used as the input unit 11 is not limited to the above-described one, and it only needs to have a function of detecting a close object and determining a distance from the detected object.

  When the input unit 11 detects the pointing object, the input unit 11 notifies the operation prediction unit 12 of the actual measurement information including the detection state regarding the approach / contact of the pointing object, the location of the pointing object, and the detection time.

  The operation predicting unit 12 performs communication processing such as the throughput of data transmitted / received via the network 30, the size of execution screen information, and the round-trip delay time based on data transmitted / received between the client terminal 10 and the server device 20. It has a communication measurement function that measures the ability. Further, the operation prediction unit 12 has an input prediction function for predicting the operation content and the operation type based on the location, detection time, and detection state of the detected pointing object. Note that each of the predicted contact position, contact time, and operation type is referred to as a predicted contact position, predicted contact time, and predicted operation type. The predicted contact position, predicted contact time, and predicted operation type are collectively referred to as operation prediction information.

  The operation prediction unit 12 calculates a moving speed based on a change in the location of the pointing object with respect to the pointing object detected by the input unit 11, and predicts a contact time or a prediction where a contact between the pointing object and the input surface is predicted. The contact position is calculated. Further, the operation prediction unit 12 predicts the operation type from the detection state of the pointing object. The operation type is, for example, a type of operation performed by the user such as a tap operation or a flick operation. The operation prediction information including the predicted contact time, the predicted contact position, and the predicted operation type derived by the operation prediction unit 12 is transmitted to the server device 20 according to the communication processing capability of the network 30.

  Further, the operation predicting means 12 is based on the measured parameters, the time required to transmit / receive data between the client terminal 10 and the server device 20 (data transmission / reception time), and the time at which the operation prediction information is transmitted to the server device 20. (Operation prediction information transmission time) is calculated. And the operation prediction means 12 is provided with the function which controls the timing which transmits operation prediction information so that an execution screen can be displayed immediately after prediction contact time.

  The data transmission / reception time is estimated using the data size exchanged between the client terminal 10 and the server device 20 and the throughput of the communication means. The time for the operation prediction information to be delivered from the client terminal 10 to the server device 20 (operation prediction information delivery time) is predicted from the data size of the operation prediction information and the throughput of the communication means. The processing time of the application is predicted based on the processing content of the application corresponding to the predicted contact position and the predicted operation type. The time (transmission delay) required to deliver the execution screen information from the server device 20 to the client terminal 10 is obtained from the propagation time of the average execution screen information size, for example.

  The display unit 13 has a function of receiving execution screen information from the server device 20 and displaying an execution screen of an application acquired using the predicted operation time calculated by the operation prediction unit 12 or the contact detection of the pointing object as a trigger. . The display unit 13 is realized as a display provided in the client terminal 10. When the input unit 11 is configured by a touch panel, an execution screen is displayed on the touch panel itself.

  The above is the outline of the client terminal 10 according to the present embodiment.

(Server device)
Next, an outline of the server device 20 according to the present embodiment will be described.

  The server device 20 is a device that executes an application designated by the client terminal 10 in a virtual environment and generates execution screen information including information (screen information) related to an execution screen to be displayed on the client terminal 10. The server device 20 is configured by a server that can construct a virtual environment.

  The server device 20 includes an execution screen information generation unit 21 and executes an application in the virtual environment 22 according to the operation prediction information transmitted from the client terminal 10. That is, the server device 20 executes the application in advance in the virtual environment 22 based on the operation prediction information. If the predicted operation position is known, the position where the pointing object touches the input surface can be predicted, and therefore the processing content of the application to be executed can be predicted.

  The execution screen information generation unit 21 generates execution screen information including information related to the execution screen displayed on the client terminal 10. The execution screen information is information obtained by converting the execution result of the application executed in the virtual environment 22 into a format including an execution screen displayed on the client terminal 10.

  The virtual environment 22 is a virtualized environment for executing an application. The virtual environment 22 is a computer (virtual machine) virtually constructed by software on the server device 20. The virtual environment 22 is realized by virtualization such as a hypervisor type or a container type. In the case of the hypervisor type, an application including an OS is executed on a virtual machine constructed on the server device 20 by the hypervisor. That is, in the hypervisor type, the virtual environment 22 is constructed using resources such as a CPU, memory, and storage provided by a virtual machine (CPU: Central Processing Unit). In the case of the container type, an application is executed in a partial area of the OS that runs on the server device 20. That is, in the container type, the virtual environment 22 is constructed using a dedicated area of the server device 20.

  The server device 20 transmits the generated execution screen information to the client terminal 10 so that the execution screen is displayed after the timing when the pointing object contacts the client terminal 10 (after the predicted contact time). That is, the server device 20 transmits the execution screen information so that the client terminal 10 can receive the execution screen information before the predicted contact time.

  The above is the outline of the server device 20 according to the present embodiment.

(network)
Finally, the network 30 according to the present embodiment will be described.

  The network 30 is a data transmission path that connects the client terminal 10 and the server device 20. If an external environment network is used, the general Internet may be used as the network 30. If an internal environment network of a specific facility is used, an intranet such as a LAN constructed in the internal environment of the facility may be used as the network 30 (LAN: Local Area Network). The network 30 may have a configuration included in the client server system 1 according to the present embodiment, or may be a configuration not included in the client server system 1. The form of the network 30 and the data transmission system inside the network 30 are not particularly limited.

  In addition, the form in which each of the client terminal 10 and the server device 20 is connected to the network 30 can be arbitrarily set. For example, the client terminal 10 and the network 30 may be wired via a connector or the like, or may be wirelessly connected using an access point or a wireless LAN. An arbitrary interface can be applied as an interface for the client terminal 10 and the server device 20 to connect to the network 30.

  The above is the description of the network 30 according to the present embodiment.

  As described above, in the client server system according to the first embodiment of the present invention, before the user operates the client terminal, the operation content or operation type by the user is predicted, and the OS or application on the server device is determined. Run ahead. At this time, the timing at which the execution screen information generated based on the predicted operation is received on the client side is controlled according to the communication processing capability of the network. Then, the execution screen is displayed without a sense of incongruity when the operation by the user is actually performed. As a result, it is possible to substantially reduce the response time from when the user operation is performed until the execution screen information is displayed.

  That is, it is possible to control the timing at which the execution screen information generated based on the predicted operation is received on the client side according to the communication processing capability of the network, and display the execution screen at an appropriate timing.

  The control method of the client server system according to the present embodiment as described above is highly versatile because it can be applied to all applications operable from the client terminal.

  The above is the description of the client server system according to the first embodiment. Details of the internal configuration and operation of the client server system 1 will be described in detail using the second and third embodiments.

(Second Embodiment)
Next, a thin client system 2 according to the second embodiment of the present invention will be described in detail with reference to the drawings. The thin client system 2 embodies the client server system 1 according to the first embodiment.

  FIG. 2 is a block diagram showing the configuration of the thin client system 2 according to the second embodiment of the present invention. The thin client system 2 has a configuration in which the client terminal 10 and the server device 20 are connected via a network 30. The thin client system 2 includes at least one client terminal 10 and at least one server device 20. For example, if a plurality of virtual environments 22 can be constructed in a single server device 20, a plurality of client terminals 10 can be connected to one server device 20. Further, if processing of a single client terminal 10 is distributed to a plurality of server devices 20, the single client terminal 10 can be connected to the plurality of server devices 20.

(Client terminal)
First, the functional configuration of the client terminal 10 according to the present embodiment will be described with reference to FIG.

  The client terminal 10 includes an input unit 11, an operation prediction unit 12, a display unit 13, and a communication unit 14. The input unit 11, the operation prediction unit 12, and the display unit 13 correspond to the constituent elements of the first embodiment.

<Input means>
The input unit 11 has a function of detecting the approach or contact of a pointing object such as a finger or a pen on an input surface that receives input from the user.

  When the input unit 11 detects the pointing object, the input prediction unit 123 inputs actual measurement information including information indicating whether the pointing object is approaching or touching, the location of the object, and the detection time. Notify

<Operation prediction means>
As shown in FIG. 2, the operation prediction unit 12 measures the communication processing capability between the client terminal 10 and the server device 20 and performs operations such as a predicted contact position, a predicted contact time, and a predicted operation type regarding the detected pointing object. Derived prediction information.

  The operation prediction unit 12 includes a communication measurement unit 121, an input prediction unit 123, and an operation prediction information transmission unit 125.

  The communication measuring unit 121 is a function for measuring communication processing capability such as the round-trip delay time and throughput between the client terminal 10 and the server device 20 and the average size of execution screen information based on the data transmitted and received by the communication unit 14. Have For example, the communication measuring unit 121 can measure the communication processing capability by measuring the time from when the measurement data is transmitted to the server device 20 until the transmission completion ACK is received (ACK: Acknowledgment). ). The round-trip delay time is also called RTT and corresponds to the time until a signal transmitted from the client terminal 10 to the server device 20 returns to the client terminal 10 (RTT: Round Trip Time).

By using the data size for measurement and the transmission time that is the time required for transmission, the throughput of the network 30 can be calculated by Equation 1.
(Throughput) = (Data size for measurement) / (Transmission time) (1)
Further, instead of the measurement data, the throughput of the network 30 may be calculated using the operation prediction information itself as in Expression 2.
(Throughput) = (data size of operation prediction information) ÷ (transmission time) (2)
Further, by dividing the average size of the execution screen information by the throughput, the transmission delay when transmitting / receiving the average size of the execution screen information can be calculated (Equation 3). The transmission delay corresponds to the time required for transmitting the execution screen information from the server device 20 to the client terminal 10.
(Transmission delay) = (Average size of execution screen information) / (Throughput) (3)
Since the transmission delay differs for each application and processing content, it may be obtained for each application and processing content. Further, the transmission delay may be obtained using Expression 4 using the data size of the execution screen information that is predicted to be generated as a result of the process executed based on the operation prediction information.
(Propagation delay) = (Predicted execution screen information size) ÷ (Throughput) (4)
The communication measurement unit 121 transmits the measured communication processing capability to the input prediction unit 123.

  When the input predicting unit 123 determines that the pointing object is in contact with the input unit 11 based on the actual measurement information received from the input unit 11, the input predicting unit 123 predicts the contact position, the contact time, and the operation type that occurs when the contact is made. It has a function to do.

  The operation prediction information transmission unit 125 calculates a data transmission / reception time, operation prediction information transmission time, and the like based on the communication processing capability measured by the communication measurement unit 121.

  Further, the operation prediction information transmission unit 125 controls the timing for transmitting the operation prediction information so that the execution screen information can be received immediately after the moving time until the pointing object contacts the input surface of the input unit 11 has elapsed. It has a function. In order to display the execution screen at the predicted contact time in the client terminal 10, the operation predicted information transmission time is calculated based on the data transmission / reception time including the operation predicted information delivery time, the application processing time, the transmission delay, and the like as parameters. Good. This system is characterized in that the application is executed in advance in the server 20 in consideration of the communication processing capability, and the execution screen is displayed in the client terminal 10 immediately after the predicted operation time. Other delay factors may be considered as necessary.

<Display means>
The display unit 13 manages the execution screen information acquired from the server device 20 and displays the execution screen included in the execution screen information in accordance with the contact or prediction of the pointing object on the input surface of the client terminal 10. It is.

  As shown in FIG. 2, the display unit 13 includes an execution screen information management unit 131 and an output unit 133.

  The execution screen information management unit 131 has a function of receiving execution screen information from the server device 20 and transmitting the received execution screen information to the output unit 133. Note that the execution screen information management unit 131 may transmit only information related to the execution screen included in the execution screen information to the output unit 133.

  The output unit 133 has a function of displaying an execution screen of an application executed by the server device 20 on a display device such as a display or a touch panel provided in the client terminal 10.

<Communication means>
The communication unit 14 is an interface connected to the network 30 and has a function of transmitting the operation prediction information transmitted by the operation prediction information transmission unit 125 to the server device 20 and a function of receiving execution screen information from the server device 20. . The communication means 14 is realized by a general data communication device.

(Server device)
Next, a functional configuration of the server device 20 according to the present embodiment will be described.

  As illustrated in FIG. 2, the server device 20 includes an execution screen information generation unit 21, a virtual environment 22, and a communication unit 23.

  The execution screen information generation unit 21 captures the screen output of the virtual environment 22 at predetermined time intervals, generates execution screen information including information (screen information) about the execution screen and generation time, and the generated execution screen information Is transmitted to the communication means 23. In addition, when the difference with the execution screen information transmitted last time is a part of the screen, only the difference may be transmitted as the execution screen information. The execution screen information generation unit 21 has a function of executing a corresponding application on the virtual environment 22 in accordance with operation prediction information and actual measurement information received from the client terminal 10. The execution screen information generation unit 21 activates the virtual environment 22 for each client terminal 10.

  The virtual environment 22 is a virtualized environment for executing an application. The virtual environment 22 is activated inside the server device 20 for each client terminal 10.

  The communication unit 23 is an interface connected to the network 30, and transmits to the client terminal 10 a function of receiving operation prediction information and actual measurement information from the client terminal 10 and execution screen information generated by the execution screen information generation unit 21. It has a function. The communication means 21 is realized by a general data communication device.

  The above is the description of the configuration of the thin client system 2 according to the second embodiment.

(Operation)
Next, the operation of the thin client system 2 according to the second embodiment of the present invention will be described.

  FIG. 3 shows an outline of a processing flow of operations related to the thin client system 2 according to the second embodiment of the present invention. The process flow of FIG. 3 includes steps such as a connection process (step S10), an actual measurement process (step S20), an operation prediction process (step S30), an execution screen information generation process (step S40), and an execution screen display process (step S50). Including. In addition, each process of the process flow shown in FIG. 3 is divided for convenience, and individual processes included in each process may be included in another process.

  In the connection process (step S10), the client terminal 10 and the server apparatus 20 are connected, the execution screen information of the application executed on the server apparatus 20 is transmitted to the client terminal 10, and the execution screen is displayed on the client terminal 10. It is.

  The actual measurement process (step S20) is a process for actually measuring the communication processing capability between the client terminal 10 and the server device 20 and the distance between the pointing object and the input surface.

  The operation prediction process (step S30) is a process for predicting the user's operation from the movement of the pointing object approaching the client terminal 10. In the operation prediction process, a time relating to data exchange between the client terminal 10 and the server device 20 and an operation time of the application are predicted, and a process of controlling the exchange of each data based on the predicted time is also performed.

  The execution screen information generation process (step S40) is a process in which the server apparatus 20 executes the application in advance in the virtual environment 22 and generates execution screen information in advance based on the operation prediction information.

  The execution screen display process (step S50) is a process for causing the output unit 133 to display an appropriate execution screen when the actual pointing object contact with the input unit 11 is detected.

  The above is the outline of the processing flow of the operation of the thin client system 2 according to the second embodiment of the present invention.

  Next, details of each processing step shown in FIG. 3 will be described with reference to FIGS. 4, 5, and 7 to 9.

<Connection process>
First, the connection process in step S10 of FIG. 3 will be described using the sequence diagram of FIG.

  In FIG. 4, first, the client terminal 10 is connected to the server device 20 (step S11).

  The server device 20 activates a dedicated virtual environment 22 for each client terminal 10 (step S12). On the virtual environment 22, a virtual terminal of each client terminal 10 is activated, and an application is executed on the activated virtual terminal.

  The execution screen information generation means 21 of the server device 20 executes the application on the virtual environment 22 and sequentially generates execution screen information including the identifier of the client terminal 10, the execution screen of the virtual terminal, and the generation time (step). S13).

  The communication unit 23 of the server device 20 transmits the generated execution screen information to the client terminal 10 to which the identifier included in the execution screen information is attached (Step S14).

  When the communication unit 14 of the client terminal 10 receives the execution screen information, the communication unit 14 transmits the received execution screen information to the execution screen information management unit 131. The execution screen information management unit 131 transmits the execution screen included in the execution screen information to the output unit 133 in accordance with the timing for displaying the execution screen (after the predicted operation time). Note that the timing for displaying the execution screen may be after the predicted operation time, and is preferably a timing at which the user does not feel uncomfortable.

  The output unit 133 displays the received execution screen (step S15).

  The above is the description regarding the connection processing.

<Measurement process>
Next, the actual measurement process in step S20 of FIG. 3 will be described using the flowchart of FIG.

  In FIG. 5, first, the communication measuring unit 121 observes data transmitted and received between the client terminal 10 and the server device 20, and measures round-trip delay time and communication throughput between the client terminal 10 and the server device 20. Is executed (step S21).

  Here, when the pointing object approaches a certain distance from the input surface, the input unit 11 starts detecting the pointing object (step S22).

  When the input unit 11 detects the pointing object (Yes in step S22), the input unit 11 measures actual measurement information including the location of the pointing object, the detection time, and the detection state (step S23).

  On the other hand, when the input means 11 has not detected the pointing object (No in step S22), the process returns to step S22. If the input unit 11 has not detected the pointing object (No in step S22), the process may return to step S21.

  Here, the location of the pointing object, the detection time, and the detection state will be described with reference to FIG.

  First, on the input surface, a plane including the X axis and the Y axis orthogonal to each other is defined as an XY plane, and an axis orthogonal to the XY plane is defined as a Z axis. Then, the location of the pointing object is specified using the X coordinate, the Y coordinate, and the Z coordinate of the three-dimensional orthogonal coordinate system based on the XY plane. If a plurality of detection points are detected, the detection point closest to the input surface may be specified as the location of the pointing object. In order to detect different pointing objects at the same time, physical characteristics such as electrical characteristics, magnetic characteristics, and optical characteristics for each pointing object, and a tendency regarding movement for each pointing object may be used.

  In the example of FIG. 6, the pointing object is located at the location (X1, Y1, Z1) at time T1, and is located at the location (X2, Y2, Z2) at time T2. If the distance between the pointing object and the input surface at the time T2 is closer than the time T1, it can be understood that the pointing object is approaching the input surface. On the other hand, if the distance between the pointing object and the input surface at time T2 is longer than time T1, it can be seen that the pointing object moves away from the input surface. In addition, when the pointing object is detected at time T1 but is not detected at time T2, it is determined that the pointing object has left the detection range.

  The detection time indicates the time when the pointing object is detected. In the example of FIG. 6, T1 and T2 correspond to detection times. The detection time may be the time when the actual measurement information is detected.

  There are four types of detection states: “entering an approach detection region”, “moving in an approach detection region”, “leaving an approach detection region”, and “in contact”. “Entering the approach detection area” is an identifier indicating that the pointing object has entered the detection range of the input unit 11, that is, that the pointing object has approached. “Movement within the approach detection area” is an identifier indicating that the pointing object is moving while maintaining a state of approaching the input unit 11. The “leaving approach detection area” is an identifier indicating that the pointing object is out of the detection range of the input unit 11 or that the pointing object has touched. “In contact” is an identifier indicating that the pointing object is in contact with the input unit 11. The detection state can be determined based on the moving speed of the pointing object calculated based on the location of the pointing object and the detection time, the approach angle with respect to the incident surface described later, and the like.

  Next, when actual measurement information is acquired in step S23 of FIG. 5, the input prediction unit 123 verifies whether or not the predicted contact time included in the operation prediction information is calculated (step S24).

  When the predicted contact time has been calculated (Yes in step S24), the input prediction unit 123 compares the predicted contact time generated immediately before with the detection time of the actual measurement information (step S25). On the other hand, if the predicted contact time has not been calculated (No in step S24), the flow in FIG. 5 ends, and the process proceeds to the next operation prediction process.

  At this time, if the detection time is after the predicted contact time (Yes in step S25), the actual measurement information is transmitted to the server device 20 (step S26). And after step S26, it transfers to the operation prediction process. On the other hand, if the detection time is earlier than the predicted contact time (No in step S25), the flow in FIG. 5 is terminated, and the process proceeds to the next operation prediction process.

  The above is the description of the actual measurement process.

<Operation prediction process>
Next, the operation prediction process in step S30 in FIG. 3 will be described using the flowchart in FIG. The operation prediction process is a process that starts after receiving the actual measurement process. Although the actual measurement process and the operation prediction process are described separately here, the actual measurement process and the operation prediction process may be executed as one step.

  In FIG. 7, the input predicting means 123 receives the result of the actual measurement process, and the position of the pointing object in contact with the input means 11 (predicted contact position), the time of contact (predicted contact time), and the operation type (predicted operation type). Whether or not prediction is possible is determined (step S31). The predicted contact position, predicted contact time, and predicted operation type are collectively referred to as operation prediction information.

  When the operation prediction information can be discriminated (Yes in step S31), the input prediction unit 123 is based on the difference between the movement distance of the location and the detection time included in the actual measurement information indicating the plurality of “movements in the approach detection area”. Next, operation prediction information is calculated (step S32). The input prediction unit 123 then passes operation prediction information including the calculated predicted contact time, predicted contact position, and predicted operation type to the operation prediction information transmission unit 125.

  On the other hand, when the operation prediction information is not discriminable (No in step S31), the process returns to step S23 in FIG.

  In the example of FIG. 6, the moving speed and moving direction of the pointing object are obtained from the change in the position coordinates of the pointing object at time T1 and time T2, and the coordinates of the pointing object at time Tp when the coordinate in the Z-axis direction is 0 are predicted. To do. Actually, an extended line of a locus drawn when connecting a plurality of location positions in time series order calculates an intersection with the XY plane of the touch panel, and the position of the intersection is determined as a predicted operation position. For example, a trajectory drawn when a plurality of locations are connected in time series order may be fitted to an expression representing a parabola, hyperbola, ellipse, circle, exponential function curve, logarithmic curve, or the like. Note that there may be a case where three or more pieces of location information are obtained and the angle formed by a vector connecting the plurality of location information with the XY plane is not constant. In that case, the predicted operation position may be determined based on the angle formed by the tangent line of the curve and the XY plane at the intersection of the curve and the XY plane fitted between a plurality of location position information. Further, when there are a plurality of pieces of location position information, the predicted contact position may be calculated for the intersection of the vector connecting the latest two points and the XY plane.

  The operation type can be determined based on the angle formed by the vector connecting the location information and the input plane (XY plane).

  In FIG. 6, an example will be described in which a tap operation and a flick operation are discriminated based on an angle θ formed by a half line drawn from a location at time T1 toward a location at time T2. In the case of the flick operation, the pointing object moves in the horizontal direction with respect to the XY plane, and thus the angle θ becomes small. Therefore, when the angle θ is small, it is determined as a flick operation. On the other hand, in the case of a tap operation, the pointing object moves in the direction toward the XY plane, so that the angle θ increases. Therefore, when the angle θ is sufficiently large, it is determined as a tap operation. Note that the angle that is a criterion for distinguishing between the tap operation and the flick operation may be set in advance or may be learned from the user's operation history.

  Further, the moving speed of the pointing object may be measured based on the moving distance and moving time between the location information, and the operation type may be determined using the measured moving speed. For example, when the moving speed of the pointing object is fast, priority is given to the determination of the operation type based on the angle described above, and when the moving speed of the pointing object is slow, it is determined that the user is losing the tap position and the tap operation is determined. Can do. For example, when the moving speed of the pointing object is slow, a closed curve or a curved surface is calculated by connecting the detection positions of the pointing object within a predetermined period, and the area on the input surface immediately below the calculated closed curve or curved surface is What is necessary is just to predict that it will be tapped. However, the determination of the magnitude of the moving speed of the pointing object can be arbitrarily set.

  When there are two or more predicted operation types, the operation type with the highest possibility may be selected using a user operation history or a known prediction method. Further, the operation type may include not only a tap operation and a flick operation but also a drag and swipe, a double tap, a double slide, a pinch in, a pinch out, and the like.

  For example, when a position that can be double-tapped is tapped, it can be predicted in advance that the double-tap will be tapped. At this time, if a predicted contact position having the same operation content as the contact position is predicted after the tap is detected, it can be determined that the possibility of a double tap is high. For example, when two different pointing objects are detected, operations such as double slide, pinch-in, and pinch-out are predicted. At this time, the distinction between pinch-in and pinch-out can be estimated by the distance between the two pointing objects. That is, when the distance between the pointing objects is small, it can be determined as pinch out, and when the distance between the pointing objects is large, it can be determined as pinch in.

  Further, the operation type can be predicted by the execution screen displayed on the screen in the operation stage. For example, when an execution screen for inputting character information is displayed, it can be determined that a flick operation is more likely than a tap operation. For example, when map information is displayed, it can be determined that there is a high possibility that dragging, pinching in, pinching out, and the like will be performed. Note that these operation types are merely examples, and the scope of the present invention is not limited to these examples.

Returning to FIG. 7, the operation prediction information transmission unit 125 calculates the data transmission / reception time (step S <b> 33).
The data transmission / reception time is a time required for the operation prediction information to be delivered from the client terminal 10 to the server device 20 and the execution screen information to be delivered from the server device 20 to the client terminal 10. For example, the data transmission / reception time is obtained by Equation 5.
(Data transmission / reception time) = (round-trip delay time) / 2 + (transmission delay) (5)
In Expression 5, the time for which the operation prediction information is delivered from the client terminal 10 to the server device 20 is reflected in “(round trip delay time) / 2”. The time for which the execution screen information is delivered from the server device 20 to the client terminal 10 is reflected in the transmission delay. However, since the application processing time in the server device 20 is not included in the expression 5, it is preferable to actually calculate the data transmission / reception time by the expression 6 in consideration of the application processing time.

Further, the data transmission / reception time may be calculated as in Expression 6.
(Data transmission / reception time) = (round-trip delay time) / 2 + (application processing time) + (transmission delay) (6)
Note that the data transmission / reception time is not limited to the calculation method of Equation 5 and Equation 6, and if the execution screen can be displayed on the client terminal 10 immediately after the predicted operation time, other delay factors may be added, or Equation 5 The parameters in Equation 6 may be adjusted or deleted.

Next, the operation prediction information transmission means 125 calculates the operation prediction information transmission time which becomes a timing which transmits operation prediction information to the server apparatus 20 using Formula 7 (step S34). The predicted contact time is calculated as the time when the coordinate in the Z-axis direction becomes 0 based on the location of the pointing object and the moving speed and moving direction of the pointing object.
(Operation prediction information transmission time) = (Contact prediction time) − (Data transmission / reception time) (7)
Then, the operation prediction information transmission unit 125 controls the timing for transmitting the operation prediction information to the server device 20 so that the execution screen can be displayed on the client terminal 10 immediately after the contact prediction time is reached (step S35). In other words, the operation prediction information transmission unit 125 performs control so that the operation prediction information is transmitted to the server device 20 at the predicted contact time.

  Thus, by calculating the time required for transmission / reception of the operation prediction information and the execution screen information, it is possible to determine the operation prediction information transmission time that is the time for transmitting the operation prediction information. When the operation prediction information transmission unit 125 receives another operation prediction information from the input unit 11 while waiting for transmission of the operation prediction information, the operation prediction information transmission unit 125 cancels the transmission of the previous operation prediction information, and new operation prediction information. Is transmitted at the operation prediction information transmission time.

  When the operation prediction information transmission time comes, the operation prediction information transmission unit 125 generates and generates one or more operation prediction information including the identifier of the client terminal 10, the predicted contact position (X coordinate and Y coordinate), and the predicted operation type. The operation prediction information thus transmitted is transmitted to the server device 20 (step S36). The identifier of the client terminal 10 is specified by, for example, the IP address of the client terminal 10 (IP: Internet Protocol). The operation prediction information may be generated before the operation prediction information transmission time.

  The above is the description of the operation prediction process.

<Execution screen information generation processing>
Next, the execution screen information generation process in step S40 of FIG. 3 will be described using the sequence diagram of FIG.

  In FIG. 8, first, operation prediction information or actual measurement information is transmitted from the client terminal 10 to the server device 20 (step S41).

  When the server device 20 receives the operation prediction information or the actual measurement information by the communication unit 23, the execution screen information generation unit 21 refers to the received operation prediction information or the actual measurement information, reproduces the execution screen information on the virtual environment 22, The target application is executed (step S42).

  Then, the execution screen information generation unit 21 generates execution screen information obtained as a result of executing the target application (step S43).

  The execution screen information generated by the execution screen information generation unit 21 is transmitted to the client terminal 10 by the communication unit 23 (step S44).

  When the communication unit 14 of the client terminal 10 receives the execution screen information, the communication unit 14 transmits the received execution screen information to the execution screen information management unit 131, and the execution screen information management unit 131 outputs the execution screen to the output unit 133. Prepare to do (step S45).

  The above is the description of the execution screen information generation process.

<Execution screen display processing>
Next, the execution screen display process of step S50 of FIG. 3 will be described using the flowchart of FIG.

  In FIG. 9, first, it is determined whether or not the pointing object has touched the operation panel (step S51).

  When the pointing object touches the operation panel (Yes in step S51), the input predicting unit 123 verifies whether or not the contact position of the pointing object matches the predicted contact position (step S52). On the other hand, if the pointing object is not in contact with the operation panel (Yes in step S51), step S51 is repeated.

  When the contact position of the pointing object matches the predicted contact position (Yes in step S52), the operation prediction unit 12 informs the execution screen information management unit 131 that the contact position of the pointing object matches the predicted contact position. Notice. The execution screen information management unit 131 transmits the prepared screen information to the output unit 133, and the output unit 133 displays the execution screen included in the received screen information (step S53).

  On the other hand, when the contact position of the pointing object does not match the predicted contact position (No in step S52), the client terminal 10 transmits the actually measured contact position to the server device 20 and acquires the corresponding execution screen information (step S54). ). Although omitted in FIG. 9, in the process of step S54, the server apparatus 20 executes the processes of step S42 to step S44 in FIG.

  And the output means 133 of the client terminal 10 acquires screen information from the execution screen information management means 131, and displays an execution screen (step S55).

  The above is the description of the execution screen display process.

  As described above, in the thin client system according to the present embodiment, the operation content to the input unit of the client terminal is predicted, and the operation prediction information is generated in accordance with the time when the pointing object is predicted to contact the input unit. Send to server device. The server device executes the corresponding application in advance in the virtual environment, and transmits execution screen information as an execution result to the client terminal. As a result, the client terminal can display the execution screen as soon as the pointing object performing the input operation touches the input surface. Therefore, according to the thin client system according to the present embodiment, the response time for screen display can be shortened.

  According to this embodiment, based on the predicted contact time when the pointing object contacts the input surface and the communication time between the client terminal and the server device, the timing for transmitting the operation prediction information, the timing for generating the execution screen information, The timing for transmitting the execution screen information can be controlled. As a result, it is possible to suppress the execution screen from being updated before the user performs an input operation, and to eliminate the uncomfortable feeling of the screen update for the operation.

(Third embodiment)
Next, a thin client system 3 according to the third embodiment of the present invention will be described.

(Constitution)
FIG. 10 is a block diagram illustrating a functional configuration of the client system 3 according to the third embodiment. Unlike the thin client system 2 according to the second embodiment, the thin client system 3 according to the third embodiment performs actual measurement processing and operation prediction processing on the server device 200 side. In FIG. 10, the same components as those of the thin client system 2 according to the second embodiment are denoted by the same reference numerals.

  A client system 3 according to the third embodiment has a configuration in which a client terminal 100 and a server device 200 are connected via a network 300.

  The client terminal 100 includes an input unit 11, an operation prediction unit 12, a display unit 13 including an execution screen information management unit 131 and an output unit 133, and a communication unit 14. The difference from the second embodiment is that the operation prediction unit 12 does not perform the actual measurement process and the operation prediction process. The operation prediction unit 12 relays the operation information input to the input unit 11 to the communication unit 14 by converting it into an input signal as necessary. Note that the operation prediction unit 12 may have a measurement processing function and an operation prediction processing function similar to those of the operation prediction unit 12 of the second embodiment, and may be configured to perform control so that the function is not normally performed. .

  The server device 200 includes an execution screen information generation unit 210, a virtual environment 22, and a communication unit 23. The execution screen information generation unit 210 includes a communication measurement unit 211, an operation prediction information measurement unit 213, and an execution screen information processing unit 215. The difference from the thin client system 2 according to the second embodiment is that the execution screen information generation unit 210 includes a communication measurement unit 211 and an operation prediction information measurement unit 213. The execution screen information processing unit 215 generates execution screen information based on the measurement results of the communication measurement unit 211 and the operation prediction information measurement unit 213.

(Operation)
The thin client system 3 according to the third embodiment performs processing according to FIG. 3 as in the second embodiment.

  First, the connection process between the client terminal 100 and the server apparatus 200 is performed in the same manner as in the second embodiment shown in FIG.

  And unlike 2nd Embodiment, the measurement process (FIG. 5) and the operation prediction process (FIG. 7) are performed by the server apparatus 200 side. 5 and FIG. 7 are executed in the same way except for the operation subject, and detailed description thereof is omitted.

  In the third embodiment, it is assumed that the actual measurement information regarding the pointing object acquired by the input unit 11 of the client terminal 100 is sequentially transmitted to the server device 20. However, the operation prediction information may be calculated on the client terminal 100 side, and the server device 200 may be configured to use the operation prediction information calculated in the client terminal 200.

  Subsequently, as illustrated in FIG. 8, the execution screen information processing unit 215 executes the application in the virtual environment 22 based on the operation prediction information, and generates execution screen information. The difference from the second embodiment is that step S41 of FIG. 8 is not required in order to generate operation prediction information on the server device 200 side.

  Furthermore, the execution screen display process triggered by the pointing object touching is the same as in the second embodiment shown in FIG. Note that the execution screen information generated in step S43 of FIG. 8 may be stored on the server device 200 side without being transmitted to the client terminal 100. In that case, it is only necessary that the operation subject in steps S51 and S52 in FIG. 9 is the server apparatus 200, and the contact determination between the pointing object and the input surface is performed on the server apparatus 200 side. At this time, with regard to step S53 of FIG. 9, execution screen information corresponding to the contact position may be transmitted from the server apparatus 200 to the client terminal 100 in accordance with the contact position of the pointing object. That is, the server device 200 may adjust the timing for generating execution screen information and the timing for transmitting execution screen information.

  The above is the description of the operation of the thin client system 3 according to the third embodiment.

  Note that the client terminal 100 may have the configuration of the client system 2 according to the second embodiment and be configured to execute the operation prediction process in the same manner as the server device 200. In such a case, the operation prediction process is normally performed on the server device 200 side, and the operation prediction process is performed on the client terminal 100 side in a situation where a communication delay occurs. Therefore, redundancy can be improved.

  As described above, according to the thin client system according to the third embodiment, the actual measurement process and the operation prediction process can be executed on the server device side. Therefore, it is possible to reduce the processing on the client terminal side as compared with the second embodiment. Further, by configuring so that the operation prediction process can be executed by the client terminal and the server device, the redundancy of the system can be improved.

(Hardware configuration)
Here, a hardware configuration for realizing the thin client system according to the embodiment of the present invention will be described. The following hardware configuration is an example for realizing the thin client system according to the present embodiment, and does not limit the scope of the present invention.

  FIG. 11 is an example of a hardware configuration of the client terminal. The client terminal includes a CPU 101, a ROM 102, a RAM 103, a buffer 104, an auxiliary storage device 105, and a communication interface 106 (ROM: Read Only Memory, RAM: Random Access Memory). Further, the client terminal includes an input / output device 110 having an input unit 111, an input control unit 112, a display unit 113, and a display control unit 114. Each element is connected to each other via a bus 107.

  FIG. 12 is an example of a hardware configuration of the server apparatus. The server device includes a CPU 201, ROM 202, RAM 203, buffer 204, auxiliary storage device 205, communication interface 206, bus 207, input unit 211, and display unit 212. Each element is connected to each other via a bus 207.

  Below, each component is demonstrated easily. For hardware having functions common to the client terminal and the server device, the server device code is shown in parentheses after the client terminal code.

  The CPU 101 (201) is a central processing unit, and is a device for executing arithmetic processing / control processing. The CPU 101 (201) uses the RAM 103 (203) as a working memory according to a control program stored in the ROM 102 (202), and controls the operation of each unit. In addition, the CPU 101 of the client terminal executes control processing of an input signal input from the input unit 111 and generated by the input control unit 112, a response signal from the server received by the communication interface 106, and a signal related to execution screen information. . The ROM 102 (202) is a non-volatile memory that stores a program for controlling the system. The RAM 103 (203) is a volatile memory that becomes a working memory for developing application programs. The buffer 104 (204) can be realized by a volatile memory that functions as a frame buffer for storing execution screen information. When the RAM 103 (203) also functions as the buffer 104 (204), the buffer 104 (204) may be omitted. The auxiliary storage device 105 (205) is a device for storing data generated as a result of executing a program on the server device. The communication interface 106 (206) is a device for establishing a network connection between a client terminal and a server device, and has standards and specifications in which a connection procedure and the like are defined.

  The input unit 111 of the client terminal is a device for inputting operation information to the client terminal 10. The input unit 111 has a touch panel function and a function of detecting a pointing object that comes close to or touches. The input unit 111 detects a change in a physical quantity such as a current value in accordance with the position of the pointing object that is approaching or touching. For example, the input unit 111 detects the pointing object that comes close to or touches by a method such as a capacitance method or an electromagnetic induction method. Further, the input unit 111 may be configured to detect a pointing object that comes close to or touches by a method such as an ultrasonic method, an electromagnetic wave method, or an infrared method. The input unit 111 also serves as a device for inputting operation information.

  The input control unit 112 is a device that converts and controls a signal detected by the input unit 111 into an input signal such as position information.

  For example, in an electromagnetic induction type touch panel, a sensor coil group intersecting perpendicularly to the input surface of the touch panel is provided inside, and a magnetic field is generated by applying an alternating current to the sensor coil in a pulse shape. When a pointing object such as an electromagnetic induction pen having a coil approaches the operation surface of the touch panel, an induced current flows through the coil inside the pointing object, and charges are stored in a capacitor inside the pointing object. Here, when the current application to the sensor coil is stopped, the electric charge stored in the capacitor of the pointing object is released, and a current flows through the coil on the pointing object side to generate a magnetic field. As a result, an electrostatic current corresponding to the positional relationship with the pointing object flows through the sensor coil inside the touch panel, and a signal indicating the position of the pointing object is detected by the input unit 111. Then, the input control unit 112 converts the signal indicating the position of the pointing object into an input signal indicating the position information.

  The display unit 113 of the client terminal is a device for displaying an execution screen included in the execution screen information transmitted from the server device to the client terminal. When the client terminal has a touch panel, the display unit 113 may be located at the same site as the input unit 111 in appearance. Further, the display unit 113 may be a display that displays an execution screen separately from the touch panel having an input function.

  The display control unit 114 is a device that controls the display of the execution screen acquired from the server device. The execution screen information acquired from the server device is temporarily stored in the buffer 104 of the client terminal. The display control unit 114 performs control so that the execution screen included in the execution screen information stored in the buffer 104 is displayed at a timing at which execution screen information is generated according to the state of contact of the pointing object with the input unit 111. .

  The input unit 211 of the server device is a device for inputting operation information to the server device. For example, the input unit 211 is a device for inputting operation information input by an input device such as a keyboard and a mouse to the server device. The display unit 213 of the server device is a device for displaying operation information and the like on the server device.

  This completes the description of the hardware configuration for realizing the thin client system according to the embodiment of the present invention.

  Each process in the client server system according to the embodiment of the present invention can be realized by software including a control program for executing the above-described process. A program recording medium storing a control program according to the embodiment of the present invention is also included in the scope of the present invention. Examples of the program recording medium include a ROM (Read Only Memory), a RAM (Random Access Memory), a semiconductor storage device such as a flash memory, an optical disk, a magnetic disk, and a magneto-optical disk.

  Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  The present invention is applied to a thin client system such as a screen transfer method. Another application example is application to a Web application. Many interfaces of cloud-type communication services are constructed by Web applications. If the present invention is applied to a Web application, the user can comfortably use the communication service by reducing the response time from when the user operates the information terminal until the content of the Web server is displayed. .

DESCRIPTION OF SYMBOLS 1 Client server system 2, 3 Thin client system 10, 100 Client terminal 11 Input means 12 Operation prediction means 13 Display means 14 Communication means 20, 200 Server apparatus 21 Execution screen information generation means 22 Virtual environment 23 Communication means 30, 300 Network 101 201 CPU
102, 202 ROM
103, 203 RAM
104, 204 Buffer 105, 205 Auxiliary storage device 106, 206 Communication interface 107, 207 Bus 110 Input / output device 111, 211 Input unit 112 Input control unit 113, 213 Display unit 114 Display control unit 121 Communication measurement unit 123 Input prediction unit 125 Operation prediction information transmission means 131 Execution screen information management means 133 Output means 211 Communication measurement means 213 Operation prediction information measurement means 215 Execution screen information processing means

Claims (10)

In a thin client system that executes processing corresponding to an operation on a client terminal on a server device,
Input means for acquiring the position information of the pointing object relative to the input surface in association with a detection time by detecting the approach and contact of the pointing object with respect to the input surface provided in the client terminal;
The communication processing capability between the client terminal and the server device is measured, and the pointing object is predicted to come into contact with the input surface based on a change in position information of the pointing object acquired by the input unit. A predicted contact position that is a position where the pointing object is predicted to contact the input surface, an identifier of the client terminal, the predicted contact position, and the predicted contact time. Operation prediction means for transmitting operation prediction information including the communication processing capability based on the communication processing capability,
The operation prediction information is received, and the process executed in response to the predicted contact position on the input surface of the client terminal to which the identifier is attached is executed in the virtual environment on the server device, thereby Execution screen information generating means for generating execution screen information including screen information that is an execution result of the process corresponding to the contact position;
The execution screen information generated by the execution screen information generation means is received, and an execution screen corresponding to the screen information included in the received execution screen information is displayed on the display provided in the client terminal after the predicted contact time. A client server system comprising display means for controlling to be displayed. The operation prediction means includes
Communication measuring means for measuring the communication processing capability between the client terminal and the server device;
Input prediction means for calculating the operation prediction information including the predicted contact position and the predicted contact time;
Data transmission / reception time required for transmitting the operation prediction information from the client terminal to the server device, and transmitting the execution screen information from the server device to the client terminal, and the operation prediction from the client device to the server device An operation prediction information transmission time that is a time to transmit information is calculated, and the operation screen information is received from the client terminal to the server device so that the execution screen information is received by the client terminal before the predicted contact time. The client server system according to claim 1, further comprising operation prediction information transmission means for controlling a timing at which the prediction information is transmitted. The operation prediction means includes
The client server system according to claim 1, wherein a prediction operation type that is an operation type predicted from a change in a location of the pointing object is included in the operation prediction information for prediction. The operation prediction means includes
The operation type is determined using an angle formed by a vector connecting the location of the pointing object at different times acquired by the input unit and the input surface, and a moving speed of the pointing object. Client server system. The communication measuring means includes
Based on the data transmitted and received between the client terminal and the server device, the throughput when transmitting and receiving data between the client terminal and the server device, and between the client terminal and the server device 5. The client according to claim 2, wherein an average size of the execution screen information to be transmitted / received and a round trip delay time when data is transmitted from the client terminal to the server device are measured as the communication processing capability. Server system. The operation prediction information transmitting means includes
The timing at which the operation prediction information is transmitted from the client terminal to the server device is controlled so that the execution screen is displayed on the display after the predicted contact time has elapsed. The client server system described in 1. When the contact position of the pointing object with respect to the input surface is different from the predicted contact position,
The client terminal is
Requesting the server device to execute processing according to the contact position;
The server device
The process according to the actual contact process is executed in the virtual area in response to a request from the client terminal to generate the execution screen information, and the generated execution screen information is transmitted to the client terminal. The client server system according to any one of 1 to 6. The server device
By including the operation prediction means, the communication processing capability is measured, the operation prediction information and the predicted operation time are predicted, the data transmission / reception time and the operation information transmission time are calculated, and the predicted contact position The execution screen information is generated for the client terminal so that the execution screen information is generated by executing the process corresponding to, and the execution screen information is received by the client terminal before the predicted contact time elapses. The client server system according to claim 1, wherein the transmission timing is controlled. In a thin client system that executes processing corresponding to an operation on a client terminal on a server device,
By detecting the approach and contact of the pointing object with respect to the input surface provided in the client terminal, the positional information of the pointing object with respect to the input surface is acquired in association with the detection time;
The communication processing capability between the client terminal and the server device is measured, and based on the obtained change in the positional information of the pointing object, the pointing object is predicted to come into contact with the input surface. An operation prediction including a predicted contact position and a predicted contact time that is a time when the pointing object is predicted to contact the input surface, and includes an identifier of the client terminal, the predicted contact position, and the predicted contact time. Sending information based on the communication processing capability;
The operation prediction information is received, and the process executed in response to the predicted contact position on the input surface of the client terminal to which the identifier is attached is executed in the virtual environment on the server device, thereby Generate execution screen information including screen information that is the execution result of the process corresponding to the contact position,
A control method for performing control so that an execution screen corresponding to the screen information included in the generated execution screen information is displayed after the predicted contact time on a display provided in the client terminal. In a thin client system that executes processing corresponding to an operation on a client terminal on a server device,
A process of acquiring position information of the pointing object relative to the input surface in association with a detection time by detecting the approach and contact of the pointing object with respect to the input surface provided in the client terminal;
The communication processing capability between the client terminal and the server device is measured, and based on the obtained change in the positional information of the pointing object, the pointing object is predicted to come into contact with the input surface. An operation prediction including a predicted contact position and a predicted contact time that is a time when the pointing object is predicted to contact the input surface, and includes an identifier of the client terminal, the predicted contact position, and the predicted contact time. Processing for transmitting information based on the communication processing capability;
The operation prediction information is received, and the process executed in response to the predicted contact position on the input surface of the client terminal to which the identifier is attached is executed in the virtual environment on the server device, thereby Processing for generating execution screen information including screen information that is the execution result of the processing corresponding to the contact position;
A control program for causing a computer to execute a process of controlling an execution screen corresponding to the screen information included in the generated execution screen information to be displayed after the predicted contact time on a display provided in the client terminal.

Images