JP2010119030A - Communication device, communication method, and communication program - Google Patents

Abstract

A communication apparatus for efficiently controlling the performance of a plurality of functions provided to a display terminal is provided.
An input analysis unit that thins out input information in accordance with a thinning rate specification, an update image generation unit that generates an update image for updating a display image in accordance with the thinned out information, an update A difference detection unit 112 that detects a difference region in which pixel information does not match between an image and a display image at a specified time interval; a compressed image generation unit 113 that generates a compressed image obtained by compressing an image in the difference region; A video generation unit 115 that generates information, a transmission unit 117a that transmits a compressed image and video information to a display device, and a decimation rate that is specified to the input analysis unit 114 when the video information generation processing is larger than a threshold value. And a control unit 116 that executes at least one of control to increase and control to increase the time interval specified for the difference detection unit 112.
[Selection] Figure 2

Description

  The present invention relates to a communication device, a communication method, and a communication program for realizing a function of sharing an application screen between devices.

  For the purpose of improving usability, a computing system in which a terminal device having a minimum input / output interface 211 is arranged on the user side, and complicated arithmetic processing is executed on a communication device (main device) installed in a remote place. Exists. For example, Patent Document 1 proposes a technique related to a system (screen transfer system) that projects screen information of a main device (personal computer, server computer, etc.) onto a remote display device (display terminal) via a network. .

  In such a system, input information (such as pen input by a digitizer) from the display terminal is similarly transmitted to the main device via the network, and the actual application program processing is executed by the main device. Thereafter, the execution result and the screen update information are transferred to the display terminal via the network. The display terminal executes output processing (drawing processing) based on the received screen update information.

  On the other hand, VNC (Virtual Network Computing) is known as a technique for efficiently transmitting screen information from a main unit on a remote network to a display terminal. In this VNC, when the update of the screen is detected, the value of the read pixel information is compared with the value of the pixel information transmitted to the previous display terminal, and the updated screen area changed from the previous time is determined. Further, after the still image compression of the updated screen area, only the compressed screen difference information is transmitted to the display terminal. Thereby, the consumption of a communication band can be suppressed. Accordingly, the amount of screen information to be transmitted increases when the screen change is large, such as movement of a window, and conversely, the amount of screen information to be transmitted decreases when the screen change is small.

  In addition, media content such as video data (video information), photo data, and music data stored in an external storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) connected to the main unit is transferred to the network. There is known a multimedia system capable of providing a video streaming function that is transmitted via a remote display terminal and viewed from a remote display terminal.

  For example, UPnP (Universal Plug and Play) negotiates technical specifications for interconnecting AV (Audio Visual) devices such as personal computers and peripheral devices in the home, televisions and HDD recorders through a network. In addition, there is a DLNA (Digital Living Network Alliance) etc. in which a wider range of technical specifications such as a communication method, a transmission control method, and a content type to be handled are negotiated. In this system, a client terminal finds a server device on a network and browses content on the server device by streaming playback.

  Here, in the multimedia system, a case is considered in which the type of video information stored in the external storage device of the main unit is different from the type of video information that can be reproduced on the display terminal. For example, it is assumed that the video information on the main device side is in the WMV format, and the video information that can be reproduced on the display terminal is in the MPEG-2 (Generic coding of moving pictures and associated audio information) format. In this case, the user cannot browse the video information of the main device via the display terminal. Therefore, in such a situation, there can be considered a method in which the main device converts (transcodes) the video information into a format that can be reproduced by the display terminal.

  The transcoding process is a process of performing encoding (encoding) into a new format after performing decoding (decoding) processing of video information. In the above-described example, video information in WMV format is decoded and then encoded into video information in MPEG-2 format conforming to a predetermined frame rate or bit rate condition. As described above, generally, transcoding processing of video information can be said to be processing with a large amount of calculation.

  Then, when the main unit provides the functions of both the screen transfer system and the multimedia system as described above, a process for generating a transmission image to be transmitted to the display terminal (a process for compressing the updated screen area) And transcoding processing of video information may occur simultaneously. In such a situation, there may arise a problem that the amount of calculation to be performed by the main device is very large.

  Patent Document 2 proposes a technique for changing an input frame rate of imaging based on a predetermined priority when a subject is detected in one direction in an imaging device having imaging devices in two directions. . Thereby, recording / transmission of video information appropriate for the situation is realized.

  Patent Document 2 is a technique for inputting video in a plurality of directions, and is not a technique for adjusting the amount of computation of a main device having a plurality of functions as described above. If the method of Patent Document 2 is applied to the main device as described above, a method of lowering the video streaming frame rate when a plurality of functions start operating in the main device can be considered.

US Pat. No. 6,784,855 JP 2003-274358 A

  However, if a method of controlling the quality of other functions to be lowered after applying a specific function as described in Patent Document 2, adjustment of the calculation amount may be inefficient. .

  For example, in the screen transfer system described above, when there is no screen update on the main unit, or when the area of the update area is small, such as when the screen is updated due to the user's mouse operation, the computation associated with transmission image generation Since the amount is small, the performance can be maintained without reducing the video streaming frame rate. However, if the method of Patent Document 2 is simply applied, the frame rate of video streaming is lowered when the screen transfer system is started, and there is a risk that the quality may be reduced unnecessarily.

  The present invention has been made in view of the above, and an object of the present invention is to provide a communication device, a communication method, and a communication program that can efficiently control the performance of a plurality of functions provided to a display terminal. And

  In order to solve the above-described problems and achieve the object, the present invention is a communication device connected to a display device for displaying an image via a network, and stores a display image to be displayed on the display device. A storage unit, an input analysis unit that accepts input information transmitted from the display device and thins out the input information in accordance with a decimation rate that indicates a rate of thinning out the input information, and according to the thinned out input information An update image generation unit for generating an update image for updating the display image, and detecting a difference area representing an area where pixel information does not match between the update image and the display image at a specified time interval A detection unit; a compressed image generation unit that generates a compressed image obtained by compressing the image of the difference area; a video generation unit that generates video information to be displayed on the display device; and the compressed image and the video information in front of each other. A transmission unit that transmits to the display device, and a first processing time that represents a time from the start of the video information generation process to the end of any of the video information generation process and the video information transmission process. When the first processing time is compared with the first threshold and the first processing time is larger than the first threshold, the control for increasing the thinning rate specified for the input analysis unit, and the specification specified for the detection unit And a control unit that executes at least one of the control for increasing the time interval.

  Further, the present invention is a method and program capable of executing the above-described apparatus.

  The present invention is also a communication device connected to a first display device and a second display device for displaying an image via a network, wherein the first video information displayed on the first display device and the second display device A video generation unit that generates second video information to be displayed on, a transmission unit that transmits the first video information to the first display device, and transmits the second video information to the second display device; A processing time from the start of one video information generation process to the end of either the first video information generation process or the first video information transmission process is compared with a predetermined threshold, and the processing time And a control unit that controls the video generation unit so as to lower a frame rate when generating the second video information when the second video information is generated.

  According to the present invention, it is possible to efficiently control the performance of a plurality of functions provided to a display terminal.

  Exemplary embodiments of an apparatus, a method, and a program according to the present invention will be described below in detail with reference to the accompanying drawings.

  The communication system according to the present embodiment includes a main device (communication device) that executes an application and a display terminal (display device) that displays a screen updated by executing the application. The main unit is controlled to reduce the processing load of the other function when the performance of one of the function for transferring image information generated by the operation of the application program and the function for streaming playback of video information decreases. To do.

  FIG. 1 is a block diagram showing a configuration of a communication system according to the present embodiment. The communication system according to the present embodiment is a system that transmits an image of an updated part due to an event occurring on the screen of the main device to a display terminal. Hereinafter, such a communication system is referred to as a screen transfer system.

  As shown in FIG. 1, a communication system 10 according to the present embodiment includes a main device 100 as a communication device, a radio base station 300 as an access point connected to the main device 100 via a network 400, and a radio Display terminals 200a and 200b (hereinafter also referred to as display terminals 200) serving as display devices that wirelessly communicate with the base station 300 via a wireless LAN are provided.

  The communication system 10 wirelessly transfers a screen of application software that operates on the main device 100 to the display terminal 200 via the radio base station 300, and displays an application screen of the main device 100 with the display terminal 200. And share functions. In the communication system 10, in order to transfer the screen updated on the main device 100 side to the display terminal 200 in real time, only the image information of the updated portion in the screen of the main device 100 is transferred. That is, the main device 100 can transmit the image information via the radio base station 300 to the display terminal 200 that displays the image information.

  Further, the main device 100 has a function of acquiring video information on the external storage device 103 (not shown in the figure), and generating and transmitting new video information to be displayed on the display terminal 200. Hereinafter, a system that converts video information and transmits the video information is referred to as a video streaming system. In the example of the figure, the main device 100 wirelessly transfers the generated new video information to one display terminal 200 via the wireless base station 300. For example, the display terminal 200a and the main device 100 realize a screen transfer system, and the display terminal 200b and the main device 100 realize a video streaming system. That is, the main device 100 can simultaneously provide a plurality of different functions to the plurality of display terminals 200.

  However, when a plurality of processes with a large amount of calculation such as generation of a transmission image and conversion of video information are simultaneously performed in the main device 100, an update screen display response on the display terminal 200 due to a lack of available resources. The problem that the Causes of this problem include an increase in transmission image generation time in the main device 100 and an increase in transmission delay due to a large amount of traffic flowing on the wireless transmission path. Furthermore, in the screen transfer system, since the user performs an input operation while confirming the update of the desktop screen of the main device 100, it is important to maintain the responsiveness regarding the display of the screen. On the other hand, in the video streaming system, if the generation of new video information and the transmission interval vary due to a decrease in the computing capability of the main device 100, the playback quality on the display terminal 200 also decreases.

  Therefore, in the present embodiment, when main device 100 that provides a plurality of different functions to a plurality of display terminals 200 detects that the performance of a specific function has decreased, the performance of the other device is decreased. Constitute. This prevents performance degradation of a plurality of functions from occurring in a chain and results in a loss of stability of the entire system.

  The display terminal 200 receives the image information from the main device 100, expands the received image information, and displays it on the corresponding part in the screen. In addition, the display terminal 200 reproduces the media data received from the main device 100 through a display or a speaker.

  The wireless base station 300 is a wireless communication base station that complies with a wireless communication protocol such as IEEE802.11. The network 400 is a network that conforms to a wired communication protocol such as IEEE802.3, for example. The network form is not limited to this, and it may be configured to connect with another protocol. Further, the display terminal 200 and the main device 100 may be connected via a wired network.

  Next, a detailed configuration of the main device 100 will be described with reference to FIG. FIG. 2 is a block diagram of main device 100 according to the present embodiment. As shown in the figure, the main device 100 includes a display 101, an input device 102, an external storage device 103, an image buffer 121, a condition storage unit 122, an event acquisition unit 111, a difference detection unit 112, A compressed image generation unit 113, an input analysis unit 114, a video generation unit 115, a control unit 116, a communication processing unit 117, and a connection determination unit 118 are provided.

  The display 101 is a display device realized by an LCD (Liquid Crystal Display) or the like. The input device 102 is realized by a mouse or the like that moves the cursor displayed on the screen of the display 101. In addition, as the input device 102, a keyboard, a trackball, or the like may be used.

  The external storage device 103 is realized by a large-capacity storage device such as an HDD or an SSD. The external storage device 103 stores multimedia contents such as video data (video information), music data, and photo data.

  The image buffer 121 is a storage unit that stores images. The condition storage unit 122 stores a processing time threshold (first threshold) as a condition used by the control unit 116 for determination.

  The image buffer 121 and the condition storage unit 122 can be configured by any commonly used storage medium such as an HDD, an optical disk, a memory card, and a RAM (Random Access Memory).

  The event acquisition unit 111 acquires an event generated by an operation of an application program (not shown) or the like. For example, the event acquisition unit 111 includes an operating system (OS) that performs overall control of the computer, a virtual display driver having a function equivalent to a display driver incorporated in the OS, and application programs such as application software that operates on the OS. It is realized with.

  The event acquisition unit 111 updates the screen (image) when the screen is updated by application software or when the cursor is moved by a mouse operation or the like to update an image of an arbitrary area in the screen. Get that as an event.

  The event acquisition unit 111 includes an update image generation unit 111a as a detailed configuration. The updated image generation unit 111a can be realized as, for example, a virtual display driver incorporated in the OS. When a screen update event is acquired, the update image generation unit 111a acquires a drawing command from the graphic engine of the OS, and generates a display image representing an image to be displayed on the display terminal 200 by performing a drawing process. The data is sequentially output to the buffer 121 and stored. As a result, display images are sequentially held in the image buffer 121.

  Hereinafter, an image held in the image buffer 121 is referred to as a display image, and an image newly generated by the update image generation unit 111a and stored in the image buffer 121 is referred to as an update image. As described above, the update image generation unit 111a generates an update image to be displayed on the display terminal 200 according to an event generated by the operation of the application program.

  The difference detection unit 112 detects a difference area representing an area where the pixel information does not match between the old and new display images sequentially held in the image buffer 121. That is, the difference detection unit 112, when notified by the update image generation unit 111a that an update image has been generated, generates new image information (update image) and image information buffered in the image buffer 121. A difference area between the (display image) and the display image is detected. For example, the difference detection unit 112 detects a minimum rectangle including a portion that does not match between two pieces of image information as a difference area. In addition, the difference detection part 112 confirms the presence or absence of a difference for every predetermined time interval.

  The compressed image generation unit 113 generates a compressed image obtained by compressing the image of the difference area detected by the difference detection unit 112 for transmission. The compressed image may be generated using lossy compression such as JPEG (Joint Photographic Experts Group), or may be generated using a lossless compression method such as Zlib.

  The compressed image generation unit 113 and the above-described difference detection unit 112 are realized by an application program for screen transfer.

  The input analysis unit 114 analyzes input information such as mouse input and keyboard input sent from the display terminal 200 and sends the input information to the event acquisition unit 111. Note that the input analysis unit 114 thins the input information at the designated thinning rate, and sends the thinned input information to the event acquisition unit 111. Examples of information regarding mouse input include mouse coordinate position (X coordinate, Y coordinate), button state (right click, left click), and the like. On the other hand, the information regarding the keyboard input includes information regarding the operation state such as a newly pressed state or a released state.

  The video generation unit 115 acquires video information on the external storage device 103 via the event acquisition unit 111 and generates new video information to be displayed on the display terminal 200. For example, when the video information stored in the external storage device 103 is in the WMV format, the video generation unit 115 converts the video information into the MPEG-2 format or the like. Therefore, the video generation unit 115 has a function of decoding original video information (decoding function) and a function of encoding new video information (encoding function).

  Furthermore, the video generation unit 115 generates a transmission video message to be transmitted to the display terminal 200 based on the generated new video information. For example, the video generation unit 115 generates a transmission video message by adding transmission control information such as a TCP (Transmission Control Protocol) / IP (Internet Protocol) header to the video information.

  The control unit 116 controls the performance of the screen transfer system and the performance of the video streaming system. For example, the control unit 116 receives the measured value of the processing time from the compressed image generation unit 113 and the video generation unit 115 and compares it with a threshold value stored in advance in the condition storage unit 122. When the actual measurement value is larger than the threshold value, that is, when it is detected that one of the performances has deteriorated, the control unit 116 controls to lower the performance of the other function. Note that to control to reduce (decrease) the performance of the function means to control to reduce the load applied to the main device 100.

  For example, when the processing time per unit area of the compressed image generation unit 113 exceeds a threshold, the control unit 116 instructs the video generation unit 115 to lower the generation frame rate of new video information. For example, when the video generation unit 115 generates video information at an interval of 30 frames per second, the control unit 116 sends a frame rate to the video generation unit 115 so as to generate at an interval of 15 frames per second. The change instruction is performed.

  For example, when the generation time of new video information by the video generation unit 115 exceeds a threshold, the control unit 116 controls the difference detection unit 112 and the input analysis unit 114 to lower the performance. . That is, the control unit 116 instructs the difference detection unit 112 to extend the difference detection interval. As described above, since the detected update images are sequentially overwritten in the image buffer 121, an effect equivalent to frame dropping can be obtained by extending the detection interval. In addition, the control unit 116 instructs the input analysis unit 114 to increase the thinning rate of the input information. By thinning out the input information, for example, in the case of mouse information, some screen updates accompanying the movement of the mouse and window can be skipped, the transmission image generation process can be reduced, and the amount of data transmitted on the communication path can be reduced. Can be reduced.

  Although not explicitly shown in the figure, the main device 100 includes a session information storage unit and a session manager.

  The session information storage unit stores session information representing information related to the display terminal 200 that is establishing a session with the main device 100. For example, the session information storage unit includes user identification information for identifying a user, status information indicating whether or not the session is being used, and transmission control indicating whether transmission control is TCP or UDP (User Datagram Protocol) Session information in which information is associated with information such as format information indicating the format of reproducible video information is stored.

  In a state where a plurality of display terminals 200 have established a session for one main apparatus 100, the session information storage unit identifies a terminal that identifies the display terminal 200 as a destination of a message transmitted from main apparatus 100. Store session information including information.

  The session manager manages communication (session) established with the main device 100. For example, when the session manager establishes a session with the main device 100, the session manager generates session information in which session state information, transmission control information, and the like are associated with each other, and stores them in the session information storage unit.

  The communication processing unit 117 transmits / receives a message to / from an external device such as the display terminal 200. The communication processing unit 117 includes a transmission unit 117a that transmits a message and a reception unit 117b that receives the message. For example, the transmission unit 117 a transmits a transmission image message including the compressed image generated by the compressed image generation unit 113 to the display terminal 200. In addition, the transmission unit 117 a transmits the transmission video message generated by the video generation unit 115 to the display terminal 200.

  The transmission unit 117a transmits a message to be transmitted via the radio base station 300 with the display terminal 200 specified by the session manager as a destination.

  The connection determination unit 118 determines whether or not the display terminal 200 that requested the connection is connectable. Details of the connection determination process for determining whether or not the display terminal 200 can be connected by the connection determination unit 118 will be described later.

  Next, a detailed configuration of the display terminal 200 will be described with reference to FIG. FIG. 3 is a block diagram of the display terminal 200 according to the present embodiment. As shown in the figure, the display terminal 200 includes a display 201, an input device 202, an antenna 203, an image buffer 221, a session information storage unit 222, an input / output interface 211, an image generation unit 212, and a video. A decoding unit 213, a session manager 214, and a wireless communication processing unit 215 are provided. In addition, although not shown in the figure, the display terminal 200 includes a speaker for outputting sound.

  The display 201 is a display device realized by an LCD or the like. The input device 202 is realized by a digitizer or a touch screen that moves a cursor displayed on the screen of the display 201. The input information acquired by the input device 202 is passed to the input / output interface 211 (described later).

  The antenna 203 transmits and receives radio waves for wireless communication with an external device such as the main device 100.

  The image buffer 221 is a storage unit that functions as a memory that stores images and a video memory that stores images.

  The session information storage unit 222 stores session information representing information related to the main device 100 that is establishing a session. For example, the session information storage unit 222 stores session information including session state information and transmission control information.

  Note that the image buffer 221 and the session information storage unit 222 can be configured by any commonly used storage medium such as an HDD, an optical disk, a memory card, and a RAM.

  The input / output interface 211 is an input / output interface for the display 201 and the input device 202, and is realized by an application program such as a GUI (Graphical User Interface).

  For example, the input / output interface 211 acquires image information from the image buffer 221 and displays it on the display 201. The input / output interface 211 acquires the image information transmitted from the main body device 100 via the image generation unit 212 and writes the image information in the image buffer 221. In addition, the input / output interface 211 is for GUI generated independently in the display terminal 200. The image information is written into the image buffer 221.

  The image generation unit 212 expands the compressed image received from the main device 100 and then writes the expanded image information to the designated drawing position of the drawing image buffer 221. That is, the image generation unit 212 displays a partial image generated by expanding the compressed image transmitted from the main device 100 and received by the wireless communication processing unit 215 at a designated position on the display 201.

  The video decoding unit 213 decodes the encoded video received from the main device 100, and then writes the decoded video information into the drawing image buffer 221 at a specified interval. At this time, if the encoding method of the encoded video information sent from the main device 100 is different from the encoding method that can be decoded by the display terminal 200, the compressed video information cannot be decoded, It cannot be played.

  The session manager 214 manages communication (session) established with the main device 100. For example, when the session manager 214 establishes a session with the main device 100, the session manager 214 generates session information in which session state information, transmission control information, and the like are associated with each other, and stores the session information in the session information storage unit 222.

  The wireless communication processing unit 215 transmits / receives a signal to / from the wireless base station 300 via the antenna 203. The wireless communication processing unit 215 includes a transmission unit 215a that transmits a message and a reception unit 215b that receives the message. The wireless communication processing unit 215 is realized by a wireless LAN function compliant with IEEE 802.11 or the like.

  For example, the reception unit 215b of the wireless communication processing unit 215 demodulates the received wireless signal to generate a packet, and passes the corresponding data to the image generation unit 212 according to the message type of the packet. For example, when the packet is a packet of a transmission image message including a compressed image, information such as the compressed image extracted from the packet and the number of pixels is passed to the image generation unit 212. For example, when the packet is a video information packet (packet of a transmission video message) of the video streaming system, the encoded video extracted from the packet is passed to the video decoding unit 213.

  Whether the display terminal 200 performs remote operation of the main body device 100 using a screen transfer system or whether to view video information using a video streaming system is determined via a GUI (Graphical User Interface) on the input / output interface 211. A method of allowing the user to select is conceivable.

  When the screen transfer system is used, the input device 202 acquires input information from the user using a digitizer, a touch screen, or the like. The input information is transmitted to the main device 100 via the wireless communication processing unit 215 after the coordinate information and the like are analyzed by the input / output interface 211. In this case, the main device 100 executes application processing based on the input information received from the display terminal 200. When an update occurs in the screen area as a result of the application process, the main device 100 acquires the image information to be updated, generates a transmission image message, and transmits the message to the display terminal 200.

  When the video streaming system is used, the user selects one of multimedia contents such as video data (video information), music data, and photo data stored on the main device 100 displayed on the GUI. . The display terminal 200 determines multimedia content to be played based on user input information.

  For example, according to the UPnP standard, identification information such as a URI (Uniform Resource Identifier) is added to the multimedia content as an identifier for identifying a location on the network. After analyzing the selection operation from the user, the display terminal 200 starts a communication session for using the video streaming function of the main device 100 based on the URI information. At this time, HTTP (Hyper Text Transfer Protocol) may be used as the communication session, but transmission control such as RTP (Real-time Transport Protocol) may be used. In the case of a video streaming system, when the main unit 100 detects that a communication session has been started from the display terminal 200, the main unit 100 converts the video information stored in the external storage device 103 into new video information by the video generation unit 115. While performing the conversion, transmission control is performed so as to transmit to the display terminal 200 via the communication processing unit 117.

  Next, a connection determination process performed by the main device 100 according to the present embodiment configured as described above will be described with reference to FIGS. 4 and 5. FIG. 4 is a flowchart showing the overall flow of the connection determination process in the present embodiment.

  First, the connection determination unit 118 receives a connection request from the display terminal 200 via the reception unit 117b of the communication processing unit 117 (step S401). The connection determination unit 118 determines whether or not the received connection request is a request from the second and subsequent display terminals 200 (step S402). For example, the connection determination unit 118 determines whether the request is from the second display terminal 200 by storing the number of connected display terminals 200 in a storage unit or the like.

  When it is a connection request from the second and subsequent display terminals 200 (step S402: YES), the connection determination unit 118 determines whether or not a connection permission / rejection condition is satisfied (step S403).

  FIG. 5 is a diagram illustrating an example of the connectability condition. The figure shows a connection including a function used by the first display terminal 200, a function requested by the second display terminal 200, and information indicating whether or not the second and subsequent display terminals 200 can be connected. The example which memorize | stored the information showing whether the availability condition and the confirmation regarding performance degradation are required is shown.

  As described above, in the communication system 10 according to the present embodiment, one main device 100 provides a plurality of different functions to a plurality of display terminals 200. That is, the main device 100 provides a function as a screen transfer system (screen transfer function) and a function as a video streaming system (video streaming function).

  In the present embodiment, it is assumed that a plurality of users cannot use the screen transfer function at the same time. Therefore, as shown in the figure, when the first display terminal 200 uses the screen transfer function, when the other display terminal 200 requests the use of the screen transfer function, the second display terminal 200 uses the screen transfer function. Deny subsequent connections. In the figure, an example is shown in which “◯” is set as a connection permission / inhibition condition when the second and subsequent connections are permitted, and “X” is set when permission / denial is permitted.

  In step S403 in FIG. 4, the connection determination unit 118 determines whether or not connection is possible based on such a connection possibility condition. Note that even when the connection of the second display terminal 200 is permitted, a decrease in performance is predicted due to the function being used by the first display terminal 200. It may be desirable to confirm whether or not to allow the above.

  For example, the encoding method of the video information acquired by the main device 100 from the external storage device 103 matches the encoding method of the video information that can be reproduced by the display terminal 200, and the bit rate and resolution need to be changed. When there is no transcoding, the transcode processing by the video generation unit 115 is not necessary. When transcoding is not performed, the video streaming function does not require a large amount of computation, so the performance does not deteriorate and confirmation with the user is unnecessary.

  On the other hand, when transcoding is performed by the video generation unit 115, since it is necessary to perform decoding processing and encoding processing of video information as described above, a large amount of calculation is required. Therefore, when operating a plurality of functions including a video streaming function for transcoding, there is a possibility that the performance of the main body device 100 is degraded. In such a case, it is desirable to confirm with the user that the performance may be degraded.

  Therefore, in the present embodiment, when the connection permission condition is satisfied, the connection determination unit 118 further confirms with the user whether or not the performance deterioration is allowed with reference to the confirmation information regarding the performance deterioration in FIG. Judge whether to do.

  That is, when it is determined in step S403 in FIG. 4 that the connection permission / inhibition condition is satisfied (step S403: YES), the connection determination unit 118 determines whether or not it is necessary to check performance degradation (step S404). Specifically, the connection determination unit 118 determines that it is necessary to check performance degradation when at least one display terminal 200 is using or requests to use a video streaming function with transcode.

  When it is determined that it is necessary to confirm the performance degradation (step S404: YES), the connection determination unit 118 displays a message confirming that the performance may be degraded on the display 201 of the display terminal 200 (step S405). ). In addition, the method of confirming performance degradation with respect to a user is not restricted to this. For example, any conventionally used method such as a method of outputting a confirmation message by voice through the speaker of the display terminal 200 can be applied.

  Next, the connection determination unit 118 determines whether or not the possibility of performance degradation has been accepted by the user (step S406). If accepted (step S406: YES), the connection determination unit 118 permits the connection of the display terminal 200 that has transmitted the connection request (step S407). If not accepted (step S406: NO), the connection determination unit 118 rejects the connection of the display terminal 200 (step S408).

  If it is determined in step S402 that the connection request is not from the second and subsequent display terminals 200 (step S402: NO), that is, if the connection request is from the first display terminal 200, the connection determination unit 118 permits the connection of the display terminal 200 (step S407). If it is determined in step S403 that the connection permission / rejection condition is not satisfied (step S403: NO), the connection determination unit 118 rejects the connection of the second and subsequent display terminals 200 (step S408). In the example of FIG. 5, the case where the second display terminal 200 requests use of the screen transfer function while the first display terminal 200 uses the screen transfer function is applicable.

  In this embodiment, after permitting the connection and function provision of the second display terminal 200 in this way, the control unit 116 measures the performance of each function, and provides a plurality of functions to the plurality of display terminals 200. provide. Then, when it is detected that the performance of the function being used by the first display terminal 200 has deteriorated, control is performed to lower the performance of the function being used by the second display terminal 200 connected later. . Details of the performance control process in which the main device 100 controls the performance of each function will be described later.

  Next, an outline of the video streaming function will be described with reference to FIG. FIG. 6 is a flowchart showing the overall flow of the video information transmission process in the present embodiment.

  First, the video generation unit 115 starts decoding video information in the external storage device 103 designated by the display terminal 200 (step S601). Next, the video generation unit 115 encodes the decoded video information into video information in a format suitable for the designated display terminal 200 (step S602). Note that the video generation unit 115 encodes the video information so that the generation frame rate specified by the control unit 116 is obtained. Then, the video generation unit 115 transmits the encoded video information to the display terminal 200 (step S603).

  In the present embodiment, the video generation unit 115 measures an actual measurement value P1 of the processing time from the start of decoding of video information to the completion of encoding (step S604). Note that the method for measuring the processing time is not limited to the above. For example, the processing time from the start of decoding the video information to the completion of transmission of the encoded video information may be measured. In addition, the index indicating the performance of the video streaming function is not limited to the above processing time. For example, an actual measurement value V of the frame rate obtained by dividing the number of generated frames by the processing time may be measured.

  Next, the video generation unit 115 determines whether or not the video information has ended (step S605). If the video information has not ended (step S605: NO), the video generation unit 115 further acquires the video information from the external storage device 103. The decoding process is repeated (step S601). If the video information has ended (step S605: YES), the video information transmission process ends.

  Next, an overview of the screen transfer function will be described with reference to FIG. FIG. 7 is a flowchart showing the overall flow of the image transmission process in the present embodiment.

  First, the input analysis unit 114 acquires user input information transmitted from the display terminal 200, thins out the input information at a specified thinning rate α, and sends the input information to the event acquisition unit 111 (step S701).

  The application program executes a predetermined process according to the input information (step S702). When the screen is updated by executing the application program, the event acquisition unit 111 acquires that the image has been updated as an event.

  The difference detection unit 112 waits until a time interval (standby time t) designated for executing the difference detection process elapses (step S703).

  After the standby process, the difference detection unit 112 detects a difference area between the new image information generated by the updated image generation unit 111a of the event acquisition unit 111 and the image information buffered in the image buffer 121. (Step S704).

  Next, the compressed image generation unit 113 generates a compressed image obtained by compressing the image in the difference area, and generates a transmission image message including the compressed image (step S705). Next, the transmission unit 117a transmits the generated transmission image message to the display terminal 200 (step S706).

  In the present embodiment, the compressed image generation unit 113 measures the measured value P2 of the processing time from the start of the difference area detection processing to the completion of the compressed image generation processing (step S707), and notifies the control unit 116 of the measured value. To do. In addition, you may comprise so that the actual value of the processing time from the detection start of a difference area | region to the completion of transmission of a compressed image may be measured.

  Next, the details of the performance control process for controlling the performance of each function with reference to the measured performance values measured in the procedure shown in FIGS. 6 and 7 will be described with reference to FIG. FIG. 8 is a flowchart showing the overall flow of the performance control process in the present embodiment.

  First, a component that provides each function of the main device 100 starts to provide any of the requested screen transfer function and video streaming function to the two display terminals 200, and the performance of the provided function Measurement is started (step S801).

  Specifically, the event acquisition unit 111, the difference detection unit 112, the compressed image generation unit 113, and the like cooperate to provide a screen transfer function. In addition, the event acquisition unit 111 and the video generation unit 115 cooperate to provide a video streaming function.

  Next, the control unit 116 determines whether or not the actual performance value of the function provided to the first display terminal 200 has reached the performance target value (step S802). For example, when the first display terminal 200 uses the screen transfer function, the control unit 116 receives the measured value P1 of the processing time from the start of decoding the video information to the end of encoding from the video generation unit 115. Then, the control unit 116 compares the target value of the processing time corresponding to the frame rate specified to the video generation unit 115 with the actual measurement value P1, and determines whether the actual measurement value P1 is equal to or less than the target value. . In the case where the video generation unit 115 is configured to measure the actual measurement value V of the frame rate, the control unit 116 has the actual measurement value V of the frame rate equal to or higher than the frame rate specified for the video generation unit 115. Or not.

  Further, for example, when the first display terminal 200 uses the video streaming function, the control unit 116 generates the compressed image generation value P2 of the processing time from the start of detection of the difference area to the completion of generation of the compressed image. Received from the unit 113. Then, the control unit 116 compares a predetermined target value with the actual measurement value P2, and determines whether or not the actual measurement value P2 is equal to or less than the target value.

  When it is determined that the actual performance value has not reached the target value (step S802: NO), the control unit 116 reduces the performance of the function provided to the second display terminal 200. The component is controlled (step S803). After step S803 is executed, the process returns to step S802, and the comparison process between the actual measurement value and the target value is repeated.

  For example, it is assumed that the main device 100 provides a video streaming function to the first display terminal 200 and provides a screen transfer function to the second display terminal 200. Then, it is assumed that the measured value of the processing time of the video streaming function exceeds the target value. In this case, the control unit 116 performs control so as to lower the performance of the screen transfer function by extending the standby time t that is a time interval at which the difference detection unit 112 detects the difference region. Alternatively, the control unit 116 controls the input analysis unit 114 to increase the thinning rate α for thinning out the input information. The control unit 116 may be configured to execute both the extension of the waiting time t and the increase of the thinning rate α.

  For example, it is assumed that the main device 100 provides a screen transfer function to the first display terminal 200 and provides a video streaming function to the second display terminal 200. Then, it is assumed that the measured value of the processing time of the screen transfer function exceeds the target value. In this case, the control unit 116 performs control so as to lower the generation frame rate v specified when the video generation unit 115 generates new video information.

  Further, for example, it is assumed that the main device 100 provides a video streaming function to both the first display terminal 200 and the second display terminal 200. It is assumed that the measured value of the processing time of the video streaming function for the first display terminal 200 exceeds the target value. In this case, the control unit 116 controls to reduce the generation frame rate v of the video information displayed on the second display terminal 200.

  If it is determined that the actual performance value has reached the target value (step S802: YES), the control unit 116 does not execute the performance control of the function, and returns to step S802 to repeat the process.

  In this way, the control unit 116 evaluates the actual measurement value of the function provided to the first display terminal 200, and when the actual measurement value does not reach the target value, the second display terminal Control is performed so as to reduce the performance of the function provided to 200. As a result, it is possible to control the performance more efficiently than in the conventional method in which the performance of other functions is lowered only when a specific function is operated.

  Next, an operation example of the main device 100 will be described. 9 and 10 are sequence diagrams showing an operation example of the main device 100 according to the present embodiment.

  FIG. 9 shows that when the video streaming function (with transcode) is first operating in the main device 100, the second display terminal 200 uses the screen transfer function, and the performance of the video streaming function has deteriorated. This is an example in the case where is detected. In the figure, an example in which the video generation unit 115 is configured to measure the actual measurement value V of the frame rate is shown.

  The control unit 116 detects that the actual measurement value V of the frame rate notified from the video generation unit 115 is below the threshold value (step S901), and instructs the screen transfer function to change the performance (step S902). That is, the control unit 116 instructs the difference detection unit 112 to increase the standby time t, and instructs the input analysis unit 114 to increase the thinning rate α of the input information (step S903).

  FIG. 10 shows that in the situation where the screen transfer function is first operating in the main device 100, the second display terminal 200 uses the video streaming function (with transcode), and the performance of the screen transfer function has deteriorated. This is an example in the case where is detected.

  The control unit 116 detects that the measured value P2 of the processing time notified from the compressed image generation unit 113 exceeds the threshold (step S1001), and instructs the video streaming function to change the performance (step S1002). . That is, the control unit 116 instructs the video generation unit 115 to lower the generation frame rate v of new video information (step S1003).

  In the above embodiment, priority is given to the function started first among a plurality of functions, and when the performance of the function started first is deteriorated, the quality of the function started later is controlled to be reduced. . The method of specifying the priority function is not limited to this, and for example, a specific function among a plurality of functions may be prioritized regardless of the start order. Moreover, you may comprise so that the function performed with the specific display terminal 200 among the several display terminals 200 may be prioritized.

  As described above, when the performance of one of the screen transfer function and the video streaming function is reduced, the communication device (main device) according to the present embodiment is controlled so as to reduce the processing load of the other function. can do. For this reason, when each function can be operated with the same quality without performing performance control, the quality is not unnecessarily degraded. Further, when it is necessary to ensure the quality of a specific function, the quality of a necessary function can be ensured by reducing the quality of other functions. That is, it is possible to efficiently control the performance of a plurality of functions provided to the display terminal.

  Next, the hardware configuration of the communication device (main device) and the display device (display terminal) according to the present embodiment will be described with reference to FIG. FIG. 11 is an explanatory diagram illustrating a hardware configuration of the communication device and the display device according to the present embodiment.

  The communication device and the display device according to the present embodiment communicate with a control device such as a CPU (Central Processing Unit) 51 and a storage device such as a ROM (Read Only Memory) 52 and a RAM 53 by connecting to a network. An I / F 54, an external storage device such as an HDD (Hard Disk Drive) and a CD (Compact Disc) drive device, a display device such as a display device, an input device such as a keyboard and a mouse, and a bus 61 that connects each part. I have.

  A communication program executed by the communication apparatus according to the present embodiment is a file in an installable format or an executable format, and is a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact). The program is recorded on a computer-readable recording medium such as a disk recordable (DVD) or a digital versatile disk (DVD).

  Further, the communication program executed by the communication apparatus according to the present embodiment may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The communication program executed by the communication device according to the present embodiment may be provided or distributed via a network such as the Internet.

  Further, the communication program of the present embodiment may be provided by being incorporated in advance in a ROM or the like.

  The communication program executed by the communication apparatus according to the present embodiment includes the above-described units (event acquisition unit, difference detection unit, compressed image generation unit, input analysis unit, video generation unit, control unit, communication processing unit, connection determination). In the actual hardware, the CPU 51 (processor) reads out the communication program from the storage medium and executes it to load each unit onto the main storage device. It is generated on the main memory.

  As described above, the communication device, the communication method, and the communication program according to the present invention have the function of sharing the application screen between the communication device and the display terminal, and the function of transmitting the video information from the communication device and displaying it on the display terminal. Suitable for the system to be realized.

It is a block diagram which shows the structure of the communication system concerning this Embodiment. It is a block diagram of the main body apparatus concerning this Embodiment. It is a block diagram of the display terminal concerning this embodiment. It is a flowchart which shows the whole flow of the connection determination process in this Embodiment. It is a figure which shows an example of a connection permission / prohibition condition. It is a flowchart which shows the whole flow of the video information transmission process in this Embodiment. It is a flowchart which shows the whole flow of the image transmission process in this Embodiment. It is a flowchart which shows the whole flow of the performance control process in this Embodiment. It is a sequence diagram which shows the operation example of the main body apparatus of this Embodiment. It is a sequence diagram which shows the operation example of the main body apparatus of this Embodiment. It is explanatory drawing which shows the hardware constitutions of the communication apparatus and display apparatus concerning this Embodiment.

Explanation of symbols

51 CPU
52 ROM
53 RAM
54 Communication I / F
61 Bus 10 Communication System 100 Main Unit 101 Display 102 Input Device 103 External Storage Device 111 Event Acquisition Unit 111a Update Image Generation Unit 112 Difference Detection Unit 113 Compressed Image Generation Unit 114 Input Analysis Unit 115 Video Generation Unit 116 Control Unit 117 Communication Processing Unit 117a Transmission unit 117b Reception unit 118 Connection determination unit 121 Image buffer 122 Condition storage unit 200a, 200b Display terminal 201 Display 202 Input device 203 Antenna 211 Input / output interface 212 Image generation unit 213 Video decoding unit 214 Session manager 215 Wireless communication processing unit 215a Transmission unit 215b Reception unit 221 Image buffer 222 Session information storage unit 300 Wireless base station 400 Network

Claims (7)

A communication device connected to a display device for displaying an image via a network,
A first storage for storing a display image to be displayed on the display device;
An input analysis unit that accepts input information transmitted from the display device and thins out the input information according to a designation of a thinning rate that represents a ratio of thinning out the input information;
An update image generation unit for generating an update image for updating the display image according to the thinned input information;
A detection unit that detects a difference region representing a region where pixel information does not match between the updated image and the display image at a specified time interval;
A compressed image generation unit that generates a compressed image obtained by compressing the image of the difference area;
A video generation unit for generating video information to be displayed on the display device;
A transmission unit for transmitting the compressed image and the video information to the display device;
Comparing a first processing time representing a time from the start of the generation process of the video information to the end of either the generation process of the video information and the transmission process of the video information with a predetermined first threshold, When the first processing time is larger than the first threshold, at least one of control for increasing the thinning rate specified for the input analysis unit and control for increasing the time interval specified for the detection unit A control unit for executing
A communication apparatus comprising: The control unit further includes a second processing time that is predetermined as a second processing time that represents a time from the start of the difference area detection processing to the end of either the compressed image generation processing or the compressed image transmission processing. Comparing the threshold value and controlling the video generation unit to lower the frame rate when generating the video information when the second processing time is greater than the second threshold value;
The communication apparatus according to claim 1. A second storage unit for storing video information;
The video generation unit acquires video information stored in the second storage unit, generates video information to be displayed on the display device from the acquired video information,
The control unit includes the first processing time indicating the time from the start of the process of acquiring the video information from the second storage unit to the completion of the process of generating the video information to be displayed on the display device; When the first processing time is compared with the first threshold and the first processing time is larger than the first threshold, the control for increasing the thinning rate specified for the input analysis unit, and the specification specified for the detection unit Performing at least one of the controls to increase the time interval;
The communication apparatus according to claim 1. A second storage unit for storing video information;
The video generation unit acquires video information stored in the second storage unit, generates video information to be displayed on the display device from the acquired video information,
The control unit includes a first processing time that represents a time from when a process of acquiring video information from the second storage unit to a process of transmitting the generated video information to the display device is completed. The first threshold value is compared, and when the first processing time is larger than the first threshold value, control for increasing the thinning rate specified for the input analysis unit, and specification for the detection unit Performing at least one of the controls to increase the time interval;
The communication apparatus according to claim 1. A communication method executed by a communication device connected to a display device for displaying an image via a network,
The communication device includes a first storage unit that stores a display image to be displayed on the display device,
An input analysis unit that accepts input information transmitted from the display device, and an input analysis step of thinning out the input information according to a designation of a thinning rate that represents a ratio of thinning out the input information;
An update image generation step, wherein an update image generation unit generates an update image for updating the display image in accordance with the thinned input information;
A detection step in which a detection unit detects a difference region representing a region where pixel information does not match between the updated image and the display image at a specified time interval;
A compressed image generation step, wherein the compressed image generation unit generates a compressed image obtained by compressing the image of the difference area;
A video generation step for generating video information to be displayed on the display device by the video generation unit;
A transmitting step for transmitting the compressed image and the video information to the display device;
The control unit has a first processing time indicating a time from the start of the video information generation process to the end of any of the video information generation process and the video information transmission process, and a predetermined first threshold value. In comparison, when the first processing time is larger than the first threshold, the control for increasing the thinning rate specified for the input analysis unit and the time interval specified for the detection unit are increased. A control step for performing at least one of the controls;
A communication method comprising: Computer
A first storage unit for storing a display image to be displayed on a display device connected via a network;
An input analysis unit that accepts input information transmitted from the display device and thins out the input information according to a designation of a thinning rate that represents a ratio of thinning out the input information;
An update image generation unit for generating an update image for updating the display image according to the thinned input information;
A detection unit that detects a difference region representing a region where pixel information does not match between the updated image and the display image at a specified time interval;
A compressed image generation unit that generates a compressed image obtained by compressing the image of the difference area;
A video generation unit for generating video information to be displayed on the display device;
A transmission unit for transmitting the compressed image and the video information to the display device;
Comparing a first processing time representing a time from the start of the generation process of the video information to the end of either the generation process of the video information and the transmission process of the video information with a predetermined first threshold, When the first processing time is larger than the first threshold, at least one of control for increasing the thinning rate specified for the input analysis unit and control for increasing the time interval specified for the detection unit A control unit for executing
Communication program to function as A communication device connected to a first display device and a second display device for displaying an image via a network,
A video generator for generating first video information to be displayed on the first display device and second video information to be displayed on the second display device;
A transmission unit for transmitting the first video information to the first display device and transmitting the second video information to the second display device;
Comparing the processing time from the start of the generation process of the first video information to the end of either the generation process of the first video information and the transmission process of the first video information with a predetermined threshold, A control unit that controls the video generation unit to lower a frame rate when generating the second video information when a processing time is greater than the threshold;
A communication apparatus comprising:

Images