WO2016072675A1 - Method and device for controlling screen sharing among plurality of terminals, and recording medium - Google Patents

Abstract

The present disclosure relates to a technology for sensor networks, machine to machine (M2M), machine type communication (MTC) and Internet of Things (IoT). The present disclosure can be utilized for intelligent services based on the aforementioned technologies (e.g. smart homes, smart buildings, smart cities, smart cars or connected cars, health care, digital education, retail businesses, security, disaster network communications and safety related services, etc.). The present disclosure relates to a method for controlling screen sharing among a plurality of terminals, in which a relay device receives, from a first terminal, a screen image of a first terminal to thus transmit the received screen image to at least one second terminal.

Description

Method, apparatus and recording medium for controlling screen sharing between a plurality of terminals

The disclosed embodiment is capable of computer readable recording a program for performing a method for controlling screen sharing between a plurality of terminals, a device for controlling screen sharing between a plurality of terminals, and a method for controlling screen sharing between a plurality of terminals. It relates to a recording medium.

The Internet has evolved from a human-centered connection network where humans create and consume information, and an Internet of Things (IoT) network that exchanges and processes information among distributed components such as things. The Internet of Everything (IoE) technology, which combines big data processing technology through connection with cloud servers and the like, is emerging. In order to implement the IoT, technical elements such as sensing technology, wired / wireless communication and network infrastructure, service interface technology, and security technology are required, and recently, a sensor network for connection between things, a machine to machine , M2M), Machine Type Communication (MTC), etc. are being studied.

In an IoT environment, intelligent Internet technology (IT) services can be provided that collect and analyze data generated from connected objects to create new value in human life. IoT is a combination of existing IT (information technology) technology and various industries, and smart home, smart building, smart city, smart car or connected car, smart grid, healthcare, smart home appliances, advanced medical services, disaster network It can be applied to fields such as communication.

The disclosed embodiment provides a method and apparatus for controlling screen sharing between a plurality of terminals that can improve quality of experience (QoE) by transmitting a screen image through a relay device when the screen is shared among a plurality of terminals. To provide.

The present disclosure relates to a method of controlling a screen sharing between a plurality of terminals that a relay device receives a screen image of a first terminal from a first terminal and transmits the received screen image to at least one second terminal.

1 is a diagram for describing a communication system for controlling screen sharing between a plurality of terminals according to an embodiment.

2 is a flowchart illustrating a method of controlling a screen sharing between a plurality of terminals by a relay device according to an embodiment.

3 is a flowchart illustrating a method of controlling a screen sharing between a plurality of terminals by a relay device according to an embodiment in more detail.

4 is a flowchart illustrating a method of controlling a screen sharing between a plurality of terminals by a first terminal according to an exemplary embodiment.

5 is a flowchart illustrating a method of controlling a screen sharing between a plurality of terminals by a second terminal according to an embodiment.

6 is a flowchart illustrating an operation of a communication system that controls screen sharing between a plurality of terminals according to an embodiment.

7 is a diagram for describing a method of controlling, by an AP device, screen sharing between a plurality of terminals as a relay device, according to an embodiment.

8 is a diagram for describing a method of controlling a screen sharing between a plurality of terminals by a relay server when a PTT service is performed in a communication system according to an embodiment.

9 is a diagram for describing a method of controlling a screen sharing between a plurality of terminals by a relay device connected to a plurality of AP devices by wire in a communication system according to an exemplary embodiment.

FIG. 10 illustrates a method in which a first terminal selects at least one of a plurality of second terminals as a relay device and transmits a screen image of the first terminal through the selected relay device in the communication system according to an embodiment. It is a flow chart.

11 is a flowchart illustrating a method of reselecting a relay device according to a performance of each of a plurality of second terminals and a situation of a network including each of a plurality of second terminals in a communication system according to an exemplary embodiment. to be.

12 illustrates a method of transmitting a screen image of a first terminal to other second terminals by a second terminal selected from among a plurality of second terminals that receive the screen image of the first terminal according to an embodiment; It is a flowchart for doing so.

FIG. 13 is a flowchart for describing a method of sharing a screen image of a first terminal among a plurality of second terminals when a relay device is changed according to an embodiment.

14 is a block diagram of a relay device according to an embodiment.

15 and 16 are block diagrams of a first terminal according to an embodiment.

17 and 18 are block diagrams of a second terminal according to an embodiment.

According to an embodiment of the present disclosure, a method of controlling screen sharing between a plurality of terminals includes: at least one second terminal receiving performance information regarding a screen image transmission and a screen image of a first terminal from which a relay device transmits a screen image; Obtaining context information of an included network; Determining an encoding quality of the screen image of the first terminal based on the performance information and the situation information of the network; Receiving the encoded screen image at the first terminal based on the determined encoding quality; And transmitting the received screen image to at least one second terminal.

According to an embodiment, the method of controlling screen sharing between a plurality of terminals further includes transmitting information regarding the determined encoding quality to the first terminal.

According to an embodiment, in a method of controlling screen sharing between a plurality of terminals, the determining of the encoding quality of the screen image may include transmitting the screen image to at least one second terminal based on context information of the network. Determining a bitrate for the device, and based on the determined bitrate, determining an encoding quality comprising at least one of a resolution and a frame rate.

According to an embodiment, in a method of controlling screen sharing between a plurality of terminals, the relay device may include performance information of each of the plurality of second terminals receiving the screen image of the first terminal and each of the plurality of second terminals. It is selected in the first terminal based on the context information of the included network.

According to an embodiment of the present disclosure, a method of controlling screen sharing between a plurality of terminals includes a performance and a network among a plurality of second terminals when a performance of a relay device and a situation of a network including the relay device are out of a preset reference range. Receiving a message from the first terminal indicating that the second terminal corresponding to a situation is selected as a new relay device; And receiving the screen image encoded in the first terminal from the selected second terminal according to the encoding quality determined in the selected second terminal.

According to an embodiment, a method of controlling screen sharing between a plurality of terminals may include obtaining identification information of at least one second terminal from a first terminal; And establishing a communication session with the at least one second terminal based on the obtained identification information.

According to an embodiment, a method of controlling screen sharing between a plurality of terminals may include obtaining identification information of a first terminal and at least one second terminal from a server; And establishing a communication session with the first terminal and the at least one second terminal based on the obtained identification information.

According to an embodiment, a method of controlling screen sharing between a plurality of terminals may include at least one of performance information regarding transmission of a screen image of a first terminal from a relay device that transmits a screen image of a first terminal to at least one second terminal. Obtaining information about an encoding quality of a screen image of a first terminal determined based on context information of a network including one second terminal; Encoding a screen image of the first terminal based on the obtained information about the encoding quality; And transmitting the encoded screen image to the relay device, wherein the encoded screen image is transmitted to the second terminal via the relay device.

According to an embodiment, the method of controlling screen sharing between a plurality of terminals further includes transmitting performance information regarding transmission of a screen image of a first terminal to a relay device.

According to an exemplary embodiment, a method of controlling screen sharing between a plurality of terminals includes a situation in which a network includes performance information of each of a plurality of second terminals receiving a screen image of a first terminal and each of a plurality of second terminals. Selecting at least one of the plurality of second terminals as the relay device based on the information; And transmitting a message indicating the selection of the relay device to each of the plurality of second terminals.

According to an embodiment of the present disclosure, a method of controlling screen sharing between a plurality of terminals includes a performance and a network among a plurality of second terminals when a performance of a relay device and a situation of a network including the relay device are out of a preset reference range. The method may further include reselecting another second terminal corresponding to the preset reference range as the relay device.

According to an exemplary embodiment, a method of controlling screen sharing between a plurality of terminals includes: transmitting, by a second terminal receiving a screen image of a first terminal, information about a situation of a network including a second terminal to a relay device; And receiving, from the relay device, the screen image encoded in the first terminal according to the encoding quality determined by the relay device, wherein the encoding quality includes performance information regarding the transmission of the screen image of the first terminal acquired by the relay device and the relay. The determination is based on context information of a network including at least one second terminal that receives the screen image of the first terminal through the device.

According to an exemplary embodiment, a relay device for controlling screen sharing between a plurality of terminals includes at least one second terminal for receiving screen information and performance information regarding screen image transmission of a first terminal for transmitting a screen image. A communication unit for obtaining situation information of a network; And a controller configured to determine an encoding quality of the screen image of the first terminal based on the performance information and the situation information of the network, wherein the communication unit receives the screen image encoded by the first terminal based on the determined encoding quality, The received screen image is transmitted to at least one second terminal.

According to an embodiment, in a relay device that controls screen sharing between a plurality of terminals, the communication unit transmits information about the determined encoding quality to the first terminal.

According to an embodiment, in a relay device that controls screen sharing between a plurality of terminals, the controller may determine a bit rate for transmitting the screen image to at least one second terminal based on context information of the network, Based on the determined bitrate, determine an encoding quality comprising at least one of resolution and frame rate.

In a relay device for controlling screen sharing between a plurality of terminals according to an embodiment, the relay device, the performance information of each of the plurality of second terminals for receiving the screen image of the first terminal and the plurality of second terminals Each is selected at the first terminal based on the context information of the included network.

According to an embodiment, in a relay device that controls screen sharing among a plurality of terminals, the communication unit may include a plurality of second terminals when a performance of the relay device and a situation of a network including the relay device are out of a preset reference range. Among them, a message indicating that a second terminal whose performance and network conditions correspond to a preset reference range is selected as a new relay device is received from the first terminal, and is encoded by the first terminal according to the encoding quality determined by the selected second terminal. The received screen image from the selected second terminal.

In a relay device for controlling screen sharing between a plurality of terminals according to an embodiment, the communication unit obtains identification information of at least one second terminal from the first terminal, the control unit based on the obtained identification information Set up a communication session with at least one second terminal.

In a relay device for controlling screen sharing between a plurality of terminals according to an embodiment, the communication unit obtains identification information of the first terminal and at least one second terminal from the server, and the control unit is configured to obtain the obtained identification information Establish a communication session with the first terminal and the at least one second terminal on the basis.

According to an exemplary embodiment, a first terminal that controls screen sharing between a plurality of terminals includes a performance related to screen image transmission of a first terminal from a relay device that transmits a screen image of the first terminal to at least one second terminal. A communication unit for obtaining information about an encoding quality of a screen image of the first terminal determined based on the information and the context information of the network including the at least one second terminal; A controller configured to determine an encoding quality of the screen image of the first terminal based on the obtained information about the encoding quality; And an encoding unit for encoding the screen image of the first terminal according to the determined encoding quality, wherein the communication unit transmits the encoded screen image to the relay device, and the encoded screen image is transmitted to the second terminal through the relay device. .

In a first terminal for controlling screen sharing among a plurality of terminals according to an embodiment, the communication unit transmits performance information regarding transmission of a screen image of the first terminal to a relay device.

In a first terminal for controlling screen sharing between a plurality of terminals according to an embodiment, the control unit, the performance information of each of the plurality of second terminals for receiving the screen image of the first terminal and the plurality of second terminals Based on the context information of each network included, at least one of the plurality of second terminals is selected as the relay device, and the communication unit transmits a message indicating the selection of the relay device to each of the plurality of second terminals.

According to an embodiment of the present disclosure, in a first terminal for controlling screen sharing between a plurality of terminals, the controller may include a plurality of second devices when the performance of the relay device and the situation of the network including the relay device are out of a preset reference range. Among the terminals, another second terminal whose performance and network conditions correspond to the preset reference range is reselected as the relay device.

According to an embodiment of the present disclosure, a second terminal for controlling screen sharing between a plurality of terminals includes a controller configured to acquire, by a second terminal receiving a screen image of the first terminal, context information of a network including the second terminal; And a communication unit configured to transmit the acquired situation information of the network to the relay device and to receive a screen image encoded by the first terminal from the relay device according to the encoding quality determined by the relay device, wherein the encoding quality is obtained by the relay device. It is determined based on the performance information on the transmission of the screen image of the first terminal and the status information of the network including at least one second terminal receiving the screen image of the first terminal through the relay device.

Terms used herein will be briefly described and the present invention will be described in detail.

The terms used in the present invention have been selected as widely used general terms as possible in consideration of the functions in the present invention, but this may vary according to the intention or precedent of the person skilled in the art, the emergence of new technologies and the like. In addition, in certain cases, there is also a term arbitrarily selected by the applicant, in which case the meaning will be described in detail in the description of the invention. Therefore, the terms used in the present invention should be defined based on the meanings of the terms and the contents throughout the present invention, rather than the names of the simple terms.

When any part of the specification is to "include" any component, this means that it may further include other components, except to exclude other components unless otherwise stated. In addition, the terms "... unit", "module", etc. described in the specification mean a unit for processing at least one function or operation, which may be implemented in hardware or software or a combination of hardware and software. .

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

FIG. 1 is a diagram for describing a communication system 10 (hereinafter, referred to as a communication system) for controlling screen sharing between a plurality of terminals 100, 300a, 300b, and 300c, according to an exemplary embodiment.

The communication system 10 according to an embodiment may include a plurality of terminals 100, 300a, 300b, and 300c and a relay device 200.

In the communication system 10 shown in FIG. 1, only components related to the present embodiment are shown. Accordingly, it will be understood by those skilled in the art that other general purpose components may be further included in addition to the components shown in FIG. 1. For example, the communication system 10 shown in FIG. 1 may further include a server or another terminal in addition to the plurality of terminals 100, 300a, 300b, and 300c and the relay device 200.

Among the plurality of terminals 100, 300a, 300b and 300c, the first terminal 100 transmits a screen image to be shared when the screen is shared with the plurality of second terminals 300a, 300b and 300c. to be. Here, the screen image represents an image output on the screen of the first terminal 100. The first terminal 100 according to an embodiment may generate a screen image by capturing an image output on the screen of the first terminal 100.

According to another exemplary embodiment, the screen image may be an image output on the screen of the external device connected to the first terminal 100. In this case, the first terminal 100 may receive an image capturing the screen of the external device from the external device.

Meanwhile, the first terminal 100 may encode the screen image according to a specific encoding quality. Here, the encoding quality may include at least one of a frame rate and a resolution. However, this is only an embodiment, and according to another example, the encoding quality may further include codec setting information such as a bit rate, a forward error correction (FEC) rate, and whether intra refresh is applied, as well as a frame rate and resolution. have.

Meanwhile, the first terminal 100 may receive information about the encoding quality of the screen image from the relay device 200. The first terminal 100 may encode the screen image of the first terminal based on the information about the encoding quality received from the relay device 200.

The first terminal 100 according to an embodiment may transmit the encoded screen image to the relay device 200. For example, the first terminal 100 may transmit the screen image encoded in data units such as packets and segments to the relay device 200.

The relay device 200 that controls screen sharing among the plurality of terminals 100, 300a, 300b, and 300c may receive an encoded screen image from the first terminal 100. In addition, the relay device 200 may transmit the received screen image to the plurality of second terminals 300a, 300b, and 300c, respectively.

Meanwhile, the relay device 200 may determine the encoding quality of the screen image of the first terminal 100 encoded by the first terminal 100. According to one embodiment, the relay device 200 is based on the performance of the first terminal 100 and the situation of the network including each of the plurality of second terminals (300a, 300b, 300c), the first terminal 100 Can determine the encoding quality of the screen image.

Here, the performance information regarding the transmission of the screen image of the first terminal 100 may include information about CPU usage, encoding processing time and frame rate of the first terminal 100. In addition, the context information of the network including each of the plurality of second terminals 300a, 300b, and 300c may include information about a round trip time (RTT), a data transmission / reception delay time, and a data loss rate. However, this is only an exemplary embodiment, and congestion of the network including the second terminals 300a, 300b, and 300c may be included in the context information of the network including each of the plurality of second terminals 300a, 300b, and 300c. Other information that may indicate may be further included.

Meanwhile, according to another exemplary embodiment, the relay device 100 may include performance information regarding screen image transmission of the first terminal 100, situation information of a network including the first terminal 100, and a plurality of second terminals ( The encoding quality of the screen image of the first terminal 100 may be determined based on the context information of the network including the respective 300a, 300b, and 300c. For example, when wireless communication is established between the relay device 100 and the first terminal 100, the relay device 100 considers the status information of the network including the first terminal 100, thereby relaying the device 100. ) Can be prevented from occurring in the network between the first terminal 100 and the first terminal 100.

The plurality of second terminals 300a, 300b, and 300c according to an exemplary embodiment are terminals that receive a screen image to be shared when the screen is shared with the first terminal 100. Each of the plurality of second terminals 300a, 300b, and 300c may receive a screen image of the first terminal 100 from the relay device 200. When each of the plurality of second terminals 300a, 300b, and 300c receives the screen image of the first terminal 100, each of the plurality of second terminals 300a, 300b and 300c may receive the first terminal 100. Share an image corresponding to the screen image of the.

Meanwhile, the plurality of terminals 100, 300a, 300b, and 300c according to an embodiment may include a personal computer (PC), a smartphone, a tablet, a wearable device, and the like. In addition, the relay device 200 may include a server, a personal computer (PC), a smartphone, a tablet, a wearable device, and the like.

2 is a flowchart illustrating a method of controlling a screen sharing between a plurality of terminals by a relay device according to an embodiment.

In step S210, the relay device obtains the performance information about the screen image transmission of the first terminal that transmits the screen image and the status information of the network including the at least one second terminal that receives the screen image.

The relay device according to an embodiment may obtain performance information regarding screen image transmission of the first terminal from the first terminal. For example, the relay device may obtain information regarding CPU usage, encoding processing time and frame rate of the first terminal 100. The relay device may acquire at least one of Real Time Control Protocol (RTCP), Real Time Protocol (RTP), Session Initiation Protocol (SIP), and Extensible Messaging and Presence Protocol (XMPP) for acquiring the performance information regarding the transmission of the screen image of the first terminal. One protocol may be used.

In addition, the relay device according to an embodiment may obtain context information of a network including at least one second terminal from at least one second terminal receiving the screen image of the first terminal. For example, the relay device may obtain information regarding at least one of an RTT, a data transmission / reception delay time, and a data loss rate of a network including at least one second terminal. The relay device may receive context information of a network including at least one second terminal based on at least one protocol of RTCP, RTP Extension, SIP, and XMPP.

In step S220, the relay device determines the encoding quality of the screen image of the first terminal based on the performance information and the situation information of the network.

The relay device according to one embodiment may not generate congestion in a network including at least one second terminal based on context information of a network received from at least one second terminal. The bit rate for screen image transmission may be determined. Here, the relay device may store a preset bitrate value according to at least one of RTT, data transmission / reception delay time, and data loss rate included in the situation information of the network. The relay device may determine the bit rate for transmitting the screen image of the first terminal according to the received situation information of the network based on the previously stored information.

In addition, the relay device may determine the encoding quality of the screen image encoded in the first terminal by using the determined bitrate and performance information regarding the transmission of the screen image of the first terminal. For example, when the frame rate set according to the determined bit rate is fps_1 and the frame rate transmittable by the first terminal is fps_2, the relay device may select a lower frame rate among fps_1 and fps_2.

However, this is only an example, and the relay device may determine the resolution, FEC rate, intra refresh, etc. of the screen image encoded by the first terminal based on the determined bit rate.

The relay device according to an embodiment determines the encoding quality in consideration of the performance information regarding the transmission of the screen image of the first terminal as well as the status information of the network including the at least one second terminal, thereby allowing the range of the allowed network. Transmission and reception of the screen image between a plurality of terminals may be controlled so that the screen image of the highest quality can be shared within the terminal.

In step S230, the relay device receives the screen image encoded at the first terminal based on the determined encoding quality.

The relay device according to an embodiment may transmit information about the determined encoding quality to the first terminal. Here, the relay device may set a channel for transmitting the information on the encoding quality to the first terminal separately from the channel for receiving the screen image from the first terminal. However, this is only an example, and the relay device may transmit information about the determined encoding quality to the first terminal using a channel for receiving the screen image from the first terminal.

The first terminal may determine at least one of frame rate, resolution, FEC rate, and intra refresh for encoding the screen image of the first terminal based on the information about the encoding quality received from the relay device.

The relay device according to an embodiment may receive the encoded screen image according to the information about the encoding quality from the first terminal. For example, the relay device may receive a screen image encoded according to data units such as packets and segments from the first terminal.

In operation S240, the relay device transmits the received screen image to at least one second terminal.

The relay device according to an embodiment may transmit the received screen image to at least one second terminal, respectively. As the at least one second terminal receives the screen image from the relay device, the at least one second terminal may share the screen image corresponding to the screen image of the first terminal.

3 is a flowchart illustrating a method of controlling a screen sharing between a plurality of terminals by a relay device according to an embodiment in more detail.

In operation S310, the relay device may acquire performance information regarding the transmission of the screen image of the first terminal that transmits the screen image and the status information of the network including the at least one second terminal that receives the screen image.

Step S310 may correspond to step S210 described above with reference to FIG. 2.

In operation S320, the relay device may determine a bit rate for transmitting the screen image of the first terminal to the at least one second terminal based on the network condition information of the at least one second terminal.

According to an embodiment, a relay device may include a network including at least one second terminal based on information on at least one of RTT, data transmission / reception delay time, and data loss rate included in network status information of at least one second terminal. The congestion of can be predicted. The relay device may determine a bitrate that may not cause congestion in the network including the at least one second terminal based on the predicted congestion degree of the network.

In operation S330, the relay device may determine an encoding quality of the screen image of the first terminal based on the determined bit rate and performance information regarding the transmission of the screen image of the first terminal.

When the relay device sets the frame rate for the screen image of the first terminal to 15 fps based on the determined bit rate, a case in which the performance of the first terminal cannot process the set frame rate may occur. The relay apparatus according to an embodiment determines the encoding quality of the screen image of the first terminal based on the performance information on the screen image transmission of the first terminal, thereby degrading the quality of the screen image shared among the plurality of terminals. Can be prevented.

For example, when the frame rate according to the determined bit rate is 15 fps and the frame rate of the screen image that can be processed by the first terminal is 3 fps, the relay device may select a frame rate at which the screen image of the first terminal is encoded as 3 fps. have.

Meanwhile, the relay device may determine not only the frame rate but also codec setting information regarding resolution, FEC rate, whether to apply infrastructure refresh, and the like based on the determined bitrate and performance information regarding transmission of the screen image of the first terminal.

In operation S340, the relay device may transmit information about the determined encoding quality to the first terminal.

The relay device according to an embodiment may transmit information about the determined encoding quality to the first terminal based on at least one protocol of RTCP, RTP Extension, SIP, and XMPP.

In operation S350, the relay device may receive the encoded screen image from the first terminal based on the determined encoding quality.

The relay device according to an embodiment may receive the encoded screen image from the first terminal according to data units such as packets and segments.

In operation S360, the relay device may transmit the received screen image to at least one second terminal.

Step S360 may correspond to step S240 described above with reference to FIG. 2.

4 is a flowchart illustrating a method of controlling a screen sharing between a plurality of terminals by a first terminal according to an exemplary embodiment.

In operation S410, the first terminal acquires information about an encoding quality of the screen image of the first terminal from a relay device that transmits the screen image of the first terminal to at least one second terminal. Here, the first terminal is a terminal for transmitting the screen image among a plurality of terminals sharing the screen image.

According to an embodiment, the first terminal is determined based on performance information regarding transmission of a screen image of the first terminal from a relay device and context information of a network including at least one second terminal receiving the screen image of the first terminal. Information about the encoding quality can be obtained.

In operation S420, the first terminal encodes the screen image of the first terminal based on the obtained information about the encoding quality.

The first terminal according to an embodiment may generate a screen image by capturing an image output on the screen of the first terminal. In addition, the first terminal may encode the screen image according to the specified frame rate based on the information about the encoding quality. However, this is only an example, and the first terminal may determine at least one of resolution, FEC rate, and whether infrastructure refresh is applied based on the information about the encoding quality.

In operation S430, the first terminal transmits the encoded screen image to the relay device.

The first terminal according to an embodiment may transmit the screen image to the relay device in data units such as packets and segments.

Meanwhile, as the situation of a network including at least one second terminal sharing a screen image with the first terminal changes, the first terminal may receive feedback on the changed encoding quality in real time. When the encoding quality is changed, the first terminal may transmit the encoded screen image to the relay device according to the changed encoding quality.

5 is a flowchart illustrating a method of controlling a screen sharing between a plurality of terminals by a second terminal according to an embodiment.

In step S510, the second terminal transmits information about the network status of the second terminal to the relay device.

According to an embodiment, the second terminal may transmit status information of a network including the second terminal to the relay device based on at least one protocol of RTCP, RTP Extension, SIP, and XMPP. Here, the status information of the network including the second terminal may include information on at least one of the RTT, data transmission and reception delay time, and data loss rate.

In operation S520, the second terminal may receive the screen image encoded by the first terminal from the relay device according to the encoding quality determined by the relay device.

According to an embodiment, the second terminal may receive a screen image encoded by the first terminal according to data units such as packets and segments from the relay device.

6 is a flowchart illustrating an operation of a communication system that controls screen sharing between a plurality of terminals according to an embodiment.

In FIG. 6, for convenience of description, only an operation of one second terminal has been described. However, this is only an example, and a plurality of second terminals may be included in the communication system. When a plurality of second terminals are included in the communication system, operations of each of the plurality of second terminals may correspond to operations of the second terminal described with reference to FIG. 6.

In operation S610, the first terminal may transmit performance information regarding screen image transmission of the first terminal to the relay device.

According to an embodiment, the first terminal may separately set a channel for transmitting performance information in addition to a channel for transmitting the screen image to the relay device. However, this is only an example, and the first terminal may transmit performance information through a channel for transmitting the screen image to the relay device.

In operation S620, the second terminal may transmit the network condition information including the second terminal to the relay device.

On the other hand, the order in which steps S610 and S620 are executed may be changed.

In operation S630, the relay device may determine the encoding quality of the screen image of the first terminal based on the performance information regarding the transmission of the screen image of the first terminal and the information of the network situation including the second terminal.

In operation S640, the relay device may transmit information about the determined encoding quality to the first terminal.

In operation S650, the first terminal may encode the screen image of the first terminal based on the information about the encoding quality.

For example, the first terminal may encode the screen image of the first terminal based on the frame rate and the resolution included in the information about the encoding quality.

In operation S660, the first terminal may transmit the encoded screen image to the relay device.

In operation S670, the relay device may transmit the received screen image to the second terminal.

As the screen image received by the relay device according to an embodiment is transmitted to the second terminal, the first terminal and the second terminal may share the screen image corresponding to each other.

FIG. 7 is a diagram for describing a method of controlling a screen sharing between a plurality of terminals by an access point (AP) device as a relay device according to an embodiment.

In an embodiment of FIG. 7, the AP device 720 may perform an operation corresponding to the relay device described above with reference to FIG. 2.

 Among the plurality of terminals 710, 730a, 730b, and 730c sharing the screen image, the first terminal 710 that transmits the screen image includes a plurality of second terminals 730a, 730b, and 730c that receive the screen image. IP information may be transmitted to the AP device 720. In addition, the first terminal 710 may transmit performance information regarding the screen image transmission of the first terminal 710 to the AP device 720. The performance information about the screen image transmission of the first terminal 710 may be used by the AP device 720 to determine the encoding quality of the screen image of the first terminal 710.

Meanwhile, the AP device 720 according to an embodiment may provide context information of a network including a plurality of second terminals 730a, 730b, and 730c, respectively, from each of the plurality of second terminals 730a, 730b, and 730c. Can be received. For example, the AP device 720 may provide context information of a network including a plurality of second terminals 730a, 730b, and 730c, respectively, from each of the plurality of second terminals 730a, 730b, and 730c through the RTCP protocol. Can be received.

The AP device 720 based on the performance information on the screen image transmission of the first terminal 710 and the network status information including each of the plurality of second terminals 730a, 730b, and 730c. ) Can determine the encoding quality of the screen image. The AP device 720 may transmit the information about the determined encoding quality to the first terminal 710.

The first terminal 710 may encode the screen image of the first terminal 710 based on the information about the encoding quality received from the AP device 720. The first terminal 710 may transmit the encoded screen image to the AP device 720.

The AP device 720 according to an embodiment may transmit the screen image received from the first terminal 710 to each of the plurality of second terminals 730a, 730b, and 730c. The AP device 720 may transmit the screen image received from the first terminal 710 to each of the plurality of second terminals based on the IP information previously received from the first terminal 710.

8 illustrates a relay server 820 controlling screen sharing between a plurality of terminals 810, 830a, 830b, and 830c when a push to talk (PTT) service is performed in a communication system 800 according to an exemplary embodiment. It is a figure for demonstrating the method.

The communication system 800 according to an embodiment may include a plurality of terminals 810, 830a, 830b, and 830c, a signaling server 815, and a relay server 820.

In the communication system 800 shown in FIG. 8, only components related to the present embodiment are shown. Accordingly, it will be understood by those skilled in the art that other general purpose components may be further included in addition to the components illustrated in FIG. 8.

Referring to FIG. 8, the PTT service performed by the communication system 800 is a one-way video transmission service, and the screen image of the first terminal 810 is transmitted to the plurality of second terminals 830a and 830b through the relay server 830. 830c).

Meanwhile, in the communication system 800, the signaling server 815 may obtain identification information of the first terminal 810 and the plurality of second terminals 830a, 830b, and 830c. In this case, the identification information may include at least one of IP information and port information. In addition, the signaling server 815 may provide the relay server 820 with the identification information of the obtained first terminal 810 and the plurality of second terminals 830a, 830b, and 830c.

The relay server 820 according to an embodiment of the present disclosure may be configured based on the identification information of the first terminal 810 and the plurality of second terminals 830a, 830b, and 830c provided from the signaling server 815. The screen image of the first terminal 810 received from the 810 may be transmitted to each of the plurality of second terminals 830a, 830b, and 830c.

Meanwhile, the relay server 820 according to an exemplary embodiment may receive performance information regarding screen image transmission of the first terminal 810 from the first terminal 810. In addition, the relay server 820 may receive context information of a network including the plurality of second terminals 830a, 830b, and 830c from each of the plurality of second terminals 830a, 830b, and 830c. For example, the relay server 820 is a context information of a network including a plurality of second terminals 830a, 830b, and 830c from each of the plurality of second terminals 830a, 830b, and 830c through the RTCP protocol. Can be received.

The relay server 820 based on the performance information regarding the transmission of the screen image of the first terminal 810 and the network situation information including each of the plurality of second terminals 830a, 830b, and 830c, the first terminal 810. ) Can determine the encoding quality of the screen image. The relay server 820 may transmit information about the determined encoding quality to the first terminal 810.

The first terminal 810 may encode the screen image of the first terminal 810 based on the information about the encoding quality received from the relay server 820. The first terminal 810 may transmit the encoded screen image to the relay server 820. The relay server 820 based on the identification information of each of the plurality of second terminals 830a, 830b, and 830c received from the signaling server 815 on the screen image received from the first terminal 810, the plurality of first images. 2 terminals 830a, 830b, and 830c, respectively.

9 illustrates a screen sharing between a plurality of terminals 910, 930a, 930b, and 930c by a relay device 920 connected to a plurality of AP devices 915 and 925 by wire in a communication system 900, according to an exemplary embodiment. A diagram for describing a method of controlling the same.

The communication system 900 according to an embodiment may include a plurality of terminals 910, 930a, 930b, and 930c, a plurality of AP devices 915, 925, and a relay device 920.

In the communication system 900 shown in FIG. 9, only components related to the present embodiment are shown. Therefore, it will be understood by those skilled in the art that other general purpose components may be further included in addition to the components illustrated in FIG. 9.

Referring to FIG. 9, in the communication system 900, the relay device 920 may be wired to the first AP device 915 and the second AP device 925.

In the communication system 900 according to an embodiment, IP address information of the first terminal 910 may be preset in the first AP device 915. The first AP device 915 may establish a communication session with the first terminal 910 based on the IP address information of the first terminal 910. As a communication session is established with the first terminal 910, the first AP device 915 may receive a screen image of the first terminal 910 encoded from the first terminal 910. The first AP device 915 may transmit the received screen image to the relay device 920.

The relay device 920 according to an embodiment may transmit the received screen image to each of the plurality of second terminals 930a, 930b, and 930c through the second AP device 925. IP address information of each of the plurality of second terminals 930a, 930b, and 930c may be preset in the second AP device 925. The second AP device 925 receives a plurality of screen images of the first terminal 910 received from the relay device 920 based on the IP address information of each of the plurality of second terminals 930a, 930b, and 930c. It may transmit to the second terminals 930a, 930b, and 930c, respectively.

Meanwhile, the relay device 920 according to an exemplary embodiment may receive performance information regarding screen image transmission of the first terminal 910 from the first terminal 910 through the first AP device 915. In addition, the relay device 920 transmits context information of a network including the plurality of second terminals 930a, 930b, and 930c to the plurality of second terminals 930a, 930b, and the second AP device 925. 930c) may be received from each.

The relay server 920 based on the performance information on the screen image transmission of the first terminal 910 and the network status information including each of the plurality of second terminals 930a, 930b, and 930c, the first terminal 910. ) Can determine the encoding quality of the screen image. The relay server 920 may transmit the information about the determined encoding quality to the first terminal 910 through the first AP device 915.

Meanwhile, in the communication system 900, the relay device 920 may perform wired communication with the first AP device 915 and the second AP device 925, respectively. As the relay device 920 performs wired communication with the first AP device 915 and the second AP device 925, respectively, it is necessary to control screen sharing between the plurality of terminals 910, 930a, 930b, and 930c. Bandwidth usage of the wireless network can be reduced.

FIG. 10 illustrates a method in which a first terminal selects at least one of a plurality of second terminals as a relay device and transmits a screen image of the first terminal through the selected relay device in the communication system according to an embodiment. It is a flow chart.

In operation S1010, the first terminal determines which one of the plurality of second terminals is based on the performance information of the plurality of second terminals receiving the screen image of the first terminal and the network condition information including each of the plurality of second terminals. One can be selected as a relay device.

The first terminal according to an embodiment may transmit the screen image of the first terminal to the plurality of second terminals, respectively. Here, the first terminal may transmit the screen image of the first terminal to each of the plurality of second terminals within a range of a preset transmission bandwidth.

Meanwhile, when the bandwidth required for transmitting the screen image exceeds a preset bandwidth, the first terminal according to an embodiment may select at least one of the plurality of second terminals as a relay device. For example, when the amount of data in the screen image increases or the number of second terminals receiving the screen image of the first terminal increases, the bandwidth required for transmitting the screen image may exceed the preset bandwidth. In this case, the first terminal may transmit the screen image of the first terminal to the other second terminals through the relay device selected from the plurality of second terminals.

The first terminal may select the relay device based on performance information of the plurality of second terminals obtained from the plurality of second terminals and context information of a network including each of the plurality of second terminals.

For example, the first terminal may include at least one of the performance and the network situation corresponding to a preset reference range based on the acquired performance information of the plurality of first terminals and the situation information of the network including each of the second terminals. The second terminal of may be selected as the relay device. Here, the reference range may be set with respect to the RTT, data transmission / reception delay time, data loss rate, frame rate, resolution, processing time of the encoder, and the like.

In addition, according to another example, the first terminal may determine the rank of the plurality of second terminals according to the performance and the situation of the network, and may select some second terminals having a higher rank as the relay device. Here, the first terminal may determine the rank of each of the plurality of second terminals according to the RTT, data transmission / reception delay time, data loss rate, frame rate, resolution, and processing time of the encoder.

On the other hand, the first terminal is the number of relay devices based on the number (N) of the plurality of second terminals, the throughput (TH) of the screen image transmittable from the second terminal, the maximum bit rate (Bmax) of the screen image, etc. (M) can be selected. For example, the first terminal may select the number M of relay devices within a range determined by Equation 1 below.

[Equation 1]

Bmax * N / TH <M <TH / Bmax

In addition, according to another embodiment, the first terminal sets the same number of relay devices connected to the first terminal and the number of second terminals connected to each relay device through the following equation (2). May be determined.

[Equation 2]

N / M = M

Meanwhile, the above-described example is merely an example for selecting the number of relay devices in the first terminal, but the present invention is not limited thereto.

In operation S1020, the first terminal may transmit a message indicating that the relay device is selected to each of the plurality of second terminals. Here, the message indicating that the relay device is selected may include identification information about the selected relay device and mode information indicating that the mode of the screen sharing is switched to the relay mode in which the screen image of the first terminal is transmitted through the relay device. The identification information about the relay device may include information about at least one of an IP address and a port number of the relay device. However, this is only an example, and identification information about the relay device is not limited thereto.

In operation S1030, the first terminal may transmit a screen image of the first terminal to the selected relay device. According to an embodiment, the screen image transmitted from the first terminal may be transmitted to each of the plurality of second terminals through the selected relay device.

The first terminal according to an embodiment may transmit a screen image of the first terminal after a preset time from the time point when each of the plurality of second terminals has received a message indicating that the relay device is selected. For example, the first terminal may transmit a screen image of the first terminal to the relay device after A ms after receiving a response message indicating that the relay device is selected to each of the plurality of second terminals. Can be.

Meanwhile, the first terminal may receive information regarding the encoding quality of the screen image of the first terminal from the selected relay device. The relay device may determine the encoding quality of the screen image of the first terminal based on the performance information about the screen image transmission of the first terminal obtained from the first terminal and the context information of the network including the other second terminals. The method of determining the encoding quality of the screen image of the first terminal in the relay device may correspond to the above description with reference to FIG. 3.

According to an embodiment, when the relay device is selected from among the plurality of second terminals, the first terminal may transmit information about a reproduction time of the screen image to the plurality of second terminals, respectively. Accordingly, when the screen image of the first terminal is transmitted through the relay device in the communication system, the screen image shared between the second terminal selected as the relay device and the other second terminals may be synchronized.

The first terminal according to an embodiment of the present invention is a relay mode for transmitting a screen image to other second terminals through a relay device selected from a plurality of second terminals, respectively, and a direct transmission of the screen images to a plurality of second terminals, respectively. By selecting any one of the modes, the screen image of the first terminal may be shared with the plurality of second terminals. For example, when the number of second terminals receiving the screen image of the first terminal increases beyond the threshold, the first terminal selects a relay mode and the number of second terminals receiving the screen image of the first terminal is selected. When decreasing below the threshold, a direct mode can be selected.

Meanwhile, even after the first terminal selects at least one of the plurality of second terminals as the relay device, the first terminal maintains a communication session with the second terminals that are not selected, thereby selectively using the relay mode and the direct mode. The screen image of the may be shared with a plurality of second terminals.

Also, in the transfer period, in which the relay mode and the direct mode are changed, the screen image of the first terminal is transmitted to each of the relay device and the plurality of second terminals, thereby changing the screen image of the first terminal. Can be prevented from being transmitted.

11 is a flowchart illustrating a method of reselecting a relay device according to a performance of each of a plurality of second terminals and a situation of a network including each of a plurality of second terminals in a communication system according to an exemplary embodiment. to be.

In operation S1110, the first terminal may include one of a plurality of second terminals based on the performance information of the plurality of second terminals receiving the screen image of the first terminal and the context information of the network including each of the plurality of second terminals. Either one can be selected as the relay device.

Meanwhile, step S1110 may correspond to step S1010 described above with reference to FIG. 10.

In operation S1120, the first terminal may transmit a message indicating that the relay device is selected to each of the plurality of second terminals. Here, second terminals not selected as the relay device among the plurality of second terminals may establish a communication session with the relay device through identification information of the relay device included in the message indicating that the relay device is selected.

Meanwhile, step S1120 may correspond to step S1020 described above with reference to FIG. 10.

In operation S1130, the first terminal may transmit a screen image of the first terminal to the selected relay device. The first terminal may transmit the screen image of the encoded first terminal to the selected relay device according to a preset encoding quality.

Meanwhile, the encoding quality of the screen image of the first terminal may be updated based on the information about the encoding quality received from the relay device. The relay device may determine the encoding quality based on network condition information including the second terminal that receives the screen image of the first terminal and performance information regarding transmission of the screen image of the first terminal. In addition, as the network situation including the second terminal is changed, the encoding quality determined by the relay device may be changed.

In operation S1140, the first terminal may determine whether the performance and network conditions of the selected relay device correspond to a preset reference range.

According to an embodiment, the first terminal may receive feedback about the performance of the selected relay device and the situation of a network including the selected relay device. Here, the situation of the network including the relay device may change over time. For example, when the network situation between the second terminals connected to the relay device is increased in congestion or when the relay device is powered off, the network condition may be changed. In addition, even if the relay device is out of network range, the network situation may change. In addition, since the second terminal connected to the relay device is added, even if the bandwidth required for the relay device to transmit the screen image exceeds the bandwidth set in the relay device, the network situation may be changed.

The first terminal may determine whether the performance of the relay device and the situation of the network including the relay device correspond to the preset reference range based on the feedback information. Here, the preset reference range may be determined with respect to the RTT, data transmission / reception delay time, data loss rate, frame rate, resolution, and processing time of the encoder.

In operation S1150, the first terminal may select one of the plurality of second terminals as a new relay device based on the performance information of the plurality of second terminals and the context information of the network including the plurality of second terminals. have.

According to an embodiment, when the performance and network conditions of the selected relay device do not correspond to the preset reference range, the first terminal may select one of the other second terminals as the new relay device. The first terminal may select, as a new relay device, a second terminal whose performance and network conditions among other second terminals correspond to a preset reference range.

In operation S1160, the first terminal may transmit a message indicating that the new relay device is selected to each of the plurality of second terminals.

As a new relay device is selected, the first terminal according to an embodiment may transmit a message including identification information about the new relay device to each of the plurality of second terminals. Here, the identification information regarding the new relay device may include at least one of IP address information and port number of the new relay device.

In operation S1170, the first terminal may transmit a screen image of the first terminal to the selected relay device. The screen image of the first terminal transmitted to the relay device may be transmitted to each of the other second terminals not selected among the plurality of second terminals through the relay device.

12 illustrates a method of transmitting a screen image of a first terminal to other second terminals by a second terminal selected from among a plurality of second terminals that receive the screen image of the first terminal according to an embodiment; It is a flowchart for doing so.

In operation S1210, a second terminal, which is one of a plurality of second terminals that receive the screen image of the first terminal, may receive a message indicating that the terminal is selected as the relay device from the first terminal. Here, the message may include identification information about the relay device. According to another example, the message may include identification information about other second terminals not selected as the relay device among the plurality of second terminals together with the identification information about the relay device. The second terminal selected as the relay device may establish a communication session with the other second terminals, respectively, using identification information about other second terminals included in the message.

In operation S1220, the relay device may transmit the screen image of the first terminal received from the first terminal to the second terminal not selected as the relay device.

According to an embodiment, the second terminal corresponding to the identification information included in the message may be selected as the relay device. The relay device may establish a communication session with other second terminals not selected as the relay device. The relay device may transmit the screen image received from the first terminal to the other second terminals through the established communication session. Here, the screen image of the first terminal may be encoded according to a frame rate and resolution preset in the first terminal.

In operation S1230, the relay device may acquire performance information about the screen image transmission of the first terminal that transmits the screen image and network condition information including at least one second terminal that receives the screen image.

Meanwhile, step S1230 may correspond to step S210 described above with reference to FIG. 2.

In operation S1240, the relay device may determine an encoding quality regarding the screen image of the first terminal based on the performance information and the situation information of the network.

Meanwhile, step S1230 may correspond to step S220 described above with reference to FIG. 2.

In operation S1250, the relay device may receive the screen image encoded by the first terminal based on the determined encoding quality.

Meanwhile, step S1230 may correspond to step S230 described above with reference to FIG. 2.

In operation S1260, the relay device may transmit the received screen image to the second terminal not selected as the relay device.

Meanwhile, step S1230 may correspond to step S240 described above with reference to FIG. 2.

FIG. 13 is a flowchart for describing a method of sharing a screen image of a first terminal among a plurality of second terminals when a relay device is changed according to an embodiment.

In operation S1310, a second terminal, which is one of a plurality of second terminals that receive the screen image of the first terminal, may receive a message indicating that the terminal is selected as the relay device from the first terminal.

Meanwhile, step S1310 may correspond to step S1210 described above with reference to FIG. 12.

In operation S1320, the relay device may transmit a screen image of the first terminal received from the first terminal to a second terminal not selected as the relay device among the plurality of second terminals.

Meanwhile, step S1320 may correspond to step S1220 described above with reference to FIG. 12.

In operation S1330, the relay device may receive a message indicating that a new relay device is selected from the first terminal.

When the performance of the relay device and the situation of the network including the relay device according to an embodiment do not correspond to the preset reference range, the first terminal is different from the second terminal selected as the relay device among the plurality of second terminals. 2 The terminal can be selected as a new relay device. As the new relay device is selected, the relay device may receive a message indicating that the new relay device is selected from the first terminal. Here, the message indicating that the new relay device is selected may be transmitted to each of the plurality of second terminals that receive the screen image from the first terminal. In addition, the message indicating that the new relay device is selected may include identification information of the new relay device.

In operation S1340, the relay device may receive the screen image of the first terminal from the new relay device.

The relay device according to an embodiment may terminate the role as a relay device as a new relay device is selected. The relay device is a second terminal that receives the screen image of the first terminal, and may receive the screen image of the first terminal through the new relay device.

14 is a block diagram of a relay device 1400 according to an embodiment.

In the relay device 1400 shown in FIG. 14, only components related to the present embodiment are shown. As shown in FIG. 14, the relay device 1400 according to an embodiment may include a communication unit 1410 and a controller 1420. However, not all illustrated components are essential components. The relay device 1400 may be implemented by more components than the illustrated components, and the relay device 1400 may be implemented by fewer components. For example, the relay device 1400 may further include a memory (not shown).

The communication unit 1410 may include a module for performing communication between the relay device 1400 and the first terminal or the relay device 1400 and at least one second terminal. For example, the communication unit 1410 may include a short range communication unit (not shown) and a mobile communication unit (not shown).

The short-range wireless communication unit includes a Bluetooth communication unit, a Bluetooth low energy (BLE) communication unit, a near field communication unit (Near Field Communication unit), a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared ray (IrDA) It may include, but is not limited to, a Data Association (W Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an ultra wideband (UWB) communication unit, an Ant + communication unit, and the like.

The mobile communication unit transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call call signal, or a text / multimedia message.

The communication unit 1410 obtains the performance information of the screen image transmission of the first terminal that transmits the screen image and the context information of the network including the at least one second terminal that receives the screen image.

In addition, the communication unit 1410 receives the screen image of the encoded first terminal from the first terminal according to the encoding quality determined based on the performance information and the situation information of the network. The communication unit 1410 transmits the received screen image to at least one second terminal.

The communication unit 1410 according to an embodiment may transmit information about the determined encoding quality to the first terminal. The communication unit 1410 may obtain identification information of at least one second terminal from the first terminal. Accordingly, a communication session may be established between the relay device and the at least one second terminal. According to another example, the communication unit 1410 may obtain identification information of the first terminal and the at least one second terminal from the server.

The controller 1420 may determine the encoding quality of the screen image of the first terminal based on the performance information and the situation information of the network. The controller 1420 according to an exemplary embodiment may determine a bit rate for transmitting the screen image to at least one second terminal based on the situation information of the network. In addition, the controller 1420 may determine an encoding quality including at least one of a resolution and a frame rate, based on the determined bit rate.

According to an embodiment, the controller 1420 may establish a communication session with at least one second terminal based on the identification information of the at least one second terminal obtained by the communication unit 1410.

According to another embodiment, the relay device 1400 is based on the performance information of each of the plurality of second terminals receiving the screen image of the first terminal and the situation information of the network including each of the plurality of second terminals. It may be selected in the first terminal. In this case, the communication unit 1410 may receive a message indicating that the relay device is selected.

In addition, when the performance of the relay device and the situation of the network including the relay device deviate from the preset reference range, the communication unit 1410 corresponds to the performance of the plurality of second terminals and the situation of the network corresponding to the preset reference range. A message indicating that the second terminal is selected as the new relay device may be received from the first terminal. Accordingly, the communication unit 1410 may receive a screen image encoded by the first terminal from the selected second terminal according to the encoding quality determined by the selected second terminal.

The memory (not shown) may store a program for processing and controlling the controller 1420, and input / output data (eg, identification information of at least one second terminal, identification information of the first terminal, and the like). You can also save it.

The memory (not shown) may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), Random Access Memory (RAM) Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic It may include a storage medium of at least one type of disk, optical disk. In addition, the relay device 1400 may operate a web storage or a cloud server that performs a storage function of a memory (not shown) on the Internet.

15 and 16 are block diagrams of a first terminal 1500 according to an exemplary embodiment.

In the first terminal 1500 illustrated in FIG. 15, only components related to the present exemplary embodiment are illustrated. As shown in FIG. 15, the first terminal 1500 according to an embodiment may include a communication unit 1510, a controller 1520, and an encoder 1530. However, not all illustrated components are essential components. The first terminal 1500 may be implemented by more components than the illustrated components, and the first terminal 1500 may be implemented by fewer components. For example, as illustrated in FIG. 16, the first terminal 1500 may further include an output unit 1540 and a memory 1550 in addition to the communication unit 1510, the control unit 1520, and the encoding unit 1530. have.

The communication unit 1510 may include a module for performing communication between the first terminal 1500 and the relay device or the first terminal 1500 and the at least one second terminal. For example, the communicator 1510 may include a short range communication unit (not shown) and a mobile communication unit (not shown).

The short-range wireless communication unit includes a Bluetooth communication unit, a Bluetooth low energy (BLE) communication unit, a near field communication unit (Near Field Communication unit), a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared ray (IrDA) It may include, but is not limited to, a Data Association (W Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an ultra wideband (UWB) communication unit, an Ant + communication unit, and the like.

The mobile communication unit transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call call signal, or a text / multimedia message.

The communication unit 1510 may transmit, from the relay apparatus transmitting the screen image of the first terminal to the at least one second terminal, performance information regarding the transmission of the screen image of the first terminal and the situation information of the network including the at least one second terminal. Obtains information about an encoding quality of a screen image of the first terminal determined based on the; The communication unit 1510 transmits the encoded screen image to the relay device. Here, the encoded screen image is transmitted to the second terminal via the relay device.

In addition, as the controller 1520 selects the relay device, the communication unit 1510 may transmit a message indicating the selection of the relay device to each of the plurality of second terminals.

The controller 1520 may determine the encoding quality of the screen image of the first terminal based on the obtained information about the encoding quality of the screen image of the first terminal. According to an embodiment of the present disclosure, the controller 1520 is configured to perform a plurality of operations based on the performance information of each of the plurality of second terminals that receive the screen image of the first terminal and the situation information of the network including the plurality of second terminals. At least one of the second terminals may be selected as the relay device.

If the performance of the relay device and the situation of the network including the relay device deviate from the preset reference range, the controller 1520 according to an embodiment of the present disclosure may have a performance and a network condition among the plurality of second terminals in the preset reference range. The other corresponding second terminal may be reselected as the relay device.

The encoder 1530 encodes the screen image of the first terminal based on the obtained information about the encoding quality. The encoder 1530 according to an embodiment may encode the screen image of the first terminal according to the frame rate and the resolution determined by the controller 1520 based on the information about the encoding quality. In addition, the encoder 1530 may generate an FEC packet for the screen image of the encoded first terminal.

The output unit 1540 may output content executed in the first terminal 1500. An image displayed on the output unit 1540 may be captured by the controller 1520 to generate a screen image of the first terminal 1500.

However, this is only an example, and the output unit 1540 may be included in a device external to the first terminal 1500. In this case, the external device may capture the image displayed on the output unit 1540 and provide it to the first terminal 1500 according to a capture command of the controller 1520.

The memory 1550 may store a program for processing and controlling the controller 1520, and input / output data (eg, identification information of the relay device, identification information of the first terminal, and screen image of the first terminal). Information about the encoding quality of the data).

The memory 1550 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), RAM Random Access Memory (RAM) Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic Disk It may include at least one type of storage medium of the optical disk. In addition, the first terminal 1500 may operate a web storage or a cloud server that performs a storage function of the memory 1550 on the Internet.

17 and 18 are block diagrams of a second terminal 1700 according to an embodiment.

In the second terminal 1700 illustrated in FIG. 17, only components related to the present exemplary embodiment are illustrated. As illustrated in FIG. 17, the second terminal 1700 according to an embodiment may include a communication unit 1710 and a controller 1720. However, not all illustrated components are essential components. The second terminal 1700 may be implemented by more components than the illustrated components, and the second terminal 1700 may be implemented by fewer components. For example, as illustrated in FIG. 17, the second terminal 1700 may further include an output 1730 and a memory 1740 in addition to the communication unit 1710 and the controller 1720.

The communication unit 1710 may include a module for performing communication between the second terminal 1700 and the relay device or the second terminal 1700 and the first terminal. For example, the communication unit 1710 may include a short range communication unit (not shown) and a mobile communication unit (not shown).

The short-range wireless communication unit includes a Bluetooth communication unit, a Bluetooth low energy (BLE) communication unit, a near field communication unit (Near Field Communication unit), a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared ray (IrDA) It may include, but is not limited to, a Data Association (W Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an ultra wideband (UWB) communication unit, an Ant + communication unit, and the like.

The mobile communication unit transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call call signal, or a text / multimedia message.

The communication unit 1710 transmits information regarding a situation of a network including the second terminal 1700 to the relay device. In addition, the communication unit 1710 receives a screen image encoded by the first terminal from the relay device according to the encoding quality determined by the relay device.

In addition, the communication unit 1710 according to an embodiment may receive a message informing the selection of the relay device as the relay device selects the relay device.

Meanwhile, when the second terminal 1700 is selected as the relay device, the communication unit 1710 may transmit the screen image received from the first terminal to other second terminals, respectively.

The controller 1720 may obtain information about a situation of a network including the second terminal 1700. For example, when receiving a screen image from a relay device, the controller 1720 may obtain information about an RTT, a data loss rate, and a data delay rate.

When the controller 1720 receives a message indicating the selection of the relay device from the first terminal, the identification information regarding the relay device included in the received message corresponds to the identification information of the second terminal 1700. Can be determined. The controller 1720 may operate as the relay device when the identification information about the relay device and the identification information of the second terminal 1700 correspond to each other. For example, the controller 1720 may control the communicator 1710 to establish a communication session with other second terminals by using identification information of other second terminals included in a message informing the selection of the relay device. .

The output unit 1730 may display a screen image of the first terminal received from the relay device. However, this is only an example, and the output unit 1540 may be included in a device external to the second terminal 1700.

The memory 1740 may store a program for processing and controlling the controller 1720, and input / output data (eg, identification information of a relay device and status information of a network including the second terminal 1700). Etc.).

The memory 1740 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), RAM Random Access Memory (RAM) Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic Disk It may include at least one type of storage medium of the optical disk. In addition, the second terminal 1700 may operate a web storage or a cloud server that performs a storage function of the memory 1740 on the Internet.

Method according to an embodiment of the present invention is implemented in the form of program instructions that can be executed by various computer means may be recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Claims (15)

1. In the method for the relay device to control the screen sharing between a plurality of terminals,

   Acquiring performance information regarding screen image transmission of a first terminal transmitting a screen image and situation information of a network including at least one second terminal receiving the screen image;

   Determining an encoding quality of a screen image of the first terminal based on the performance information and the situation information of the network;

   Receiving an encoded screen image at the first terminal based on the determined encoding quality; And

   And transmitting the received screen image to the at least one second terminal.
2. The method of claim 1,

   And transmitting the information on the determined encoding quality to the first terminal.
3. The method of claim 1, wherein determining the encoding quality of the screen image comprises:

   Determining a bit rate for transmitting the screen image to the at least one second terminal based on the situation information of the network;

   Based on the determined bitrate, determining the encoding quality comprising at least one of a resolution and a frame rate.
4. The method of claim 1, wherein the relay device,

   A plurality of terminals selected from the first terminal based on performance information of each of a plurality of second terminals receiving the screen image of the first terminal and context information of a network including each of the plurality of second terminals; How to control screen sharing between people.
5. In the method for the first terminal to control the screen sharing between a plurality of terminals,

   From the relay device for transmitting the screen image of the first terminal to at least one second terminal, based on the performance information on the transmission of the screen image of the first terminal and the status information of the network including the at least one second terminal Obtaining information regarding an encoding quality of the determined screen image of the first terminal;

   Encoding a screen image of the first terminal based on the obtained information about the encoding quality; And

   Transmitting the encoded screen image to the relay device,

   And the encoded screen image controls screen sharing between a plurality of terminals transmitted to the second terminal through the relay device.
6. The method of claim 5,

   At least one of the plurality of second terminals based on performance information of each of the plurality of second terminals receiving the screen image of the first terminal and context information of the network including the plurality of second terminals, respectively. Selecting as the relay device; And

   And transmitting a message informing of selection of the relay device to each of the plurality of second terminals.
7. In the method of the second terminal to control the screen sharing between a plurality of terminals,

   Transmitting, by the second terminal receiving the screen image of the first terminal, information regarding the situation of the network including the second terminal to the relay device; And

   Receiving the screen image encoded at the first terminal from the relay device according to the encoding quality determined at the relay device,

   The encoding quality is a situation of a network including performance information on the screen image transmission of the first terminal obtained by the relay device and at least one second terminal receiving the screen image of the first terminal through the relay device. A method of controlling screen sharing among a plurality of terminals, the information being determined based on the information.
8. A communication unit for acquiring performance information on screen image transmission of a first terminal transmitting a screen image and situation information of a network including at least one second terminal receiving the screen image; And

   And a controller configured to determine an encoding quality of the screen image of the first terminal based on the performance information and the situation information of the network.

   The communication unit,

   And a screen device between the plurality of terminals that receives the screen image encoded by the first terminal based on the determined encoding quality and transmits the received screen image to the at least one second terminal.
9. The method of claim 8, wherein the communication unit,

   Relay device for controlling the screen sharing between a plurality of terminals for transmitting the information on the determined encoding quality to the first terminal.
10. The method of claim 8, wherein the control unit,

    Determine a bitrate for transmitting the screen image to the at least one second terminal based on the context information of the network, and based on the determined bitrate, the encoding comprising at least one of a resolution and a frame rate Relay device for controlling the screen sharing between a plurality of terminals for determining the quality.
11. The method of claim 8, wherein the relay device,

    A plurality of terminals selected from the first terminal based on performance information of each of the plurality of second terminals receiving the screen image of the first terminal and context information of a network including the plurality of second terminals; Relay device that controls screen sharing between them.
12. From the relay device for transmitting the screen image of the first terminal to at least one second terminal, based on the performance information on the transmission of the screen image of the first terminal and the status information of the network including the at least one second terminal A communication unit for obtaining information regarding an encoding quality of the determined screen image of the first terminal;

    A controller configured to determine an encoding quality of a screen image of the first terminal based on the obtained encoding quality information; And

    An encoding unit for encoding the screen image of the first terminal according to the determined encoding quality;

    The communication unit,

    Send the encoded screen image to the relay device,

    The encoded terminal image is transmitted to the second terminal through the relay device, the first terminal to control the screen sharing between a plurality of terminals.
13. The method of claim 12, wherein the control unit,

    At least one of the plurality of second terminals based on performance information of each of the plurality of second terminals receiving the screen image of the first terminal and context information of the network including the plurality of second terminals, respectively. Is selected as the relay device,

    The communication unit,

    A first terminal for controlling screen sharing among a plurality of terminals, sending a message informing the selection of the relay device to each of the plurality of second terminals.
14. A control unit for obtaining, by the second terminal receiving the screen image of the first terminal, context information of a network including the second terminal; And

    A communication unit which transmits the obtained situation information of the network to a relay device and receives a screen image encoded by the first terminal from the relay device according to an encoding quality determined by the relay device,

    The encoding quality is a situation of a network including performance information on the screen image transmission of the first terminal obtained by the relay device and at least one second terminal receiving the screen image of the first terminal through the relay device. A second terminal for controlling screen sharing among a plurality of terminals, determined based on the information.
15. A computer-readable recording medium having recorded thereon a program for performing the method of claim 1.

Images