HK1181936B - Controlling public displays with private devices - Google Patents

Description

Controlling public displays with private devices

Technical Field

This application relates to controlling public displays, and more particularly to controlling public displays with private devices.

Background

Electronic displays are becoming increasingly common in public areas. Such a display may display content such as text, images, video, or graphics, and the content may change over time (e.g., in different advertisement sequences). Sometimes, the electronic display of the public area is connected to a computer at a kiosk at a fixed location. A user at the kiosk can input information into the computer and the display can change in response to the input information. However, such kiosks typically can accommodate only one user at a time, so the number of users that can affect the output of the display is limited. Moreover, such displays typically do not accept input from other computing devices. In some public environments, such as a stadium, the display can display SMS messages sent by the mobile phone to a particular phone number (e.g., a phone number controlled by the public display). However, the content of SMS messages is limited to plain text, and waiting for an SMS message to be delivered to a destination phone number before being displayed or may result in significant latency (e.g., delays of seconds, minutes, or more) while waiting for the content of the message to be interpreted and filtered (e.g., by a human moderator).

With the increasing popularity and sophistication of mobile computing devices (e.g., tablet computers, smart phones), there is a need for enhanced, real-time interaction with public displays.

Disclosure of Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

In various examples herein, techniques and tools are described for using a client device to control a public display device over a network (e.g., the internet). In some examples, the time-limited proximity code is displayed by a public display device, and the client device may provide the code over a public network to verify that the client device is present within a proximity zone (e.g., a visual proximity zone) and allowed to control the public display. Once authorized, the client device may provide control data to control the visual content on the public display. A relay service may be used to relay data between the client device and the public display device over the network through the communication connection. For example, a relay service may be implemented on a server connected to a network and may use HTTP (hypertext transfer protocol) to relay data (such as control data) from one device to another through messages. The relay service may be useful, for example, where a security protocol restricts communication with a public display device or a client device (e.g., restricts direct access to the public display device or restricts incoming communications on the client device).

The foregoing and other objects, features and advantages of the invention will become more apparent from the following detailed description, which proceeds with reference to the accompanying figures.

Drawings

FIG. 1 illustrates a general example of a mobile computing device in which one or more of the described embodiments may be implemented.

FIG. 2 is a block diagram showing a generalized system in which one or more of the described embodiments may be implemented.

Fig. 3, 4, 5 and 6 are diagrams showing exemplary arrangements in which a common display device according to one or more described embodiments may be controlled by one or more mobile devices.

FIG. 7 is a flow diagram showing an exemplary technique for obtaining authorization for a client device to control content displayed on a common display device in accordance with one or more described embodiments.

FIG. 8 is a flow diagram illustrating an exemplary technique for displaying visual content based on control data received over a public network from client devices located within a proximity zone in accordance with one or more described embodiments.

Fig. 9 is a flow diagram illustrating an example technique for providing relay services to facilitate communication between a public display device and a client device controlling the public display device in accordance with one or more described embodiments.

FIG. 10 illustrates a generalized example of a suitable computing environment in which one or more of the described embodiments may be implemented.

Detailed Description

Techniques and tools are described for using a client device (e.g., a private computing device such as a mobile device) to control a public display device (or a public output device with other types of controllable output) over a network (e.g., the internet). The described techniques and tools use proximity and/or authentication information to manage users that are licensed to control a common display. Security may be enhanced by limiting access by distance and/or anonymous users. In some examples, a proximity code (e.g., a visual proximity code such as a text string or an image such as a barcode) is displayed on the public display, and the user may provide the code (e.g., by typing the code on a keyboard, capturing an image of the code, or by some other method appropriate to the code) to verify that the user is present within the proximity region and allowed to control the public display. A relay service may be used to relay data between the client device and the public display device over the network through the communication connection. For example, a relay service may be implemented on a server connected to a network and may use HTTP (hypertext transfer protocol) to relay data from one device to another through messages. The described techniques and tools may allow a user to control a common display in a synchronized or near-synchronized manner, allowing for direct control and interaction with relatively low latency.

The described techniques and tools may be used by common output devices in a variety of environments (e.g., large electronic billboards in times squares, scoreboards in stadiums, interactive artwork in galleries, museums or large building lobbies, billboards, or devices provided by vendors for ordering goods in retail stores). The described techniques and tools may be used in various interactive contexts. For example, a user may cause a public display to display the user's name or image, or perform more complex actions, such as playing a game or controlling an avatar associated with the user (e.g., waving a hand, jumping, displaying a symbol, making a gesture such as sign language). As another example, a user within a restaurant may cause a public display (e.g., in text, images, and/or graphics) to display the user's order. As another example, a user within a retail store may request a demonstration of a product on a public display or configure the product with options selected by the user. As another example, a user within a stadium may cause images of favorite players or artists to appear on a display at the stadium. Such actions may be personalized. For example, an avatar, restaurant order, or image of a favorite athlete may be displayed along with the user's personal information (e.g., image, name, or user ID) if the user desires. Multiple users may interact with the same public display (e.g., controlling multiple avatars), or users may interact with each other on a public display (e.g., playing games such as tennis or ping-pong). The number of users may be unlimited, but based on factors such as screen size, system resources (e.g., memory, communication bandwidth), and/or content being controlled (e.g., games with a limited number of players), a limit on the number of users (e.g., the first 100 users in an avatar display scenario, the first two users in a tennis game scenario) may be desirable.

The described techniques and tools may be used by any number of displays of any size or type (e.g., a matrix of several small displays configured to display content as a single large virtual display, individual displays located at different locations (e.g., on different sides of a room) linked together and displaying related content). The described techniques and tools may be used to control other types of output in addition to visual output. For example, a user may provide input that results in an audio output (such as synthesized speech or other sounds).

The described techniques and tools provide benefits over other approaches for affecting public displays. For example, kiosks connected to a public display allow users to interact with the display but do not necessarily limit the number of users that can interact with them (typically one user per kiosk). The described techniques and tools are not limited by input hardware (such as a kiosk) that can be connected to a public display. As another example, while SMS messages sent by users over mobile phones may be used to affect public displays, the free-form content of SMS messages typically requires human intervention to approve messages before they are displayed, thereby increasing latency even beyond the normal latency present in the delivery of SMS messages. The described techniques and tools allow for a discrete set of commands that can be used to avoid inappropriate interaction with public displays and reduce latency (e.g., latency introduced by a human filtering process). While the user may be instructed to send only a particular SMS message (e.g., "1" to perform the first action and "2" to perform the second action), this may become difficult for the user. The described techniques and tools provide a more intuitive, richer interface to control public displays. SMS messages are typically sent to publicly available telephone numbers, allowing the user to send the message from a remote location. The described techniques and tools may be used to verify the presence of a user within a proximity zone and limit access by distance and/or unauthorized users.

The examples described herein should not be considered as limiting in any way. Rather, the invention is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, alone and in various combinations and subcombinations with one another. The disclosed systems, methods, and apparatus are not limited to any specific aspect or feature or combination thereof, nor do the disclosed embodiments require that any one or more specific advantages be present or problems be solved.

Although some of the operations of the disclosed methods are described in a particular, sequential order for convenient presentation, it should be understood that this method of description encompasses rearrangement, unless a particular ordering is required by specific language set forth below. For example, in some cases, operations described sequentially may be rearranged, omitted, repeated, or performed concurrently. As another example, the system described with reference to the system diagram may be changed by changing the ordering of the elements or processing stages shown in the figure, by repeating or omitting certain elements or processing stages, and so on. Moreover, for the sake of brevity, the attached figures do not show the various ways in which the disclosed things and methods can be used in conjunction with other things and methods. In addition, this specification sometimes uses terms like "determine," "transmit," and "receive" to describe the disclosed methods. These terms are high-level abstractions of the actual operations that are performed. The actual operations that correspond to these terms may vary depending on the particular implementation and are readily discernible by one of ordinary skill in the art.

Any of the disclosed methods may be implemented as computer-executable instructions stored on one or more computer-readable media (e.g., non-transitory computer-readable media, such as one or more volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as hard drives)) and executed on a computer (e.g., any commercially available computer, including smart phones or other mobile devices that include computing hardware). Any of the computer-executable instructions for implementing the disclosed techniques, as well as any data created and used during implementation of the disclosed embodiments, can be stored on one or more computer-readable media (e.g., non-transitory computer-readable media). The computer-executable instructions may be part of a dedicated software application or a software application that is accessed or downloaded, for example, via a web browser or other software application, such as a remote computing application. For example, such software can be executed on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such network) using one or more network computers.

For clarity, only certain selected aspects of software-based implementations are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language, or program. The disclosed techniques may be implemented by software written in C + +, C #, Java, Perl, JavaScript, HTML5, or any other suitable programming language. Also, the disclosed techniques are not limited to any particular computer or type of hardware. Certain details of suitable computers and hardware are well known and need not be set forth in detail in this disclosure.

Further, any of the software-based embodiments (including, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) may be uploaded, downloaded, or remotely accessed through suitable communication means. Such suitable communication means include, for example, the internet, the world wide web, an intranet, software applications, electrical cables (including fiber optic cables), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communication means.

I.Exemplary Mobile device

Fig. 1 is a system diagram depicting an exemplary mobile device 100 including various optional hardware and software components, shown generally at 102. Any component 102 in the mobile device may communicate with any other component, but not all connections are shown for ease of illustration. The mobile device can be any of a variety of computing devices (e.g., a cellular telephone, a smart phone, a handheld computer, a tablet computer, etc.) and can allow wireless two-way communication with one or more mobile communication networks 104, such as a cellular or satellite network.

The illustrated mobile device 100 may include a controller or processor 110 (e.g., signal processor, microprocessor, ASIC, or other control and processing logic circuitry) for performing tasks such as signal coding, data processing, input/output processing, power control, and/or other functions. The operating system 112 can control the allocation and usage of the components 102 in a variety of ways, and provide support for one or more application programs 114. The application programs may include public mobile computing applications (e.g., image capture applications, email applications, calendars, contact managers, web browsers, messaging applications), or any other computing application.

The illustrated mobile device 100 may include memory 120. Memory 120 may include non-removable memory 122 and/or removable memory 124. The non-removable memory 122 may include RAM, ROM, flash memory, a hard disk, or other well-known memory storage technologies. The removable memory 124 may include flash memory or a Subscriber Identity Module (SIM) card, which is well known in GSM communication systems, or other well known memory storage technologies, such as smart cards. The memory 120 may be used to store data and/or code for running the operating system 112 and the application programs 114. Example data may include web pages, text, images, sound files, video data, or other data sets sent to and/or received from one or more network servers or other devices via one or more wired or wireless networks. The memory 120 can be used to store a subscriber identifier, such as an International Mobile Subscriber Identity (IMSI), and an equipment identifier, such as an International Mobile Equipment Identifier (IMEI). These identifiers may be transmitted to a network server to identify the user and equipment.

The mobile device 100 may support one or more input devices 130, such as a touch screen 132, microphone 134, camera 136, physical keyboard 138, trackball 140, and/or proximity sensor 142, and one or more output devices 150, such as a speaker 152 and one or more displays 154. Other possible output devices (not shown) may include piezoelectric or tactile output devices. Some devices may provide more than one input/output function. For example, the touch screen 132 and the display 154 may be combined in a single input/output device.

The wireless modem 160 may be coupled to an antenna (not shown) and may support bi-directional communication between the processor 110 and external devices, as is well understood in the art. The modem 160 is shown generically and may include a cellular modem and/or other radio-based modem (e.g., bluetooth 164 or Wi-Fi 162) for communicating with the mobile communications network 104. The wireless modem 160 is typically configured for communication with one or more cellular networks, such as a GSM network for data and voice communication within a single cellular network, between cellular networks, or between a mobile device and a Public Switched Telephone Network (PSTN).

The mobile device may further include at least one input/output port 180, a power supply 182, a satellite navigation system receiver 184, such as a Global Positioning System (GPS) receiver, an accelerometer 186, a gyroscope (not shown), and/or a physical connector 190, which may be a USB port, an IEEE1394 (firewire) port, and/or an RS-232 port. The illustrated components 102 are not required or all-inclusive, as any components can be deleted and other components can be added.

II.Generalized system

Fig. 2 is a block diagram of an exemplary system 200 in which techniques described herein may be implemented. In this example, a client device 210 (e.g., a mobile device such as a smartphone or tablet computer) communicates with a public output device 220 (e.g., a display device in a public environment such as a building lobby or city street) over a network 230 (e.g., a publicly accessible network such as the internet). As used herein, the term "common output device" refers to a common device that provides an output that is discernable by a user, such as a visual output, an audio output, a combination of visual and audio outputs, or some other type of output. As used herein, the term "public" (e.g., a public display device) as applied to a device is used to describe a device that can be controlled by more than one user and by more than one client device, but does not require that such a device be accessible by all possible users or be located at a location accessible by the public. For example, the public display device may be located in a private residence or company and may be accessible only by residents or employees, respectively.

Client device 210 may be a general purpose computing device (e.g., a smartphone or tablet computer) or a special purpose device (e.g., a device specifically designed to interact with public output device 220). The client device 210 need not be owned by a user who wishes to control the common output device 220. For example, client device 210 may be lent to a user for the purpose of controlling common output device 220. The borrowed device may be used in an environment such as a museum where users viewing a particular exhibit using a common display device may verify proximity (e.g., by verifying their presence in the room containing the exhibit) and then control the common output device. The common output device 220 receives control data (e.g., display control data from the client device 210) and performs output operations (e.g., displaying content) in response to the control data. The recipient module 222 may interpret the control data at the common output device 220. For example, the recipient module 222 receives and interprets control data from the client device 210 to determine output operations to be performed by the common output device 220 (e.g., display an avatar, display user actions in a game).

In any of the examples described, a relay service may be used to relay data between client device 210 and common output device 220 over network 230 through a communication connection. For example, the relay service may be implemented on a server (not shown) connected to the network 230. The relay service may use simple HTTP (hypertext transfer protocol) to relay data from one device to another. The relay service provides flexibility for communication between devices in certain scenarios, such as when security considerations prevent direct communication between client device 210 and common output device 220. For example, mobile phone operators sometimes require mobile phones to use NAT (network address translation) and/or firewalls to restrict inbound connections. A common output device may have similar limitations for inbound connections but be more tolerant of outbound connections that may involve a lesser degree of security risk.

Some examples described herein use messages sent over HTTP to establish a connection between a sender (e.g., client device 210) and a recipient (e.g., public output device 220) and send data using such a connection. For example, recipient module 222 of common output device 220 may generate messages to open a communication session, close a communication session, and receive control data from client device 210. The common output device control module 212 of the client device 210 may generate a message to register with the open communication session and send control data to the common output device 220. The common output device control module 212 and the recipient module 222 may be implemented in different manners (e.g., in hardware or software). For example, common output device control module 212 may be implemented as a custom control application (not shown) on client device 210, or a user may navigate to a web page (not shown) associated with control of common output device 220. A custom control application or web page may provide computer program code to serve as the common output device control module 212.

In some scenarios, it may be desirable to limit access to the common output device to users within a local area (e.g., to avoid damage by intruders at remote locations). As used herein, the term "proximity" is used to describe a measure of proximity (e.g., physical distance) between devices (e.g., client device 210 and common output device 220). In the example shown in fig. 2, client device 210 is within a proximity zone 290 that also includes common output device 220. The presence of client device 210 within proximity zone 290 indicates that the client device is considered to be sufficiently recent to control common output device 220.

In any of the embodiments described herein, proximity may be based on distance from a common output device, context (e.g., the client device and the common output device are co-located within a locality), or some combination of distance and context. For example, a user holding a mobile phone within a stadium that is located 150 meters from the public display on the opposite side of the stadium may be considered to be within the proximity zone, while a user outside the stadium but only 100 meters from the public display may be considered to be outside the proximity zone. Proximity can be measured explicitly. For example, the location of the client device may be calculated by GPS or network-based positioning. The location of the client device may also be approximate. For example, a communication technology (e.g., bluetooth) with a limited range (e.g., 50 meters) may be used to determine whether the client device is within range of the common output device, even if the exact location of the client device is not known. As another example, if the client device joins a local wireless network to which the common output device is connected, the system may imply that the client device is within a proximity region that encompasses the range of the local wireless network. The user of the client device may be given notification of any collection of location data and an opportunity to provide or deny consent to the collection of location data. Consent may be given in the form of opt-out consent, where the user may take an affirmative action to prevent collection of location data until it is collected, or opt-in consent, where the user may take an affirmative action to give consent to collection of location data until it is collected.

Although useful, many of the proximity metrics described above also have drawbacks. For example, the GPS coordinates may be spoofed by an intruder who may attempt to gain unauthorized control of the common output device or by a legitimate user who may attempt to circumvent the proximity restrictions. As another example, requiring bluetooth or local network connectivity to verify proximity may be inconvenient for some users. Thus, in any of the described examples, other proximity metrics (e.g., visual proximity, audio proximity, or some other proximity metric) may be used. For example, a proximity code displayed at a common output device (e.g., a display device) may be used to determine visual proximity. In some described examples, by displaying the proximity code on a common display device, if a user is able to provide the code, the system may suggest that the user is within a visual proximity area, even if the exact location of the client device is not known. The size and shape of the visual proximity region may vary based on factors such as the size of the code being displayed or the size of the display itself or the viewing angle range. As another example, a tone or other audio signal (e.g., audible or non-audible by a human ear) may be used to determine audio proximity. In some described examples, tones transmitted by a public display device or client device may be received by other devices and used to verify audio proximity. The size and shape of the audio proximity region may vary based on factors such as the intensity of the audio signal or the sound effect of a particular location. Alternatively, other signals or protocols may be used to verify proximity.

Different proximity metrics may be used to verify different levels of proximity (e.g., different ranges or directions of distance from a common output device). For example, the public display device may display codes that are visible by users within a particular location (e.g., a stadium) and/or within a limited viewing angle. As another example, the public display device may transmit tones that may be detected by client devices inside or outside of a particular venue and/or in any direction with the public display device. Different proximity metrics may be used alone or in combination with one another.

A user interacting with the common output device may send commands to control the common output device. These commands may come from a limited set of commands (e.g., commands for controlling particular visual content on a display device). A limited set of commands may be useful, for example, to keep content suitable for public viewing, although other data (e.g., free-form text or images) may also be sent to a common output device.

Exemplary client devices, common output devices, proximity metrics, user interfaces, controllable content, communication protocols, and control scenarios are described in detail below.

In actual practice, a system such as system 200 described herein may be more complex, have additional functionality, have more complex relationships between components of the system, and so forth. The techniques described herein may be generally applicable to the details of operating systems or hardware, and may be applied in various environments to take advantage of the features.

III.Exemplary arrangement

Fig. 3 is a diagram of an exemplary arrangement 300 in which techniques described herein may be implemented. In this example, a mobile device 310 (e.g., mobile phone, tablet computer) controlled by a user 312 is authorized to control a public display device 320 (e.g., by controlling an avatar 380) over a network 330. The common display device 320 includes a controllable display area 370 and also displays instructions 350 and proximity codes 360. As shown, the instructions 350 direct the user to enter a proximity code 360 to control an avatar 380 within a controllable display area 370. Alternatively, the instructions 350 are different from those shown in fig. 3 or omitted. The mobile device 310 includes a display 314 (e.g., a touch screen display) and a user interface 316 on which a proximity code 360 is displayed as entered by the user 312.

In any of the examples described herein, authorization of the user and/or client device to control the disclosure output device may be determined from the confirmation of the proximity code. Confirmation of the proximity code may involve, for example, comparing the proximity code transmitted by the client device with one or more legitimate codes (e.g., a single legitimate code, a set of legitimate proximity codes in a database). The legitimate code may be generated and the transmitted code confirmed by a public display device or some other device, such as a server. The legitimate code may be stored, for example, in memory or other storage of the public display device or some other device, such as a server. When the transmitted proximity code is confirmed, the client device may receive authorization data. The authorization data may include, for example, an authorization code or identifier (such as a session ID) for the communication session with the common display device. The same proximity code may be used by more than one user. The proximity code may be used in combination with other information to verify proximity. For example, to verify the user's presence in the stadium, the user may provide a proximity code along with, for example, GPS location information, a seat number, or an additional code on a printed ticket.

In any of the examples described herein, the proximity code may be a time-limited proximity code that is valid for a limited period of time. Over time, different proximity codes for different time periods may be displayed. Such a code may be referred to as a rotation code. Confirmation of the time-limited proximity code may show that the user has not used an expired code (e.g., a code that the user used on a previous day). The user's current presence may be implied from the confirmation of the code.

In any of the examples described herein, the proximity code may include visual information, such as text, graphics (e.g., a linear barcode or a two-dimensional barcode such as a QR code), an image, or any other suitable visual information. The proximity code may also include audio information, such as a single tone or a group of tones, or any other suitable audio information. In any of the examples described herein, the user may enter the proximity code by typing on a touchscreen or keyboard, by speaking the code into a microphone, by capturing an image of the code using a camera, or by some other user input mechanism. The type of input used may vary based on, for example, the input devices available, user preferences, and the type of code. For example, a camera device may be used to capture an image of a two-dimensional barcode. Alternatively, the proximity code may be obtained by the client device in some other manner (e.g., without user interaction).

In the example shown in fig. 3, the mobile device 310 transmits the proximity code 360 over the network 330. When the communicated proximity code is validated (e.g., by a server (not shown) on the network 330), the mobile device 310 receives the authorization data. An avatar 380 displayed in the controllable display area 370 is about to be controlled by the user 312. The avatar 380 may be associated with the user 312 or another user, or not with a particular user.

Fig. 4 is a diagram of another exemplary arrangement 400 in which techniques described herein may be implemented. In this example, a mobile device 410 controlled by a user 412 sends control data to control a common display device 420 through a network 430. The common display device 420 includes a controllable display area 470 and also displays instructions 450 and proximity codes 460. As shown, the instructions 450 direct the user 412 to activate a button labeled "1" or "2" (e.g., a software button in a graphical user interface) to cause the avatar 480 to wave its hand or jump, respectively. Alternatively, the instructions 450 are different from those shown in fig. 4 or omitted.

The mobile device 410 includes a display 414 (e.g., a touch screen display) and a graphical user interface 416 in which software buttons labeled "1" and "2" are displayed, with the button labeled "1" highlighted to display the activation made by the user 412. The user interface 416 may be provided by a custom control application, by a web page, or by some other software or hardware component.

In any of the examples described herein, a user may activate a user interface element (e.g., a software button) in a user interface by tapping or holding a key on a keyboard on a touchscreen, by speaking into a microphone, or by some other input mechanism. The type of input used may vary based on, for example, the input devices available and user preferences.

In the example shown in fig. 4, the proximity of the mobile device 410 has been confirmed by the transmission of the proximity code 460. The mobile device 410 transmits control data (e.g., control data corresponding to the user's selection of the button labeled "1") to the public display device 420 via the network 430. In response to control data sent by the mobile device 416, the avatar 480 displayed in the controllable display area 470 is waving its hand.

Fig. 5 is a diagram of another exemplary arrangement 500 in which techniques described herein may be implemented. In this example, mobile devices 510, 540 controlled by users 512, 542, respectively, communicate with public display device 520 over public network 530 (e.g., the internet). The public display device 520 includes a controllable display area 570 having game elements 580, 582 controllable by users 512, 542. The public display device 520 also displays instructions 550, a status area 552, and proximity codes 560. As shown, instructions 550 direct the user to enter a proximity code 560 along with a user ID (e.g., "Gamertag") to join the game displayed in controllable display area 570. Alternatively, instructions 550 are different from those shown in fig. 5 or omitted.

In the example shown in FIG. 5, the user ID is used to authenticate the user 512, 542. The exemplary user IDs shown in fig. 5 are "gamer tags," which are public visible user IDs unique to the users 512, 542 and which were previously registered with an online service (e.g., microsoft Xbox live). The player tag may also be associated with a personalized avatar (not shown in FIG. 5) presented on public display device 520. Alternatively, other types of user IDs may be used for authentication (e.g., private user IDs not displayed on public display device 520) or authentication data other than user IDs may be used. For example, cryptographic information such as a security token (e.g., a security token issued by a Security Token Service (STS)), authentication credentials stored on the client device, or other information may be used for authentication. As indicated in status area 552, game element 580 ("Player 1") is being controlled by user 542 (player tag: "Player 1"). User 512 (player tag: "player 2") is joining the game to control game element 582 ("player 2"). Alternatively, the information in the status area 552 is different from the information shown in FIG. 5 or omitted.

In any of the examples described herein, various authentication mechanisms (e.g., OpenID, OAuth, microsoft windows liveid) may be used either alone or in combination with each other or with other authentication mechanisms described herein. For example, the microsoft windows phone platform may use windows live id and corresponding player tags to automatically authenticate some applications. Applications may also use custom authentication mechanisms that are specific to the application. For example, a custom control application downloaded to the mobile device may use a custom user ID and password to authenticate a user who wishes to control the public display device. The token may be passed to the server through a communication protocol described herein, and the server may analyze the token to determine whether particular users and client devices will be licensed to control a common display device. Alternatively, the user may be identified without strong authentication credentials. For example, a user may enter their name to be displayed on the screen (e.g., after pre-registering their name with the public display device service), an unregistered user ID, or a name or user ID of another person. As another alternative, the user may be anonymous.

In the example shown in FIG. 5, users 512, 542 use their mobile devices 510, 540 to communicate with public display device 520 over public network 530 via HTTP in a shared communication Session (labeled "Session _ ID1 (Session _ ID 1)"). The server 532 provides relay services to relay messages generated by the public display device 520 and the mobile devices 510, 540. The public display device 520 may generate messages to open and close communication sessions and may generate messages to obtain control data (e.g., control data in control messages sent from the mobile devices 510, 540). The mobile devices 510, 540 may generate messages to register with the open communication session and send control data that may be used to control content on the common display device 520.

The mobile device 510 includes a display 514 (e.g., a touch screen display) and a user interface 516 on which a proximity code 560 and player tag ("player 2") are displayed for input by the user 512. Alternatively, the proximity code 560 and player tag (or other authentication information) may be provided to the mobile device 510 without user action and/or without being displayed on the display 514. In the example shown in fig. 5, the mobile device 510 transmits the proximity code 560 along with authentication information (e.g., a player tag associated with the user 512) over the public network 530. The mobile device 510 receives a response (e.g., from the server 532) that includes a Session identifier ("Session _ ID1 (Session _ ID 1)") corresponding to an open HTTP Session between the mobile device 510, 540 and the public display device 520. Mobile device 510 transmits a message (e.g., to server 532) including the session identifier to register with the open communication session and allow user 512 to join the game.

The mobile device 540 includes a display 544 (e.g., a touch screen display) on which display 544 up and down arrow buttons are displayed in a graphical user interface 546. The graphical user interface 546 may be provided by a custom control application, by a web page, or by some other software or hardware component. In the example shown in fig. 5, the proximity of mobile device 540 has been confirmed (e.g., by transmission of proximity code 560), and mobile device 540 has registered with an open communication session (e.g., by sending a message with a session identifier) to allow user 542 to join the game. Mobile device 540 transmits control data (e.g., display control data corresponding to user selection of game element 580 to move up or down) via public network 530 through control messages. The mobile device 540 may also receive response data (e.g., from the server 532) that may indicate, for example, that the control message has been successfully relayed to the public display device 520.

Fig. 6 is a diagram of another exemplary arrangement 600 in which techniques described herein may be implemented. In this example, mobile devices 612, 640, controlled by users 610, 642, respectively, communicate with a public display device 620 over a public network 630 (e.g., the internet). The public display device 520 includes a controllable display region 670 having avatars 680, 682 associated with users 612, 642, respectively. The public display device 620 also displays instructions 650, 654 and a status region 652. As shown, the instructions 650 direct the user to enter a user ID (e.g., "player tag") to control the avatar associated with the user ID in the controllable display area 670. Instructions 654 direct the user to activate a button labeled "1" or "2" (e.g., a software button in a graphical user interface) to cause the avatar to wave its hand or jump, respectively. As indicated in the status region 652, the avatar 680 is being controlled by the user 642 (Gaster tag: "Gaster 1"), the user 612 (Gaster tag: "Gaster 2") is joining to control the avatar 682, and another portion of the controllable display region 670 is indicated as being available to another user (not shown) to display and control another avatar (not shown) (as indicated by the text "join now!"). Alternatively, the instructions 650, 654 and/or information in the status region 652 is different than the information shown in FIG. 6 or omitted.

In the example shown in FIG. 6, users 612, 642 use their mobile devices 610, 640 to communicate with public display device 620 via public network 630 via HTTP in separate communication sessions, labeled "Session _ ID1 (Session _ ID 1)" and "Session _ ID2 (Session _ ID 2)", respectively. Server 632 provides relay services to relay messages generated by public display device 620 and mobile devices 610, 640. The public display device 620 may generate messages to open and close communication sessions and may generate messages to obtain control data (e.g., control data in control messages sent from the mobile devices 610, 640). The mobile devices 610, 640 may generate messages to register with the open communication session and send control data that may be used to control content on the common display device 620.

The mobile device 610 includes a display 614 (e.g., a touch screen display) on which a player tag ("player 2") associated with the user 612 is displayed on the display 614. Alternatively, player tabs (or other authentication information) may be provided to the mobile device 610 without user action and/or without being displayed on the display 614. In the example shown in fig. 6, the mobile device 610 transmits the player tag along with proximity data (e.g., GPS location data and/or proximity code) over the public network 630. The mobile device 610 receives a response (e.g., from the server 632) that includes a session identifier ("session _ ID 2") corresponding to the open HTTP communication session. The mobile device 610 transmits a message (e.g., to the server 632) including the session identifier to register with the open communication session and allow the user 612 to control the avatar 682.

The mobile device 640 includes a display 644 (e.g., a touch screen display) and a graphical user interface 646 in which software buttons labeled "1" and "2" are displayed, with the button labeled "1" highlighted to display the activation made by the user 642. The user interface 646 may be provided by a custom control application, by a web page, or by some other software or hardware component. In the example shown in fig. 6, the proximity of mobile device 640 has been confirmed (e.g., by the transfer of proximity data), and mobile device 640 has registered with an open communication session associated with session _ ID1 (e.g., by sending a message with the associated session ID) to allow user 642 to control avatar 680. The mobile device 640 communicates control data (e.g., control data corresponding to the user's selection of an action to be performed by the avatar 680) via the public network 630 through control messages. Mobile device 640 may also receive response data (e.g., from server 632) that may indicate, for example, that the control message has been successfully relayed to public display device 620.

Alternatives to the examples shown in arrangements 300, 400, 500, 600 are possible. For example, the common display device 320, 420, 520, 620 may display different user-controllable content. As another example, the content displayed by the common display device 320, 420, 520, 620 may be controlled in some other manner. In actual practice, arrangements such as arrangements 300, 400, 500, 600 described herein may be more complex, have additional functionality, have more complex relationships between devices, and the like. The techniques described herein may be generally applicable to the details of operating systems or hardware, and may be applied in various environments to take advantage of the features.

IV.Exemplary protocol

This section describes exemplary communication protocols for controlling public displays with private devices. The exemplary communication protocol may be used in the arrangements described above, such as arrangements 500, 600, or in some other arrangement.

In this exemplary communication protocol, there are two ends to exchange messages and data through a relay service: a receiver side (e.g., a public display device) and a sender side (e.g., a client device used to control the public display device). The receiver side may be referred to as the receiver and the sender side may be referred to as the sender. The relay service is implemented on a relay server that communicates with the receiver and the sender through HTTP. In order to achieve fast communication between a sender and a receiver, a relay server performs connection maintenance (paring). For example, when a recipient attempts to receive data from a sender, if there is no data to receive (e.g., because the sender has not yet sent any data), the relay server may maintain the recipient's connection by keeping the recipient's connection open until there is data to receive. As another example, the relay service may maintain the sender's connection waiting for an open connection to become available from the recipient. The time that a connection can be left open may be limited (e.g., to avoid keeping connections that are not actually used) to a limited period of time, such as minutes for a connection to a recipient while waiting for data from the sender, seconds for a connection to the sender while waiting for an open connection from the recipient. When a client makes a request to a server and the server holds the request, the server may determine the amount of time to maintain the connection. However, the underlying operating system library on the client may affect how long (e.g., one minute) the client will wait back for the connection to be held. The server may respond to requests that are held but still do not have a response (e.g., within a minute or some other suitable period of time, which may be implemented by the client library). By maintaining the connection, the relay service can efficiently process asynchronous requests in a near synchronous manner.

In this exemplary communication protocol, the recipient has three message types: OpenSession, CloseSession, and Receive. OpenSession opens a session for a sender to connect to. It passes in two parameters: a session identifier (or session ID) and an access token. For example, the session identifier may be a GUID that is cryptographically unique to a session or interaction with the user, and the access token may be a string of characters (e.g., a short text code to facilitate easy entry by the user (e.g., by typing)) or some other type of information (e.g., an image of a barcode, audio information such as a tone or a set of tones). The CloseSession closes the session and takes the session identifier as a parameter. OpenSession and CloseSession appear as synchronous calls and return immediately. Receive takes the session identifier and returns the data sent from the sender. The Receive returned data includes control data from the sender as well as the identity of the sender. If there is no data to return, the relay server remains connected until there is data to return or until a timeout threshold is reached. When the recipient receives data (e.g., sender-generated data or a message from the relay server indicating that no data was received after a timeout), the recipient may perform an action. For example, the recipient may cause the display to change based on control data received from the sender, close an inactive session (e.g., after a timeout threshold has been reached), open a new session (which can also be held if appropriate), and/or require more data from the relay service (e.g., via another Receive message). If the connection is maintained, after a period of time (e.g., one minute) during which no data is available, the relay server may return a code indicating that no data is available and the recipient may issue a new request for data as an alternative to closing the connection, allowing the recipient to be more forgiving of waiting for data.

In this exemplary communication protocol, the sender has two message types: register and Send. The Register message is used to connect to an open session. The Register message takes the access token as a parameter and returns the session identifier (if available). The sender may store the session identifier in an http cookie, for example. If no open session is available, the relay service may return an error or hold the sender's connection for a period of time (e.g., seconds) to see if an open session becomes available from the recipient. The Send message is used to transmit the command to the recipient. The Send message takes a command (e.g., byte array or string) and a session identifier as arguments and returns a flag indicating "true" if the command is sent to the recipient and "false" if the command is not sent to the recipient. The relay server waits until a command has been sent to the recipient before returning "true" to the sender. If the data is not sent to the recipient (e.g., if there is no corresponding open session, or if there is no Receive request from the recipient), the relay server returns "false" (e.g., after a timeout threshold such as a few seconds has been reached) and the sender may take appropriate action (e.g., alert the user that the command was not received by the display that the user is attempting to control). A client device that is a sender may generate and send control data using a client application installed on the client device. Alternatively, the client device may use a web browser to navigate to a website and use the website to generate and send control data. If the client device uses a website rather than a client application, the website may act as a sender on behalf of the client device.

In an exemplary scenario, a large, public LED display in a building lobby shows a time-limited proximity code and registers itself with a cloud server on the internet as a recipient waiting for a session to be conducted with the time-limited proximity code. The recipient creates a unique, time-limited proximity code, opens a session with the relay server that is passed in the code and displays the code. At the website or in a custom client application, the user enters the time-limited proximity code and their player tag into a client device (e.g., a smartphone). Alternatively, the user may input a different user ID, or user identification may be performed without the user inputting a user ID. The code and player tag are passed (e.g., via an access token in a Register message) over the internet to the same cloud server that is used as the relay server. The cloud server may then confirm the accuracy of the time-limited proximity code. Alternatively, another entity (e.g., a recipient) may confirm the accuracy of the code. The cloud server generates and sends a unique session identifier to the client device. The cloud server also sends the session identifier and the user's player tag to a recipient that displays an avatar associated with the user's player tag. Alternatively, another entity (e.g., a recipient) may generate and/or transmit the session identifier. Avatar data (e.g., personalized avatar appearance data) may be obtained from a cloud server or some other entity. As the connection is now established, the client device provides an interface (e.g., six buttons in a graphical user interface) through which the user may issue commands to cause the avatar to perform an action (e.g., laugh, cry, jump). When the user activates the button, the Send message is sent to the cloud server, which then relays control data corresponding to the user's command to the recipient. The display may then perform operations to cause the avatar to respond to the user's commands. The lack of user action may cause the session to time out.

In this exemplary communication protocol, a common HTTP relay service relays messages between a sender (e.g., a closed sender) and a recipient (e.g., a closed recipient), where one-way messages are passed from the sender to the recipient. The recipient establishes an active session with a sender. However, various extensions and alternatives to the exemplary protocols described herein are possible. For example, while some described examples demonstrate one-way communication from a sender to a recipient, the protocol may be extended to pass messages from the recipient to the sender (e.g., in a two-way communication scenario). As another example, multiple sessions or shared sessions may be used.

V.Exemplary techniques

Fig. 7 is a flow diagram showing an exemplary technique 700 for obtaining authorization for a client device to control content displayed on a public display device. A client device or other device, such as mobile device 100, performs the technique 700.

At 710, the client device obtains a time-limited proximity code from a public display device. The time-limited proximity code may include, for example, visual information (e.g., text, graphics, or images) or audio information (e.g., tones or groups of tones). At 720, the client device transmits the time-limited proximity code from the client device to the authorizer over the public network. The authorizer may be, for example, a server connected to a public network. At 730, the client device receives authorization data from the authorizer over the public network. The authorization data confirms that the client device is authorized to control content displayed on the public display device. The authorization data may include a session identifier that identifies the HTTP session. The client device may transmit control data (e.g., over an HTTP session) to the common display device. For example, the client device may display a graphical user interface (e.g., on a touch screen) that includes elements (e.g., software buttons) corresponding to commands for controlling content displayed on the common display device. The client device may receive user input through the graphical user interface and transmit control data based on the user input. The control data may include, for example, avatar control commands for causing the public display device to display avatar actions (e.g., swing, jump) or game control commands for causing the public display device to display game actions in a game (e.g., move game elements in a single-player or multi-player game). The client device may transmit authentication data (e.g., user ID, password information such as player tag) that may be entered by the user (e.g., in response to a prompt) or obtained in some other manner (e.g., from authentication credentials already in storage on the client device). The client device may use a custom display device control application, a web browser application, or some other application to perform actions related to control of the common display device.

Fig. 8 is a flow diagram illustrating an exemplary technique 800 for displaying visual content based on control data received over a public network from client devices located within a proximity zone. A public display device or other computing device performs the technique 800.

At 810, the public display device displays a proximity code associated with a proximity region for the public display device. At 820, the public display device receives control data (e.g., display control data such as avatar control commands) from client devices located within the proximity region over the public network (e.g., over an HTTP session). At 830, the public display device displays visual content (e.g., an avatar, a user ID associated with the avatar, and/or other content) based at least in part on the received control data. The common display device may also receive other control data from one or more additional client devices located within the proximity zone, and the displayed visual content may be further based on the control data received from the additional client devices.

Fig. 9 is a flow diagram illustrating an exemplary technique 900 for providing relay services to a common display device and a client device controlling the common display device. A computing device such as a server or other computing device performs the technique 900.

At 910, the server receives a first message from the public display device, the first message including a request to open an HTTP session identified by a session identifier (e.g., a session identifier generated by the public display device or the server). At 920, the server receives a second message from the client device, the second message including a request to connect to the HTTP session identified by the session identifier. At 930, the server sends the session identifier to the client device in response to the second message. At 940, the server receives a third message from the client device, the third message including control data for controlling visual content displayed by the common display device. At 950, the server receives a fourth message from the public display device, the fourth message including a request for available control data. In response to the fourth message, the server transmits the control data included in the third message to the common display device, at 960.

Any combination of the commands and operations described herein may be applied in any of the above techniques. Depending on the desired implementation and type of processing, the processing stages shown in the example techniques may be rearranged, added, omitted, divided into multiple stages, combined with other stages, and/or replaced with similar stages.

VI.Exemplary computing Environment

FIG. 10 illustrates a generalized example of a suitable computing environment 1000 in which the described technology may be implemented. The computing environment 1000 is not intended to suggest any limitation as to scope of use or functionality, as the techniques may be implemented in diverse general-purpose or special-purpose computing environments.

Referring to fig. 10, the computing environment 1000 includes at least one processing unit 1010 coupled to memory 1020. In fig. 10, this basic configuration 1030 is included within the dashed line. Processing unit 1010 executes computer-executable instructions. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. The memory 1020 may be non-transitory memory, such as volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. Memory 1020 may store software 1080 implementing any of the techniques described herein.

The computing environment may have additional features. For example, the computing environment 1000 includes storage 1040, one or more input devices 1050, one or more output devices 1060, and one or more communication connections 1070. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing environment 1000. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing environment 1000 and coordinates activities of the components of the computing environment 1000.

The storage 1040 may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, CD-RWs, DVDs, or any other non-transitory computer-readable medium which can be used to store information and which can be accessed within the computing environment 1000. Storage 1040 may store software 1080 containing instructions for any of the techniques described herein.

The input device (1050) may be a touch input device such as a keyboard, touch screen, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing environment 1000. The output device 1060 may be a display, a printer, a speaker, a CD or DVD recorder, or another device that provides output from the computing environment 1000. Some input/output devices, such as touch screens, may include both input and output functionality.

The communication connection 1070 allows communication through a communication mechanism to another computing entity. The communication mechanism conveys information such as computer-executable instructions, audio/video or other information, or other data. By way of example, and not limitation, communication mechanisms include wired or wireless techniques implemented with an electrical, optical, Radio Frequency (RF), infrared, acoustic, or other carrier.

The techniques herein may be described in the general context of computer-executable instructions, such as those included in program modules, being executed in a computing environment on a target real or virtual processor. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or separated between program modules as desired in various embodiments. Computer-executable instructions for program modules may be executed in local or distributed computing environments.

Any of the storage acts described herein may be implemented by storage in one or more computer-readable media (e.g., non-transitory computer-readable storage media or other tangible media). Any of the things described as being stored may be stored in one or more computer-readable media (e.g., computer-readable storage media or other tangible media).

Any of the methods described herein may be implemented by (e.g., encoded on) computer-executable instructions in one or more computer-readable media (e.g., non-transitory computer-readable storage media or other tangible media). These instructions may cause a computer to perform the method. The techniques described herein may be implemented in various programming languages.

Any of the methods described herein can be implemented by computer-executable instructions stored in one or more non-transitory computer-readable storage devices (e.g., memory, CD-ROM, CD-RW, DVD, etc.). These instructions may cause a computer to perform the method.

Extension and substitution

Various alternatives to the implementations described herein are possible. For example, the user interface described with reference to the illustrations may be changed by changing the content or arrangement of the user interface features shown in the figures, by omitting certain features, and the like. As another example, while certain implementations have been described with reference to particular devices and user input mechanisms (e.g., mobile devices with touch screen interfaces), the described techniques and tools may also be used with other devices and/or user input mechanisms.

In view of the many possible embodiments to which the principles of the disclosed invention may be applied, it should be recognized that the illustrated embodiments are only preferred examples of the invention and should not be taken as limiting the scope of the invention. Rather, the scope of the invention is defined by the appended claims. Accordingly, all that comes within the spirit and scope of these claims is claimed as the invention.

Claims (10)

1. A computerized method comprising:

obtaining, by a client device, a time-limited proximity code from a public display device;

transmitting the time-limited proximity code from the client device to an authorizer over a public network; and

receiving, by the client device, authorization data from the authorizer over the public network based at least in part on the transmitted time-limited proximity code, the authorization data confirming authorization of the client device to control content displayed on the public display device; and

transmitting control data from the client device to the public display device via a hypertext transfer protocol, HTTP, relay server, the HTTP relay server performing a connection hold comprising at least one of:

maintaining an open connection with the client device while waiting for a connection to be established with the public display device; or

Maintaining an open connection with the public display device while waiting for control data from the client device.

2. The method of claim 1, wherein the authorization data comprises a session identifier that identifies an HTTP session established with the HTTP relay server.

3. The method of claim 1, wherein the control data comprises at least one of game control commands or avatar control commands operable to cause the public display device to display game actions in a game.

4. The method of claim 3, wherein the control data comprises the game control commands, and wherein the game comprises a multiplayer game.

5. The method of claim 1, further comprising transmitting authentication data from the client device over the public network, the authentication data including at least one of user ID or password information.

6. The method of claim 1, wherein the time-limited proximity code comprises visual information or audio information.

7. The method of claim 1, wherein the client device includes at least one of a custom display device control application or a web browser application, and wherein the client device performs one or more of the described steps using at least one of a custom display device control application or a web browser application.

8. The method of claim 1, wherein the client device comprises a mobile device having a touchscreen, the method further comprising:

displaying a graphical user interface on the touch screen, the graphical user interface including one or more elements corresponding to commands for controlling content displayed on the public display device;

receiving user input via the graphical user interface; and

transmitting control data from the client device to the public display device, wherein the control data is based on the user input received via the graphical user interface.

9. A computerized method, the method comprising:

displaying a proximity code associated with a proximity region for a common display device;

receiving, over a public network, first control data from a first client device located within the proximity area through a relay server configured to perform connection maintenance comprising at least one of:

maintaining an open connection with the first client device while waiting for a connection to be established with the public display device; or

Maintaining an open connection with the public display device while waiting for control data from the first client device; and

the visual content is displayed based at least in part on the received first control data.

10. A method performed by a server comprising one or more processors, memory, and a storage medium storing computer-executable instructions for causing the server to perform the method, the method comprising:

receiving a first message from a public display device, the first message comprising a request to open an HTTP session identified by a session identifier;

receiving a second message from a client device, the second message comprising a request to connect to the HTTP session identified by the session identifier;

in response to the second message, sending the session identifier identifying the HTTP session to the client device;

receiving a third message from the client device, the third message including control data for controlling visual content displayed by the public display device;

receiving a fourth message from the public display device, the fourth message comprising a request for available control data; and

in response to the fourth message, sending the control data of the third message to the public display device via a hypertext transfer protocol, HTTP, relay server, the HTTP relay server performing a connection hold comprising at least one of:

maintaining an open connection with the client device while waiting for a connection to be established with the public display device; or

Maintaining an open connection with the public display device while waiting for control data from the client device.