[go: up one dir, main page]

WO2012164076A1 - Systems and methods for configurable remote control - Google Patents

Systems and methods for configurable remote control Download PDF

Info

Publication number
WO2012164076A1
WO2012164076A1 PCT/EP2012/060402 EP2012060402W WO2012164076A1 WO 2012164076 A1 WO2012164076 A1 WO 2012164076A1 EP 2012060402 W EP2012060402 W EP 2012060402W WO 2012164076 A1 WO2012164076 A1 WO 2012164076A1
Authority
WO
WIPO (PCT)
Prior art keywords
commands
control device
control
computer program
capabilities
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2012/060402
Other languages
French (fr)
Inventor
Robert DE WIT
Chris DE BACKER
Peter VERSIEREN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
OKEEZ bvba
Original Assignee
OKEEZ bvba
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by OKEEZ bvba filed Critical OKEEZ bvba
Publication of WO2012164076A1 publication Critical patent/WO2012164076A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
    • H04N21/4222—Remote control device emulator integrated into a non-television apparatus, e.g. a PDA, media center or smart toy
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H04N21/42226—Reprogrammable remote control devices
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/4508—Management of client data or end-user data
    • H04N21/4532—Management of client data or end-user data involving end-user characteristics, e.g. viewer profile, preferences
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8166—Monomedia components thereof involving executable data, e.g. software
    • H04N21/8186—Monomedia components thereof involving executable data, e.g. software specially adapted to be executed by a peripheral of the client device, e.g. by a reprogrammable remote control
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4126—The peripheral being portable, e.g. PDAs or mobile phones
    • H04N21/41265—The peripheral being portable, e.g. PDAs or mobile phones having a remote control device for bidirectional communication between the remote control device and client device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
    • H04N21/42222—Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
    • H04N21/42224—Touch pad or touch panel provided on the remote control

Definitions

  • the invention relates to arrangements of electronic devices and a computer program, enabling interaction between at least a pair of electronic devices, the underlying computer-implemented methods at set-up, corresponding computer programs, used at run-time, the use of such arrangements by users and application designers, and electronic devices specially adapted for use in such arrangements.
  • the symptom translates to users being confronted with different means to interact with one and the same service or application, such as an install-based desktop application, using keyboard and mouse, and advances user interface features such as drag & drop; a desktop browser, using keyboard & mouse, with no access to advances user interface features; a mobile handheld or tablet browser app, generally with a touch-based interface, and limited user interface functionality; an install-based mobile handheld app, with access to specific mobile platform functionalities; an install or SaaS-based IPTV app, with remote-control interaction and very limited user interface functionality.
  • an install-based desktop application using keyboard and mouse
  • advances user interface features such as drag & drop
  • a desktop browser using keyboard & mouse
  • a mobile handheld or tablet browser app generally with a touch-based interface, and limited user interface functionality
  • an install-based mobile handheld app with access to specific mobile platform functionalities
  • an install or SaaS-based IPTV app with remote-control interaction and very limited user interface functionality.
  • the main frustration with regard to this phenomenon is having to invest time and effort
  • the invention provides for the field of consumer computing technology in the form of arrangements of electronic devices (of which part might be handheld) and a computer program, the underlying computer-implemented methods at set-up, corresponding computer programs, used at run-time, which enable remote control interaction, the invention aims technically at scalable solutions, while providing improved user interface functionality with less devices. Summary of the invention
  • a computer program for enabling remote control (interaction) of any apparatus of a predefined first set by any control device of a predefined second set.
  • the computer program (set of instructions) comprising means (code) for inputting remote control capabilities of a control device (of the first set) and an apparatus (of the second set); and means (code) to configure, based on said input, the arrangement of the control device and the apparatus, in order to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus.
  • the control device is not specifically designed for the apparatus. In an embodiment thereof the set of remote control capabilities of the control device, selected for remote control of the apparatus is less than the required remote control capabilities of the apparatus.
  • the means to configure arranges that one or more of the remote control capabilities of the control device represents a grouping of at least two of said apparatus remote control capabilities.
  • the means (code) to configure comprises means (code) to link the control device capabilities into first predefined abstract capabilities, means (code) to link the apparatus capabilities into second predefined abstract capabilities; and means (code) to link said control device and apparatus capabilities by deriving mapping from predefined information between their respective abstract capabilities.
  • the grouping whereby the grouping results in grouping of commands selected from the group of: keyboard commands, mouse commands, media commands, explorer commands, system commands, window commands, application specific commands, API commands, emulator commands, batch commands, script commands, control line commands, remote procedural call commands, inter process call commands, service commands, file system commands, database commands, executable commands, process commands, messaging commands, user interface commands, menu commands, virtual machine commands, just in time commands, screen capture and drawings commands, input commands, output commands, data capturing commands, data scraping commands or browser commands. More in particular the grouping may result in a grouping of commands, whereby at least two are from another category. In a further embodiment at least three are from another category.
  • a computer implemented method for enabling an arrangement comprising a control device and an apparatus to remote control of the apparatus by the control device comprises of providing remote control capabilities of a control device and an apparatus as input for a computer program; configuring the arrangement of the control device and the apparatus by the computer program based on those inputs, such that such arrangement becomes capable to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus.
  • a computer implemented method for remote control of an apparatus by a control device comprising performing the method of the second aspect; receiving user instruction on said control device to generate control data; adapting said generated control data in accordance with the configuration and transmit said adapted such adapted control data to the apparatus; and receive said transmitted adapted control data by the apparatus and perform the control operation in accordance therewith.
  • an arrangement comprising a control device, an apparatus and means enabling transmission of at least control data from said control device to said apparatus, wherein at least one of said control device, apparatus or said transmission means, being adapted for adapting in the control device generated control data in response to user instructions, in way suitable for the apparatus, said arrangement being configured by the computer program as discussed above.
  • run-time computer programs for adapting control data used for remote control of an apparatus by a control device, the control data being generated at the control device, is provided; the computer program is being executed on either one of the control device, the apparatus or any transmission means used for transmitting the control data from said control device to said apparatus, the computer program comprising means (code) for receiving control data; and means (code) for adapting said received control data, said adapting means being provided by use of the computer program as discussed above.
  • a method for computer aided design of a user remote control interface for an apparatus, suitable for use on a control device comprising the steps of providing remote control capabilities of a control device and an apparatus as input for a computer program as discussed above; providing at least one by the user demanded grouping of said apparatus remote control capabilities into a single remote control capabilities of the control device as input for a computer program as discussed above; configuring the arrangement of the control device and the apparatus by the computer program based on those provided inputs, such that such arrangement becomes capable to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus.
  • a seventh aspect of the invention the method of computer aided creation of computer program product of a target apparatus as described above is disclosed, the method comprising the steps of inputting the remote control capabilities of the target apparatus; loading a generic template of the computer program product; adapting the generic template based on the inputs to obtain the computer program product.
  • the generic template describes a generic flow with a plurality of steps; and wherein the step of adapting comprising selecting those steps of the generic flow being of use for the computer program product specially targeting a target apparatus.
  • the symbol representation can be selected from the group of: icon representation, text button representation, or hyperlink representation.
  • the invention in its various aspects is designed for mastering a variety of heterogeneous devices, in particular with respect to their control capabilities and/or requirements while being scalable, and therefore an optimal design in terms of re-use of code, by performing abstractions and derivation from generic components.
  • the invention hence provides systems and methods for configurable remote control, customizable by a user or a process designer, and further enables user friendly interfacing by use of symbol representation, such as but not limited to icon representation.
  • FIG. 1 Context of the invention with the various devices and the computer program product.
  • FIG. 1 Generic flow Figure 3 shows the three step flowchart of the method.
  • Figure 4 shows the various API used with the invented technology.
  • Figure 5 shows the steps of converting and mapping, via the notion of abstract capabilities.
  • Figure 6-11 shows specific flows derived from the template generic flow of figure 2. Detailed description of the invention
  • the invention provides technology enabling interaction between at least two electronic devices, more in particular enabling control by a first device (10), further denoted control device throughout the entire description, of a second device (20), further denoted apparatus throughout the entire description.
  • Electronic devices are hardware devices and/or or hardware devices running one ore multiple software processes, such as an application, for instance a game, or portions thereof.
  • the technology is in particular provided in the form of computer programs (40) and the underlying methods therein, such technology being provided either in the form of a storage medium wherein such computer programs are stored, or by download from a network (for example, the internet, local, wide, private and VPN), or by running the computer program by the user via a network at a place distant from the user premise.
  • a network for example, the internet, local, wide, private and VPN
  • the computer program execution (from the storage medium or after download from the storage place where it is downloaded to or at least the connection with the distant location) is on a third electronic device (30), typically at the user side, such third electronic device may coincide with either the first or second device or may be another device or a combination of those (in case of the computer program is distributed).
  • the arrangement provided by the invention hence comprises the first and second devices and the computer program and the third device, running such computer program or at least supporting a network connection to execute such computer program elsewhere.
  • the invention is capable to support multi-user, say a user at control side and a user at receive side, such users not necessarily be in the same location, for example: a remote presentation controller on a mobile (user 1) controlling the slides on a PC or TV in a meeting room(user 2).
  • the invention provides technology enabling control by the control device of the apparatus, such control device and apparatus, being remote from each other, meaning that (control) data exchange happens via a separate transmission, which can be wired, wireless or combinations thereof, such transmission is at least unidirectional from the control device to the apparatus, but may also be bidirectional, such transmission (50) can further be direct or indirect via a fourth electronics device (60), which in itself may comprise of a plurality of devices, such fourth device may be localized at the user premises or even extend to networks or the cloud.
  • a separate transmission which can be wired, wireless or combinations thereof, such transmission is at least unidirectional from the control device to the apparatus, but may also be bidirectional, such transmission (50) can further be direct or indirect via a fourth electronics device (60), which in itself may comprise of a plurality of devices, such fourth device may be localized at the user premises or even extend to networks or the cloud.
  • the invention provides technology enabling control by the control device of the apparatus, such control device and apparatus, being remote from each other, whereby such control device and apparatus where initially not intended nor designed for acting as a control-receive pair, therefore the computer program (product), enables remote control of any apparatus by any device, of course within the ambit of the supported list of control devices and apparatuses by such computer program.
  • the computer program (product) uses, one can therefore distinguish between (a) the set-up, whereby a user, after installing (in one of the above discussed manners) the computer program, will arrange the entire arrangement of his particular control device, the to be controlled apparatus and the computer program or portions thereof or other installed run-time programs, (b) the actual execution, wherein at least portions of the computer program or by the computer program configured run-time programs, will provide the necessary steps in either the control device, the apparatus or any transmission supporting fourth device, during the process of remote controlling, (c) acts of control device and/or apparatus (or processes like games) designers for further enriching the supported list for such embodiments wherein the computer program (product) is set-up as an semi-open environment.
  • apparatus can also be a class of apparatuses instead.
  • the apparatus might be a game running on a computing engine, whereby the computer program product is targeted for this single game but for instance supporting its various versions (in the development of the game or even its different versions for different OS).
  • the control capabilities are the same or nearly the same, one may state that the computer program product has some dedication for such game.
  • the invention also provides for a method of computer aided creation of computer program product for a target apparatus. More in particular such method will require a step of inputting the remote control capabilities of the target apparatus.
  • one wants to perform code re-use as much as possible therefore one loads a generic template of the computer program product and subsequently adapts the generic template based on the inputs to obtain the computer program product.
  • the generic template describes a generic flow (of which examples are given below) with a plurality of steps; and the step of adapting comprises selecting those steps of the generic flow being of use for the computer program product specially targeting a target apparatus.
  • the invention is of course applicable to control devices having already some degrees of universal control capabilities and/or apparatuses being capable of being remote controlled by a plurality of interfaces.
  • the key is that it is the computer program (product) with its input means, which arranges by configuring the entire set-up, and therefore adds extra possibilities on top of the flexibility that might already be present, to thereby provide other remote control experiences, designed by the user itself.
  • the computer program (product) will need to be able to configure the entire arrangement of the first, second and any fourth device if present for all possibilities it is designed for.
  • the computer program product may also come in different flavors (fast, slow, more or less advanced features), depending on the capabilities of the third devices, whereon it is executed.
  • the computer program product will be executable software for a computing engine or engines (if the distributed case), it might also be firmware or contains portions which can be considered firmware, or the devices (control devices, apparatus or either involved devices) may have firmware, supporting the computer program product, or the run-time code portions on either one of the device may use support of firmware available thereon.
  • the invention supports a dynamic process whereby the eventual flow in the arrangement will depend on the capabilities of the pairs, the following capabilities can for example be present: in a smartphone, display, touch, movement, optic (camera), in a game controller, touch and A/D (analog/digital), in Wii controller, touch, movement and A/D, in a voice control, sonic and A/D, in Kinect, movement, optic and A/D,
  • the inputting of remote control capabilities can be made relative easy by actually selecting the control device and apparatus from the predefined sets, for instance from a drop down menu or by inputting other identification information, wherein the computer program based on such selection loads the corresponding capabilities of the device, and in further preferred embodiments also other capabilities such as transmission (transmitting and receiving) capabilities and/or computing capabilities.
  • the inputting of the remote control or other capabilities can be more sophisticated, whereby the user actually specifies those, e.g. also from drop down menu's, or parameter ranges. Also the combination with the easier embodiment is possible, wherein a portion of the capabilities come with the identification, while the other get specified by the user, or another combination wherein the identification provides default parameters and possibilities which can be further tuned by the users, or even a combination of all such approaches.
  • the inputting of the remote control or other capabilities happens on an automated detection basis, whereby the computer program based on reception of signals received either as such or upon request by the program, derives such capabilities. Again such advanced approach is combinable with any of the above input possibilities.
  • the computer program product has the necessary information, based thereon, the arrangement of the control device and the apparatus and even other involved device (as the fourth device for transmission) get configured, in order to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus.
  • the computer program Given the heterogeneity of devices considered the computer program is designed to be able to decide on how to use the available transmission capabilities (e.g. select the most optimal one or at least suggest to the user the possibilities) and to decide where and which corresponding computer programs will be stored at which device (first, second or any other device in the chain) that will perform data format adaptations. Note that such computer programs are in essence different from the overall computer program product (as used to configure) as these focus on the data adaptation and run-time control (after configuration).
  • These computer programs might be computer programs preinstalled on the devices, which just get reconfigured (for instance by receiving certain settings) from the configuration computer program.
  • the run-time computer programs might be derived from the configuration computer program and hence be considered portions thereof. Combination of the above mentioned approaches are possible.
  • FIG 3 shows the three step flowchart of the method, wherein two inputting steps (100)(110) for the controller and receiver respectively, feed the configuration step (120).
  • the above computer program can be used in a static way, in the sense that the user (buyer, licensee) configures for one or more control devices (further also denoted controller) and to be controlled apparatuses, and thereafter keep such configuration, and/or share this with other users.
  • control devices further also denoted controller
  • the dynamics of the to be controlled apparatus might be high, for instance if such apparatus is an application like a games with various levels or screens or scenes, each requiring other (different or slightly different) control capabilities.
  • Such circumstance requires reconfiguration on a more real-time basis.
  • the computer program product supports flexible switching from one configuration to another configuration, either under the control of the user or even invoked by circumstances, originating from the apparatus itself (e.g. certain state values).
  • the technology supports user interaction, meaning interaction going beyond mere switching on or off the apparatus, or setting some parameters within said apparatus. Indeed, while the above might be an integral part thereof, real-time exchange of control data, created by actions of the user on the control device, for steering, guiding, playing the process on the apparatus is envisaged, as the user wants to see the effect of his actions in real-time on the apparatus, for instance on its display.
  • control device has limited remote control capabilities, not necessarily by itself, but the user might elect to use only a few of those, for instance he finds the other inconvenient or he might reserve them for any other purpose.
  • control capabilities of the apparatus or processes Indeed on the receive side, a large amount of control capabilities, e.g. by keyboard, mouse or combination thereof might be presented.
  • the technology must configure the devices and computer programs in that that one or more of the remote control capabilities of the control device represent a grouping (for instance a concatenation of or another logic sequence (even with some if then else conditions) at least two of said apparatus remote control capabilities, to thereby reduce the difference in amount of capabilities and further provide user comfort.
  • the computer program product supports a computer implemented set-up or design time method for enabling an arrangement comprising a control device and an apparatus to remote control of the apparatus by the control device, such method having steps of providing remote control capabilities of a control device and an apparatus as input for a computer program; configuring the arrangement of the control device and the apparatus by the computer program based on those inputs, such that such arrangement becomes capable to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus.
  • a computer implemented run-time method for actual remote control of an apparatus by a control device can be executed, the method starting with performing the design time method; receiving user instruction on said control device to generate control data; adapting said generated control data in accordance with the configuration and transmit said adapted such adapted control data to the apparatus; receive said transmitted adapted control data by the apparatus and perform the control operation in accordance therewith.
  • Execution of the design time method results in an arrangement or system comprising of a control device, an apparatus and means enabling transmission of at least control data from said control device to said apparatus, wherein at least one of said control device, apparatus or said transmission means, being adapted for adapting in the control device generated control data in response to user instructions, in way suitable for the apparatus.
  • the invention provides technology, enabling bridging, mapping, control, interfacing and data exchange between computing devices with operating systems or applications running on these computing devices, and controller devices, the technology is designed for universal application and extendable ((semi-)open environment, scalable approach), the technology builds on local, online or distributed applications or services, communication/conversion protocols and/or hardware.
  • the invention offers a solution to simplify interaction with products or services by allowing one computing device, application or service that we designate the controller to be used to send commands and user interactions to another computing device, application or service that we designate as the receiver through a range of connection, control and communication technologies that we designate as the bridging, including for pairs of controllers and receivers that were never designed or intended to communicate with or be used with each other in such a manner.
  • the generic flow diagram of Figure 2 illustrates an abstract of the interaction between a controller and a receiver through a bridge with underlying binding that entails the entire interaction flow.
  • the interaction can be initiated by a user or process between the components, and is based on one or several means of communication that are either within the capabilities of the controller and receiver, or provided by the bridging component.
  • the bridging component also provides translation of interaction and information exchange between controller and receiver, and, if required, emulates interaction events on the receiver. In this way, the technology allows communication between controllers and receivers that were not specifically designed or intended to interact with each other.
  • Controllers are local or remote devices, applications, application services or processes that may can provide one or several of the following: allow input in the form of user, environment, bridging, binding, and/or process interactions and events; allow output of information and/or user experiences from the bridging, binding and/or receiver components. Controllers can communicate with the bridging, binding and/or receiver components either through built-in means of communication and supported protocols, or these communication and protocols can be provided by the bridging component. It is possible that a controller in the form of an application or process resides on the same device as the receiver. Controllers can be virtual.
  • Receivers are devices, applications, application services or processes that may can provide one or several of the following: allow input in the form of user, environment, and/or process interactions and events, either directly as transmitted by a controller device or binding components, or indirectly, where the controller device input has been translated by the bridging component; allow output of information and/or user interfaces to itself, or the bridging, binding and/or controller components.
  • Receivers can communicate with the bridging, binding and/or controller components either through built-in means of communication and supported protocols, or these communication and protocols can be provided by the bridging component. It is possible that a receiver in the form of an application or process resides on the same device as the controller. Receivers can be virtual.
  • Bridging refers to hardware and/or software that is required to allow exchange and transformation or translation of events and information between controllers, receivers and the binding layer(s). Bridging is the core of the technology, that allows communication between controllers and receivers, even if not specifically designed or intended to interact with each other. Indeed the invention provides a computer program product to configure the entire arrangement of controllers, receivers and any intermediate device needed to realize the binding layer, the invention further provides then the run-time software installed under this configuration process control on one or more of the involved devices, to do at run-time the necessary data exchange and transformations/adaptations. The invention also describes hardware or the firmware thereof especially designed for support of the above described approach.
  • the bridging component(s) also interact with the binding layer, that manages aspects such as identification and facilitating connections.
  • Bridging components can be virtual and may reside on the controller, the receiver, on a separate local or remote device or service, or may be distributed among the controller, receiver and separate device or service.
  • the Bridging component generally performs the following sequence:
  • the binding layer is a series of local or distributed data, applications and/or services that span the entire interaction flow between a controller, a bridge and a receiver.
  • the main role of the binding layer is to provide functionality and services such as: facilitating the identification and connection process between controller and receiver; managing user profiles associated with controller/receiver pairs; managing security & permissions associated with controller/receiver pairs; serving user Interfaces for the controller and/or receiver; perform or delegate translation of controller-device and device-controller communication.
  • Translation may be delegated to the controller, receiver, bridge or a third-party service; provide install-based or zero-install distributions and updates of applications, services and/or drivers for controllers, receivers and bridges; serving bridging applications, configurations, translation maps and/or data; performing analytics on interactions ; persisting all of the above
  • API application programming interface
  • controller APIs bridging APIs
  • receiver APIs binding APIs
  • the controller subsection of the APIs contain the following specifications and functionality: controller identification; controller connection profile management; controller-to-receiver or controller-to-bridge connection establishment; user profile, security and authorization queries; controller-to-receiver or controller-to-bridge communication; receiver-to-controller or bridge-to-controller communication; controller output emulators (see below);binding layer interaction.
  • the bridging subsection of the APIs contain the following specifications and functionality: controller & receiver capability determination; transformation and passing of control ler-to- receiver and receiver-to-controller communication; binding Layer Interaction.
  • the controller subsection of the APIs contain the following specifications and functionality: receiver identification; receiver connection profile management; receiver-to-controller or receiver-to- bridge connection establishment; user profile, security and authorization queries; receiver-to- controller or receiver-to-bridge connection communication; controller-to-receiver or controller-to-bridge communication; receiver input emulators (see below); binding layer interaction.
  • An emulator in computing duplicates provides an emulation of) the functions of one system using a different system, so that the second system behaves like (and appears to be) the first system.
  • Emulators within the invented technology can be implemented for the controller and/or the receiver. Controller-side emulators are used to render (series of) data and/or results returned from the bridging and/or receiver as natively supported information, in case these results or data need a form of emulation to be entered, stored, or visualized on the controller.
  • Receiver-side emulators are used to input (series of) data, commands, events from the controller or transformed (series of) data, commands, events from the bridge into the receiver, in case a form of emulation for these events or data is required to be entered, stored, or visualized on the receiver.
  • the invention turns a portable, mobile device, such as a smart phone or pad, into a wireless controller for computers, applications, presentations and games.
  • a touch- capable mobile device like an iPhone or an Android device, a Touch-capable tablet or pad, like an iPad or a Windows Tablet, a Touch-capable Media Player, like a Music Pod, or any non- touch capable portable device that has a built-in web browser.
  • Wi-Fi wireless network
  • JavaScript capable web browser you can use it to control your desktop computer with the concepts of the invention. You control a function or program on your computer through a Macro. You are probably familiar with a Media Keyboard, i.e.
  • a macro in the context of the invention can be a combination of any of the following: Keyboard commands, like pressing or holding down one or more keys; Mouse commands, like moving the mouse or clicking any of the mouse buttons; Media commands, like volume and playback control for audio and video; Explorer commands, like exploring & selecting files on your desktop from your touch device; System commands, like logging off and hibernating your computer; Window commands, like selecting, moving or resizing a window; Browser commands, like opening a web site in your favorite web browser.
  • the invention can do more than only use static icons to represent macros.
  • it is very useful to display an area of a desktop program on your controller and to "touch through” from your controller to your desktop screen, for example in a video game, like shown here.
  • the invention does this by what we call live areas from your computer screen. In essence, you define what area(s) you want to stream as constantly updated images from your desktop to your controller display. This can result in extremely intuitive controller interfaces for certain types of applications and games.
  • Touch-based User touch Controller interface Built-in WiFi or SmartPhone interaction with a and (transformed) USB, Device or Tablet controller user interaction results Driver
  • interface e.g. a from the receiver in dedicated app or the form of textual browser-based and graphical controller interface
  • USB cable protocol & device driver
  • Controller elements e.g. remote controller
  • buttons e.g. LCD display
  • Video Images/Video Depending on output FireWire, USB, Camera sequence capabilities of the Wi-Fi,

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Selective Calling Equipment (AREA)

Abstract

The invention relates to a computer program, enabling remote control of any apparatus of a predefined first set by any control device of a predefined second set, said computer program comprising means for inputting remote control capabilities of a control device and an apparatus; and means to configure, based on said input, the arrangement of the control device and the apparatus, in order to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus. The invention is designed for mastering a variety of heterogeneous devices with respect to their control capabilities and/or requirements while being scalable, and therefore an optimal design in terms of re-use of code, by performing abstractions and derivation from generic components. The invention achieves a natural way of user interaction with distant processes by grouping different types of instructions, represented in user friendly formats.

Description

SYSTEMS AND METHODS FOR CONFIGURABLE REMOTE CONTROL
Field of the invention
The invention relates to arrangements of electronic devices and a computer program, enabling interaction between at least a pair of electronic devices, the underlying computer-implemented methods at set-up, corresponding computer programs, used at run-time, the use of such arrangements by users and application designers, and electronic devices specially adapted for use in such arrangements.
Background of the invention
The success of cloud computing and the growth in handheld device usage has brought a syndrome we will refer to as the five remotes to consumer computing. This refers to the fact that the average household has five different remote controls on their living room table such as a TV Remote control, a DVR/VCR Remote Control, a Set Top Box controller, a Game Console controller, and a Hi-Fi set remote control. All of these remotes are used to control a specific device. In consumer computing, the symptom translates to users being confronted with different means to interact with one and the same service or application, such as an install-based desktop application, using keyboard and mouse, and advances user interface features such as drag & drop; a desktop browser, using keyboard & mouse, with no access to advances user interface features; a mobile handheld or tablet browser app, generally with a touch-based interface, and limited user interface functionality; an install-based mobile handheld app, with access to specific mobile platform functionalities; an install or SaaS-based IPTV app, with remote-control interaction and very limited user interface functionality. For consumers, the main frustration with regard to this phenomenon is having to invest time and effort learning each of these interfaces, and that these different ways to access a product or service do not necessarily offer the same range of functionality. For businesses, the main challenge lies in having to develop and maintain all these different interfaces.
Aim of the invention
The invention provides for the field of consumer computing technology in the form of arrangements of electronic devices (of which part might be handheld) and a computer program, the underlying computer-implemented methods at set-up, corresponding computer programs, used at run-time, which enable remote control interaction, the invention aims technically at scalable solutions, while providing improved user interface functionality with less devices. Summary of the invention
In a first aspect of the invention a computer program (or the storage media with program) for enabling remote control (interaction) of any apparatus of a predefined first set by any control device of a predefined second set is disclosed. The computer program (set of instructions) comprising means (code) for inputting remote control capabilities of a control device (of the first set) and an apparatus (of the second set); and means (code) to configure, based on said input, the arrangement of the control device and the apparatus, in order to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus. Note that the control device is not specifically designed for the apparatus. In an embodiment thereof the set of remote control capabilities of the control device, selected for remote control of the apparatus is less than the required remote control capabilities of the apparatus.
In another embodiment the means to configure arranges that one or more of the remote control capabilities of the control device represents a grouping of at least two of said apparatus remote control capabilities.
In a further embodiment the means (code) to configure, comprises means (code) to link the control device capabilities into first predefined abstract capabilities, means (code) to link the apparatus capabilities into second predefined abstract capabilities; and means (code) to link said control device and apparatus capabilities by deriving mapping from predefined information between their respective abstract capabilities.
In a further embodiment thereof the grouping whereby the grouping results in grouping of commands selected from the group of: keyboard commands, mouse commands, media commands, explorer commands, system commands, window commands, application specific commands, API commands, emulator commands, batch commands, script commands, control line commands, remote procedural call commands, inter process call commands, service commands, file system commands, database commands, executable commands, process commands, messaging commands, user interface commands, menu commands, virtual machine commands, just in time commands, screen capture and drawings commands, input commands, output commands, data capturing commands, data scraping commands or browser commands. More in particular the grouping may result in a grouping of commands, whereby at least two are from another category. In a further embodiment at least three are from another category. In a second aspect of the invention a computer implemented method for enabling an arrangement comprising a control device and an apparatus to remote control of the apparatus by the control device is provided, the method comprises of providing remote control capabilities of a control device and an apparatus as input for a computer program; configuring the arrangement of the control device and the apparatus by the computer program based on those inputs, such that such arrangement becomes capable to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus.
In a third aspect of the invention a computer implemented method for remote control of an apparatus by a control device is disclosed, the method comprising performing the method of the second aspect; receiving user instruction on said control device to generate control data; adapting said generated control data in accordance with the configuration and transmit said adapted such adapted control data to the apparatus; and receive said transmitted adapted control data by the apparatus and perform the control operation in accordance therewith. In a fourth aspect of the invention an arrangement (system) is provided, comprising a control device, an apparatus and means enabling transmission of at least control data from said control device to said apparatus, wherein at least one of said control device, apparatus or said transmission means, being adapted for adapting in the control device generated control data in response to user instructions, in way suitable for the apparatus, said arrangement being configured by the computer program as discussed above. In a fifth aspect of the invention run-time computer programs for adapting control data, used for remote control of an apparatus by a control device, the control data being generated at the control device, is provided; the computer program is being executed on either one of the control device, the apparatus or any transmission means used for transmitting the control data from said control device to said apparatus, the computer program comprising means (code) for receiving control data; and means (code) for adapting said received control data, said adapting means being provided by use of the computer program as discussed above. In a sixth aspect of the invention a method for computer aided design of a user remote control interface for an apparatus, suitable for use on a control device, is presented, the method comprising the steps of providing remote control capabilities of a control device and an apparatus as input for a computer program as discussed above; providing at least one by the user demanded grouping of said apparatus remote control capabilities into a single remote control capabilities of the control device as input for a computer program as discussed above; configuring the arrangement of the control device and the apparatus by the computer program based on those provided inputs, such that such arrangement becomes capable to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus.
In a seventh aspect of the invention the method of computer aided creation of computer program product of a target apparatus as described above is disclosed, the method comprising the steps of inputting the remote control capabilities of the target apparatus; loading a generic template of the computer program product; adapting the generic template based on the inputs to obtain the computer program product.
In an embodiment thereof the generic template describes a generic flow with a plurality of steps; and wherein the step of adapting comprising selecting those steps of the generic flow being of use for the computer program product specially targeting a target apparatus. In an eighth aspect of the invention the use of an symbol representation on the control device of the grouping as discussed above. The symbol representation can be selected from the group of: icon representation, text button representation, or hyperlink representation. The invention in its various aspects is designed for mastering a variety of heterogeneous devices, in particular with respect to their control capabilities and/or requirements while being scalable, and therefore an optimal design in terms of re-use of code, by performing abstractions and derivation from generic components. Moreover the increased user experience that one wants to achieve by providing more natural ways of user interaction with distant processes (like a game) results in grouping of in terms of type of capabilities different types of instructions, such groupings to be represented in user friendly formats just as an icon. The invention hence provides systems and methods for configurable remote control, customizable by a user or a process designer, and further enables user friendly interfacing by use of symbol representation, such as but not limited to icon representation.
Brief description of the drawings
Figure 1 Context of the invention with the various devices and the computer program product.
Figure 2 Generic flow Figure 3 shows the three step flowchart of the method.
Figure 4 shows the various API used with the invented technology.
Figure 5 shows the steps of converting and mapping, via the notion of abstract capabilities.
Figure 6-11 shows specific flows derived from the template generic flow of figure 2. Detailed description of the invention
As shown in Figure 1, the invention provides technology enabling interaction between at least two electronic devices, more in particular enabling control by a first device (10), further denoted control device throughout the entire description, of a second device (20), further denoted apparatus throughout the entire description. Electronic devices are hardware devices and/or or hardware devices running one ore multiple software processes, such as an application, for instance a game, or portions thereof. The technology is in particular provided in the form of computer programs (40) and the underlying methods therein, such technology being provided either in the form of a storage medium wherein such computer programs are stored, or by download from a network (for example, the internet, local, wide, private and VPN), or by running the computer program by the user via a network at a place distant from the user premise. The computer program execution (from the storage medium or after download from the storage place where it is downloaded to or at least the connection with the distant location) is on a third electronic device (30), typically at the user side, such third electronic device may coincide with either the first or second device or may be another device or a combination of those (in case of the computer program is distributed). The arrangement provided by the invention hence comprises the first and second devices and the computer program and the third device, running such computer program or at least supporting a network connection to execute such computer program elsewhere. In an embodiment of the invention the invention is capable to support multi-user, say a user at control side and a user at receive side, such users not necessarily be in the same location, for example: a remote presentation controller on a mobile (user 1) controlling the slides on a PC or TV in a meeting room(user 2).
More in particular the invention provides technology enabling control by the control device of the apparatus, such control device and apparatus, being remote from each other, meaning that (control) data exchange happens via a separate transmission, which can be wired, wireless or combinations thereof, such transmission is at least unidirectional from the control device to the apparatus, but may also be bidirectional, such transmission (50) can further be direct or indirect via a fourth electronics device (60), which in itself may comprise of a plurality of devices, such fourth device may be localized at the user premises or even extend to networks or the cloud. Even more in particular the invention provides technology enabling control by the control device of the apparatus, such control device and apparatus, being remote from each other, whereby such control device and apparatus where initially not intended nor designed for acting as a control-receive pair, therefore the computer program (product), enables remote control of any apparatus by any device, of course within the ambit of the supported list of control devices and apparatuses by such computer program. In the computer program (product) uses, one can therefore distinguish between (a) the set-up, whereby a user, after installing (in one of the above discussed manners) the computer program, will arrange the entire arrangement of his particular control device, the to be controlled apparatus and the computer program or portions thereof or other installed run-time programs, (b) the actual execution, wherein at least portions of the computer program or by the computer program configured run-time programs, will provide the necessary steps in either the control device, the apparatus or any transmission supporting fourth device, during the process of remote controlling, (c) acts of control device and/or apparatus (or processes like games) designers for further enriching the supported list for such embodiments wherein the computer program (product) is set-up as an semi-open environment.
While the above description suggests a many-to-many context, one might choose to deliver (in one or another form) the computer program (product) alternatively for a many (control) devices to 1 apparatus context, thereby delivering a computer program (product) targeted for such an dedicated apparatus. Therefore the corresponding set may comprise just one apparatus. It must be noted that apparatus here can also be a class of apparatuses instead. As an example the apparatus might be a game running on a computing engine, whereby the computer program product is targeted for this single game but for instance supporting its various versions (in the development of the game or even its different versions for different OS). In essence if the control capabilities are the same or nearly the same, one may state that the computer program product has some dedication for such game. Alternatively formulated the invention also provides for a method of computer aided creation of computer program product for a target apparatus. More in particular such method will require a step of inputting the remote control capabilities of the target apparatus. In a preferred embodiment thereof one wants to perform code re-use as much as possible, therefore one loads a generic template of the computer program product and subsequently adapts the generic template based on the inputs to obtain the computer program product. In an embodiment thereof the generic template describes a generic flow (of which examples are given below) with a plurality of steps; and the step of adapting comprises selecting those steps of the generic flow being of use for the computer program product specially targeting a target apparatus. Further note that the invention is of course applicable to control devices having already some degrees of universal control capabilities and/or apparatuses being capable of being remote controlled by a plurality of interfaces. The key is that it is the computer program (product) with its input means, which arranges by configuring the entire set-up, and therefore adds extra possibilities on top of the flexibility that might already be present, to thereby provide other remote control experiences, designed by the user itself.
As the any control device and any apparatus context leads to a variety of such devices with substantially different control, transmission or computer program (portion) execution possibilities, the computer program (product) will need to be able to configure the entire arrangement of the first, second and any fourth device if present for all possibilities it is designed for. Naturally the computer program product may also come in different flavors (fast, slow, more or less advanced features), depending on the capabilities of the third devices, whereon it is executed.
More over while most often the computer program product will be executable software for a computing engine or engines (if the distributed case), it might also be firmware or contains portions which can be considered firmware, or the devices (control devices, apparatus or either involved devices) may have firmware, supporting the computer program product, or the run-time code portions on either one of the device may use support of firmware available thereon.
As already discussed the invention supports a dynamic process whereby the eventual flow in the arrangement will depend on the capabilities of the pairs, the following capabilities can for example be present: in a smartphone, display, touch, movement, optic (camera), in a game controller, touch and A/D (analog/digital), in Wii controller, touch, movement and A/D, in a voice control, sonic and A/D, in Kinect, movement, optic and A/D, In one embodiment of the invention the inputting of remote control capabilities can be made relative easy by actually selecting the control device and apparatus from the predefined sets, for instance from a drop down menu or by inputting other identification information, wherein the computer program based on such selection loads the corresponding capabilities of the device, and in further preferred embodiments also other capabilities such as transmission (transmitting and receiving) capabilities and/or computing capabilities.
In other embodiments of the invention the inputting of the remote control or other capabilities can be more sophisticated, whereby the user actually specifies those, e.g. also from drop down menu's, or parameter ranges. Also the combination with the easier embodiment is possible, wherein a portion of the capabilities come with the identification, while the other get specified by the user, or another combination wherein the identification provides default parameters and possibilities which can be further tuned by the users, or even a combination of all such approaches. In yet another embodiment of the invention the inputting of the remote control or other capabilities happens on an automated detection basis, whereby the computer program based on reception of signals received either as such or upon request by the program, derives such capabilities. Again such advanced approach is combinable with any of the above input possibilities. Once the computer program product has the necessary information, based thereon, the arrangement of the control device and the apparatus and even other involved device (as the fourth device for transmission) get configured, in order to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus. Given the heterogeneity of devices considered the computer program is designed to be able to decide on how to use the available transmission capabilities (e.g. select the most optimal one or at least suggest to the user the possibilities) and to decide where and which corresponding computer programs will be stored at which device (first, second or any other device in the chain) that will perform data format adaptations. Note that such computer programs are in essence different from the overall computer program product (as used to configure) as these focus on the data adaptation and run-time control (after configuration). These computer programs might be computer programs preinstalled on the devices, which just get reconfigured (for instance by receiving certain settings) from the configuration computer program. Alternatively the run-time computer programs might be derived from the configuration computer program and hence be considered portions thereof. Combination of the above mentioned approaches are possible.
Figure 3 shows the three step flowchart of the method, wherein two inputting steps (100)(110) for the controller and receiver respectively, feed the configuration step (120).
The above computer program can be used in a static way, in the sense that the user (buyer, licensee) configures for one or more control devices (further also denoted controller) and to be controlled apparatuses, and thereafter keep such configuration, and/or share this with other users. Alternatively the dynamics of the to be controlled apparatus (also denoted receiver) might be high, for instance if such apparatus is an application like a games with various levels or screens or scenes, each requiring other (different or slightly different) control capabilities. Such circumstance requires reconfiguration on a more real-time basis. The computer program product supports flexible switching from one configuration to another configuration, either under the control of the user or even invoked by circumstances, originating from the apparatus itself (e.g. certain state values). It must be particularly emphasized that the technology supports user interaction, meaning interaction going beyond mere switching on or off the apparatus, or setting some parameters within said apparatus. Indeed, while the above might be an integral part thereof, real-time exchange of control data, created by actions of the user on the control device, for steering, guiding, playing the process on the apparatus is envisaged, as the user wants to see the effect of his actions in real-time on the apparatus, for instance on its display.
In an embodiment of the invention a context is envisioned wherein the control device has limited remote control capabilities, not necessarily by itself, but the user might elect to use only a few of those, for instance he finds the other inconvenient or he might reserve them for any other purpose. With limited is meant with respect of the control capabilities of the apparatus or processes. Indeed on the receive side, a large amount of control capabilities, e.g. by keyboard, mouse or combination thereof might be presented. As the user prefers to use a remote control, not necessary to one-to-one link capabilities at controller and receiver side, the technology must configure the devices and computer programs in that that one or more of the remote control capabilities of the control device represent a grouping (for instance a concatenation of or another logic sequence (even with some if then else conditions) at least two of said apparatus remote control capabilities, to thereby reduce the difference in amount of capabilities and further provide user comfort. In an embodiment of the invention the computer program (product) comprises portions of code, for each possible control-receive part, and control logic portions, selecting the appropriate one of said portions, depending on the input. Those portions when elected and executed perform the necessary steps to configure the related devices and where possible install the necessary run-time computer programs, for providing the necessary control data adaptation or conversion steps.
As the above embodiment might be useful for a relative small set of devices to be supported, this approach definitely has difficulty in scaling for larger set of devices. A more elaborated approach (described in Figure 5) therefore exist in understanding the fundamental aspects of user interactions (type of interactions, how this translate in certain type of data) and exploit those understandings by providing a kernel, acting on so- called abstract control capabilities of first and second devices, and providing a predefined mapping between those. The computer program (product) is then further completed with means to link the real control capabilities to such abstract control capabilities. While the kernel is a stable aspect of the product, to be changed only by the developer of it, the convertors can be added by other developers (like drivers in an OS). Therefore such computer program is or arranged to be an open environment to a certain extent. Therefore within the means to configure the arrangements, means (code) to link the control device capabilities into first predefined abstract capabilities; means (code) to link the apparatus capabilities into second predefined abstract capabilities; and means (code) to link said control device and apparatus capabilities by exploiting a predefined mapping between their respective abstract capabilities. In the use of the above realization of the computer program, within the necessary control data adaptation or conversion steps, one will hence recognize code, originating from those different means.
Note that in the above execution of code is to be interpreted broadly as running code on the CPU or GPU or any other processes in a device but also storing in local memory or change a state or a setting locally. Note that with exploiting a predefined mapping between their respective abstract capabilities does not require that this predefined mapping is fixed. On the contrary, from the real control capabilities input, one will construct in an automated way the required mapping, by this construction exploits predefined code portions, such as libraries. Therefore one may state that actually a mapping is derived from predefined information (code portions, libraries).
The computer program product supports a computer implemented set-up or design time method for enabling an arrangement comprising a control device and an apparatus to remote control of the apparatus by the control device, such method having steps of providing remote control capabilities of a control device and an apparatus as input for a computer program; configuring the arrangement of the control device and the apparatus by the computer program based on those inputs, such that such arrangement becomes capable to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus. Subsequently a computer implemented run-time method for actual remote control of an apparatus by a control device can be executed, the method starting with performing the design time method; receiving user instruction on said control device to generate control data; adapting said generated control data in accordance with the configuration and transmit said adapted such adapted control data to the apparatus; receive said transmitted adapted control data by the apparatus and perform the control operation in accordance therewith. Execution of the design time method results in an arrangement or system comprising of a control device, an apparatus and means enabling transmission of at least control data from said control device to said apparatus, wherein at least one of said control device, apparatus or said transmission means, being adapted for adapting in the control device generated control data in response to user instructions, in way suitable for the apparatus. Hence at least one of the devices (control device, transmission means, apparatus) is actually changed, by changing setting, parameter or installing of run-time code in them. The invention provides technology, enabling bridging, mapping, control, interfacing and data exchange between computing devices with operating systems or applications running on these computing devices, and controller devices, the technology is designed for universal application and extendable ((semi-)open environment, scalable approach), the technology builds on local, online or distributed applications or services, communication/conversion protocols and/or hardware.
The invention offers a solution to simplify interaction with products or services by allowing one computing device, application or service that we designate the controller to be used to send commands and user interactions to another computing device, application or service that we designate as the receiver through a range of connection, control and communication technologies that we designate as the bridging, including for pairs of controllers and receivers that were never designed or intended to communicate with or be used with each other in such a manner.
To explain how the different main components of the technology work, it is important to understand the definition and scope of the main terms we use to describe it. The document-specific core terms we will use and define here are: controllers, receivers, bridging, binding layer. Other components whose meaning & scope is close to the general use of these terms in IT specifications that we'll define for clarity include: application Programmer Interface (API) and emulators. The best way to start defining the core components is by illustrating the generic flow of interaction that these components will go through in any specific implementation of the technology. We will describe specific implementations for controller/receiver pairs in the next chapters to illustrate how the technology is implemented for each of those specific use cases. It is however important to remember that the generic flow that will be illustrated here will always be at the basis of those specific implementations. It is important to first understand the scope and role of each of the core components in the generic flow, as each of these components can take on different forms in actual implementations.
The generic flow diagram of Figure 2 illustrates an abstract of the interaction between a controller and a receiver through a bridge with underlying binding that entails the entire interaction flow. The interaction can be initiated by a user or process between the components, and is based on one or several means of communication that are either within the capabilities of the controller and receiver, or provided by the bridging component. The bridging component also provides translation of interaction and information exchange between controller and receiver, and, if required, emulates interaction events on the receiver. In this way, the technology allows communication between controllers and receivers that were not specifically designed or intended to interact with each other.
Controllers are local or remote devices, applications, application services or processes that may can provide one or several of the following: allow input in the form of user, environment, bridging, binding, and/or process interactions and events; allow output of information and/or user experiences from the bridging, binding and/or receiver components. Controllers can communicate with the bridging, binding and/or receiver components either through built-in means of communication and supported protocols, or these communication and protocols can be provided by the bridging component. It is possible that a controller in the form of an application or process resides on the same device as the receiver. Controllers can be virtual.
Receivers are devices, applications, application services or processes that may can provide one or several of the following: allow input in the form of user, environment, and/or process interactions and events, either directly as transmitted by a controller device or binding components, or indirectly, where the controller device input has been translated by the bridging component; allow output of information and/or user interfaces to itself, or the bridging, binding and/or controller components. Receivers can communicate with the bridging, binding and/or controller components either through built-in means of communication and supported protocols, or these communication and protocols can be provided by the bridging component. It is possible that a receiver in the form of an application or process resides on the same device as the controller. Receivers can be virtual.
Bridging refers to hardware and/or software that is required to allow exchange and transformation or translation of events and information between controllers, receivers and the binding layer(s). Bridging is the core of the technology, that allows communication between controllers and receivers, even if not specifically designed or intended to interact with each other. Indeed the invention provides a computer program product to configure the entire arrangement of controllers, receivers and any intermediate device needed to realize the binding layer, the invention further provides then the run-time software installed under this configuration process control on one or more of the involved devices, to do at run-time the necessary data exchange and transformations/adaptations. The invention also describes hardware or the firmware thereof especially designed for support of the above described approach. Next to allowing controllers and receivers to communicate with, and understand, each other, the bridging component(s) also interact with the binding layer, that manages aspects such as identification and facilitating connections. Bridging components can be virtual and may reside on the controller, the receiver, on a separate local or remote device or service, or may be distributed among the controller, receiver and separate device or service. The Bridging component generally performs the following sequence:
1. Finding & Identifying the controller/receiver pair and linking them to a user profile
2. Determining the capabilities of the controller/receiver pair through the binding layer
3. Determining the communication carrier and protocol(s) to be used through the binding layer
4. Communicate with the binding layer to determine & persist security
5. Receive input data from the controller
6. Passes relevant input data to the binding layer
7. Perform transformations or translations of the received controller input into a format that the receiver understands
8. Communicates the transformed controller input to the receiver
9. Inputs the transformed controller input into the receiver by means of communication protocols, programmer interfaces, specific input mechanisms provided by the controller, or by emulating any of these. 10. Receive or capture output events, data, and/or state changes from the receiver
11. Passes relevant result data to the binding layer 12. Perform transformations or translations of the result into a format that the controller understands
13. Communicates the transformed result to the controller
Depending on the capabilities of the controller/receiver pair, certain steps in this generic flow may not be required. It is important to note that this flow allows combining any controller type with any receiver type.
The binding layer is a series of local or distributed data, applications and/or services that span the entire interaction flow between a controller, a bridge and a receiver. The main role of the binding layer is to provide functionality and services such as: facilitating the identification and connection process between controller and receiver; managing user profiles associated with controller/receiver pairs; managing security & permissions associated with controller/receiver pairs; serving user Interfaces for the controller and/or receiver; perform or delegate translation of controller-device and device-controller communication. Translation may be delegated to the controller, receiver, bridge or a third-party service; provide install-based or zero-install distributions and updates of applications, services and/or drivers for controllers, receivers and bridges; serving bridging applications, configurations, translation maps and/or data; performing analytics on interactions ; persisting all of the above
An application programming interface (API) is a particular set of rules and specifications that software programs can follow to communicate with each other. It serves as an interface between different software programs and facilitates their interaction, similar to the way the user interface facilitates interaction between humans and computers. Generally, the term API refers to one set of specifications that allows interacting with an application or service. Within the invented technology, all of the core components have their own specific API implementation, and all of these individual API subsets are interdependent and complementary. As shown in Figure 4, the invention therefore also provides for an overall API that is therefore the sum of the interrelated subsets of controller APIs; bridging APIs; receiver APIs; binding APIs, each with their own characteristics. The controller subsection of the APIs contain the following specifications and functionality: controller identification; controller connection profile management; controller-to-receiver or controller-to-bridge connection establishment; user profile, security and authorization queries; controller-to-receiver or controller-to-bridge communication; receiver-to-controller or bridge-to-controller communication; controller output emulators (see below);binding layer interaction. The bridging subsection of the APIs contain the following specifications and functionality: controller & receiver capability determination; transformation and passing of control ler-to- receiver and receiver-to-controller communication; binding Layer Interaction. The controller subsection of the APIs contain the following specifications and functionality: receiver identification; receiver connection profile management; receiver-to-controller or receiver-to- bridge connection establishment; user profile, security and authorization queries; receiver-to- controller or receiver-to-bridge connection communication; controller-to-receiver or controller-to-bridge communication; receiver input emulators (see below); binding layer interaction.
An emulator in computing duplicates (provides an emulation of) the functions of one system using a different system, so that the second system behaves like (and appears to be) the first system. Emulators within the invented technology can be implemented for the controller and/or the receiver. Controller-side emulators are used to render (series of) data and/or results returned from the bridging and/or receiver as natively supported information, in case these results or data need a form of emulation to be entered, stored, or visualized on the controller. Receiver-side emulators are used to input (series of) data, commands, events from the controller or transformed (series of) data, commands, events from the bridge into the receiver, in case a form of emulation for these events or data is required to be entered, stored, or visualized on the receiver.
The invention turns a portable, mobile device, such as a smart phone or pad, into a wireless controller for computers, applications, presentations and games. This can mean a touch- capable mobile device, like an iPhone or an Android device, a Touch-capable tablet or pad, like an iPad or a Windows Tablet, a Touch-capable Media Player, like a Music Pod, or any non- touch capable portable device that has a built-in web browser. Generally speaking, if a portable device supports connecting to a wireless network (Wi-Fi) and has a built-in JavaScript capable web browser, you can use it to control your desktop computer with the concepts of the invention. You control a function or program on your computer through a Macro. You are probably familiar with a Media Keyboard, i.e. a PC keyboard that has a couple of extra macro keys to control music playback, volume, open your mail application, etc. You may also be familiar with Gaming Keyboards that add a dozen or so extra keys that can hold macros that can be programmed by the user. While these specialized keyboards offer some basic extensions to a standard keyboard, through macros, they have limitations as these specialized keyboards only offer a limited number of possible extra keys, the extra keys are usually just labeled like "G1...G12", meaning the user has to remember which key hold a particular macro without any visual support, the extra macro keys are located on either the left or the right side of the keyboard and cannot be moved or switched. The invention is designed to eliminate these limitations. In an embodiment of the invention macros are visually represented on the controller by icons. A macro in the context of the invention can be a combination of any of the following: Keyboard commands, like pressing or holding down one or more keys; Mouse commands, like moving the mouse or clicking any of the mouse buttons; Media commands, like volume and playback control for audio and video; Explorer commands, like exploring & selecting files on your desktop from your touch device; System commands, like logging off and hibernating your computer; Window commands, like selecting, moving or resizing a window; Browser commands, like opening a web site in your favorite web browser. Generally, you will represent a single macro by a single icon on your controller. The icon on your controller is linked to a macro that executes on your desktop. You can use any existing icon image file on your computer or found in online icon collections, or create your own icon images. However the invention can do more than only use static icons to represent macros. In some cases, it is very useful to display an area of a desktop program on your controller and to "touch through" from your controller to your desktop screen, for example in a video game, like shown here. The invention does this by what we call live areas from your computer screen. In essence, you define what area(s) you want to stream as constantly updated images from your desktop to your controller display. This can result in extremely intuitive controller interfaces for certain types of applications and games.
The tables below show a non-exhaustive list of examples of controllers to which this technology can be applied for and a non-exhaustive list of examples of receivers to which this technology can be applied for. Controller Input Output Communication
Touch-based User touch Controller interface Built-in WiFi or SmartPhone interaction with a and (transformed) USB, Device or Tablet controller user interaction results Driver
interface (e.g. a from the receiver in dedicated app or the form of textual browser-based and graphical controller interface) elements.
Motion Motion events N/A Built-in WiFi or
Controller created by a user of USB, Device
Camera an object in the Driver
environment
monitored by the
device
Audio Audio events created N/A Built-in WiFi or
Recording by a user of an object USB, Device
Device in the environment Driver
monitored by the
device
Game User interaction with Force Feedback Built-in WiFi or Controller game controller USB, Device elements(e.g. buttons Driver and joysticks)
Brain- User Brainwave N/A Built-in WiFi or Computer captured data USB, Device Interface Driver Mobile Phone SMS Message SMS Message 3G or Mobile
Phone Network
Keyboard User keystroke N/A Keyboard or events USB cable, protocol & device driver
Mouse User mouse events N/A Keyboard or
USB cable, protocol & device driver
Computer Application process Textual, data Wired or Application or event structure, graphical, Wireless LAN, process interaction Inter-Process
Communication, Email, Service, Log file, API, etc.
Controller Input Output Communication
Online Service Service event, Service event, Wired LAN, process or message process or message Inter-Process
Communication, Email, Service, Log file, API, etc
(Universal) User interaction with Depending on output Infrared, Wi-Fi
Remote controller capabilities of the
Controller elements(e.g. remote controller
buttons) (e.g. LCD display)
(Video) Images/Video Depending on output FireWire, USB, Camera sequence capabilities of the Wi-Fi,
camera (e.g. LCD BlueTooth display)
Hardware Sensor Capture Data Depending on output Built-in WiFi or Sensor capabilities of the USB, Device sensor (e.g. LCD Driver, Custom display) Protocols
Dedicated Depending on Depending on Depending on Hardware hardware/ firmware hardware/ firmware hardware/ (e.g. home capabilities capabilities firmware automation) capabilities
Figure imgf000022_0001
with • Operating System Commands, Inter-Process
Operating Communication
• Application Interface calls,
System and , Email, Service, application/ • Data input emulation, Log file, API, services etc.
• Batch or shell files/commands,
• Macro playback
• User interaction emulation (e.g.
selecting a window, clicking a specific
button in an application, selecting a
menu in an application, inputting text)
Game • Keyboard, Mouse, Game Controller Built-in WiFi or Console emulation, USB
• System/Firmware Commands (e.g.
initiate built-in video playback support)
Touch- • SMS Message Wi-Fi or 3G based
• Push notification
SmartPhon
e or Tablet • Mobile browser pages/metadata
• Native app data
Connected • Firmware Remote Control Command Wi-Fi, LAN or TV emulation Infrared
• CTV App user interaction (e.g. emulate
keystrokes)
Traditional • Controller commands converted to Infrared TV Infrared remote control commands Home • Controller commands converted to Infrared
Entertainm Infrared remote control commands
ent Device
(e.g. STB,
DVR)

Claims

A computer program, enabling remote control of any apparatus of a predefined first set by any control device of a predefined second set, said computer program comprising: means for inputting remote control capabilities of a control device (of the first set) and an apparatus (of the second set); and means to configure, based on said input, the arrangement of the control device and the apparatus, in order to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus.
The computer program of claim 1, wherein the set of remote control capabilities of the control device, selected for remote control of the apparatus, being less than the required remote control capabilities of the apparatus.
The computer program of claim 2, wherein the means to configure arranges that one or more of the remote control capabilities of the control device represent a grouping of at least two of said apparatus remote control capabilities.
The computer program of claim 1, 2 or 3, wherein the means to configure, comprising:
means to link the control device capabilities into first predefined abstract capabilities;
means to link the apparatus capabilities into second predefined abstract capabilities; and
means to link said control device and apparatus capabilities by deriving a mapping from predefined information between their respective abstract capabilities.
5. The computer program of claim 3, whereby the grouping results in grouping of commands selected from the group of: keyboard commands, mouse commands, media commands, explorer commands, system commands, window commands, application specific commands, API commands, emulator commands, batch commands, script commands, control line commands, remote procedural call commands, inter process call commands, service commands, file system commands, database commands, executable commands, process commands, messaging commands, user interface commands, menu commands, virtual machine commands, just in time commands, screen capture and drawings commands, input commands, output commands, data capturing commands, data scraping commands or browser commands. 6. A computer implemented method, for enabling an arrangement comprising a control device and an apparatus, to remote control the apparatus by the control device, the method comprising: providing remote control capabilities of a control device and an apparatus as input for a computer program; and configuring the arrangement of the control device and the apparatus by the computer program based on those inputs, such that such arrangement becomes capable to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus.
7. A computer implemented method for remote control of an apparatus by a control device, the method comprising: performing the method of claim 6; receiving user instruction on said control device to generate control data; adapting said generated control data in accordance with the configuration and transmit said adapted such adapted control data to the apparatus; and receive said transmitted adapted control data by the apparatus and perform the control operation in accordance therewith.
8. An arrangement of a control device, an apparatus and means enabling transmission of at least control data from said control device to said apparatus, wherein at least one of said control device, apparatus or said transmission means, being adapted for adapting in the control device generated control data in response to user instructions, in way suitable for the apparatus, said arrangement being configured by the computer program as in any of the claims 1 to 5.
9. A computer program for adapting control data, used for remote control of an apparatus by a control device, the control data being generated at the control device; the computer program being executed on either one of the control device, the apparatus or any transmission means used for transmitting the control data from said control device to said apparatus, the computer program comprising means for receiving control data; and means for adapting said received control data, said adapting means being provided by use of the computer program of any of the claims l to 5.
10. A method for computer aided design of a user remote control interface for an apparatus, suitable for use on a control device, the method comprising the steps of: providing remote control capabilities of a control device and an apparatus as input for a computer program of any of the claims 1 to 5;
providing at least one by the user demanded grouping of said apparatus remote control capabilities into a single remote control capabilities of the control device as input for a computer program of any of the claims 1 to 5; and
configuring the arrangement of the control device and its remote control user interface and the apparatus by the computer program based on those provided inputs, such that such arrangement becomes capable to adapt control data and transmit such adapted control data from said control device to said apparatus for remote control of said apparatus.
11. A method of computer aided creation of computer program product of a target apparatus as described in any of the claims 1-5, the method comprising the steps of: inputting the remote control capabilities of the target apparatus; loading a generic template of the computer program product; and adapting the generic template based on the inputs to obtain the computer program product. 12. The method of claim 11, wherein the generic template describes a generic flow with a plurality of steps; and wherein the step of adapting comprising selecting those steps of the generic flow being of use for the computer program product specially targeting a target apparatus. 13. Use of a symbol representation on the control device of the grouping of claim 3 or 5.
14. The use of claim 13, wherein the symbol representation being selected from the group of: icon representation, text button representation, or hyperlink representation.
PCT/EP2012/060402 2011-06-01 2012-06-01 Systems and methods for configurable remote control Ceased WO2012164076A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201161492295P 2011-06-01 2011-06-01
US61/492,295 2011-06-01

Publications (1)

Publication Number Publication Date
WO2012164076A1 true WO2012164076A1 (en) 2012-12-06

Family

ID=46354173

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2012/060402 Ceased WO2012164076A1 (en) 2011-06-01 2012-06-01 Systems and methods for configurable remote control

Country Status (1)

Country Link
WO (1) WO2012164076A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002007122A2 (en) * 2000-07-13 2002-01-24 Universal Electronics Inc. Customizable and upgradable devices and methods related thereto
WO2002031978A2 (en) * 2000-10-10 2002-04-18 Koninklijke Philips Electronics N.V. Programmable remote control device
WO2010111426A1 (en) * 2009-03-24 2010-09-30 Logitech Europe S.A. Apparatus and method of operation for a remote control system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002007122A2 (en) * 2000-07-13 2002-01-24 Universal Electronics Inc. Customizable and upgradable devices and methods related thereto
WO2002031978A2 (en) * 2000-10-10 2002-04-18 Koninklijke Philips Electronics N.V. Programmable remote control device
WO2010111426A1 (en) * 2009-03-24 2010-09-30 Logitech Europe S.A. Apparatus and method of operation for a remote control system

Similar Documents

Publication Publication Date Title
US10552518B2 (en) Administration of web page
US10708534B2 (en) Terminal executing mirror application of a peripheral device
US9621434B2 (en) Display apparatus, remote control apparatus, and method for providing user interface using the same
US9971417B2 (en) Network control system, control apparatus, controlled apparatus, and apparatus control method
US20120084729A1 (en) Proactive browsing method with swing gesture in free space
US20120075529A1 (en) Techniques for displaying data on a secondary device while displaying content on a television
KR20140144104A (en) Electronic apparatus and Method for providing service thereof
KR101462057B1 (en) Apparatus and Computer Readable Recording Medium Storing Program for Providing User Interface for Sharing Media content in Home-Network
KR20120079208A (en) Methods and apparatus for remote controlling peripheral device in mobile communication terminal
KR20150043422A (en) Input device using input mode data from a controlled device
US20160006971A1 (en) Display apparatus and controlling method thereof
KR102344580B1 (en) Providing method, apparatus and computer-readable medium of providing a content control interface through interworking with an open type display device and a user terminal
US20120278724A1 (en) Control method of a terminal display device
CN112449227B (en) Interaction method and device for touch screen application compatible with remote controller operation and smart television
US20120079532A1 (en) Techniques for developing a television user interface for a secondary device
US20120079533A1 (en) Techniques for developing a customized television user interface for a secondary device
CN117915141A (en) A local key control method and display device
WO2012164076A1 (en) Systems and methods for configurable remote control
KR102303286B1 (en) Terminal device and operating method thereof
CN117807339A (en) Display equipment and method for loading webpage through browser
CN102761651A (en) Terminal display device and control method thereof
CN102810102A (en) Method for file sharing operation
KR102042177B1 (en) Terminal and method of providing user interface thereof
KR102330475B1 (en) Terminal and operating method thereof
Huang et al. A smart universal remote control based on audio-visual device virtualization

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12729394

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12729394

Country of ref document: EP

Kind code of ref document: A1