GB2518410A - Entertainment Device and Method - Google Patents

Abstract

An entertainment device comprises communication means operable to receive game input data packets over a network, a buffer operable to store game input data obtained from the game input data packets and to read out stored game input data to a videogame, a target packet acceptance percentage setter operable to set a target packet acceptance percentage, and a target latency setter operable to set a target latency. The communication means is operable to accept received game input data packets having a network latency within the target latency and to discard received game input data packets having a network latency greater than the target latency. The target latency setter is operable to modify the target latency until, in operation, the communication means accepts at least the target packet acceptance percentage of received game input data packets. The adjustment of the target latency can extend the period during which data packets can be accepted by the communication means. Also disclosed is a portable device which executes an application which causes the portable device to emulate an input peripheral of an entertainment device and generate game input data appropriate to the input peripheral and transmit the game data to an entertainment device.

Description

ENTERTAINMENT DEVICE AND METHOD

The present invention relates to an entertainment device and method.

Conventional videogames use a variety of input mechanisms. Most common among these are keyboards. mice and handheld game controllers such as the Sony ® SixAxis ® controller.

More recendy, a number of videogames use a monoscopic. stereoscopic or depth-sensitive camera as an input device.

However, there is also a class of games that are characterised by the use of bespoke input hardware. Such games include Buzz (which uses a controller with a single large quiz show' style button), SingStar Guitar (which uses a mock guitar with control buttons on the frets) and the original SingSiar family of games. which use one or more wired or wireless hand held microphones that are visually distinguished using a coloured cuff and are also logically distinguishable by the entertainment device hosting the game, so that the performances of different singers can be distinguished and scored.

However, this latter class of games has the disadvantage that whilst the bespoke input hardware contributes to the gameplay experience, it also contributes to the cost of the game.

This acts as a barrier to entry for the game and so can limit sales. Moreover, the use of bespoke hardware also acts as a barrier to certain forms of game distribution, such as the increasingly popular digital download market.

The present invention aims to mitigate these proNems of cost and distribution.

In a first aspect, an entertainment device is provided in accordance with claim 1.

In a second aspect, a portable device is provided in accordance with claim 6.

In a third aspect, a method of processing game input data is provided in accordance with claim i2.

In a fourth aspect, a method of generating a virtual peripheral for an entertainment device is provided in accordance with claim 17.

Further respective aspects and features of the invention are defined in the appended claims.

Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which: Figure 1 is a schematic diagram of an entertainment device in accordance with an embodiment of the present invention.

Figure 2 is a schematic diagram of an entertainment system in accordance with an embodiment of the present invention.

Figure 3 is a schematic diagram of a signalling scheme in accordance with an embodiment of the present invention.

Figures 4A-C ate schematic diagrams of packet latency distributions and acceptance cut-off timings, in accordance with an embodiment of the present invention.

Figure 5 is a flow diagram of a method of processing game input data, in accordance with an embodiment of the present invention.

Figure 6 is a flow diagram of a method of generating a virtual peripheral for an entertainment device, in accordance with an embodiment of the present invention.

An entertainment device and method are disclosed. In the following description, a number of specific detafis are presented in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to a person skilled in the art that these specific details need not be employed to practice the present invention.

Conversely, specific details known to the person skilled in the art are omitted for the purposes of clarity where appropriate.

Examples of suitable entertainment devices include but are not limited to videogame consoles, such as the PlayStation 3 ® and also the PlayStation 4 ®.

Figure 1 schematically illustrates the overall system architecture of the Sony® PlayStation 3® entertainment device. A system unit 10 is provided, with various peripheral devices connectable to the system unit.

The system unit 10 comprises: a Cell processor 100; a Rambus® dynamic random access memory (XDRAM) unit 500; a Reality Synthesiser graphics unit 200 with a dedicated video random access memory (VRAM) unit 250; and an 1/0 bridge 700.

The system unit 10 also comprises a Blu Ray® Disk BD-ROM® optica] disk reader 430 for reading from a disk 440 and a removable s'ot-in hard disk drive (HDD) 400. accessible through the I/O bridge 700. Optionally the system unit also comprises a memory card reader 450 for reading compact flash memory cards, Memory Stick® memory cards and the like, which is similarly accessible through the 110 bridge 700.

The I/O bridge 700 also connects to four Universal Serial Bus (USB) 2.0 ports 710; a gigabit Ethernet port 720; an IEEE 802.1 lb/g wireless network Wi-Fi) port 730; and a Bluetooth® wireless link port 740 capable of supporting up to seven Bluetooth connections.

In operation the I/O bridge 700 handles all wireless, JJSB and Ethernet data, including data from one or more game controllers 751. For example when a user is playing a game, the 1/0 bridge 700 receives data from the game controller 751 via a Bluetooth link and directs it to the Cell processor 100, which updates the culTent state of the game accordingly.

The wireless. USB and Ethernet ports also provide connectivity for other peripheral devices in addition to game controllers 751, such as: a remote control 752; a keyboard 753; a mouse 754; a portable entertainment device 755 such as a Sony PlayStation Portable® entertainment device; a video camera such as an EyeToy® video camera 756; and a microphone headset 757. Such peripheral devices may therefore in principle be connected to the system unit 10 wirelessly; for example the portable entertainment device 755 may communicate via a Wi-H ad-hoc connection, whilst the microphone headset 757 may communicate via a thuetooth link.

The provision of these interfaces means that the PlayStation 3 device is also potentially compatible with other peripheral devices such as digita' video recorders (DVRs), set-top boxes, digital cameras, portable media players, Voice over IP telephones, mobile telephones, printers and scanners.

In addition, a legacy memory card reader 410 may be connected to the system unit via a USB port 710, enabling the reading of memory cards 420 of the kind used by the PlayStation® or PlayStation 2® devices.

Typically the game controller 751 is operable to communicate wirelessly with the system unit via the Bluetooth link. However, the game controller 751 can instead be connected to a USB port, thereby also providing power by which to charge the battery of the game controller 751. In addition to one or more analog joysticks and conventional control buttons, the game controller is sensitive to motion in 6 degrees of freedom, corresponding to translation and rotation in each axis. Consequently gestures and movements by the user of the game controller may be translated as inputs to a game in addition to or instead of conventional button or joystick commands. Optionally, other wirelessly enabled peripheral devices such as the portable entertainment device 755 or the PlayStation Move (RTM) 758 may be used as a controller. In the case of the portable entertainment device, additional game or control information (for example. control instructions or number of lives) may be provided on the screen of the device. In the case of the PlayStafion Move, control information may be provided both by internal motion sensors and by video monitoring of the light on the PlayStation Move device. Other alternative or supplementary control devices may also be used, such as a dance mat (not shown), a light gun (not shown), a steering wheel and pedals (not shown) or bespoke controllers, such as a single or several large buttons for a rapid-response quiz game (also not shown).

The remote control 752 is also operable to communicate wirdessly with the system unit 10 via a Bluetooth link. The remote control 752 compnses controls suitable for the operation of the Blu Ray Disk BD-ROM reader 430 and for the navigation of disk content.

The Blu Ray Disk BD-ROM reader 430 is operable to read CD-ROMs compatible with the PlayStation and PlayStation 2 devices, in addition to conventional pre-recorded and recordable CDs, and so-called Super Audio CDs. The reader 430 is also operable to read DVD-ROMs compatible with the PlayStation 2 and PlayStation 3 devices, in addition to conventional pre-recorded and recordable DVDs. The reader 430 is further operable to read BD-ROMs compatiNe with the fflayStation 3 device, as weB as conventional pre-recorded and recordable Blu-Ray Disks.

The system unit 10 is operable to supply audio and video, either generated or decoded by the PlayStation 3 device via the Reality Synthesiser graphics unit 200, through audio and video connectors to a display and sound output device 300 such as a monitor or television set having a display 305 and one or more loudspeakers 310. The audio connectors 210 may include conventional analogue and digital outputs whilst the video connectors 220 may variously include component video, S-video, composite video and one or more High Definition Multimedia Interface (HDMI) outputs. Consequently, video output may be in formats such as PAL or NTSC, or in 720p, I 080i or 1080p high definition.

Audio processing (generation, decoding and so on) is performed by the Cell processor 100.

The PlayStation 3 device's operating system supports Dolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), and the decoding of 7.1 surround sound from Blu-Ray® disks.

The system unit may communicate with a video camera 756 comprising a single charge coupled device (CCD), an LED indicator, and hardware-based real-time data compression and encoding apparatus so that compressed video data may be transmitted in an appropriate format such as an intra-image based MPEG (mofion picture expert group) standard for decoding by the system unit 10. In embodiments the camera LED indicator is arranged to illuminate in response to appropriate control data from the system unit 10, for example to signify adverse lighting conditions. Embodiments of the video camera 756 may variously connect to the system unit 10 via a USB, Bluetooth or Wi-fl communication port.

Embodiments of the video camera may include one or more associated microphones and also be capable of transmitting audio data. In embodiments of the video camera, the CCD may have a resolution suitable for high-definition video capture. In use, images captured by the video camera may for example be incorporated within a game or interpreted as game control inputs.

In general, in order for successful data communication to occur with a peripheral device such as a video camera or remote control via one of the communication ports of the system unit 10, an appropriate piece of software such as a device dnver should be provided. Device driver technothgy is well-known and will not be described in detail here, except to say that the skilled man will be aware that a device driver or similar software interface may be required in the present embodiment described.

Software instructions implemented by the Cell processor 100 and/or the RSX 200 may be supplied at manufacture and stored on the HDD 400, andlor may be supplied on a data calTier or storage medium such as an optical disk or solid state memory, or via a transmission medium such as a wired or wireless network or internet connection, or via combinations of these.

The software supplied at manufacture comprises system firmware and the PlayStation 3 device's operating system OS). In operation, the OS provides a user interface enabfing a user to select from a variety of functions, including playing a game, listening to music, viewing photographs. or viewing a video. The interface takes the form of a so-called cross media-bar (XMB), with categones of function arranged honzontally. The user navigates by moving through the function icons (representing the functions) horizontally using the game controller 751, remote control 752 or other suitable control device so as to highlight a desired function icon, at which point options pertaining to that function appear as a vertica'ly scrollable list of option icons centred on that function icon, which may be navigated in analogous fashion.

However, if a game, audio or movie disk 440 is inserted into the BD-ROM optical disk reader 430, the PlayStation 3 device may select appropriate options automatically (for example, by commencing the game), or may provide relevant options (for example, to select between playing an audio disk or compressing its content to the HDD 400).

In addition, the OS provides an on-line capability, including a web browser, an interface with an on-line store from which additional game content, demonstration games (demos) and other media may be downloaded, and a friends management capability, providing on-line communication with other PlayStation 3 device users nominated by the user of the current device; for example, by text, audio or video depending on the peripheral devices available.

The on-line capability also provides for on-line communication, content download and content purchase dunng play of a suitably configured game, and for updating the firmware and OS of the PlayStation 3 device itself. It will be appreciated that the term "on-line" does not imply the physical presence of wires, as the term can also apply to wireless connections of various types.

Turning now to Figure 2, in an embodiment of the present invention, the PS3 10 is connected to a local network, such as a home network. Typically such a network is co-ordinated by a router 1000 having a wireless capability (though the PS3 itself may have a wired connection to the router). The router itself is typicafly able to connect to the internet via a telephone landline or cable using ADSL or similar, thereby providing on-line access to devices logically connected to the router.

In addition, in the embodiment one or more users' own portable devices (2000A,B,C), having WiFi® capability, are also logically connected to the local network, for example via the router.

In the embodiment, such a portable device 2000 comprises a display and a microphone, and is capable of running a downloaded app. Examples of such a device include the Apple ® iPhone ®, iPod touch ®, iPad ®, iPad mini ® or the like, typically running a version of the loS operating system; or similarly a smartphone or tablet running a version of Android ® or Windows 8 ® or similar, or again similarly the Sony PS Vita ® or PSP ®.

In an embodiment of the present invention, the portable device is operable to run a companion application, or app', described later herein, that relays signals from the microphone of the portable device to the entertainment device.

The entertainment device similarly runs a program, such as a version of SingStar. In the description below, it will be appreciated that SingStar is a non-limiting example provided for clarity and simplicity of explanation.

Turning now to Figure 3, an initial operation of the entertainment device and portable device is described in accordance with an embodiment of the present invention.

In a first step (s30 I) the entertainment device connects to the network.

On a portable device, upon running the companion app stored thereon, the app requests / polls (302) device information from the network (for example by performing a network broadcast on a predetermined UDP port), and typically receives responses (s303) from the respective devices on the network via the router. The companion app then displays to a user those devices that are capable of hosting the SingStar game, so that the user can select (s304) the appropriate entertainment device. For example. the companion app may only list entertainment devices identifying themselves as a PS3 or PS4. If there is only one such entertainment device currently on the network, then optionally the display step is bypassed and that entertainment device is selected automatically.

The companion app and the entertainment device then optionally perform any suitable mutual authentication process (s306A,B). for example to mitigate the risk of malware on the portable device attempting to link to the entertainment device and compromise it. This authentication may use a locally held key in the app and/or device, and/or may involve authentication for one or both parties via an external server (not shown) accessed via the router and the internet.

Optionally the entertainment device then sends a signal (s307) to the companion app, providing an identifier for the microphone (such as ID#l, ID#2 etc) that the app is to emulate.

The app may then optionally provide a graphical illustration of the microphone in accordance with this ID non its own display. For example, a picture of a legacy SingStar microphone may be displayed with a colour cuff selected in response to the ID (for example, a red or blue cuff). Due to the virtual nature of the microphone, other features may easily be included on it or with it, such as an audio level meter responsive to the user's voice.

Consequently, different portable devices can connect to the entertainment device and each may display a virtual microphone with a respective visual identifier familiar to users of previous versions of the SingStar game and the original bespoke input devices.

On the entertainment device, the relevant steps above may be carried out by the operating system, the SingStar game, or a combination of the two.

It will be appreciated that optionally the companion app may be run on a portable device before the SingStar game is run on the entertainment device. In this case, once the relevant entertainment device has been sdected by the app at step s304, the app may send a request (s305) to start the SingStar game on the entertainment device. This request may be automatic, manually initialised by a user, and/or triggered after a predetermined period in which one of the subsequent steps of Figure 3 has not occurred. In this way, the game can be launched using the portable device, without the need to locate and use the conventional game controllers to navigate to it.

It will be appreciated that this principle can be extended further to a wake-on-WiFi option, wherein the entertainment device is turned on by a signal from the companion app via the network and then bunches the SingStar game, so avoiding the need to use any controller other than the portable device itself. If the entertainment device only has a wake-on-Bluetooth ® function then if necessary to emulate a conventional controller using such a Bluetooth function, a Bluetooth enabled portable device can be paired with the entertainment device in advance for this purpose.

Referring back to Figure 2, the or each companion app is now in communication with the SingStar game on the entertainment device (10) using WiFi via the respective host portable devices (2000A,B,C) and the router (1000).

The user may now control the game in a conventional manner, for example using a conventional hand-held controller to navigate through the game and select a game mode, if applicable, and a song to sing along to. Alternatively the (2000A,B,C) app can provide virtual buttons for navigation, the data identifying button presses being sent via the local network to the SingStar game. If the game navigation is relatively comp'ex, the dispbyed virtual buttons may be context sensitive, based on information about the current game state provided to the app from SingStar via the local network. Where multiple portable devices are connected, optionally only one companion app may be enabled in this way at any time; for example either the first device to connect may be treated as the master navigation device for all play, or some or all navigation control may pass to each player's device during turn-based play.

Once a song has been selected, the user of the portable device can press a virtual start' button displayed by the app to begin playback.

A SingStar game. as an example of a typical game mechanic, operates by storing a marked-up pitch track that is synchronised to a corresponding music video. A user's performance of the song is then compared to the marked-up track; the pitch and optionally the timing accuracy of the user's performance is compared to the marked-up pitch track and used to generate a score.

Notably this comparison is preferably performed in close to real time, so that the game feels responsive to the user by providing visual feedback of their accuracy.

In order to provide this rapid comparison and feedback, audio data representing the user's speech is typically transmitted by the companion app every 5.3ms (it will be understood that this is a non-limiting example, and that time slices in the order of l-2Oms or more are also possible, depending on the desired responsiveness of the particu'ar game).

The audio data is placed in a circular buffer capable of holding for example up to I second of audio data. The SingStar game then typically reads data from the buffer using a first position offset, whilst received data is written to the buffer at a second position offset.

Preferably, the latency between singing a portion of the song and comparing it / displaying feedback is in the order of 20-4Oms. This can be used to determine the required relative offset between the reading and writing positions in the buffer, or equally to determine how full the buffer is allowed to get whilst being read.

When using the original bespoke hardware, comprising a wired microphone, this preferred latency was easy to achieve as it was only a function of transmission and/or packetisation and processing between the microphone and the entertainment device. Similarly with bespoke wireless microphones using a Bluetooth-type connection, the latencies were typically small in the short direct link between the microphone and the entertainment device.

However, for a WiFi system using an arbitrary portable device linked to the entertainment device via a router that is serving a household, the latency is less predictable.

It will be appreciated that if the audio data is delayed in the network, the buffer can become empty, resulting in apparent silence. However, typically the audio data is not all delayed by a fixed amount -rather, different packets of data are delayed by different amounts, resulting in intermittent and out-of order reception.

If packets are received within the preferred latency period, they can be inserted into their proper place in the buffer without any perceptible effect on the audio then sent to the SingStar game. However, if an audio packet is not received within the preferred latency period, then one must either wait for the packet (resulting in variable latencies of unknown length) or treat the packet as lost, resulting in gaps in the buffered audio used by the game.

Referring to Figure 4A, this illustrates a possible distribution of packet latency in the home network between the portaNe device and the entertainment device via the router, with time in milliseconds on the x-axis, and relative packet count on an arbitrary scale on the y-axis.

Typically the distribution is a skewed Gaussian distribution, with a large percentage of packets arriving within a prefelTed latency penod T of, for example, l5ms. In the example shown in Figure 4A, 95% of packets are received within the preferred latency period T. However, an appreciable portion of the packets (in this example 5%) arrive after this period.

Consequently, as suggested above one option is to simply discard packets that have a latency greater than the preferred latency, and to set the preferred latency to, for example, l5ms, or 35ms. The choice of i5ms or 35ms is illustrative and corresponds to an example desired overall latency of 2Oms or 4Oms, which also includes the approximately Srns needed for the accumulation of the audio to send in the transmitted packet. 20-4Ums corresponds to 1024- 2048 samples at 48kHz.

The result is that the continuity of the audio data in the circular buffer will be dependent upon the latency distribution of the received audio data packets in the network. If the latency in the home network is predominantly less than the preferred latency, then the continuity of the audio data will be high because almost all packets are received in time to be included in the circular buffer (i.e. within the preferred latency). However, if the latency in the home network is longer, so that an appreciable proportion of packets take longer to reach the entertainment device than is permitted by the preferred latency, then the continuity of the audio data in the circular buffer will be poor.

Because the performance of any given user's home network is unknown, and moreover can change significantly in response to changes in environmental conditions (for example. when a user is hosting a party with lots of people in the house, resulting in an influx of mobile phones and the like also transmitting at 2,4GHz I 50Hz or other WiFi frequencies that may cause packet loss for the present system). this makes it difficult to anticipate the performance of the game, or to set a reasonable preferred latency at the point of game publication.

Consequently in an embodiment of the present invention, a goal-oriented control of the latency is used. Referring now to Figure 4B (using the same axes as Figure 4A), a target latency is set in response to a target packet loss. Hence for example the latency may be set to a time TG within which 95% of packets are received (or equally within which 5% of packets are lost). As can be seen from Figure 4W in adverse WiFi / network conditions. T > T This target latency T6 may be updated periodically (for example. every 10 seconds). It may also be initialised during the game navigation phase for a portable device, prior to singing; that is to say, the portable device may transmit ambient audio from the microphone prior to transmitting the begin playing song' signal, so that an initial suitable target latency TG for current conditions can be evaluated using sacrificial audio data, and then updated during play.

The target latency T may also have an upper bound. However, because this upper bound is not used unconditionally (like the 2Oms or 4Oms Tp examples above) it can be set higher, representing a temporary tolerance to a pathological condition. Hence for example the upper bound may be lOOms for receiving 95% of packets. Above this point, a fail-safe operation may occur in the game. It will be appreciated that different upper bounds may be provided for different percentage goals.

RefelTing now also to figure 4C, similarly the target latency T6 may have a lower bound.

That is to say, if the target latency TG falls to below the preferred latency Tp (for example, to within l5ms). then rather than reduce the latency further, it maybe preferable to remain at the preferred latency T and allow the percentage of received packets to increase above the goal.

Hence in Figure 4C, the value of T0 to achieve 95% packet reception would be (for example) lOms, but since a latency Tp of lSms is acceptable, then 97% packet reception can be achieved at this higher latency. It will be appreciated that different lower bounds may be provided for different percentage goah.

It will also be appreciated that optionafly different percentage goals may be associated with different songs. or with different time points within a song. This is because the information density of different songs (or parts of songs) vanes, and so it is possible that the associated density of comparison points (tones or phonemes, for example) also varies within or between songs. Consequently the need for a more complete set of audio data in the circular buffer can also vary between and within songs.

Hence for example, during an initial quiet period of a track with no singing, the packet percentage goal may be set comparatively low, for example 80%, as there may be no active scoring during this period. Meanwhile for a song with a tht of sustained vocalisations, then during the song (or at least the vocalisations) the packet percentage goal may be higher. for example 95%. This may provide a sufficiently good supply of pitch values to track the singing. Meanwhile dunng a fast rap the percentage goal may be higher still, for example 98%, in order to accurately track phoneme matches in an almost complete audio sequence, if such matching is required.

Thus more generally, a target permissible packet latency TG for incoming audio data packets is calculated in response to a target percentage of packets to be received and the actual distribution of packets received during an evaluation period. This target permissible packet latency may optionally be subject to a lower bound representing a prefelTed packet latency T, so that if T < T2. the percentage of received packets is allowed to exceed the goal percentage, but if the network latency increases so that the goal percentage is not met at the preferred packet latency Tp, then a new target permissible packet latency TG is again calculated periodically to meet the goal percentage; subject to an upper bound latency indicative of a pathological network problem.

In an embodiment of the present invention, as noted above the target permissible packet latency TG incrementally changes in response to the percentage of audio data packets received within the target latency in a given evaluation period. If the percentage of audio data packets received within the target latency does not meet the goal percentage in that period, then the target permissible packet latency is increased, either in fixed steps or responsive to the percentage shortfall. If the percentage of audio data packets received within the target latency exceeds the goal percentage in a period, then the target permissible packet latency is decreased, either in fixed steps or responsive to the percentage surplus. Optionally, hysteresis is provided by preventing the decrement ofT0 for N evaluation periods after incrementing it, andlor decrement steps can be smaller than increment steps.

Using the above techniques, advantageous'y it is possible to set a target for the percentage of audio data packets that should be received by the entertainment device, and consequently the entertainment device can modify a packet latency cut-off point in order to meet that percentage target, responsive to local conditions and optionally subject to upper and lower bounds.

This enables the SingStar game to be more responsive to the p'ayer whilst still receiving a desired proportion of audio packets. However it will also be appreciated that, as a result, there will frequently be a notable percentage of rejected or unprocessed audio packets received after the cut-off point, and hence a notable number of gaps in the audio data in the circular buffer.

Gaps in this audio data can have two effects. Firstly, they can mean that playback of the usei' s performance over the reference video performance may sound patchy. Secondly. they may make scoring the user's performance more difficult.

To mitigate the reduced playback quality, one or more of the following strategies may be adopted.

In an embodiment of the present invention, at least one portion of the received audio data is copied to a separate buffer. When the game encounters a gap in the audio in the circular buffer, then optionally the audio from the separate buffer is played instead of the gap.

To ensure that the substitute audio from the separate buffer is likely to be similar to the expected audio, optionally the audio in the separate buffer comprises one or more blocks of the most recently played valid audio from the circular buffer (for example one, two or more 5.3ms blocks of audio corresponding to one or two packet's worth of audio data).

Optionally this audio may be cross-faded into the existing audio at reproduction to reduce perceived discontinuities. Similarly the sdection, precise timing and/or start point of the substitute audio can be adjusted to reduce discontinuities with the audio data terminating at the start of the gap.

Where there is a gap in the audio of several such blocks, optionally then multiple instances of the substitute audio from the separate buffer can be played in succession, again optionally cross-faded to reduce apparent discontinuities.

Alternatively or in addition, other methods of generating representative audio for an arbitrary period may be employed, such as the use of granular synthesis based upon the audio in the separate buffer (and/or from the circular buffer).

Optionally it is aho possible to make use of the fact that there is a marked-up pitch associated with the reference performance in the played video, and also a reasonable expectation that the user is culTently trying to match the current pitch. As a result, it is possible to use synthesis effects such as granular synthesis and pitch-shifting to approximate a calculated or extrapolated version of what the user might be expected to be singing during the period of the missing audio.

Similarly, optionally it is possible to build up a corpus of vocalisations of the user at different pitches during one or more performances, so that these can be sdected as substitutes for gaps in accordance with the marked-up pitch. Optionally, such elements from a corpus can be cross-faded with the existing audio, and can also be blended with audio from the above mentioned separate buffer, or included as further source audio in a granular synthesis process.

For some SingStar games, more detailed mark-up may exist, such as an indicator of expected voiced and unvoiced sounds, in addition to pitch. 1-lence the substitution schemes above can select either voiced or unvoiced earlier sounds as applicable to the gap, and similarly the corpus can include voiced and unvoiced example sounds.

This principle can be expanded in accordance with the sophistication of the mark-up available; if in addition to pitch, voiced and unvoiced markers, the voiced markers indicate an expected vowel sound for example, then a corpus of substitute sounds can be accumulated and selected appropriately. Similarly if the mark-up includes phonemes, then a corpus of phonemes can be accumulated for a user, and the most likely current phoneme based upon immediately preceding recognition results can be selected for a gap; that is to say, a phoneme can be selected responsive to what phoneme is expected at the current time based upon the markup, andior what phoneme is expected to follow the detected phoneme uttered prior to the gap, based upon the markup.

In this way, increasingly sophisticated substitution strategies are facilitated by the increasing level of detail of audio mark-up available for the current reference audio/video track: Without specific reference to the mark-up, the most recendy played audio in the circular buffer can be repeated dunng a gap in a variety of ways, including cross-faded repetition and use in granular synthesis.

With reference to pitch mark-up, such audio can either be pitch-shifted appropriately, or exemplars at appropriate pitches can be stored and selected as substitutes. Again blending and synthesis may be used as appropriate to generate the actua' substitute audio presented to the game.

With reference to voicing mark-up, appropriate vowel or fricative exemplars may be chosen for use as described above.

With reference to phoneme mark-up, appropriate phoneme exemplars may be chosen for use as described above.

Using such techniques, it is possible to mitigate the discarding of audio data packets received late from the portable device by the entertainment device.

Next, to mitigate scoring the user's performance, one or more of the following strategies may be adopted.

If a gap in the audio data coincides with a first game measuring/scoring point, then optionally evaluation at that first game measuring/scoring point is suspended until the next measuring/scoring point has been evaluated. The value to ascribe to the first game measuring/scoring point may then the interpolated from scores obtained for the previous and next measuring/scoring points.

Alternatively or in addition, the substituted audio data described previously herein could be evaluated at that game measuring/scoring point; although in effect this means that the game is evaluating its own ability to substitute audio. However, the number of evaluation points at which this will occur will be comparatively small if packet acceptance rates are in the order of 95% or more, and so the effect on overall score accuracy will also be small.

Alternately or in addition. some pre-processing of the audio may be performed by the companion app running on the portaNe device. For example, simple pitch tracking and optionally also voiced/unvoiced/silence evaluations can be performed by use of a signal energy measure to determine silence vs. speech, followed by an autocorrelation measure to determine voiced (correlated) vs. unvoiced (uncorrelated) speech, and the correlation offset of the voiced speech will correspond to the pitch. Other pitch tracking strategies such as use of a comb filter will be appreciated by the skilled person. Computational load on the device will be relatively low, but can be lowered further by starting the correlation search based upon the pitch calculated for the preceding audio block or frame, and/or a pitch value sent to the portable device from the entertainment device that corresponds to the marked-up target pitch in the current reference track.

A bit pattern for a simple pitch value may then be transmitted. For example, a first bit could be a flag set to indicate 0 = voiced, I = unvoiced. Then N bits could be used to indicate pitch, to a 2N granularity. A flag of 0 and a pitch of 0 could indicate silence as the third possible state.

This bit pattern could occupy unassigned bits in a packet or other header, for example in a separate short packet sent between the portable device and the entertainment device. This short packet would be likely to have a different delay pattern in the network to the corresponding longer audio data packet, so improving the chance of at least one representation of the current audio being received within the target latency period T0, and so improving the rate of success for for scoring pitch and voiced/unvoiced timing within the game. Similarly, where the audio data is transmitted using UDP. this separate short packet may be send using the TCP protocol to ensure arnval at the entertainment device. Such pitch information could also be used to inform the audio gap masking strategies described previously, for example to appropriately pitch-shift substitute audio.

Alternatively or in addition, it can be appreciated that whilst a low latency is most perceptible for the user in relation to the reproduction of their audio by the game, latency with respect to scoring may be considered less time sensitive. If this is the case, then optionally the received audio data may also be buffered in a further scoring buffer used for audio analysis and scoring, and which is associated with a target latency Tus that is greater than the target latency T(} of the audio buffer used for audio playback, so that late packets which are rejected by the audio buffer may be accepted into the scoring buffer. In other words, a greater tolerance to latency in scoring the performance will reduce the number of gaps that need to be concealed in such a scoring buffer.

It will be appreciated that whilst the example SingStar game may be written to make use of the techniques described herein, it is also envisaged that a wrapper program similarly making use of the techniques described herein could be used in conjunction with a legacy SingStar game.

In this case, the wrapper program emulates the reception of the audio as if using the real bespoke microphones. by using the above techniques as appropriate in conjunction with the companion application on the or each portable device, and presents this transparently to the legacy game. The wrapper may also emulate some or all of the other input conuols described previously herein. In this way, legacy SingStar games may be made easily distributable by download, for example on the PlayStation Store ®. with little or no further modification beyond that required to run on the current entertainment device. Typically, such a wrapper program integrates with the legacy game by intercepting I/O calls from the original game, and/or by substituting a library or function of the original game with functions calling to the wrapper program.

Another example of such a wrapper program is where a network enabled TV, set-top box or other device accesses a cloud-hosted version of the game, provided by a service such as Gakkai ®, where the game is hosted on a cloud server, with outputs streamed to the TV, etc., and inputs sent from the TV to the cloud server. In this case, the wrapper program on the TV, etc.. receives the data from the companion application and presents it as appropriate input to the cloud-hosted version of the game (whether this is in a conventional input format, as if playing using original hardware, or in a format modified for a conventional use of the hardware inputs associated with the TV, set-top box, portable device etc., or a direct relay of the received inputs to the cloud server). Hence the TV etc., acts as an intermediaiy device operable to receive game input data packets from the or each portable device and to transmit game input data packets (in whatever suitable form) to the cloud system.

It will also be appreciated that the above example of SingStar and a microphone are non-limiting examples. More generally, the companion application on the portable device at least partially emu'ates the original bespoke input hardware, whether this is a microphone, guitar frets, drum kit, dance mat, or quiz button, thereby providing a virtual peripheral on the portable device for interaction with the main game on the entertainment device. Similarly, a video camera built in to such a portable device may be used in lieu of a bespoke video camera normally coupled to the entertainment device, to the extent applicable to the capabilities of the video camera on the portable device. Any such virtual peripheral will experience the same issues of variable signal timing over the WiFi network compared with an original real peripheral having a dedicated communications link to the entertainment device, and so can benefit from the techniques described herein.

Finally, it will be appreciated that whilst the above descnption has assumed a separate router in the network, it is envisaged that an entertainment device may provide direct WiFi network access to the or each portable device itself, for example using the WiFi Direct standard (also known as WiFi P2P). Hence to the extent that a router is mentioned in the above embodiments, its functions may be assumed to be those of a suitably enabled entertainment device, adapted as app] icable.

Hence in a summary embodiment of the present invention, an entertainment device (such that the PS3 10, a PS4, or other suitable device), comprises communication means (730) operable to receive game input data packets over a network; buffer means (500) operable to store game input data obtained from the game input data packets, and to read out stored game input data 1 5 to a videogame; target packet acceptance percentage setting means (e.g. the cell processor 100) operable to set a target packet acceptance percentage; and target latency setting means (e.g. the cell processor 100) operable to set a target latency TG -and wherein the communication means is operable to accept received game input data packets having a network btency within the target atency T0, and to discard received game input data packets having a network latency greater than the target latency T; and wherein the target latency setting means is operable to modify the target latency until in operation the communication means accepts at least the target packet acceptance percentage of received game input data packets.

Hence as explained previously, the target latency T0 is modified until the target percentage of packets is received within the target atency period, optiona'ly subject to maximum and/or minimum bounds on the target latency.

In an instance of this summary embodiment, the communication means is operable to accept received game input data packets having a network latency within the longer of the target latency TG or a predetermined preferred latency Tp, and to discard received game input data packets having a network latency greater than the longer of the target btency or the predetermined preferred latency. Hence as described previously, when the target latency falls below a prefelTed latency, rather than needlessly discard packets received after the target latency period, packets are accepted if they are received within the preferred latency period, thereby increasing the percentage of accepted packets above the target packet acceptance percentage when network conditions are comparatively good.

In an instance of this summary embodiment, the videogame comprises a user performance evaluation phase (i.e. the main karaoke game, dance game, etc.), and the target latency setting means is operable to modify the target latency responsive to a percentage of received game input data packets accepted by the communication means prior to the user performance evaluation phase. Hence as described previously, the companion application can send game input data before the user is actively playing the game, so that the latency target can be modified in advance.

In an instance of this summary embodiment, the target packet acceptance percentage setting means is operable to set the target packet acceptance percentage responsive to properties of all or part of a reference performance associated with the videogame. 1-lence as described previously, different songs or other game tasks (such as dance performances) may have associated with them on or more the target packet acceptance percentages, responsive to either the complexity of the song or dance, or responsive to the number or temporal density of comparison points in the song or dance.

In an instance of this summary embodiment, the entertainment device comprises audio generation means (e.g. cell processor 100 and/or RSX 200) operable to generate audio data to read out to the videogame in the place of gaps in the buffer caused by discarded game input data packets; wherein the audio generation means is operable to generate audio one or more processes selected from the list consisting of: i. substituting the gap with previously accepted audio data from the buffer; ii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target pitch value at the time of the gap; iii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target phoneme at the time of the gap; and iv. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to the detected phoneme uttered prior to the gap.

Hence as described previously, the entertainment device may provide substitute audio comprising duplicated or synthesised voca' elements, or a combination of the two; and the accumulation and selection of vocal dements can increase in sophistication responsive to the sophistication of the mark-up in the reference peiformance.

In a summary embodiment of the present invention, a portable device 2000 (such as a smartphone or tablet), comprises processing means (e.g. a known central processing unit) arranged in operation to run an application. The application in turn is operable to at least partially emulate a hardware input peripheral of an entertainment device 10, and operable to generate game input data appropriate to the emulated hardware input peripheral in response to a user interaction. Meanwhile, the portable device comprises wireless communication means (e.g. WiFi) operable to transmit game input data packets comprising the generated game input data over a network to an entertainment device. The emulation may take the form of using an input of the portable device in lieu of the original hardware input peripheral (e.g. real or virtual buttons, microphone and/or camera) and formatting the output data in a manner acceptable to a corresponding application on the entertainment device, such as for example a videogame or videogame wrapper application, as described previously. The display of the portable device may illustrate some or all of the emulated input peripheral, for example to provide visual feedback about the emulated state of the input peripheral (such as the assigned colour of the microphone cuff, andlor audio levels).

In an instance of this summary embodiment, the application is operable to also emulate some or all of a conventional input controller of the entertainment device. Hence for example, the application may display virtual buttons corresponding to some or all of an input controller's buttons.

In an instance of this summary embodiment the application is operable to cause the portable device to send a signal to the entertainment device causing the entertainment device to run an application. As described previously herein, this may use WiFi or Bluetooth as appropriate, and may involve waking the entertainment device first.

In an instance of this summary embodiment the wireless communication means of the portable device is operable to receive a signal from the entertainment device assigning a player number to the hardware input peripheral being at least partially emulated by the application. As noted previously herein, this may be used for example to assign a colour cuff to the virtual microphone emulated by the portable device.

In an instance of this summary embodiment the portable device comprises pre-processing means (e.g. the CPU) operable to generate from the game input data further data relating to a scoring metric of an application of the entertainment device that uses the game input data (for example pitch track data for a singing game, or inter-frame difference data, or skeletal pose data, for a dance game); and in which the wireless communication means operable to transmit the further data over the network to the entertainment device (for example in a separate packet, typically being smaller than the packets comprising the game input data).

iS In a summary embodiment of the present invention an entertainment system comprises an entertainment device as described in the relevant summary embodiment above, and one or more portable devices as described in the relevant summary embodiment above.

Turning now to Figure 5. a method of processing game input data comprises: In a first step sSlO, receiving game input data packets over a network; In a second step s520, storing game input data obtained from the game input data packets, for reading out to a videogame; In a third step s530, setting a target packet acceptance percentage; In a fourth step s540, setting a target latency; In a fifth step s550 being a communications step, accepting received game input data packets having a network latency within the target latency. and discarding received game input data packets having a network latency greater than the target latency; and wherein the fourth step of setting a target latency comprises periodically modifying the target latency until in the communications step at least the target packet acceptance percentage of received game input data packets are accepted.

It will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the apparatus as described and claimed herein are considered within the scope of the present invention, including but not limited to: -the communications step comprising accepting received game input data packets having a network latency within the longer of the target latency or a predetermined preferred latency, and discarding received game input data packets having a network latency greater than the longer of the target latency or the predetermined preferred latency; -the videogame comprising a user performance evaluation phase, and the step of setting a target latency comprises modifying the target atency responsive to a percentage of received game input data packets accepted by the communication means prior to the user performance evaluation phase; -the step of setting the target packet acceptance percentage comprising setting the target packet acceptance percentage responsive to properties of all or part of a reference performance associated with the videogame; and -the game input data comprising audio data, and generating audio to read out to the videogarne in the place of gaps in the buffer caused by discarded game input data packets using one or more sub-steps selected from the list consisting of i. substituting the gap with previously accepted audio data from the buffer; ii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target pitch value at the time of the gap; iii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target phoneme at the time of the gap; and iv. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to the detected phoneme uttered prior to the gap.

Turning now to Figure 6, a method of generating a virtual peripheral for an entertainment device on a portable device comprises: In a first step s610, emulating at least partially the functionality of a hardware input peripheral of an entertainment device; In a second step s620, generating game input data appropriate to the emulated hardware input peripheral in response to a user interaction; and In a third step s630, transmitting game input data packets comprising the generated game input data over a network to an entertainment device.

It will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the apparatus as described and claimed herein are considered within the scope of the present invention, including but not limited to: -the emulation step also comprising emulating some or all of a conventional input controller of the entertainment device; -sending a signal to the entertainment device causing the entertainment device to run an application; -receiving a signal from the entertainment device assigning a player number to the hardware input peripheral that is being at least partially emulated; and -generating from the game input data further data relating to a scoring metric of an application of the entertainment device that uses the game input data, and transmitting the further data over the network to the entertainment device.

It will be appreciated that the above methods may be carried out on conventional hardware suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware.

Thus the required adaptation to existing parts of a conventional equivalent device may be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, PROM. RAM, flash memory or any combination of these or other storage media, or realised in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use in adapting the conventional equivalent device. Separately, such a computer program may be transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these of other networks.

Claims (14)

1. CLAIMS1. An entertainment device, comprising: communication means operable to receive game input data packets over a network; buffer means operable to store game input data obtained from the game input data packets, and to read out storcd game input data to a vidcogame; target packet acceptance percentage setting means operable to set a target packet acceptance percentage; target latcncy setting mcans operable to set a target latency; wherein the communication means is operable to accept received game input data packets having a network latency within the target latency, and to discard received game input data packets having a network latency greater than the target latency; and wherein the target atency setting means is operable to modify the target latency until in operation the communication means accepts at least the target packet acceptance percentage of received game input data packets.
2. 2. The entertainment device according to claim 1, in which: communication means is operable to accept received game input data packets having a network latency within the longer of the target latency or a predetermined preferred latency, and to discard received game input data packets having a nctwork latency grcater than the longcr of the target latcncy or the prcdctermincd preferred latency.
3. 3. The entertainment device according to claim I or claim 2, in which: the vidcogamc comprises a uscr pcrformance eva'uation phasc; and the target latency setting means is operable to modify the target latency responsive to a percentage of received game input data packets accepted by the communication means prior to the user perrormance evaluation phase.
4. 4. The entertainment device according to any one of the preceding claims, in which: target packet acceptance percentage setting means is operable to set the target packet acceptance percentage responsive to properties of all or part of a reference performance associated with the videogame.
5. 5. The entertainment device according to any one of the preceding claims in which the game input data comprises audio data, the device comprising: audio generation means operable to generate audio data to read out to the videogame in the place of gaps in the buffer caused by discarded game input data packets; wherein the audio generation means is operable to generate audio one or more processes selected from the list consisting of: i. substituting the gap with previously accepted audio data from the buffer; ii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target pitch value at the time of the gap; iii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target phoneme at the time of the gap; and iv. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to the detected phoneme uttered prior to the gap.
6. 6. A portable device, comprising: processing means arranged in operation to run an application, the application in turn operable to at least partially emulate a hardware input peripheral of an entertainment device, and operable to generate game input data appropriate to the emulated hardware input peripheral in response to a user interaction; and wireless communication means operable to transmit game input data packets comprising the generated game input data over a network to an entertainment device.
7. 7. The portable device of claim 6, in which: the application is operable to also emulate some or all of a conventional input controller of the entertainment device.
8. 8. The portable device of claim 6 or claim 7, in which: the application is operable to cause the portable device to send a signal to the entertainment device causing the entertainment device to run an application.
9. 9. The portable device of any one of claims 6 to 8, in which: the wireless communication means is operable to receive a signal from the entertainment device assigning a player number to the hardware input peripheral being at least partially emulated by the application.
10. 10. The portable device of any one of claims 6 to 9, comprising: pre-processing means operable to generate from the game input data further data relating to a scoring metric of an application of the entertainment device that uses the game input data; and in which the wireless communication means operable to transmit the further data over the network to the entertainment device.
11. 11. An entertainment system, comprising: an entertainment device according to any one of claims I to 5; and one or more portable devices according to any one of claims 6 to 10; 12. An entertainment system according to claim 11, in which the entertainment device is hosted in a cloud system, and comprising: an intermediary device operable to receive game input data packets from the or each portable device and to transmit game input data packets to the cloud system.13. A method of processing game input data, comprising the steps of: receiving game input data packets over a network; storing game input data obtained from the game input data packets, for reading out to a videogame; setting a target packet acceptance percentage; setting a target latency; in a communications step, accepting received game input data packets having a network latency within the target latency, and discarding received game input data packets having a network latency greater than the target latency; and wherein the step of setting a target latency comprises periodically modifying the target latency until in the communications step at least the target packet acceptance percentage of received game input data packets are accepted.14. The method of claim 13, in which: the communications step comprises accepting received game input data packets having a network latency within the longer of the target latency or a predetermined preferred latency, and discarding received game input data packets having a network latency greater than the longer of the target latency or the predetermined preferred latency.15. The method of claim 13 or claim 14, in which: the videogame comprises a user performance evaluation phase; and the step of setting a target latency comprises modiiiing the target latency responsive to a percentage of received game input data packets accepted by the communication means prior to the user performance evaluation phase.16. The method of any one of claims 13 to 15, in which: The step of setting the target packet acceptance percentage comprises sefting the target packet acceptance percentage responsive to properties of all or part of a reference performance associated with the videogame.17. The method of any one of claims 13 to 16 in which the game input data comprises audio data, the method comprising the step of: generating audio to read out to the videogame in the place of gaps in the buffer caused by discarded game input data packets; and wherein the step of generating audio comprises one or more sub-steps selected from the list consisting of: i. substituting the gap with previously accepted audio data from the buffer; ii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target pitch value at the time of the gap; iii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target phoneme at the time of the gap; and iv. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to the detected phoneme uttered prior to the gap.18. A method of generating a virtual peripheral for an entertainment device on a portable device, comprising the steps of: emulating at least partially the functionality of a hardware input peripheral of an entertainment device; generating game input data appropriate to the emulated hardware input peripheral in response to a user interaction; and transmitting game input data packets comprising the generated game input data over a network to an entertainment deviec.19. Thcmcthodofclaiml8,inwhich: the emulation step also comprises emulating some or all of a conventional input controller of the entertainment device.20. The method of claim 18 or claim 19, comprising the step of: sending a signal to the entertainment device causing the entertainment device to run an application.21. The method of any one of claims 18 to 20, comprising the step of: receiving a signal from the entertainment device assigning a player number to the hardware input peripheral that is being at least partially emulated.22. Thc method of any onc of claims 18 to 21, comprising thc steps of generating from the game input data further data relating to a scoring metric of an application of the entertainment device that uses the game input data; and transmitting the further data over the network to the entertainment device.23. A computer program for implementing the steps of any preceding method claim.AMENDMENTS TO CLAIMS HAVE BEEN FILED AS FOLLOWSCLAIMS1 An entertainment device, comprising: communication means operable to receive game input data packets over a network; communication means operable to receive separate short packets over the network, comprising frirther data rehiting to a scoring metric of an application of the entertainment device, generated from pre-processed game input data; buffer means operable to store game input data obtained from the game input data packets, and to read out stored game input data to a videogame; target packet acceptance percentage setting means operable to set a target packet acceptance percentage; target latency setting means operable to set a target latency; wherein the communication means is operable to accept received game input data packets and separate short packets having a network latency within the target latency, and to discard received game input data packets and separate short packets having a network latency greater than the target latency; C\I and wherein the target latency setting means is operable to modify the target latency until in operation the communication means accepts at least the target packet acceptance o percentage of at least one of received game input data packets and separate short packets.2. The entertainment device according to claim 1, in which: communication means is operable to accept received game input data packets having a network latency within the longer of the target latency or a predetermined preferred latency, and to discard received game input data packets having a network latency greater than the longer of the target latency or the predetermined preferred latency.3 The enlertainmen i device according to cI aim I or cI aim 2, in which: the videogame comprises a user performance evaluation phase; and the target latency setting means is operable to modify the target latency responsive to a percentage of received game input data packets accepted by the communication means prior to the user performance evaluation phase.4. The entertainment device according to any one of the preceding claims, in which: target packet acceptance percentage setting means is operable to set the target packet acceptance percentage responsive to properties of all or part of a reference performance associated with the videogame.5, The entertainment device according to any one of the preceding claims in which the game input data comprises audio data, the device comprising: audio generation means operable to generate audio data to read out to the videogame in the place of gaps in the buffer caused by discarded game input data packets; wherein the audio generation means is operable to generate audio one or more processes selected from the list consisting of: i, substituting the gap with previously accepted audio data from the buffer; ii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target pitch value at the time of the gap; i iii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target phoneme at the C\I time of the gap; and iv. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to the detected phoneme uttered prior to the gap.6. A portable device, comprising: processing means arranged in operation to run an application, the application in turn operable to at least partially emulate a hardware input peripheral of an entertainment device, and operable to generate game input data appropriate to the emulated hardware input peripheral in response to a user interaction; pre-processing means operable to generate from the game input data further data relating to a scoring metric of an application of the entertainment device that uses the game input data; and wireless communication means operable to transmit game input data packets comprising the generated game input data over a network to an entertainment device, and operable to transmit the further data over the network to the entertainment device as a separate short packet.7, The portable device of claim 6, in which: the application is operable to also emulate some or all of a conventional input controller of the entertainment device.8. The portable device of claim 6 or claim 7, in which: the application is operable to cause the portable device to send a signal to the entertainment device causing the entertainment device to mn an application.9. The portable device of any one of claims 6 toS, in which: the wireless communication means is operable to receive a signal from the entertainment device assigning a player number to the hardware input peripheral being at least partially emulated by the application.10. An entertainment system, comprising: an entertainment device according to any one of claims ito 5; and one or more portable devices according to any one of claims 6 to 9; 11. An entertainment system according to claim 10, in which the entertainment device is hosted in a cloud system, and comprising: an intermediary device operable to receive game input data packets from the or each portable device and to transmit game input data packets to the cloud system.
12. 12. A method of processing game input data, comprising the steps of receiving game input data packets over a network; receiving separate short packets over the network, comprising further data relating to a scoring metric of an application of an entertainment device, generated from pre-processed game input data; storing game input data obtained from the game input data packets, for reading out to a videogame; setting a target packet acceptance percentage; setting a target latency; in a communications step, accepting received game input data packets and separate short packets having a network latency within the target latency, and discarding received game input data packets and separate short packets having a network latency greater than the target latency; and wherein the step of setting a target latency comprises periodically modifring the target latency until in the communications step at least the target packet acceptance percentage of at least one of received game input data packets and separate short packets are accepted.
13. 13. The method of claim 12, in which: the communications step comprises accepting received game input data packets having a network latency within the longer of the target latency or a predetermined preferred latency, and discarding received game input data packets having a network latency greater than the longer of the target latency or the predetermined preferred latency.
14. 14. The method of claim 12 or claim 13, in which: the videogame comprises a user performance evaluation phase; and __ the step of setting a target latency comprises modifying the target latency responsive to a percentage of received game input data packets accepted by the communication means prior to the user performance evaluation phase. cy)15. The method of any one of claims 12 to 14, in which: The step of setting the target packet acceptance percentage comprises setting the target packet acceptance percentage responsive to properties of all or part of a reference performance associated with the videogame.16. The method of any one of claims 12 to 15 in which the game input data comprises audio data, the method comprising the step of: generating audio to read out to the videogame in the place of gaps in the buffer caused by discarded game input data packets; and wherein the step of generating audio comprises one or more sub-steps selected from the list consisting of: i. substituting the gap with previously accepted audio data from the buffer; ii. substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target pitch value at the time of the gap; iii, substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to an expected target phoneme at the time of the gap; and iv, substituting the gap with previously stored audio data copied from the buffer and selected or processed responsive to the detected phoneme uttered prior to the gap.17. A method of generating a virtual peripheral for an entertainment device on a portable device, comprising the steps of emulating at least partially the functionality of a hardware input peripheral of an entertainment device in an application; generating game input data appropriate to the emulated hardware input peripheral in response to a user interaction; generating from the game input data further data relating to a scoring metric of an application of the entertainment device that uses the game input data; wirelessly transmitting game input data packets comprising the generated game input C\I data over a network to an entertainment device; and wirelessly transmitting the frirther data over the network to the entertainment device in separate short packets.18. The method of claim 17, in which: the emulation step also comprises emulating some or all of a conventional input controller of the entertainment device, 19. The method of claim 17 or claim 18, comprising the step of sending a signal to the entertainment device causing the entertainment device to mn an application.20. The method of any one of claims 17 to 19, comprising the step of receiving a signal from the entertainment device assigning a player number to the hardware input peripheral that is being at least partially emulated, 21. A computer program for implementing the steps of any preceding method claim.