HK1165582A - Realtime kernel - Google Patents

Description

Real-time kernel

Background

Interest in avatar-based virtual reality communication systems has grown with the increased availability of computing systems with high processing power and high bandwidth network connections. The main goal of such virtual reality systems is to create a virtual space in which users can interact and communicate using real-time data streams, such as audio, video, and text chat streams. A virtual space is generally defined by a computer graphic description that describes the virtual geometry of the space, the colors and textures mapped onto the virtual geometry, the collision properties that control how the user moves in the space, and the auditory properties of the space (such as reverberation and sound absorption properties).

In a typical virtual reality system, users communicate with each other from respective computers through an interface that is a source, sink, or both source and sink of one or more real-time data streams supported by the system. The virtual reality software application running on each user computer configures its own audio and graphical presentation based on position information describing the position of the avatar in the virtual space. The location information is typically received directly from the computers of other users or indirectly from a central presence server. By default, a virtual reality software application typically connects each source represented in a virtual space to each sink represented in the virtual space, subject to global switching rules, local user preferences, and conditions specified in object properties in the virtual space. These conditions are typically specified in terms of relative distance between objects. For example, some virtual reality software applications are configured such that real-time data stream connections are not established if the separation distance between avatars exceeds a maximum threshold distance.

A successful virtual reality communication system should generally have relatively low computational resource requirements so that real-time communication performance can be achieved using currently available computing devices and network bandwidth limitations. Furthermore, such systems should typically be implemented in a manner that encourages area designers to develop virtual areas that enhance the adoption of the system by users.

Summary of The Invention

In one aspect, the invention features a method in accordance with which one or more flow handling instructions are received at a local network node from a remote network node. The one or more stream handling instructions include a description of a stream handler for processing at least one real-time data stream. At the local network node, a stream handler is created from the description. The resulting data stream is generated at the local network node. In this process, the real-time data stream is processed by the created stream handler.

In another aspect, the invention features a method in accordance with which a description of a real-time stream handler is parsed from one or more stream handling instructions. In this process, an input source identifier, an output sink identifier, and a respective identifier for each of one or more data processing objects are parsed from the one or more stream handling instructions. Real-time stream handling objects corresponding to respective ones of the identifiers are instantiated. A directed graph is created from the description that includes some instantiated real-time stream handle objects. A real-time data stream is received from an input source corresponding to an input source identifier. The resulting data stream is generated at the output sink corresponding to the output sink identifier. In this process, a real-time data stream is processed through the directed graph.

In another aspect, the invention features a method in accordance with which at least one real-time data flow connection is established between a local network node and at least one remote network node. At the local network node, at least one real-time data stream originating from the remote network node is processed. In this process, the at least one real-time data stream is processed by one or more real-time data processing operations to produce a resultant data stream. The process is monitored. In response to determining that the process deviates from a performance goal based on the monitoring, the process is modified according to a real-time performance goal routine.

In another aspect, the invention features a method in accordance with which a first session is established at a local network node with a remote network node on a transport stream in accordance with a connectionless transport protocol. One or more channels are automatically opened for communicating data between the local network node and the remote network node in the first session on behalf of one or more software entities on the local network node. In a first session, a table is maintained. The table identifies some channels that are open and associates corresponding attribute values with the identified channels. In response to determining that the first session has failed, automatically attempting to establish a second session with the remote network node on a second transport stream according to a connectionless transport protocol. Each channel identified in the table is automatically opened in response to successfully establishing the second session.

In another aspect, the invention features a method in accordance with which a list of kernel components including one or more kernel service components is parsed. All kernel components in the parsed list that are missing in the local repository are determined. Each kernel component determined to be absent is retrieved. Kernel services are instantiated from some kernel services kernel components. The instantiated kernel service is executed to communicate with one or more remote network nodes in a communication environment defined with respect to the virtual area.

In another aspect, the invention features a method performed on a local network node. According to the method, the local network node is configured to support real-time communication with at least one remote network node in a context defined by a virtual area. The configuration process comprises the following steps: in response to a call to enumerate all plugins that support a specified Application Programming Interface (API), returning a list containing identifiers of all plugins in a plugin database that are associated with the specified API; in response to a call to enumerate variables of a given API supported by an identified one of the plug-ins, passing a list containing identifiers of all variables of the given API supported by the identified plug-in; and in response to a call to the identified one variable instantiating the identified API supported by the identified one plug-in, loading the identified plug-in and providing a pointer to the instance of the identified variable. At least one real-time data stream connection is established between the configured local network node and the at least one remote network node.

The invention also features apparatus for implementing the inventive methods described above and computer readable media storing computer readable instructions for causing a computer to implement the inventive methods described above.

Other features and advantages of the invention will become apparent from the following description, including the drawings and claims.

Brief Description of Drawings

Fig. 1 is a diagram of an embodiment of a virtual area communication environment including a first client network node, a second client network node, and an area server network node 16 interconnected by a network.

FIG. 2 is a flow diagram of an embodiment of a method performed by an embodiment of a real-time kernel.

Figure 3A is an illustration of an embodiment of a virtual area communication environment in which network nodes communicate in a peer-to-peer architecture.

Fig. 3B is an illustration of an embodiment of a virtual area communication environment in which network nodes communicate in a server-arbitrated architecture.

Fig. 4 is an illustration of an embodiment of a network node comprising a graphical user interface presenting a depiction of a virtual area.

FIG. 5A is an illustration of an embodiment of a heads-up display (HUD) superimposed on a graphical user interface presenting a depiction of a virtual area.

Fig. 5B is an illustration of the HUD shown in fig. 5A.

Fig. 5C is an illustration of an expanded view of the HUD shown in fig. 5A.

Fig. 6 is a flow diagram of an embodiment of a method implemented by an embodiment of a local area network infrastructure service.

FIG. 7 is a flow diagram of an embodiment of a method implemented by an embodiment of a real-time kernel.

Figure 8 is a block diagram of an embodiment of a client network node including an embodiment of a real-time kernel.

FIG. 9 is a flow diagram of an embodiment of a method implemented by an embodiment of the real-time kernel of FIG. 8 in response to a real-time kernel API call requesting a connection to a virtual area.

FIG. 10 is a flow diagram of an embodiment of a method implemented by an embodiment of the real-time kernel of FIG. 8 in response to a real-time kernel API call requesting entry into a virtual area.

FIG. 11 is a flow diagram of an embodiment of a method implemented by an embodiment of the real-time kernel of FIG. 8 in response to a stream handling instruction received from a regional service.

FIG. 12 is a block diagram of an embodiment of a stream handler created by the stream handler configuration manager.

FIG. 13 is a flow diagram of an embodiment of a method implemented by an embodiment of the real-time kernel of FIG. 8 in scheduling tasks for execution by the real-time kernel.

FIG. 14 is a flow diagram of an embodiment of a method implemented by an embodiment of the real-time core of FIG. 8 based on monitoring of processing of at least one real-time data stream.

FIG. 15 is a block diagram of an embodiment of the real-time kernel of FIG. 8.

FIG. 16 is a flow diagram of an embodiment of a method of authenticating an account server through credentials of the account server.

FIG. 17 is a flow diagram of an embodiment of a method implemented by a loader component of an embodiment of the real-time kernel of FIG. 8.

Fig. 18 is a flow diagram of an embodiment of a session management method implemented by the STRAW service component of the embodiment of the real-time kernel of fig. 8.

FIG. 19 is a flow diagram of an embodiment of a method implemented by a component of an embodiment of the real-time kernel of FIG. 8 in response to a remote stream handling instruction received from a local area network infrastructure service.

Fig. 20 is a diagram of transport protocol components implemented by the STRAW service component of an embodiment of the real-time kernel of fig. 8.

Fig. 21 illustrates an embodiment of a method of establishing a server flow between a client network node 344 and a server 346.

Referring to FIG. 22, each session is identified by a new GUID generated by the originating server.

FIG. 23 illustrates elements of an exemplary embodiment of a four-correspondent audio processing diagram.

FIG. 24 illustrates an embodiment of a computer system that enables people to communicate with virtual area communicants via a non-virtual area based communication application.

FIG. 25 shows a diagram of an embodiment of a plug-in class hierarchy.

FIG. 26 is a diagram of an embodiment of a set of plug-in base classes, where each plug-in base class is associated with a respective set of one or more derived variable classes.

FIG. 27 is a block diagram of an embodiment of a plug-in architecture.

FIG. 28 is a block diagram of an embodiment of a plug-in architecture including a plug-in manager, a plug-in directory containing a set of plug-in containers, a plug-in database, and a caller.

FIG. 29 is a flow diagram of an embodiment of a method implemented by an embodiment of the plug-in manager of FIG. 28 in registering an available plug-in on a client network node.

FIG. 30 is an illustration of an embodiment of a plug-in database.

FIG. 31 is a flow diagram of an embodiment of a method implemented by the embodiment of the plug-in manager of FIG. 28 in response to receiving an API call from a caller.

Detailed Description

In the following description, like reference numerals are used to identify like elements. Furthermore, the drawings are intended to illustrate major features of exemplary embodiments in a diagrammatic manner. The drawings are not intended to depict every feature of actual embodiments nor relative dimensions of the depicted elements, and are not drawn to scale.

I. Definition of terms

A "computer" is any machine, device, or apparatus that processes data according to computer-readable instructions stored temporarily or permanently on a computer-readable medium.

A "computer operating system" is a software component of a computer that manages and coordinates task execution and software and hardware resource sharing. A "kernel" is a collection of software components that can be called by a software application to provide specific functionality for accessing computer resources (e.g., CPU, memory, network links, and peripheral resources). A "software application" (also referred to as software, an application, computer software, a computer application, a program, and a computer program) is a set of instructions that a computer can interpret and execute to perform one or more tasks.

An "application programming interface" (or API) is a set of functional (or procedural) declarations that an operating system, library, or service provides to support requests made by software applications. The API specifies the behavior of an interface and the identifier specified in the interface. An implementation of an API refers to software application code that provides the functionality described by the API. A "computer data file" is a block of information that persistently stores data for use by a software application.

A "service" is a process that performs tasks autonomously (independently of other processes).

The "manager" is the gateway for the service to perform tasks. The manager performs the task involuntarily.

A "database" is an organized collection of records that can be searched by a computer, presented in a standardized format. The database may be stored on a single computer-readable data storage medium on a single computer, or may be distributed across multiple computer-readable data storage media on one or more computers.

A "data sink" (referred to herein simply as a "sink") is any of a device, a portion of a device (e.g., a computer), or software that receives data.

A "data source" (referred to herein simply as a "source") is any of a device, a portion of a device (e.g., a computer), or software that originates data.

A "network node" (also referred to simply as a "node") is a junction or connection point in a communication network. Exemplary network nodes include, but are not limited to, terminals, computers, and network switches. A "server network node" is a host computer on a network that responds to information or service requests. A "client network node" is a computer on a network that requests information or services from a server. A "network connection" is a link between two communication network nodes. The term "local network node" refers to the network node that is currently the subject of the primary discussion. The term "remote network node" refers to a network node that is connected to a local network node by a network communication link.

"presence" refers to the ability or willingness of a networked entity (e.g., a correspondent, service, or device) to communicate, wherein such willingness affects the ability to detect and obtain information about the status of the entity on the network and the ability to connect to the entity.

A "real-time data stream" is data structured and processed in the form of a continuous stream and is designed to be received without delay or with only an imperceptible delay. The real-time data stream includes digital representations of speech, video, user movements, facial expressions, and other physical phenomena, as well as data within the computing environment that may benefit from rapid transmission, rapid execution, or both rapid transmission and rapid execution, including, for example, avatar movement instructions, text chat, real-time data feeds (e.g., sensor data, machine control instructions, transaction streams, and stock price information feeds), and file transfers.

"stream mixing" is a combination of two or more real-time data streams of the same or semantically consistent type (e.g., audio, video, chat, and motion data). For example, a set of voice streams may be mixed into a single voice stream, or a voice stream may be mixed into the audio portion of a video stream.

A "switching rule" is an instruction that specifies the connection or disconnection of one or more real-time data sources to one or more real-time data sinks in compliance with one or more conditional precedents.

A "virtual area" (also referred to as a "region" or "place") is a representation of a space or scene managed by a computer. Virtual regions are typically one-dimensional, two-dimensional, or three-dimensional representations; although in some embodiments the virtual area may correspond to a single point. Virtual areas are often designed to simulate physical real-world space. For example, using a conventional computer monitor, the virtual area may be visualized as a two-dimensional graphic of a three-dimensional space generated by a computer. However, the virtual area does not require an associated visualization to implement the switching rules. A virtual area generally refers to an instance of a virtual area pattern, where the pattern defines the structure and content of the virtual area in a variable manner, while the instance defines the structure and content of the virtual area in a manner of values that have been solved from a particular context.

A "virtual area application" (also referred to as a "virtual area description") is a description of a virtual area used when creating a virtual area communication environment. Virtual area applications typically include definitions of geometry, physics, and real-time switching rules associated with one or more zones of a virtual area.

A "virtual communication environment" is a representation of a computer-managed space that includes at least one virtual area and supports real-time communication between communicants.

A "zone" is a zone in a virtual area associated with at least one switching rule or dominance rule. The switching rules control the switching (e.g., routing, connecting, and disconnecting) of real-time data flows between network nodes communicating in the context of the virtual area. The governing rules control a communicant's access to a resource (e.g., an area, a region of an area, or contents of the area or region), the scope of the access, and subsequent results of the access (e.g., requirements that an audit record regarding the access must be recorded).

"location" in a virtual area refers to the location of a point or area or volume in the virtual area. The points are typically represented by a single set of one-, two-, or three-dimensional coordinates (e.g., x, y, z) that define the points in the virtual area. The area is typically represented by three-dimensional coordinates of three or more coplanar vertices defining the boundaries of the closed two-dimensional shape in the virtual area. A volume is typically represented by three-dimensional coordinates of four or more non-coplanar vertices that define the closed boundaries of a three-dimensional shape in a virtual area.

In the context of a virtual area, an "object" is a discrete element in the virtual area that can be effectively treated and is independent of the geometry of the virtual area. Exemplary objects include doors, portals, windows, viewing screens, and loudspeakers. Objects typically have attributes and characteristics that are separate and distinct from those of the virtual area. An "avatar" is an object in a virtual area that represents a correspondent.

A "communicant" is a person who communicates or otherwise interacts with others over a network connection, where the communication or interaction may or may not occur in the context of a virtual area. A "user" is a correspondent that is operating a particular network node that defines a particular point of view for descriptive purposes.

A "zone server" is a network node that includes a zone network infrastructure service that manages a virtual zone hosting virtual zone applications by managing sessions of client nodes associated with objects in the virtual zone.

As used herein, the term "including" means including but not limited to, and "includes" means including but not limited to.

Introduction II

Embodiments described herein provide a real-time kernel that supports real-time communications between communicants operating on respective network nodes. The real-time kernel handles the complex tasks of connecting to correspondents, virtual areas, and other network resources, switching these connections in response to user input, and mixing real-time data streams. The real-time kernel enables developers to focus on developing high-level communication functions rather than low-level nexus code. The real-time kernel imposes relatively low computational resource requirements so that real-time communication performance can be achieved using the wide range of computing devices and network connections currently available.

In some embodiments, the real-time kernel supports remote configuration and execution of audio and graphics rendering engines, as well as switching of real-time data streams, in response to instructions (also referred to as definitions) received from a remotely hosted virtual area application. In this way, the real-time kernel enables virtual area designers to maintain control over the presentation of an immersive virtual communication environment on remote client network nodes, thereby encouraging the development of many different types of virtual areas and increasing the number of users who would like to employ the communication system.

In some embodiments, the real-time kernel monitors the processing of the real-time data stream and adapts the processing based on the deviation of the processing from the performance goal. In this way, the kernel increases the likelihood that real-time performance can be achieved regardless of the computing environment in which real-time data stream processing is being performed.

In some embodiments, the real-time kernel implements a streaming protocol that is efficient at connecting and disconnecting, and at the time of transmission. In some such embodiments, the streaming protocol provides a connection-oriented encrypted connection over a connectionless transport protocol (e.g., UDP). The real-time kernel additionally provides a reconnect mechanism between the client application and the transport layer that automatically attempts to reestablish the failed connection without intervention by the client application, thereby increasing reliability over inherently unreliable communication protocols.

In some embodiments, the real-time kernel has a plug-in architecture that allows the functionality of the kernel components to be provided by one or more plug-ins that can be dynamically loaded onto the client network node. In this way, kernel components can be developed independently and managed and updated remotely. The plug-in architecture additionally allows the installation coverage area of the real-time kernel to be substantially reduced, thereby allowing the kernel to be installed on a wide range of client devices, including those with significant computational and storage resource limitations.

Overview

A. Introduction to the design reside in

Fig. 1 illustrates an embodiment of an exemplary virtual area communication environment 10 including a first client network node 12, a second client network node 14, and an area server network node 16 interconnected by a network 18. The first client network node 12 includes an embodiment of a real-time core 20 including one or more configurable stream handlers 22, and input/output (I/O) hardware 24. The second client network node 14 is typically configured in substantially the same manner as the first client network node 12. The area server network node 16 includes an area network infrastructure service 26 (also referred to simply as a "area service") that manages the virtual area 28 by managing sessions for the first and second client nodes 12, 14 in the virtual area 28. The virtual area 28 hosts a virtual area application 30 that includes a description of the virtual area for creating a virtual area communication environment. The zone service 26 manages the virtual zone 28 according to the virtual zone application 30.

In creating the shared virtual area communication environment, the area service 26 remotely configures the real-time kernels in the first and second client network nodes 12, 14 according to the remote virtual area application 30 as influenced by a set of restrictions 32 on the virtual area application 30. The restrictions 32 typically include control over access to the virtual area. Access control is typically based on one or more capabilities (where access is granted to communicants or client nodes with appropriate capabilities or permissions) and an access control list (where access is granted to communicants or client nodes with identities on the list). In some embodiments, restrictions 32 are managed by a secure network infrastructure service (described below). Client software applications operating on the first and second client network nodes 12, 14 allow the communicants to access the shared virtual area communication environment by presenting respective views of the virtual area according to data received from the area service 26 via the real-time kernel 20 and by providing an interface for receiving commands from the communicants. The communicants are typically represented in the virtual area 32 by respective avatars that move about the virtual area in response to commands entered by the communicants at their respective network nodes. The view of the virtual area seen by each communicant is typically presented from the perspective of the communicant's avatar, which enhances the level of immersion experienced by the communicant. Each communicant is generally able to see any portion of the virtual area around his or her avatar. The real-time kernels operating on the first and second client network nodes 12, 14 establish real-time data stream connections with other network nodes sharing the virtual area communication environment based at least in part on the position of the communicant's avatar in the virtual area.

Fig. 2 shows an exemplary embodiment of a method implemented by the real-time kernel 20. According to the method, the real-time kernel 20 establishes a session with the regional service 26 (FIG. 2, block 34). Whether in response to a correspondent input or automatically, the real-time kernel 20 requests entry into an instance of the virtual area 28 (FIG. 2, block 36). If the restrictions 32 on the communicant's access to the virtual area instance are satisfied, the area service 26 communicates configuration data including current state information to the real-time kernel 20, where the current state information includes the position of the avatar in the virtual area. The real-time kernel 20 receives configuration data from the regional services 26 (fig. 2, block 38). The real-time kernel 20 configures the I/O hardware 24 to present a human-perceptible virtual zone communicator environment according to instructions received from the zone service 26 (fig. 2, block 40).

In some embodiments, the process of configuring the I/O hardware 24 involves dynamically configuring at least one stream handler 22 according to instructions and location data received from the remote network node 14. For example, the virtual area application 30 may specify one or more audio effects that should be applied to the audio streams associated with the current objects in the virtual area, in which case the area service 26 sends instructions to the real-time kernel executing on the first and second client network nodes 12, 14 that configure their respective audio stream handlers in accordance with the positions of the respective objects in the virtual area to achieve the specified effects.

The real-time kernel 20 processes the real-time data stream associated with the correspondent object through each configured stream handler 22 to produce a corresponding output 33. Depending on its content, the output 33 may be stored on a computer readable medium or converted to a human perceptible output by I/O hardware operating on the first and second network nodes 12, 14. For example, audio output signals are converted to audible sound by audio hardware (e.g., a sound card and speakers), while graphical output signals are converted to visible images by graphics hardware (e.g., a video card and a display). In some embodiments, the output 33 produced by at least one stream processor 22 is processed by one or more downstream software components, which in turn produce an output that can be stored on a computer-readable medium or converted to a human-perceptible output.

B. Exemplary operating Environment

The real-time kernel 20 operates in the context of a virtual area communication environment 10, the virtual area communication environment 10 including a network 18 and a network infrastructure services environment, the latter including a number of network infrastructure services including an area service 26. The real-time kernel 20 and the network infrastructure service environment constitute a platform for creating a virtual area communication environment for the communicants.

1. Network environment

Network 18 may include any of a Local Area Network (LAN), a Metropolitan Area Network (MAN), and a Wide Area Network (WAN), such as the Internet. Network 18 typically includes several computing platforms and transmission facilities that support the transmission of various different media types (e.g., text, voice, audio, and video) between network nodes.

The real-time kernel 20 typically operates on network nodes that include software and hardware resources that, along with administrative policies, user preferences (including preferences regarding the egress of user presence and user connections to regions and connection objects), and other settings, define local configurations that affect the management of real-time connections with other network nodes. Network connections between network nodes may be arranged in a variety of different stream handling topologies, including peer-to-peer architectures, server-arbitrated architectures, and hybrid architectures that combine aspects of peer-to-peer and server-arbitrated architectures.

Fig. 3A illustrates an embodiment 42 of the virtual area communication environment 10, wherein the first and second network nodes 12, 14 and the remote network node 16 are interconnected by a communication network 18 of a peer-to-peer architecture. In this architecture, each of the network nodes 12-16 transmits a state change (e.g., avatar movement in the virtual area 28) to each of the other network nodes. One of the network nodes (typically the network node that initiates the communication session) operates as a "zone server". In the illustrated embodiment, the network node 16 assumes the role of a zone server. The regional server network node 16 maintains global state information and acts as a data server for the other network nodes 12, 14. The global state information includes a list of all objects in the virtual area and their corresponding locations in the virtual area. The area server network node 16 sends instructions to configure the other network nodes 12, 14. The area server network node 16 also registers and transmits initialization information to other network nodes requesting to join the communication session. In this process, the area server network node 16 transmits to each joining client network node a list of components (e.g., plug-ins) required to render the virtual area 28 on the client network node in accordance with the virtual area application 30. The real-time kernel on the client network node 12, 14 typically retrieves any missing components on the list from a remote server (e.g., a plug-in server). The regional server network node 16 also ensures that in the event of a communication failure, the other network nodes 12, 14 can synchronize to a global state.

Fig. 3B illustrates an embodiment 44 of the virtual area communication environment 10 in which the network nodes 12-16 (also referred to in this architecture as "area client" network nodes) communicate in an architecture that is arbitrated by an area server 46. In this embodiment, the zone server 46 assumes the zone server functions performed by the network node 16 in the peer-to-peer architecture embodiment shown in FIG. 3A. In this manner, the regional server 46 maintains global state information and serves as a data server for the regional client network nodes 12-16. The architecture allows real-time data stream exchange between the regional client nodes 12-16 to be handled in a variety of topologies, including peer-to-peer topologies, full server arbitrated topologies where the regional server 46 operates as a communications agent between the network nodes 12-16, and hybrid topologies that combine aspects of the peer-to-peer and full server arbitrated topologies. Exemplary topologies of these types are described in U.S. application Ser. Nos. 11/923,629 and 11/923,634, both of which were filed on 24/10 of 2007.

2. Network infrastructure services

One or more network infrastructure services typically cooperate with the real-time kernel 20 in establishing and managing network connections with other network nodes. Network infrastructure services may run on a single network node or may be distributed across multiple network nodes. Network infrastructure services typically run on one or more dedicated network nodes (e.g., server computers or network devices that perform edge services such as routing and switching). However, in some embodiments, one or more network infrastructure services run on at least one correspondent network node. The network infrastructure services included in the exemplary embodiment of the virtual area communication environment 10 are account services, security services, area services 26, aggregation services, and interactive services, among others.

The account service manages the correspondent account in the network infrastructure service environment. The account service also manages the creation and issuance of authentication tokens that can be used by client network nodes to authenticate themselves to any network infrastructure service.

The security service controls access by communicants to assets and other resources of the virtual regional communication environment 10. The access control method implemented by a security service is typically based on one or more capabilities (where access is granted to entities with appropriate capabilities or permissions) and an access control list (where access is granted to entities with identities on the list). After a particular communicant has been granted access to resources, the communicant typically interacts in the virtual area communication environment 10 using functionality provided by other network infrastructure services.

The area service 26 manages virtual areas. In this process, the zone service 26 manages connections associated with the virtual zones, maintains global state information for the virtual zones, and serves as a data server for client network nodes participating in a shared communication session in the context defined by the virtual zones, according to the capabilities of the requesting entity. The global state information includes a list of all objects in the virtual area and their corresponding locations in the virtual area. The regional service 26 sends instructions to configure the client network node. The regional service 26 also registers and communicates initialization information to other client network nodes requesting to join the communication session. In this process, the zone service 26 transmits to each joining client network node a list of components (e.g., plug-ins) required to render the virtual zone 28 on the client network node in accordance with the virtual zone application 30. The regional service 26 also ensures that client network nodes can synchronize to a global state in the event of a communication failure.

The aggregation service manages the collection, storage, and distribution of presence information according to the capabilities of the requesting entities and provides the network nodes with a mechanism for communicating with each other (e.g., by managing the distribution of connection handles). The aggregation service typically stores presence information in a presence database.

The interaction service maintains an interaction database that records interactions between communicants and supports queries against the interaction database based on capabilities of the requesting entity. For each interaction between communicants, one or more services (e.g., regional services 26) in the virtual regional communication environment 10 communicate interaction data to the interaction service. In response, the interaction service generates one or more corresponding interaction records in a relational database. Each interaction record describes the context of the interaction. For example, in some embodiments, the interaction record contains an identifier for each communicant, an identifier for the interaction locale (e.g., a virtual area instance), a description of the level of the interaction locale (e.g., describing how the interaction space relates to a larger area), the start and end times of the interaction, and a list of all files and other streams shared during the interaction. Thus, for each real-time interaction, the interaction service tracks when, where, and what has occurred with respect to the communicants involved (e.g., entry and exit), objects that are activated/deactivated, and files that are shared during the interaction.

The interaction service can present the results of queries to the interaction database records in a sorted order (e.g., most frequent or recent) based on locale. The query results can be used to derive a frequency classification of contacts that the communicant has encountered within which virtual areas, and a classification of contacts that the communicant has encountered regardless of the virtual area, and a classification of the virtual area that the communicant most frequently visits. The query results may also be used by application developers as part of a heuristic system that automates certain tasks based on relationships. Examples of this type of elicitation are elicitations that allow a communicant who has visited a particular virtual area more than 5 times to gain entry without having to tap by default, or elicitations that allow a communicant who exists in an area at a particular time to modify and delete a file created by another communicant who exists in the same area at the same time. Queries to the interaction database may be combined with other searches. For example, queries to the interaction database may be combined with queries for contact history data generated using communication systems outside the domain of the network infrastructure services environment (e.g., Skype, Facebook, and Flickr) to interact with contacts.

3. Virtual area

The real-time kernel 20 manages real-time connections with network nodes in a communication context defined by an instance of a virtual area. The virtual area instance may correspond to an abstract (non-geometric) virtual space defined with respect to abstract coordinates. Alternatively, the virtual region instance may correspond to a visual virtual space defined in terms of one-, two-, or three-dimensional geometric coordinates associated with a particular visualization. The abstract virtual areas may or may not be associated with respective visualizations, while the visual virtual areas are associated with respective visualizations.

As explained above, communicants are typically represented by respective avatars in virtual areas with associated visualizations. The avatars move around in the virtual area in response to input commands entered by the communicants at their respective network nodes. The view of the virtual area instance seen by a communicant with the associated visualization is typically presented from the perspective of the communicant's avatar, and each communicant is typically able to view any portion of the visual virtual area surrounding his or her avatar, thereby enhancing the level of immersion experienced by the communicant.

Fig. 4 illustrates an embodiment of an exemplary network node implemented by a computer system 48. The computer system 48 includes a display monitor 50, a computer mouse 52, a keyboard 554, speakers 56, 58, and a microphone 60. The display monitor 50 displays a graphical user interface 62. The graphical user interface 62 is a window-based graphical user interface that may include a plurality of windows, icons, and pointers 64. In the illustrated embodiment, the graphical user interface 62 presents a two-dimensional depiction of a shared virtual area 66 associated with a three-dimensional visualization representing a gallery. The communicants are represented in the virtual area 66 by respective avatars 68, 70, 72, each of which may have a respective character (e.g., a curator, an artist, and a guest) within the context of the virtual area 66.

As explained in detail below, the virtual area 66 includes zones 74, 76, 78, 80, 82 associated with respective rules governing real-time data flow switching among the network nodes represented by the avatars 68-72 in the virtual area 66. (during a typical communication session, the dashed lines demarcating zones 74-82 in FIG. 4 are not visible to the communicants, although there may be visual cues associated with such zone boundaries.) the switching rules specify how the local connection process performed on each network node establishes communication with other network nodes based on the location of the communicant's avatar 68-72 in the zones 74-82 of the virtual area 66.

The virtual area is defined by a specification that includes a description of the geometric elements of the virtual area and one or more rules, including switching rules and governing rules. The switching rules govern real-time streaming connections between network nodes. The governing rule controls access of the communicant to resources such as the virtual area itself, the zone having the virtual area, and objects within the virtual area. In some embodiments, the geometric elements of the virtual area are specified according to the COLLADA-Digital Asset set Schema Release 1.4.1 April 2006 specification (4-month Digital Asset plan version 1.4.1, 2006), from http://www.khronos.org/collada/Derived), and the switching rules use the extensible markup language (XML) text format (referred to herein as the virtual space description grid) in accordance with the COLLADA stream reference specification described in U.S. application nos. 11/923,629 and 11/923,634Equation (VSDL)) is described.

The geometric elements of the virtual area typically include the physical geometry and collision geometry of the virtual area. The physical geometry describes the shape of the virtual area. The physical geometry is typically formed by triangular, quadrilateral, or polygonal surfaces. Colors and textures are mapped onto the physical geometry to create a more realistic appearance of the virtual region. For example, lighting effects may be provided by depicting light rays visually geometrically and modifying texture, color, or brightness in the vicinity of the light rays. The collision geometry describes the invisible surface that determines the way an object can move in a virtual area. The collision geometry may be consistent with the visual geometry, correspond to a simpler approximation of the visual geometry, or relate to application-specific requirements for the virtual area designer.

The switching rules typically include a description of the conditions for connecting the source and sink of the real-time data stream by location in the virtual area. Each rule typically includes attributes that define the type of real-time data stream to which the rule applies and the location(s) to which the rule applies in the virtual area. In some embodiments, each rule optionally may include one or more attributes specifying a required role for the source, a required role for the sink, a priority level for the flow, and a requested flow handling topology. In some embodiments, if there is no explicit switching rule defined for a particular portion of the virtual area, one or more implicit or default switching rules may be applied to that portion of the virtual area. One exemplary default switching rule is a rule that connects each source within a region to each compatible sink in accordance with policy rules. Policy rules may apply globally to all connections between regional clients or only to corresponding connections with individual regional clients. An example of a policy rule is a proximity policy rule that only allows connection of source and compatible sinks associated with respective objects within a prescribed distance (or radius) from each other in the virtual area.

In some embodiments, governance rules are associated with a virtual area to control who can access the virtual area, who can access its content, what the scope of access to the content of the virtual area is (e.g., what the user can do with the content), and what the subsequent results of accessing the content are (e.g., record keeping, such as audit logs, and payment requirements). In some embodiments, the entire virtual area or a zone of the virtual area is associated with a "dominating grid". In some embodiments, the dominating grid is implemented in a manner similar to the implementation of the zone grid described in U.S. application nos. 11/923,629 and 11/923,634. The governance grid enables a software application developer to associate governance rules with a virtual area or zone of a virtual area. This avoids the need to create an individual license for each file in the virtual region and avoids the need to deal with the complexity that can arise when the same document needs to be treated differently depending on the context.

In some embodiments, a virtual area is associated with a governing grid that associates one or more zones of the virtual area with Digital Rights Management (DRM) functions. The DRM function controls access to one or more of the virtual area or one or more zones within the virtual area or objects within the virtual area. The DRM function is triggered each time the correspondent crosses a dominant grid boundary within the virtual area. The DRM function determines whether the trigger action is permitted and, if so, the extent of the permitted action, whether payment is required, and whether an audit record needs to be generated. In an exemplary implementation of a virtual area, the associated governing grid is configured such that if a communicant is able to enter the virtual area, he or she can perform actions on all documents associated with the virtual area, including manipulating documents, viewing documents, downloading documents, deleting documents, modifying documents, and re-uploading documents. In this way, the virtual area may become a repository for information that is shared and discussed in the context defined by the virtual area.

Additional details regarding the specifications of the virtual areas are set forth in U.S. application Ser. Nos. 61/042714 (filed on 4/2008), 11/923,629 (filed on 24/10/2007), and 11/923,634 (filed on 24/10/2007).

4. Other platform assemblies

The real-time kernel 20 is designed to work as a component of a local network node that is part of a client software package, wherein the client software further comprises:

a. a Heads Up Display (HUD) software application;

b. a local Human Interface Device (HID) and an audio playback device;

a So3D graphical display, an avatar, and a physics engine;

d. a system database and a storage facility.

a. Head-up display (HUD)

A Heads Up Display (HUD) is an application interface to the real time kernel 20 operating on each client network node. A HUD is a small, lightweight interface that a user can always hold and run on his or her desktop. It is a user interface for launching a virtual area application, providing him or her immediate access to real-time contacts and real-time collaboration sites (or areas). The virtual region is integrated with the user's desktop through the HUD and real-time kernel 20, enabling the user to drag and drop files into the virtual region communication environment, utilize a native client software application independent of the virtual region communication environment to use files stored in association with the virtual region while also being present in the virtual region, and more generally treat the presence and location within the virtual region as an aspect of its operating environment similar to other operating system functions rather than just one of several applications.

Fig. 5A and 5B illustrate an embodiment 84 of the HUD implemented by a semi-transparent user interface that sits in the lower right corner of the communicator's desktop. The HUD 84 is an application interface to the platform. The characteristics of the HUD 84 include:

small, lightweight applications that are intended to run all the time on a user's desktop; and

provide an interface for the user to facilitate viewing and interacting with the contact and viewing and interacting with the virtual area where the interaction occurred.

In this embodiment, the HUD 84 is implemented by a substantially transparent (semi-transparent) user interface overlay that provides permanent interface and access to the controls. In the embodiment shown in FIG. 5A, the HUD 84 is transparent, except for a limited set of one or more of the following semi-transparent elements of the interface:

the outline of the progressive immersion control;

the outline of the user's current location;

a sub-graph representing real-time contacts in the virtual area 86; and

lines delimiting the HUD region.

The communicant is able to work in a common desktop computing environment while the real-time kernel 260 and HUD 84 are running and ready to initiate a real-time communication session. For example, the correspondent can be with a communication system such as Microsoft Windows ExcelAnd the like cooperate to create documents that can be later shared in a real-time communication session. The virtual area 86 is integrated with the communicator's desktop so that the communicator can drag and drop files into the virtual area, utilize a native client software application independent of the virtual area communication environment to use files stored in association with the virtual area while present in the virtual area, and more generally treat the presence and location within the virtual area as an aspect of its operating environment similar to other operating system functions rather than as one of several applications.

The HUD 84 provides the communicant with independent control over the visualization he or she desires as he or she interacts in the virtual area 86. For example, the communicant may display a minimized view (minimized) of the virtual areaTo the lower right corner of the desktop) and engage in an audio conversation with another communicant in the virtual area while operating, for example, microsoftExcelAnd the like in various applications. The communicant can then change his or her visualization mode and enter a more immersive three-dimensional presentation of the virtual area 86. This is accomplished by changing the setting of the progressive immersion slider 88 in the HUD 84 from "desktop" to "3D". Once in the 3D visualization mode, the communicator's desktop displays a 3D rendering of the virtual area 86 (as shown in FIG. 5A). The communicants (represented by sub-graphics 90, 92, 94 in desktop mode) now take the form of three-dimensional avatars 96, 98, 100 as shown in FIG. 5A.

Any data associated with the virtual area 86 may be displayed on the viewing screens 102, 104, 106. The viewing screen is a generic data presentation component that can be used to present any arbitrary data. Examples of the types of data that may be presented on a viewing screen include:

microsoft PowerPoint report

Video

Output of the web camera

Real-time data directly from an organizational Enterprise Resource Planning (ERP) System

As shown in FIG. 5C, the HUD 84 is designed to act as a real interface to display information and provide access to controls with minimal obstruction to the underlying portion of the graphical user interface 62 presented on the display monitor of the communicant. HUD 84 efficiently displays:

the real-time contacts of the correspondent that are currently online,

where the communicant's real-time contacts with the communicant are currently "located" in the virtual area 86,

progressive immersion control to control visualization of virtual area 86, and

navigation controls that enable a user to quickly connect to a particular venue.

In particular, the HUD 84 provides the communicant with instant access to their real-time contacts and virtual areas where real-time collaboration occurs. The HUD 84 allows navigation through the area based on the location of the person and allows viewing of the virtual area. These virtual areas can be accessed in a number of ways: most frequently used, most recently used, or application specific manner.

The HUD 84 displays an ordered set of place tiles 108, 110, 112. Clicking on one of the place tiles takes the user to the virtual area represented by the selected place tile. For a person, there is a basic metaphor for going (to the area of the correspondent) and taking (bringing them to the area of the user). This is improved in the HUD 84 by allowing the correspondent to queue a go or a take request and communicate with a person via text or voice without "moving". The HUD 84 notifies another correspondent upon receiving each communication request from the correspondent. The correspondent can accept the request, ignore it, or add it to the communication queue. In this way, the correspondent can respond to the non-priority communication at a later time. For example, a correspondent can queue a communication received during a time when the correspondent is busy (e.g., busy with a current communication session), and after the correspondent is free, the correspondent can respond to a communication request in the communication queue.

As described above, the interaction network infrastructure service maintains an interaction database that records who the correspondent encounters and where. The interaction service responds to queries to the relational database with query results that may be presented in a place-based sort order (e.g., most frequent or recent). In this way, the relational database information can be used to derive a frequency classification of the people the correspondent meets in which areas, as well as a classification of the people the correspondent has met regardless of area, and a classification of the area the correspondent most frequents. This data is used in the HUD 84. This data can also be used as part of a heuristic system with the virtual area application developer (e.g., a rule to admit a person who has visited a particular virtual area more than 5 times without a default knock, or a rule to allow a person who is present in the virtual area at a particular time to modify and delete a file created by another communicant who is there at the same time).

In FIG. 5C, the HUD 84 presents a series of place tiles 108, 110, 112 representing respective virtual areas. Each of these virtual areas is tied to a query to a relational database. With respect to each virtual area, the aggregation service queries the relational database for all contacts that the user has encountered in that virtual area. The aggregation service typically presents the identified contacts in a list sorted by frequency or by recency of interaction (e.g., the contact with which the correspondent last interacted). In other embodiments, the contacts may be categorized in some other application-dependent manner.

Queries to the relational database may be combined with other searches. For example, a query to a relational database may be combined with a query to contact history data generated for interactions with contacts using another communication system (e.g., Skype, Facebook, and Flickr). In one example, the Skype virtual area 112 can be associated with a query for the communicant's relationship data associated with the Skype virtual area 112 and the communicant's Skype history data to produce a sorted list of the user's real-time contacts associated with the Skype virtual area 112.

FIG. 5C illustrates the basic navigation of contacts and virtual areas in the HUD 84. Clicking on the left-facing arrow associated with each of the virtual area tiles 108-112 displays a list of real-time contacts sorted by interaction frequency in a given locale. For example, clicking on the left facing arrow 114 of the main HUD tile 84 (labeled "office") displays the real-time contacts that the user communicates most frequently in the virtual area 86. The list of contacts (represented by respective icon tiles) is sorted by frequency. The first contact in the list (DVW in this example) represents the contact that the user most frequently collaborates in virtual area 86, followed by a PJB, Tim, etc. Clicking on the up arrow 116 displays a set of place tiles representing some or all of the virtual areas that the correspondent has visited. The set of place tiles is typically sorted by frequency, recency, or other ordering. The virtual area site tile displays real-time activity that is currently occurring in the corresponding virtual area. For example, DVW, Kim, and Joe (represented by the corresponding sub-graphics in the main virtual area tile 108) are all in the main virtual area and are talking in real time, while Jeff, Ann, and Jane (represented by the corresponding sub-graphics in the virtual area tile 110) are all in the Facebook virtual area.

Assuming that any correspondent exits or enters the virtual area, the presence indicator (i.e., a small graphic shown by a circle, typically associated with a name or other identifier) in this virtual area will automatically be updated in real-time. This feature demonstrates the ability of the virtual area designer to input application specific real-time data into the locale tile. The locale tile may appear associated with the communicant or with the communicant's locale. For example, a game developer may output a map of the location of a communicant in their gaming environment, whereby others connected to the communicant through a relational database receive a real-time feed of the communicant's current activities. The person may use the virtual area tile to navigate to the correspondent, communicate with him or her, or contact him or her (e.g., send an invitation to enter the virtual area). The HUD84 manages this interface with contacts and virtual areas for many different virtual areas simultaneously.

The real-time data used in the HUD virtual area tiles 84, 108, 110, 112 is provided by an interface managed by the area service hosting the relevant area via the real-time kernel 120. Each region service may provide the communicants with tile data for a different respective HUD virtual region based on the communicants' permission to view the hosted virtual region. For example, if a communicant enters a virtual area that the communicant is not allowed to view, the HUD virtual area tile may display limited or no detailed information. Further, the HUD virtual area tile data feed provided by the hosting area service may be customized by the virtual area provider operating the area service to present an application-specific view of the virtual area to the subscribing HUD.

b. A local Human Interface Device (HID) and an audio playback device;

the local HID device enables a communicant to input commands and other signals to the client network node while participating in a virtual area communication session. Exemplary HID devices include a computer keyboard, a computer mouse, a touch screen display, and a microphone.

An audio playback device enables a communicant to play back audio signals received during a virtual zone communication session. An exemplary audio playback device includes audio processing hardware (e.g., a sound card) for manipulating (e.g., mixing and applying special effects) audio signals, and a speaker for outputting sound.

So3D graphical display, avatar, and physics engine

The So3D engine is a three-dimensional visualization engine that controls the display of virtual areas and corresponding views of objects within the virtual areas on a display monitor. The So3D engine typically interfaces with graphical user interface drivers and HID devices to present a view of the virtual area and to allow the communicator to control the operation of the HUD application.

The So3D engine typically receives graphics rendering instructions from the regional service 26 via the real-time kernel 20. In some embodiments, the So3D engine also reads a correspondent avatar database that contains images needed to render the correspondent's avatar in the virtual area. Based on this information, the So3D engine generates visual representations (i.e., images) of the virtual area and objects within the virtual area from the perspective (position and orientation) of the communicants' avatars within the virtual area. The visual representation is typically passed to a graphics rendering component of the operating system that drives the graphics rendering hardware to render the visual representation of the virtual area on the client network node.

The communicant may control the rendered view of the virtual area by communicating commands from the HID device (e.g., a computer mouse) to the realtime kernel 20 (the realtime kernel 20 communicates view control commands to the So3D engine). The So3D engine updates the view of the virtual area according to the view control command. The So3D engine also updates the graphical representation of the virtual area on the display monitor according to the updated object location information received from the regional service 26 via the real-time kernel 20.

d. System database and storage facility

The system databases and storage facilities store various types of information used by the platform. Exemplary information typically stored by a storage facility includes a presence database, an interaction database, an avatar database, a Real User Id (RUID) database, a style (art) buffer database, and a virtual area description database. The information may be stored on a single network node or may be distributed across multiple network nodes.

C. Exemplary communication sessions

Referring again to fig. 4, during the communication session, each client network node generates a respective set of real-time data streams (e.g., a motion data stream, an audio data stream, a chat data stream, a file transfer data stream, and a video data stream). For example, each communicant manipulates one or more input devices (e.g., the computer mouse 52 and the keyboard 54) that generate a stream of motion data that controls the movement of his or her avatar within the virtual area 66. In addition, communicator speech and other sounds generated locally near the computer system 48 are captured by the microphone 60. The microphone 60 generates an audio signal that can be converted into a real-time audio stream. Respective copies of the audio stream are transmitted to other network nodes represented by avatars in the virtual area 66. Sounds generated locally at these other network nodes are converted to real-time audio signals and transmitted to the computer system 48. The real-time core 20 converts audio streams generated by other network nodes into audio signals that are rendered by the speakers 56, 58. The motion data stream and the audio data stream may be transmitted directly or indirectly from each communicant node to other client network nodes. In some flow handling topologies, each client network node receives copies of the real-time data flow transmitted by other client network nodes. In other flow handling topologies, one or more client network nodes receive one or more flow mixes derived from real-time data flows originating from (originating from) some other network nodes.

In some embodiments, the region service 26 maintains global state information including a current description of the virtual region, a current registry of objects in the virtual region, and a list of any stream mixes that are currently being generated by the network node hosting the region service 26. The object registration typically includes, for each object in the virtual area, a respective object identifier (e.g., a tag that uniquely identifies the object), a connection handle (e.g., a URI such as an IP address) that can establish a network connection with the network node associated with the object, and interface data that identifies the real-time data source and sink associated with the object (e.g., the source and sink of the network node associated with the object). The object registry also typically includes one or more alternate role identifiers for each object that can be explicitly assigned to the object by the correspondent or the regional service 26, or can be inferred from other attributes of the object. In some embodiments, the object registration also includes the current location of each object in the virtual area, as determined by the area service 26 from an analysis of real-time motion data streams received from network nodes associated with the objects in the virtual area. In this regard, the zone service 26 receives real-time motion data streams from network nodes associated with objects in the virtual zone and tracks the communicant's avatar and other objects entering, leaving, and moving around in the virtual zone based on the motion data. The regional service 26 updates the object registration based on the current location of the tracked object.

In managing real-time data flow connections with other network nodes, the area service 26 maintains a set of configuration data for each client network node, including interface data, a list of zones, and the location of objects currently located within the virtual area. The interface data includes, for each object associated with each client network node, a respective list of all sources and sinks for the real-time data stream type associated with the object. The zone list is a registration of all zones in the virtual area that are currently occupied by avatars associated with the respective client network nodes. When the correspondent first enters the virtual area, the area service 26 typically initializes the current object location database with location initialization information. The area service 26 then updates the current object location database with the current location of the object in the virtual area, as determined from an analysis of the real-time motion data streams received from the other client network nodes sharing the virtual area.

Fig. 6 illustrates an embodiment of a method according to which an embodiment of the area service 26 determines a set of required real-time data stream connections to be established when a subscriber enters a virtual area or crosses a boundary between zones of the virtual area. The area service 26 establishes a list of occupied zones for each communicant based on the virtual area description and the location of the communicant's avatar in the virtual area instance (fig. 6, block 180). In this process, the zone service 26 retrieves the current position of the avatar in the virtual zone instance from the current object position database, which contains the coordinates of the avatar's current position in the virtual zone instance. The area service 26 then compares the current location of the communicator's avatar with the zone definitions in the virtual area description. The area service 26 compiles an occupied zone list from all zones in the virtual area description that coincide with the current position of the correspondent avatar. For example, in some embodiments, the occupied zone list includes all zones whose mesh contains the current location of the communicator avatar.

The area service 26 determines a set of target real-time data stream types defined for the zones in the occupied zone list (fig. 6, block 182). The zone service 26 then determines the required set of real-time data stream data from the set of target real-time data stream types, the location of the object in the virtual zone instance, and the switching rules defined in the virtual zone description (fig. 6, block 184).

In some example embodiments, after the regional service 26 has determined a set of real-time data flow data that enables users to participate in collaborative communication sessions with other network nodes in the shared virtual region instance (fig. 6, block 184), the regional service 26 determines a real-time data flow connection that will result in the delivery of the required data flow data to the computer system 120.

In some embodiments, regional service 26 determines a real-time data stream handling topology that communicates the set of real-time data streams to computer system 120 based at least in part on the bandwidth capabilities of computer system 120. In this process, the regional service 26 determines a respective form of receiving each real-time data stream from the unmixed real-time data streams and a stream mix derived from a combination of the real-time data streams. The regional service 26 also determines a network route that receives each real-time data stream from the direct peer-to-peer network routes and the network routes arbitrated by one or more other network nodes. After the flow handling topology has been determined, regional service 26 sends instructions to a real-time kernel operating on computer system 120. The instructions specify real-time data stream connections required between the computer system 120 and some other network nodes according to the determined stream handling topology.

FIG. 7 illustrates an embodiment of a method implemented by the real-time kernel 20 in determining the topology of a real-time data stream connection that delivers required data stream data to the computer system 120.

According to the method, the real-time kernel 20 determines whether the computer system 120 has sufficient bandwidth to receive the required real-time data stream data set 186 directly from other network nodes (fig. 7, block 188). In this process, other network nodes transmit link requests to computer system 120. The link request indicates a respective bandwidth requirement required by computer system 120 to transmit a respective set of real-time data streams. The real-time kernel 20 compares the total bandwidth required to establish the required direct connection with the download bandwidth currently available to the computer system 120.

If the available bandwidth is at least equal to the total required bandwidth, the real-time kernel 20 establishes a direct connection with other network nodes providing the required real-time data stream data (fig. 7, block 190). In this process, the real-time kernel 20 creates a socket (e.g., a TCP socket or a performance optimized dedicated real-time socket) between the computer system 120 and one or more other network nodes. The real-time kernel 20 processes the real-time data streams, including encrypting them, recording them, and passing the processed data streams to downstream software components needed for presentation in the user interface and transmission over the network 18.

If the available bandwidth is less than the requested bandwidth (FIG. 7, block 188), the real-time kernel 20 checks the stream mix list to determine if the regional service 26 is generating a stream mix that provides the requested real-time data stream data (FIG. 7, block 192). If the required stream mix is available, the real-time kernel 20 establishes a connection with the regional service 26 that transfers a copy of the required real-time data stream mix from the regional server 28 to the computer system 120 (FIG. 7, block 194). If the desired stream mix is not available, the real-time kernel 20 sends a stream mix request to the regional service 26 (FIG. 7, block 196). The regional service 26 generates the required stream mixes in response to the stream mix requests, if possible.

System architecture

A. Introduction to the design reside in

The communicants typically connect to the network 18 from client network nodes, which are typically implemented by general purpose computer systems or special purpose communication computer systems (or "consoles," such as network-enabled video game consoles). The network nodes perform a communication process that establishes real-time data stream connections with other network nodes and typically perform a visual rendering process that renders a view of each virtual area into which the communicant enters.

Fig. 8 illustrates an embodiment of a client network node implemented by computer system 120. Computer system 120 includes a processing unit 122, a system memory 124, and a system bus 126 that couples processing unit 122 to the various components of computer system 120. The processing unit 122 may include one or more data processors, each of which may be in the form of any one of commercially available computer processors. The system memory 124 may include Read Only Memory (ROM) and Random Access Memory (RAM) that store a basic input/output system (BIOS) that contains the start-up routines for the computer system 120. The system bus 126 may be a memory bus, a peripheral bus, or a local bus, and may be compatible with any bus protocol, including PCI, VESA, Microchannel, ISA, and EISA. Computer system 120 also includes a persistent storage memory 128 (e.g., hard disk drive, floppy disk drive, CD ROM drive, tape drive, flash memory device, and digital video drive) coupled to system bus 126 and containing one or more computer-readable media disks providing non-volatile or persistent storage of data, data structures, and computer-executable instructions.

A communicant may interact (e.g., enter commands or data) with computer system 120 using one or more input devices 130 (e.g., one or more keyboards, computer mice, microphones, cameras, joysticks, physical motion sensors such as Wii input devices, and touch pads, etc.). The information may be presented through a Graphical User Interface (GUI) presented to the correspondent on the display monitor 132, which is controlled by the display controller 134. The computer system 120 may also include other input/output hardware 136 (e.g., peripheral output devices such as speakers and printers). Computer system 120 is connected to other network nodes 138, 140, and 142 through a network adapter 138 (also referred to as a "network interface card" or NIC).

A number of program modules may be stored in the system memory 124, including an Operating System (OS)144 (e.g., Windows XP, available from Microsoft corporation of Redmond, Wash., U.S.A.)An operating system), the real-time kernel 20, drivers 146 (e.g., GUI drivers), network protocols 148, native software applications 150 (e.g., HUDs 84), and data (e.g., input data, output data, program data, registry 156, and configuration settings 152).

B. Operating system

Operating system 144 hosts software applications by providing the underlying operating system services used to create runtime execution environments on computer system 120. Exemplary types of services that are typically provided by operating systems are resource management, file management, security, authentication, verification, notification, and user interface (e.g., windows, menus, dialogs, etc.).

Services related to the management of resources (e.g., memory, processors, and I/O devices) of computer system 120 are typically implemented by the operating system kernel. File management may be implemented by the operating system kernel or may be implemented with a separate file system manager (e.g., an installable file system, which is available at some Microsoft WindowsWindowsProvided in the operating system). In opening a file (e.g., a computer data file or a software application file), the file system manager typically invokes an appropriate file system driver that looks up the disk storage location of the file in a database (e.g., a file allocation table such as FAT, FAT98, VFAT, MFT, and CDFS) that maps out the storage location of the file on disk. Other operating system functions such as security, authentication, verification, notification, and user interface may be performed by one or more other components of the operating system (e.g., some Microsoft Windows WindowsAn executable service layer in an operating system).

Exemplary types of services that are typically provided by an operating system kernel are process management, memory management, device management, and system call handling. Process management includes running applications and providing Application Programming Interfaces (APIs) to hardware components of a computer system. During the execution of a software application, the operating system kernel typically establishes an address space for the software application in memory, loads a file containing the software application code in the address space, and executes the loaded software application code. Memory management involves managing access to system memory 124 by software applications. Device management involves providing access to hardware devices through device drivers. System call handling involves providing APIs that expose operating system kernel services to user mode software applications. By calling APIs (e.g., through inter-process communication mechanisms and system calls), the software application may request services from the operating system kernel, pass parameters, and receive results generated by the services in response to the requests.

Operating system 144 typically stores hardware and software configuration information, user preferences, and settings information in registry 156. For example, the registry 156 typically contains the following information: parameter values required to boot and configure the system, system-wide software settings to control the operation of the operating system 144, security databases, and per-user profile settings. In some embodiments, connection rules 32 are stored in registry 156 rather than in a separate database.

C. Network protocol

Network protocol 148 controls or enables connections, communications, and data transfers between computer system 120 and other network nodes. Exemplary network protocol types include transmission control protocol/internet protocol (TCP/IP), user datagram protocol/internet protocol (UDP/IP), real-time transport protocol (RTP), and Session Initiation Protocol (SIP).

TCP/IP includes a TCP portion and an IP portion. The TCP portion of the protocol provides transport functions by breaking messages into smaller packets, reassembling the packets at the other end of the communication network, and resending any packets lost on the way. The IP portion of the protocol provides routing functionality by assigning addresses of the destination network and the target node at the destination network to the data packets. Each data packet communicated using TCP/IP includes a header containing TCP and IP information. IP does not provide guarantees of packet delivery to upper layers of the communications stack. TCP, on the other hand, provides guaranteed queued packet delivery to the next end-to-end transport service to the connection. In this way, the TCP protocol provides a reliable transport layer connection.

UDP is a message-oriented transport layer protocol that provides an interface between the application layer and the internet layer. UDP does not guarantee delivery of the message to the application layer. UDP is a connectionless protocol because no tree is built for establishing a dedicated end-to-end connection. The UDP sender does not retain state information about UDP messages after sending them. The communication is based on the transmission of messages in one direction from the source to the destination without checking the status of the receiver.

RTP defines a standardized packet format for communicating audio and video over a network connection. Various network protocols may be used to transmit and receive RTP data between network nodes, including peer-to-peer networking frameworks, centralized servers using TCP sockets alone or in combination with UDP, and multicast protocols.

SIP provides users with a means for locating each other, establishing communication sessions, and terminating active sessions. With SIP transactions, the session negotiation procedure is handled according to the Session Description Protocol (SDP).

D. Device driver

The device drivers 146 are typically implemented by software applications that enable other software applications (e.g., user-mode software applications and operating systems) to interact with hardware devices connected to the computer system 120. Device drivers typically provide APIs to functions that can be invoked by calls made by software processes in order to translate commands and data passed between the software processes and the hardware devices.

E. Real-time kernel

1. Introduction to the design reside in

The real-time kernel 20 includes services that control the processing and switching of real-time data flows between the computer system 120 and other network nodes sharing a virtual area communication environment, as well as services that render corresponding views of the virtual area and objects in the virtual area on the display monitor 132. In these processes, the real-time kernel interfaces with operating system functions that communicate with drivers 148 to pass commands and data to and from the hardware components of computer system 120 to exchange real-time data streams with other network nodes and present the communicant with an immersive virtual area communication experience.

The implementation of the real-time kernel 20 includes one or more discrete modules or discrete libraries (e.g., dynamically linked libraries), which are not limited to any particular hardware, firmware, or software configuration. In general, these modules may be implemented in any computing or data processing environment, including in digital electronic circuitry (e.g., an application specific integrated circuit such as a Digital Signal Processor (DSP)) or in computer hardware, firmware, device drivers, or software. In some embodiments, the functionality of the modules is combined into a single data processing component. In some embodiments, the respective functionality of each of the one or more modules is performed by a respective set of multiple data processing components. In some implementations, the process instructions (e.g., computer readable code such as computer software) for implementing methods performed by embodiments of the real-time kernel 20, as well as the data they generate, are stored on one or more computer readable media. Storage devices suitable for tangibly embodying these instructions and data include all forms of non-volatile computer-readable memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; DVD-ROM/RAM and CD-ROM/RAM.

2. Exemplary real-time Kernel functionality

The real-time kernel 20 cooperates with one or more network infrastructure services to establish and manage network connections between the computer system 120 and other network nodes sharing a virtual area communications environment. Exemplary functionalities involved in establishing and managing network connections are session management, remote control flow handling, and real-time task scheduling.

e. Session management

Fig. 9 illustrates an embodiment of a method implemented by the real-time kernel 20 in response to a real-time kernel API call requesting a connection to a virtual area.

According to the method of FIG. 9, the real-time kernel 20 determines the designation of a virtual region (FIG. 9, block 160). In some embodiments, this functionality of the real-time kernel 20 is invoked by initiating a real-time kernel API call to a real-time kernel service that includes a virtual region designation. The real-time kernel API calls may be made by any software application, operating system service.

The real-time kernel 20 establishes a session with the network infrastructure service hosting the specified virtual area (fig. 9, block 162). In this process, the real-time kernel 20 establishes a session with the regional service 26. The real-time kernel 20 then transmits a request to the zone service 26 for connection to the specified virtual zone. The region service 26 determines an instance of the virtual region specified in the request received from the real-time kernel 20. After determining the instance of the virtual area instance, the area service 46 determines whether the user capabilities satisfy the capability requirements associated with the virtual area instance. If the user capabilities meet the capability requirements, then the zone service 46 transmits a message indicating the availability of status data describing the current state of the virtual zone instance (e.g., a list of objects currently in the virtual zone instance, and the names of the communicants associated with those objects).

The real-time kernel 20 subscribes to state data describing the current state of the virtual area instance (fig. 9, block 164). In response to the subscription request, the area service 26 publishes status information to a channel on the link between the real-time kernel 20 and the area service 26.

In some embodiments, the So3D engine of the realtime kernel 20 invokes user interface services of the operating system 144 to present a human-perceptible view of the state data. For example, the So3D engine may invoke an interface service to present on display 132 a representation of each communicant associated with the object currently within the area. In some embodiments, the correspondent may be represented by an icon, thumbnail, or other graphic that is optionally labeled with the correspondent name. In some embodiments, the state data is presented in a graphical interface of the software application that triggers invocation of the real-time kernel 20. In some embodiments, the status data is presented in an embodiment of a Heads Up Display (HUD) interface 84 (see fig. 5A-5C).

After a connection has been established with the virtual area instance, the software application triggering the real-time kernel 20 to invoke may give the user the option of requesting entry into the virtual area instance or may automatically request entry into the virtual area instance on behalf of the user.

Fig. 10 illustrates an embodiment of a method implemented by the real-time kernel 20 in response to a real-time kernel API call requesting entry into a virtual area.

According to the method of fig. 10, the real-time kernel 20 declares an intent to enter a virtual area to the network infrastructure service hosting the virtual area (fig. 10, block 168). In this process, the real-time kernel 20 sends a message containing the assertion to the regional service 26. The message may be sent on a channel of an existing link with regional service 26 or on a new link established by real-time core 20 with regional service 26. In response, the zone service 26 determines whether the user capabilities satisfy the capability requirements associated with the virtual zone instance. If the user capabilities meet the capability requirements, the regional service 26 returns configuration data to the real-time kernel 20. From the description of the virtual area instance, the configuration data typically includes a definition of the virtual area instance, a registration of objects currently within the virtual area instance, and a set of real-time data stream sources and sinks associated with the objects in the virtual area.

The real-time kernel 20 initiates the transfer of at least one real-time data stream over at least one network connection with at least one real-time data stream source, wherein the at least one data stream source is respectively associated with at least one object in a virtual area (fig. 10, block 170). In this process, the real-time kernel 20 determines one or more network nodes associated with an instance of the virtual area based on the configuration data received from the area service 26. The real-time kernel 20 then initiates a transfer of at least one real-time data stream over at least one network connection with at least one determined network node. The connection between the real-time kernel 20 and other network nodes may be a peer-to-peer connection or a server-arbitrated connection. With respect to peer-to-peer connections, the connection target network node and the real-time kernel 20 typically authenticate each other and subsequently establish a link via which at least one real-time data stream is transferred to or from the connection target. The link is typically unidirectional and is requested by the transmitter and accepted or rejected by the receiver.

In the illustrated embodiment, the real-time kernel 20 processes the initiated real-time data stream according to at least one stream handling definition in the description of the virtual area instance (fig. 10, block 172). In this process, the real-time kernel 20 assembles a set of stream processing objects into a directed graph according to the stream processing configuration defined in the virtual area description.

f. Remotely controlled stream handling

Fig. 11 illustrates an embodiment of a method implemented by the real-time kernel 20 in response to a handle instruction received from the virtual area service 26.

According to the method of fig. 11, the real-time core 20 receives one or more stream handling instructions from a regional service 26 operating on a remote network node, wherein the stream handling instructions include a description of a stream handler for processing at least one real-time data stream (fig. 11, block 200). The real-time kernel 20 creates a stream handler from the stream handler description (fig. 11, block 202). A stream handler typically includes mixing functionality described in one or more stream handling instructions. The mixing function is for mixing the real-time data stream with at least one other real-time data stream to produce a mixed real-time data stream. The real-time core 20 generates the resulting data stream in a process that includes processing the real-time data stream by the created stream handler (fig. 11, block 204). In some embodiments, the process involves determining a configuration parameter value from real-time status information specified in one or more stream handling instructions, and dynamically configuring the stream handler with the configuration parameter value.

Fig. 12 illustrates an embodiment of a stream handler 206 created by a stream handler configuration manager 208 (which is a component of the real-time kernel 20) from stream handling instructions 210 received from the regional services 26. The stream processor configuration manager 208 is typically comprised of one or more other components that make up the service and real-time kernel 20. Stream handler configuration manager 208 constructs stream handler 206 from a set of processing objects (also referred to as processing primitives or PGEs). Each processing object is a software object capable of performing a specific function (e.g., a transform function, a discrete function, and a hybrid function) on a data stream. The stream handler configuration manager 208 instantiates one or more of the processing objects specified in the stream handling instructions and assembles the instantiated processing objects into directed graph components 212 of the stream handler 206 according to the description. In some embodiments, the flow handling instructions specify the processing objects with respective unique identifiers and the flow processor configuration manager 208 instantiates the processing objects by initiating calls to the processing object API that include respective ones of the identifiers. The stream handler 206 is configured to process a plurality of data streams 214 of a particular data type (e.g., audio, video, and motion data types) through respective processing chains 216-218, which are made up of respective ones of the processing objects. Stream handler 206 additionally includes a hybrid object 220 (which is specified in one or more stream handling instructions). In operation, stream handler 206 executes hybrid object 220 to generate hybrid real-time data stream 222 from the combination of processed real-time data streams 216-218. In some embodiments, at least one instantiated processing object encapsulates a corresponding call to a driver module 224, which controls the hardware components of the local network node based at least in part on the resulting data flow 222.

g. Real-time task scheduling

FIG. 13 illustrates an embodiment of a method implemented by the real-time core 20 in scheduling tasks for execution by the real-time core 20.

According to the method of fig. 13, the real-time kernel 20 establishes at least one real-time data stream connection with at least one remote network node (fig. 13, block 230).

The real-time core 20 processes at least one real-time data stream originating from a remote network node (fig. 13, block 232). In this process, the real-time core 10 processes the at least one real-time data stream through one or more real-time data processing operations to produce a resultant data stream.

The real-time kernel 20 monitors the processing of the at least one real-time data stream (fig. 13, block 234). In some embodiments, the real-time kernel 10 monitors one or more of the following parameters: a rate of resulting data generation, utilization of at least one processor of the local network node, and bandwidth utilization of at least one networking resource of the local network node. In some embodiments, the real-time data stream is packetized into frames and the real-time core 20 monitors the processing of each frame during each successive fixed-length interval set according to a local clock, which is typically synchronized with the remote master clock service. Based on this monitoring, the real-time core 20 determines whether the processing of the real-time data stream deviates from the performance goal. In some embodiments, the performance goal comprises a time-based threshold for producing the resulting data stream. For example, in some embodiments, the performance goal is a predicate (i.e., a condition) on the rate at which frames of the resulting data stream are produced. Exemplary performance targets of this type include a target threshold and a target range.

In response to determining that the processing of the at least one real-time data stream deviates from the performance goal, the real-time core 20 modifies the processing according to a real-time performance goal routine (FIG. 13, block 236).

Fig. 14 shows an embodiment of a method performed by the real-time core 20 based on monitoring of the processing of at least one real-time data stream. In response to determining that the processing of the at least one real-time data stream does not meet the performance goal (FIG. 14, block 238), the real-time core 20 reduces the computing resource load to a lower level (FIG. 14, block 240). Depending on the design of the real-time performance goal routine, the real-time kernel 20 typically reduces computing resources in one or more of the following ways: the real-time core 20 may ignore the processing of one or more portions of the real-time data stream (fig. 14, block 242); the real-time core 20 may ignore one or more real-time data processing operations (fig. 14, block 244); and the real-time core 20 may replace at least one real-time data processing operation with a different respective data processing operation (fig. 14, block 246). In ignoring the processing of one or more portions of the real-time data stream (fig. 14, block 242), the real-time core 20 typically performs at least one of the following operations: ignoring one or more data processing operations characterized by corresponding performance values outside of the performance target; and prioritizing the one or more data processing operations based on priority values respectively assigned to the respective data processing operations.

If the processing of the at least one real-time data stream meets the performance goal (FIG. 14, block 238) and the computational load has been reduced to a lower level by any of the methods described above (FIG. 14, block 248), the real-time kernel 20 increases the computational load from the lower level (FIG. 14, block 250). The real-time kernel 20 typically increases the computing resources according to the heuristic by reversing one or more operations for reducing the computing resource load at block 240. If the processing of the at least one real-time data stream meets the performance objectives (FIG. 14, block 238) and the computational load is not reduced to a lower level by any of the methods described above (FIG. 14, block 248), the real-time kernel 20 maintains current processing of the real-time data stream.

In some embodiments, the real-time kernel 20 instantiates a processing object that performs a respective number of data processing operations on at least one real-time data stream. The real-time kernel 20 builds a directed graph from some instantiated processing objects and processes the at least one real-time data stream through the directed graph. Depending on the real-time performance goal routine, the real-time kernel 20 may modify the processing of the real-time data stream by deleting one or more instantiated processing objects from the directed graph. In some embodiments, processing objects are assigned respective priority values, and the real-time kernel 20 deletes processing objects by removing some instantiated processing objects from the directed graph based on the assigned priority values. For example, in some of these embodiments, the deleting includes removing from the directed graph some instantiated processing objects that are assigned respective priority values that do not satisfy the priority threshold.

In some embodiments, the real-time kernel 20 builds a second directed graph from some instantiated processing objects, the second directed graph being used to process a second real-time data stream originating from one of the local network node and the at least one remote network node. In some of these embodiments, the first and second directed graphs are assigned respective priority values, and the real-time kernel modifies processing of the first and second real-time data streams by preferentially modifying one of the first and second directed graphs based on the assigned priority values. For example, the real-time kernel may offload the one of the first and second directed graphs that is assigned the lowest priority value.

In some embodiments, the real-time kernel 20 processes a second real-time data stream through the directed graph, wherein the second real-time data stream originates from one of the local network node and the at least one remote network node. In some of these embodiments, the first and second real-time data flows are assigned respective priority values, and the real-time core 20 prioritizes processing of one of the first and second real-time data flows based on the assigned priority values.

In some embodiments, the real-time kernel 20 establishes respective real-time data stream connections between the local network node and a plurality of remote network nodes. The real-time kernel 20 processes the real-time data streams originating from the respective ones of the remote network nodes through the directed graph. In some of these embodiments, the real-time data flows are assigned respective priority values, and the real-time core 20 prioritizes processing of one or more real-time data flows based on the assigned priority values. A directed graph typically includes a plurality of directed chains of respective instantiated processing objects. The real-time kernel 20 typically processes a respective one of the real-time data streams through each of the directed chains. In some of these embodiments, the real-time kernel 20 iteratively modifies the processing of the real-time data stream until the processing falls within specified performance goals. During each iteration, the modification typically includes performing one or more of: (i) removing one or more chains from the directed graph and (ii) deleting one or more instantiated processing objects from the directed graph.

Exemplary real-time Kernel embodiments

A. Introduction to the design reside in

Fig. 15 illustrates an embodiment 260 of the real-time kernel 20. The real-time kernel 260 supports remote configuration and execution of 2D/3D graphics rendering engines, audio mixing and switching engines on different network nodes to create the perception of physical presence between two or more communicants. In managing all communicants interacting in a single virtual area instance at the same time, regional service 26 remotely configures sessions between real-time kernel 260 and other network nodes via a packet transport component of real-time kernel 260 (referred to herein as STRAW service 268). In some embodiments, the real-time kernel 260 configures the data streams (e.g., real-time audio data streams) point-to-point (P2P) to minimize communication with the zone services 26. The regional service 26 may also mix and deliver the data streams to the client network nodes as necessary. Real-time kernel 260 will report P2P connection failures to area service 26 so that area service 26 can determine when to mix data streams for client network nodes. The real-time kernel 260 has a small initial coverage area and loads updates and add-on functionality as plug-ins over the network connection.

The real-time kernel 260 includes a set of managers and services. The real-time kernel manager has a connection and service mix manager 262, an area/zone manager 264, and a plug-in manager 266. The real-time kernel services include STRAW service 268, SODA handler service 270, media service 271, audio streaming service 272, So3D interface service 274, asset caching service 275, one or more social servers 277, record, playback and transport bus service 276, real-time scheduler service 278, time service 280, SIP service 282, local HID/RDS driver handler service 284, and services for including local audio playback 286, local speaker 288, local microphone 290, and Skype An interface service of a local audio device for audio. In one exemplary embodiment, the real-time kernel 260 is implemented by the following runtime packaging components:

library:

other plug-ins:

encryption algorithm

Compression algorithm

And (3) authentication algorithm:

voucher(s)

Audio mixing

Audio source

Audio codec

Audio frequency calculation

Graphic effects

Physical extension

Script extension

Input device main memory

B. Real-time kernel design

As shown in fig. 15, the real-time kernel 260 is designed as a collection of services, plug-ins, and real-time schedulers, which constitute a platform for presenting virtual regional communication environments according to instructions received from the regional services 26. The services cooperate to implement the platform to operate at different levels depending on network characteristics through audio and graphical presentation configurations. The plug-ins are heterogeneous, each attached to a plug-in management API and each having its own class API. The real-time scheduler 278 ensures that audio and graphics rendering occurs at a uniform frame rate. The platform is configured by the area service 26 from an instance of a virtual area by SODA definition records communicated over a STRAW UDP socket (see section VI, which contains the SODA definition for an exemplary set of SODA records). The STRAW service 268 demultiplexes SODA record streams using a publish/subscribe model. The SODA record is only transmitted when the subscriber is present on the other end of the STRAW socket. The received SODA records are delivered to one or more subscribers upon arrival. The service supports native APIs for use by the So3D graphics engine and HUD 84.

The following sections describe the installation, design, and operation of embodiments of real-time kernel 260 and its components.

1. Mounting of

a. Overview

In some embodiments, the virtual area-based presentation platform is downloaded as an installation package via the internet as a software package. Which is delivered from the download server by HTTP download. In operation Microsoft WindowsOn the client network node of the operating system, the platform software package is the. msi package. The initial installation is a single package that is modified on the download server when the update becomes available. When a new client network node performs a current installation, no further updates are needed until such time as a subsequent update is created.

The real-time kernel 260 utilizes plug-ins to customize applications. The necessary plug-ins are included in the installation package. From time to time, components may be updated independently (e.g., real-time kernel services may be repurposed and plug-ins may be added). In this case, a separate Windows release may be created for the fix issueMsi install package and register with update server. The installed platform software will be notified of the update and will provide the communicator with an upgrade option. Some communicants may delay an upgrade until more than one update is available. When the correspondent finally agrees to the upgrade, all available updates will be loaded and applied sequentially.

Multiple versions of a plug-in may be presented at the same time on the client network node. This is because client network nodes typically agree on featuresAnd selecting plug-ins that fit the API and version requirements. Each plug-in advertises its APIs and variables. These plug-ins will have different file names to avoid name conflicts in the file system. Two plug-ins with the same API and different variables are different implementations and the selection is made by the service requesting the plug-in (perhaps by, for example, negotiation with the server). This is a bug fix when a plug-in is loaded that has the same API and variables as the existing plug-in. The new plug-in replaces the old plug-in. The service is always replaced by an upgrade. Never have two services with the same API. Windows (R) WindowsWindows installing usage manifests and bundlesRelated DLL to guarantee no matter WindowsHow the updated state of the environment is effective. Windows (R) WindowsThe side-by-side feature is used to avoid conflicts with other product installation requirements.

b. Update server

The update server contains an installation package for each supported hosting operating environment, and an upgrade package for each previously supported installation package.

The client network node and the update server communicate over a reliable STRAW channel. The update server publishes the available upgrade definitions for each supported hosting operating environment. The virtual area-based rendering platform software installed on the client network node may then subscribe to the upgrade. The update server begins sending the desired software blocks.

In some embodiments, a client version and upgrade tool is installed on each client network node to allow the user to view the current client software version, list available upgrades, and initiate and monitor the upgrade process. The client network node will maintain a table of GUIDs for the upgrade package that have been applied. The table will present the list to the update server and in return get a list of pending updates by GUID in application order. They will have the attached description, size and date attributes.

Upgrades are only marked as "applied" when the download is complete and the automatic installation report is successful. The automatic installation process includes stopping any running SORK service so that the DLL can be rewritten. Downloading an upgrade is done through a series of SODA records so the process can be interrupted and resumed without repeating any data transfers. The record includes the update GUID and the offset.

Since there is no requirement for rollback or reinstallation, there is no need for any Microsoft Windows"side-by-side" library inventory. Any required libraries may be loaded as part of the upgrade.

The upgrade tool will generate a registry key or maintain a file containing the GUID of the upgrade being applied, any GUIDs of the currently loaded upgrade, its offset, and a reference to the file on disk containing the data "so far". Once applied, the upgrade package is deleted. If it is desired to cache upgrade packets for multiple client network nodes, each of them is pointed to at the same client agent that will do the caching.

c. Updating a local database

The client network node stores the virtual area-based rendering platform services and plug-ins in an asset directory in a local file system. These services and plug-ins are self-describing through APIs and accompanying resources. No other information about the client network node software state is retained. When a client network node re-installs virtual-area-based rendering platform software, the existing plug-ins are typically re-authenticated, perhaps after an OS (operating system) upgrade. The new installation includes all basic services and plug-ins, but there may be optional or application specific plug-ins on the machine that are typically deleted or re-verified. In some embodiments, the binary content of a valid plug-in is hashed and unidirectionally encrypted, and the resulting value is stored as an attached resource for checking whether the plug-in is trusted. To verify a suspect plug-in, the current plug-in content is re-hashed and encrypted, and the resulting value is compared to the existing resources. If the content does not match the resource, the plug-in is invalid.

d. Client authentication

Network authentication typically occurs each time the real-time kernel 260 boots. In some embodiments, an account server running an account network infrastructure service is used to authenticate a correspondent and establish a true user identifier (RUID) for the correspondent. The account server creates a token (subsequently included as part of the RUID) and gives the token to the client network node to authenticate itself to other servers. In this process, the client network node is securely issued a credential at installation. The credential is typically a CA-defined credential issued by a certificate authority. The certificate contains a private key and a public key. The virtual area based presentation platform installation package creates a new credential containing only the public key. The private key is securely stored on the client network node. The virtual area based presentation platform installation package creates a signature using a private key to encrypt a digest of a password provided by the correspondent and securely transmits the signature to the account server. The account server recovers the digest and stores it as the client identification secret.

When a connection is established, the live kernel 260 shares credentials with the account server. The account server responds with its credentials (e.g., a server-side certificate). The client network node and the account server verify the credentials using a registration authority. Once verified, the server-side credentials are valid for any server anywhere.

In some embodiments, the account server also provides the client network node with a random 128-bit challenge phrase. The client network node hashes the challenge phrase with a cryptographic digest of the password provided by the correspondent and returns it as a response. The account server also hashes the challenge phrase with the previously obtained digest of the correspondent and verifies that the responses from the client network nodes are a match. The network connection is now authenticated and the correspondent is identified as the owner of the private key.

In some embodiments, the account server assigns a random client ID with an attached signature to the correspondent. The signature is a 128-bit hash of the client ID encrypted using the account server private key. The signature can only be created by the account server. Anyone who receives the token can authenticate the correspondent by decrypting the digest using the public key published by the account server and comparing the digest with the client ID.

e. Account server authentication

FIG. 16 illustrates an embodiment of a method of authenticating an account server 296 through credentials of the account server. According to the method, the client network node 294 and the account server 296 exchange credentials (fig. 16, blocks 298, 300). The client network node 294 issues a server ID and server token to the account server 296 for subsequent quick authentication of the account server 296 with the client network node 294 (fig. 16, block 302). The account server 296 then issues the client ID and the attached identification token to the client network node 294 (fig. 16, block 304). The authentication phrase on the stream to the account server is encrypted using the public key of the participant.

2. Initialization sequence

Fig. 17 illustrates an embodiment of a method implemented by the loader component of the real-time kernel 260 each time the operating system on a client network node is launched. In this process, the loader parses a static kernel component list that includes one or more kernel service components (FIG. 17, block 320). The loader determines all kernel components in the resolved list that are missing in the local repository (e.g., a directory in the local file system) (fig. 17, block 322). The loader retrieves each kernel component determined to be missing (FIG. 17, block 324). In some embodiments, the loader instantiates an update service on the client network node that retrieves the missing kernel component from a remote network node (e.g., a download server or an update server). After the missing kernel components have been retrieved, the loader instantiates kernel services from respective ones of the kernel service components (FIG. 17, block 326). The instantiated kernel service is executed to communicate with one or more remote network nodes in a communication environment defined with respect to the virtual area (fig. 17, block 328). For example, in some embodiments, the HUD84 invokes a kernel service to communicate with the regional server in order to establish a HUD session or regional session as described in detail herein.

In some embodiments, the following services of the real-time kernel 260 are loaded as Windows at boot timeAnd the service DLL comprises the following steps:

STRAW service 268

SODA service 270

Media service 271

Audio streaming service 272

Connection and server mixing manager 262

Area/zone manager 264

Asset caching service 275

Real-time scheduler service 278

·HUD 84

Default plug-in

In these embodiments, the service is loaded by name rather than by GUID. Only one copy of each service is presented at a time on the client network node. After loading, the SODA channel service 270, media service 271, audio streaming service 272, region/zone manager, and real-time scheduler services wait idle. The connection and server mix manager causes the audio to not be configured and waits for the definition of a connection to the zone server. The default plug-in is registered as an API class object by the GUID. The default plug-in is loaded when referenced in the definition by the GUID. The HUD 84 contacts the account server, authenticates, and identifies the correspondent. The HUD 84 creates a stream to the aggregated network infrastructure service and the interactive network infrastructure service and populates its Most Recently Used (MRU) friends and area list and its frequent friends and area list. The asset caching service 275 typically contacts the pattern database server upon initiation and begins caching the digital asset and updating its GUID mapping.

3. Conversation

The real-time kernel 260 manages sessions between the client network node and other network nodes. During the session, data is shared between the server and the client network nodes as SODA defined records on STRAW sockets. Data is shared in a publish/subscribe model. The real-time kernel 260 subscribes only to data required by the client network node. For subscription, the real-time kernel 260 creates a STRAW channel with the desired server. The STRAW channel is agreed upon by the well-known GUID for a particular virtual area. In some embodiments, the STRAW sockets are connected using addresses provided by the configured DNS.

The regional service will send a publish message indicating the data streams available to the correspondent and tagging each data stream with a GUID handle. The real-time kernel 260 then sends a subscription message for the desired data stream. Any changes to the regional service data for subscribed channels are sent as SODA definition records to all client network nodes that have subscribed to those channels.

There are two main types of sessions: (a) a HUD session involving the display of current relationship and presence information in the HUD 84; and (b) a zone session, which involves latent or entry into a virtual zone instance.

HUD session

In the HUD session, the HUD 84 contacts the account server, RUID server, and rendezvous server and subscribes to the correspondent's own account and relationship information over the STRAW channel. The HUD 84 then subscribes to presence information for the closely related contacts and virtual areas. At this point, the HUD 84 may display dynamic presence information for closely related contacts.

b. Regional conversation

In the zone session, the HUD 84 subscribes to information about the relevant virtual zone. In some embodiments, the directory server is consulted to determine the current region server hosting the virtual region specified by the HUD. A STRAW stream to the current area server is created.

The HUD subscribes to presence data associated with the virtual area and updates its 2D heads-up display with the names of other communicants currently participating in the virtual area. At this point, the correspondent is "latent" in the virtual area. The presence of the correspondent may be displayed in a pop-up list and the icon displayed in the HUD area representation (e.g., in the office space shown in fig. 5A-5C).

If the communicant directs the HUD 84 to enter the virtual area, then the real-time kernel notifies the rendezvous service that the communicant requests entry into the virtual area. Other communicants who subscribe to presence information associated with a virtual area are notified of new communicants who have entered the virtual area.

The real-time kernel 260 directs the So3D engine to start the interactive environment. The So3D engine subscribes to zone server environment data (e.g., presence and motion data). The regional server begins streaming the requested regional server environment data to the real-time kernel 260. The real-time kernel passes the requested data to the So3D engine, which presents the data according to the current visualization mode (e.g., 2D top view, low resolution view, or fully immersive 3D view).

The zone server defines raw microphone audio media streams between client network nodes associated with objects in the virtual zone. The zone server also creates definitions of audio mixing elements from the audio handling instructions (e.g., spatial effect definitions and zone definitions) in the virtual zone description. The connection and server mixing manager 262 listens for audio definitions that include a GUID handle for each P2P audio stream and creates a media stream for each definition. Each media stream is registered with the local transport bus 276 and the appropriate audio mixing component is created with the audio streaming service 272. The area/zone manager 264 also subscribes to SODA definitions for audio and for avatar motion and orientation data. The region/zone 264 controls the gain/mute of each audio stream as the communicant's avatar navigates within the virtual area.

In some embodiments, the area/zone manager 264 additionally subscribes to relationship data that the area/zone manager 264 uses to control the orientation/movement/pose of the avatar in the virtual area via the social processor 277 (see fig. 15). In this process, the area/zone manager 264 sets parameter values for the social processor 277 based on the avatar's location in the virtual area and the relationship database. In this way, when the communicant speaks, the relationship may be indicated by changing the position and orientation of the avatar's head (e.g., turning it to face another avatar when the avatar enters a zone of the virtual area, or orienting the avatar to get the best view on the viewing screen when entering a media zone of the virtual area). In some embodiments, social processor 277 is defined by a third party developer and passed to the client network node via a plug-in. Each social processor 277 is a set of instructions that is automatically executed upon the occurrence of a particular event (e.g., automatic movement triggered by proximity to other avatars, or a location in an area, or both). Social processor 277 may be any arbitrary programmatic routine that controls the movement of an avatar or object in a virtual area. For example, in some embodiments, if an avatar is close to the viewing screen, one type of social processor automatically quickly moves the avatar to the network defined in the virtual area description and centers the avatar in front of the viewing screen so that the user can easily see the content of the viewing screen. In this way, the need for complex manipulations of the mobile avatar is eliminated. Another type of social processor 277 automatically rotates and turns the avatar to confirm the presence of another user. For example, embodiments of social processors of this type are configured to automatically redirect avatars in a virtual area from facing each other to respective orientations in which the avatars face a new communicant in response to the new communicant entering the virtual area. In this case, the communicant associated with the avatar originally in the virtual area does not have to manually manipulate his avatar; instead, the social processor automatically rotates its head to confirm the presence of the new communicant.

4. Managing sessions

Fig. 18 illustrates an embodiment of a session management method implemented by the STRAW service 268.

According to the method of fig. 18, at the local network node, the STRAW service 268 establishes a first session with the remote network node over the transport stream according to a connectionless transport protocol (e.g., UDP) (e.g., fig. 18, block 362). The STRAW service 268 creates a definition of the session, where the definition includes an Internet Protocol (IP) address, a port address, and a globally unique identifier of a transport protocol. The STRAW service 268 sends the definition to the remote network node. The STRAW service 268 determines the first station definition assigned to the remote network node and stores the first station definition in a table as an attribute of each open channel. In this process, the STRAW service 268 parses the station definition records received from the remote network node. The station definition record includes a set of fields, where each field is defined by a corresponding field type and an associated field value, and each field type is identified by a corresponding Globally Unique Identifier (GUID).

The STRAW service 268 automatically opens one or more channels for transmitting data between the local network node and the remote network node in the first session on behalf of one or more software entities on the local network node (fig. 18,. block 364). In this process, the STRAW service 268 sends to the remote network node a record defining the local published channels and a record for each local subscribed channel with an identifier that matches the identifier of one of the remote published channels.

In the first session, the STRAW service 268 maintains a table identifying some channels that are open and associates corresponding attribute values with the identified channels (fig. 18, block 366). The STRAW service 268 records attributes of local published channels available from local network nodes, local subscribed channels requested by one or more software entities, remote published channels available from remote network nodes, and remote subscribed channels requested by remote network nodes. In this process, the STRAW service 268 maintains a record for each local published channel that includes an identifier of one of the software entities indicating the ability to publish data on the local published channel, an identifier of the remote network node that subscribes to the local published channel, and an identifier of the local published channel. The STRAW service 268 maintains a record for each local subscription channel that includes an identifier of one of the software entities that subscribes to the local subscription channel, an identifier of the remote network node that indicates the ability to publish data on the local subscription channel, an identifier of the local subscription channel, and one or more network transmission parameters associated with the local subscription channel. The STRAW service 268 maintains a record for each remote published channel that includes an identifier of the remote network node indicating the ability to publish data on the remote published channel, and an identifier of the remote published channel.

The STRAW service 268 communicates data between the local network node and the remote network node on one or more open channels in a session. In some embodiments, the data is transmitted in a form in which each record includes a set of fields. Each field of a record is defined by a corresponding field type and associated field value, and each field type is identified by a corresponding GUID. Some of the records are media records that contain media data that includes packets of renderable data. The other record is a configuration record containing configuration data that includes a definition of the configuration settings. Media recording and configuration recording are typically encapsulated in transport recording on a transport stream. Media records are typically compressed using a first data compression service, while configuration records are typically compressed using a second data compression service. Upon transmission, the STRAW service 268 associates the transmission records with identifiers of respective ones of the channels on which the transmission records are transmitted, encrypts the transmission records, and arranges the encrypted transmission records in order. Upon receipt, the STRAW service 268 decrypts the transport records and dispatches the media records and configuration records contained in the decrypted transport records to the subscribing software entities.

In response to determining that the first session failed, the STRAW service 268 automatically attempts to establish a second session with the remote network node on a second transport stream according to the connectionless transport protocol (fig. 18, block 368). In some embodiments, the STRAW service 268 determines that the first session failed in response to determining that the current station definition assigned to the remote network node is different from the first station definition assigned to the remote network node when the first session was established.

In response to successfully establishing the second session, the STRAW service 268 automatically opens each channel identified in the table (fig. 18, block 370).

5. Processing a data stream

The real-time kernel 260 supports remote configuration of stream handlers for processing data streams received by client network nodes from other network nodes. In response to instructions received from regional services 26, various services and other components of real-time kernel 260 cooperate to construct and configure a directed graph of processing elements as a stream handler for processing data streams. The zone service instructions configure the stream handler according to the virtual zone application hosted by the virtual zone managed by the zone service 26.

Fig. 19 illustrates an embodiment of a method implemented by components of real-time kernel 260 in response to remote stream handling instructions received from regional services 26.

According to the method of FIG. 19, the real-time kernel 260 parses a description of the real-time stream handler from one or more stream handling instructions (FIG. 19, block 330). In this process, STRAW service 268 receives a SODA definition from regional service 26 for configuring the stream handlers. The STRAW service 268 dispatches the SODA definition to the connection and server mix manager 262 and the area/zone manager 264. The connection and server mixing manager 262 parses the input source identifier, the output sink identifier, and the respective identifier for each of the one or more data processing objects from the one or more stream handling instructions.

The connection and server mixing manager 262 instantiates real-time stream handle objects corresponding to respective ones of the identifiers (fig. 19, block 332). The connection and server hybrid manager 262 registers the instantiated objects with the transport bus 276.

The transport bus 276 creates a directed graph including some instantiated real-time stream handle objects according to the description (fig. 19, block 334). The area/zone manager 264 passes the audio calculation SODA definition to the audio calculation objects specified within the directed graph.

The STRAW service 268 receives the real-time data stream from the input source corresponding to the input source identifier (fig. 19, block 336). The STRAW service 268 passes the real-time data stream to the media service 271, which processes the stream and passes it to the transport bus 276. The transport bus 276 sequentially executes the processing primitives of the stream processor to perform the specified processing of the real-time data stream.

The stream handler generates the resulting data stream at the output sink corresponding to the output sink identifier (fig. 19, block 338). The resulting data stream is then passed to a rendering component of the client network node.

6. Services and other components of a real-time kernel

The components of the real-time kernel 260 include services, plug-ins, and libraries.

a. Compressor bank

The compressor library implements an optional compression layer for the transmitted data. It is not intended to compress protocol headers or link negotiation exchanges.

The compressor library is used by the SODA channel service 270 and by the media service 271. When encryption is configured, these services 270, 271 create two compressor instances and pipeline the channel data. One compressor instance is used for transmission and another compressor instance is used for reception.

The compressor library uses compression/decompression plug-ins configured by variables. The compression variables are agreed upon by the service and provided to the compressor.

b. Audio streaming service

In some embodiments, the audio streaming service 272 is WindowsThe DLL is served.

The audio streaming service 272 manages audio streaming mixing. It defines an API for creating and configuring audio processing primitives (also referred to as audio component objects) that are manipulated by zone/zone manager 264. The audio streaming service 272 is a client of the transport bus 276. All audio processing primitives are registered with the transport bus 276.

The audio processing primitives are created by the plug-in manager 266(plug inmgr) via the following API calls:

API enumeration (gui plug-in API)

Plug-in manager variable enumeration (guid identifier, guid plug-in Api)

Plug-in manager create instance (guid identifier, guid plug-in Api, guid variable)

Create instance () API produces a guid plug-in that represents an instance of the API's desired variables.

The plug-in APIs for audio processing primitives are:

audio mixing

Audio source

Audio insertion

Audio transmission

The calling program provides a variable that is passed only by the audio streaming service 272. The variables represent the implementation of the audio processing primitives, as defined in the virtual zone application. The audio streaming service 272 then creates an audio component object that encapsulates the plug-in instance (guid plug-in). The audio component uses the plug-in manager 266 to access the method of the plug-in. The audio streaming service 272 creates the correct type of derived audio component for each audio API:

audio mixing

Audio source

Audio insertion

Audio transmission

The audio component object is registered with the transport bus 276. The transport bus API is used to link components into a graph. The audio component API supports the operation of the transport bus 276 and the zone/zone manager 264. The base class audio component has APIs for audio sources and audio sinks, both aspects of the same plug-in.

STRAW service

(i) Overview

In some embodiments, STRAW service 268 is WindowsThe DLL is served.

The STRAW service 268 implements a STRAW transport protocol that enables connection-oriented cryptographically secure sockets between network nodes over a connectionless transport protocol (e.g., UDP). The STRAW transport protocol uses a fixed length Globally Unique Identifier (GUID) to identify all records and all field types in the records. For example, in some embodiments, a network node (or station) is defined by an IP _ address and a port. In these embodiments, the STRAW station identification record defines a particular network node with the following GUID set: { GUID1, GUID2, GUID3, GUID4, GUID5, GUID6}, wherein

GUID1 identifies the STRAW record as a SODA record;

GUID2 identifies the STRAW record as a network node identification record;

GUID3 is an IP address field tag;

GUID4 is the IP address of the network node;

GUID5 is a port number field tag; and

GUID6 is the port number of the network node

In these embodiments, the station identifier record contains binary data that can be easily compressed to a smaller size. In some embodiments, the size of one or more STRAW records may be further reduced by omitting the field flag. In these embodiments, the transmitter and receiver of the STRAW record are aware of the format of the STRAW record so that the semantics of the field value are known without reference to any field tag.

Referring to fig. 20, the STRAW service 268 manages a session 340 on a transport stream 342. In some embodiments, a flow in the context of a STRAW session is defined by a pair of (IP, port) addresses and a transport GUID. The session includes zero or more logical channels, where a channel is a series of records that are appropriate for a particular kernel manager (e.g., So3D graphics engine 274, connection and server hybrid manager 262, and zone/zone manager 264). More than one kernel manager may receive records from the same stream, the records being distinguished by channels.

The STRAW service 268 manages two types of channels: a media channel containing streaming data (e.g., audio); and a SODA channel containing SODA definition records (or instructions). STRAW records encapsulate SODA records and media records on the stream. The STRAW record is encrypted, serialized, and includes a message integrity field. This sequence is independent of the source or destination of the record, which is a link-level feature for detecting out-of-order or missing records.

The STRAW record is identified by a channel. The GUID is used as a channel identifier. The SODA and media records may be compressed at the channel level into a stream independent of the STRAW record encapsulation. Each SODA record contains one or more SODA definitions 344. Examples of SODA definitions include processing primitives (e.g., audio mixes and audio effects), 3D rendering assets (e.g., textures and meshes), and RDS (e.g., avatar motion checkpoint). Each media recording contains one media packet 346. Examples of media packets include audio codecs and text.

The application publishes the channel using the well-known GUID ID on the session. The kernel manager subscribes to the channel. The publish/subscribe model is connectionless. The kernel manager that subscribes to a channel registers to receive notifications of channel state changes and channel records as they arrive.

(ii) Stream-session-channel-recording

In the context of a STRAW session, a flow is a bi-directional UDP socket between two network nodes defined by two IP address/port pairs and a transport GUID. The stream supports a session of the channel. A session is a logical node-to-node connection. A session is a two-node transport channel. A session may be transmitted through one or more agent stations and through a stream that may contain multiple sessions.

A channel is a logical structure that conveys SODA or media records between two network nodes in a session. The channel may be reliable or unreliable, compressed or uncompressed. The content of the channel is identified by a content GUID. The channel records are transmitted in a series of STRAW records that share the same header channel _ client ID and have a sequential packet number and MAC field. The MAC calculation depends on the packet sequence on a given channel in only one direction. All records transmitted on a single channel share a single set of configuration parameters (e.g., { client, reliable, compressed }). The records on a single channel are compressed into a serial stream. Typically only reliable channels can be compressed. In some embodiments, unreliable channels may be compressed with a compression process in which compression is restarted on every key frame. In the case of lost packets on an unreliable channel, records on that channel are discarded until key frames are reached (because they cannot be decompressed out of order).

Compression uses compression. To improve compression, the channel definition may include preloaded data that may run through the compressor but is not transmitted. The goal is to populate the compression state table with the common phrase. The compression state table is reset and reconstructed each time a key frame is received.

(iii) SODA recording

The SODA records are nested structures having an initial GUID ID and one or more SODA definitions. The SODA definition has a definition type, a definition length, and one or more fields. The definition type is a well-known GUID (e.g., GUID asset). The length indicates the total size of the field. Fields are a combination of type-specific fixed fields and nested SODA definitions. That is to say that the first and second electrodes,

and (3) recording by using the SODA:

guid ID

definition of SODA

…

The definition of SODA:

guid definition type

Long length

[ field ] -dependent on definition type

Field:

fixed field

Or SODA definition

For example,

the SODA record is encapsulated in a STRAW record:

(iv) channel reliability and link level protocol

The STRAW record is numbered and contains a channel ID. After receiving the packet and a short delay, the transmission sends an ACK record containing the next expected packet number for each channel so that the sender can acknowledge that the transmitted record was received and can free up local resources. There is no reliability feature for this ACK except for periodic transmissions. This scheme achieves reliability with minimal network resources, assuming that nearly all records are successfully received.

The MAC field is calculated for each STRAW record transmitted. It is checked on receipt.

For reliable channels:

if a record in the channel is received out of order, a NACK is transmitted for the missing record. MAC failure also results in a NACK being transmitted for the desired record. A single record allows up to four NACKs and then the transmission queues the failure message to any subscribing kernel manager and erases the channel definition.

For unreliable channels:

if a record in a channel is received out of order, the missing packet number is signaled to any manager subscribing to the channel and no NACK is sent. MAC failure is also indicated as a lost packet to any kernel manager subscribing to the channel and no NACK is sent. There is no threshold for lost packets and the channel is never turned off by the transmission.

There is no need to "close" the channel. If all kernel managers do not subscribe, data transfer on the channel stops. Since the channel is a logical entity, operating system resources are not used.

(v) Publish/subscribe

The STRAW service 268 maintains a list of locally published and subscribed items. Each item includes:

create local manager for the item

Server identifier

Channel identifier

Publishing or subscribing to

Transmission parameters (for subscription)

Initialize lists with

{ STRAW service, GUID _ NULL, Session, subscribe, reliable, uncompressed }

Thus, the STRAW service 268 subscribes to all arriving SODA records that arrive on any session channel. These records include publication and subscription definitions. GUID _ null channels are never published and are assumed by each server to be subscribed with a well-known channel ID on each stream.

The STRAW service 268 also maintains a table of all arrival publication definitions for use in the case of a subsequent registration of a subscription in the local list.

{ ID client, ID Server, ID channel }

In the case where the ID client is the (possibly empty) GUID of the particular client for which the channel is intended, the ID server is the remote source of the channel record and the ID channel is the well-known GUID of the channel.

When the STRAW service 268 receives a session definition for a desired connection to another station, the STRAW service 268 establishes a flow, sends the session definition, and sends all local table publications in the SODA record on the session channel. When the publication definition arrives at the STRAW service 268, the STRAW service 268 enters the definition into a publication definition table and then sends a subscription definition for the session channel for each subscription entry in the local list that has a matching channel ID in the publication record. When the subscription definition arrives, the STRAW service 268 begins sending definition updates (piped from the published application) on a given channel as STRAW records containing SODA records for that definition. These records may be sent on more than one channel.

When a kernel manager wants to participate in a channel with a server, it defines a subscription request, regardless of whether there are any STRAW streams destined for any server. If the virtual area application publishes later (i.e., after the flow is established), the change in the local table triggers a resend of the published entry in the table, which automatically triggers any potential subscriptions at the other end of the link. The STRAW service 268 automatically sends a subscription request if the kernel manager later subscribes to and publishes an entry in the table. This process ensures that the channel data is only transmitted on the link when needed by the receiver.

(vi) Channel record assignment

The STRAW record is decrypted when it arrives. If valid, its embedded record is decompressed and then dispatched to all subscribing kernel managers. The list of locally subscribed items is checked and all items matching the channel ID (in the subscription transport information) receive a copy of the record in its message queue.

The subscription kernel manager is responsible for releasing messages as they are processed. The large block data portion of the message is not copied but is directed to the original network buffer containing the STRAW record. Each kernel manager releases messages so that network buffers can be reused when messages are released.

(vii) Establishing STRAW flows

The client network node is fluidly connected to the session with the server and the peer network node. In this process, each party authenticates itself to the other party.

The STRAW stream is trusted and secure. This means that:

the client network node is certain of the identity of the partner;

the message is private;

receiving a message that the message can prove was sent (not modified somewhere in the middle); and

messages will be interpreted by both parties in the same way.

Part of the session definition is a list of streaming plug-in GUIDs. If a client network node responding to the definition supports at least one of these GUIDs, it loads a plug-in and uses the plug-in to establish a session. The server that creates the definition may examine the support list of each involved client network node and decide which transport plug-in GUID to include in the definition.

Part of the session definition is a list of stream encryption plug-ins GUIDs. If a client network node responsive to the definition supports at least one of the GUIDs, it loads a plug-in and encrypts the session using the plug-in. The server that creates the definition may examine the support list of each involved client network node and decide which stream encryption plug-in GUID to include in the definition.

(viii) Server streaming

In some embodiments, the flow from the client network node 344 to the server 346 is established using an address obtained from a server (such as a directory server, a mapping server, or an area server). Exemplary purposes of the stream include obtaining presence information, rendering a common space using a rendering definition from a mapping server, and rendering a virtual area using a rendering definition from an area server.

Fig. 21 illustrates an embodiment of a method of establishing a server flow between a client network node 344 and a server 346. According to the method, the client network node 344 sends the client credentials and the stream ID to the server 346 (fig. 21, block 348). The server 346 replies with the server credential and the pre-master secret (fig. 21, block 350). Once the flow is created, the connected client network node 344 agrees on the set of passwords and then authenticates by presenting its identification token (fig. 21, block 352). The server 346 presents a server token (selected by the stream ID and passed to the server 346 by the account server) appropriate for that client network node 344 (fig. 21, block 354).

(ix) Client streaming

Referring to FIG. 22, each session is identified by a new GUID generated by the originating server. The network nodes involved in the stream are notified of the session definition, and each network node communicates with the other node using the hash of the session GUID and the client ID as a stream encryption key. In the exemplary embodiment shown in fig. 22, the zone server 356 defines a session between two client network nodes 358, 360. Each of the client network nodes 358, 360 authenticates to the zone server and uses the encrypted channel to communicate definitions (including session definitions). The client network nodes 358, 360 need not share any further authentication information with each other. Each of the client network nodes 358, 360 is identified by the server 346 with a respective GUID. Each session definition identifies both client network nodes 358, 360 with its GUID. The client network nodes 358, 360 may use this information to decide which channel to publish on the session.

If a flow or session fails for one of the client network nodes 358, 360, the client network node uses the session failure SODA definition to notify the regional server 356 of the failure. Reasons for failure include, for example, no compatible transmission, no available channels, and reliable channel failure. In some embodiments, the regional server 356 responds to the session failure SODA definition by attempting to reroute the stream (e.g., by reflecting the audio stream off of a proxy or server).

In some embodiments, the client network nodes 358, 360 communicate P2P according to the simple traversal over UDP (STUN for short) network protocol to Network Address Translator (NAT). In these embodiments, the clients 358, 360 operate by responding to NATs. A server (e.g., regional server 356) acts as a STUN server that listens on the public side of the NAT for two IP addresses in the network and reports a mapped IP address and port outside the NAT. From this information, the client network node 358, 360 can discover the existence and specific type of NAT and obtain the mapped (external) IP address (NAT address) and port number that the NAT has assigned for the client to remote host UDP connection. The client network nodes 358, 360 then communicate with another P2P using the external IP address according to the UDP protocol. Additional details regarding the STUN Protocol may be obtained from "STUN-simple translation of User Datagram Protocol (UDP) Through Network Address Translators (NATs)" (STUN-simple traversal of User Datagram Protocol (UDP) Through Network Address Translators (NATs), internet proposal standard RFC 3489 (3 months 2003)) by Jonathan Rosenberg et al.

(x) Remain effective

In some embodiments, once a flow is established, a transmission on the client network node issues a periodic idle flow keep alive definition. The partner network node returns a flow with a timeout set to the maximum interval it can tolerate remains well defined. The purpose of this message is to keep any NAT firewall holes active. The partner network node records the desired timeout and extends the interval each time. If the next message is from a different IP and port, the NAT times out and a new hole is created by remaining active. The interval should then be shortened.

If the client network node or partner network node notices that the flow remains valid as lost due to expiration of the idle timer and no message received, or no reply to the keep-alive message received, it issues an immediate flow keep-alive with a very small timeout. This is to distinguish between a lost station and a lost UDP packet. Several retries may be attempted. If no reply is received after the retry, a local flow failure event is generated with the failed session ID and the flow definition is deleted.

In some embodiments, the STRAW service 268 responds to the broken link by automatically reestablishing the link based on the local published and subscribed entries and re-linking all subscriptions and data flows without application (or client) intervention.

f. Media channel service

In some embodiments, media channel service 271 is WindowsThe DLL is served.

Media channel service 271 is used to robustly deliver P2P audio and text chats. Media channel service 271 will compress the stream if configured to do so. The compressor library completes compression as required. The media channel service uses an audio codec plug-in configured by a variable. The variable GUID is taken from the stream definition.

SODA channel service

In some embodiments, SODA channel service 270 is WindowsThe DLL is served.

SODA channel services 270 are used to robustly communicate SODA definitions. The SODA channel service 270 will compress the stream if configured to do so. The compressor library completes compression as required. This is where the data structure is converted into network endianness. In some embodiments, little-end (intel) network endianness is used.

h. Connection and server hybrid manager

In some embodiments, the connection and server mixing manager 262 is WindowsThe DLL is served.

In some embodiments, the connection and server mixing manager 262 outputs a program API available to the local meeting scheduling tool to initiate a session with the zone server. This API is the initial item for creating a session. The connection and server hybrid manager 262 publishes the active session definition. The regional server receives the session definition on the SODA channel.

The connection and server mixing manager 262 also constructs an audio map from the audio map processing elements. The audio map processing element is configured by the regional server 26 either directly through the SODA record or indirectly through the VSDL script. In any case, the SODA definition is the result. In some embodiments, the connection and server hybrid manager 262 processes the following SODA definitions sent by the area service 26:

audio equipment

Audio streams

Audio mixing

Audio effects

Audio calculation

Audio recording

Audio playback

These SODA definitions are described in the following paragraphs.

An audio device is a definition of a local audio device to register as an audio source with the audio transport bus (which is a component of the transport bus 276). The well-known ID of the standard local audio source (microphone, headphones) or registered local audio recording (file, streaming CD audio) is provided, along with the instance ID (which headphone, if multiple headphones are available, it indicates). The device is given a new ID to use on the audio transport bus. The connection and server mixing manager 262 creates an instance of the appropriate variable for the audio device plug-in using the well-known audio source ID and hands it over to the audio transport bus.

An audio stream is a definition of an incoming audio stream that will register as an audio source to the audio transport bus. An incoming audio stream is defined by the channel ID that conveys the audio stream. For dynamic mixing purposes (by the area/zone manager 264), the stream must be associated with an avatar ID. The device uses the channel ID as its audio transport bus ID. The connection and server mixing manager 262 creates an instance of the appropriate variable of the audio stream plug-in based on the channel type ID and hands it over to the audio transport bus.

Audio mixing is a definition of a combined audio source and audio sink plug-in. This definition fully specifies the plug-in APIID, the variable ID, one or two audio transmission bus source IDs, and the associated avatar ID (for the zone/zone manager 264). The connection and server mixing manager 262 creates the indicated variable of the audio mixing plug-in based on the provided ID and hands it over to the audio transport bus.

The audio effect is a definition of a combined audio source and audio sink plug-in. The definition fully specifies the plug-in APIID, the variable ID, one audio transmission bus source ID, and the associated avatar ID (for the zone/zone manager 264). The connection and server mixing manager 262 creates the indicated variable of the audio effect plug-in based on the provided ID and hands it over to the audio transport bus.

Audio computing is a definition of an audio computing plug-in. The definition specifies comprehensively a plug-in API ID, a variable ID, an associated audio transport bus source object ID, the component's own audio transport bus ID, and two case-specific parameters. The audio computational object does not directly process audio data in the audio chain. Instead, the audio calculation object is based on a "domain object model", external information such as manual settings (mute, volume control in HUD), avatar position and motion, reverberation space, and Windows Settings (e.g., speaker selection in the control panel) to calculate settings for other audio map components. The audio computational objects are executed based on different presentation timer events (typically less than normal audio presentation). Because the data they use as input to the computation changes slowly. The connection and server mixing manager 262 creates the indicated variable of the audio computing plug-in based on the provided ID and hands it over to the audio transport bus.

Audio recordings are a definition of audio sink plug-ins. This definition links points in the audio map to the storage component. At presentation time, the audio recording component does not trigger the presentation itself. But if the rendering is triggered by another audio sink component, the rendered audio data is provided to the audio recording object for delivery to the indicated storage component. The connection and server mixing manager 262 creates the audio sink plug-in and hands it over to the audio transport bus.

Audio playback is a definition of an audio source plug-in. This definition links points in the audio map to the storage component. If the audio chain references the component, audio data for one time slice is fetched from the storage component at the frame preparation time and provided as output by the component. The connection and server mix manager 262 creates the audio source plug-in and hands it over to the audio transport bus.

The connection and server mix manager 262 configures the transport bus 276 and the audio streaming service 272 according to the definitions received from the zone servers.Each definition results in the creation of an audio processing primitive that is an audio stream plug-in, an audio computation plug-in, or an audio source plug-in. Local audio devices (e.g., microphone, speaker, and Skype audio) may be configured according to settings selected by the HID configuration tool. The HID configuration tool allows a user to select keyboard and mouse options for navigating in the 3D collaboration space. For example, a shortcut key may be defined and mouse gestures are limited to avatar behavior. In some embodiments, the audio output selection follows WindowsThe "sound and audio equipment" settings for audio and speech in the control panel. This ensures that the audio settings for the virtual area communications are the same as those for a conventional VOIP session.

If the regional server session transmission fails, the connection and server hybrid manager 262 attempts recovery. It drops the session and restarts it on a different regional server. If the media stream fails, the connection and server mixing manager 262 attempts recovery. In this process, the connection and server hybrid manager 262 attempts to reconnect to the client network node. If the reconnection attempt fails, the connection and server hybrid manager 262 defines the correspondent status as inaudible to the zone server.

i. Area/zone manager

In some embodiments, the area/zone manager 264 is WindowsThe DLL is served.

The area/zone manager 264 adjusts the audio streaming service 272 mixing parameters according to the zone definition and avatar position definition. The area service 26 publishes to the area/zone manager 264 a SODA definition that associates each avatar with audio processing primitives that respond to the avatar's motion. The avatar position data is used to mix audio streams from each client network node participating in the virtual area according to the local zone definition in a manner that allows each communicant to hear the other communicants at appropriate audio locations and at appropriate volumes. The parameter values applied to the audio processing primitives typically depend on calculations including relative positions, orientation of the communicant, zone definitions, audio attributes of the virtual areas, and manual settings (e.g., mute, volume) configured by the communicant.

In some embodiments, the area/zone manager 264 processes the following SODA definitions that pertain to the overall characteristics of the simulated audio space in which the current audio map is presented.

Audio reverberation

Audio aperture

Audio disturbance

These SODA definitions are described in the following paragraphs.

Audio reverberation is the definition of a "hollow space" of reverberation space that results in some reverberation or echo effect. The definition identifies simple geometric shapes with location. When an audio computing object is invoked, the definition is provided to all audio computing objects in a Domain Object Model (DOM).

The audio aperture is the definition of the connection between the two reverberation spaces. It identifies the two reverberant spaces by ID and specifies the audio connection between the two spaces. The connection is circular in one position and orientation. When an audio computing object is invoked, the definition is provided to all audio computing objects in a Domain Object Model (DOM).

An audio barrier is a definition of a physical barrier to sound propagation. It is modeled as a sphere at a location. When an audio computing object is invoked, the definition is provided to all audio computing objects in a Domain Object Model (DOM).

The SODA definition described above is an input to an audio computing object, which is a script computing plug-in that employs the following parameters as arguments:

physical properties of both the sound source and the sound trap;

zone definitions for source and sink;

manual settings (mute alone, volume/AGC);

environment settings (global silence, volume); and

Room audio characteristics.

The initial audio computing plug-in includes:

manual muting;

manual volume;

a location;

doppler shift;

orientation (facing/away);

a zone; and

room reverberation.

Some calculations are appropriate for individual audio sources; while some calculations are appropriate for the final mix of the entire room. The virtual area application can optionally introduce new plug-ins by referencing them in the audio definition. The area/zone manager 264 will subscribe to plug-ins it does not have and receive its definition from the area server.

Fig. 23 illustrates an example of a four-correspondent audio processing diagram 380 that may be specified by a zone server application. For simplicity, certain audio processing primitives that are typically present in a complete audio processing diagram (e.g., codec, network, filtering, special effects, and error concealment primitives) have been omitted from this example.

Arrows 382, 384, 386, 388, 390 represent audio sources, which are all mono dry audio sources. Avatars 1, 2, and 3 are network streams from remote client network nodes. Whisper is an optional local audio feed from a specified source. Everything to the left of the audio panning control (panner) is mono with a series of additional effects. These effects include adjusting the volume according to zone and speaker orientation and applying doppler shifts to account for the relative velocities of the talker and listener. The audio panning control places each adjusted mono signal within three hundred sixty degrees of audio space of the current occupancy zone of the virtual area. The speaker's position relative to the listener is used. Everything to the right of the audio lip control is 5.1 audio. The room audio processing primitives calculate room acoustic effects on the audio signal. It takes into account the location of the speaker and listener, room characteristics, and obstacles. The final mixing audio processing primitive adds all the processed audio signals together to produce a resulting stream that is piped to the designated audio output device (i.e., SPKR, which in the illustrated example represents a local speaker).

Some audio processing primitives (insertions) have fixed parameters and, therefore, are not associated with any runtime computing plug-in script. These components include echo and noise cancellation, Automatic Gain Control (AGC), silence detection, fixed source sound image control, and final mixing.

j. Asset caching service

In some embodiments, asset caching service 275 is WindowsThe DLL is served.

Assets are recorded in a local database or table service indexed by GUID. The asset data is retained in the amorphous storage pool. The ID for a particular asset never changes, thereby avoiding any cache coherency issues. Assets are cached by the SODA record. This means that large assets can be stored as many pieces, since the SODA records are limited to the size of the UDP MTU (about 1000 bytes). The asset cache must index the records by GUID and data offset, which is a field in the SODA record.

(i) Asset indexing

Assets are represented in class tables, and optionally attribute tables.

The asset class table maps asset GUIDs to class GUIDs and data storage references.

Asset ID	Assets class	Data storage
			guid surface	guid-like texture	<Image 1>
guid meeting table surface	guid-like texture	<Image 2>
			guid Sococo conference surface	guid-like texture	<Image 3>
guidDog display grid	guid-like mesh	<Image 4>
			guid name Soeoco conference name	guid-like text	Meeting "
guid author DVW	guid-like text	“David Van Wie”

The asset attribute table appends tagged attribute scalar values to the asset.

(ii) Data storage

The asset storage interface of the virtual area based presentation platform allows for stacking data, transactionally storing separate indexes, and fetching unused asset storage for reuse. The combination of database and file will be used for asset storage. The database contains two tables: an allocation table is stored, along with a GUID/offset index. Files are created in a fixed size according to the configured cache size. Assets are stored in files using a heap-bucket algorithm.

The asset cache data store will be indexed by GUID and offset. The heap is divided into memory segments by data size in powers of 2. The smallest memory segment is 32 bytes; the largest memory segment is 2 kbytes, which constitutes 7 memory segments in total. Each storage segment is budgeted for the same amount of storage, which means that half of the data items are present in each successive storage segment. Each storage segment is a heap table that is large enough to hash enough data items to complete a storage budget that makes the chance of hash collisions quite small.

(iii) SODA asset definition record

The assets are packaged in the SODA record for transmission. The definition includes the asset GUID, its contents (unstructured data) and its offset (if more than one record), and the attribute table. The SODA record of the packaged asset never contains any reference to the storage system.

The offset and data fields may repeat as long as they fit into one record.

The query for the asset is a definition.

Using definitions to ignore assets

(iv) Attribute ranking

Assets have attributes, the most important of which are type and default. The type specifies the use of the asset. A default designation may be used to replace the underlying asset of a given asset. Exemplary attribute ratings are shown in the following table:

name (R)	Type (B)	By default	Value of
				Blank space	Texture	Air conditioner	<Image 5>
Wood particle size 7	Texture	Blank space	<Image 6>
				Watch top 1	Texture	Wood particle size 7	<Image 7>
Conference watch wood	Texture	Watch top 1	<Image 8>
				Sococo watch	Texture	Conference watch wood	<Image 9>

In this example, a virtual area scene that includes the Sococo table texture would be rendered using the first available texture, searching from the Sococo table to the meeting table wood, then table top 1, and so on.

The style base will have a large asset level tree based on default assets, all of which are ultimately based on a smaller number of base assets. These base assets are installed as part of the client software package. This feature is intended to allow the design of a hierarchy in which assets with a specified style are called out, and to allow presentation of all style assets before they are actually designed. It may also be desirable to initiate the rendering of the virtual area before loading all of the styling assets.

Optional attributes include default, author, application, and Collada ID as a reference to the Collada source from which the asset was derived. A browsing tool running on the author's station will index the assets by any and all attributes.

k. Audio transmission bus

The audio transport bus is a component of the transport bus 276 that handles audio streaming. In some embodiments, the audio transport bus is implemented by a library that manages audio maps as a set of component objects. All audio map objects are registered with the audio transport bus by a unique ID. The audio transport bus is responsible for managing audio map objects when rendering audio. The audio transport bus tracks the audio map component by ID. In this process, the audio transport bus calls each audio map component in turn, providing audio data from the input component named by the ID.

The audio transport bus buffers one time interval of each audio stream available on the client network node. The audio transport bus feeds these streams to zero or more subscribers configured by the audio streaming service 272. Streaming data uses a pull model in which the final output stage invokes the previous stage to obtain the required data. Each stage calls one previous stage until the original audio stream source is reached. If the source needs to control the rate (flow control), it is usually buffered on its own and has its own specific signaling scheme. For example, the local file source may double buffer and read forward one time interval while processing the previous one. The network file source may signal the flow rate and buffer limitations to the server over the network. On the other hand, the local microphone source has no ability to control the flow rate at all.

The audio transport bus operates in two phases: upon occurrence of a presentation timer event, it provides existing presentation data to the audio sink component; the audio transport bus then traverses the audio map causing the audio data for the next time slice to be rendered and buffered. This technique gives the audio map a good opportunity to provide continuous playback even in the presence of variable latency audio source data.

In some embodiments, the audio transport bus measures the rendering latency of each audio graph component and aggregates each rendering chain latency by aggregating all relevant (source) audio component latencies. The audio transport bus collects and registers presentation latency statistics. Based on these statistics, the real-time scheduler 278 determines when and how the audio map should be modified to achieve the audio map processing goal. In some embodiments, the real-time scheduler 278 performs one or more of the methods described above in connection with fig. 13 and 14 in determining when and how the audio map should be modified to achieve the audio map processing goals.

Another function of the audio transport bus is to periodically invoke audio computational objects. The audio computational objects are used to change the settings of some of the associated audio map processing elements. The period of audio computation execution is typically much longer (less frequent) than the audio map rendering period.

The audio transport bus typically has the capability to record streams and play back recorded streams. The original audio stream is typically recorded so that during playback the mix can be re-rendered depending on the viewer's angle. Some embodiments include a hub that receives all of the raw audio streams. In these embodiments, the hub typically handles session recording. The audio transport bus typically only records audio streams on the client network node when a re-rendering of the session is not desired.

An audio source object is the basis for all audio sources. The object conveys data when polled and defines its expected latency and channel (e.g., mono, stereo, 5.1). The export objects include the output side of the microphone, media streams, clips, waveform files, DirectX audio, and mixing plug-ins.

The audio sink object is a basic object of the audio output device. The object requests data from an audio source when polled. The derived objects include the input side of the mixing plug-in, the media stream, and the speakers.

(i) Audio plug-in API

In some embodiments, the audio plug-in incorporates a VST audio effect C + + object available from Steinberg media technology GmbH. In particular, the audio plug-in incorporates a VST object packaged as a plug-in. A shim library (shim library) is provided for wrapping VST objects as audio plug-ins. The wrapper provides an audio plug-in API. The API of the VST object will be used as an audio plug-in class specific API. The API includes:

setting bus arrangement (input, n-input, output, n-output)

Get bus arrangement (Direction, index, speaker &)

Can handle sample size (size)

Obtain latency samples ()

Set-up processing (treatment)

Setting process (status)

Treatment (data &)

Obtaining a Tail sample ()

In these embodiments, the VST plug is packaged as an audio source and an audio sink. For example, the audio source:: frame (data &, size) call will be implemented as a call to the previous audio source:: frame (data &, size &), followed by setup processing (processing) and processing (data &). The configuration (latency &, channel placement &) call is implemented according to the acquisition latency sample () and acquisition bus arrangement (output, i, channel placement &) for each supported channel. The presence of the wrapper means that VST source code is required to populate the audio bus with existing VST plug-ins.

(ii) OpenAL (open audio library)

Most of the audio processing graph mixing and effects elements are performed using the OpenAL cross-platform audio API available from www.openal.org. The OpenAL library is able to use the best features of the available sound cards to compute all the parameters listed above in the area/zone manager section to implement these features. In particular, OpenAL sources, listeners, and program buffers are created for each operation in the audio map from mute to final mix. Before each update, the buffer parameters are modified according to the compute plug-in.

In some embodiments, the stream processing component (plug-in) is implemented using a GIPS componentized audio library available from Global IP Solutions, Inc. The GIPS audio library directly supports the following audio plug-ins: codec, error cancellation, jitter control, echo and noise cancellation, AGC, and silence checking.

I. Local stream source

The local stream sources are microphones, local sound sources such as recorded waveform files and music, and sound pattern resources. Windows (R) WindowsAn operating system API is used to attach each of these sources and present them to the audio transport bus for distribution. Each source is "wrapped" into an audio source derived class. When a definition is received (see audio device SODA definition in section VI), a source object wrapper is created. Windows-based for microphone, clip, Skype, and CD soundUsing a direct sound API. RTP streaming is simply a wrapper of audio sources around a real-time protocol (RTP) service that is used to deliver streaming data (e.g., audio) over a UDP socket. Streaming audio over the internet supports web-based downloading and playing and also microsoft media server streaming.

In some embodiments, the real-time kernel 260 supports the origination and mixing of sessions of virtual-zone based communications with non-virtual-zone based communications (e.g., Skype and VOIP audio). In these embodiments, the real-time kernel intercepts and renders non-virtual zone based communications as a local audio source. The non-virtual area based communication session is initiated by one of the communicants on a client network node responsible for sharing the original audio of the non-virtual area based communication session with other network nodes. The hosting client network node also mixes audio associated with the virtual area communication environment for communicants communicating via the non-virtual area based communication application.

Fig. 24 illustrates an embodiment of a computer system 389 (labeled "system 2") that enables people to communicate with virtual area communicants via different communication applications (e.g., Skype and VOIP). In fig. 24, audio communication channels are established between four network nodes (i.e., system 1, system 2, system 3, and system 4) sharing a virtual area. System 1 represents a client network node that does not run a virtual area-based communication application; instead, the system 1 is configured to operate an alternate communication application (e.g., Skype). System 2 represents a correspondent network node that is running an embodiment of the realtime kernel 260 that includes a stream handler 391 that issues and mixes virtual zone-based sessions with system 1. In this process, the stream handler 391 uses an integrated component 393 that virtualizes the playback and audio capture streams of the system 1. Systems 3 and 4 represent two other client terminals that are running virtual area based communication applications. The components of the system shown in fig. 24 include:

stream handler 391: the audio signal processing device is composed of the following audio image processing elements:

the O integration component 393 virtualizes the alternate playback and capture streams. In this process, integrated component 393 sends microphone (Mic)1 to C-split 1 received from virtualization replacement playback, receives microphone 2, 3, and 4 hybrid from P-hybrid, and sends virtualization replacement capture for transmission to system 1.

O C split 1 receives microphone 1 from integrated component 393 and sends microphone 1 to both the P-route and I/O multiplexer/demultiplexer.

C-split 2 captures receive microphone 1 from system 2 and sends microphone 2 to both P-hybrid and I/O multiplexer/demultiplexer.

P routes microphone 1 from C split 1 and microphones 2 and 3 from I/O multiplexer/demultiplexer. P-routing also plays back the sending microphones 1, 3 and 4 to system 2 and sends the microphones 3 and 4 to P-hybrid.

P-hybrid receives microphone 2 from C-split 2 and microphones 3 and 4 from P-route, sends the mix of microphones 2, 3, 4 to stream handler 391 for transmission outside of virtualization.

C: audio capture by virtual area-based communication applications

P: audio playback by virtual zone-based communication applications

·C_A: audio capture by alternate audio application of system 1

·P_A: audio playback by alternate audio application of system 2

Virtual microphone: virtual microphone associated with alternate audio for system 2

Virtual speakers: virtual speakers associated with alternate audio for system 2

In operation of computer system 389, the I/O multiplexer/demultiplexer sends audio signals 1 and 2 received from systems 1 and 2 to both systems 3 and 4. The I/O multiplexer/demultiplexer also sends the audio signals 3 and 4 received from the systems 3 and 4 to the P-routing component of the stream handler 391. The P-routing component sends audio signals 1, 3, and 4 to the playback component of system 2 and passes audio signals 3 and 4 to the P-mixing component of system 2. The P-mix component of the stream processor 391 mixes the audio signals 2, 3, 4 and passes the mixed signals to the integrated components of the system 2. The integration component 393 passes the mixed signal to an audio capture component of an alternate communication application (e.g., Skype) that is running on system 2 and corresponds to the communication application 395 (e.g., Skype) used by system 1. Alternative audio capture system (C) _A) The captured mixed signal 2+3+4 is passed to the playback component of the replacement communication application 395 running on the system 1.

In some embodiments of communications infrastructure 389, P-hybrid direct subscription I/O multiplexer/demultiplexer makes the system more symmetric. In these embodiments, P routes become P hybrid 1 and receive 3, 4 from I/O and 1 from C split 1. Because these signals are sent as independent channels, the output of C-split 1 may be sent directly to the playback component, but this is not flexible enough (because P-mixing may perform the actual mixing instead of the delivery of independent channels, see 3 below). In this case, P-hybrid becomes P-hybrid 2 and receives 3, 4 from I/O and 2 from C-split 2. The output of the mixer is a true mix because the alternative audio system is a mono communication system (even if the channels are stereo, there is no multi-track mixer on the other end that combines the signals from multiple sources).

Fig. 24 does not show the interaction of system 3 and system 4 with each other, but only with system 2 and, by extension, with system 1. The interaction between systems 3 and 4 may be peer-to-peer or server-mediated as described above.

In fig. 24, at any time, the two streams are comma separated (meaning it is a multi-channel route), the system can also send the mixed streams to save internal communication resources (e.g., outside the I/O multiplexer/demultiplexer). The streams to be mixed are indicated by a plus sign (i.e., by integration component 393 towards replacement capture component C_AThe transmitted virtualized microphone signal).

m. real-time scheduler

In some embodiments, the real-time scheduler service 278 is WindowsThe DLL is served.

In some embodiments, the rendering of audio and 3D scenes is done on a frame-by-frame basis. Initially, the flow is initiated, and then after a delay, the real-time scheduler service 278 begins processing the first frame. The delay is calibrated by the combined expected latency of each audio and video processing chain. The real-time scheduler service 278 initiates the consumption of the previous preparation frame and then processes the next frame on a time clock having a 50 millisecond period.

The final render objects in each chain are registered with the real-time scheduler service 278. These objects are derived from the SoFrameRenderer class, which is a method

Frame presentation (time frame Ms)

The method prepares a frame with the indicated time from a data source (audio or video) that is specific to the presentation chain. The SoFrameRenderer class includes another approach

Frame transfer ()

The method passes the previously prepared frame to the final destination, which is specific to the presentation chain. The SoFrameRenderer object does not require that more than one complete frame be able to be buffered. The real-time scheduler service 278 will frame pass the previously prepared frame on a schedule and then invoke frame rendering to prepare the frame for the next interval.

WindowsThe operating system does not have "hard-scheduling" capabilities. In some embodiments, the real-time scheduler service 278 is configured to invoke one or more SoFrameRenderer classes, which include audio processors, graphics processors, physical modeling, and scripts. The soframerender class enables the real-time scheduler service 278 to readjust frame processing in response to determining that the client network node cannot keep up with the target processing level. In some embodiments, the real-time scheduler 278 implements one or more of the methods described above in connection with fig. 13 and 14. In some embodiments, the real-time scheduler service 278 measures the presentation time of all frames and makes statistics available through the SODA interface. If the statistics fall outside of the range (e.g., taking too long to prepare a frame), a log event is generated. The heuristic will be triggered to attempt to "catch up", perhaps by skipping a frame, dropping out of range presentation chains (which are typically out of range due to hardware or network errors), or by dropping low priority presentation chains. For the purpose of implementing priority-based scheduling, the soframerender class defines the method:

Frame priority ()

The method returns a number, with smaller numbers being more important. The heuristic may determine from the latency and priority of the chains which chain(s) should be dropped to yield the greatest benefit with the least impact on overall priority.

In some embodiments, the real-time scheduler service 278 can additionally drop stages in the chain. To do so, the real-time scheduler service 278 can invoke the method:

waiting time frame deletion (priority)

Where the processing chain is responsible for "dropping" links below the indicated priority. The real-time scheduler service 278 may initiate a call at maximum priority and count down until all frames are present within the desired total latency. The combined heuristic of iteratively discarding low-priority chains and deleting the chains themselves typically terminates at a certain priority level. If the priority level is below a threshold, a log entry may be made and, in some cases, the session closed.

The real-time scheduler service 278 is also responsible for managing the interface to the SIS time service (network resources) to synchronize the client network node clocks with the clocks of other client network nodes in the network. In some embodiments, the time service is implemented by a plug-in, as described below.

7. Fault recovery

The real-time kernel 260 makes best efforts to continue operation in the event of network and server failures. In this process, the real-time kernel 260 implements a two-layer fault recovery algorithm. First, upon failure, SODA channel service 270 and media service 271 independently attempt to reestablish the connection. Media recovery will allow the conference to continue in the presence of a single audio channel failure or to recover in the event of NAT timeouts. Second, if the SODA channel service 270 and the media service 271 fail to re-establish a connection, they will signal the failure to their clients. In some embodiments, the act of multiple communicants attempting recovery simultaneously may be coordinated by registering the client node session with a regional vault server that can synchronize recovery attempts.

In some embodiments, the area/zone manager 264 also attempts recovery in the event of a server failure. The area/zone manager 264 takes over the stream failure and then drops the session and restarts the session on a different area server.

8. Plug-in component

In some embodiments, the live kernel 260 has a componentized, open, and platform-independent architecture that allows developers to independently develop and remotely add and update components of the live kernel 260.

a. Plug-in design

The plug-in is a platform-specific binary image that is downloaded from a plug-in server. Based on WindowsThe plug-in is implemented as a DLL (e.g.,. NET or COM style plug-in). However, the plug-in is loaded differently than the normal dynamic link library. In particular, during loading, no linking to the plug-in library is required; nor does it require any compilation of software code. Instead, the plug-in manager 266 simply loads the plug-ins contained in the plug-in directory. In this way, plug-ins can be added to or removed from the client network node without requiring any further configuration of the station (e.g., registry keys), simply by downloading or deleting executables. These plug-ins are connected to each plug-in host in a well-defined manner.

As shown in FIG. 25, the active instance of a plug-in 392 is derived from a corresponding plug-in variable class 394, which in turn is derived from a corresponding plug-in base class 396. In some embodiments, plugin instance 392 is instantiated by creating a base plugin object and casting the base plugin object to the identified variable by inheritance. Each base class 396 and variable class 394 represents that it provides a service by declaring an interface to be implemented.

FIG. 26 illustrates an exemplary set of plug-in base classes 398, 400, each associated with a respective set of one or more derived variable classes 402, 404 and 406, 408. Each plug-in base class 398, 400 defines a specific functionality class, while each plug-in variable class defines a specific use or class behavior (e.g., API, mouse) of the functionality of the corresponding base class. Plug-ins typically use header files to define an interface, a functional prototype of an interface function, and a macro for invoking the interface function.

The plug-in is identified by a GUID and a human readable name. A single plug-in may have multiple APIs supported and multiple variables on the same API. In some embodiments, the plug-in API is a C language data structure containing function pointers to plug-in functions that implement services defined by the interface.

The following table shows an example of a hypothetical physical plug-in identified as guid physical setting 1.

Filename	Internal names	Identifier	API	Variables of
					Physical DLL	Frictionless physical setting 1.0	guid physical settings 1	guid joining	Guid joint ball
Physical DLL	Frictionless physical setting 1.0	guid physical settings 1	guid joining	guid joint hinge
					Physical DLL	Frictionless physical setting 1.0	guid physical settings 1	guid joining	guid engagement cursor
Physical DLL	Frictionless physical setting 1.0	guid physical settings 1	guid joining	guid joint spring
					Physical DLL	Frictionless physical setting 1.0	guid physical settings 1	guid conflict	guid conflict octal
Physical DLL	Frictionless physical setting 1.0	guid physical settings 1	guid conflict	guid conflict dynamics

In this example, there are four variables of the "join" API and two variables of the "conflict detect" API. The file name is the name with which the plug-in is stored in the file system. The internal names are intended to describe the plug-ins as a whole. The identifier identifies the entire plug-in binary image and is used only by the plug-in manager 266. The API is a well-known GUID that specifies the programming API supported by the plug-in. Variables are well known GUIDs that a client application can use to select a plug-in for a particular purpose. For example, the SODA channel service 270 may agree on a compression plug-in variable for use on a session.

FIG. 27 shows a graphical representation of Microsoft WindowsAn exemplary architecture for COM (component object model) architecture compatible plug-in 410. The plug-in 410 includes a set of entry point functions 412, a first API class 414 including first and second variables 416, 418, and a second API class 420. The plug-in 410 implements all functions in all supported API classes 414, 420 and in all variables in each API class. Each API class and each variable in an API class is associated with a corresponding entry point function (also referred to as a "factory"). When the plug-in 410 is loaded, the corresponding entry point function is registered with the plug-in host for each supported API class. The plug-in host uses an entry point function associated with the API class to instantiate the API class and obtain a pointer to its Iunknown interface. The plug-in host uses the Iunknown interface to search for entry point functions for variables supported by the API class. The plug-in host uses an entry point function associated with a variable to instantiate the variable and obtain a pointer to its Iunknown interface. The plug-in host queries the variable for the interface function table of the interface using the Iunknown interface. The function table gives the host access to the API implementation of the variable.

Each plug-in executable supports a core plug-in API that allows the plug-in manager 266 to manage the plug-in. The core plug-in API makes the plug-in executable self-describing. In particular, each plug-in outputs a function that can be called by the plug-in manager 266 to allow the plug-in to register itself. In some embodiments, each plug-in executable outputs the following core plug-in APIs:

(i) plug-in variables

The active instance of a plug-in variable is derived from the class "plug-in variable". Plug-in variables are always created using plug-in objects. All useful plug-ins extend this class to include API-specific methods. The only common method is the common configuration method.

Plug-in variables: arrangement ()	Initiating a dialog
		Plug-in variables: configuration settings (parameters)	Setting configuration parameters
Plug-in variables: configuration acquisition (parameter)&)	Obtaining configuration parameters

(ii) Plug-in manager

FIG. 28 illustrates an embodiment of a plug-in architecture including a plug-in manager 266, a plug-in directory 422 containing a set of plug-in containers 424, a plug-in database 426, and a caller 428. The plug-in manager 266 queries the plug-in container 424 in the plug-in directory 422, registers plug-ins in the plug-in database 426, and responds to requests from the caller 428 to send information about available plug-ins and to instantiate specified plug-in variables. The caller 428 is typically a kernel component (e.g., an installation loader component, kernel managers such as the connection and server hybrid manager 262 and the area/zone manager 264, and kernel services); although in some embodiments the caller 428 may correspond to a software application or service running on a client node or a remote network node (e.g., a server).

FIG. 29 illustrates an embodiment of a method implemented by the plug-in manager 266 in registering available plug-ins on a client network node.

According to the method of FIG. 29, the plug-in manager 266 discovers the plug-ins available on the client network node (FIG. 29, block 430). In some embodiments, all plugins are stored in a shared plugin directory created in the file system of the client network node. In these embodiments, the plugin manager 266 discovers available plugins by examining the shared plugin directory. In other embodiments, the plug-in manager 266 may be configured to examine other file locations to find available plug-ins.

The plug-in manager 266 queries the discovered plug-ins to discover all the APIs they support (fig. 29, block 432). In this process, the plug-in manager 266 makes calls to the core API to enumerate parameter values associated with the available plug-ins. For example, the plug-in manager 266 queries the plug-ins with core API calls that return the plug-in GUID, GUIDs of the supported APIs, and GUIDs of the supported variables for a given API.

Based on the query results, the plug-in manager 266 stores the associations between the plug-ins and the APIs that the plug-ins respectively support in the plug-in database 426 (FIG. 29, block 434). In this process, the plug-in manager 266 sorts all plug-ins in the directory 422 through the APIs supported by the plug-ins. The plug-in manager 266 automatically enters plug-ins under all supported APIs in the plug-in database 426. In this way, plug-in database 426 allows variable queries to be made on plug-ins with the same API. Further, the plug-ins in plug-in database 426 may be enumerated by APIs and variables as well as instances created when referenced by the virtual area application. FIG. 30 illustrates an exemplary embodiment of creating a database 436.

The plug-in manager outputs the following APIs:

FIG. 31 illustrates an embodiment of a method implemented by the plug-in manager 266 in response to receiving an API call from the caller 428. According to the method, in response to receiving a call to enumerate all plugins that support a specified one of the APIs (FIG. 31, block 440), the plugin manager 266 returns a list that includes identifiers of all plugins in the plugin database that are associated with the specified API (FIG. 31, block 442). In response to receiving a call to enumerate variables of an identified one of the APIs supported by the identified one of the plug-ins (fig. 31, block 444), the plug-in manager 266 returns a list including identifiers of all variables of the given API supported by the identified plug-in (fig. 31, block 446). In response to receiving a call to the identified one variable instantiating the identified one API supported by the identified one plug-in (FIG. 31, block 448), the plug-in manager 266 loads the identified plug-in and returns a pointer to the instance of the identified variable (FIG. 31, block 450). In this process, a specified variable is located, loaded into the caller's address space, and the internal pointer table is populated with the function table addresses of the variable functions that implement the services defined by the API.

(iii) Plug-in server

The plug-ins exist on the server in a downloadable form appropriate for each supported platform. The server always supports a minimal set of encryption, compression, and authentication plug-ins, so the client network node always succeeds in attempting server connections.

When first registered on the server by a third party developer, the plug-in is checked by the automation utility to ensure that it conforms to the API description and to check for unacceptable client site API references. For example, dynamic binding to any native interface that is not already in the plug-in environment is not allowed. The card is then scanned for viruses. All images on the server are then virus-safe.

The plug-in performs access control by authenticating the user. In this way, a user who has paid for the access can use the plug-in at any location. In some embodiments, the user is authenticated for plug-in download via the e-commerce engine.

b. Plug-in components

All plugins conform to the plugin API. Some plug-ins are specific to certain tools (e.g., OpenGL) and have their own standards. Each plug-in class has a class-specific API. In some embodiments, the plug-in depends on the native application runtime that all client sites access. In each component that uses plug-ins, the protocol allows features that use GUID namespaces to be agreed upon. For example, in some instances, when connecting network streams, the provider client network node provides its list of encrypted plug-ins GUID in a preferred order, and the recipient network node selects among these offers and responds with a selection or a rejection.

In some exemplary embodiments, the following plug-in classes are defined and used by developers to develop plug-ins for use in creating virtual area communicator environments.

Encryption algorithm

Compression algorithm

Authentication algorithm

Certificate

Graphic effects

Physical extension

Script extension

Input device Main memory

Audio mixing

Audio source

Audio insertion

Flow transmission

Time service

These plug-in classes are defined in the following paragraphs.

Encryption algorithm

Encryption is a streaming feature and is agreed upon at stream creation time. The session definition will include an encryption variable ID. Exemplary encryption variables include AES and RC4 block encryption.

Compression algorithm

Compression is an optional channel content feature. The channel selects compression when the transmission agrees with the channel definition. The subscribing station provides available compression options, while the publishing station indicates its selection when it provides publication. Audio codecs choose to skip compression because their content is already compressed. Exemplary audio compression plug-in variables implement ITU-T v.44 compression processing for channel-specific prepare (primary) streams.

Authentication algorithm

The network infrastructure server requires some authentication protocol in order to connect. The third party server may have specific authentication requirements. The authentication protocol can be modified at any time. In addition to the core plug-in API, the authentication plug-in has an API for executing authentication protocols and gaining access to local credentials. Exemplary authentication plug-in variables include plug-in variables that support SSL type authentication of the initial server and subsequent server login using the signed ID as a token.

Voucher(s)

Credentials may be created and must be packaged in a plug-in to be stored and used by an authentication algorithm. An exemplary credential is one that contains a public key.

Graphic effects

OpenGL and Collada support scripted shaders, samplers, profiles, and annotations. Some embodiments support Collada cg _ surface _ type and glsi _ surface _ type as shaders, and gl _ sampler X and cg _ sampler X as samplers.

Physical extension

In some embodiments, an Open Dynamic Engine (ODE) is modified to include a specific hook (hook) in a dynamic behavior loop. Virtual area applications typically specify physical properties by applying physical attributes to entities in a scenegraph. The physical extension plug-in has an API for querying the associated property GUID processed by the plug-in. In this way, physical plug-ins are only invoked for the relevant objects and virtual areas. Some embodiments support Collada rigid _ constraints that are joints within the model, and OpenGL collision checking that is a global motion algorithm for avatars and artifacts.

Script extension

The script extension plug-in has additional APIs that allow the script runtime (e.g., Java or JavaScript) to be wrapped and provided with the native application runtime. The script is defined by a SODA record that includes the GUID of the script extension plug-in.

Input device main memory

Input device plug-ins generate input events for standard processing by application logic (e.g., by computer keyboard, computer mouse, XboxController, and WiiController generated events).

Audio mixing, audio source, and audio insertion

Audio processing is desirable at the beginning (e.g., microphone) and takes effect and mixes at the end (e.g., speaker). Audio plug-ins are typically unable to change audio network routing because this affects other usersAnd (6) experiencing. Windows (R) WindowsThe audio plug-in on the platform is a DLL that conforms to the audio plug-in API. They are registered with the real-time kernel 260 and are available for reference by the SODA. The virtual area developer can request an audio plug-in as part of the virtual area application (e.g., using extended Collada fields or VSDL semantics).

The avatar script may also request an audio plug-in.

Mixing variables include echo, echo cancellation, reverberation, and synthesis.

Source variables include microphone, clip, media, Skype, and streaming files.

The insertion variables include sound image control, volume, iLBC, RCU, iPCM-wb, equalizer, LPF, HPF, AGC, noise cancellation, error cancellation, jitter control, mute, delay, silence check, comfort noise.

Streaming

STRAW uses a streaming plugin to host a session on the network. Stream packets and provide reliability, authentication, and encryption.

Time service

The real-time scheduler 260 is interested in synchronizing the time between the client network nodes. To this end, each client network node will synchronize with an internet time standard, such as SIS, NTP (network time protocol), or ITS (internet time service).

c. Plug-in class API

All plug-in class objects are based on plug-in variables.

Encryption algorithm

The plug-in is based on a plug-in variable. It implements a block encryption algorithm that includes keying. The plug-in is stateless, i.e. each encryption is independent of the other, except for the key.

Encryption: Ctor (Algorithm, Key)

Encryption:encryption (data, size, object)

Encryption:: decryption (data, size, object)

Encryption Dtor ()

Compression algorithm

The plug-in is based on a plug-in variable. It performs one or more lossless compression/decompression algorithms. It compresses the continuous stream and may maintain an internal state table. The algorithm restarts at each key frame and may include processing "prepare" blocks that are not issued but contribute to the compression state.

Compression: Ctor (Algorithm)

Compression:: key frame (preload)

Compression: (data, size, object)

Compression:: decompression (data, size, object)

Compression: (Dtor)

And (3) authentication algorithm:

the plug-in is based on a plug-in variable. It accesses the local credentials and implements a client authentication state machine of one or more states. The authentication state machine does not control the communication link to the authentication server. It only processes the message and generates subsequent messages.

Authentication: Ctor (Algorithm, voucher)

Authentication initial State ()

Authentication status (State)

Authentication: advanced (message data, message consumer &)

Authentication:: session key (key &)

Authentication Dtor ()

Voucher(s)

The plug-in is based on a plug-in variable. It accesses a platform-specific credential store.

Voucher: Ctor ()

Voucher: selection (index)

Receipt (data, size, index &)

Certificate private key (secret key &)

Certificate (Key &)

Certificate (password phrase &)

Voucher: Dtor ()

Graphics effect-shader

The plug-in is based on a plug-in variable. It affects 3D rendering.

The shader API supports a shader (script surface generator).

Shader: Ctor (gl &)

Shader: (format, size, mip, type, code, technique)

Shader-setting texture parameters (texture)

Shader:: set texture stream (id stream)

Shader:: present (transfer)

Shader: Dtor ()

Pattern effect sampler

The sampler API is based on plug-in variables. It supports mapping textures to surfaces.

Sampler: Ctor (gl &)

Sample:initialization (type, pack, small filter, large filter, mip filter, color, mip map _ max level, mip map _ offset)

Sampler Dtor ()

Physical extension-joining

The plug-in is based on a plug-in variable. It extends the dynamics of the motion.

The coaptation API supports coaptation constraints and dynamics on the avatar and artifact subcomponents.

Joint: Ctor (unbounded n size, ode)

Initialization (body 1, Anchor 1, orientation 1, body 2, Anchor 2, orientation 2, axis, erp, cfm)

Joint torque

Obtaining status (fps &, erp &, jLin1&, jAng1&, jLin2&, jAng2&, c &, cfm &, low &, high &)

Physical extension-conflict

The conflict API is based on plug-in variables. It supports fast computational conflict detection. In some embodiments, the conflict plug-in variables may implement a quadtree-like dx space algorithm in an Open Dynamic Engine (ODE).

Conflict: (space, ode)

Conflict (step)

Script extension

The plug-in is based on a plug-in variable. It supports the execution of code provided by the virtual area application. The code is in binary form and is marked up by an API that is supported by the language and the particular code definition. The plug-in varies with language. It loads a specific code definition and calls the code definition to provide a native application runtime object that fits the API.

Script extension: Ctor (sonar &)

Script extension execution (code, api)

Script extension Dtor ()

Input device main memory

The plug-in is based on a plug-in variable. It supports user event capture (in addition to audio). This includes everything from the mouse to the video camera with face recognition. The plug-in generates user events to the queue.

Input device: Ctor (arg, queue)

Configuration of input device (arg)

Audio mixing

The plug-in is based on audio components. It combines the two audio streams.

Audio mixing Ctor (Audio scene 1, Audio scene 2, parameters)

Audio mixing configuration (parameters)

Audio mixing Dtor: ()

Audio source

The plug-in is based on audio components. It provides a pollable audio stream.

Audio source Ctor ()

Audio Source:: configuration (arg)

Audio source:: acquisition latency (latency &)

Audio source poll (data)

Audio insertion

The plug-in is based on audio components. It interprets the audio stream. It can compress, decompress, or simply modify the stream.

Audio insertion: Ctor (Audio source &)

Audio insertion: get latency (latency &)

Audio insertion procedure

Audio transmission

The plug-in is based on audio components. It splits the audio stream into two equal streams (replicating the stream). Any audio source may be routed to any other audio element.

Audio transmission: Ctor (Audio source &)

Audio transmission Dtor ()

Audio transmission:: copy (data &)

Streaming

The plug-in is based on a plug-in variable. An exemplary streaming plug-in variable supports IP UDP transport with configurable authentication and encryption.

Transmission: Ctor (Port, station, Session, encryption)

Transmission MTU ()

Transmission:Hello ()

Transmission: challenge (credential)

Response (challenge, encryption)

Transmission response (response, encryption)

Transmission of identification token ()

Transmission identification token (token)

Transmission: Transmit (data, size, id channel)

Transmission: reception (recording)

Transmission: Dtor ()

Time service

The plug-in is based on a plug-in variable. It synchronizes the client network node time with the internet standard.

Time service Ctor ()

Time service: (synchronization)

Time service Dtor: ()

SODA definition

Conversation

ID station 1 GUID

ID station 2 GUID

ID session GUID

Session failure

ID session GUID

The reason is long

Length of parameter

Station

ID GUID

STRAW _ ADDRESS IP, Port

STRAW _ TRANSMISSION [ GUID ]

Disclose (a)

ID client GUID

ID server GUID

ID channel GUID

ID compression [ GUID ]

Subscription

Flow maintenance is effective

ID session GUID

The overtime is long

Flow failure

ID session GUID

Channel failure

ID channel index GUID

Correspondent status

ID itself GUID

ID communicator GUID

Short state

HID

ID device class GUID

ID device GUID

ID GUID

Audio parameters

ID GUID

ID parameter GUID

Short value

Audio equipment

ID device class GUID

ID device GUID

ID GUID

Regional conversation

ID area GUID

ID GUID

Physical checkpoint

Ground zone

ID GUID

Shaped mesh

Starting point double X3

Avatar [ GUID ]

Audio aperture

ID GUID

Radius double

Starting point double X3

Oriented double X3

Audio frequency obstacle

ID GUID

Radius double

Starting point double X3

Audio reverberation

ID GUID

Shaped mesh

Starting point double X3

Audio mixing

Audio effects

Audio streaming

ID Audio GUID

Avatar GUID

Audio frequency calculation

Plug-in component

Type GUID

Name text

ID GUID

Supported API [ GUID ]

Escalating dependency lists

Foundation installation GUID

Description text

Upgrade ID [ GUID ]

Upgrading

ID GUID

Offset length

Data [ ]

Audio recording

ID GUID

ID Audio GUID

ID storage GUID

Audio playback

ID GUID

ID Audio GUID

ID storage GUID

Network DNS

Address string

Network proxy

Address string

Debug information

ID GUID

Statistical GUID

Reset boolean

Debugging traps

ID GUID

Trap GUID

Rearming boolean

HID

IDKvm GUID

HID events

VII. conclusion

Embodiments described herein provide a real-time kernel that supports real-time communications between communicants operating on respective network nodes. The real-time kernel handles the complex tasks of connecting to correspondents, virtual areas, and other network resources, switching these connections in response to user input, and mixing real-time data streams. The real-time kernel enables developers to focus on developing high-level communication functions rather than low-level nexus code. The real-time kernel imposes relatively low computational resource requirements so that real-time communication performance can be achieved using a wide range of currently available computing devices and network connections.

Other embodiments are within the scope of the following claims.

Claims (139)

1. A computer-implemented method for remotely controlled real-time data stream handling, comprising:

receiving, at a local network node (12), one or more stream handling instructions from a remote network node (16), wherein the one or more stream handling instructions comprise a description of a stream handler (22) for processing at least one real-time data stream;

creating, at the local network node (16), a stream handler (22) from the description; and

generating a resultant data stream at the local network node (16), wherein the generating comprises processing the real-time data stream by the created stream handler (22).

2. The method of claim 1, wherein the generating comprises

Determining a configuration parameter value from real-time status information specified in the one or more stream handling instructions, an

Dynamically configuring the stream handler (22) with the configuration parameter values.

3. The method of claim 1, wherein the creating comprises establishing the stream handler (22) with a mixing function specified in the one or more stream handling instructions, and the generating comprises mixing the real-time data stream with at least one other real-time data stream in accordance with the mixing function to generate a mixed real-time data stream.

4. The method of claim 1, wherein:

the creating comprises instantiating the processing objects specified in the one or more stream handling instructions and assembling the instantiated processing objects into directed graph components of the stream handler (22) according to the description; and

the processing includes processing the real-time data stream through the directed graph.

5. The method of claim 4, wherein the generating comprises

Determining from real-time state information specified in the one or more stream handling instructions, a configuration parameter value for the instantiated processing object, an

Configuring the processing object with the configuration parameter values.

6. The method of claim 4, wherein the instantiating comprises instantiating a mix object specified in the one or more stream handling instructions, and the generating comprises executing the mix object to generate a mixed real-time data stream from a combination of the real-time data stream and at least one other real-time data stream.

7. The method of claim 4, wherein at least one instantiated processing object encapsulates a corresponding call into a driver module, the driver module controlling a hardware component of the local network node (16) based at least in part on the resulting data flow.

8. The method of claim 4, wherein the one or more stream handling instructions specify the processing object with respective unique identifiers and the instantiating comprises issuing a call to a processing object application program interface program (API) that includes respective ones of the identifiers.

9. The method of claim 1, wherein the one or more stream handling instructions comprise a second description of a second stream handler (22) for processing at least one real-time data stream having a data type different from the first real-time data stream, the creating comprises creating the second stream handler (22) from the second description, and the generating comprises processing the real-time data stream by the second stream handler (22).

10. The method of claim 1, wherein the local network node (16) is associated with a first object in a virtual area (28), and the method further comprises receiving a real-time data stream from a second remote network node associated with a second object in the virtual area (28).

11. The method of claim 10, wherein the one or more stream handling instructions include real-time location information describing respective locations of the first and second objects in the virtual area (28), and the generating includes determining configuration parameter values from the real-time location information and configuring the stream handler (22) with the configuration parameter values.

12. The method of claim 11, further comprising receiving a second real-time data stream from a third remote network node associated with a third object in the virtual area (28), wherein the creating comprises establishing the stream handler (22) with a mixing function specified in the one or more stream handling instructions, and the generating comprises combining the first real-time data stream and the second real-time data stream according to the mixing function.

13. The method of claim 11,

the one or more stream handling instructions comprise a second description of a second stream handler (22) for handling at least one real-time data stream having a data type different from the first real-time data stream,

the method also includes receiving a second real-time data stream from a third remote network node.

Wherein the creating comprises creating a second stream handler (22) according to the second description, and the generating comprises configuring the second stream handler (22) according to one or more of the configuration parameter values and processing the first and second real-time data streams by the second stream handler (22).

14. The method of claim 10, wherein:

the one or more stream handling instructions comprise real-time location information describing respective locations of the first and second objects in the virtual area (28);

the creating comprises establishing the stream handler (22) with one or more object property handling functions specified in the one or more stream handling instructions;

the generating comprises determining object configuration parameter values from the real-time location information, and dynamically configuring the one or more object property processing functions with the object configuration parameter values while generating the resulting data stream; and

the generating includes presenting a visual output including visual representations of the first and second objects within the virtual area (28) based at least in part on the resulting data stream.

15. The method of claim 14, wherein at least one of the one or more object property processing functions is configured to dynamically control the visual representation of the first object over at least one of an orientation, a position, a movement, and a pose based on a position of the first object in the virtual area (28).

16. The method of claim 1, wherein the specified data type is audio and the generating comprises generating an audible output.

17. The method of claim 16, wherein the creating comprises establishing the stream handler (22) with one or more audio processing functions specified in the one or more stream handling instructions.

18. The method of claim 17, wherein the network node is associated with a first object in a virtual area (28), the method further comprising receiving the real-time data stream from a second network node associated with a second object in the virtual area (28), and wherein the one or more stream handling instructions comprise real-time location information describing respective locations of the first and second objects in the virtual area (28), and the generating comprises determining an audio processing configuration parameter value from the real-time location information and configuring the stream handler (22) with the audio processing configuration parameter value.

19. The method of claim 16, wherein the creating comprises establishing the stream handler (22) with a mixing function specified in the one or more stream handling instructions, and the generating comprises combining the real-time data stream with at least one other real-time audio data stream according to the mixing function.

20. The method of claim 1, further comprising, at the local network node (16):

storing a remote publishing definition received from the remote network node, wherein the definition describes a real-time data stream available from one or more remote network nodes;

storing a local subscription definition of a real-time data stream requested by a local real-time kernel component of the local network node (16); and

sending a respective live data stream request to one or more of the remote network nodes for each of the local subscription definitions having a matching remote publication definition.

21. The method of claim 20, further comprising directing, at the local network node (16), real-time data streams received from the one or more remote network nodes in response to the request to the respective local real-time kernel component according to the local subscription definition.

22. The method of claim 1, further comprising generating a human perceptible output at the local network node (16) based at least in part on the resulting data flow.

23. An apparatus, comprising:

a computer readable medium storing computer readable instructions; and

A data processing unit coupled to the memory to execute the instructions and to perform operations based at least in part on the execution of the instructions, the operations comprising:

receiving, at a local network node (16), one or more stream handling instructions from a remote network node, wherein the one or more stream handling instructions comprise a description of a stream handler (22) for processing at least one real-time data stream;

creating a stream handler (22) at the local network node (16) according to the description, and

generating a resultant data stream at the local network node (16), wherein the generating comprises processing the real-time data stream by the created stream handler (22).

24. At least one computer readable medium having computer readable program code embodied therein, the computer readable program code adapted to be executed by a computer to implement a method comprising:

receiving, at a local network node (16), one or more stream handling instructions from a remote network node, wherein the one or more stream handling instructions comprise a description of a stream handler (22) for processing at least one real-time data stream;

creating, at the local network node (16), a stream handler (22) from the description; and

Generating a resultant data stream at the local network node (16), wherein the generating comprises processing the real-time data stream by the created stream handler (22).

25. A computer-implemented method for real-time data stream handling, comprising:

parsing a description of a real-time stream handler (206) from one or more stream handling instructions (210), wherein the parsing includes parsing an input source identifier, an output sink identifier, and a respective identifier for each of one or more data processing objects from the one or more stream handling instructions (210);

instantiating real-time stream handling objects corresponding to respective ones of the identifiers;

creating a directed graph (212) comprising some instantiated real-time stream handling objects from the description;

receiving a real-time data stream from an input source corresponding to the input source identifier; and

generating a resulting data stream at an output sink corresponding to the output sink identifier, wherein the generating comprises processing the real-time data stream through the directed graph (212).

26. The method of claim 25, wherein the parsing includes extracting identifiers of data processing objects from the one or more stream handling instructions (210), and the creating includes assembling the directed graph (212) from instantiations of real-time stream handling objects corresponding to the extracted identifiers.

27. The method of claim 25, wherein the parsing comprises extracting identifiers of audio processing objects from the one or more stream handling instructions (210), and the processing comprises processing the real-time data stream by one or more instantiated audio processing objects corresponding to respective ones of the extracted identifiers.

28. The method of claim 25, wherein the instantiating comprises querying an object library for the respective identifier, retrieving an object from the object library in response to the query, and instantiating the retrieved object.

29. The method of claim 28, wherein the instantiating comprises issuing a call to a process object Application Program Interface (API) program that includes respective ones of the identifiers.

30. The method of claim 25, wherein the generating comprises

Determining a configuration parameter value from real-time status information specified in the one or more stream handling instructions (210), an

Dynamically configuring one or more of the instantiated real-time flow handling objects in the directed graph (212) with the configuration parameter values.

31. The method of claim 25, wherein:

The receiving comprises receiving a second real-time data stream from a second input source corresponding to a second input source identifier in the description; and

the generating comprises processing the second real-time data stream through the directed graph (212).

32. The method of claim 31, wherein the instantiating comprises instantiating a real-time stream mix object (220) corresponding to a respective one of the parsed identifiers, the creating comprises merging the instantiated real-time stream mix object into the directed graph (212), and the generating comprises executing the instantiated real-time stream mix object (220) to combine the first and second real-time data streams.

33. The method of claim 32, wherein the first input source corresponds to a definition of an incoming real-time data stream (214) from a first remote network node and the second input source corresponds to a definition of an incoming real-time data stream (214) from a second remote network node.

34. The method of claim 25, wherein:

the parsing includes parsing a second description of a second real-time stream handler (22) from the one or more stream handling instructions (210), the second description including a second input source identifier, a second output sink identifier, and a respective identifier for each of one or more data processing objects;

The creating comprises creating a second directed graph from some instantiated real-time stream handling objects according to the second description;

the receiving comprises receiving a second real-time data stream from a second input source corresponding to the second input source identifier, wherein the first and second real-time data streams differ in data type; and

the generating includes processing the second real-time data stream through the second directed graph to a second output sink corresponding to the second output sink identifier.

35. The method of claim 34, wherein the first real-time data stream and the second real-time data stream are processed concurrently.

36. The method of claim 25, wherein at least one of the instantiated real-time flow handle objects encapsulates a corresponding call to a driver module that controls a local hardware component.

37. The method of claim 25, wherein the parsing, the instantiating, the creating, the receiving, the producing, and the generating are performed on a local network node (16) associated with a first object in a virtual area (28), and the receiving comprises receiving the real-time data stream from a remote network node associated with a second object in the virtual area (28).

38. The method of claim 37, wherein the one or more stream handling instructions (210) include real-time location information describing respective locations of the first and second objects in the virtual area (28), and the generating includes determining configuration parameter values from the real-time location information and dynamically configuring one or more of the instantiated real-time stream handling objects in the directed graph with the configuration parameter values.

39. The method of claim 38, wherein the receiving comprises receiving a second real-time data stream from a second remote network node associated with a third object in the virtual area (28), the creating comprises building the directed graph with instantiated hybrid objects corresponding to respective ones of the parsed identifiers, and the generating comprises executing the instantiated hybrid objects (220) to combine the first real-time data stream and the second real-time data stream.

40. The method of claim 38, wherein:

the parsing includes parsing a second description of a second real-time stream handler (22) from the one or more stream handling instructions (210), the second description including a second input source identifier, a second output sink identifier, and a respective identifier for each of one or more data processing objects;

The creating comprises creating a second directed graph from some instantiated real-time stream handling objects according to the second description;

the receiving comprises receiving a second real-time data stream from a second input source corresponding to the second input source identifier, wherein the second input source corresponds to a definition of an incoming real-time data stream from a second remote network node, and the first and second real-time data streams differ in data type; and

the generating includes processing the second real-time data stream through the second directed graph to a second output sink corresponding to the second output sink identifier.

41. The method of claim 38, wherein:

the parsing comprises extracting an identifier of a graphical object appearance processing object from the one or more stream handling instructions (210); and

the generating comprises processing the real-time data stream by one or more instantiated graphical object appearance processing objects corresponding to respective ones of the extracted identifiers, and dynamically configuring the instantiated graphical object appearance processing objects to control a visual representation of the first object in at least one of orientation, position, movement, and pose based at least in part on a position of the first object in the virtual area (28).

42. The method of claim 25, wherein the parsing, the instantiating, the creating, the receiving, the producing, and the generating are performed on a remote network node, and the method further comprises associating an interface element of a user-level application executing on a local network node (16) with the real-time data stream and transmitting the real-time data stream from the local network node (16) to the remote network node.

43. The method of claim 42, further comprising transmitting the one or more flow handling instructions from the local network node (16) to the remote network node.

44. A method as defined in claim 43, wherein the user-level application is a desktop application.

45. The method of claim 44, wherein the user-level application is Microsoft OfficeA desktop application.

46. The method of claim 25, wherein the stream handling instructions (210) specify the real-time stream handler (22) without referencing a function.

47. The method of claim 25 wherein the stream handling instructions (210) specify the real-time stream handler (22) without referencing a stream type.

48. The method of claim 25, wherein the generating comprises traversing the directed graph (212) from the output sink to the input source during each successive fixed length interval.

49. The method of claim 25, wherein the parsing comprises parsing one or more records, each record comprising a respective definition type, a respective definition length, and at least one of a definition type-specific field and a respective definition type-specific definition.

50. The method of claim 25, further comprising generating a human perceptible output based at least in part on the resulting data stream.

51. An apparatus, comprising:

a computer readable medium storing computer readable instructions; and

a data processing unit coupled to the memory to execute the instructions and to perform operations based at least in part on the execution of the instructions, the operations comprising:

parsing a description of a real-time stream handler (22) from one or more stream handling instructions (210), wherein the parsing includes parsing an input source identifier, an output sink identifier, and a respective identifier for each of one or more data processing objects from the one or more stream handling instructions,

Instantiating real-time stream handling objects corresponding to respective ones of the identifiers,

creating a directed graph (212) comprising some instantiated real-time flow handle objects from the description,

receiving a real-time data stream (214) from an input source corresponding to the input source identifier, an

Generating a resulting data stream at an output sink corresponding to the output sink identifier, wherein the generating comprises processing the real-time data stream through the directed graph (212).

52. At least one computer readable medium having computer readable program code embodied therein, the computer readable program code adapted to be executed by a computer to implement a method comprising:

parsing a description of a real-time stream handler (22) from one or more stream handling instructions (210), wherein the parsing includes parsing an input source identifier, an output sink identifier, and a respective identifier of each of one or more data processing objects from the one or more stream handling instructions;

instantiating real-time stream handling objects corresponding to respective ones of the identifiers;

creating a directed graph (212) comprising some instantiated real-time stream handling objects from the description;

receiving a real-time data stream (214) from an input source corresponding to the input source identifier; and

Generating a resulting data stream at an output sink corresponding to the output sink identifier, wherein the generating comprises processing the real-time data stream through the directed graph (212).

53. A computer-implemented method for real-time data stream handling, comprising:

establishing at least one real-time data flow connection between the local network node (16) and at least one remote network node (230);

processing, at the local network node (16), at least one real-time data stream originating from the remote network node, wherein the processing comprises processing the at least one real-time data stream by one or more real-time data processing operations to produce a resultant data stream (232);

monitoring the process (234); and

in response to determining that the process deviates from a performance goal based on the monitoring, the process is modified (236) according to a real-time performance goal routine.

54. The method of claim 53, wherein the modifying (236) comprises omitting processing of one or more portions of the real-time data stream in response to determining that the processing does not meet the performance goal.

55. The method of claim 53, wherein the modifying (236) comprises omitting one or more of the real-time data processing operations in response to determining that the processing does not meet the performance goal.

56. The method of claim 55, wherein said omitting comprises omitting one or more of said data processing operations characterized by corresponding performance values outside of said performance goal.

57. The method of claim 55, wherein some of said data processing operations are assigned respective priority values, and said omitting comprises preemptively omitting one or more of said data processing operations based on said assigned priority values.

58. The method of claim 53, wherein the modifying (236) comprises replacing at least one of the real-time data processing operations with a different respective real-time data processing operation in response to determining that the processing does not meet the performance goal.

59. The method of claim 53, wherein one or more of the real-time data processing operations are assigned respective priority values, wherein the real-time performance goal routine comprises a heuristic that determines data processing operations to be omitted from the data processing operations based on the assigned priority values, and wherein the modifying (236) is performed in accordance with the heuristic.

60. The method of claim 59, wherein said heuristic determines data processing operations to omit from said data processing operations based on a weighting of said assigned priority values by time-based performance statistics.

61. The method of claim 53, wherein the modifying (236) comprises iteratively modifying the processing until the processing falls within a specified performance target.

62. The method of claim 53, wherein the processing (232) comprises: instantiating processing objects, some of the processing objects for performing respective ones of the data processing operations; building a directed graph from some instantiated processing objects; and processing the real-time data stream through the directed graph.

63. The method of claim 62, wherein the modifying (236) comprises deleting one or more of the instantiated processing objects from the directed graph.

64. The method of claim 63, wherein some of said processing objects are assigned respective priority values, and said deleting comprises removing some of said instantiated processing objects from said directed graph based on said assigned priority values.

65. The method of claim 64, wherein the deleting comprises removing from the directed graph some instantiated processing objects that are assigned respective priority values that do not satisfy a priority threshold.

66. The method of claim 62, wherein the processing (232) comprises building a second directed graph from the instantiated processing objects and processing a second real-time data stream originating from the local network node (16) and one of the at least one remote network node through the second directed graph.

67. The method of claim 66, wherein the first and second directed graphs are assigned respective priority values, and the modifying (236) comprises preferentially modifying one of the first and second directed graphs based on the assigned priority values.

68. The method of claim 67, wherein the modifying (236) comprises omitting processing of one of the first and second real-time data streams in a corresponding one of the first and second directed graphs assigned a lowest priority value.

69. The method of claim 62, wherein the processing (232) comprises processing a second real-time data stream originating from one of the local network node (16) and the at least one remote network node through the directed graph.

70. The method of claim 69, wherein the first and second real-time data streams are assigned respective priority values, and wherein said modifying (236) comprises prioritizing the processing of one of the first and second real-time data streams based on the assigned priority values.

71. The method of claim 62, wherein the establishing (230) comprises establishing respective real-time data stream connections between the local network node (16) and a plurality of remote network nodes, and the processing (232) comprises processing real-time data streams originating from respective ones of the remote network nodes through the directed graph.

72. The method of claim 71, wherein the real-time data streams are assigned respective priority values, and wherein said modifying (236) comprises preferentially modifying processing of one or more of the real-time data streams based on the assigned priority values.

73. The method of claim 71, wherein the directed graph includes a plurality of directed chains for respective ones of the instantiated processing objects, and the processing (232) includes processing a respective one of the real-time data streams through each of the directed chains, wherein each of the real-time data streams originates from a respective one of the local network node (16) and the remote network node.

74. The method of claim 73, wherein the modifying (236) comprises iteratively modifying the processing until the processing falls within the specified performance goal, and during each iteration, the modifying comprises performing one or more of: (i) removing one or more of the chains from the directed graph and (ii) deleting one or more of the instantiated processing objects from the directed graph.

75. The method of claim 53, wherein the performance goal comprises a time-based threshold for generating the resultant data stream.

76. The method of claim 53, further comprising receiving, at the local network node (16), one or more flow handling instructions from a remote network node, the flow handling instructions comprising assigning respective priority values to the one or more real-time data processing operations, wherein the modifying (236) comprises modifying the processing based on the assigned priority values.

77. The method of claim 53, wherein the modifying (236) comprises reducing the computing resource load to a lower level in response to determining that the processing does not meet the performance goal.

78. The method of claim 77, wherein the modifying (236) comprises increasing the computing resource load from the lower level in response to determining that the processing meets the performance goal.

79. The method of claim 53, wherein said real-time data stream is grouped into frames, and said monitoring is performed on each of said frames.

80. The method of claim 79, wherein the processing (232) comprises processing the frames of the real-time data stream during each successive fixed length interval set according to a local clock.

81. The method of claim 80, further comprising synchronizing the local clock with a remote master clock service at the local network node (16).

82. The method of claim 53, wherein the monitoring (234) comprises monitoring utilization of at least one processor of the local network node (16).

83. The method of claim 53, wherein the monitoring (234) comprises monitoring bandwidth utilization of at least one networking resource of the local network node (16).

84. The method of claim 53, further comprising generating a human perceptible output at the local network node (16) in response to the resulting data stream.

85. The method of claim 84 wherein the local network node (16) and the remote network node are associated with respective objects in a virtual area (28), and the human perceptible output is a visualization of the objects in the virtual area (28) on a display.

86. An apparatus, comprising:

a computer readable medium storing computer readable instructions; and

a data processing unit coupled to the memory to execute the instructions and to perform operations based at least in part on the execution of the instructions, the operations comprising:

Establishing at least one real-time data flow connection between the local network node (16) and at least one remote network node (230),

processing, at the local network node (16), at least one real-time data stream originating from the remote network node, wherein the processing comprises processing the at least one real-time data stream by one or more real-time data processing operations to produce a resultant data stream (232);

monitoring said process (234), and

in response to determining that the process deviates from a performance goal based on the monitoring, the process is modified according to a real-time performance goal routine (236).

87. At least one computer readable medium having computer readable program code embodied therein, the computer readable program code adapted to be executed by a computer to implement a method comprising:

establishing at least one real-time data flow connection between the local network node (16) and at least one remote network node (230);

processing, at the local network node (16), at least one real-time data stream originating from the remote network node, wherein the processing comprises processing the at least one real-time data stream by one or more real-time data processing operations to produce a resultant data stream (232);

Monitoring the process (234); and

in response to determining that the process deviates from a performance goal based on the monitoring, the process is modified according to a real-time performance goal routine (236).

88. A computer-implemented method, comprising:

establishing a first session (362) with a remote network node on a transport stream according to a connectionless transport protocol on a local network node (16);

automatically opening one or more channels (364) for communicating data between the local network node (16) and the remote network node in the first session on behalf of one or more software entities on the local network node (16);

maintaining a table identifying some of the channels that are open and associating corresponding attribute values with the identified channels in the first session (366);

in response to determining that the first session failed, automatically attempting to establish a second session with the remote network node on a second transport stream according to the connectionless transport protocol (368); and

in response to successfully establishing the second session, automatically opening each of the channels identified in the table (370).

89. The method of claim 88 wherein the establishing (362) comprises determining a first station definition assigned to the remote network node and the maintaining comprises storing the first station definition in the table as an attribute of each of the open channels.

90. The method of claim 89, further comprising discovering a failure of the first session, wherein the discovering comprises determining a current station definition assigned to the remote network node and determining that the first session failed in response to determining that the current station definition is different from the first station definition.

91. The method of claim 89, wherein the determining comprises parsing a station definition record received from the remote network node, wherein the station definition record comprises a set of fields, each field defined by a respective field type and an associated field value, and each field type identified by a respective Globally Unique Identifier (GUID).

92. The method of claim 88, further comprising recording attributes of local published channels available from the local network node (16), local subscription channels requested by the one or more software entities, remote published channels available from the remote network node, and remote subscription channels requested by the remote network node.

93. The method of claim 92, wherein the recording comprises maintaining a record for each of the local published channels, the record comprising an identifier of one of the software entities indicating an ability to publish data on the local published channel, an identifier of a remote network node subscribing to the local published channel, and an identifier of the local published channel.

94. The method of claim 92, wherein the recording comprises maintaining a record for each of the local subscription channels, the record comprising an identifier of one of the software entities subscribing to the local subscription channel, an identifier of a remote network node indicating an ability to publish data on the local subscription channel, an identifier of the local subscription channel, and one or more network transmission parameters associated with the local subscription channel.

95. The method of claim 92, wherein the recording comprises maintaining a record for each of the remote publish channels, the record including an identifier of a remote network node indicating an ability to publish data on the remote publish channel and an identifier of the remote publish channel.

96. The method of claim 92, wherein the opening comprises sending a record to the remote network node defining the local published channel.

97. The method of claim 92, wherein the opening comprises sending a record for each of the local subscription channels to the remote network node with an identifier that matches an identifier of one of the remote published channels.

98. The method of claim 88, wherein the establishing (362) comprises: creating a definition of the session, wherein the definition includes an Internet Protocol (IP) address, a port address, and a globally unique identifier of a transport protocol; and sending the definition to the remote network node.

99. The method of claim 88, further comprising transmitting data between the local network node (16) and the remote network node on the one or more open channels in the session.

100. The method of claim 99, wherein the transmitting comprises transmitting the data in records, each of the records comprising a set of fields, each of the fields being defined by a corresponding field type and an associated field value, and each of the field types being identified by a corresponding GUID.

101. The method of claim 99, wherein the transmitting comprises transmitting a media record containing media data comprising packets of renderable data and transmitting a configuration record containing configuration data comprising definitions of configuration settings.

102. The method of claim 101, wherein the transmitting comprises encapsulating media records and configuration records in transport records on the transport stream.

103. The method of claim 102 wherein the encapsulating comprises compressing some of the media records using a first data compression service and compressing some of the configuration records using a second data compression service.

104. The method of claim 102 wherein the encapsulating comprises associating the transmission records with identifiers of respective ones of the channels on which the transmission records were transmitted, encrypting the transmission records, and arranging the encrypted transmission records in order.

105. The method of claim 102, further comprising receiving, at the local network node (16), some of the transmission records transmitted from the remote network node, wherein the receiving comprises decrypting the transmission records and assigning the media records and the configuration records contained in the decrypted transmission records to some of the software entities subscribing to the software entities.

106. An apparatus, comprising:

a computer readable medium storing computer readable instructions; and

a data processing unit coupled to the memory to execute the instructions and to perform operations based at least in part on the execution of the instructions, the operations comprising:

Establishing a first session (362) with a remote network node on a transport stream according to a connectionless transport protocol on a local network node (16);

automatically opening one or more channels (364) for communicating data between the local network node (16) and the remote network node in the first session on behalf of one or more software entities on the local network node (16);

maintaining a table identifying some of the channels that are open and associating corresponding attribute values with the identified channels in the first session (366);

in response to determining that the first session failed, automatically attempting to establish a second session with the remote network node on a second transport stream according to the connectionless transport protocol (368); and

in response to successfully establishing the second session, automatically opening each of the channels identified in the table (370).

107. At least one computer readable medium having computer readable program code embodied therein, the computer readable program code adapted to be executed by a computer to implement a method comprising:

establishing a first session (362) with a remote network node on a transport stream according to a connectionless transport protocol on a local network node (16);

Automatically opening one or more channels (364) for communicating data between the local network node (16) and the remote network node in the first session on behalf of one or more software entities on the local network node (16);

maintaining a table identifying some of the channels that are open and associating corresponding attribute values with the identified channels in the first session (366);

in response to determining that the first session failed, automatically attempting to establish a second session with the remote network node on a second transport stream according to the connectionless transport protocol (368); and

in response to successfully establishing the second session, automatically opening each of the channels identified in the table (370).

108. A method performed on a local network node (16), the method comprising:

parsing a list of kernel components including one or more kernel service components (320);

determining all kernel components in the parsed list that are missing in the local repository (322);

retrieving each of the kernel components determined to be absent (324);

instantiating a kernel service (326) from some of the kernel service kernel components;

executing the instantiated kernel service to communicate (328) with one or more remote network nodes in a communication environment defined with respect to a virtual area (28).

109. The method of claim 108, wherein the instantiating (326) comprises instantiating a stream connection kernel service, and the executing (328) comprises executing the stream connection kernel service to dynamically load at least one stream transport plug-in that provides at least one of a reliability function, an authentication function, and an encryption function.

110. The method of claim 109, wherein the executing (328) comprises dynamically loading the at least one streaming plug-in response to a network connection definition received from one of the remote network nodes.

111. The method as recited in claim 108, wherein the instantiating (326) includes instantiating a real-time scheduler kernel service and the executing (328) includes executing the real-time scheduler kernel service to dynamically load a time service plug-in that provides time synchronization functionality.

112. The method of claim 111 wherein the executing (328) comprises dynamically loading the time service plug-in response to a network connection definition received from one of the remote network nodes.

113. The method of claim 108, wherein the instantiating (326) comprises instantiating one or more flow handling kernel services, and the executing (328) comprises executing the one or more flow handling kernel services to dynamically create and configure at least one flow handler (22) from one or more plug-ins that provide flow processing functionality.

114. The method of claim 113 wherein the executing (328) comprises dynamically creating and configuring the stream handler (22) in response to a stream handler description received from one of the remote network nodes.

115. The method of claim 114 wherein the executing (328) comprises assembling a plurality of the plug-ins into a directed graph according to the stream handler description.

116. The method of claim 115 wherein the executing (328) comprises assembling one of the plug-ins into a second directed graph according to a second stream handler description received from the one remote network node, the first and second directed graphs having different respective plug-in topologies.

117. The method of claim 114 wherein the configuring comprises dynamically configuring the stream handler (22) while the stream handler (22) is processing a real-time data stream received from the one remote network node.

118. The method of claim 114 wherein the one or more remote network nodes are associated with respective objects in the virtual area (28), and the configuring comprises determining configuration parameter values based on locations of objects in the virtual area (28) and configuring ones of the plug-ins in the stream handler (22) with the configuration parameter values.

119. The method of claim 108, further comprising, at the local network node (16):

declaring an intent to enter the virtual area (28) to a network infrastructure service, an

An identification of a plug-in required for presenting the virtual area (28) is received from the network infrastructure service.

120. The method of claim 119, further comprising, at the local network node (16):

retrieving from a remote network node some of the plug-ins missing from the local network node (16).

121. The method of claim 119, further comprising loading at least one of the plug-ins and invoking one or more functions of the loaded plug-in during the presenting of the virtual area (28) on the local network node (16).

122. The method of claim 108, further comprising establishing a network session between the local network node (16) and one of the remote network nodes, wherein the establishing comprises receiving an identification of a set of identified transport Application Program Interfaces (APIs), selecting one variable from the set, and invoking one or more functions of the selected variable in passing a real-time data stream between the local network node (16) and the one remote network node in the session.

123. The method of claim 122, wherein the calling comprises loading a transport plug-in that supports the identified transport API and calling one or more functions of the transport plug-in.

124. An apparatus, comprising:

a computer readable medium storing computer readable instructions; and

a data processing unit coupled to the memory to execute the instructions and to perform operations based at least in part on the execution of the instructions, the operations comprising:

parsing a list of kernel components including one or more kernel service components (320),

determining all kernel components in the parsed list that are missing in the local repository (322),

retrieving each of the kernel components determined to be absent (324),

instantiating a kernel service (326) from some of the kernel service kernel components, an

Executing the instantiated kernel service to communicate (326) with one or more remote network nodes in a communication environment defined with respect to a virtual area (28).

125. At least one computer readable medium having computer readable program code embodied therein, the computer readable program code adapted to be executed by a computer to implement a method comprising:

Parsing a list of kernel components including one or more kernel service components (320);

determining all kernel components in the parsed list that are missing in the local repository (322);

retrieving each of the kernel components determined to be absent (324);

instantiating a kernel service (326) from some of the kernel service kernel components, an

Executing the instantiated kernel service to communicate (328) with one or more remote network nodes in a communication environment defined with respect to a virtual area (28).

126. A method performed on a local network node (16), comprising:

configuring the local network node (16) to support real-time communication with at least one remote network node in a context defined by a virtual area (28), the configuring comprising

In response to a call to enumerate all plugins that support a specified Application Programming Interface (API), returning a list containing identifiers of all plugins in the plugin database that are associated with the specified API (442),

in response to a call to enumerate variables of a given API supported by an identified one of the plug-ins, passing a list (446) including identifiers of all variables of the given API supported by the identified plug-in, and

In response to a call to the identified one variable instantiating the identified API supported by the identified one of the plug-ins, loading the identified plug-in and providing a pointer to the instance of the identified variable (450); and

establishing at least one real-time data flow connection between the configured local network node (16) and the at least one remote network node.

127. The method of claim 126, wherein the step of applying comprises applying a voltage to the substrate

The loading (450) includes creating a base plug-in object and a returned pointer to the base plug-in object, an

Further comprising executing the identified variable instance, wherein the executing comprises throwing the base plug-in object to the identified variable by inheritance.

128. The method of claim 126, wherein the communicating comprises instantiating some of the plugins in a set in response to a call that includes an identifier of the plugins in the set and an identifier of one or more of the APIs respectively supported by the plugins in the set, and assembling the instantiated plugins into a directed graph.

129. The method of claim 128, wherein the assembling is performed according to a flow handling description received from a remote network node.

130. The method of claim 126, wherein the returning, the passing, and the loading are performed independently of API type.

131. The method of claim 126, wherein at least one of the plug-ins supports a plurality of APIs and a plurality of variables for at least one of the plurality of APIs that the plug-in supports.

132. The method of claim 126, further comprising communicating with one or more remote network nodes in a communication environment defined with respect to a virtual area (28), wherein the communicating comprises executing the identified variable instance.

133. The method of claim 132, wherein the communicating comprises instantiating at least one of the plug-ins that provides streaming functionality.

134. The method of claim 132, wherein the communicating comprises instantiating at least one of the plug-ins that provides audio stream processing functionality.

135. The method of claim 132, wherein the communicating comprises instantiating at least one of the plug-ins that provides a visual graphic effect function.

136. The method of claim 132, wherein the communicating comprises instantiating at least one of the plug-ins that provides a collision detection function.

137. The method of claim 126, further comprising:

discovering a plug-in (430) available on the local network node (16);

querying the plug-in for all Application Programming Interfaces (APIs) it supports (432); and

based on the query, the associations between the plug-ins and their respectively supported APIs are stored in a plug-in database on the computer readable medium (434).

138. An apparatus, comprising:

a computer readable medium storing computer readable instructions; and

a data processing unit coupled to the memory to execute the instructions and to perform operations based at least in part on the execution of the instructions, the operations comprising:

configuring a local network node (16) to support real-time communication with at least one remote network node in a context defined by a virtual area (28), the configuring comprising

In response to a call to enumerate all plugins that support a specified Application Programming Interface (API), returning a list containing identifiers of all plugins in the plugin database that are associated with the specified API (442),

in response to a call to enumerate variables of a given API supported by an identified one of the plug-ins, passing a list (446) including identifiers of all variables of the given API supported by the identified plug-in, and

In response to a call to the identified one variable instantiating the identified API supported by the identified one of the plug-ins, loading the identified plug-in and providing a pointer to the instance of the identified variable (450); and

establishing at least one real-time data flow connection between the configured local network node (16) and the at least one remote network node.

139. At least one computer readable medium having computer readable program code embodied therein, the computer readable program code adapted to be executed by a computer to implement a method comprising:

configuring a local network node (16) to support real-time communication with at least one remote network node in a context defined by a virtual area (28), the configuring comprising

In response to a call to enumerate all plugins that support a specified Application Programming Interface (API), returning a list containing identifiers of all plugins in the plugin database that are associated with the specified API (442),

in response to a call to enumerate variables of a given API supported by an identified one of the plug-ins, passing a list (446) including identifiers of all variables of the given API supported by the identified plug-in, and

In response to a call to the identified one variable instantiating the identified API supported by the identified one of the plug-ins, loading the identified plug-in and providing a pointer to the instance of the identified variable (450); and

establishing at least one real-time data flow connection between the configured local network node (16) and the at least one remote network node.