HK1165581A - Pervasive realtime framework - Google Patents

Description

Pervasive real-time framework

Background

Advances in high-speed networking and computer processing resources have led to an expansion of a variety of different real-time software applications, including real-time communication systems (e.g., text chat, voice, and video communication systems) and real-time data streaming systems that require fast response times (e.g., online financial transaction systems). Real-time software applications operate on a computer in an application environment created by the computer's operating system. Computer operating systems typically provide a standardized, consistent Application Programming Interface (API) between real-time software applications and the computer system hardware. An API typically allows real-time software applications to interface with or access computer system hardware in a standardized manner through a set of low-level primitives. These low-level primitives must be integrated through low-level pipeline code to support the higher-level functionality of the real-time functionality required by the real-time software application. The minimal development support provided by the computer operating system places a significant burden on the developer of the real-time software application; not only is the amount of effort required to write the underlying pipeline code, but the inherent complexity of wiring code interacting directly with low-level primitives inevitably increases the time required to develop real-time software applications.

Summary of The Invention

In one aspect, the invention features a method in accordance with which an instance of a virtual area is determined based on a designation of the virtual area in a connection rule associated with at least one of a software application and a computer data file. One or more network nodes associated with an instance of the virtual area are ascertained. The transfer of the at least one real-time data stream is initiated via at least one network connection with at least one of the network nodes in a context defined by an instance of the virtual area.

In another aspect, the invention features a method in accordance with which a specification of at least one connection target in a connection rule associated with at least one of a software application and a computer data file is resolved to at least one corresponding network node connection handle. The transfer of the at least one real-time data stream is initiated via at least one network connection with at least one network node respectively associated with the at least one respective network node connection handle.

In another aspect, the invention features a method in accordance with which an Application Programming Interface (API) is published. The API declarations perform functions that include operations to manage network connections for real-time data flows based on connection rules. Each of the connection rules includes at least one of a respective designation of a virtual area and a respective designation of one or more connection targets associated with the virtual area. In response to a call to the API, managing real-time data stream connections with at least one network node based on a respective one of the connection rules associated with at least one of a software application and a computer data file and at least one location in the virtual area.

In another aspect, the invention features a method in accordance with which an Application Programming Interface (API) is invoked with a call to establish a network connection with at least one connection target in a context defined by an instance of a virtual area in accordance with connection rules associated with at least one of a software application and a data file on which the software application is operable. The connection rules include respective designations of virtual areas and respective designations of one or more connection targets associated with the virtual areas. The API is invoked with a call that initiates a transfer of at least one real-time data stream with respect to the connection target via the network connection based on the location in the virtual area instance.

In another aspect, the invention features a method in accordance with which at least the following is performed in response to a call by an Application Programming Interface (API) that includes a definition of a location in at least one of a software application and a computer data file: determining a connection rule associated with the location definition and comprising a designation of a virtual area; establishing a session with a network infrastructure service hosting an instance of the virtual area and publishing state data describing a current state of the virtual area instance; subscribing to the status data; and presenting a human-perceptible view of the status data.

In one aspect, the invention features a method in accordance with which at least the following is performed in response to an Application Programming Interface (API) call that includes a definition of a location in at least one of a software application and a computer data file: determining a connection rule associated with the location definition and comprising a specification of at least one connection target; establishing a session with a network infrastructure service that manages connection handle distribution of network nodes; declaring, to the network infrastructure service, an intent to connect to one or more of the connection targets specified in the connection object; receiving at least one respective network node connection handle from the network infrastructure service; and initiating delivery of the at least one real-time data stream via at least one network connection with the network node associated with the at least one respective network node connection handle.

In one aspect, the invention features a method in accordance with which at least the following is performed in response to an Application Programming Interface (API) call that includes a definition of a location in at least one of a software application and a computer data file: determining a connection rule associated with the location definition and comprising a specification of at least one connection target; establishing a session with a network infrastructure service that manages an exchange of presence data between network nodes; and declaring, to the network infrastructure service, an intent to output presence data including a definition of the location to at least one of the network nodes respectively corresponding to the at least one connection target.

The invention also features apparatus for implementing the inventive methods described above and computer readable media storing computer readable instructions for causing a computer to implement the inventive methods described above.

Other features and advantages of the invention will become apparent from the following description, including the drawings and claims.

Brief Description of Drawings

FIG. 1 is an illustration of an embodiment of an operating environment comprising a pervasive real-time framework and a network infrastructure services environment.

Fig. 2 is a flow diagram of an embodiment of a method performed by an embodiment of the pervasive real-time framework of fig. 1.

Fig. 3 is a block diagram of an embodiment of the pervasive real-time framework and network infrastructure service environment of fig. 1 to establish real-time communications in a communication context defined by a virtual area instance.

Fig. 4 is an illustration of an embodiment of a network node that includes a graphical user interface presenting a two-dimensional depiction of a shared virtual area.

FIG. 5A is a block diagram of an embodiment of a computer data file.

FIG. 5B is a diagram of an embodiment of a computer data file segment definition database storing records defining segments of the computer data file of FIG. 5A.

FIG. 6A is a block diagram of an embodiment of a software application file.

FIG. 6B is a diagram of an embodiment of a software application section definition database storing records defining sections of the software application of FIG. 6A.

FIG. 7A is an illustration of an embodiment of an abstract virtual space with zones mapped to sections of the software application in FIG. 6A.

FIG. 7B is an illustration of an embodiment of two visual virtual spaces with zones mapped to the segments of the software application in FIG. 6A.

FIG. 8 is a diagram of an embodiment of a connection object association database containing records with connection object identifiers pointing to corresponding records in an embodiment of the connection object database.

FIG. 9 is an illustration of an embodiment of a connection object database.

Fig. 10 is a block diagram of an embodiment of a network node connected to three other network nodes in an embodiment of the operating environment of fig. 1.

FIG. 11 is a flow diagram of an embodiment of a method of logging into an embodiment of the network infrastructure services environment of FIG. 1.

FIG. 12 is a flow diagram of an embodiment of a method implemented by an embodiment of the pervasive real-time framework of FIG. 1.

Fig. 13 is a flow diagram of an embodiment of a method implemented by an embodiment of the pervasive real-time framework of fig. 1 in ascertaining a connection target via an embodiment of a rendezvous service.

Fig. 14 is a flow diagram of an embodiment of a method implemented by an embodiment of the pervasive real-time framework of fig. 1 in ascertaining connection targets via an embodiment of a zone service.

FIG. 15 is an illustration of an embodiment of the operating environment shown in FIG. 10.

FIG. 16 is an illustration of an embodiment of an operating system and an embodiment of the pervasive real-time framework of FIG. 1.

Fig. 17 is a flow diagram of an embodiment of a method implemented by an embodiment of a zone connectivity service of the pervasive framework of fig. 1.

Fig. 18 is a flow diagram of an embodiment of a method implemented by an embodiment of a zone entry service, a flow switching service, and a flow handler service in an embodiment of the pervasive framework of fig. 1.

Fig. 19 is a flow diagram of an embodiment of a method implemented by an embodiment of a stream switching service in an embodiment of the pervasive framework of fig. 1.

Fig. 20 is a flow diagram of an embodiment of a method implemented by an embodiment of a stream switching service in an embodiment of the pervasive framework of fig. 1.

Fig. 21 is a flow diagram of an embodiment of a method implemented by an embodiment of a stream switching service in an embodiment of the pervasive framework of fig. 1.

Fig. 22 is a flow diagram of an embodiment of a method implemented by an embodiment of a target connectivity service in an embodiment of the pervasive framework of fig. 1.

Detailed description of the invention

In the following description, like reference numerals are used to identify like elements. Furthermore, the illustrations are intended to illustrate major features of example embodiments in a diagrammatic manner. The illustrations are not intended to depict every feature of actual embodiments or relative dimensions of the depicted elements, and are not drawn to scale.

I. Definition of terms

A "computer" is any machine, device, or apparatus that processes data according to computer-readable instructions stored temporarily or permanently on a computer-readable medium. A "computer operating system" is a software component of a computer system that manages and coordinates the execution of tasks and the sharing of computing and hardware resources. A "software application" (also known as software, an application, computer software, a computer application, a program, and a computer program) is a set of instructions that a computer can interpret and execute to perform one or more specific tasks. An "application programming interface" (or API) is a set of functional (or procedural) declarations that an operating system, library, or service provides to support requests made by software applications. The API specifies the interface and behavior of the identifier specified in the interface. An implementation of an API refers to software application code that provides the functionality described by the API. A "computer data file" is a block of information that persistently stores data for use by a software application.

An "interaction space" is an abstract space having one dimension for each of at least one of a software application and a computer data file and having "locations" corresponding to different sections of the software application and/or data file (e.g., chapters or presentation slides in the computer data file, and different functions or entry points of the software application). The "current focus of the user with respect to the interaction space" is the section of the software application and/or computer data file that is currently active on the user's network node.

A "database" is an organized collection of records presented in a standardized format that can be searched by a computer. The database may be stored on a single computer-readable data storage medium, or may be distributed across multiple computer-readable data storage media.

A "data sink" (referred to herein simply as a "sink") is any of a device (e.g., a computer), a portion of a device, or software that receives data.

A "data source" (referred to herein simply as a "source") is any of a device (e.g., a computer), a portion of a device, or software that originates data.

A "framework" is a set of classes of reusable, co-operating high-level functions and protocols that govern the manner in which low-level functions (e.g., operating system primitives and kernel primitives) may be combined, including rules that define how these primitives may be invoked by software applications and how the framework responds to such invocations.

A "network node" is a node or connection point in a communication network. Exemplary network nodes include, but are not limited to, terminals, computers, and network switches. A "network connection" is a link between two communication network nodes.

A "connection rule" specifies at least one of a virtual area and a connection target, and includes an optional set of one or more connection conditions that direct the behavior of an appropriately configured software application or service in initiating a network connection. "connection target" refers to an identifier or connection handle (e.g., a Uniform Resource Identifier (URI)) that may be used to establish a network connection with a correspondent, resource, or service on a network node. A "connection condition" specifies one or more parameters that affect the establishment of a network connection, the management of a network connection, or the processing of data communicated across a network connection. For example, the connection condition may describe a predicate on the operating environment that should be satisfied before the network connection is attempted or established.

An "object" is any type of discrete element in a virtual area that can be usefully treated differently from the geometry of the virtual area. Exemplary objects include doors, portals, windows, viewing screens, and loudspeakers. Objects typically have attributes and characteristics that are separate and distinct from those of the virtual area. An "avatar" is an object representing a correspondent in a virtual area.

"presence" refers to the ability or willingness of a networked entity (e.g., a correspondent, service, or device) to communicate, wherein such willingness affects the ability to detect and obtain information about the status of the entity on the network and the ability to connect to the entity.

A "real-time data stream" is data structured and processed in the form of a continuous stream and is designed to be received without delay or with only an imperceptible delay. The real-time data streams may fall into different priority categories from a higher priority hard real-time data stream (e.g., a voice stream) to a lower priority soft real-time data stream (e.g., a screen sharing data stream). The real-time data stream includes digital representations of voice, video, user movements, facial expressions, and other physical phenomena, as well as data within the computing environment that may benefit from rapid transmission, rapid execution, or both rapid transmission and rapid execution, including, for example, avatar movement instructions, text chatting, real-time data feeds (e.g., sensor data, machine control instructions, transaction streams, and stock quote information feeds), and file transfers.

"stream mixing" is a combination of two or more real-time data streams of the same or semantically consistent type (e.g., audio, video, chat, and motion data). For example, a set of voice streams may be mixed into a single voice stream or a voice stream may be mixed into the audio portion of a video stream.

A "switching rule" is an instruction that specifies the connection or disconnection of one or more real-time data sources to one or more real-time data sinks in accordance with one or more conditional precedents.

A "virtual area" (also referred to as an "area" or "place") is a representation of a space or scene managed by a computer. Virtual regions are typically one-dimensional, two-dimensional, or three-dimensional representations; although in some embodiments the virtual area may correspond to a single point. Virtual areas are often designed to simulate physical, real-world spaces. For example, using a conventional computer monitor, the virtual area may be visualized as a two-dimensional graphic of a three-dimensional space generated by a computer. However, the virtual area does not require an associated visualization to implement the switching rules. A virtual area generally refers to an instance of a virtual area pattern, where a pattern defines the structure and content of a virtual area in the form of variables, and an instance defines the structure and content of a virtual area in the form of values that have been parsed from a particular context.

The "virtual area specification" is a virtual area description used when creating a shared virtual area communication environment.

A "virtual communication environment" is a representation of a computer-managed space that includes at least one virtual area and supports real-time communication between communicants.

A "zone" is a zone in a virtual area associated with at least one switching rule or dominating rule. Switching rules control the switching (e.g., routing, connecting, and disconnecting) of real-time data streams between network nodes communicating through a shared virtual area. The governing rules control a communicant's access to a resource (e.g., an area, a region of an area, or contents of the area or region), the scope of the access, and subsequent results of the access (e.g., requirements that an audit record regarding the access must be recorded).

A "location" in a virtual area refers to the location of a point or area or volume in the virtual area. The points are typically represented by a single set of one-, two-, or three-dimensional coordinates (e.g., x, y, z) that define the points in the virtual area. The area is typically represented by three-dimensional coordinates of three or more coplanar vertices defining the boundaries of the closed two-dimensional shape in the virtual area. A volume is typically represented by three-dimensional coordinates of four or more non-coplanar vertices that define the closed boundaries of a three-dimensional shape in a virtual area.

A "communicant" is a person who communicates or otherwise interacts with others over a network connection, where the communication or interaction may or may not occur in the context of a shared virtual area. A "user" is a correspondent that is operating a particular network node that defines a particular point of view for descriptive purposes.

"predicates" are the conditional part of a rule.

As used herein, the term "including" means including but not limited to, and "comprises" means including but not limited to.

Introduction II

Embodiments described herein provide a pervasive real-time framework that supports execution of real-time software applications by high-level functionality that significantly reduces the effort and time required to develop real-time software applications in a new operating environment paradigm in which real-time connections between network nodes are pervasive. The pervasive real-time framework handles the complex tasks of connecting to communicants, virtual areas, and other network resources, and exchanging these connections in response to user input, and thereby enables software application developers to concentrate on developing advanced real-time software application functionality.

In some embodiments, the pervasive real-time framework provides functionality that integrates traditional operating system functionality (e.g., process management functionality, file management functionality, memory management functionality, storage management functionality, device management functionality, and network management functionality) with real-time functionality (e.g., real-time scheduling functionality, real-time connection functionality, and real-time data stream handling functionality).

In some embodiments, a pervasive real-time framework includes a library of pre-coding solutions for common pervasive real-time tasks, a kernel that supports the execution of these tasks, and tools for configuring and building applications that can take advantage of the functionality provided by the framework. In this manner, the pervasive real-time framework brings a set of real-time advanced functions and primitives in a peering relationship with traditional operating system primitives.

Some embodiments of the pervasive real-time framework enable software application designers to define semantics for locations in a software application or computer data file and associate the location semantics with at least one of a virtual area and a connection target. These embodiments include functionality that can be invoked by software applications to use location as a switching metaphor in a spatial network connectivity system. These features allow locations in a software application or computer data file to be used, for example, to drive connections to virtual areas, access to virtual areas, connections to communicants and other sources or sinks of real-time data streams, and determination of presence data related to communicants and network resources and services.

Overview

A. Pervasive Real Time (PRT) framework

Fig. 1 illustrates an embodiment of a Pervasive Real Time (PRT) framework 12 that manages network connections with various network nodes 14, 16, 18, 20, 22, 24, 26 based on a location in at least one of a computer data file 28 and a software application 30 and based on one or more connection rules 32 associated with at least one of the software application 28 and the computer data file. Each of the connection rules 32 specifies at least one of a virtual area and a connection target, and includes an optional set of one or more connection conditions that direct the behavior of an appropriately configured software application or service in managing network connections.

Fig. 2 illustrates an exemplary embodiment of a method implemented by the PRT framework 12. According to this method, the PRT framework 12 issues an Application Programming Interface (API) declaring functions to perform operations including managing network connections for real-time data flows based on the connection rules 32 (fig. 2, block 34). Each of the connection rules 32 includes at least one of a respective designation of a virtual area and a respective designation of one or more connection targets associated with the virtual area. In response to the call 36 to the API, the PRT framework 12 manages real-time data flow connections with at least one network node based on a respective one of the connection rules 32 associated with at least one of the software application 30 and the computer data file 28 and at least one location in the virtual area (fig. 2, block 38).

In the exemplary embodiment shown in FIG. 1, computer data file 28 contains three sections (S1, S2, S3), each section being defined as a respective location in computer data file 28 (e.g., a respective paragraph or chapter in a word processing document); the software application 30 contains six segments (S4, S5, S6, S7, S8, S9), each segment being defined as a respective location in the software application 30 (e.g., a respective function or entry point of the software application). Each of the sections (S1-S3) of the computer data file 28 and each of the sections (S4-S9) of the software application 30 are associated with at least one of the connection rules 32. In this way, the PRT framework 12 is able to connect each of the segments S1-S9 to a virtual area or one or more connection targets, or both, in real-time, including outputting presence information to the associated connection target, inviting the associated connection target to participate in a communication session in the associated virtual area, and establishing a real-time connection with the associated connection target.

For example, in implementing the method of fig. 2 for the embodiment shown in fig. 1, the PRT framework 12 manages respective real-time data flow connections with one or more of the network nodes 14-26 based on reported locations in at least one of the computer data files 28 and the software applications 30, and based on connection rules associated with the reported locations. When the reported location is S1, the PRT framework 12 manages real-time data stream network connections with the network node 14 (data source 1; e.g., real-time stock quote service) according to connection rules associated with location S1. When the reported location is S2, the PRT framework 12 manages the video conference network connection with the network nodes 16, 18, and 20 (business contacts) according to the connection rules associated with location S2. When the reported location is S8, the PRT framework 12 manages real-time data streaming connections with the network node 22 (data source 2; e.g., a real-time music or video streaming service) according to the connection rules associated with location S8. When the reported location is S9, the PRT framework 12 manages voice and text chat network connections with the network nodes 24, 26 (friends) according to the connection rules associated with location S9.

Thus, the association of the PRT framework 12 with software applications, computer data files, and portions thereof through connection rules enables software application developers to construct software applications that can operate in a pervasive, real-time connection environment. The PRT framework 12 manages real-time connections and real-time data stream processing as required by the connection rules and thereby allows software application developers to concentrate on the value-added tasks of generating software applications that can take advantage of the new paradigm of such pervasive network connectivity.

B. Operating environment

As shown in fig. 1 and 3, the PRT framework 12 operates in the context of an operating environment 40 that includes a network 42 and a network infrastructure services environment 43.

1. Network environment

Network 42 may include any of a Local Area Network (LAN), a Metropolitan Area Network (MAN), and a Wide Area Network (WAN) (e.g., the Internet). Network 42 typically includes a number of different computing platforms and transport facilities that support the transmission of a wide variety of different media types (e.g., text, voice, audio, and video) between network nodes.

The PRT framework 12 generally operates on network nodes that include software and hardware resources that, together with management policies, user preferences (including output regarding the presence of users and preferences of users' connections to regions and connection targets), and other settings, define local configurations 58 that affect management of real-time connections with other network nodes. Network connections between network nodes may be arranged in a variety of different stream handling topologies, including peer-to-peer architectures, server-arbitrated architectures, and hybrid architectures that combine aspects of peer-to-peer and server-arbitrated architectures. Exemplary topologies of these types are described in U.S. application nos. 11/923,629 and 11/923,634 filed on 24/10/2007.

2. Infrastructure service environment

The network infrastructure services environment 43 provides one or more network infrastructure services that cooperate with the PRT framework 12 to manage network connections with the network nodes 14-26. The network infrastructure services may run on a single network node or may be distributed across multiple network nodes. Network infrastructure services typically run on one or more dedicated network nodes (e.g., server computers or network devices that perform edge services such as routing and switching). However, in some embodiments, one or more of the network infrastructure services run on at least one of the communicants' network nodes. In some embodiments, one or more of the network infrastructure services run on one or more virtual machines, which may be running on the same physical hardware. Included in the exemplary operating environment 40 are network infrastructure services such as security service 44, regional service 46, rendezvous service 48, and interaction service 50.

The security service 44 controls the communicants' access to the resources of the operating environment 40. The access control method implemented by security service 44 is typically based on a capability (where access is granted to entities with appropriate capabilities or permissions) or an access control list (where access is granted to entities whose identities are on the list). After security service 44 has authorized access to a particular correspondent, the correspondent typically interacts in operating environment 40 using functionality provided by another network infrastructure service.

The zone service 46 hosts a virtual zone. In this process, the zone service 46 manages connections to the virtual zones in accordance with capabilities of the requesting entity, maintains global state information for the virtual zones, and acts as a data server for network nodes participating in the shared communication session in the context defined by the virtual zones. The global state information includes a list of all objects in the virtual area and their respective corresponding locations in the virtual area. Regional service 46 periodically sends global state information to participating network nodes. The regional service 46 also registers and communicates initialization information to other network nodes requesting to join the communication session. In this process, the area service 46 transmits to each joining network node a copy of the virtual area specification, which may be stored in a local or remote database. The regional service 46 also ensures that participating network nodes can synchronize to a global state in the event of a communication failure.

The rendezvous service 48 manages the collection, storage, and distribution of presence information and provides a mechanism for network nodes to communicate with each other (e.g., by managing the distribution of connection handles) in accordance with the capabilities of the requesting entity. The rendezvous service 48 typically stores presence information in a presence database.

The interaction service 50 maintains an interaction database that records interactions between communicants and supports queries of the interaction database in accordance with capabilities of the requesting entity. For each interaction between the communicants, one or more services in operating environment 43 (e.g., regional service 46) communicate interaction data to interaction service 50. In response, interaction service 50 generates one or more corresponding interaction records in a relational database. Each interaction record describes the context of the interaction. For example, in some embodiments, the interaction record contains an identifier for each communicant, an identifier for the interaction site (e.g., a virtual area instance), a description of the hierarchy of the interaction site (e.g., a description of what the interaction has a relationship with a larger area), the start and end times of the interaction, and a list of all files and other streams shared during the interaction. Thus, for each real-time interaction, interaction service 50 tracks when it occurs, where it occurs, and what happens during the interaction in terms of the communicants involved (e.g., entering and exiting), objects activated/deactivated, and files shared.

The interaction service 50 can present the results of queries to the interaction database records in a place-based sort order (e.g., most frequent or recent). The query results can be used to drive the ranking of the frequency of who the correspondent has met in which virtual areas, as well as the ranking of who the correspondent has met regardless of virtual area and the ranking of the virtual areas that the correspondent most frequently visits. The query results may also be used by software application developers as part of a heuristic system that automates certain tasks based on relationships. Examples of this type of heuristic operation are a heuristic that by default permits a communicant who has visited a particular virtual area more than 5 times to enter without knocking, or a heuristic that allows a communicant who is present in an area at a particular time to modify and delete a file created by another communicant who is present in the same area at the same time. Queries to the interaction database may be combined with other searches. For example, queries to the interaction database may be combined with queries for contact history data generated using communication systems (e.g., Skype, Facebook, and Flickr) outside the domain of the network infrastructure services environment 43 to interact with contacts.

3. Virtual area

Referring to fig. 3, in some embodiments, the PRT framework 12 manages real-time connections with network nodes in a communication context 54 defined by an instance 56 of a virtual area. The virtual area instance 56 may correspond to an abstract virtual space defined in terms of abstract coordinates (e.g., coordinates defined by a location in an associated computer data file or software application, or in embodiments where the customer service database is an area, each record in the database constitutes a zone). Alternatively, virtual area instance 56 may correspond to a visual virtual space defined with respect to one-, two-, or three-dimensional geometric coordinates associated with a particular visualization. The abstract virtual areas may or may not be associated with respective visualizations, while the visual virtual areas are associated with respective visualizations.

The communicants typically access the virtual area instances 56 from respective network nodes executing respective computing environments that are able to determine motion in the virtual area and establish real-time data stream connections with other network nodes. The communicants are typically represented by respective avatars in virtual areas with associated visualizations. The avatars move about the virtual area in response to input commands entered by the communicants at their respective network nodes. The view seen by a communicant with the associated visualized virtual area instances is typically presented from the perspective of the communicant's avatar, and each communicant is typically able to view any portion of the visual virtual area around his or her avatar, thereby enhancing the communicant's immersive level.

Fig. 4 illustrates an embodiment of an exemplary network node implemented by a computer system 60. The computer system 60 includes a display monitor 62, a computer mouse 64, a keyboard 66, speakers 68, 70, and a microphone 72. The display monitor 62 displays a graphical user interface 74. The graphical user interface 74 is a window-based graphical user interface that can include a plurality of windows, icons, and pointers 76. In the illustrated embodiment, the graphical user interface 74 presents a two-dimensional depiction of a shared virtual area 78 associated with a three-dimensional visualization representing a gallery. The communicants are represented in the virtual area 78 by respective avatars 80, 82, 84, each of which may have a respective character (e.g., a curator, an artist, and a guest) within the context of the virtual area 78.

As explained in detail below, the virtual area 78 includes zones 86, 88, 90, 92, 94 that are associated with respective rules governing the exchange of real-time data streams between network nodes represented by the avatars 80-84 in the virtual area 78. (during a typical communication session, the dashed lines demarcating zones 86-94 in FIG. 4 are not visible to the communicants, although there may be visual cues associated with such zone boundaries.) the exchange rules specify how the local connection process performed on each network node establishes communication with other network nodes based on the location of the communicant's avatar 80-84 in the zone 86-94 of the virtual area 78.

During the communication session, each correspondent network node generates a respective set of real-time data streams (e.g., a motion data stream, an audio data stream, a chat data stream, a file transfer data stream, and a video data stream). For example, each communicant manipulates one or more input devices (e.g., the computer mouse 64 and the keyboard 66) that generate a motion data stream, which controls the movement of his or her avatar in the virtual area 78. In addition, the communicator's voice and other sounds generated locally near the computer system 60 are captured by the microphone 72. Microphone 72 generates an audio signal that can be converted to a real-time audio stream. Respective copies of the audio stream are transmitted to other network nodes represented by avatars in the virtual area 78. Sounds generated locally at these other network nodes are converted to real-time audio signals and transmitted to the computer system 60. The computer system 60 converts audio streams generated by other network nodes into audio signals that are rendered by speakers 68, 70. The motion data stream and the audio data stream may be transmitted directly or indirectly from each communicant node to the other communicant network nodes. In some flow handling topologies, each correspondent network node receives copies of the real-time data flow transmitted by other correspondent network nodes. In other flow handling topologies, one or more correspondent network nodes receive one or more flow mixes derived from real-time data flows originating from (originating from) some other network nodes.

The virtual area is defined by a specification that includes a description of the geometric elements of the virtual area and one or more rules, including exchange rules and governing rules. The switching rules govern the real-time streaming connections between the network nodes. The governing rule controls access of the communicant to resources such as the virtual area itself, the zone having the virtual area, and objects within the virtual area. In some embodiments, the geometric elements of the virtual area are in accordance with COLLADA, a 2006 4-month digital asset plan version 1.4.1 Specification (available from Collada @), 2006http://www.khronos.org/collada/Derived) and switching rules are described in accordance with the COLLADA stream reference specifications described in U.S. application nos. 11/923,629 and 11/923,634.

The geometric elements of the virtual area typically include the physical geometry and collision geometry of the virtual area. The physical geometry describes the shape of the virtual area. Physical geometries are typically formed by surfaces that are triangular, quadrilateral, or polygonal. Colors and textures are mapped onto the physical geometry to create a more realistic appearance of the virtual region. For example, lighting effects may be provided by tracing the lighting on the visual geometry and modifying the texture, color, or brightness in the vicinity of the lighting. Collision geometry describes an invisible surface that determines the way an object can move in a virtual area. The collision geometry may be consistent with the visual geometry, correspond to a simpler approximation of the visual geometry, or relate to application-specific requirements for the virtual area designer.

The switching rules typically include a description of the conditions for connecting the source and sink of the real-time data stream by location in the virtual area. Each rule typically includes attributes that define the type of real-time data stream to which the rule applies and the location(s) to which the rule applies in the virtual area. In some embodiments, each rule optionally may include one or more attributes specifying a required role for the source, a required role for the sink, a priority level for the flow, and a requested flow handling topology. In some embodiments, if no explicit exchange rules are defined for a particular portion of the virtual area, one or more implicit or default exchange rules may be applied to that portion of the virtual area. One exemplary default switching rule is a rule that connects each source within the region to each compatible sink in accordance with the policy rules. Policy rules may apply globally to all connections between regional clients or only to corresponding connections with individual regional clients. An example of a policy rule is a proximity policy rule that only allows connection of sources to compatible traps associated with respective objects within a prescribed distance (or radius) from each other in the virtual area.

In some embodiments, governance rules are associated with a virtual area to control who can access the virtual area, who can access its content, what the scope of access to the content of the virtual area is (e.g., what the user can do with the content), and what the subsequent results of accessing the content are (e.g., log tracking, such as audit logs, and payment requirements). In some embodiments, the entire virtual area or a zone of the virtual area is associated with a "dominating grid". In some embodiments, the dominating grid is implemented in a manner similar to the implementation of the zone grid described in U.S. application nos. 11/923,629 and 11/923,634. The governance grid enables a software application developer to associate governance rules with a virtual area or zone of a virtual area. This avoids the need to create an individual license for each file in the virtual region and avoids the need to deal with the complexity that can arise when the same document needs to be treated differently depending on the context.

In some embodiments, a virtual area is associated with a governing grid that associates one or more zones of the virtual area with Digital Rights Management (DRM) functions. The DRM function controls access to one or more of the virtual area or one or more zones within the virtual area or objects within the virtual area. The DRM function is triggered each time the correspondent crosses a dominant grid boundary within the virtual area. The DRM function determines whether the trigger action is permitted and, if so, the extent of the permitted action, whether payment is required, and whether an audit record needs to be generated. In an exemplary implementation of a virtual area, the associated governing grid is configured such that if a communicant is able to enter the virtual area, he or she can perform actions on all documents associated with the virtual area, including manipulating documents, viewing documents, downloading documents, deleting documents, modifying documents, and re-uploading documents. In this way, the virtual area may become a repository for information that is shared and discussed in the context defined by the virtual area.

Additional details regarding the specifications of the virtual areas are set forth in U.S. application nos. 61/042714 (filed on 4/2008), 11/923,629 (filed on 24/10/2007), and 11/923,634 (filed on 24/10/2007).

4. Location in software applications and computer data files

Some embodiments of the pervasive real-time framework 12 enable software application designers to define semantics for locations in a software application or computer data file. These location definitions may be used, for example, to drive connections to virtual areas, ingress of virtual areas, connections to communicants and other sources or sinks of real-time data streams, and determination of presence data related to communicants, network resources, and network services through association with respective connection rules.

As described above, a computer data file is any block of information that persistently stores data for use by a software application (e.g., information that is used as input to the software application, and/or written as output by the software application). The computer data file may be designed for use with any type of software application, including consumer and enterprise software applications, and may be stored in any type of open or closed data file format. The computer data file includes one or more components whose contents and structure depend, at least in part, on the type and use of the data stored in the computer data file and on the software used to create it. For example, computer data files for desktop publishing software applications typically have components corresponding to predefined sections or categories of data, such as text, diagrams, tables, rows/columns, pages, sketches, presentation slides, and spreadsheets arranged in sentences, paragraphs, headings, and blocks, or functional features such as security features or authentication features. A software application developer or end user may define one or more sections of a computer data file that include one or more constituent components of the computer data file. A segment may comprise the entire computer data file or a portion of the computer data file and may overlap, in whole or in part, another segment.

FIG. 5A illustrates an exemplary computer data file 96 comprising a data file container 97, the data file container 97 holding a hierarchical arrangement of components A, B, C, D, sub-components B (1), B (2), and B (3) of component B, and sub-components C (1), C (2), and C (3) of component C. The computer data file 96 is divided into three sections (S1, S2, S3), each section being associated with one or more of the components and subcomponents of the computer data file 96. For example, section S1 is associated with subassembly B (1), section S2 is associated with assembly C, and section S3 is associated with subassembly (B3) and assembly C.

Associations between components and sections of computer data files may be stored in a variety of different data structure formats. For example, FIG. 5B illustrates an exemplary computer data file segment definition database 98 containing records defining segments in a computer data file 96. The segment definition records are indexed by record identifiers and identifiers of the computer data files 96. In some embodiments, the PRT framework 12 provides functionality that allows software application developers to design software applications with one or more dialog boxes that may be used by end users to create records in the computer data file segment definition database 98.

As described above, a software application is a set of instructions that a computer can interpret and execute after having been loaded into a storage medium (e.g., a hard drive, memory, or random access memory). A software application includes one or more logical components (e.g., hardwired, temporary, contextual, look-up table components) or functional components (e.g., functions or entry points) whose content and structure depend, at least in part, on the functionality of the software application and its design architecture. For example, some software applications use a componentized framework composed of several components, each of which may be contained in a separate library and may expose a respective set of interfaces that enable it to be hosted by a host executable. A software application developer may define one or more sections of a software application that include one or more components of the software application. A section may include the entire software application or a portion of the software application and may overlap, in whole or in part, another section.

FIG. 6A illustrates an exemplary software application 99 that includes an application container 100, the application container 100 maintaining a hierarchical arrangement of component E, F, sub-components E (1), E (2), and E (3) of component E, sub-components F (1), F (2) of component F, sub-components F (i) and F (ii) of sub-component F (1), and sub-component F (iii) of sub-component F (2). Software application 100 is divided into six sections (S4, S5, S6, S7, S8, S9), each section being associated with one or more of the components and subcomponents of software application 99. For example, section S4 is associated with subassembly B (1), section S5 is associated with assembly F, section S6 is associated with subassembly F (1), section S7 is associated with subassembly F (2), section S8 is associated with subassemblies F (ii) and F (iii), and section S9 is associated with subassemblies E (3) and F (i).

Associations between components and segments of a software application may be stored in a variety of different data structure formats. For example, FIG. 6B illustrates an exemplary software application segment definition database 102 containing records defining segments in a software application 99. The segment definition records are indexed by a record identifier and an identifier of the software application 99. In some embodiments, the PRT framework 12 provides one or more dialog boxes that allow a software application developer to create database records in the software application segment definition database 102.

Referring to fig. 7A and 7B, in some embodiments, the one or more sections of the computer data file or software application are associated with respective zones of the virtual area.

For example, in some embodiments, the one or more sections of the computer data file or software application are associated with respective zones of an abstract virtual area defined with respect to coordinates in a one-dimensional or multi-dimensional abstract topological space having a one-to-one mapping to locations in the associated computer data file or software application. For example, FIG. 7A illustrates an exemplary mapping of segments S4-S9 of software application 99 to respective coordinates in a two-dimensional topological space having coordinates that map directly to respective components and subcomponents of software application 99.

In other embodiments, the one or more sections of the computer data file or software application are associated with respective zones of one or more visual virtual areas, each of which may be defined with respect to one-, two-, or three-dimensional geometric coordinates associated with a respective visualization. For example, FIG. 7B illustrates an exemplary mapping of segments S4-S9 of software application 99 to the respective coordinates of the two visual virtual areas 104 and 78. In this example, section S4 is associated with the three-dimensional visualized visual virtual area 104 associated with the meeting room, and sections S5-S9 are associated with respective zones of the three-dimensional visualized visual virtual area 78 associated with the gallery.

5. Connection rules

The PRT framework 12 provides functionality that enables software application developers to incorporate software application protocols (e.g., dialogs and the like) that allow end users to associate computer data files or software applications with one or more connection rules. The computer data file or software application may be associated with the connection rule in a variety of different ways. In some embodiments, a computer data file is associated with a connection rule by storing or referencing the connection rule in an attribute database managed by a file manager service of an operating system (e.g., in an extensible file attribute that may be associated with the computer data file at a file system level of the operating system). In some embodiments, the software application is associated with the connection rule by storing the connection rule or a reference to the connection rule in a header of a software application file (e.g., a header or a fragment that describes how the software application should be loaded into memory by a program loader service of the operating system). In some embodiments, a computer data file or software application is associated with a connection rule by storing the connection rule or a reference to the connection rule in a separate database record indexed by an identifier of the computer data file or software application.

In the illustrated embodiment, a computer data file and a software application are associated with instances of reusable connection objects that each encapsulate one or more connection rules and one or more optional methods. Each connection rule specifies at least one of a virtual area and a connection target and includes an optional set of one or more connection conditions that affect establishment of a network connection, management of a network connection, or processing of data communicated across a network connection. The types of methods encapsulated in the connection object include, for example, methods that invoke functionality of the PRT framework 12 and methods that expose connection rules that may be used by the PRT framework 12, the operating system, or may be presented to the user in a graphical interface.

A segment of a computer data file or software application is typically associated with a respective instance of a connection object by way of a respective record in a connection object association database that indexes each record with an identifier of the computer data file or software application and an identifier of the associated segment. FIG. 8 illustrates an exemplary connection object association database 106 that includes, for each segment of a computer data file or software application, a respective record associating the segment with one or more references (e.g., O1, O2, O3, O4, etc.) to one or more respective connection objects indexed in the connection object database 108.

FIG. 9 illustrates an embodiment 110 of the connection object database 108 that includes, for each connection object, an exemplary set of possible attributes (or fields) indexed by an object ID attribute key whose value corresponds to an identifier (e.g., O1) of the connection object. Exemplary attributes included in the connection object database 108 are an area specifying attribute, a connection target specifying attribute, and a connection condition attribute.

The region-specifying attribute contains a region-specifying value that identifies or can be used to identify an instance of the virtual region. For example, the region designation value may correspond to any of a schema identifier (e.g., schema _ ID) identifying a schema of the virtual region, a region identifier (e.g., region _ ID) identifying an instance of the virtual region, a query to the interaction database returning an identifier of the virtual region instance, and a reference to another region (e.g., a virtual region outside of the domain of the associated software application or computer data file).

The connection target designation attribute contains one or more connection target designations that identify or can be used to identify one or more connection targets. For example, the connection target designation may include at least one of information identifying a single communicant, information identifying a group of communicants, and a definition of a role associated with at least one communicant. The connection target specifies identifiers of one or more queries that may correspond to one or more specific connection targets (e.g., fixed network resources and specific communicants) and return identifiers of the connection targets. Exemplary queries include a query to the rendezvous service 48 for any connection targets currently in the specified virtual area instance, a query to the rendezvous service 48 for specific connection targets currently in the specified virtual area instance, a query to the rendezvous service 48 for connection targets associated with one or more attributes (e.g., a particular role attribute value or a particular group identifier value), and a query to the interaction service 50 for connection targets that have interacted with the user and optionally are associated with one or more other attributes (e.g., connection targets associated with the specified virtual area instance or a specified attribute value such as a particular role attribute value or a group identifier value).

The connection condition attribute contains one or more connection condition definitions specifying one or more connection conditions. Each connection condition specifies one or more parameters that affect the operation of the PRT framework 12 when connecting to a virtual area instance or connection target specified in the associated connection rule. For example, the connection target may specify one or more parameters that affect any of establishment of a network connection, management of a network connection, processing of data communicated across a network connection, and presentation of information related to the connection to a user.

Some connection conditions describe predicates on the current connection context that should be satisfied before a network connection is attempted or established. Such exemplary conditions include connection conditions that limit when network connections are permitted, conditions that limit the establishment of network connections to network nodes that meet specified resource requirements, conditions that limit the establishment of network connections to network nodes that meet specified node configuration requirements, conditions that limit the establishment of network connections to network nodes that meet specified network node location requirements, and conditions that limit the establishment of network connections to times that meet specified connection target availability requirements.

Other connection conditions include conditions on the instantiation behavior of the virtual area (e.g., a new instance of the virtual area mode should be instantiated each time the software application is run), and conditions on the manner in which the query results should be handled (e.g., results are presented to the user in a graphical interface or automatically connected to a connection target only if a single identified connection target is returned in response to the query).

Connection objects and their constituent connection rules may be added, modified, or deleted by a variety of different functions and services depending on the implementation of the operating environment. For example, in some embodiments, connection rules may be added, modified, or deleted by a software application developer, a local system administrator, an end user, the PRT framework 12, or one or more network infrastructure services in the network infrastructure services environment 43.

System architecture

A. Overview

1. Introduction to the design reside in

The communicants typically connect to the network 42 (see FIG. 1) from network nodes, which are typically implemented by general purpose computer systems or special purpose communication computer systems (or "consoles" such as network-enabled video game consoles). The network nodes perform a communication process that establishes real-time data stream connections with other network nodes and typically perform a visualization process that presents a view of each virtual area that the user enters. In some embodiments, multiple communicants may share a single network node.

Figure 10 illustrates an embodiment of a network node implemented by a computer system 120, the computer system 120 including a processing unit 122, a system memory 124, and a system bus 126 that couples the processing unit 122 to the various components of the computer system 120. The processing unit 122 may include one or more data processors, each of which may be in the form of any one of commercially available computer processors. The system memory 124 may include Read Only Memory (ROM) that stores a basic input/output system (BIOS) that contains the start-up routines for the computer system 120, and may also include Random Access Memory (RAM). The system bus 126 may be a memory bus, a peripheral bus, or a local bus, and may be compatible with any bus protocol, including PCI, VESA, Microchannel, ISA, and EISA. Computer system 120 also includes a persistent storage memory 128 (e.g., hard disk drive, floppy disk drive, CD ROM drive, tape drive, flash memory device, and digital video disk) connected to system bus 126 and containing one or more computer-readable media disks providing non-volatile or persistent storage of data, data structures, and computer-executable instructions.

A user can interact with (e.g., enter commands or data) computer system 120 using one or more input devices 130 (e.g., one or more keyboards, computer mice, microphones, cameras, joysticks, physical motion sensors (such as Wii input devices), and touch pads, etc.). The information may be presented through a Graphical User Interface (GUI) presented to the correspondent on the display monitor 132, which is controlled by the display controller 134. The computer system 120 may also include other input/output hardware 136 (e.g., peripheral output devices such as speakers and printers). The computer system 120 is connected to other network nodes 138, 140, 142 through a network adapter 138 (also referred to as a "network interface card" or NIC).

A number of program modules may be stored in the system memory 124, including an Operating System (OS)144 (e.g., Windows XP, available from Microsoft corporation of Redmond, Wash., USA)An operating system), PRT framework 12, drivers 146 (e.g., GUI drivers), network protocols 148, PRT-aware software applications 150, PRT-agnostic software applications 152 connected to PRT framework 12 through shims 154, and data (e.g., input data, output data, program data, registries 156, and connection rules 32). In some embodiments, the shim 154 is implemented by an extension module (e.g., a plug-in) or a macro.

2. Operating system

Operating system 144 hosts software applications by providing the underlying operating system services used to create runtime execution environments on computer system 120. Exemplary types of services that are typically provided by operating systems are resource management, file management, security, authentication, verification, notification, and user interface (e.g., windowing, menus, dialogs, etc.).

Services related to the management of resources (e.g., memory, processors, and I/O devices) of computer system 120 are typically implemented by the kernel. File management may be implemented by the kernel, or may be by a separate file system manager (e.g., in some Microsoft WindowsWindowsAn installable file system provided in an operating system). In opening a file (e.g., a computer data file or a software application file), the file system manager typically invokes an appropriate file system driver that looks up the disk storage locations of the file in a database (e.g., a file allocation table such as FAT, FAT98, VFAT, MFT, and CDFS) that maps out the storage locations of the file on the disk. Other operating system functions (such as security, authentication, verification, notification, and user interface) may be performed by one or more other components of the operating system (e.g., some Microsoft WindowsWindowsAn executive services layer in an operating system).

Exemplary types of services that are typically provided by a kernel are process management, memory management, device management, and system call handling. Process management includes running applications and providing Application Programming Interfaces (APIs) to hardware components of a computer system. During the execution of a software application, the kernel typically establishes an address space for the software application in memory, loads a file containing the software application code in the address space, and executes the loaded software application code. Memory management involves managing access to system memory 124 by software applications. Device management involves providing access to hardware devices through device drivers. System call handling involves providing an API exposing kernel services to user mode software applications. By calling the API (e.g., through interprocess communication mechanisms and system calls), the software application may request services from the kernel, pass parameters, and receive results generated by the services in response to the requests.

Operating system 144 typically stores hardware and software configuration information, user preferences, and settings information in registry 156. For example, the registry 156 typically contains the following information: parameters required to boot and configure the system; system-wide software settings that control the operation of the operating system 144; a secure database, and per-user profile settings. In some embodiments, connection rules 32 are stored in registry 156 rather than in a separate database.

3. Network protocol

Network protocol 148 controls or enables connections, communications, and data transfers between computer system 120 and other network nodes. Exemplary network protocol types include transmission control protocol/internet protocol (TCP/IP), user datagram protocol/internet protocol (UDP/IP), and real-time transport protocol (RTP).

TCP/IP includes a TCP portion and an IP portion. The TCP portion of the protocol provides transport functionality by breaking messages into smaller packets, reassembling the packets at the other end of the communication network, and resending any packets lost on the way. The IP portion of the protocol provides routing functionality by assigning data packets with the address of the destination network and the address of the destination node on the destination network. Each data packet communicated using TCP/IP includes a header containing TCP and IP information. IP does not provide guarantees of packet delivery to upper layers of the communications stack. TCP, on the other hand, provides connection-oriented, end-to-end transport services with guaranteed in-order packet delivery. In this way, the TCP protocol provides a reliable transport layer connection.

UDP may be used instead of TCP without requiring reliable delivery. For example, UDP/IP may be used for real-time audio and video traffic, simply ignoring lost data packets for reasons of no time retransmission or any degradation in overall data quality being acceptable.

RTP defines a standardized packet format for delivering audio and video over a network connection. Various network protocols may be used to transmit and receive RTP data between network nodes, including peer-to-peer networking frameworks, central servers using TCP sockets alone or in conjunction with UDP, and multicast protocols.

4. Device driver

The device drivers 146 are typically implemented by software applications that enable other software applications (e.g., user-mode software applications and operating systems) to interact with hardware devices connected to the computer system 120. Device drivers typically provide APIs for functions that can be called by other software applications to translate commands and data passed between the software applications and the hardware device.

PRT framework

The PRT framework 12 includes a PRT API and a PRT kernel.

The PRT API exposes high-level functionality that significantly reduces the effort and time required to develop real-time software applications in a new operating environment paradigm in which real-time connections between network nodes are pervasive. Some of these functions use locations in at least one of computer data files and software applications to control connections to virtual areas, access to virtual areas, connections to communicants and other sources or sinks of real-time data streams, and determination of presence data related to communicants and network resources and services.

The PRT core includes services that control the exchange of real-time data streams between the computer system 120 and other network nodes 138, 140, 142. In some embodiments, the PRT kernel also includes a process that controls the presentation of the virtual area and the corresponding views of the objects in the virtual area on the display monitor 132. In this regard, the PRT kernel interfaces with operating system functions that communicate with the driver 148 to translate commands and data to and from the display controller 134 and the user input 130 to present a view of the virtual area and to allow the communicant to control interactions in the virtual area.

Implementations of the PRT API and PRT kernel include one or more discrete modules or discrete libraries (e.g., dynamic link libraries) that are not limited to any particular hardware, firmware, or software configuration. In general, these modules may be implemented in any computing or data processing environment, including in digital electronic circuitry (e.g., an application specific integrated circuit such as a Digital Signal Processor (DSP)) or in computer hardware, firmware, device drivers, or software. In some embodiments, the functionality of the modules is combined into a single data processing component. In some embodiments, the respective functionality of each of the one or more modules is performed by a respective set of multiple data processing components. In some implementations, the process instructions (e.g., computer readable code, such as computer software) for implementing methods performed by embodiments of the PRT API and PRT kernel, as well as the data they generate, are stored on one or more computer readable media. Storage devices suitable for tangibly embodying these instructions and data include all forms of non-volatile computer-readable memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks and removable hard disks, magneto-optical disks, DVD-ROM/RAM, and CD-ROM/RAM.

6. Exemplary System level functionality

The PRT framework 12 cooperates with the network infrastructure services environment 43 to manage network connections between the computer system 120 and other network nodes 138, 140, 142. Exemplary system-level functionality involved in managing network connections is logging into the network infrastructure service environment 43 and performing connection rule-based PRT framework operations, which include determining instances of virtual areas and ascertaining connection targets via at least one of the rendezvous service 48 and the area service 46.

a. Logging into a network infrastructure service environment

In some embodiments, authentication with respect to the network infrastructure service environment 43 is performed upon each initiation of the PRT framework 12. The authentication process is typically based on credentials that are securely issued to the user when the PRT framework 12 is installed on the computer system 120. The credential is typically a certificate issued by a certificate authority. The certificate contains a private key and a public key. The PRT framework 12 creates a new credential containing only the public key and securely stores the private key on the computer system 120. The computer PRT framework 12 creates a signature using the private key to encrypt a digest of the password provided by the user and passes the signature to the security service 44. The security service 44 recovers the digest and stores it as the user's identification secret.

Fig. 11 illustrates an embodiment of a method for the PRT framework 12 to log into the network infrastructure services environment 43 after the security service 44 has received the user's identification secret. According to this method, the PRT framework 12 establishes a session with the security service 44 and sends the user's credentials to the security service 44 (fig. 11, block 160). The security service authenticates the credentials received from the PRT framework 12 (fig. 11, block 162). If the user's credentials pass authentication, the security service sends an authentication token to the PRT framework 12; otherwise, the security service notifies the PRT framework 12 that the authentication failed (fig. 11, block 164). If authentication fails (FIG. 11, block 166), the PRT framework 12 notifies the user and the login process stops (FIG. 11, block 168). If the authentication of the user credentials is successful (fig. 11, block 166), the PRT framework 12 validates the token received from the security service 44 (fig. 11, block 170). If the token is valid (FIG. 11, block 170), the PRT framework waits for a software application or operating system call to call the PRT API (FIG. 11, block 172) before proceeding. After verification, the security service token can authenticate the user to any network infrastructure service. In some embodiments, the PRT framework 12 may execute background tasks (e.g., cache interaction data and other resources to improve responsiveness of the PRT framework 12 in managing connections and executing other tasks) while waiting for calls of the PRT API.

b. Connection rule based PRT framework operation

i. Overview

The connection rules may be associated with definitions of at least one respective section of the interaction space defined with respect to at least one of the software application and the computer data file. The PRT framework may determine a connection rule based at least in part on a current focus for the at least one segment in the interaction space, and then initiate delivery of one or more real-time data streams with respect to one or more network nodes in accordance with the connection rule. In these embodiments, the PRT framework 12 is invoked through a PRT API call that typically includes a definition of a location in at least one of a software application and a computer data file. The location definition may be provided by, for example, a PRT-aware software application that determines a current focus of the user with respect to an interaction space defined by at least one of the software application and the computer data file. Alternatively, the location definition may be provided by an operating system service (e.g., a program loader service or a file manager service) that determines the location definition from metadata associated with at least one of the software application and the computer data file (e.g., in a header of the software application file or in a file properties database associated with the computer data file) or from a user interface service of an operating system that controls the windowing environment.

Fig. 12 illustrates an embodiment of a method implemented by the PRT framework 12 in response to a PRT API comprising a location definition in at least one of a software application and a computer data file.

According to this method, the PRT framework 12 retrieves one or more connection rules associated with at least one of the software application and the computer data file (fig. 12, block 174). In some embodiments, the PRT framework 12 queries from the connection object association database 106 (see fig. 8) the identifier of at least one connection object associated with the section of the software application or computer data file corresponding to the location definition contained in the PRT API call. The PRT framework 12 retrieves the connection object corresponding to the connection object identifier from the connection object database 108 (see fig. 8) and determines at least one connection rule defined in the connection object.

As explained above, the connection rules specify at least one of a virtual area and a connection target, and may include one or more optional connection conditions that direct the behavior of appropriately configured software applications or services in managing network connections. If a virtual area is specified in the connection rule, the PRT framework 12 determines a virtual area instance based on the virtual area specification (fig. 12, block 176). The PRT framework 12 ascertains one or more connection targets from the connection rules (fig. 12, block 178). For example, connection targets can be ascertained through their designation in connection rules or through their association with a virtual area instance (e.g., ascertained connection targets are those associated with objects currently in the virtual area instance). If the connection rules specify one or more connection targets and do not specify a virtual area, the PRT framework 12 typically ascertains the connection targets via the rendezvous service 48. If the connection rule specifies a virtual area, then the PRT framework 12 typically ascertains the connection objective via one or both of the area service 46 and the rendezvous service 48.

After having ascertained the connection targets, the PRT framework 12 initiates respective real-time data stream connections with one or more of these connection targets (fig. 12, block 180).

Ascertaining connection targets via rendezvous service

Fig. 13 illustrates an embodiment of a method implemented by the PRT framework 12 in connecting to a connection target through the rendezvous service 48.

According to the method of fig. 13, the PRT framework 12 resolves the specification of the at least one connection target in the connection rule into at least one corresponding node connection handle (fig. 13, block 182). In this process, the PRT framework 12 authenticates the user to the rendezvous service 48 with the token received from the security service 44. After the user has been authenticated, the PRT framework 12 establishes a session with the rendezvous service 48. The PRT framework 12 then transmits a request to the rendezvous service 48 for a respective connection handle for each connection target corresponding to the connection target designation in the connection rule.

The rendezvous service 48 identifies one or more connection targets specified by the corresponding connection target. If the connection rule specifies a particular connection target with a target identifier, then the rendezvous service 48 queries the presence database for the status and capability requirements of the connection target corresponding to the specified target identifier. If the connection rule specifies a connection target in a set of one or more attribute values, the rendezvous service 48 queries the presence database for the status and capability requirements associated with a connection target having an attribute value matching the specified attribute value. The rendezvous service 48 compares the capabilities of the user with the capability requirements associated with each of the identified connection targets. The rendezvous service 48 communicates to the PRT framework 12 a respective connection handle for each of the identified connection targets whose capability requirements are satisfied.

Upon receiving the query results from the rendezvous service 48, the PRT framework 12 determines the applicability of any connection conditions contained in the connection rules. For example, the connection rules may include conditions on the manner in which query results returned by the rendezvous service 48 should be handled. For example, the connection condition may specify that the PRT framework 12 should present the results to the user in a graphical interface so that the user can select one or more of the matching connection targets with which he or she wants to communicate. In another example, the connection condition may specify that the PRT framework 12 should automatically connect to a connection target whose connection handle is returned in the query result. The connection rules may also include connection conditions that describe predicates about the current connection environment that should be satisfied prior to making the network connection. Such exemplary conditions include connection conditions that limit when network connections are permitted, conditions that limit the establishment of network connections to network nodes that meet specified resource requirements, conditions that limit the establishment of network connections to network nodes that meet specified node configuration requirements, conditions that limit the establishment of network connections to network nodes that meet specified network node location requirements, and conditions that limit the establishment of network connections to times that meet specified connection target availability requirements.

Referring back to fig. 13, after resolving the designation of the connection target into at least one corresponding connection handle (fig. 13, block 182), the PRT framework 12 initiates delivery of at least one real-time data flow via at least one network connection with the network node associated with the connection handle in accordance with any applicable connection conditions specified in the connection rule (fig. 13, block 184). As explained above, the connections between the PRT framework 12 and other network nodes may be peer-to-peer connections or server-arbitrated connections. With respect to peer-to-peer connections, the connection target network node and the PRT framework 12 typically authenticate each other and then establish a link via which to communicate at least one real-time data stream to or from the connection target. The link is typically unidirectional and is requested by the transmitting party and accepted or rejected by the receiving party.

Ascertaining connection targets via a regional service

Fig. 14 illustrates an embodiment of the method implemented by the PRT framework 12 in connecting to a virtual area through the area service 46.

According to the method of fig. 14, the PRT framework 12 determines an instance of a virtual area based on the designation of the virtual area in the connection rule (fig. 14, block 186). In this process, the PRT framework 12 authenticates the user to the regional service 46 via the token that the user receives from the security service 44. After the user has been authenticated, the PRT framework 12 establishes a session with the area service 46. The PRT framework 12 then transmits a request to the zone service 46 for an instance to connect to the virtual zone specified in the connection rule.

The zone service 46 determines an instance of the virtual zone corresponding to the designation of the virtual zone in the connection rule. This process typically depends on the manner in which the virtual area is specified (e.g., by a schema identifier, a virtual area instance identifier, or by one or more query attributes).

If the connection rule specifies a virtual area by reference to a pattern identifier, area service 46 retrieves the pattern corresponding to the pattern identifier and creates an instance of the retrieved pattern with the associated virtual area instance identifier. If the user's capabilities meet the capability requirements associated with the virtual area instance, the area service 46 returns configuration data to the PRT framework 12. The configuration data typically includes a definition of the virtual area instance and a registration of the object currently in the virtual area instance.

If the connection rule specifies a virtual area by reference to an instance identifier, area service 46 determines the status of the identified virtual area instance. If the virtual area instance is available (e.g., currently running) and the user's capabilities satisfy the capability requirements associated with the virtual area instance, the area service 46 returns configuration data to the PRT framework 12, where the configuration data typically includes the definition of the virtual area instance and the registration of the object currently in the virtual area instance. If the virtual area instance is not available (e.g., the virtual area instance is not currently running or cannot accept any new communicants) but the user's capabilities satisfy the capability requirements associated with the virtual area instance, the zone service 46 may create a new instance of the virtual area based on the schema of the original instance and return configuration data to the PRT framework 12, where the configuration data includes a definition of the new virtual area instance and a registration of the object currently in the new virtual area instance.

If the connection rule specifies a virtual area with a set of one or more query attribute values, area service 46 transmits a request to interaction service 50 to query all virtual area instances in the interaction database that have attributes that match the specified attribute values. Region service 46 determines the status and capability requirements associated with the virtual region instance identified in the query results returned by interaction service 50. The zone service 46 communicates to the PRT framework 12 a respective identifier for each of the identified virtual zones for which the capability requirements are satisfied. Upon receiving the query results from the regional service 46, the PRT framework 12 determines the applicability of any connection conditions contained in the connection rules. For example, the connection rules may include conditions regarding the manner in which query results returned by the regional service 46 should be handled. For example, the connection condition may specify that the PRT framework 12 should present the results to the user in a graphical interface so that the user can select which of the matching virtual area instances he or she wants to enter. In another example, the connection condition may specify that the PRT framework 12 should automatically enter one of the virtual area instances that is based on a specified criteria (e.g., most frequently visited).

Referring back to fig. 14, after determining a virtual area instance (fig. 14, block 186), the PRT framework 12 ascertains one or more network nodes associated with the instance of the virtual area (fig. 14, block 188). In this process, the PRT framework 12 reads the object registration received from the zone service 46 to determine the network node associated with the object currently in the virtual zone instance. In some embodiments, the PRT framework 12 also ascertains, via the rendezvous service 48, the network nodes associated with the connection targets specified in the connection rules as described in the previous section.

The PRT framework 12 initiates the transfer of at least one real-time data stream via at least one network connection with at least one of the ascertained network nodes in a context defined by an instance of the virtual area (fig. 14, block 190). The connection between the PRT framework 12 and other network nodes may be a peer-to-peer connection or a server-arbitrated connection. With respect to peer-to-peer connections, the connection target network node and the PRT framework 12 typically authenticate each other and then establish a link via which to communicate at least one real-time data stream to or from the connection target. The link is typically unidirectional and is requested by the transmitting party and accepted or rejected by the receiving party.

As explained above, the initiation of each real-time data stream connection is in accordance with any applicable connection conditions specified in the connection rules. For example, the connection rule may include a connection condition that describes a predicate on the current connection context that should be satisfied before the network connection is made. Such exemplary conditions include connection conditions that limit when network connections are permitted, conditions that limit the establishment of network connections to network nodes that meet specified resource requirements, conditions that limit the establishment of network connections to network nodes that meet specified node configuration requirements, conditions that limit the establishment of network connections to network nodes that meet specified network node location requirements, and conditions that limit the establishment of network connections to times that meet specified connection target availability requirements.

B. Exemplary System architecture embodiments

1. Introduction to the design reside in

Fig. 15 illustrates an embodiment of the operating environment of fig. 10 including an embodiment 200 of computer system 120 and an embodiment 202 of network node 140.

Network node 202 (node C) hosts regional service 46. The regional service 46 maintains global state information and the network node 202 acts as a data server for the network nodes 200, 138, 142. The global state information maintained by the regional service 46 is a current description 204 of the virtual region, a current registration 206 of objects in the virtual region, and a list 208 of any flow mixes currently being generated by the network node 202.

The object registration 206 generally includes, for each object in the virtual area, a respective object identifier (e.g., a tag that uniquely identifies the object), a connection handle (e.g., a URI such as an IP address) that enables establishment of a network connection with the network node associated with the object, and interface data that identifies the real-time data source and sink associated with the object (e.g., the source and sink of the network node associated with the object). The object registration 206 also typically includes one or more optional role identifiers for each object that can be explicitly assigned to the object by the correspondent or the regional service 46, or can be inferred from other attributes of the object. In some embodiments, the object registration 206 also includes the current location of each object in the virtual area, as determined by the area service 46 from an analysis of real-time motion data streams received from network nodes associated with the objects in the virtual area. In this regard, the zone service 46 receives real-time motion data streams from network nodes associated with objects in the virtual zone and tracks avatars and other objects of communicants entering, leaving, and walking around the virtual zone based on the motion data. The regional service 46 updates the object registration 206 based on the current location of the tracked object.

Computer system 200 includes an embodiment 210 of PRT frame 12. Among other functions, PRT framework 210 manages real-time data stream connections with other network nodes. In this process, the PRT framework 210 maintains a configuration data set that includes interface data 212, a zone list 214, and the location 216 of objects currently in the virtual area. For each object associated with the computer system 200, the interface data 212 includes a respective list of all sources and sinks of the real-time data stream type associated with the object. Zone list 214 is a registration of all zones in a virtual area currently occupied by an avatar associated with computer system 200. When a user first enters a virtual area, the PRT framework 210 typically initializes the current object location database 216 with location initialization information downloaded from the network node 202. Thereafter, the PRT framework 210 updates the current object location database 216 with the current location of the object in the virtual area determined from analysis of the real-time motion data streams received from, for example, the computer mouse 218 and one or more of the network nodes 138, 202, 142. The configuration data maintained by the PRT framework 210 also includes object registrations 206, a stream mix list 208, and copies 220, 222, 224 of the virtual area specification 204, respectively; these copies 220, and 222 are typically downloaded from regional service 46 and represent local caches of these data. In some embodiments, the object location 216 is incorporated into the object registry 220.

2. Operating system

Fig. 16 illustrates an embodiment 230 of the operating system 144.

The operating system 230 includes an OS API 234 and an OS kernel 236. The OSAPI 234 exposes functionality provided by the operating system 230, including file manager services 238, user interface services 240, and functionality provided by services of the OS kernel 236. File manager service 238 manages the storage and retrieval of files and metadata (e.g., file attributes) associated with computer data files and software application files. The user interface service 240 manages the windowing environment, including the presentation of dialog boxes and graphics on the display 132. In the illustrative environment shown in FIG. 16, the OS kernel 236 includes a program loader service 242, a device manager service 244, and a memory manager service 246. The program loader service 242 manages execution of the software application (e.g., loads at least one executable of the software application into memory, prepares the executable for execution, and executes the prepared executable). The device manager service 244 manages access to the hardware devices through device drivers. Memory manager service 246 manages access to computer system memory 124 and persistent storage 128.

PRT framework

a. Introduction to the design reside in

Fig. 16 also shows an embodiment 232 of the PRT frame 12.

The PRT framework 232 includes a PRT API250, a set of PRT non-kernel services, and a PRT kernel 252 that includes a set of PRT kernel services. The PRT API250 exposes all services provided by the PRT framework 232. The PRT uncore services include a connection object manager service 253, a zone connection service 254, a zone entry service 255, a target connection service 256, and an output presence service 260. The PRT non-kernel services 254-260 are also capable of invoking services of the PRT kernel 252 and the operating system 230 as needed in implementing their respective functionality. The PRT kernel services include login manager service 262, session manager service 264, stream exchange service 266, stream handler service 268, real-time scheduler service 270, visualization engine service 272, and bandwidth monitor service 274. PRT kernel services 262 and 274 can also invoke services of operating system 250 in performing their respective functions. Implementation of at least some of the services 254 and 274 involves integrating traditional operating system functions (e.g., process management functions, file management functions, memory management functions, storage management functions, device management functions, and network management functions) with real-time functions provided by the kernel (e.g., real-time scheduling functions, real-time connection functions, and real-time data flow handling functions). In this way, the real-time primitives and the legacy operating system primitives are exposed to the software application developer in a peer-to-peer relationship.

PRT kernel services

Each time the PRT framework 232 boots up (typically at the time of computer system 200 boot up), the login manager service 262 implements the process of logging into the network infrastructure services environment 43. In some embodiments, login manager service 262 performs the functions of logging into network infrastructure services environment 43 through security service 44 as described above in connection with FIG. 1. The login manager service 262 also typically handles the exit of the network infrastructure service environment 43.

Session manager service 264 manages sessions between computer system 120 and target network nodes in other network nodes. In response to invocation of the session manager service 264 (either through an interprocess communication mechanism or through a system call), the session manager service 264 negotiates a link with the target network node. In this process, the session manager service 264 typically authenticates the user to the target network node (e.g., based on the security token received from the security service 44), negotiates a streaming protocol, and negotiates a streaming encryption protocol. The transmission of messages between the session manager service 264 and the target network node may be performed according to various different messaging paradigms. In some embodiments, messages are exchanged asynchronously according to a publish/subscribe model, where messages are segmented into classes, subscribers express interest in one or more message classes, and publishers only deliver messages corresponding to the subscribed classes. In some embodiments, session manager service 264 may also provide a connection recovery mechanism for reestablishing abnormally broken connections.

The stream switching service 266 manages the switching of network connections according to the switching rules defined by the virtual area. Stream exchange service 266 handles the entry and exit of avatars and any other objects associated with computer system 200 into and out of the virtual area. The stream switching service 266 also automatically determines how to switch (e.g., route, connect, and disconnect) real-time data streams between the computer system 200 and the other network nodes 138, 202, 142. Stream switching service 266 makes these determinations based on the switching rules contained in the virtual area specification, the current locations of the avatars and other objects in the virtual area, and the real-time data stream types associated with the avatars and other objects in the virtual area. In these embodiments, the stream exchange service 266 also considers the upload and download bandwidth constraints of the computer system 200 (as determined by the bandwidth monitor service 274) and any of the other network nodes 138, 202, 142 in these determinations. In addition, the stream switching service 266 re-evaluates the current connection set in response to events (e.g., upload or download bandwidth failures, and requests to enter or exit the virtual area), either periodically, or both. As a result of the reevaluation of the current connection, the flow exchange service 266 may take any of the following actions, for example: requesting flow mixing from the network node 202, dropping one or more direct links with one or more of the other network nodes 138, 142, or forming one or more direct links with one or more of the other network nodes 138, 142.

The stream handler services 268 include respective stream handler services for processing each type of real-time data stream (e.g., a motion data stream, an audio data stream, a chat data stream, a file transfer data stream, and a video data stream) communicated between the computer system 200 and other network nodes. Real-time data stream processing typically includes applying a transformation to the real-time data stream. In some embodiments, when a real-time data stream is being processed in the context of a shared virtual area, the stream handler service 268 processes the real-time data stream according to a stream processing configuration defined by the virtual area specification. In some embodiments, one or more of the flow handler services 268 include a manager that assembles a set of flow processing objects into a directed graph according to a flow processing configuration defined by the virtual area specification.

Some embodiments include a chat stream handler service that provides an interface for outgoing text messages received from a local text input device (e.g., keyboard) of the computer system 200 and an interface for incoming chat streams received from other network nodes 138, 202, 142. The chat stream handler service processes text messages entered into the real-time chat stream by the correspondent through a text input device and passes the real-time chat stream to the device manager service 244, which converts the chat stream into a format that can be transmitted to the other network nodes 138, 202, 142 via the network 42. The chat stream handler service also processes the incoming text stream and passes the processed text stream to the device manager service 244, which converts the text stream into a signal that can be presented on the display monitor 132.

Some embodiments include an audio stream handler service that processes incoming audio signals received from other network nodes 138, 202, 142. The audio stream handler service passes the processed incoming audio signals to the device manager service 244, which converts the audio signals into a format that can be rendered by the speakers 278, 280 in the communicant's headphones 282. The audio stream handler service also processes outgoing audio signals generated by a microphone 284 in the headphones 282. The audio stream handler service passes the processed outgoing audio signals to the device manager service 244, which converts the audio signals into a format that can be transmitted to the other network nodes 138, 202, 142 via the network 42.

Some embodiments include a video stream handler service that processes incoming video signals received from other network nodes 138, 202, 142. The video stream handler service passes the processed incoming video signals to the device manager service 244, which converts the video signals into a format that can be presented by the display 132 and speakers 278, 280 in the communicator's headset 282. The video stream handler service also processes outgoing video signals generated, for example, by a local camera attached to the computer system 200. The video stream handler service passes the processed outgoing video signals to the device manager service 244, which converts the video signals into a format that can be transmitted to other network nodes 138, 202, 142 via the network 42.

Some embodiments include a motion data stream handler service that processes incoming motion data signals received from other network nodes 138, 202, 142. The motion data stream handler service passes the processed incoming motion data signals to the visualization engine service 272. The visualization engine service 272 updates the presentation of the shared virtual area communication session on the display 132 with the motion data signal. The motion data stream handler service also processes outgoing motion data signals generated by the user input device. The motion data stream handler service passes the processed outgoing motion data signals to the device manager service 244, which converts the motion data signals into a format that may be transmitted to other network nodes 138, 202, 142 via the network 42.

Some embodiments include a file delivery stream handler service that processes file signals received from other network nodes 138, 202, 142. The file delivery stream handler service passes the processed file signals to the memory manager service 246, which converts the file signals into a format that may be stored in the persistent storage memory 128. The filestream handler service also calls the memory manager service 246 to retrieve the computer data file from the persistent storage 128, process the computer data file into a stream of file signals, and pass the processed file signals to the device manager service 244, which converts the file signals into a format that can be transmitted to other network nodes 138, 202, 142 via the network 42.

The visualization engine service 272 presents a view of the virtual area and any objects in the virtual area on the display monitor 132. In this process, the visualization engine service 272 reads the virtual area specification data 224, the object registration 220, and the current object location database 216. In some embodiments, the visualization engine service 272 also reads the correspondent avatar database 248 that contains images needed to render the correspondent's avatar in the virtual area. Based on this information, the visualization engine service 272 generates a one-, two-, or three-dimensional representation (i.e., an image) of the virtual area and the objects within the virtual area from the viewpoint (position and orientation) of the communicant avatar within the virtual area. The visualization engine service 272 then passes the representation of the virtual region to the device manager service 244 of the operating system 230, which controls the rendering of the image of the virtual region on the display monitor 132. In some embodiments, the visualization engine service 272 determines the visibility of the communicant's avatar to limit the amount of data that must be exchanged, processed, and presented to that portion of the virtual area that is visible on the display monitor 132.

The user may control the view of the virtual area or the position of the avatar in the virtual area presented by communicating commands from an input device (e.g., computer mouse 218) to a user interface service 240 of the operating system 230. The visualization engine service 272 updates the view of the virtual area and the location of the object in the virtual area according to the updated location in the current object location database 216 and re-renders the updated version of the graphical representation of the virtual area for presentation on the display monitor 132. The visualization engine service 272 may update the rendered image periodically or only in response to movement of one or more objects in the virtual area.

The real-time scheduler service 270 manages the scheduling of tasks executed by other PRT kernel services (e.g., stream handler services) to achieve real-time performance. For example, in some embodiments, the real-time scheduler service 270 schedules tasks based at least in part on a comparison between processing performance and performance goals. In some embodiments, the real-time scheduler service 270 is implemented by a kernel mode or driver process that manages the execution of PRT framework tasks. In some embodiments, the real-time scheduler service 270 additionally manages the execution of operating system tasks (e.g., scheduling and memory management) and synchronization primitives (e.g., semaphores and signaling and messaging mechanisms) in a manner that provides deterministic execution and blocking times.

In some embodiments, the real-time scheduler service 270 multiplexes the resources of the computer system 200 between one or more executing software applications based on their respective resource requirements according to a priority-based or proportional-share-based scheduling model. In these embodiments, each resource is typically associated with a respective scheduler that controls the order in which tasks access the resource. Such exemplary schedulers include a processor scheduler that multiplexes access to one or more processors in processing unit 122, a memory scheduler that multiplexes bandwidth of persistent storage 128, and a network scheduler that multiplexes link bandwidth of network adapter 138. The global scheduler typically sets the policies of the individual schedulers.

In some embodiments, the real-time scheduler service 270 is implemented by a best effort real-time scheduling process that satisfies the real-time software application deadlines as best as possible but does not guarantee that the real-time software application will satisfy these deadlines. In some of these embodiments, the real-time scheduler service 270 serves as a user-mode process that takes high priority threads and attempts to guarantee a specified processing rate (e.g., a target frame rate) through one or more stream handler services 268. In some of these embodiments, the real-time scheduler service 270 monitors the performance of at least one processing task executed by one of the stream handler services 268 and schedules the processing tasks based at least in part on the measured performance with respect to fixed or adaptive performance targets. For example, the real-time scheduler service 270 may adaptively modify the configuration of the monitored stream handler service to meet a target frame rate of data delivered to a device driver (e.g., audio driver).

PRT off-kernel services

In the illustrative environment shown in fig. 16, the non-kernel services of the PRT framework 232 include a connection object manager service 253, a zone connection service 254, a zone entry service 255, a target connection service 256, and an output presence service 260. The connection object manager service 253 provides functionality for managing connection objects and instantiating connection objects in response to PRT API calls. The zone connection service 254 provides a function for connecting to a virtual zone. The zone entry service 255 provides a function for entering a virtual zone. The target connection service 256 provides a function for connecting to a connection target. The outgoing presence service 260 provides functionality for outputting presence information to other network nodes. Each of the PRT uncore services may typically be invoked by calls from any of software applications, other PRT framework services, and operating system services.

The functionality provided by the uncore services of the PRT framework 232 is described in the following sections.

i. Managing connected objects

The connection object manager service 253 provides functions for managing connection objects, including adding connection objects, modifying connection objects, and deleting connection objects. In this regard, the connection object manager service 253 responds to some connection object management-related software application calls to the PRT API250 by invoking the user interface service 240 of the operating system 230 to create dialogs and the like that allow a user (e.g., a correspondent or local system administrator) to add, modify, and delete connection objects and their respective associations with computer data files and software applications.

In some embodiments, the software application may include tools that allow a user to manage connection objects associated with computer data files (e.g., document files produced by a desktop publishing software application) or segments of the software application. The tools may include a command for associating a new connection object instance with the computer data file or software application, a command for modifying a connection object instance associated with the computer data file or software application, and a command for deleting a connection object instance associated with the computer data file or software application.

In response to selecting a command to associate a new connection object instance with a section of the computer data file or software application (e.g., a section corresponding to the user's current focus of attention), the software application makes a call to PRT API250 that invokes connection object manager service 253. The connection object manager service 253 instantiates a new connection object instance and associates the new connection object instance with the specified section of the computer data file or software application by creating a record in a connection object association database indexed by the identifier of the specified section and the identifier of the computer data file or software application. The connection object may be an instance of a default connection object associated with the calling software application or computer data file, or may be a specific connection object specified in the PRT API call. After instantiating the new connection object instance, the connection object manager service 253 invokes the user interface service 240 to create a dialog box that allows the user to specify the values of the attributes of the connection object instance.

In response to a command selecting to modify a connection object instance associated with a section of the computer data file or software application (e.g., a section corresponding to the user's current focus of attention), the software application makes a call to PRT API250 that invokes connection object manager service 253. The connection object manager service 253 queries a connection object association database that associates each segment of the computer data file or software application with at least one corresponding connection object instance. The query returns a connection object instance identifier. The connection object manager service 253 instantiates a connection object instance corresponding to the connection object identifier. After instantiating the connection object instance, the connection object manager service 253 invokes the user interface service 240 to create a dialog box that allows the user to add, modify, or delete values for the properties of the connection object instance.

In response to a command selecting to delete a connection object instance associated with a section of the computer data file or software application (e.g., a section corresponding to the user's current focus of attention), the software application makes a call to PRT API250 that invokes connection object manager service 253. The connection object manager service 253 queries a connection object association database that associates each segment of the computer data file or software application with at least one corresponding connection object instance. The query returns a connection object identifier corresponding to the connection object instance associated with the section of the computer data file or software application. The connection object manager service 253 invokes the user interface service 240 to create a dialog box confirming the user's intent to delete the identified connection object instance.

Handling PRT API calls for real-time connections based on location in a computer data file or software application

The connection object manager service 253 also typically handles initial PRT API calls from software applications and operating system services (e.g., the program loader service 242 and the file manager service 238) that contain computer data files or specified locations in the software application. In this process, the connection object manager service 253 queries a connection object association database that associates each segment of the computer data file or software application with at least one respective connection object instance. The query returns a connection object identifier corresponding to the connection object instance associated with the location in the computer data file or software application specified in the PRT API call. The connection object manager service 253 instantiates a connection object instance corresponding to the connection object identifier. After instantiating the connection object instance, the connection object manager service 253 invokes one or more of the other PRT services that establish a connection with a virtual area or connection target according to the attribute values of the instantiated connection object instance.

Connecting to a virtual area

Fig. 17 illustrates an embodiment of a method implemented by the zone connection service 254 of the PRT framework 232 in response to a PRT API call requesting a connection to a virtual zone.

According to the method of FIG. 17, the zone connection service 254 determines a designation of a virtual zone (FIG. 17, block 290). In some embodiments, the region connection service 254 is invoked by any one of a software application, an operating system service, and a PRT framework service with a PRT API call that includes a virtual region designation. For example, the connection object manager service 253 may invoke the region connection service 254 with a PRT API call that includes a virtual region designation that the connection object manager service 253 extracts from an instance of a connection object associated with a computer data file or software application.

The zone connection service 254 establishes a session with the network infrastructure service hosting the specified virtual zone (fig. 17, block 292). In this process, the regional connectivity service 254 invokes the session manager service 264 to establish a session with the regional service 46 in the manner described above. The zone connection service 254 then transmits a request to the zone service 46 to connect to the specified virtual zone. The zone service 46 determines an instance of the virtual zone specified in the request received from the zone connection service 254. As explained above, this process typically depends on the manner in which the virtual area is specified (e.g., by a schema identifier, a virtual area instance identifier, or by one or more query attributes). After determining the instance of the virtual area instance, the area service 46 determines whether the user capabilities satisfy the capability requirements associated with the virtual area instance. If the user capabilities meet the capability requirements, the zone service 46 transmits a message indicating the availability of status data (e.g., a list of objects currently in the virtual zone instance, and the names of the communicants associated with those objects) that describes the current status of the virtual zone instance.

The zone connectivity service 254 subscribes to status data (fig. 17, block 294). In response to the subscription request, the area service publishes the state data to a channel on the link between session manager service 264 and area service 46.

The zone connection service 254 invokes the user interface service 240 of the operating system 230 to present a human-perceptible view of the state data (FIG. 17, block 296). For example, the area connection service 254 may invoke the interface service 240 to present a representation of each communicant associated with the object currently within the area on the display 132. In some embodiments, the correspondent may be represented by an icon, thumbnail, or other graphic that is optionally labeled with the correspondent name. In some embodiments, the state data is presented in a graphical interface of the software application that triggers invocation of the zone connection service 254. In some embodiments, the status data is presented in an embodiment of a Heads Up Display (HUD) interface described in U.S. patent application No.61/042714 filed 4/2008.

Entering a virtual area

After a connection has been established with a virtual area instance, the software application that triggers invocation of the area connection service 254 may give the user the option of requesting entry into the virtual area instance or may automatically request entry into the virtual area instance on behalf of the user.

Fig. 18 illustrates an embodiment of a method implemented by the region entry service 255, the stream exchange service 266, and the stream handler service 268 of the PRT framework 232 in response to a PRT API call requesting entry into a virtual region.

According to the method of fig. 18, the region entry service 255 declares an intent to enter a virtual region to the network infrastructure service hosting the virtual region (fig. 18, block 298). In this process, the zone entry service 255 sends a message containing the declaration to the zone service 46. The message may be sent on a channel of an existing link with the regional service 46 or on a new link with the regional service established by the session manager 264. In response, the zone service 46 determines whether the user capabilities satisfy the capability requirements associated with the virtual zone instance. If the user's capabilities meet the capability requirements, then the regional service 46 returns configuration data to the regional entry service 255, which includes, according to the specification of the virtual region instance, a definition of the virtual region instance, a registration of the object currently in the virtual region instance, and a set of real-time data stream sources and sinks associated with the object in the virtual region.

Stream switching service 266 initiates delivery of at least one real-time data stream via at least one network connection with at least one real-time data stream source, wherein the at least one real-time data stream source is respectively associated with at least one object in a virtual area (fig. 18, block 300). In this process, stream switching service 266 ascertains one or more network nodes associated with an instance of the virtual area based on configuration data received from area service 46. Stream switching service 266 then initiates delivery of at least one real-time data stream via at least one network connection with at least one ascertained network node. The connection between the stream switching service 266 and other network nodes may be a peer-to-peer connection or a server-arbitrated connection. With respect to peer-to-peer connections, the connection target network node and the session manager service 264 typically authenticate each other and then establish a link via which to communicate at least one real-time data stream to or from the connection target. The link is typically unidirectional and is requested by the transmitting party and accepted or rejected by the receiving party.

In the illustrative embodiment, the flow handler service 268 processes the initiated real-time data flow according to at least one flow handling definition in the specification of the virtual area instance (fig. 18, block 302). In this process, one or more of the flow handler services 268 include a manager that assembles a set of flow processing objects into a directed graph according to a flow processing configuration defined by the virtual area specification.

Fig. 19 illustrates an embodiment of a method according to which an embodiment of a stream switching service 266 processes configuration data received from the area service 46 to determine a required set of real-time data stream connections to be established when a subscriber enters a virtual area or crosses a boundary between zones of the virtual area. As explained above, the configuration data includes a copy of the virtual area specification 204 (see fig. 15) and a copy of the updated object registration 206 (see fig. 15). In some embodiments, the configuration data additionally includes a stream mix list 208 (see fig. 15) that identifies the mix (or combination) of real-time data streams generated by the network nodes 138, 142 that the regional service 46 is currently producing.

Stream switching service 266 initializes local object registration 220 (see fig. 15) with a copy of object registration 206 received from regional service 46 (fig. 19, block 304). The stream switching service 266 also initializes the local stream mix list 222 (see fig. 15) with a copy of the stream mix list 208 received from the regional service 46 (fig. 19, block 306). The stream switching service 266 additionally initializes the local virtual area specification cache 224 (see fig. 15) with a copy of the virtual area specification 204 received from the area service 46 (fig. 19, block 308).

The stream switching service 266 builds the list 214 of occupied zones (see fig. 15) from the virtual area specification 244 and the position of the user avatar in the virtual area instance (fig. 19, block 310). In this process, the stream exchange service 266 retrieves the current location of the customization avatar in the virtual area instance from the current object location database 216 containing the coordinates of the avatar's current location in the virtual area instance. These coordinates are determined from a real-time motion data stream received from an input device, such as a computer mouse 218. The stream switching service 266 then compares the current location of the user avatar with the zone definitions in the virtual area specification 224. The stream switching service 266 compiles a list of occupied zones 214 from all zones within the virtual area specification that are consistent with the current location of the communicator's avatar. For example, in some embodiments, occupied zone list 214 includes all zones whose mesh contains the current location of the user avatar.

The stream switching service 266 determines a target set of real-time data stream types defined for the zones in the occupied zone list 214 and a target list of supported features that take into account the classification of the client (e.g., voice-only client to full-feature client) (fig. 19, block 312). The stream switching service 266 then determines the required set of real-time data stream data from the target set of real-time data stream types, the location of the object in the virtual region instance, and the switching rules defined in the virtual region specification 224 (fig. 19, block 314). Additional details regarding the process of determining a set of target real-time data stream types and the process of determining the set of real-time data stream data required are described in U.S. application nos. 11/923,629 and 11/923,634, both filed on 24/10/2007.

In some example embodiments, after the stream exchange service 266 has determined the set of real-time data stream data that enables the user to participate in the collaborative communication session with other network nodes in the shared virtual area instance (fig. 19, block 314), the stream exchange service 266 determines a real-time data stream connection that will result in the delivery of the needed data stream data to the computer system 200.

In some of these embodiments, stream exchange service 266 determines a real-time data stream handling topology that delivers the set of real-time data streams to computer system 200 based at least in part on the bandwidth capabilities of computer system 200. In the process, the stream switching service 266 determines the respective form in which each real-time data stream is received from the unmixed real-time data streams and from the stream mix derived from the combination of the real-time data streams. Stream switching service 266 also determines the network route over which each real-time stream is received from the direct peer-to-peer network routes and the network routes arbitrated by one or more other network nodes. After the flow handling topology has been determined, flow switching service 266 establishes real-time data flow connections between computer system 200 and other network nodes according to the determined flow handling topology.

Fig. 20 illustrates an embodiment of a method implemented by the stream switching service 266 in determining the topology of a real-time data stream connection that delivers the required data stream data to the computer system 200.

According to the method, the stream switching service 266 determines whether the computer system 200 has sufficient bandwidth to receive the required set of real-time data stream data 316 directly from other network nodes (fig. 20, block 318). In this process, other network nodes transmit link requests to the computer system 200. The link request indicates a respective bandwidth requirement for a respective set of real-time data streams required to transmit the computer system 200. The stream switching service 266 compares the overall bandwidth required to establish the required direct connection with the download bandwidth currently available to the computer system 200 as reported by the bandwidth monitor service 274 (see FIG. 16).

If the available bandwidth is at least equal to the overall required bandwidth, the stream switching service 266 establishes a direct connection with other network nodes that provide the required real-time data stream data (fig. 20, block 320). In this process, the session manager service 264 creates a socket (e.g., a TCP socket or a performance optimized dedicated real-time socket) between the computer system 200 and one or more other network nodes 138, 202, 142. For each real-time data stream type, the created sockets typically include one socket for carrying the real-time data stream and one socket for carrying control information (e.g., quality of service information) associated with the transmission and reception of the associated real-time data stream packets. The stream handler service 268 processes the real-time data stream, including encrypting it, recording it, and delivering the processed data stream to the visualization engine service 272, the operating system user interface service 240, and the operating system device manager service 244 as needed for presentation in a user interface and transmission over the network 42.

If the available bandwidth is less than the requested bandwidth (FIG. 20, block 318), the stream switching service 266 checks the stream mix list 222 (see FIG. 15) to determine if the regional service 46 is generating a stream mix that provides the requested real-time data stream data (FIG. 20, block 322). If the desired stream mix is available, the stream switching service 266 establishes a connection with the regional service 46 over which a copy of the desired real-time data stream mix is transmitted from the regional server 46 to the computer system 200 (FIG. 20, block 324). If the required stream mix is not available, the stream switching service 266 sends a stream mix request to the regional service 46 (FIG. 20, block 326). The regional service 46 generates the required stream mixes in response to the stream mix requests, if possible.

v. connecting to a connection target

Fig. 21 illustrates an embodiment of a method implemented by the target connection service 256 of the PRT framework 232 in response to a PRT API call requesting a connection to a connection target.

According to the method of fig. 21, the target connection service 256 determines a designation of at least one connection target (fig. 21, block 330). In some embodiments, the target connection service 256 is invoked by any of a software application, an operating system service, and a PRT framework service with a PRTAPI call that includes a connection target designation. For example, the connection object manager service 253 may invoke the target connection service 256 with a PRT API call that includes a connection target designation that the connection object manager service 253 extracted from an instance of a connection object associated with a computer data file or software application.

The target connection service 256 establishes a session with a network infrastructure service that manages the connection handle distribution of the network node (fig. 21, block 332). In this process, the regional target connection service 256 invokes the session manager service 264 to establish a session with the conferencing service 48 in the manner described above.

The target connection service 256 declares to the network infrastructure service an intent to connect to one or more connection targets corresponding to the connection target designation (fig. 21, block 334). The rendezvous service 48 identifies one or more connection targets specified by the corresponding connection target. If the connection rule specifies a particular connection target with a target identifier, then the rendezvous service 48 queries the presence database for the status and capability requirements of the connection target corresponding to the specified target identifier. If the connection rule specifies a connection target in a set of one or more attribute values, the rendezvous service 48 queries the presence database for the status and capability requirements associated with a connection target having an attribute value matching the specified attribute value. The rendezvous service 48 compares the capabilities of the user with the capability requirements associated with each of the identified connection targets. The rendezvous service 48 communicates to the connection service 256 a respective connection handle for each of the identified connection targets whose capability requirements are satisfied.

Upon receiving at least one respective network node connection handle from the network infrastructure service (fig. 21, block 336), the target connection service 256 invokes the flow switching service 266 to initiate delivery of the at least one real-time data flow via at least one network connection with the network node associated with the at least one respective network node connection handle (fig. 21, block 338). The connection between the stream switching service 266 and other network nodes may be a peer-to-peer connection or a server-arbitrated connection. With respect to peer-to-peer connections, the connection target network node and the session manager service 264 typically authenticate each other and then establish a link via which to communicate at least one real-time data stream to or from the connection target. The link is typically unidirectional and is requested by the transmitting party and accepted or rejected by the receiving party.

Output Presence

Figure 22 illustrates an embodiment of a method implemented by the output presence service 260 in response to a PRT API call requesting output of presence information to one or more connection targets, the presence information describing a current location of a user in at least one of a computer data file or a software application.

According to the method of fig. 22, the output presence service 260 determines a designation of at least one connection target (fig. 22, block 340). In some embodiments, output presence service 260 is invoked by any of a software application, an operating system service, and a PRT framework service with a PRT API call that includes a connection target designation and a definition of a user location. For example, the connection object manager service 253 may invoke the output presence service 260 with a PRT API call that includes a connection target designation that the connection object manager service 253 extracted from an instance of a connection object associated with a computer data file or software application.

The outgoing presence service 260 establishes a session with a network infrastructure service that manages the exchange of presence data between network nodes (fig. 22, block 342). In this process, the outgoing presence service 260 invokes the session manager service 264 to establish a session with the conference service 48 in the manner described above.

The output presence service 260 declares an intent to the network infrastructure service to output presence data describing the user's location to at least one network node corresponding to the connection target designation (fig. 21, block 334). In response to the declaration, the rendezvous service 48 identifies one or more connection targets specified by the corresponding connection targets. If the connection rule specifies a particular connection target with a target identifier, then the rendezvous service 48 queries the presence database for the status and capability requirements of the connection target corresponding to the specified target identifier. If the connection rule specifies a connection target in a set of one or more attribute values, the rendezvous service 48 queries the presence database for the status and capability requirements associated with a connection target having an attribute value matching the specified attribute value. The rendezvous service 48 compares the capabilities of the user with the capability requirements associated with each of the identified connection targets. The rendezvous service 48 communicates the presence data of the user to each of the identified connection targets whose capability requirements are satisfied.

d. Invoking PRT framework functionality

The functions of the PRT framework 232 are invoked through calls to the PRT API 250. These calls may be generated in a variety of different ways. In the illustrated embodiment, the call to the PRT API250 may be made by any one of a software application executing on the computer system 200 or a remote network node, an operating system executing on the computer system 200 or a remote network node, and a network infrastructure service.

In some embodiments, a software application developer designs a software application to call the PRT API250 to establish a real-time application environment for running the software application. In this regard, the software application may be designed to call the PRT API250 at one or more particular locations of the application (e.g., at startup or when particular functions of the software application are invoked), or each time the user crosses a boundary between sections of the software application. In these embodiments, the software application typically calls the PRT API250 with a call that includes a definition of the current section in at least one of the software application and the computer data file being processed by the software application.

In some of these embodiments, the software application is configured to invoke the PRT API250 with a call to establish a network connection with at least one connection target. For example, the software application may invoke the connection object manager service 253 with a specified location in a computer data file or software application. The connection object manager service 253 instantiates a connection object instance associated with a location in the computer data file or software application. The connection object typically includes a respective designation of a virtual area and a respective designation of one or more connection targets associated with the virtual area. The software application may present the user with an option to enter the virtual area or to connect to one or more of the connection targets. If the user chooses to enter a virtual area or connect to a connection target, the software application calls the PRT API250 with a PRT API call that initiates a transfer of at least one real-time data stream on the network connection with respect to the connection target based on the location in the virtual area instance. For example, in response to the PRT API call, the connection object manager service 253 invokes one or more other PRT services (e.g., area connection service 254 or target connection service 256) that establish a connection with a virtual area or connection target according to the attribute values of the instantiated connection object instance. One or more of the services of the operating system 230 and the PRT framework 232 process the real-time data streams into a format that can be presented as human-perceptible output (e.g., visual images on the display 132 or audio output through the speakers 278, 280).

In some embodiments, the PRT API250 is called in a method implemented by the computer operating system that calls the software application or opens the computer data file.

For example, the PRT API250 may be called by the file manager service 240 of the operating system 230 in the process of opening a computer data file. In some embodiments, this process includes identifying a software application associated based on its association with a file extension associated with the computer data file, and executing the associated software application to open the computer data file. In some of these embodiments, a PRT-aware file manager service of the operating system manages a file attribute database (e.g., in an extended file attribute that may be associated with a computer data file at a file system level of the operating system) that includes a PRT enable attribute that indicates whether the associated computer data file is configured with PRT features. In response to a request to open a computer data file, the file manager service 238 reads the file attribute database. If the PRT enabled attribute value indicates that the computer data file is configured with PRT features, the file system manager service calls the connection object manager service 253 with a call to the PRT API250 that includes an identifier of the computer data file. In some of these embodiments, the file attribute database additionally includes an optional definition of an initial location in the computer data file. In these embodiments, file manager service 238 passes the initial location definition to join object manager service 253 in the PRTAPI call. The connection object manager service 253 uses the initial location definition to determine a connection object associated with the computer data file.

The PRT API250 may also be called by the program loader service 242 of the operating system 230 in the process of creating an initial operating environment for the software application. In some embodiments, this process includes loading at least one executable of the software application into memory, preparing the executable, and executing the prepared executable. In some of these embodiments, the software application file may also be associated with a PRT-enabled attribute that can be read by a PRT-aware file manager service. In these embodiments, a user request to run the software application triggers the file system manager service to invoke the connection object manager service 253 with a call to the PRT API250 that includes an identifier of the computer data file and, optionally, an initial location definition. In some embodiments, the software application developer incorporates a PRT enable attribute or reference to a connection object in a header of a software application file (e.g., a header or fragment that describes how the software application should be loaded into memory by a program loader service of the operating system). In some of these embodiments, the software application header additionally includes an optional definition of an initial position in the software application. In these embodiments, a command line in the software application header may instruct the program loader service to call the connection object manager service 253 with a call to the PRT API250 that includes an identifier of the computer data file and optionally an initial location definition. If the software application header stores a reference to a connection object indexed in the connection object database, a command line in the software application header may instruct the program loader service to call the connection object manager service 253 with a call containing the connection object reference.

Exemplary applications

The pervasive real-time framework 12 supports the development of a variety of real-time software applications that can take advantage of a new operating environment paradigm in which real-time connections between network nodes are pervasive. In a first embodiment, a personal information manager software application (e.g., Microsoft WindowsOutlookSoftware applications) (originally or through plug-in modules or macros) are designed to take advantage of the functionality provided by the PRT framework 12. Herein, the inventionIn an embodiment, the personal information manager software application includes an email task, a calendar task, and a contact management task. Each task is defined as a separate section and is associated with a corresponding connection object. The segment corresponding to the email task may, for example, be associated with a connection object that specifies that, when reading an electronic message sent to or from any contact in a specified workgroup, the PRT framework 12 should connect to a specified virtual area associated with the workgroup and output presence information to contacts in the workgroup that are not currently in the virtual area.

In this first embodiment, when the user's attention is focused on an email message in an email task, the personal information manager software application calls the connection object manager service 253 with a PRT API call that includes location definitions corresponding to the email task and a list of senders and recipients of the email message. The connection object manager service 253 retrieves a connection object associated with the email function based on the location definition. The connection object manager service 253 invokes the zone connection service 254, which attempts to connect to the virtual zone specified in the connection object in accordance with the user's preferences and the capability requirements of the virtual zone. The connection object manager service 253 also invokes a target connection service 256 that attempts to output presence information to contacts in the workgroup that are not currently in the virtual area in accordance with the user's preferences and the capability requirements of the virtual area. In this way, the email task of the personal information manager software application may utilize the functionality of the PRT framework 12 to provide the user with real-time connectivity to the relevant contacts based on the user's focus on the email task. Other tasks of the personal information manager software application may also be designed to take advantage of the functionality of the PRT framework 12. In a second embodiment, a web browser software application (e.g., Microsoft WindowsInternetExplorerSoftware application, FirefoxSoftware application, and SafariSoftware applications) (originally or through plug-in modules or macros) are designed to take advantage of the functionality provided by the PRT framework 12. In this embodiment, the web browser software application includes a tabbed document interface that allows a user to switch between different web pages without switching the top-level window. One or more web pages (e.g., default primary web pages and consumer service web pages) are associated with respective connection objects. The section corresponding to the primary webpage may, for example, be associated with a connection object that specifies that the PRT frame 12 should connect to a specified virtual area associated with the user (e.g., the user's personal virtual area or a virtual area associated with the user's workgroup) when the primary webpage is viewed in a tab of a web browser software application. The section corresponding to the consumer services web page may be associated with a connection object that specifies that when the consumer services web page is viewed in a tab of the web browser software application, the PRT frame 12 should connect to the specified consumer services virtual area and automatically connect to any communicants in the consumer services virtual area that have role attribute values corresponding to the consumer services representative.

In this second embodiment, when the user's attention is focused on the tab presenting the default primary web page, the web browser software application calls the connection object manager service 253 with a PRT API call that includes a definition of the location corresponding to the default primary page document. The connection object manager service 253 retrieves a connection object associated with the default main page document based on the location definition. The connection object manager service 253 invokes the zone connection service 254, which attempts to connect to the virtual zone specified in the connection object in accordance with the user's preferences and the capability requirements of the virtual zone.

When the user's attention is focused on the tab presenting the consumer services web page, the web browser software application calls the connection object manager service 253 with a PRT API call that includes a definition of the location corresponding to the consumer services page document. The connection object manager service 253 retrieves a connection object associated with the default main page document based on the location definition. The connection object manager service 253 invokes the zone connection service 254, which attempts to connect to the virtual zone specified in the connection object in accordance with the user's preferences and the capability requirements of the virtual zone. If anyone among the current occupants of a virtual area has a consumer service representative role attribute value, the connection service 254 invokes a zone entry service 255 that attempts to enter the virtual area in accordance with the user's preferences and the capabilities requirements of the virtual area. If the zone entry attempt is successful, the stream exchange service 266 automatically initiates a connection with a network node associated with an object (e.g., avatar) in the virtual zone.

In a third embodiment, a spreadsheet software application (e.g., Microsoft Windows @) ExcelSoftware applications) (originally or through plug-in modules or macros) are designed to take advantage of the functionality provided by the PRT framework 12. In this embodiment, the spreadsheet software application includes a tabbed worksheet interface that allows a user to switch between different worksheets without switching the top-level window. One or more worksheets (e.g., stock analysis worksheets) are associated with the respective connection objects. The section corresponding to the stock analysis worksheet may, for example, be associated with a connection object that specifies that when the stock analysis worksheet is viewed in a tab of the spreadsheet software application, the PRT framework 12 should connect to a specified virtual area associated with the user (e.g., a virtual area associated with a specified type of financial analysis) and to a specified data source connection target (e.g., an online stock quote service) to request specific data or data types (e.g., for a particular data or data type)Real-time stock quote information for a set of stocks listed on the stock analysis worksheet).

In this third embodiment, when the user's attention is focused on the tab presenting the stock analysis worksheet, the spreadsheet software application calls the connection object manager service 253 with a PRT API call that includes a definition of the location corresponding to the stock analysis worksheet. The connection object manager service 253 retrieves a connection object associated with the stock analysis data table based on the location definition. The connection object manager service 253 invokes the zone connection service 254, which attempts to connect to the virtual zone specified in the connection object in accordance with the user's preferences and the capability requirements of the virtual zone. The connection object manager service 253 also invokes a target connection service 256, which attempts to connect to a specified data source and retrieve the requested data or data type. If the connection attempt is successful, the stream handler service 268 may be configured to assemble a directed graph of real-time processing objects to perform transformations on the real-time stock quote information according to real-time processing specifications, which may be provided by the stock analysis worksheet or by the virtual region.

Conclusion VI

The embodiments described herein provide a high-level functionality for a pervasive real-time framework that supports execution of real-time software applications that significantly reduces the effort and time required to develop real-time software applications in a new operating environment paradigm in which real-time connections between network nodes are pervasive. The pervasive real-time framework handles the complex tasks of connecting to communicants, virtual areas, and other network resources, and exchanging these connections in response to user input, and thereby enables software application developers to concentrate on developing advanced real-time software application functionality.

Other embodiments are within the scope of the following claims.

Claims (67)

1. A computer-implemented method, comprising:

determining an instance (56) of a virtual area based on a designation of the virtual area in a connection rule (32) associated with at least one of a software application (28) and a computer data file (30);

ascertaining one or more network nodes (16-26) associated with the instance (56) of the virtual area; and

initiating transfer of at least one real-time data stream via at least one network connection with at least one of the network nodes (16-26) in a context defined by the instance (56) of the virtual area.

2. The method of claim 1, wherein said determining, said ascertaining, and said initiating are performed on a computer (60) in a computer operating system implemented method that invokes said software application (28).

3. The method of claim 2, further comprising instantiating one or more objects in the computer operating system implemented method, the objects including one or more methods that perform operations that comprise executing the software application (28).

4. The method of claim 3, wherein executing the software application comprises loading at least one executable of the software application (28) into memory, preparing the executable, and executing the prepared executable.

5. The method of claim 1, wherein said determining, said ascertaining, and said initiating are performed on a computer (60) in a computer operating system-implemented method of opening the computer data file (30).

6. The method of claim 5, further comprising instantiating one or more objects in the computer operating system implemented method, the objects including one or more methods that perform operations that comprise opening the computer data file (30) with an associated software application (28).

7. The method of claim 6, wherein the opening comprises identifying the associated software application (28) based on an association of the software application (28) with a file extension associated with the computer data file (30), and executing the associated software application (28) to open the computer data file (30).

8. The method of claim 1, wherein the determining, the ascertaining, and the initiating are performed on a computer in response to at least one call to an Application Programming Interface (API) through an implementation of the API.

9. The method of claim 8, further comprising instantiating one or more objects in response to the at least one call to the API, the objects comprising performing one or more methods comprising the determining, the ascertaining, and the initiating operation.

10. The method of claim 1, wherein the determining comprises invoking one or more connection objects associated with at least one of the software application (28) and the computer data file (30), and each of the connection objects comprises a respective set of one or more connection rules.

11. The method of claim 10, wherein the invoking comprises querying a database (106) that includes one or more records that each associate at least one of the software application (28) and the computer data file (30) with at least one of the connection objects.

12. The method of claim 11, wherein the querying comprises querying the database (106) with a query that includes an identifier of at least one of the software application (28) and the computer data file (30).

13. The method of claim 12, wherein the query further includes a definition of a location of at least one of the software application (28) and the computer data file (30).

14. The method of claim 1, wherein the determining comprises querying a database that indexes the connection rules (32) by location with respect to at least one of the software application (28) and the computer data file (30).

15. The method of claim 1, wherein the connection rule (32) is associated with a definition of at least one respective section in an interaction space defined with respect to at least one of the software application (28) and the computer data file (30), and the initiating comprises initiating the transfer of one or more real-time data streams with respect to one or more of the network nodes in accordance with the connection rule (32) and a specified location in the interaction space.

16. The method of claim 15, further comprising receiving a user definition of a current focus of the interaction space, wherein

The determining comprises determining the connection rule (32) based at least in part on the current focus with respect to the at least one section in the interaction space,

said ascertaining comprises ascertaining one or more of said network nodes (16-26) associated with said current focus, and

the initiating includes initiating respective transfers of real-time data flows in relation to one or more of the network nodes (16-26) ascertained as being associated with the current focus in accordance with the one or more connection rules (32) determined to be associated with the current focus.

17. The method of claim 1, wherein the determining comprises passing the virtual area designation to a defined network infrastructure service (43) for resolving, at least in part, the virtual area designation to the virtual area instance (56).

18. The method as recited in claim 1, wherein the connection rule (32) additionally includes a connection target designation for one or more connection targets, and the ascertaining includes resolving the designation as a respective connection handle for at least one of the one or more network nodes (16-26).

19. The method of claim 18, wherein the resolving comprises passing the connection target designation to a network infrastructure service (43) that manages a connection handle distribution for the one or more network nodes.

20. The method of claim 18, wherein the connection target designation comprises at least one of information identifying a single communicant, information identifying a group of communicants, and a definition of a role associated with at least one communicant.

21. The method of claim 1, wherein the ascertaining comprises declaring an intent to enter the virtual area to a network infrastructure service (43) and receiving a list of network nodes associated with objects in the virtual area from the network infrastructure service (43).

22. The method of claim 1, wherein the connection rule (32) additionally includes one or more connection conditions, and the method further comprises establishing the at least one network connection in response to a respective determination that the one or more connection conditions are satisfied.

23. The method of claim 22, wherein the connection rules (32) include conditions that limit when the established connection is permitted.

24. The method of claim 22, wherein the connection rules (32) include conditions that limit the establishment to network nodes that meet specified resource requirements.

25. The method of claim 22, wherein the connection rules (32) include conditions that limit the establishment to network nodes that meet specified node configuration requirements.

26. The method of claim 22, wherein at least one of the connection rules (32) includes a condition that limits the establishment to network nodes that meet specified network node location requirements.

27. The method of claim 22, wherein at least one of the connection rules (32) includes a condition that limits the establishment to a time that satisfies a specified connection target availability requirement.

28. The method of claim 1, further comprising processing the real-time data stream into human perceptible output.

29. The method of claim 28, further comprising scheduling at least one task for execution in the process based at least in part on a performance measured for the process relative to a performance target.

30. The method of claim 1, wherein said determining, said ascertaining, and said initiating are performed on at least one computer (60).

31. At least one computer-readable medium (124, 128) storing computer-readable instructions that, when executed by a computer (60), cause the computer (60) to perform operations comprising:

determining an instance (56) of a virtual area based on a designation of the virtual area in a connection rule (32) associated with at least one of a software application (28) and a computer data file (30);

ascertaining one or more network nodes (16-26) associated with the instance (56) of the virtual area; and

initiating a transfer of at least one real-time data stream via at least one network connection with at least one of the network nodes in a context defined by the instance (56) of the virtual area.

32. An apparatus, comprising:

a computer readable memory (124, 128) storing computer readable instructions; and

a data processing unit (122) coupled to the memory to execute the instructions and perform operations based at least in part on the execution of the instructions, the operations comprising:

determining an instance (56) of a virtual area based on a designation of the virtual area in a connection rule (32) associated with at least one of a software application (28) and a computer data file (30),

ascertaining one or more network nodes (16-26) associated with the instance (56) of the virtual area, and

initiating a transfer of at least one real-time data stream via at least one network connection with at least one of the network nodes in a context defined by the instance (56) of the virtual area.

33. A computer-implemented method, comprising:

resolving a specification of at least one connection target in a connection rule (32) associated with at least one of a software application (28) and a computer data file (30) into at least one respective network node connection handle (182); and

initiating delivery (184) of at least one real-time data stream via at least one network connection with at least one network node respectively associated with the at least one respective network node connection handle.

34. The method of claim 33, wherein the parsing (182) and the initiating (184) are performed on a computer (60) in a computer operating system implemented method that invokes the software application (28).

35. The method of claim 33, wherein the parsing (182) and the initiating (184) are performed on a computer (60) in a computer operating system implemented method of opening the data file (30).

36. The method of claim 33, wherein the parsing (182) and the initiating (184) are performed on a computer in response to at least one call to an Application Programming Interface (API) through an implementation of the API.

37. The method of claim 36, further comprising instantiating one or more objects in response to the at least one call to the API, the objects including one or more methods that perform operations including the parsing (182) and the initiating (184).

38. The method of claim 33 wherein the initiating (184) is performed in a context defined by a virtual area associated with at least one of the software application (28) and the computer data file (30).

39. The method of claim 33, wherein the determining comprises querying a database comprising the connection rules (32) indexed by location with respect to at least one of the software application (28) and the computer data file (30).

40. The method of claim 33, wherein the connection rule (32) is associated with a definition of at least one respective section in an interaction space defined with respect to at least one of the software application (28) and the computer data file (30), and the initiating comprises initiating the transfer of one or more real-time data streams with respect to one or more of the network nodes in accordance with the connection rule (32) and a specified location in the interaction space.

41. The method of claim 33, wherein the resolving (182) comprises assigning the connection target to a network infrastructure service that manages a distribution of connection handles for the one or more network nodes, and receiving the at least one corresponding network node connection handle from the network infrastructure service.

42. The method of claim 33, wherein the connection target designation comprises at least one of information identifying a single communicant, information identifying a group of communicants, and a definition of a role associated with at least one communicant.

43. The method of claim 33, wherein the connection rule (32) additionally includes one or more connection conditions, and the method further comprises establishing the at least one network connection in response to a respective determination that the one or more connection conditions are satisfied.

44. At least one computer-readable medium (124, 128) storing computer-readable instructions that, when executed by a computer (60), cause the computer (60) to perform operations comprising:

resolving a specification of at least one connection target in a connection rule (32) associated with at least one of a software application (28) and a computer data file (30) into at least one respective network node connection handle (182); and

initiating delivery (184) of at least one real-time data stream via at least one network connection with at least one network node respectively associated with the at least one respective network node connection handle.

45. An apparatus, comprising:

a computer readable memory (124, 128) storing computer readable instructions; and

a data processing unit (122) coupled to the memory to execute the instructions and perform operations based at least in part on the execution of the instructions, the operations comprising:

resolving a specification of at least one connection target in a connection rule (32) associated with at least one of a software application (28) and a computer data file (30) into at least one corresponding network node connection handle (182), and

initiating delivery (184) of at least one real-time data stream via at least one network connection with at least one network node respectively associated with the at least one respective network node connection handle.

46. A computer-implemented method, comprising:

publishing (34) an Application Programming Interface (API) declaring a function to perform an operation comprising managing network connections of a real-time data stream based on connection rules (32), each connection rule (32) comprising at least one of a respective designation of a virtual area and a respective designation of one or more connection targets associated with the virtual area; and

managing (38), in response to the call to the API, real-time data stream connections with at least one network node based on a respective one of the connection rules (32) associated with at least one of a software application (28) and a computer data file (30) and at least one location in the virtual area.

47. The method of claim 46, wherein the managing (38) is performed in response to a call by the software application (28) to the API and is based on connection rules (32) associated with at least one of the software application (28) and computer data files (30) on which the software application (28) is operable.

48. The method of claim 46, wherein the managing (38) is performed in response to a call to the API by a loader operating system component that performs operations comprising loading at least one executable of a software application (28) into memory, preparing the executable for execution, and executing a prepared executable.

49. The method of claim 46 wherein the managing (38) is performed in response to a call to the API by a file manager operating system component that performs an operation that includes opening a computer data file (30).

50. The method of claim 46, wherein the managing (38) comprises initiating the delivery of at least one real-time data stream via the real-time data stream connection and processing the real-time data stream.

51. The method as recited in claim 50, wherein the managing (38) includes determining an instance (56) of the virtual area based on a designation of the virtual area in the respective connection rule (32), ascertaining one or more network nodes (16-26) associated with the virtual area instance, and initiating the communicating of the real-time data stream in a context defined by the instance (56) of the virtual area.

52. The method of claim 50, wherein the managing (38) comprises resolving the designation of at least one connection target in the respective connection rule (32) into at least one respective network node connection handle, and initiating the passing of the realtime data streams with respect to the network node associated with the at least one respective network node connection handle.

53. The method of claim 50, wherein the processing comprises processing the real-time data stream through a driver that provides an interface for a computer hardware device.

54. The method of claim 50, further comprising scheduling at least one task for execution in said process based at least in part on a performance measured for said process relative to a performance target.

55. The method of claim 50, wherein said publishing (34) and said managing (38) of said real-time data stream connections are performed on at least one computer (60).

56. At least one computer-readable medium (124, 128) storing computer-readable instructions that, when executed by a computer (60), cause the computer (60) to perform operations comprising:

publishing (34) an Application Programming Interface (API) declaring a function to perform an operation comprising managing network connections of a real-time data stream based on connection rules (32), each connection rule (32) comprising at least one of a respective designation of a virtual area and a respective designation of one or more connection targets associated with the virtual area; and

managing (38), in response to the call to the API, real-time data stream connections with at least one network node based on a respective one of the connection rules (32) associated with at least one of a software application (28) and a computer data file (30) and at least one location in the virtual area.

57. An apparatus, comprising:

a computer readable memory (124, 128) storing computer readable instructions; and

a data processing unit (122) coupled to the memory to execute the instructions and perform operations based at least in part on the execution of the instructions, the operations comprising:

publishing (34) an Application Programming Interface (API) declaring a function to perform an operation comprising managing network connections of a real-time data stream based on connection rules (32), each connection rule (32) comprising at least one of a respective designation of a virtual area and a respective designation of one or more connection targets associated with the virtual area; and

managing (38), in response to the call to the API, real-time data stream connections with at least one network node based on a respective one of the connection rules (32) associated with at least one of a software application (28) and a computer data file (30) and at least one location in the virtual area.

58. A computer-implemented method, comprising:

invoking an Application Programming Interface (API) with a call to establish a network connection with at least one connection target in a context defined by an instance (56) of a virtual area according to a connection rule (32), the connection rule (32) being associated with at least one of a software application (28) and a data file on which the software application (28) is operable and including a respective designation of the virtual area and a respective designation of one or more connection targets associated with the virtual area;

invoking the API with a call to initiate a transfer of at least one real-time data stream with respect to the connection target via the network connection based on a location in the virtual area instance; and

presenting a human perceptible output derived from the real-time data stream.

59. The method of claim 58, wherein the calling is performed by the software application (28).

60. At least one computer-readable medium (124, 128) storing computer-readable instructions that, when executed by a computer (60), cause the computer (60) to perform operations comprising:

invoking an Application Programming Interface (API) with a call to establish a network connection with at least one connection target in a context defined by an instance (56) of a virtual area according to a connection rule (32), the connection rule (32) being associated with at least one of a software application (28) and a data file on which the software application (28) is operable and including a respective designation of the virtual area and a respective designation of one or more connection targets associated with the virtual area;

invoking the API with a call to initiate a transfer of at least one real-time data stream with respect to the connection target via the network connection based on a location in the virtual area instance; and

presenting a human perceptible output derived from the real-time data stream.

61. An apparatus, comprising:

a computer readable memory (124, 128) storing computer readable instructions; and

a data processing unit (122) coupled to the memory to execute the instructions and perform operations based at least in part on the execution of the instructions, the operations comprising:

with establishment according to connection rules (32) in a context defined by an instance (56) of a virtual area

A call to a network connection with at least one connection target to invoke an Application Programming Interface (API), the connection rules (32) being associated with at least one of a software application (28) and a data file on which the software application (28) is operable and including respective designations of the virtual area and respective designations of one or more connection targets associated with the virtual area;

invoking the API with a call to initiate a transfer of at least one real-time data stream with respect to the connection target via the network connection based on a location in the virtual area instance; and

presenting a human perceptible output derived from the real-time data stream.

62. A computer-implemented method, comprising:

in response to an Application Programming Interface (API) call that includes a definition of a location in at least one of a software application (28) and a computer data file (30),

determining a connection rule (32) associated with the location definition and comprising a specification of a virtual area (290),

establishing a session (292) with a network infrastructure service that hosts an instance (56) of the virtual area and publishes state data describing a current state of the virtual area instance,

subscribing to the status data (294), and

a human-perceptible view of the state data is presented (296).

63. The method of claim 62, further comprising declaring an intent to enter the virtual area instance (56) to the network infrastructure service.

64. The method of claim 63, further comprising initiating delivery of at least one real-time data stream via at least one network connection with at least one real-time data stream source respectively associated with at least one object in the virtual area.

65. The method of claim 64, further comprising processing the at least one real-time data stream according to at least one stream handling definition in a specification of the virtual area instance (56).

66. A computer-implemented method, comprising:

in response to an Application Programming Interface (API) call that includes a definition of a location in at least one of a software application (28) and a computer data file (30),

determining a connection rule (32) associated with the location definition and comprising a specification of at least one connection target (330),

a session is established (332) with a network infrastructure service that manages the distribution of connection handles of network nodes,

declaring an intent to connect to one or more of the connection targets specified in the connection object to the network infrastructure service (334),

receiving at least one corresponding network node connection handle (336) from the network infrastructure service, and

initiating delivery (338) of at least one real-time data flow via at least one network connection with the network node associated with the at least one respective network node connection handle.

67. A computer-implemented method, comprising:

in response to an Application Programming Interface (API) call that includes a definition of a location in at least one of a software application (28) and a computer data file (30),

determining a connection rule (32) associated with the location definition and comprising a specification of at least one connection target (340),

establishing a session (342) with a network infrastructure service that manages the exchange of presence data between network nodes, and

declaring an intent (344) to the network infrastructure service to output presence data including the location definition to at least one of the network nodes respectively corresponding to the at least one connection target.