HK1162079A - System and method for context enhanced messaging - Google Patents

Description

System and method for context enhanced messaging

This application contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the patent and trademark office files or records, but otherwise reserves all copyright rights whatsoever.

Technical Field

The present invention relates to systems and methods for delivering media enhanced messages over a network, and more particularly to systems and methods for messages including media that involves user-defined context using, in part, data collected and stored by multiple devices on the network.

Background

When people use electronic devices, such as mobile phones and cable set-top boxes, a large amount of information is generated. Such information, e.g., location, applications used, social networks, physical and online locations visited (to name a few), can be used to deliver useful services and information to end users, and to provide advertisers and retailers with business opportunities. However, due to the lack of the way in which such information can be captured, a large portion of such information is actually discarded. For example, in the case of a mobile phone, information is typically not collected when the mobile phone is idle (i.e., not being used by the user). Nor will other information (such as the presence of others in the neighborhood, the time and frequency of messages for other users, and the activity of the user's social network) be effectively captured.

Disclosure of Invention

In one embodiment, the invention is a method. A message delivered from a message sender to at least one message recipient is received via a network. The message includes at least one content criterion. A query is formulated based on content criteria to search, via a network, for user profile data, social network data, spatial data, temporal data, and topical data, the data being available via the network and related to the content criteria and media files, to identify at least one media file relevant to the content criteria. The media file is inserted into the message and the message is sent to the recipient via the network.

In another embodiment, the invention is a method. A message delivered from a message sender to at least one message recipient is received via a network. The message includes at least one delivery criterion and at least one content criterion. When it is determined via the network that the delivery criteria for the message are satisfied, a query is formulated based on the content criteria to search via the network for user profile data, social network data, spatial data, temporal data, and topical data that are available via the network and that relate to the content criteria and the media files to identify at least one media file that is relevant to the content criteria. The media file is inserted into the message and the message is sent to the recipient via the network.

In another embodiment, the invention is a system comprising: a context-enhanced message receiving module that receives a message from a sender, wherein the requests each include at least one recipient, at least one delivery criteria, and at least one content criteria; a delivery criteria evaluation and tracking module that determines, for each message received by the context-enhanced message-receiving module, whether delivery criteria for the message are satisfied; a media retrieval module that uses content criteria for each message received by the context-enhanced message receiving module to formulate a query based on the content criteria such that user profile data, social network data, spatial data, temporal data, and topic data are searched via the network, the data being available via the network and relating to the content criteria and media files so as to identify at least one media file that pertains to the content criteria; a context-enhanced message update module that, for each message, inserts the media file identified by the media retrieval module into the message; and a context-enhanced message sending module that sends each message updated by the context-enhanced message updating module to at least one recipient of the message.

Drawings

The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of preferred embodiments as illustrated in the accompanying drawings in which reference characters refer to the same parts throughout the various views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.

FIG. 1 illustrates the relationship between Real World Entities (RWEs) and Information Objects (IOs) on one embodiment of a W4 communication network (W4 COMN).

FIG. 2 illustrates metadata defining relationships between RWEs and IOs on one embodiment of a W4 COMN.

FIG. 3 illustrates a conceptual model of one embodiment of the W4 COMN.

FIG. 4 illustrates the functional layers of one embodiment of the W4COMN architecture.

FIG. 5 illustrates an analysis component of one embodiment of the W4 engine as shown in FIG. 2.

FIG. 6 illustrates one embodiment of a W4 engine displaying different components within the sub-engines shown in FIG. 5.

FIG. 7 illustrates one embodiment of a data model showing how a W4COMN can store media files and associate those files to RWEs (e.g., people and locations) and IOs (e.g., topics and other types of metadata).

FIG. 8 illustrates one embodiment of a system capable of supporting context enhanced messaging between users known to the network.

FIG. 9 illustrates one embodiment of how a network (e.g., W4 COMN) containing temporal, spatial, and social networking and topical data for multiple users, devices, and media may be used for the process of making a media-enhanced message have a complex user context with delivery and content criteria.

FIG. 10 illustrates one embodiment of a context enhanced message engine capable of supporting the process illustrated in FIG. 9.

Detailed Description

The present invention is described below with reference to block diagrams and operational illustrations of methods and apparatus to select and present media on a particular topic. It will be understood that each block of the block diagrams or operational illustrations, and combinations of blocks in the block diagrams or operational illustrations, can be implemented by analog or digital hardware and computer program instructions.

These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, ASIC, or other programmable data processing apparatus to produce a machine, such that the functions/acts specified in the block diagrams or operational block or blocks are implemented, when executed via the computer or other programmable data processing apparatus.

In some alternative implementations, the functions/acts noted in the blocks may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved.

For the purposes of this disclosure, the term "server" should be understood to refer to a service point that provides processing, databases, and communication facilities. By way of example, and not limitation, the term "server" may refer to a single physical processor with associated communications and data storage and database facilities, or it may refer to a networked or clustered collection of: the processor and associated network and storage devices, as well as an operating system and one or more database systems and application software that support the services provided by the server.

For the purposes of this disclosure, the term "end user" or "user" should be understood to refer to a consumer of data provided by a data provider. By way of example, and not limitation, the term "end user" may refer to a person receiving data provided by a data provider via the internet in a browser session, or may refer to an automated software application that receives data and stores or processes the data.

For purposes of this disclosure, the terms "media" and "media content" should be understood to refer to binary data containing content that may be of interest to an end user. By way of example, and not limitation, the terms "media" and "media content" may refer to multimedia data (e.g., video data or audio data), or any other form of data capable of being transformed into a form recognizable by an end user. Further, such data may be encoded in any form currently known or developed in the future for a particular purpose. By way of example, and not limitation, data may be encrypted, compressed, and/or may contain embedded metadata.

For the purposes of this disclosure, a computer-readable medium stores computer data in a machine-readable form. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer.

For purposes of this disclosure, a module is a software, hardware, or firmware (or combination thereof) system, process, or function, or component thereof, that performs or facilitates the processes, features, and/or functions described herein (with or without human interaction or enhancement). The module may include sub-modules. Software components of the module may be stored on a computer readable medium. The modules may be integral to, or loaded and executed by, one or more servers. One or more modules may be grouped into an engine or application.

For purposes of this disclosure, an engine is a software, hardware, or firmware (or combination thereof) system, process, or function that performs or facilitates the processes, features, and/or functions described herein (with or without human interaction or enhancement).

Embodiments of the present invention utilize network-provided information that is capable of providing data collected and stored by multiple devices on a network. Such information may include, but is not limited to, temporal information, spatial information, and user information relating to a particular user or hardware device. User information may include, but is not limited to, user demographics, user preferences, user social networks, and user behavior. One example of such a network is the W4 communications network.

The "W4 communication network" or W4COMN provides information relating to "who, what, when and where" interact within the network. In one embodiment, the W4COMN is a collection of users, devices, and processes that facilitate synchronous and asynchronous communications between users and their agents, providing an instrumented network of sensors that provide data identification and collection in a real-world environment about any subject, location, user, or combination thereof.

In one embodiment, the W4COMN can handle routing/addressing, scheduling, filtering, prioritizing, replying to, forwarding, storing, deleting, privacy, processing, triggering, propagating changes, decoding, and concatenating of new messages. Further, these actions may be performed on any communication channel accessible to the W4 COMN.

In one embodiment, the W4COMN uses a data modeling strategy for creating profiles and any kind of user-defined data not only for users and locations, but also for any device on the network, with user-specified conditions. Using the available social, spatial, temporal, and logical data about a particular user, subject, or logical data object, each entity known to the W4COMN can be mapped and represented against all other known entities and data objects in order to create a minigraph for each entity and a global graph that correlates all known entities to one another. In one embodiment, these relationships between entities and data objects are stored in a global index of the W4 COMN.

In one embodiment, the W4COMN network relates to what may be referred to as a "real world entity" (hereinafter referred to as an RWE). RWE refers to, without limitation, a person, device, location, or other physical thing known to the W4 COMN. In one embodiment, each RWE known to the W4COMN is assigned a unique W4 identification number to identify the RWE within the W4 COMN.

RWEs can interact with the network directly or through proxies, which themselves can be RWEs. Examples of RWEs that interact directly with the W4COMN include any device, such as a sensor, motor, or other hardware component connected to the W4COMN for the purpose of receiving or transmitting data or control signals. An RWE may include all devices that may be used as network nodes or to generate, request, and/or consume data in a networked environment, or may be controlled by a network. These devices include any kind of "dumb" device (e.g., mobile phones, cable television set-top boxes, fax machines, telephones and Radio Frequency Identification (RFID) tags, sensors, etc.) that are intended to interact with a network.

Examples of RWEs that can use agents to interact with the W4COMN network include non-electronic entities including physical entities such as people, locations (e.g., states, cities, residences, buildings, airports, highways, etc.) and things (e.g., animals, pets, livestock, parks, physical objects, automobiles, airplanes, works of art, etc.) and intangible entities such as business entities, legal entities, groups of people, or sports teams. Additionally, "smart" devices (e.g., computing devices such as smartphones, smart set-top boxes, smart cars, laptops, personal computers, server computers, satellites, etc. that support communication with other devices or networks) can be considered RWEs that use agents to interact with a network, where software applications execute on the devices that serve as device agents.

In one embodiment, the W4COMN may cause associations between RWs to be determined and tracked. For example, a particular user (RWE) can be associated with any number and type of other RWEs (including others, mobile phones, smart credit cards, personal digital assistants, email and other communication service accounts, networked computers, smart appliances, set-top boxes and receivers for cable television and other media services, and any other networked devices). The association may be made explicitly by the user, such as when an RWE is installed into the W4 COMN.

An example of this is the establishment of a new mobile phone, cable service, or email account, where the user explicitly identifies an RWE (e.g., a user phone for mobile phone service, a user set-top box and/or location for cable service, or a username and password for online service) as being directly associated with the user. The explicit association may include the user identifying a particular relationship between the user and the RWE (e.g., this is my device, this is my home appliance, this person is my friend/father/son/etc., this device is shared between me and other users, etc.). RWEs can also be explicitly associated with users based on current circumstances. For example, a weather sensor on the W4COMN can be explicitly associated with a user based on information indicating that the user lives or is passing near the sensor location.

In one embodiment, the W4COMN network may additionally include what may be referred to as "information objects" (hereinafter referred to as IOs). An Information Object (IO) is a logical object that can store, maintain, or provide data for use by an RWE and/or a W4 COMN. In one embodiment, data within an IO may be modified by the behavior of an RWE. The IOs within the W4COMN may be provided with a unique W4 identification number to identify the IOs within the W4 COMN.

In one embodiment, the IOs include passive objects such as communication signals (e.g., digital and analog telephony signals, streaming media, and interprocess communications), email messages, transaction records, virtual cards, event records (e.g., time-identifying data files that may be combined with one or more RWEs such as users and locations, which may be further associated with known subject matter/activity/milestones such as concerts, gatherings, meetings, sporting events, etc.), telephone call records, calendar entries, web pages, database entries, electronic media objects (e.g., media files containing songs, videos, pictures, images, audio messages, telephone calls, etc.), electronic files, and associated metadata.

In one embodiment, IOs include any executing process or application that consumes or generates data, such as an email communication application (e.g., OUTLOOK by MICROSOFT or YAHOO! MAIL by YAHOO!), a calendar application, a word processing application, an image editing application, a media player application, a weather monitoring application, a browser application, and a web server application. These active IOs may or may not act as proxies for one or more RWEs. For example, voice communication software on a smartphone may be used as a proxy for both the smartphone and the smartphone owner.

In one embodiment, for each IO, there are at least three associated RWEs. The first is an RWE that owns or controls the IO, whether as a creator or a rights owner (e.g., an RWE that has editing rights or usage rights to the IO). The second is the RWEs to which the IOs refer, such as by containing information about or identifying the RWEs. The third is any RWE that accesses an IO for some purpose in order to obtain data from the IO.

In the context of the W4COMN, "available data" and "W4 data" refer to data present in IOs or data that can be collected from known IOs or RWEs (e.g., deployed sensors). In the context of the W4COMN, "sensor" refers to any source of W4 data, including PCs, telephones, portable PCs or other wireless devices, home appliances, automobiles, appliances, security scanners, video surveillance, RFID tags on clothing, products and locations, online data, or any other source of information about real world users/subjects/things (RWEs) or logic-based agents/processes/subjects/things (IOs).

FIG. 1 illustrates one embodiment of the relationship between RWEs and IOs on a W4 COMN. The user 102 is an RWE that possesses a unique network ID. The user 102 may be a person communicating with the network using proxy devices 104, 106, 108, 110 associated with the user 102, all of which are RWEs having unique network IDs. These agents may communicate directly with the W4COMN, or may communicate with the W4COMN using IOs such as applications executing on or by the agent devices.

In one embodiment, the proxy devices 104, 106, 108, 110 may be explicitly associated with the user 102. For example, one device 104 may be a smart phone connected to a network by a cellular service provider, and another device 106 may be a smart vehicle connected to the network. Other devices may be implicitly associated with the user 102.

For example, when two RWEs 104, 108 are co-located, one device 108 can be a "dumb" weather sensor located at a location that matches the current location of the user's mobile phone 104, and thus implicitly associated with the user 102. Another implicit association device 110 may be a sensor 110 for a physical location 112 known to the W4 COMN. The location 112 is known to be associated with the first user 102 either explicitly (through a user-specified relationship, e.g., this is my family, place of work, parent, etc.) or implicitly (as data from the sensor 110 at the location 112 indicates that the user 102 is often co-located with the RWE 112).

The user 102 may be directly associated with one or more people 140 and indirectly associated with more people 142, 144 through a directly associated chain. These associations may be explicit (e.g., the user 102 may identify the associated person 140 as his/her father, or may identify the person 140 as a member of the user's social network) or implicit (e.g., they share the same address). Tracking associations between people (and other RWEs) allows the concept of "intimacy," which can be defined as a measure of the degree of association between two people or RWEs, to be established. For example, each removal between RWEs can be considered a lower degree of affinity and assigned a lower affinity score. Affinity may be based solely on explicit social data, or may be extended to include all W4 data (including spatial and temporal data).

In one embodiment, as shown, each RWE 102, 104, 106, 108, 110, 112, 140, 142, 144 of the W4COMN can be associated with one or more IOs. Fig. 1 shows two IOs 122, 124 associated with the mobile phone device 104. One IO 122 may be a passive data object such as an event record used by scheduling/calendar software on a mobile phone, a contact IO used by an address book application, a history of transactions made using device 104, or a copy of a message sent from device 104. Another IO 124 may be an active software process or application that acts as a device proxy for the W4COMN by sending or receiving data via the W4 COMN. Voice communications software, scheduling/calendaring software, address book applications, or text communications applications are all examples of IOs that may communicate with other IOs and RWEs over a network. Additionally, IOs can relate to topics of interest to one or more RWEs, including but not limited to musicians, music genres, locations, and the like.

The IOs 122, 124 may be stored locally on the device 104 or remotely on some node or data store accessible to the W4COMN, such as a message server or mobile telephone service data center. The IO 126 associated with the vehicle 108 may be an electronic file containing the specification and/or current state of the vehicle 108, such as make, model, identification number, current location, current speed, current status, current owner, and the like. The IO 128 associated with the sensor 108 may identify a current state of the object monitored by the sensor 108, such as current weather or current traffic. The IO 130 associated with the mobile phone 110 may be information in a database identifying the amount of charge recently called or currently billed.

RWEs (e.g., people 102, 140, 142, 144), computing devices 104, 106, and locations 112 that can interact with the W4COMN only through proxies can have one or more IOs 132, 134, 146, 148, 150 directly associated with them that contain RWE-specific information for the associated RWE. For example, IOs associated with a person 132, 146, 148, 150 can include a user profile containing email addresses, phone numbers, physical addresses, user preferences, device identifications, and other RWEs associated with the user. In addition, the IOs can include past records of the user's interactions with other RWEs on the W4COMN (e.g., transaction records, copies of messages, a list that records the user's time and location combinations of the user's whereabouts in the past), a unique W4COMN identifier for the location and/or any relationship information (e.g., explicit user specification of the user's relationship to relatives, employers, co-workers, neighbors, service providers, etc.).

Another example of IOs associated with persons 132, 146, 148, 150 includes remote applications through which a person may communicate with the W4COMN, such as accounts using a web-based email service (e.g., Yahoo. The IO 134 for a location may contain information such as the exact coordinates of the location, driving directions to the location, classifications of the location (residential, commercial, public, non-public, etc.), information about services or products available at the location, a W4COMN identifier unique to the location, a business located at the location, a photograph of the location, etc.

In one embodiment, RWEs and IOs are correlated to identify relationships between them. RWEs and IOs can be associated using metadata. For example, if the IO is a music file, the metadata for the file may include data identifying the artist, song, etc., the album art, and the format of the music data. This metadata may be stored as part of the music file, or in one or more different IOs associated with the music file, or both. In addition, the W4 metadata may include the owner of the music file and the rights the owner has in the music file. As another example, if the IO is a photograph taken by an electronic camera, the photograph may include metadata (identifying when the photograph was taken, where the camera was when the photograph was taken, what camera was taking the photograph, who is associated with the camera (e.g., designated as the owner of the camera), if any, and who and what the object in/of the photograph, in addition to the raw image data from which the image may be created on the display). The W4COMN uses all available metadata to identify implicit and explicit associations between entities and data objects.

FIG. 2 illustrates one embodiment of metadata defining relationships between RWEs and IOs on a W4 COMN. In the illustrated embodiment, IO 202 includes object data 204 and five separate items of metadata 206, 208, 210, 212, 214. Some items of metadata 208, 210, 212 may contain information that relates only to the object data 204 and not to any other IO or RWE. Such as a creation date, text, or image of object data 204 to be associated with IO 202.

On the other hand, some items of metadata 206, 214 can identify relationships between the IO 202 and other RWEs and IOs. As shown, the IO 202 is associated with an RWE 220 through an item of metadata 206, the RWE 220 further being associated with two IOs 224, 226 and a second RWE 222 based on some information known to the W4 COMN. For example, a relationship can be described between an image (IO 202) containing metadata 206 identifying an electronic camera (first RWE 220) and a user (second RWE 224) that the system knows is the owner of the camera 220. Such ownership information may be determined, for example, from one or the other of the IOs 224, 226 associated with the camera 220.

FIG. 2 also shows metadata 214 associating an IO 202 with another IO 230. The IO 230 itself is associated with three other IOs 232, 234, 236, which further IOs 232, 234, 236 are associated with different RWEs 242, 244, 246. This portion of fig. 2 may, for example, describe the relationship between music files (IOs 202) that contain metadata 206 identifying a digital rights file (first IO 230) that defines the scope of usage rights associated with that music file 202. The other IOs 232, 234, 236 are other music files associated with usage rights and currently associated with a particular owner (RWEs 242, 244, 246).

FIG. 3 illustrates one embodiment of a conceptual model of the W4 COMN. The W4COMN 300 creates an instrumented communication infrastructure in the form of a global logical network cloud that is conceptually subdivided into networked clouds for each of the 4W (who, where, what, and when). In the who cloud 302 are all users that act as senders, recipients, data points or confirmation/authentication sources, and user agents in the form of processes, devices, agents, calendars, etc. of user programs.

In where cloud 304 are all physical locations, events, sensors, or other RWEs associated with a spatial reference point or location. When cloud 306 is composed of natural time events (i.e., events not associated with a particular location or person, such as days, times, seasons) as well as common user time events (holidays, anniversaries, elections, etc.) and user-defined time events (birthdays, intelligent timing programs).

What cloud 308 consists of all known data accessible to the W4 COMN-web or private, business or user, including, for example, environmental data such as weather and news, RWE generated data, IO data, user data, models, processes, and applications. Thus, conceptually, most of the data is contained in what cloud 308.

Certain entities, sensors, or data may potentially exist in multiple clouds at different times or simultaneously. Additionally, certain IOs and RWEs can be hybrids in that they combine elements from one or more clouds. These mixtures can be categorized as appropriate in order to determine associations between RWEs and IOs. For example, events consisting of location and time may be similarly classified within when cloud 306, what cloud 308, and/or where cloud 304.

In one embodiment, the W4 engine 310 is the intelligent hub for the W4COMN that makes all decisions in the W4 COMN. The W4 engine 310 controls all interactions between each layer of the W4COMN and is responsible for executing any approved user or application targets supported by the W4COMN operations or interoperating applications. In an embodiment, the W4COMN is an open platform with standardized published APIs for requesting (among other things) synchronization, disambiguation, user or topic addressing, access rights, prioritization, or other value-based ranking, intelligent scheduling, automation, and topic, social, spatial, or temporal alerts.

One function of the W4COMN is to collect data relating to all communications and interactions conducted via the W4COMN, which can include storing a copy of the IO and information identifying all RWEs, as well as other information relating to the IO (e.g., who, what, when, where information). Other data collected by the W4COMN can include information about the status of any particular RWE and IO at any particular time, such as location, operating status, monitored conditions (e.g., current weather conditions monitored for an RWE as a weather sensor, or its current location based on the cellular tower with which it is in contact for an RWE as a mobile phone), and current status.

The W4 engine 310 is also responsible for identifying RWEs and relationships between RWEs and IOs from data and communication streams passing through the W4 COMN. Identifying the functionality associated with an IO or an RWE involved in an IO, as well as the actions performed by other RWEs, can be referred to as entity extraction. Entity extraction can include simple actions such as identifying the sender and receiver of a particular IO, more complex analysis of the data collected by and/or available to the W4COMN, such as determining when and where a message lists an upcoming event and associating the event with the sender and receiver of the message based on the context of the message, or determining that a RWE is stuck in traffic congestion based on the correlation of the RWE's location to the status of a co-located traffic monitor.

It should be noted that when performing entity extraction from an IO, the IO may be an opaque object where only the W4 metadata related to the object is visible, while the internal data of the IO (i.e., the actual raw or object data contained within the object) is not visible, and thus the extraction of metadata is limited to metadata. Alternatively, if the internal data of an IO is visible, it can also be used for entity extraction, e.g., strings within an email are extracted and associated as RWEs for determining relationships between senders, users, subjects, or other RWEs or IOs affected by an object or process.

In the illustrated embodiment, the W4 engine 310 may be a computing device or a set of distributed computing devices, such as a general purpose Personal Computer (PC) or a specially designed server computer connected to the W4COMN via communications hardware and/or software. These computing devices may be a single device or a group of devices functioning together. A computing device may be equipped with any number of program modules and data files stored in local or remote mass storage devices and local memory (e.g., RAM) of the computing device. For example, as mentioned above, the computing device may include an operating system suitable for controlling the operation of a networked computer, such as the WINDOWS XP or WINDOWS SERVER operating systems from MICROSOFT CORPORATION.

Some RWEs may also be computing devices such as, but not limited to, smart phones, web-enabled appliances, PCs, laptops, and Personal Digital Assistants (PDAs). The computing device may be connected to one or more communication networks, such as the internet, a public switched telephone network, a cellular telephone network, a satellite communication network, a wired communication network (e.g., cable television), or a private network. The computing device may connect to any such network via a wired data connection or a wireless data connection (e.g., wi-fi, WiMAX (802.36), bluetooth, or cellular phone connection).

The local data structures (including the separate IOs) may be stored on a computer readable medium (not shown) connected to or part of any of the computing devices described herein including the W4 engine 310. For example, in one embodiment, the data backbone of the W4COMN (discussed below) includes a plurality of mass storage devices that maintain IOs, metadata, and data necessary to determine relationships between RWEs and IOs as described herein.

FIG. 4 illustrates one embodiment of the functional layers of the W4COMN architecture. At the lowest level, referred to as the sensor level 402, is a network 404 of actual devices, users, nodes, and other RWEs. Sensors include known technologies like web analytics, GPS, cell tower ping, usage logs, credit card transactions, online purchases, explicit and implicit user profiles obtained through behavioral targeting, search analytics, and other analytical models for optimizing a particular network application or function.

Data layer 406 stores and catalogs data generated by sensor layer 402. The data may be managed by a network of sensors 404 or a network infrastructure 406 created on top of an instrumented network of users, devices, agents, locations, processes, and sensors. Network infrastructure 408 is the core integrated network infrastructure that includes the hardware and software necessary to receive data sent from sensors, devices, etc. of network 404. It also includes the processing and storage capabilities needed to meaningfully sort and track the data created by the network 404.

The user profiling layer 410 performs the user profiling functions of the W4 COMN. This layer 410 may be further distributed between the network infrastructure 408 and user applications/processes 412 executing on the W4 engine or a separate user computing device. Personalization is supported over any single communication channel and mode or combination thereof, including email, IM, text (SMS, etc.), photo blogging, audio (e.g., phone calls), video (teleconferencing, live broadcasts), gaming, data trust processes, security, authentication, or any other W4COMN process call for available data.

In one embodiment, the user profile layer 410 is a logic-based layer above all sensors to which sensor data is sent in the most raw form to be mapped and placed into the W4COMN data backbone 420. The data (collected and simplified, correlated and deduplicated, synchronized and disambiguated) is then stored in one or a set of related database-enabled applications approved on the W4 COMN. Network-initiated actions and communications are based on the scope of the data backbone, and some of these actions make themselves a record somewhere in the backbone, such as checkout, while the completion of other actions (e.g., fraud detection, synchronization, disambiguation) has no impact on the profiles and models within the backbone.

Actions initiated from outside the network (e.g., RWEs such as users, locations, proxies, and processes) come from the application layer 414 of the W4 COMN. Some applications may be developed by the W4COMN operator and appear to be implemented as part of the communication infrastructure 408, such as email or calendar programs, because they operate so close to the sensor processing and user profile layer 410. The applications 412 also act as sensors because the applications 412, through their actions, generate data back to the data layer 406 via a data backbone that refers to any data created or available as a result of the execution of the applications.

In one embodiment, the application layer 414 may also provide a User Interface (UI) based on device, network, carrier, and user selection or based on security customization. Any UI may operate within the W4COMN as long as it is instrumented to provide data regarding user interactions or actions back to the network. In the case of a mobile device supporting the W4COMN, the UI may also be used to confirm or disambiguate real-time incomplete W4 data, as well as correlation, triangulation, and synchronization sensors for other nearby devices that support or do not support the W4 COMN.

At some point, devices that adequately support network effects allow the network to collect complete or nearly complete data (sufficient for profiling and tracking) for unsupported devices because of the periodic crossing and detection by supported devices in their real-world locations.

Hosted above or within the application layer 414 is a communication delivery network 416. The communications delivery network may be operated by the W4COMN operator, or be a separate third party carrier service. Data may be delivered via synchronous or asynchronous communications. In either case, the communication delivery network 414 will send or receive data on behalf of the request of a particular application or network infrastructure 408.

Communication delivery layer 418 also has elements that act as sensors, including W4 entity extraction from phone calls, emails, blogs, etc., as well as specific user commands within the delivery network context. For example, saying "save and prioritize this call" before the end of the call may trigger saving of a record of the previous session and analysis and increase of the weight prioritization decision in the personalization/user profile layer 410 for the W4 entities within the session.

FIG. 5 illustrates one embodiment of an analysis component of the W4 engine as shown in FIG. 3. As described above, the W4 engine is responsible for identifying RWEs and relationships between RWEs and IOs from data and communication streams passing through the W4 COMN.

In one embodiment, the W4 engine connects, interoperates, and instruments all network participants through a series of sub-engines that perform different operations in the entity extraction process. The attribute engine 504 tracks real-world ownership, control, publishing, or other conditional rights of any RWE in any IO. Whenever the W4 engine 502 detects a new IO, for example, by creating or transmitting a new message, a new transaction record, a new image file, etc., ownership is assigned to the IO. The attribute engine 504 creates this ownership information and also allows this information to be determined for each IO known to the W4 COMN.

The correlation engine 506 may have two capabilities: first, the associated RWEs and IOs and their relationships are identified (e.g., by creating a combined graph of any combination of RWEs and IOs and their attributes, relationships and reputations within a context or environment), and second, as a sensor analysis preprocessor for noted events from any internal or external source.

In one embodiment, the functionality of the correlation engine 506 to identify associated RWEs and IOs is accomplished by graphing the available data, for example, using one or more histograms. Histogram is a mapping technique that counts the number of observations that fall into various disjoint categories (i.e., bins). Relationships between RWEs, IOs and other known parameters can be identified by selecting each IO, RWE and other known parameters (e.g., time, date, location, etc.) as different bins and mapping the available data. A histogram of all RWEs and IOs is created from which graph-based correlations can be derived.

As a preprocessor, the correlation engine 506 monitors information provided by RWEs to determine if any conditions are identified that can trigger actions on the part of the W4 engine 502. For example, if a delivery condition has been associated with a message, when correlation engine 506 determines that the condition is satisfied, it may send appropriate trigger information to W4 engine 502 to trigger delivery of the message.

Note that engine 508 instruments all suitable network nodes, clouds, users, applications, or any combination thereof, and includes close interaction with correlation engine 506 and attribute engine 504.

FIG. 6 illustrates one embodiment of a W4 engine showing different components within the sub-engines described above with reference to FIG. 4. In one embodiment, the W4 engine 602 includes an attention engine 608, an attribute engine 604, and a relevance engine 606, with several sub-managers based on basic functionality.

Note that engine 608 includes a message ingest and generation manager 610 and a message delivery manager 612 that work closely with a message matching manager 614 and a real-time communication manager 616 to deliver and instrument all communications via the W4 COMN.

The attribute engine 604 operates within the user profile manager 618 and, along with all other modules, identifies, processes/verifies and represents ownership and entitlement information related to RWEs, IOs and combinations thereof.

The correlation engine 606 dumps data from both of its channels (sensors and processes) into the same data backbone 620 organized and controlled by the W4 analysis manager 622. The data backbone 620 includes aggregated and separately archived versions of data from all network operations including user logs 624, attention registration location logs 626, web index and environment logs 618, e-commerce and financial transaction information 630, search indexes and logs 632, sponsor content or conditions, ad copies and any other data used in W4COMN processes, IO or events. Because of the amount of data that the W4COMN will potentially store, the data backbone 620 includes many database servers and data stores in communication with the W4COMN to provide sufficient storage capacity.

The data collected by the W4COMN includes spatial data, temporal data, RWE interaction data, IO content data (e.g., media data), and user data including explicitly provided and inferred social and relationship data. Spatial data can be any data that identifies a location associated with an RWE. For example, the spatial data may include any passively collected location data, such as cell tower data, General Packet Radio Service (GPRS) data, Global Positioning Service (GPS) data, WI-FI data, private network data, IP address data, and data from other network access points, or actively collected location data, such as unknown data entered by a user.

Temporal data is time-based data (e.g., a timestamp) that relates to a particular time and/or event associated with a user and/or electronic device. For example, the time data may be passively collected time data (e.g., time data from a clock resident on the electronic device, or time data from a network clock), or the time data may be actively collected time data, such as time data input by a user of the electronic device (e.g., a user-maintained calendar).

Logical and IO data refer to data contained by and associated with an IO, such as time of creation, owner, associated RWE, time the IO was last accessed, subject or object of the IO (such as from a message content or "reply" or title bar, as some examples), and so forth. For example, the IO may relate to media data. The media data may include any data related to presentable media, such as audio data, visual data, and audiovisual data. The audio data may be data related to downloaded music, such as genre, artist, album, etc., and include data about ringtones, call backs, purchased media, playlists, and shared media, to name a few. The visual data may be data relating to images and/or text received by the electronic device (e.g., via the internet or other network). The visual data may be data relating to images and/or text sent from and/or captured at the electronic device.

The audiovisual data may be data associated with any video captured at, downloaded to, or otherwise associated with the electronic device. Media data includes media presented to a user via a network (e.g., using the internet), and includes data relating to text (e.g., search terms) entered and/or received by the user using the network, as well as interactions with the network media (e.g., click data (such as billboard clicks, bookmarks, click patterns, etc.)). Thus, the media data may include data relating to the user's RSS feeds, subscriptions, group memberships, gaming services, alerts, and the like.

The media data may include non-network activities such as image capture and/or video capture using an electronic device (e.g., a mobile phone). The image data may include user-added metadata, or other data associated with the image, such as, for example, in terms of a photograph, such as a location from which the photograph was taken, a direction from which the photograph was taken, the content of the photograph, and a time period, to name a few. The media data may be used, for example, to infer activity information or preference information, such as culture and/or purchase preference information.

Relationship data can include data relating to the relationship of an RWE or IO to another RWE or IO. For example, the relationship data may include user identity data such as gender, age, race, name, social security number, photographs, and other information associated with the user's identity. The user identity information may also include an email address, a login name, and a password. Relationship data can also include data identifying explicitly associated RWEs. For example, relationship data for a mobile phone may represent the user who owns the mobile phone and the company that provides services to the phone. As another example, the relationship data for a smart car may identify the owner, a credit card associated with the owner for electronic toll payments, those users allowed to drive the car, and a service stop for the car.

The relationship data may also include social networking data. Social network data includes data relating to any relationship explicitly defined by a user or other RWE, such as data relating to a user's friends, family, colleagues, business relationships, and so forth. The social network data may include, for example, data corresponding to an electronic address book maintained by the user. Relationship data, for example, may be correlated to location data in order to infer social network information, such as primary relationships (e.g., user-spouse, user-child, and user-parent relationships) or other relationships (e.g., user-friend, user-colleague, user-business contact relationships). Relationship data may also be used, for example, to infer activity information.

The interaction data may be any data, whether active or passive, associated with a user interaction with the electronic device. Examples of interaction data include interpersonal communication data, media data, relationship data, transaction data, and device interaction data, all of which are described further below. Table 1 below is a non-exhaustive list of examples that include electronic data.

Table 1-example of electronic data

The interaction data includes communication data between any RWEs that are communicated via the W4 COMN. For example, the communication data can be data associated with an incoming or outgoing Short Message Service (SMS) message, an email message, a voice call (e.g., a mobile phone call, an IP phone call), or other type of interpersonal communication involving an RWE. The communication data may be correlated with temporal data, for example, to infer information about communication frequencies, including focused communication patterns (which may represent user activity information).

The interaction data may also include transaction data. The transaction data may be any data associated with a commercial transaction engaged in or conducted at the mobile electronic device, such as vendor information, financial institution information (e.g., bank information), financial account information (e.g., credit card information), merchandise information and fee/price information, and purchase frequency information, to name a few. For example, transaction data may be utilized to infer activity and preference information. The transaction information may also be used to infer the types of devices and/or services that the user owns and/or may be interested in.

The interaction data can also include device or other RWE interaction data. These data include data generated by interactions between users and RWEs on the W4COMN and interactions between RWEs and the W4 COMN. RWE interaction data can be any data related to an RWE's interaction with an electronic device not included in any of the above categories, such as a custom pattern associated with electronic device data using other modules/applications, such as data regarding which applications are used on the electronic device and how often and when those applications are used. As described in more detail below, device interaction data may be correlated with other data in order to infer information about user activity and patterns associated therewith. Table 2 below is a non-exhaustive list of examples that include interaction data.

Table 2-example of interaction data

Context enhanced messaging

One of the most important functions of many communication and data networks is the ability of users to send messages to each other. Communication methods include, among others, email, instant messaging, photo communication, video conferencing, and telephone or cellular voice communication. However, these communication methods are naturally limited. It is often the case that more communications are desired by the network user than can be easily expressed verbally. Messages can be significantly enhanced by including very personalized media content, such as music and images that are well suited to the sender's message and the recipient's personal preferences.

Appropriate media may evoke a user's rudimentary memory and produce a picture, impression, sensation of time or place, person or group of persons, or even an abstract idea for the user to evoke invocation of certain kinds, business, and/or personal actions. This is simple enough to retrieve a playlist or video list of a single music artist. But a person may wish to capture a more complex context, for example a person may wish to create a playlist of songs representing the recipient's favorite music relating to an object, time, place or mood. Furthermore, communications may also be enhanced by fine tuning the delivery of messages to correspond to a particular time or time and date.

When a user creates an enhanced message, the user may be considered mentally having a particular context for the content or delivery of the message. In one embodiment, a message context may be defined as a set of criteria that describe or define one or more related ideas that are dominant to the message, sender and recipient in that context, and thus the set of criteria may be used to create a model for the message content and delivery options in that case. The criteria can be conceptually divided into four categories: who, what, when and where.

The "who" criteria are persons, devices or agents involved in the ideas embodied in the context. "who" may be a known person, such as a message sender, a message receiver, or a particular person known to the user. "who" may also be a list of particular people, such as a list of contacts stored on the user's PDA, a list of guests at a party, or people listed as friends on the user's social network profile. As an alternative, "who" may be a general description of the person concerned, e.g. a person of interest in surfing, a 40's individual woman who drives a motorcycle and likes yoga, a man who likes football and commutes with a bus, a person who passes a billboard more than three times a week, and/or a customer in a particular restaurant who also drives BMW.

The "what" criteria are concrete or abstract objects or topics that relate to ideas embodied in the context. "what" may be in the form of media of interest to the message sender or message recipient, such as photos, music, or video. What may be an object such as a car, a piece of jewelry, or other object of interest shared. What may be a genre of music or video, such as country or rock. "what" may be a topic targeted in the media, such as a love song or even a particular lyric phrase. As an alternative, "what" may be an emotion or atmosphere, such as happy, sad, energetic, or relaxed. As an indicator of topic relevance, the "what" criteria is that a human creates, pays attention to, and associates or triggers a determined infinite set of things.

The "when" criteria are temporal constructs, such as dates and times related to ideas embodied in the context. "what" may be the current date and time. "what" may also be a particular date and time in the past or future, or a range of dates and times in the past or future, such as a duration of time (such as two hours, four weeks, one year). "when" may be a conditional event when a specified condition or criteria is met. "when" may be an offset from a particular date, such as the past ten days, or an offset from a conditional event, ten days after the mortgage loan has expired. Alternatively, "when" may be an event on a calendar, such as a birthday, season or holiday, or an event of personal or social/social importance, such as the last time a favorite sports team won the champion.

The "where" criteria are physical locations that relate to ideas embodied in the context. Where may be the current location of the user. Where may be a particular location, e.g., country, state, city, neighborhood. "where" may be defined as the location of an event (e.g., a concert or some other event of reported value), or as a candidate may be defined as the user's personal location when the user listens to the event, e.g., where you were when you were listening to 9/11. As an alternative, "where" may be a general description of the place of interest, such as bruise or jazz club, or a conditional location depending on the satisfaction or resolution of a specified condition. For example, "where" may be the most popular club in real time for the age of 24 to 35 years, or "where" may be the research laboratory that ultimately cured the breast cancer.

In one embodiment, the context enhanced message includes four elements: recipient, message body, delivery criteria, and content criteria. The recipient is one or more real-world entities that are to receive the message. The recipients may be, but are not limited to, one or more specific persons, may be a set of email addresses, or may be a general description of a class of recipients, such as parents of children in my children's football team, or everyone in a person's social network. The recipient may be, or may include, a message sender, such as a message reminder for someone himself.

The message body is a text or media object representing a particular message. For example, if the context-enhanced message is an email, the message body will typically contain some text message of any length, such as "see me at 7 pm" or "happy birthday". The message body may include an audio file containing, for example, a voice message. The message body may include an image file containing, for example, a photograph of the sender, or a video message from the user.

The delivery criteria are conditions under which the message will be delivered to the recipient. These conditions may include "where" or spatial conditions, such as when the recipient is at a particular location, within a certain proximity of the location, the recipient is a certain person or object. These conditions may include "when" or a time condition, such as a particular time or date, or when a particular event occurs. These criteria may also include "who" or social criteria, such as music liked by one or more people in the sender's social network. These criteria may also utilize "what" or subject criteria, such as when the emotional manifestation of the recipient is sad, as determined, for example, by the content of the most recent message sent by the recipient.

The content standard describes the media files to be included with the message. These messages may contain criteria that are adapted to the context of the recipient or sender at the time the message is sent, the context of the subject of the message, or the context when the message is to be delivered. These criteria may include spatial criteria, such as different media files being included in the message depending on the physical location of the sender or recipient when sending or receiving the message. These criteria may include temporal criteria, for example, different media files may be included in the message depending on the time of day, day of the week, or whether it is the recipient's birthday. These criteria may include social criteria, for example, different media files are included in the message depending on the music the recipient likes. These criteria may include topical criteria, for example, different media files being included in the message depending on the mood of the recipient.

Content criteria may also include any combination of spatial, temporal, social, or topical criteria that do not relate to the context of the recipient or sender when sending or delivering the message. For example, the message may include criteria that describe the type of media file to be delivered. For example, the standard may define a playlist of top ten songs in the 70's of the 20 th century where the lyrics mentioned the city of new york.

The following examples of the invention illustrate the use of the invention within the W4 COMN. However, it is to be understood that the present invention may be implemented using any networked system, whether virtual or real, integrated or distributed throughout multiple parties, so long as it is capable of collecting, storing, accessing and/or processing user profile data, as well as temporal, spatial, topical and social data relating to users and their devices. Thus, for convenience, the term W4COMN is used herein to describe a system and/or network having features, functions and/or components described throughout.

FIG. 7 illustrates one embodiment of a data model showing how a W4COMN can store media files and associate those files to RWEs (e.g., people and locations) and IOs (e.g., topics and other types of metadata).

In the illustrated embodiment, the media is stored as media objects 710. A media object is a passive IO that refers to a media file containing audio content, visual content, or both. These media files may contain content such as songs, videos, pictures, images, audio messages, phone calls, and the like. These media objects themselves contain metadata 712. These data may be specific to the object data 710 and do not involve any other IO or RWE. At the simplest level, these metadata may relate to basic file attributes, such as creation date, text or image associated with the media file to which the IO relates.

Additionally, there is an existing database 720 that may reside inside or outside the network, which database 720 may provide a broad collection of descriptive metadata relating to particular songs, videos, and other types of media. For example, the Allmusic database (formerly the All Music Guide owned by the AllMedia Guide) provides metadata that includes:

● basic metadata such as name, genre, resource provenance, copyright information, product number.

● descriptive content such as style, tone, mood, theme, nationality, etc.

● related content such as similar artists and albums, effects, etc.

● editorial content such as biographies, comments, rankings, and the like.

Other types of databases that may be used as a source of metadata related to songs and videos include:

● historical billboard rankings at local, regional, or national level, or historical billboard rankings on foreign billboards.

● music and video industry news.

● lyrics of music.

● music blogs or industry data integration services.

● music consumption transaction and trend data from online and offline sources.

In one embodiment, metadata from these databases may be extracted from the source database and embedded 712 in the media object 710 itself. Alternatively, or in addition, the media object may relate to an IO 740 that contains or relates to metadata. The metadata may include one or more keywords or topics that describe or categorize the data (including rating or ranking information for one or more users). For example, an IO relating to metadata may be a theme relating to all songs in a genre (e.g., rock) or all songs performed on a particular festival (e.g., the Woods Stokes). A subject IO that relates to metadata may be associated with an IO 742 that relates to a higher level subject. For example, a composer may be associated with a theme such as a Barrock style music, which is itself associated with a higher level of IO for classical music.

Alternatively, or in addition, a metadata server and its associated database can be defined as an RWE 722 within the W4COMN, and media objects and other IOs can be associated with the RWE 722. In one embodiment, metadata related to a media object may be retrieved on demand, rather than being stored as static metadata or persistent IOs. The metadata retrieved on demand may be selected based on the needs of users having potential interest in the media object. For example, a user initially selecting a media object based on a theme may then retrieve metadata relating to the pitch or mood of music associated with the media object as desired.

If the user wishes to select a media object using a non-existent theme (e.g., the first ten singles in uk in 1975), a metadata server capable of providing such information may be queried to retrieve a list of such songs. In one embodiment, a list of songs may be used to create an IO related to a topic, such as IO 740, by associating a media object related to the list of songs with the newly created IO. In one embodiment, the IO is created by a correlation engine within the W4 engine (see, e.g., above and FIGS. 4-6). The IO is then available for use in subsequent queries.

In one embodiment, the media object is associated with other RWEs, such as music rights holders 730 (i.e., owners and licensed people), and interested listeners 750. In one embodiment (where the owner 730 of a media object can be identified), a property engine within the W4 engine tracks real-world ownership, control, publishing, or other conditional rights of any RWE in any media IO whenever a new object is detected.

In one embodiment, users 750, 752, and 754 may be identified as interested in a particular song 710 or topic IO 740 or 742 by a relevance engine within the W4 engine. In one embodiment, the correlation engine identifies relationships between user RWEs and media or IOs related to metadata by creating a combined graph of RWEs and IOs and their attributes, relationships and reputations. For example, a user may explicitly state in the user profile that they are interested in a particular musical artist. Alternatively, the correlation engine may determine that the user is interested in a topic or song or scenery based on the user's interaction data content, detected attention events from any internal or external source (including transaction history), online paths and browsing history, and physical real-world paths and attention data.

In one embodiment, over time, the W4COMN builds a user's profile by collecting data from the user or from information sources available to the network in order to gain knowledge of where they were born, where they lived in the past, and where they now lived. Using social data, the W4COMN can also create an overlapping social network profile that places the user in the temporal, geographic, and social graphs, thus determining when and where the user lives. A user RWE can also be associated with other RWEs through interaction data, co-location data, or co-existence data. Users interested in the same time/place may state their interests and connect to a topic-based social network, for example, through an IO involving the topic. In the embodiment illustrated in FIG. 7, users 750 and 752 are identified as being in social network 760.

Thus, media objects may be stored and associated with temporal, spatial, social, and topical data derived from (without limitation) traditional metadata sources, user profile data, social networking, and interaction data, building a relationship network across the universe of media and users. These relationships can be built on demand and, if necessary or as a candidate, continually updated based on the rest of the world involving users, their agents, declared and implied interests, and real and online. These relationships may then support queries for media that satisfy simple or complex contextual criteria.

FIG. 8 illustrates one embodiment of a system 800 capable of supporting context enhanced messaging between users known to the network.

The backbone of the system is a W4COMN 850 or similar network that provides data storage, processing, and real-time tracking capabilities. Within the W4COMN is a server that provides a context-based communication facility, as will be described in more detail below. The data relationships described above in FIG. 7 are stored in the W4 COMN. In one embodiment, the data relationships between all real-world entities and logical data are stored in a global index in the W4COMN 850 maintained by processes in the W4 COMN.

The media objects may be stored by servers in the W4COMN 850, may be stored in a distributed manner on the end-user device, or may be stored by third-party data providers 840, or both. Third party providers 840 may provide additional data to network 850, third party providers 840 being, for example, metadata providers or social networking sites known to the network.

A message sender 802 wishing to send a context enhanced message to a recipient enters the message and context criteria (including delivery criteria and content criteria) into a user agent device 804, which user agent device 804 sends the message to a network 850. The message is processed by a server within the network to add context-specific content to the message and deliver the enhanced message to the proxy device 812 of the message recipient 810 if delivery criteria are met.

The network knows real world entities including a message sender 802, a message receiver 810, proxy devices 804 and 812 of the message sender and the message receiver, respectively, friends 826 and 830 of the message sender, a retail location 820, a restaurant 824, and a friend's home 828. For each entity, the network (without limitation) tracks the physical location of the entity, builds and stores profile data, and stores and analyzes interaction data. The network also receives data from remote sensors 832, which remote sensors 832 may include traffic sensors, GPS devices, weather sensors, video surveillance, cellular towers, Bluetooth, Wi-Fi, and the like.

FIG. 9 illustrates one embodiment of how a network (e.g., W4 COMN) containing temporal, spatial, and social networking and topical data for multiple users, devices, and media may be used for the process of making a media-enhanced message have a complex user context with delivery and content criteria.

The process begins when a message (containing at least one recipient and delivery criteria and content criteria) is received 910 from a message sender. The message sender may use any type of proxy device (e.g., portable media player, PDA, computer, or mobile phone) to enter the message, delivery, and content criteria. The delivery criteria and content criteria may be any combination of spatial, temporal, social, or topical criteria.

In one embodiment, the criteria may be correlated using standard relational or set operators. In one embodiment, the criteria may be declared as a natural language query. In one embodiment, the criteria may be ranked according to relative importance for each request and prioritized appropriately in the generated universe of enhanced content. The request may be considered to include by default criteria that specify the user making the request (i.e., the request is derived from the perspective of the user making the request). The message need not contain delivery criteria and content criteria, but may contain only content criteria or only delivery criteria. Multiple recipients may also have multiple divergent or overlapping sets of content and delivery criteria. For example, the requesting user may send a message for a party and instructions for the system to enhance the content of the invitation with the "party" emotional music currently liked by each recipient. Thus, each recipient may receive a different song that is added to their invitation, with each invitation being created by a single standard request.

The process then determines whether delivery conditions are met 920 using data available to the network (including network database 922 and sensors 924). When the delivery criteria are not initially met 930, the process saves the message for a fixed length of time and periodically or continually re-evaluates the delivery criteria until the delivery conditions are met. The process may use the network-available database 920 and the sensors 924 to monitor any spatial, temporal, social, or topical data known to the network.

When the delivery conditions are met 930, the process retrieves media that relates to the content criteria 940. The criteria are used to search, via the network, user profile data, social network data, spatial data, temporal data, and topical data (which are available via networks 922 and 924 and relate to the content criteria and media files) to identify at least one media file that is relevant to the content criteria. The media file is then inserted into message 950 and the message is then sent to message recipient 960. In an alternative embodiment, media files relating to the content criteria may be retrieved prior to evaluating the delivery conditions, and the message may be updated and sent when the delivery conditions are met.

FIG. 10 illustrates one embodiment of a context enhanced message engine capable of supporting the process illustrated in FIG. 9.

The context enhanced message engine 1000 resides on a server in the W4 COMN. The contextual query engine 1000 may be defined to W4COMN as an RWE, or as a candidate as an active IO. The context query engine may be a component of the W4 engine, or alternatively, may use services provided by the W4 engine or a component of any of its constituent engines.

The context enhanced message engine 1000 includes: a content enhanced message receiving module 1100 that receives a message containing delivery and content criteria from a message sender; a delivery criteria evaluation and tracking module 1200 that determines whether delivery criteria are met and tracks data relating to the delivery criteria; a media retrieval module 1400 that retrieves media related to content standards; a context enhanced message update module 1500 that inserts media files into a message; and a context-enhanced messaging module 1600 that sends the updated message to the intended recipient. Any of the foregoing modules or communications between modules (e.g., delivery or context criteria) may be stored on a computer-readable medium for transient, temporary, or permanent storage.

The context enhanced message receiving module 1100 may provide a user interface to a message sender for entering a context enhanced message request. The interface provided may be a graphical user interface displayable on a mobile phone, gaming device, computer or PDA, including HTTP documents accessible via the internet. These interfaces may also take other forms, including text files (e.g., SMS), email, and APIs that may be used by software applications located on the computing device. The interface provides input of delivery or content criteria including spatial, temporal, social, or topical criteria.

In one embodiment, delivery and content criteria may be correlated using standard relational or set operators. In one embodiment, temporal and spatial data obtained from sensors within the user device may be included in the delivery or content criteria. For example, the current location of the device associated with the user may be automatically identified and included in the criteria, as well as the current time and date, and the like. The message sender creating the context may be automatically identified and automatically included in the context by associating a proxy device with the user in the network.

The delivery criteria evaluation and tracking module 1200 uses all data known to the network to evaluate delivery conditions. Such data may include network database 1220 and real-time sensors 1240. Sensor data can include data relating to the physical location of any real-world entity and can include message senders and message receivers, as well as any other known RWEs that can be specified in delivery conditions. The end-user device may include positioning or other sensors that detect various aspects of the physical environment surrounding the user, such as the user's geographic location, altitude, and direction vector. The sensors may also include other environmental sensors, such as temperature and lighting sensors, or may also include biometric sensors, such as heart rate, brain waves, and the like.

The delivery criteria may relate to any combination of spatial, temporal, social, or topical data available to the network. In one embodiment, the delivery criteria evaluation and tracking module 1200 tracks data relating to delivery criteria in a message when the delivery criteria are not immediately satisfied. In one embodiment, the delivery criteria are periodically re-evaluated. In another embodiment, data relating to delivery conditions is tracked in real time and changes in the values trigger re-evaluation of the delivery conditions.

For example, the delivery criteria may specify that the message be processed at a future point in time, periodically, or upon the occurrence of a particular event. For example, delivery may specify that the message be reprocessed upon the occurrence of a trigger condition, such as hourly, when the physical location of the entity associated with the delivery condition changes, when a calendar event occurs (e.g., anniversary), when a news event occurs (e.g., a favorite sports team wins a game), when a space, social, time, or theme interaction occurs (e.g., when two or more friends arrive at a favorite bar to watch soccer), or when the interests of the sender or recipient change.

The media retrieval module 1400 searches one or more network databases 1220 and sensors 1240 for user profile data, social network data, spatial data, temporal data, and topical data (which are available via the network and relate to context and media files) to identify at least one media file that pertains to the content criteria. These searches are done using network database 1220 and its supporting infrastructure capabilities.

In one embodiment, the criteria are interpreted to utilize the best available data within the network. For example, if context-related data resides on a relational database, the query module may execute a series of SQL statements or a procedural language containing embedded SQL for retrieving data from the relational database. Queries may be nested or otherwise structured to retrieve data from a set of entities and use the result set to drive additional queries against other entities, or to use recursive data retrieval.

In the case of the W4COMN, the content standard can be mapped and represented for all other known entities and data objects in order to create a mini-graph for each entity and a global graph that correlates all known entities to each other and thereby identifies context-relevant media objects. In one embodiment, these relationships between entities and data objects are stored in a global index within the W4 COMN.

When the query criteria relate to simple descriptive content (e.g., date and time of creation), the relationship may be identified using metadata embedded in the media object. When the criteria relates to a topic (e.g., a style of music), relationships may be identified by IOs (whether currently existing or dynamically generated) relating to the topic, which may then be used to identify media objects associated with the topic.

When a criterion relates to a relationship between two or more IOs or RWEs (e.g., all friends of a particular user), the social network relationships supported by the W4COMN can be used to identify the IOs and RWEs involved. When a particular media object is selected, the media search module may use ownership data in or associated with the media object to further determine whether the message recipient or an agent of the message recipient receiving the context is permitted to access the content of the media file.

The context enhanced message update module 1500 may update the context enhanced message by any means that allows the message recipient to access the selected media file. In one embodiment, the actual media file is inserted into the message and opened or begins playing when the recipient opens the enhanced message. In one embodiment, the inserted file includes a link to a media file. In one embodiment, the media files include one or more playlists of a plurality of objects or files. In an alternative implementation, the content criteria are inserted into the message and not evaluated until the message recipient opens the message. In one such embodiment, the media retrieval module 1400 does not process the content criteria until the message recipient opens the message.

Context enhanced messaging module 1600 may send a message to a single recipient or a group of recipients having a set of characteristics that define a limited set of users known to the network. For example, the message may be sent to a user in the sender's social network who is single and likes rock music, or to a fan of last night band who is at a show and also has their own blog.

In one embodiment, the enhanced message may contain advertisements and enhanced content criteria related to the advertisements. Thus, advertisements may provide basic advertising content and media designed for a particular user. For example, an advertisement for a sports car may be associated with a user's favorite music artist and song that specifies a fast tempo or is significantly related to speed, or the context of the year 1975 in which the user last possessed the sports car.

The context enhanced message may provide additional data to the network regarding the sender and recipient of the message. Collecting interaction data contained in context-enhanced messages between individuals and groups regarding preferences and communication patterns may provide data that may inform a pair or group or individuals within the pair or group of an advertiser's communication. For example, a context-enhanced message may reveal that the user loves a butterfly, and an advertiser with a butterfly-themed good may wish to contact the user.

Specific examples of contexts

The present disclosure will now discuss specific examples of the above principles. The examples given below are intended to be illustrative, not limiting.

In one example, if a user wishes to send a message containing a song that is automatically played for a recipient at a specified time (e.g., a music wake-up call), the message sender may create a message with delivery criteria for a particular time and content criteria for a specified song. The delivery criteria evaluation and tracking module will track the current time and pass the message to the media retrieval module for processing when the specified time is reached. The media retrieval module will retrieve the particular song for insertion into the message.

In another example, if the message sender wishes to send a message that a media object was played when the recipient arrived at a particular location (e.g., a message that a shopping list audio file was played when the recipient arrived at a grocery store), the message sender may create a message with delivery criteria specifying delivery when the recipient was at a particular location and content criteria specifying a particular media object. The delivery criteria evaluation and tracking module will track the current location of the recipient and pass the message to the media retrieval module for processing when the user arrives at the location. The media retrieval module will retrieve the particular media object for insertion into the message.

As an alternative, the message sender may wish to send a message containing a romantic song that is played as the recipient drives past a venue that has romantic significance to the sender and the recipient (e.g., a venue at which they are married). The message sender may create a message with delivery criteria specifying delivery when the recipient is at a particular location and content criteria specifying a romantic song. The delivery criteria evaluation and tracking module will track the current location of the recipient and pass the message to the media retrieval module for processing when the user arrives at the location. The media retrieval module may then search the recipient's profile data and recent playlists for songs having metadata representing romantic or traumatic songs, for example.

In another example, a message sender wishes to send a message that includes a collection of event-based music pieces to be played for a recipient, the music pieces varying based on the recipient's actions or location at different times of the day. The message sender may create a message in which the delivery criteria specify that on certain days

Delivery is made (e.g., the recipient's birthday) and the content criteria specify different music collections for breakfast, lunch and dinner. The content standard may further specify that the composition is to be played as the location of the recipient changes. For example, the content criteria may specify one music piece when the recipient enters a Thailand restaurant for lunch and another music piece when the recipient eats at a French restaurant.

In each case, the delivery criteria evaluation and tracking module will track the current date and time, as well as the current location of the recipient, and transmit the message to the message retrieval module for processing when the delivery criteria are met. The media retrieval module may then, for example, search for media that meets content criteria, such as music relating to food and thailand or france.

The sender may wish to express emotion using context-enhanced messaging. The message sender may create a message for immediate delivery that specifies content criteria for selecting a song that reflects the sender's current mood. The media retrieval module may attempt to determine the sender's current mood by scanning the text in the sender's recent email and text messages. The media retrieval module may then select songs from songs liked by the sender (e.g., in the user's profile, or the most common historically played songs) that have associations that suggest that the songs are responsive to the mood.

The context may define a push or pull operation. The sender may wish to express her mood to her maiden with a media-enhanced message that the system sends on her behalf, as described above, or alternatively, the sender's maiden may wish to inquire of her mood. For example, a sender's maiden may send her a message containing content criteria specifying her maiden mood. For example, if he thinks she, the system may respond with a song that expresses the emotion between their two people.

In another example, assume that a message sender wishes to send a message containing text that matches the lyrics of a song in order to express an emotional connection between the sender and the recipient. For example, the sender knows that the recipient likes butterflies. The message sender may create a message for immediate delivery with content criteria that specifies a romantic song with lyrics containing the word "butterfly". The content retrieval module may then, for example, search for songs whose lyrics include "butterfly" and have metadata representing romantic or nociceptive songs, and then rank the songs individually for the particular user based on the user profile and past consumption data for the particular user.

Those skilled in the art will recognize that the methods and systems of the present disclosure may be implemented in many ways and, thus, are not limited by the foregoing example embodiments and examples. In other words, functional elements being implemented by single or multiple components in various combinations of hardware and software or firmware, as well as individual functions, may be distributed among software applications at either the client or server side or both. In this regard, any number of the features of the different embodiments described herein may be combined into single or multiple embodiments, and alternate embodiments having fewer than, or more than, all of the features described herein are possible. The functionality may also be distributed, in whole or in part, across multiple components, in manners now known or to become known. Thus, myriad software/hardware/firmware combinations are possible in implementing the functions, features, interfaces and preferences described herein. Moreover, the scope of the present disclosure covers conventionally known manners for implementing the described features and functions and interfaces, as well as those variations and modifications that may be made to the hardware or software or firmware components described herein as would be understood by those skilled in the art now and hereafter.

Furthermore, the embodiments of the methods as set forth and described in the flowcharts of the present disclosure are provided merely as examples in order to provide a more complete understanding of the techniques. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.

While various embodiments have been described for purposes of this disclosure, these embodiments should not be considered to limit the teachings of this disclosure to those embodiments. Various changes and modifications may be made to the elements and operations described above to achieve results that remain within the scope of the systems and processes described in this disclosure.

Claims (73)

1. A method, comprising the steps of:

receiving a message delivered from a message sender to at least one message recipient via a network, wherein the message comprises at least one content criterion;

formulating a query based on the at least one content criterion to search, via the network, for user profile data, social network data, spatial data, temporal data, and topical data, the data being available via the network and related to the at least one content criterion and media files, to identify at least one media file related to the at least one content criterion;

inserting the at least one media file into the message;

sending the message to the at least one message recipient via the network.

2. The method of claim 1, wherein user profile data comprises data matching at least one user, and preferences of the at least one user are used as part of the query.

3. The method of claim 1, wherein the at least one content criterion is a spatial criterion.

4. The method of claim 1, wherein the at least one content criterion is a temporal criterion.

5. The method of claim 1, wherein the at least one content criterion is a social criterion.

6. The method of claim 1, wherein the at least one content criterion is a topical criterion.

7. The method of claim 1, wherein the at least one content criterion is a current mood of a person known to the network.

8. The method of claim 1, wherein the at least one content criterion is lyrics of a song.

9. The method of claim 1, wherein the message contains at least two content criteria, wherein a first of the at least two content criteria is used to formulate the query when the message is sent a first time and a second of the at least two content criteria is used to formulate the query when the message is sent a second time.

10. The method of claim 1, wherein a playlist referencing the at least one media file is inserted into the message.

11. The method of claim 1, wherein an HTTP link referencing the at least one media file is inserted into the message.

12. A method, comprising the steps of:

receiving a message delivered from a message sender to at least one message recipient via a network, wherein the message comprises at least one delivery criterion and at least one content criterion;

determining, via the network, that delivery criteria for the message are satisfied;

formulating a query based on the at least one content criterion to search, via the network, for user profile data, social network data, spatial data, temporal data, and topical data, the data being available via the network and related to the at least one content criterion and media files, to identify at least one media file related to the at least one content criterion;

inserting the at least one media file into the message;

sending the message to the at least one recipient via the network.

13. The method of claim 12, wherein if at least one delivery condition is not met when the message is initially received, the message is stored until at least one change in data on the network causes the delivery criteria to be met.

14. The method of claim 12, wherein the at least one delivery criterion specifies that the message is to be sent at a particular time.

15. The method of claim 12, wherein the at least one delivery criterion specifies that the message is to be sent at a particular date and time.

16. The method of claim 12, wherein the at least one delivery criterion specifies that the message is to be sent upon the occurrence of a particular event.

17. The method of claim 12, wherein the at least one delivery criterion specifies that the message is to be sent when the at least one recipient is in a particular location.

18. The method of claim 12, wherein the at least one delivery criterion specifies that the message is to be sent when the at least one recipient is at a particular location at a particular time.

19. The method of claim 12, wherein user profile data includes data matching at least one user, and preferences of the at least one user are used as part of the query.

20. The method of claim 12, wherein the at least one content criterion is a spatial criterion.

21. The method of claim 12, wherein the at least one content criterion is a temporal criterion.

22. The method of claim 12, wherein the at least one content criterion is a social criterion.

23. The method of claim 12, wherein the at least one content criterion is a subject criterion.

24. The method of claim 12, wherein the at least one content criterion is a current mood of a person known to the network.

25. The method of claim 12, wherein the at least one content criterion is lyrics of a song.

26. The method of claim 12, wherein the message contains at least two content criteria, wherein a first of the at least two content criteria is used to formulate the query when the message is sent a first time and a second of the at least two content criteria is used to formulate the query when the message is sent a second time.

27. The method of claim 12, wherein a playlist referencing the at least one media file is inserted into the message.

28. The method of claim 12, wherein an HTTP link referencing the at least one media file is inserted into the message.

29. A computer-readable medium having computer-executable instructions for a method, the method comprising the steps of:

receiving a message delivered from a message sender to at least one message recipient via a network, wherein the message comprises at least one content criterion;

formulating a query based on the at least one content criterion to search, via the network, for user profile data, social network data, spatial data, temporal data, and topical data, the data being available via the network and related to the at least one content criterion and media files, to identify at least one media file related to the at least one content criterion;

inserting the at least one media file into the message;

sending the message to the at least one message recipient via the network.

30. The computer-readable medium of claim 29, wherein user profile data includes data that matches at least one user, and preferences of the at least one user are used as part of the query.

31. The computer-readable medium of claim 29, wherein the at least one content criterion is a spatial criterion.

32. The computer-readable medium of claim 29, wherein the at least one content criterion is a temporal criterion.

33. The computer-readable medium of claim 29, wherein the at least one content criterion is a social criterion.

34. The computer-readable medium of claim 29, wherein the at least one content criterion is a subject criterion.

35. The computer-readable medium of claim 29, wherein the at least one content criterion is a current mood of the person known to the network.

36. The computer-readable medium of claim 29, wherein the at least one content criterion is lyrics of a song.

37. The computer-readable medium of claim 29, wherein the message contains at least two content criteria, wherein a first of the at least two content criteria is used to formulate the query when the message is first sent and a second of the at least two content criteria is used to formulate the query when the message is sent a second time.

38. The computer-readable medium of claim 29, wherein a playlist referencing the at least one media file is inserted into the message.

39. The computer-readable medium of claim 29, wherein an HTTP link referencing the at least one media file is inserted into the message.

40. A computer-readable medium having computer-executable instructions for a method, the method comprising the steps of:

receiving a message delivered from a message sender to at least one message recipient via a network, wherein the message comprises at least one delivery criterion and at least one content criterion;

determining, via the network, that delivery criteria for the message are satisfied;

formulating a query based on the at least one content criterion to search, via the network, for user profile data, social network data, spatial data, temporal data, and topical data, the data being available via the network and related to the at least one content criterion and media files, to identify at least one media file related to the at least one content criterion;

inserting the at least one media file into the message;

sending the message to the at least one recipient via the network.

41. The computer-readable medium of claim 40, wherein if at least one delivery condition is not met when the message is initially received, the message is stored until at least one change in data on the network causes the delivery criteria to be met.

42. The computer-readable medium of claim 40, wherein the at least one delivery criterion specifies that the message is to be sent at a particular time.

43. The computer-readable medium of claim 40, wherein the at least one delivery criterion specifies that the message is to be sent at a particular date and time.

44. The computer-readable medium of claim 40, wherein the at least one delivery criterion specifies that the message is to be sent upon the occurrence of a particular event.

45. The computer-readable medium of claim 40, wherein the at least one delivery criterion specifies that the message is to be sent when the at least one recipient is at a particular location.

46. The computer-readable medium of claim 40, wherein the at least one delivery criterion specifies that the message is to be sent when the at least one recipient is at a particular location at a particular time.

47. The computer-readable medium of claim 40, wherein user profile data includes data that matches at least one user, and preferences of the at least one user are used as part of the query.

48. The computer-readable medium of claim 40, wherein the at least one content criterion is a spatial criterion.

49. The computer-readable medium of claim 40, wherein the at least one content criterion is a temporal criterion.

50. The computer-readable medium of claim 40, wherein the at least one content criterion is a social criterion.

51. The computer-readable medium of claim 40, wherein the at least one content criterion is a subject criterion.

52. The computer-readable medium of claim 40, wherein the at least one content criterion is a current mood of a person known to the network.

53. The computer-readable medium of claim 40 wherein the at least one content criterion is lyrics of a song.

54. The computer-readable medium of claim 40, wherein the message contains at least two content criteria, wherein a first of the at least two content criteria is used to formulate the query when the message is first sent and a second of the at least two content criteria is used to formulate the query when the message is sent a second time.

55. The computer-readable medium of claim 40, wherein a playlist referencing the at least one media file is inserted into the message.

56. The computer-readable medium of claim 40, wherein an HTTP link referencing the at least one media file is inserted into the message.

57. A system, comprising:

a context-enhanced message receiving module that receives a message from a sender, wherein the requests each include at least one recipient, at least one delivery criteria, and at least one content criteria;

a delivery criteria evaluation and tracking module that determines, for each message received by the context-enhanced message-receiving module, whether the at least one delivery criteria for the message is satisfied;

a media retrieval module that uses content criteria for each message received by the context-enhanced message receiving module to formulate a query based on the content criteria such that user profile data, social network data, spatial data, temporal data, and topical data are searched via the network, the data being available via the network and related to the content criteria and media files, so as to identify at least one media file that is relevant to the content criteria;

a context-enhanced message update module that, for each message, inserts the at least one media file identified by the media retrieval module within the message; and

a context-enhanced message sending module that sends said each message updated by said context-enhanced message updating module to at least one recipient of said message.

58. The system of claim 57, wherein if the delivery criteria for a message are not met when the message is initially evaluated, the message is saved and the delivery criteria evaluation and tracking module tracks data related to delivery criteria for the message until the delivery criteria for a message are met.

59. The system of claim 57, wherein for at least some messages, the at least one delivery criterion specifies that the message is to be sent at a particular time.

60. The system of claim 57, wherein for at least some messages, the at least one delivery criterion specifies that the message is to be sent at a particular date and time.

61. The system of claim 57, wherein for at least some messages, the at least one delivery criterion specifies that the message is to be sent upon the occurrence of a particular event.

62. The system of claim 57, wherein for at least some messages, the at least one delivery criterion specifies that the message is to be sent when the at least one recipient is at a particular location.

63. The system of claim 57, wherein for at least some messages, the at least one delivery criterion specifies that the message is to be sent when the at least one recipient is at a particular location at a particular time.

64. The system of claim 57, wherein for at least some messages, user profile data includes data that matches at least one user, and preferences of the at least one user are used as part of the query.

65. The system of claim 57 wherein the at least one content criterion is a spatial criterion for at least some messages.

66. The system of claim 57, wherein the at least one content criterion is a time criterion for at least some messages.

67. The system of claim 57, wherein the at least one content criterion is a social criterion for at least some messages.

68. The system of claim 57 wherein the at least one content criterion is a subject criterion for at least some messages.

69. The system of claim 57 wherein the at least one content criterion is a current mood of a person known to the network for at least some messages.

70. The system of claim 57 wherein the at least one content criterion is lyrics of a song for at least some messages.

71. The system of claim 57, wherein for at least some messages, the message contains at least two content criteria, wherein a first of the at least two content criteria is used to formulate the query when the message is first sent and a second of the at least two content criteria is used to formulate the query when the message is sent a second time.

72. The system of claim 57, wherein the context enhanced message update module inserts a playlist referencing the at least one media file into the message.

73. The system of claim 57, wherein the context-enhanced message update module inserts an HTTP link referencing the at least one media file into the message.