HK1153063B - System and method for delivery of augmented messages - Google Patents

Description

System and method for delivering augmented messages

Background

When people use electronic devices, such as mobile phones and cable set-top boxes, a large amount of information is generated. Such information, such as location, applications used, social networks, physical and online locations accessed, and the like, may be used to deliver useful services and information to end users and provide business opportunities to advertisers and retailers. However, due to the deficiencies of the way this information is captured, most of it is actually discarded. For example, for a mobile phone, information is typically not collected when the mobile phone is idle (i.e., not in use by the user). Other information such as the presence of other people in the vicinity, the time and frequency of messages to other users, and the activity of the user's social network are also not effectively captured.

Disclosure of Invention

In conventional messaging systems, when a message is sent by a sender, the message is received by each recipient in substantially the same form, regardless of whether the message is an electronic message such as an email, a recorded voicemail message, an instant message, or other type of electronic message that may be addressed by the sender to one or more recipients. In other words, the form of the content of the message arrived at by the recipient(s) is the same as it was sent by the sender. In some cases, known email systems may insert advertising messages or background messages that may or may not be under the control of the sender, but in any event, conventional messaging systems known to date do not utilize information about the sender and/or each recipient of the message to adjust and create, respectively, an augmented message (augmented message) that contains other information in addition to that expected by the sender, and that is derived by the network sending the message based on information known or available to the network about the message and the sender and/or each recipient of the electronic message.

The present disclosure describes systems and methods for improving performance of services provided via a network using data collected and stored by multiple devices on the network. In particular, the present disclosure describes systems and methods of delivering communications associated with delivery conditions (associations), where the occurrence of a delivery condition is determined by monitoring information received from multiple sources via multiple communication channels. The message delivery system allows for delivery of messages from any "Who, What, When, Where" to any "Who, What, When, Where" upon detection of the occurrence of one or more "Who, What, Where" delivery conditions. The message, which may be any data object, including a text-based message, an audio-based message (e.g., a voicemail or other audio such as music), or a video-based pre-recorded message, is delivered according to delivery conditions based on any available data, including topical data, spatial data, temporal data, and/or social data. In addition, because the system coordinates the delivery of messages via multiple communication channels and through multiple devices, the communication channel for delivering messages may be dynamically determined based on delivery conditions.

One aspect of the present disclosure is a method for delivering a message, comprising receiving a message from a sender for delivery over a network, the message comprising message content and an identity of an intended recipient. The message content is analyzed to extract logical data, which includes the subject matter of the message. Network-available recipient data is collected by searching social, spatial, temporal, and logical data about the intended recipient and the extracted logical data. Correlating (correct) the collected recipient data, such that the collected data is refined into correlated data (correct data) relating the logical data to the intended recipient. The method collects content available to the network relating to the associated data and determines which of the collected content is to be inserted into the message. Inserting the determined collected content into a message to form an enhanced message.

Another aspect of the disclosure is a computer readable medium tangibly encoded with instructions for performing a method for delivering a message. The method includes receiving a message from a sender for delivery over a network, the message including message content and an identity of an intended recipient. The message content is analyzed to extract logical data, which includes the subject matter of the message. Network-available recipient data is collected by searching social, spatial, temporal, and logical data about the intended recipient and the extracted logical data. Correlating the collected recipient data such that the collected data is refined into correlated data relating the logical data to the intended recipient. The method collects content available to the network relating to the associated data and determines which of the collected content is to be inserted into the message. Inserting the determined collected content into a message to form an enhanced message.

In another aspect, the present disclosure describes a computer system that includes a plurality of processors. The system also includes an attention engine implemented on one of the plurality of processors for receiving a message for an intended recipient via a network, wherein the message contains message content and an identity of the intended recipient. A attribution engine implemented on one of the plurality of processors extracts logical data from the message content, the logical data including a subject matter of the message. A message fetch manager implemented on one of the processors collects network-available recipient data by searching social, spatial, temporal, and logical data about the intended recipient and the extracted logical data. An correlation engine implemented on one of the plurality of processors correlates the collected recipient data to refine the collected data into correlated data relating the logical data to the intended recipient. An identification engine implemented on one of the processors collects content available to the network relating to the correlated data and determines which of the collected content is to be inserted into the message, and a message enhancement manager implemented on one of the processors inserts the determined collected content into the message to form an enhanced message.

These and various other features and advantages will be apparent from a reading of the following detailed description and a review of the accompanying drawings. Additional features are set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the described embodiments. The benefits and features will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the disclosure as claimed.

Drawings

The following drawings, which form a part of the present application, illustrate the embodiment systems and methods described below and are not intended to limit the scope of the disclosure in any way, which scope should be based on the appended claims.

FIG. 1 illustrates an example of the relationship between RWEs and IOs on a W4 COMN.

FIG. 2 illustrates an example of metadata defining relationships between RWEs and IOs on a W4 COMN.

FIG. 3 illustrates a conceptual model of the W4 COMN.

FIG. 4 illustrates the functional layers of the W4COMN architecture.

FIG. 5 illustrates an embodiment of an analysis component of the W4 engine as shown in FIG. 2.

FIG. 6 illustrates an embodiment of a W4 engine showing different components within the sub-engines generally described above with reference to FIG. 5.

Fig. 7A-B illustrate elements of an embodiment of a W4 engine suitable for performing W4 message delivery as described herein.

FIG. 8 illustrates an embodiment of a method of delivering messages over a network based on social data, temporal data, spatial data, and topical data of entities on the network.

FIG. 9 illustrates a flow diagram of an embodiment of augmented messaging.

Fig. 9A illustrates a flow chart of an alternative embodiment.

10A-B illustrate other embodiments of systems for delivering augmented messages over a network.

FIG. 11 shows a non-limiting example of an augmented message sent from a sender to three different recipients.

Detailed Description

The present disclosure describes a communication network, referred to herein as the "W4 communication network" or W4COMN, that uses information about "Who, What, When and Where" relating to interactions with the network to provide improved services to users of the network. The W4COMN is a collection of users, devices, and processes that support synchronous and asynchronous communications between users and their agents. It includes a provisioning network of sensors that provide data identification and collection in a real-world environment with respect to any of the principals, locations, users, or combinations thereof.

As a communication network, the W4COMN handles routing/addressing, scheduling, filtering, prioritization, replying, forwarding, storing, deleting, privacy, transaction processing, triggering of new messages, propagation changes, transcoding and linking. Additionally, these actions may be performed over any communication channel accessible to the W4 COMN.

The W4COMN uses a data modeling strategy for creating profiles for not only users and locations but also for any device on the network and any kind of user-defined data, with user-specified conditions from a wide variety of possibilities. With the social, spatial, temporal, and logical data available about a particular user, topic, or logical data object, each entity known to the W4COMN can be mapped and represented relative to all other known entities and data objects in order to create a micro-graph for each entity and a global graph that correlates all known entities and their relationships with attributes.

To describe the operation of the W4COMN, two elements, namely, real world entities and information objects, on which the W4COMN is built must first be introduced. These distinctions are made to enable correlations from which the relationship between the electronic/logical object and the real object can be determined. Real World Entities (RWEs) refer to people, devices, locations, or other physical things known to the W4 COMN. Each RWE known to the W4COMN is assigned or otherwise provided with a unique W4 identification number that absolutely identifies that RWE within the W4 COMN.

RWEs can interact with the network directly or through proxies, which themselves can be RWEs. Examples of RWEs that interact directly with the W4COMN include any device, such as a sensor, motor, or other hardware, that is connected to the W4COMN in order to receive or transmit data or control signals. Because the W4COMN can be adapted to use all types of data communications, devices that can act as RWEs include all devices that are capable of acting as network nodes or generating, requesting, and/or consuming data in a networked environment or that are controllable via a network. Such devices include any kind of "dumb" device designed for the specific purpose of interacting with a network (e.g., cellular phones, cable television set-top boxes, facsimile machines, telephones, and Radio Frequency Identification (RFID) tags, sensors, etc.). Typically, such devices are primarily hardware, and their operation cannot be considered separately from the physical devices.

Examples of RWEs that must use agents to interact with the W4COMN network include all non-electronic entities, including physical entities such as people, locations (e.g., countries, cities, houses, buildings, airports, roads, etc.) and things (e.g., animals, pets, livestock, gardens, physical objects, vehicles, airplanes, crafts, etc.), and intangible entities such as business entities, legal entities, people, or sports teams. In addition, "smart" devices (e.g., computing devices such as smart phones, smart set-top boxes, smart vehicles that support communication with other devices or networks, laptops, personal computers, server computers, satellites, etc.) are also considered RWEs that must use agents to interact with the network. Smart devices are electronic devices that can run software via an internal processor to interact with a network. For smart devices, the software application(s) that interact with the W4COMN and act as agents for the device are actually running.

The W4COMN allows associations between RWEs to be determined and tracked. For example, a given user (RWE) can be associated with any number and type of other RWEs, including other personas, cellular phones, smart credit cards, personal data assistants, email and other communication service accounts, networked computers, smart appliances, set-top boxes and receivers for cable television and other media services, and any other networked devices. The association can be made explicitly by the user, such as when the RWE is installed into the W4 COMN. One example of this is the setup of a new cell phone, cable service, or email account, where a user explicitly identifies an RWE (e.g., the user's phone for cell phone service, the user's set-top box and/or location for cable service, or username and password for online service) as being directly associated with the user. The explicit association can include the user identifying a particular relationship between the user and the RWE (e.g., this is my device, this is my home appliance, this person is my friend/father/son/etc., this device is shared between me and other users, etc.). RWEs can also be implicitly associated with a user based on the current situation. For example, a weather sensor on the W4COMN may be implicitly associated with a user based on information indicating that the user lives near or is passing by the location of the sensor.

An Information Object (IO), on the other hand, is a logical object that stores, maintains, generates, serves as a source of, or otherwise provides data for use by an RWE and/or a W4 COMN. IOs differ from RWEs in that IOs represent data, whereas RWEs can create or consume data (typically by creating or consuming IOs) during their interaction with the W4 COMN. Examples of IOs include passive objects such as communication signals (e.g., digital and analog telephony signals, streaming media, and interprocess communications), email messages, transaction records, virtual cards, event records (e.g., data files identifying time, possibly in conjunction with one or more RWEs (e.g., users and locations), which may be further associated with known topics/activities/meanings (e.g., concerts, meetings, sporting events, etc.), records of telephone calls, calendar entries, web pages, database entries, electronic media objects (e.g., media files containing songs, videos, pictures, images, audio messages, telephone calls, etc.), electronic files, and associated metadata.

Further, IOs include any running process or application that consumes or generates data, such as email communication applications (e.g., OUTLOOK by MICROSOFT or YAHOO! MAIL), calendar applications, word processing applications, image editing applications, media player applications, weather monitoring applications, browser applications, and web server applications. Such an active IO may or may not act as a proxy for one or more RWEs. For example, voice communication software on a smartphone may act as a proxy for the smartphone and the owner of the smartphone.

The IO in the W4COMN can be provided with a unique W4 identification number that absolutely identifies the IO within the W4COMN 4 identification number. While the data in the IO can be modified by the actions of the RWE, the IO is still a passive logical data representation or data source and thus not an RWE.

For each IO, there are at least three classes of associated RWEs. The first type is an RWE that owns or controls the IO, whether as a creator or a rights holder (e.g., an RWE that has editing rights or usage rights to the IO). A second class is RWEs (one or more) that an IO is involved in by, for example, containing information about or identifying the RWEs. A third class is any RWE that subsequently places any attention on the IO (either directly or through a proxy process), where "placing attention" refers to accessing the IO for some purpose to obtain data from the IO.

"available data" and "W4 data" refer to data that exists somewhere in the IO in some form or that can be collected from known IOs or RWEs (e.g., deployed sensors) as needed. "sensor" refers to any source of W4 data, such as a PC, phone, portable PC or other wireless device, home appliance, vehicle, appliance, security scanner, video monitor, RFID tag in clothing, product and location, online data, or any other source of information about real world users/topics/transactions (RWEs) or logic-based agents/processes/topics/transactions (IOs).

FIG. 1 illustrates an example of the relationship between RWEs and IOs on a W4 COMN. In the illustrated embodiment, the user 102 is an RWE of the network having a unique network ID. The user 102 is a human being in communication with the network via proxy devices 104, 106, 108, 110 associated with the user 102, the proxy devices 104, 106, 108, 110 all being RWEs of the network and having their own unique network IDs. Some of these agents may communicate directly with the W4COMN, or with the W4COMN via IOs such as applications running on or by the device.

As described above, the proxy devices 104, 106, 108, 110 may be explicitly associated with the user 102. For example, one device 104 may be a smart phone connected to a network through a cellular service provider and another device 106 may be a smart vehicle connected to the network. Other devices may be implicitly associated with the user 102. For example, one device 108 can be a "dumb" weather sensor at a location that matches the current location of the user's cellular telephone 104, thereby implicitly associating with the user 102 when the two RWEs 104, 108 are at the same location. Another implicitly associated device 110 can be a sensor 110 for a physical location 112 known to the W4 COMN. The known location 112 is associated with the first user 102 either explicitly (through a user-specified relationship, e.g., this is my home, work place, parent, etc.) or implicitly (the user 102 is often in the same location as the RWE 112 as shown by data from the sensor 110 at that location 112).

The user 102 may also be directly associated with others (e.g., the illustrated persona 140) and then indirectly associated with others 142, 144 through their association, as shown. Again, such associations may be explicit (e.g., the user 102 may have identified the associated person 140 as its parent, or have identified the person 140 as a member of the user's social network) or implicit (e.g., they share the same address).

Tracking associations between people (and other RWEs) allows the creation of a concept of "intimacy". Intimacy is a measure of the degree of association between two persons or RWEs. For example, each degree of migration between RWEs can be considered a lower level of affinity and assigned a lower affinity score. Affinity may be based solely on explicit social data, or may be extended to include all W4 data, including spatial data and temporal data.

Each RWE 102, 104, 106, 108, 110, 112, 140, 142, 144 of the W4COMN can be associated with one or more IOs, as shown. Continuing with the above example, fig. 1 illustrates that two IOs 122, 124 are associated with the cellular telephone 104. One IO 122 may be a passive data object, such as an event record used by scheduling/calendaring software on a cell phone, a contact IO used by an address book application, a history of transactions conducted with device 104, or a copy of messages sent from device 104. Other IOs 124 may be active software processes or applications that act as proxies for devices to the W4COMN by sending or receiving data via the W4 COMN. Voice communication software, scheduling/calendaring software, address book applications, or text messaging applications are all examples of IOs that may communicate with other IOs and RWEs on a network. The IOs 122, 124 may be stored locally on the device 104 or remotely on some node or data storage accessible to the W4COMN, such as a message server or cellular telephone service data center. The IO 126 associated with the vehicle 108 may be an electronic file containing the specifications and/or current conditions (e.g., brand, model, identification number, current location, current speed, current conditions, current owner, etc.) of the vehicle 108. The IO 128 associated with the sensor 108 may identify the current state of the hero(s) monitored by the sensor 108, such as current weather or current traffic. The IO 130 associated with the cellular telephone 110 may be information in a database identifying the amount of charge on the most recent call or current bill.

In addition, those RWEs that can only interact with the W4COMN through agents, such as people 102, 140, 142, 144, computing devices 104, 106, and location 112, can have one or more IOs 132, 134, 146, 148, 150 directly associated therewith. Examples include IOs 132, 134 that contain contacts and other RWE-specific information. For example, a person's IOs 132, 146, 148, 150 can be a unique W4COMN identifier that contains an email address, a phone number, a physical address, user preferences, identification of devices and other RWEs associated with the user, records of the user's past interactions with other RWEs on the W4COMN (e.g., transaction records, message copies, a list of time and location combinations that record the user's past whereabouts), a location, and/or any relationship information (e.g., explicit user specification of the user's relationship to relatives, employers, co-workers, neighbors, service providers, etc.). Another example of one person's IOs 132, 146, 148, 150 includes a remote application through which one person may communicate with the W4COMN, such as an account in a web-based email service (e.g., Yahoo!mail). The IO 134 for a location may contain information such as the exact coordinates of the location, driving directions to the location, classifications of the location (residential, commercial, public, non-public, etc.), information about services or products available at the location, a unique W4COMN identifier for the location, a business at the location, a photograph of the location, and so forth.

To correlate RWEs and IOs to identify relationships, the W4COMN heavily uses existing metadata and generates additional metadata as necessary. Metadata is loosely defined as data that describes data. For example, given an IO such as a music file, the core, main or object data of the music file is the actual music data that is converted by the media player into audio that is heard by the listener. The metadata for the same music file may include data identifying the artist, song, etc., album art, and the format of the music data. The metadata may be stored as part of the music file or in one or more different IOs associated with the music file, or both. Further, the W4 metadata for the same music file may include the owner of the music file and the rights the owner has with the music file. As another example, if the IO is a picture taken with an electronic camera, the picture may include metadata identifying when the picture was taken, where the camera was when the picture was taken, what camera taken the picture, what person (if any) was associated with the camera (e.g., designated as the owner of the camera), and who and what were the principal corner of the picture or in the picture, in addition to the primary image data from which the picture may be created on the display. The W4COMN uses all available metadata to identify implicit and explicit associations between entities and data objects.

FIG. 2 illustrates an example of metadata defining relationships between RWEs and IOs on a W4 COMN. In the illustrated embodiment, IO 202 includes object data 204 and five discrete items of metadata 206, 208, 210, 212, 214. Some items of metadata 208, 210, 212 may contain information that is relevant only to the object data 204 and not to any other IO or RWE. Such as a creation date, text, or image to be associated with object data 204 of IO 202.

On the other hand, some items of metadata 206, 214 can identify relationships between the IO 202 and other RWEs and IOs. As shown, the IO 202 is associated with an RWE 220 through an item of metadata 206, the RWE 220 in turn being associated with two IOs 224, 226 and a second RWE 222 based on some information known to the W4 COMN. This portion of FIG. 2 can describe, for example, a relationship between a picture (IO 202) containing metadata 206 identifying an electronic camera (first RWE 220) and a user (second RWE 224) of which the system is known to be the owner of the camera 220. Such owner information may be determined, for example, from one or the other of the IOs 224, 226 associated with the camera 220.

FIG. 2 also illustrates metadata 214 associating an IO 202 with another IO 230. The IO230 itself is associated with three other IOs 232, 234, 236, which in turn are associated with different RWEs 242, 244, 246. This portion of fig. 2 may, for example, describe the relationship between music files (IOs 202) that contain metadata 206 identifying a digital rights file (first IO 230) that defines the scope of usage rights associated with this music file 202. Other IOs 232, 234, 236 are other music files associated with usage rights and currently associated with a particular owner (RWE 242, 244, 246).

FIG. 3 illustrates a conceptual model of the W4 COMN. W4COMN 300 creates a provisioning messaging infrastructure in the form of a global logical network cloud that is conceptually subdivided into networked clouds of each of 4W: who, Where, What, and When. In the Who cloud 302 are all users, whether acting as senders, recipients, data points or confirmation/credential sources, and user agents in the form of user program processes, devices, agents, calendars, and the like. In the Where cloud 304, are all physical locations, events, sensors, or other RWEs associated with a spatial reference point or location. The When cloud 306 includes natural time events (i.e., events not associated with a particular location or person, such as dates, times, seasons) as well as collective user time events (holidays, anniversaries, elections, etc.) and user-defined time events (birthdays, smart timing programs). What cloud 308 includes all known data accessible to the W4 COMN-web or private, business or user-including, for example, environmental data such as weather or news, RWE generated data, IO and IO data, user data, models, processes and applications. Thus, conceptually, most of the data is contained in the What cloud 308.

Since this is only a conceptual model, it should be noted that some entities, sensors, or data will naturally exist in multiple clouds at different times or simultaneously. Further, some IOs and RWEs can be composite in that they combine elements from one or more clouds. Such composites can be classified or unclassified as appropriate to assist in determining associations between RWEs and IOs. For example, events consisting of location and time may be equally classified in the When cloud 306, the What cloud 308, and/or the Where cloud 304.

The W4 engine 310 is the center of the W4 COMN's central intelligence for making all decisions in the W4 COMN. An "engine" as referred to herein is a software, hardware, or firmware (or combination thereof) system, process, or function to describe (with or without human interaction or augmentation) the execution or assistance of the processes, features, and/or functions described herein. The W4 engine 310 controls all interactions between each layer of the W4COMN and is responsible for running any approved user or application target enabled by the applications operated or collaborated by the W4 COMN. In one embodiment, the W4COMN is an open platform on which anyone can write applications. To support this, it includes standard publishing APIs for request synchronization, disambiguation, user or topic addressing, access rights, prioritization or other value-based ranking, intelligent scheduling, automation, and topic, social, spatial, or temporal alerts (etc.).

One function of the W4COMN is to collect data about all communications and interactions conducted via the W4COMN, which can include storing a copy of the IO and information identifying all RWEs and other information related to the IO (e.g., who, what, when, where information). Other data collected by the W4COMN can include information about the status of any given RWE and IO at any given time, such as location, operating status, conditions monitored (e.g., current weather conditions monitored for an RWE as a weather sensor, or its current location based on the cellular tower it is contacting for an RWE as a cellular phone), and current conditions.

The W4 engine 310 is also responsible for identifying RWEs and relationships between RWEs and IOs from data and communication streams passing through the W4 COMN. The function of identifying RWEs associated with or implied by actions performed by IOs and other RWEs is referred to as entity extraction. Entity extraction includes both simple actions, such as identifying the sender and receiver of a particular IO, as well as more sophisticated analysis of the data collected and/or available by the W4COMN, such as determining when and where a message lists an upcoming event based on its context and associating the event with the sender and receiver(s) of the message, or determining that an RWE is stuck in a traffic jam based on its location correlating to the conditions of traffic monitors at the same location.

It should be noted that when extracted from an IO execution entity, the IO may be an opaque object with only W4 metadata related to the object (e.g., creation date, owner, recipient, RWE sent and received, IO type, etc.) and no knowledge about the interior of the IO (i.e., the actual primary or object data contained within the object). Knowing the contents of the IO does not prevent the collection of W4 data about the IO (or RWE). The content of the IO may also be used for entity extraction if known (if available), but entity extraction is performed by the network based on the available data regardless of the available data. Similarly, W4 data extracted around an object can be used to imply attributes about the object itself, while in other embodiments full access to IOs is possible, so RWEs can also be extracted by analyzing the contents of the object, e.g., strings within an email are extracted and associated as RWEs for determining relationships between senders, users, subjects, or other RWEs or IOs affected by the object or process.

In one embodiment, the W4 engine 310 represents a set of applications that run on one or more computing devices that are nodes of the W4 COMN. For purposes of this disclosure, a computing device is a device that includes a processor and memory for storing data and executing software (e.g., applications) that perform the described functions. The computing device may have an operating system that allows software applications to be run to manipulate data.

In the illustrated embodiment, the W4 engine 310 may be one or a group of distributed computing devices, such as a general purpose Personal Computer (PC) or a dedicated server computer, connected to the W4COMN by suitable communications hardware and/or software. Such a computing device may be a single device or a group of devices working together. The computing device may have any number of program modules and data files stored in local or remote mass storage devices and local memory (e.g., RAM) of the computing device. For example, as mentioned above, the computing device may include an operating system suitable for controlling the operation of a networked computer, such as the WINDOWS XP or WINDOWS SERVER operating systems from MICROSOFT CORPORATION.

Some RWEs may also be computing devices such as smart phones, web-enabled appliances, PCs, laptop computers, and Personal Digital Assistants (PDAs). The computing device may be connected to one or more communication networks, such as the internet, a public switched telephone network, a cellular telephone network, a satellite communication network, a wired communication network (such as cable television), or a private network. The computing device may connect to any such network via a wired data connection or a wireless connection such as a wi-fi, WiMAX (802.36), bluetooth, or cellular phone connection.

The local data structures, including the discrete IOs, may be stored on a mass storage device (not shown) connected to or part of any of the computing devices described herein, including the W4 engine 310. For example, in one embodiment, the data backbone of the W4COMN described below includes a plurality of mass storage devices that maintain IOs, metadata, and data necessary to determine relationships between RWEs and IOs as described herein. The mass storage device includes some form of computer readable media and provides non-volatile storage of data and software for retrieval and subsequent use by one or more computing devices. Although the description of computer-readable media contained herein refers to a mass storage device (e.g., a hard disk or CD-ROM drive), it should be appreciated by those skilled in the art that computer-readable media can be any available media that can be accessed by a computing device.

By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer.

FIG. 4 illustrates the functional layers of the W4COMN architecture. At the lowest level, referred to as the sensor layer 402, is a network 404 of actual devices, users, nodes, and other RWEs. Provisioning network nodes to use them as sensors includes known techniques such as web analytics, GPS, cell tower ping, usage logs, credit card transactions, online purchases, explicit and implicit user profile profiling by behavioral targeting, search analytics, and other analytical models for optimizing specific network applications or functions.

The next layer is a data layer 406 in which the data generated by the sensor layer 402 is stored and cataloged. This data may be managed by the network of sensors 404 or by a network infrastructure 408, the network infrastructure 408 being built on top of a provisioned network of users, devices, agents, locations, processes and sensors. Network infrastructure 408 is the core, under-shell network infrastructure that includes the hardware and software necessary to receive and transmit data from sensors, devices, etc. of network 404. It also includes the processing and storage capabilities necessary to meaningfully catalog and track the data created by the network 404.

The next level of the W4COMN is a user profile profiling layer 410. This layer 410 may be further distributed between the network infrastructure 408 and user applications/processes 412 running on the W4 engine or different user computing devices. Personalization is enabled in the user profile profiling layer 410, which acts as the user profile profiling layer 410 of the W4COMN, over any single communication channel and mode, or a combination thereof, including email, IM, text (SMS, etc.), photo blogging, audio (e.g., phone calls), video (teleconferencing, live broadcasts), gaming, data trusted processes, security, credentials, or any other W4COMN process invocation on available data.

In one embodiment, the user profile profiling layer 410 is a logic-based layer above all sensors to which sensor data is sent in the most primitive form to be mapped and placed into the W4COMN data backbone 420. The data (collected and refined, related and deduplicated, synchronized and disambiguated) is then stored in one or a set of related databases, which are available to all processes of all applications approved on the W4 COMN. All network originated actions and communications are based on fields of the data backbone, and some of these actions are: they themselves become records somewhere in the backbone (e.g., invoicing), while others (e.g., fraud detection, synchronization, disambiguation) can be done without affecting the profiles and models within the backbone.

Actions originating from anything other than the network (e.g., RWEs such as users, locations, agents, and processes) come from the application layer 414 of the W4 COMN. Some applications may be developed by the W4COMN operator and appear to be implemented as part of the communication infrastructure 408, such as an email or calendar program, because of how closely they work with the sensor processing and user profile profiling layer 410. The applications 412 also act somewhat as sensors in that they generate, through their actions, data relating to any data created or available as a result of the application running, which is returned to the data layer 406 via the data backbone.

The application layer 414 also provides a personalized User Interface (UI) based on device, network, operator, and user-selected or security-based customizations. Any UI, if configured to provide data back to the network regarding user interactions or actions, may operate within the W4 COMN. This is the basic sensor functionality of any W4COMN application/UI, and while the W4COMN can cooperate with applications/UIs that are not provisioned, it is just a delivery role, and those applications/UIs will not be able to provide any data (let alone the rich data otherwise available from W4-capable devices).

In the case of a W4COMN mobile device, the UI may also be used to validate and disambiguate incomplete W4 data in real time, as well as correlate, triangulate, and synchronize sensors for other nearby enabled or disabled devices. At some point, the network effect of enough enabled devices allows the network to collect complete or near complete data (sufficient for profiling and tracking) for the devices that are not enabled, as it periodically intersects and is sensed by the enabled devices in their real-world locations.

Above (and sometimes hosted within) the application layer 414 is communication delivery network(s) 416. This may be operated by the W4COMN operator or be an independent third party operator service, but in either case it functions to deliver data via synchronous or asynchronous communications. In each case, the communications delivery network 416 will send or receive data (e.g., http or IP packets) on behalf of a particular application or network infrastructure 408 request.

The communication delivery layer 418 also has elements that act as sensors, including W4 entity extraction from phone calls, emails, blogs, etc., and delivery of specific user commands within the network context, e.g., "save and prioritize this call" spoken before the call may trigger a recording of a previous conversation to be saved and cause the W4 entity within the conversation to be analyzed in the personalization/user profile profiling layer 410 and promoted in the weighted prioritization decision.

FIG. 5 illustrates an embodiment of an analysis component of the W4 engine shown in FIG. 3. As described above, the W4 engine is responsible for identifying RWEs and relationships between RWEs and IOs from data and communication streams passing through the W4 COMN.

In one embodiment, the W4 engine connects, coordinates, and provisions all network participants through a series of sub-engines that perform different operations in the entity extraction process. One such sub-engine is a home engine 504. The home engine 504 tracks real-world ownership, control, publishing, or other conditional rights of any RWE in any IO. Whenever the W4 engine 502 detects a new IO, for example, through the creation or transmission of a new message, a new transaction record, a new image file, or the like, ownership is assigned to the IO. The home engine 504 creates this ownership information and also allows this information to be determined for each IO known to the W4 COMN.

The W4 engine 502 also includes an association engine 506. The correlation engine 506 serves two functions: first, identify associated RWEs and IOs and their relationships (e.g., by creating a combined graph of any combination of RWEs and IOs and their attributes, relationships, and reputations within a context or situation), and second, as a sensor analysis preprocessor for events of interest from any internal or external source.

In one embodiment, the functionality of the correlation engine 506 to identify associated RWEs and IOs is accomplished by plotting the available data into a graph. In this embodiment, a histogram of all RWEs and IOs is created from which graph-based associations can be made. The act of drawing a graph or creating a histogram is oneComputer science methods, i.e. identifying the distribution of data in order to identify relevant information and make associations between data. In a more general mathematical sense, a histogram is a map m_iIt counts the number of observations that fall into various separate categories (called bins), and the graph of a histogram is simply one way to represent a histogram. Relationships between RWEs, IOs and other parameters can be identified by selecting each IO, RWE and other known parameters (e.g., time, date, location, etc.) as distinct cylinders and mapping the available data.

In one embodiment, the W4 data is processed and analyzed using data models that, rather than treating the data as abstract signals stored in a database, are treated as IOs that represent RWEs that actually exist, or are about to exist in real space, real time, and are real people, objects, places, times, and/or events. As such, the data model of W4IO representing W4 RWEs (Where/When/Where/What) will not only model the signals recorded from or about RWEs, but will also represent these RWEs and their interactions in a manner that models the availability and constraints of entities and activities in the physical world. One notable aspect is modeling data related to RWEs embodied and located in a real-world context such that similarity calculations, clustering, distances, and inferences take into account the states and actions of the RWEs in the real-world and the context and patterns of those states and actions.

For example, for time data, the calculation of the time distance and the similarity in the W4 data model cannot treat time as a linear function only. The temporal distance and similarity between two times depends not only on the absolute linear time difference between them (e.g., the number of hours between "11 month 20 days, tuesday, pacific time 4:00 pm" and "11 month 20 days, tuesday, pacific time 7:00 pm"), but more on the context and activity that determine the importance of these times in the physical world and the other W4 RWEs (people, places, objects and events, etc.) associated with them. For example, in terms of distance and similarity, "11 month 20 day, tuesday, pacific time 4:00 pm" and "11 month 27 day, tuesday, pacific time 4:00 pm" may be modeled closer together than "11 month 20 day, tuesday, pacific time 4:00 pm" and "11 month 20 day, tuesday, pacific time 7:00 pm" in the W4 time data model, since this is a dinner in the home at a 4:00pm weekly meeting at the company versus 7pm at tuesday. The contextual and periodic pattern in time is important for modeling the temporal data in the W4 data model.

A simpler time data modeling problem is modeling the various periodic patterns of daily life, such as day and night (and sub-periods therein, such as morning, noon, afternoon, night, etc.), as well as the differences between weekdays and weekends. In addition, significant periods such as the seasons of the year and significant events such as holidays also affect the modeling of the time data to determine similarities and distances. In addition, modeling temporal data representing the IOs of an RWE should correlate temporal data, spatial and weather data to account for the physical conditions of time at different places on the Earth. The duration of the day varies at different latitudes, or even is reversed between the northern hemisphere and the southern hemisphere. Similar contextual and structural data modeling problems arise when modeling RWEs from and data about RWEs for people, groups of people, objects, places, and events.

Using an appropriate data model of IOs representing data from or about RWEs, a variety of machine learning techniques can be applied to analyze W4 data. In one embodiment, the W4 data can be modeled as a "feature vector," where the vector includes not only raw sensing data from or about the W4RWE, but also higher order features that take into account the contextual and periodic pattern of the state and action of the W4 RWE. Each of these features in the feature vector may have a numeric or symbolic value that may be compared to other numeric or symbolic values in the feature space to determine a degree of similarity. Each feature may also be modeled with an additional value from 0 to 1 (a certainty value) to represent the probability that the feature is true. By modeling W4 data about RWEs in features and higher order features with or without certainty values in a manner that takes into account their availability and constraints in context and patterns in the physical world, this data (whether represented in feature vectors or by other data modeling techniques) can then be processed to determine similarities, differences, clustering, hierarchical and graphical relationships, and speculative relationships between features and feature vectors.

A wide variety of statistical and machine learning techniques can be applied to the W4 data, from simple histograms to Sparse Factor Analysis (SFA), Hidden Markov Models (HMM), Support Vector Machines (SVM), bayesian methods, and the like. Such learning algorithms may be populated with data models containing features and higher order features that not only represent the "content" of signals stored as IOs (e.g., raw W4 data), but also model the context and patterns of RWEs that exist, or will exist in the physical world from which these data were captured.

As a preprocessor, the correlation engine 506 monitors information provided by RWEs to determine if any conditions are identified that can trigger actions on the part of the W4 engine 502. For example, if a delivery condition is associated with a message, when association engine 506 determines that the condition is satisfied, it may send appropriate trigger information to W4 engine 502 that triggers delivery of the message.

The attention engine 508 provisions all suitable network nodes, clouds, users, applications, or any combination thereof, and includes close interaction with both the correlation engine 506 and the attribution engine 504.

FIG. 6 illustrates an embodiment of the W4 engine showing the different components within the sub-engines generally described above with reference to FIG. 4. In one embodiment, the W4 engine 600 includes a focus engine 608, a home engine 604, and a correlation engine 606 that have several sub-managers based on basic functionality.

The attention engine 608 includes a message fetch and generation manager 610 and a message delivery manager 612 that work in close conjunction with both a message matching manager 614 and a real-time communication manager 616 to deliver and provision all communications on the W4 COMN.

The attribution engine 604 works in conjunction with all other modules within the user profile manager 618 to identify, process/authenticate, and represent ownership and entitlement information related to RWEs, IOs, and combinations thereof.

The correlation engine 606 dumps data from its two channels (sensors and processes) into the same data backbone 620, which data backbone 620 is organized and controlled by a W4 analytics manager 622, and includes user logs 624, focus place logs 626, web index and environment logs 618, e-commerce and financial transaction information 630, search indexes and logs 632, sponsor content or conditions, advertising copy, and any and all other data used in any W4COMN process, IO, or event. Because the W4COMN may store an amount of data, the data backbone 620 includes a number of database servers and data storage devices in communication with the W4COMN to provide sufficient storage capacity.

As described above, the data collected by the W4COMN includes spatial data, temporal data, RWE interaction data, IO content data (e.g., media data), and user data (including explicitly provided and inferred social and relationship data). Spatial data can be any data that identifies a location associated with an RWE. For example, the spatial data may include any passively collected location data, such as cell tower data, Global Packet Radio Service (GPRS) data, Global Positioning Service (GPS) data, WI-FI data, personal area network data, IP address data, and data from other network access points, or actively collected location data, such as location data entered by a user.

Temporal data is time-based data (e.g., a timestamp) that relates to a particular time and/or event associated with a user and/or electronic device. For example, the time data may be passively collected time data (e.g., time data from a clock present on the electronic device or time data from a network clock), or the time data may be actively collected time data, such as time data input by a user of the electronic device (e.g., a user-maintained calendar).

The interaction data may be any data associated with user interaction (whether active or passive) of the electronic device. Examples of interaction data include inter-human communication data, media data, relationship data, transaction data, and device interaction data, all of which are described in more detail below. Table 1 below is a non-exhaustive list of examples that include electronic data.

TABLE 1 example of electronic data

For interactive data, communication between any RWEs can generate communication data that is communicated via the W4 COMN. For example, the communication data can be any data associated with an incoming or outgoing Short Message Service (SMS) message, an email message, a voice call (e.g., a cellular telephone call, a voice over IP call), or other type of inter-human communication related to an RWE, such as information about who is sending and receiving a communication. As described above, the communication data may be correlated with, for example, temporal data to infer information about the frequency of communications, including a centralized communication pattern, which may indicate user activity information.

Logical and IO data refer to data contained by and associated with the IO, such as creation time, owner, associated RWE, time of last access to the IO, and the like. If the IO is a media object, the term "media data" may be used. The media data may include any data related to presentable media, such as audio data, visual data, and audio-visual data. For example, the audio data may be data related to downloaded music, such as genre, artist, album, and the like, and include data on ringtones, ring-back tones, purchased media, playlists, and shared media, and the like. The visual data may be data related to images and/or text received by the electronic device (e.g., via the internet or other network). The visual data may be data related to images and/or text sent from and/or captured at the electronic device. The audiovisual data may be data associated with any video captured at, downloaded to, or otherwise associated with the electronic device. Media data includes media presented to a user via a network, such as use of the Internet, and includes data related to text (e.g., search terms) entered and/or received by the user using the network, as well as data related to interaction with the network media, such as click data (e.g., advertisement banner clicks, bookmarks, click patterns, etc.). Thus, the media data may include data related to the user's RSS feeds, subscriptions, group memberships, game services, reminders, and the like. Media data also includes non-network activities such as image capture and/or video capture with an electronic device, such as a mobile phone. The image data may include metadata added by the user or other data associated with the image, such as, for example, in the case of a photograph, the location at which the photograph was taken, the direction from which the photograph was taken, the content and time of the photograph, and so forth. As described in more detail below, the media data may be used, for example, to infer activity information or preference information, such as culture and/or purchase preference information.

Relationship data can include data about the relationship of an RWE or IO to another RWE or IO. For example, the relationship data may include user identity data, such as gender, age, race, name, social security number, photos, and other information associated with the user's identity. The user identity information may also include an email address, a login name, and a password. The relationship data can also include data that explicitly identifies an associated RWE. For example, the relationship data for a cellular telephone may indicate the user who owns the cellular telephone and the company that provides service to the telephone. As another example, the relationship data for the smart vehicle may identify the owner, a credit card associated with the owner for payment of the electronic toll, those users permitted to drive the vehicle, and a service station for the vehicle.

The relationship data may also include social networking data. Social network data includes data relating to any relationship explicitly defined by a user or other RWE, such as data relating to a user's friends, family, colleagues, business relationships, and so forth. The social network data may include, for example, data corresponding to an electronic address book maintained by the user. Relationship data may be correlated with, for example, location data to infer social network information, such as primary relationships (e.g., user-spouse, user-child, and user-parent relationships) or other relationships (e.g., user-friend, user-co-worker, user-business associate relationships). Relationship data may also be used, for example, to infer activity information.

The interaction data may also include transaction data. The transaction data may be any data associated with a commercial transaction engaged in by or at the mobile electronic device, such as vendor information, financial institution information (e.g., bank information), financial account information (e.g., credit card information), merchandise information and cost/price information, and purchase frequency information, among others. The transaction data may be used, for example, to infer activity and preference information. The transaction information may also be used to infer the types of devices and/or services that the user owns and/or may be interested in.

The interaction data can also include device or other RWE interaction data. Such data includes data generated by interactions between users on the W4COMN and RWEs, as well as interactions between RWEs and the W4 COMN. RWE interaction data can be any data regarding an RWE's interaction with an electronic device that is not included in any of the above categories, such as habitual patterns associated with the use of electronic device data by other modules/applications, such as data regarding which applications are used on the electronic device and how often and when those applications are used. As described in more detail below, the device interaction data may be correlated with other data to infer information about user activity and patterns associated therewith.

Table 2 below is a non-exhaustive list including examples of interaction data.

TABLE 2 example of interaction data

Augmented delivery of messages over W4COMN or other networks

One notable aspect of the W4COMN is the ability to use the W4 data to allow users to adjust when, how messages are delivered to other users or their agents. Information about the W4 entity obtained from any source or communication channel can be used as a basis for delivery conditions for any message delivered via the W4COMN over any communication channel that cooperates with the W4 COMN.

Delivery of messages is a network Personal Information Management (PIM) operation that allows explicit and implicit automation of W4COMN circuits, processes and events through logic-based conditions, including means for expressing, weighting and prioritizing senders, recipients and delivery conditions in a W4 analysis process for testing delivery conditions or network environment conditions. The W4 message delivery includes a message generated by a user, process, or system for any intersection of a subject variable, a spatial variable, a temporal variable, and/or a social variable.

To continue the "When, What, When, Where," concept described above, W4 message delivery allows messages to be delivered from any "When, What, When, Where," When "delivery conditions are detected to occur to any" When, What, When, Where, "delivery conditions are detected to occur. Table 3 below provides a matrix of some examples of different "Who, What, When, Where" combinations that may be used in W4 message delivery. The list in table 3 is not complete or exhaustive, but is provided to give a concept with respect to the many different message delivery options offered by the W4 COMN.

Table 3-example of W4 message delivery

The list provided in table 3 is a very limited list of possibilities for message delivery via the W4 COMN. It should be noted that the delivery condition can be a simple condition, such as a time or detection of a designated RWE at a certain location, or a more complex condition based on the occurrence of multiple conditions, either simultaneously or in a particular order, such as delivery to RWEs near a designated location only on the day of a baseball game. In the example of a baseball game, the baseball game may be considered an event, with its own unique W4 identifier, associated with a location and a time period. For events such as sporting events, meetings, holidays, and the like, one or more IOs may exist on the W4COMN or electronic calendar, which are agents of the event from which the time, location, and other relevant data of the event may be obtained.

In a broad sense, W4 message delivery allows messages (which may be any IO, including text-based messages, audio-based messages (e.g., voicemail or other audio such as music), or pre-recorded messages over video) to be delivered according to delivery conditions based on any combination of available W4 data types, including topical data, spatial data, temporal data, and/or social data. Additionally, because the W4COMN coordinates the delivery of messages via multiple communication channels and through multiple devices and other RWEs, it allows the communication channel for delivering messages to be dynamically determined upon detecting that delivery conditions are satisfied. Examples include social alarm clocks, place-based messages, social proximity-based messages, and time-offset message delivery, to name just a few applications of the W4 message delivery functionality.

The predetermined set of W4 delivery conditions may be packaged and provided to the user in the form of a common kit, such as a parent's kit, a boss's kit, a vehicle maintenance kit, and so forth. These suites may include predetermined message content, delivery conditions, and delivery condition templates that allow the user to quickly construct delivery conditions for the message that will be easily and clearly interpreted by the W4 COMN's message delivery subsystem.

Fig. 7A illustrates elements of an embodiment of a W4 engine suitable for performing W4 enhanced message delivery as described herein. The W4 engine 700 includes the correlation engine 506, the attribution engine 504, and the attention engine 508 as described above. The W4 engine 700 has a message fetch manager 702 that is adapted to receive messages and their associated delivery conditions from senders via various communication channels in cooperation with the W4 COMN. The W4 engine 700 also includes a message delivery manager 704, an identification engine 706, and a message enhancement manager 708.

As described above, it should be understood that multiple RWEs and IOs can be associated with a single message as senders. For example, a user may create and send an email message with delivery conditions using a laptop computer. In addition, the laptop is an RWE with its own unique W4 identifier. The email application on the laptop may be tracked as an IO with its own W4 identifier. In one embodiment, some or all of the user, laptop computer, and email application may be considered to be senders of IOs as email messages. In this case, the user may be considered the originating sender, and the laptop computer and email application may be considered agents of the originating sender. The concept of an agent has been described above and is particularly important herein where it is expected that human actors, whether as senders, recipients or entities dependent on delivery conditions, will be primarily known to the W4COMN through information obtained from their agents, e.g., agent RWEs (such as their smartphones, computing devices, sensors, smart vehicles, home phones, etc.) and agent IOs (such as email accounts, communication software, credit card accounts, data objects containing data generated by RWEs, data objects containing data about RWEs or events, etc.).

In one embodiment, such a determination of the sender of the message, including a determination of who the user (if any) should be considered the original sender, can be performed by the attribution engine 504 as described above. It should also be noted that some messages may be sent programmatically by the process, for example automatically during the running of the program, so that there is no human sender to identify, only the sender IO. In addition, home engine 504 parses the received message and identifies W4 data for RWEs and IOs associated with the message. Alternatively, the home engine 504 can identify only the sending RWE that actually placed the message in the W4COMN, while any other associated RWEs (e.g., proxies and/or originating senders) can be identified by the message fetch manager 702 in conjunction with the correlation engine 506.

The W4 engine 700 also includes a recognition engine 706 adapted to recognize message types and match content output from the correlation engine 506 to message IO. Recognition engine 706 analyzes a range of possible extensions in view of the W4 entities within each message type.

Upon receiving a message with delivery conditions, message fetch manager 702 identifies the recipient of the message and the delivery conditions as described below. This may include requesting the correlation engine 506 to correlate the recipient's channel-specific identifier with other W4 data in order to identify the target human recipient (if any) and any agents for that recipient. Further, the same information may need to be determined for RWEs identified in the delivery conditions.

It should be understood that any human or non-networked entity that is the sender, recipient, or subject of a delivery condition of a message may be identified only by a proxy RWE or IO. For example, an email may be sent by or to "john.smithyahoo.com", or a telephone call may be directed to "(720) 555-. In both cases, the identifiers used to identify humans (i.e., "john. smithyahoo. com" and "(720) 555-. Based on W4 data known to the W4COMN, these identifiers of agents can be analyzed by, for example, the correlation engine 506 to determine unique W4 identifiers of RWEs accessed by, represented by, or operated on by agent RWEs or agent IOs. For example, "John smithyahoo. com" and "(720) 555-.

Message enhancement manager 708 combines the matched content and message IO output from recognition engine 706 for delivery to the recipient. The W4 engine 700 also includes a message delivery manager 704 that controls the delivery of messages. In one embodiment, the message delivery manager 704 records the delivery conditions of the message and monitors the W4 data for the occurrence of delivery conditions. The message delivery manager 704 then delivers the message to the recipient when/if the delivery condition is satisfied. This can include selecting a delivery route or communication channel and selecting an appropriate proxy RWE (if applicable) to deliver the message, possibly including reformatting the message for the selected RWE. For example, when a recipient is at a specified location, for an email message to be delivered to the recipient, the email message may be reformatted into text for display via a cellular telephone or an in-vehicle display device that is one of the recipient's proxy devices and delivered to the device upon determining that the device is at the specified location. Similarly, a voicemail to be delivered to a recipient when a certain team wins a game may be reformatted and sent to any agent device that the recipient may be using when the team wins the game. Thus, the voicemail may be delivered to a cellular telephone number, a voicemail inbox, an email inbox (as an attachment to an email) or a work telephone number depending on what the recipient is doing when the team wins the game.

To determine when delivery conditions are met, the message delivery manager 704 may utilize the correlation engine 506 to monitor the W4 data. Additionally, the determination that the delivery condition is satisfied may simply be a graphical drawing and identification of relationships through associations between IOs and RWEs known based on the W4COMN as performed in the association engine 506. The relationship may be determined in response to a request to deliver the message, triggered by some other input, or may be automatically determined periodically by the correlation engine and stored for later use.

Fig. 7B illustrates another embodiment of the above-described W4 engine 700 that appears in fig. 7A. The W4 engine 700 is associated with a content server 710. The content server 710 may be one or more servers in one or more locations that are essentially the source of the content (whether media content, advertising content, graphics, video, or text) that a third party is willing to pay for to have the content embedded in the message as part of an augmented message to a network operator or intermediary. In this manner, the augmented messaging system may monetize a particular type of content via the W4 engine 700. Examples of content are commercial data, advertising data, audio data, video data, literature data, and all other types of content that may be entered into a digital message.

The additional content provides an opportunity for monetization of the augmented messaging system described herein. Thus, for example, augmented content may take the form of advertising content or paid media placement that will be highly targeted to recipients based on the context of the message, information about the recipient known or available to the network, and the sender-to-recipient or recipient-to-event relationship described in the message, in the context of the system described herein. Thus, the augmented message will contain content by which revenue can be obtained for the system administrator or operator providing the augmented messaging platform.

Alternative embodiments of the content server 710 may place the content server 710 within or as part of the W4 engine 700. In this arrangement, the content server 710 will interact directly with the message enhancement manager 708 during the combining of the message IO and the enhanced content in preparation for delivery.

FIG. 8 illustrates an embodiment of a method of delivering messages over a network based on social data, temporal data, spatial data, and topical data of entities on the network. In the embodiments described below, the operations described may be performed by one or more of the various components, engines, and managers described above, depending on how the architecture is implemented. Further, sub-engines may be created and used to perform specific operations to improve the performance of the network, if necessary.

As described above, one aspect of the W4COMN that allows for conditional message delivery is the collection and maintenance of W4 data in real-time from RWEs interacting with the network. In one embodiment, this collection and maintenance is a separate operation 899 of the W4COMN, so that current W4 social data, temporal data, spatial data, and topical data are always available for testing delivery conditions. Further, part of this data collection operation 899 includes determining associations and ownership of different RWEs and different IOs as described above, including prioritizing between particular groups and subgroups of RWEs and IOs. Thus, each IO is owned/controlled by at least one RWE that has a known unique identifier on the W4COMN and each IO may have many associations with other RWEs that are known to the W4 COMN.

In the illustrated embodiment, the method 800 is initiated when the W4COMN detects a message with a delivery condition in a receive delivery request operation 802. Such requests may be generated by software on a user-operated computing device, by an automated process, or by a "dumb" device such as a cell phone or sensor. As described above, one or more RWEs can be identified as senders of messages. This identification may be from the data of the message, the source of the request, or a combination of both. Further, inferences can be made regarding the user being the sender of the message based on the W4 data for known devices or software senders of the message, as described above.

The delivery request will also identify one or more recipients of the message. As described above, each recipient can be identified by a channel-specific identifier of the recipient's proxy RWE. Thus, there may be multiple recipients associated with a message, similar to the sender's scenario. For example, the recipient of the email may be identified as "bill. Using the W4 data, it may be determined that the user associated with the email address has multiple agents on the W4COMN, including, for example, an email account identified by "bill. In one embodiment, a request to deliver a message to a recipient of an agent determined to be a user (or other RWE, such as an enterprise or location) can be interpreted as a request to deliver a message to a user (or other RWE) accessible via the agent, as described below.

In one embodiment, the delivery condition may be part of the delivery request or may be included in the IO that constitutes the message (e.g., as data or metadata). In this case, the address string or information is associated with the IO to be delivered. Such address strings or address information may also be part of the IO to be delivered. The request may also occur when an address string is detected, such as a user entering the address string into a field in an email composition screen or speaking the address string into a microphone on the device.

In one embodiment, the delivery conditions may be specified by the sender of the message, which may be an RWE (typically a user), or an IO (e.g., a process running on a computing device). Any suitable manner of selecting and associating delivery conditions with the message may be used, so long as the W4COMN can recognize the resulting data embodying the delivery conditions. For example, for an email message, delivery conditions may be entered by the user into a delivery options interface provided by the email application, such delivery conditions then being stored as metadata of the message: this metadata is then decoded by the W4COMN to identify the delivery conditions. For a telephone call, the message delivery system may use a voice or keyboard data entry system to allow the caller to assign or select delivery conditions from an audible menu to the voice message. Delivery conditions (which may also be subsequent and/or additional in alternative embodiments) may also be automatically generated and added to the message, for example by an application such as a parental control application that sends the message after certain activities or content are detected. Other methods of associating delivery conditions with a message are possible, and any suitable method may be used in conjunction with embodiments of the systems and methods described herein.

In the method 800, RWEs and IOs can be identified in delivery conditions by any identifier, which can be unique or non-unique, communication channel specific or global, so long as the identifier can be resolved by the W4COMN to its intended target RWEs or IOs. Resolution of the channel-specific identifier may be accomplished by correlating the channel-specific identifier with other W4 data. Non-unique identifiers (e.g., identifiers such as "mom", "father", "Debby", "Bob", "Starbucks") may have to be disambiguated based on known W4 data about the sender and the message to be delivered, and any suitable disambiguation method may be used for this purpose.

In this embodiment, a message with delivery conditions may be considered detected when it is received from (one or more), but the reader will understand that the message may not actually be sent when the delivery conditions are received. It is contemplated that in most cases, any sender that can be found is already known to the W4COMN and has a unique W4 identifier and at least one communication channel-specific address (which is another form of unique identifier).

As described above, the receive delivery request operation 802 may include receiving an actual IO (e.g., a message file or a short piece of text) from an RWE or IO (e.g., email software run by the RWE) that is to be sent as indicated by an address string or pointer to an IO at another address or location. The IO may contain data such as text or content of the communication as well as additional information in the form of metadata. The contained data can be evaluated to identify delivery conditions, additional RWEs associated with the message (e.g., people listed in the text of the message but which are neither the sender nor the recipient), other IOs contained in the message (e.g., hyperlinks to IOs, attachments, etc.), and any subject matter discussed in the message.

The receive delivery request operation 802 may be considered to occur at any point in the delivery chain within the W4COMN, such as by any one of the engines used for IO fetching, routing or delivering. For example, depending on how the implementer of the W4COMN chooses to implement network functionality, any of a message fetch and generation manager, a user profile manager, a message delivery manager, or any other engine or manager in the communication delivery chain of the W4COMN may receive and initially analyze the message and route the information to the correlation engine and addressing engine.

Upon detecting a delivery condition associated with the message, the delivery condition is analyzed in a delivery condition identifying operation 804. The delivery condition identifying operation 804 includes identifying each RWE in the delivery conditions and what the actual delivery conditions are for those RWEs. This may require parsing a string containing the delivery conditions or some other analysis of the data or metadata of the message. For example, an IO may be sent by a sender in email form and addressed to a recipient (identified by an email address) with delivery conditions that the message should only be delivered if the recipient is at/with a specified RWE (e.g., another person, a location such as a park, or a business such as a grocery store, laundromat, coffee shop, etc.). In this example, the delivery condition identifying operation 804 would identify the recipient, the designated RWE, and a maximum distance or range of distances between the two that indicate that the delivery condition is satisfied (which may or may not be explicitly provided in the delivery condition).

As described above, any RWE in the recipient and delivery conditions can be a proxy for a user or other RWE. Identifying operation 804 includes determining whether the recipient and the delivery condition RWE are proxies for the message or actual targets. Further, if the designated RWE is a proxy, the identifying operation 804 also includes identifying any of the following RWEs: such RWEs can act as proxies for designated RWEs and designated RWEs can themselves act as proxies for any RWE that they proxy.

For example, given the relationship described in FIG. 1, the sender of the IO may specify the email address of user 102 as the recipient and the delivery conditions that require that the recipient (also identified by the email address) and user 144 (which may also be identified by some sort of agent identifier, such as a phone number or email address) must be together (i.e., co-located). By retrieving the W4 data for the email address, it may be determined that it is an agent of the user 102 and that the user 102 has many other agents available for identifying the location of the user 102, including the vehicle 106 and the cellular telephone 104. The identifying operation 804 will also identify the following IOs (e.g., IOs 122, 124, 126): from these IOs, current location information for each of the identified agents for the current location of user 102 may be obtained. This process is repeated for user 144 to identify user 144, whether user 144 has any agents, and where to obtain the current location information for each agent from the RWE identifier provided in the delivery conditions.

The identifying operation 804, in identifying the agent to be used to determine whether the delivery condition is satisfied, may distinguish the agent based on the data available to the agent and the delivery condition. For example, the user's working email account may be a proxy for the user, but if the current location data is not derivable from the user's use of the email account (e.g., the email account may be accessible from multiple devices and/or from multiple or any location), the working email account may not be identified as a proxy for the user's location. However, if the device that the user uses to access the working email account at any time can be identified and its location determined (e.g., a public computer on a network that the user uses to access their email account), then the device can be used as a proxy for the user's location because of its current relationship to the user's email account, although the device has never been used by the user before.

In one embodiment, for any identified RWE that is explicitly designated as a proxy for another RWE, the W4 engine can assume that the other RWE is the intended entity and replace the identified RWE with it. For example, an IO sent by a sender in email and addressed to a recipient (identified by an email address) with delivery conditions that the message should only be delivered if the recipient is at/with the designated RWE, the IO can be delivered when the recipient's cell phone (e.g., the recipient's proxy, but not necessarily the recipient's email account) is determined to be sufficiently close to the cell phone of the designated RWE (e.g., the recipient's proxy) and can also be delivered when both the designated RWE and the recipient RWE are determined to be in attendance at the same meeting/event based on message traffic, financial data (e.g., confirming an event ticket purchase or detecting a simultaneous sale at the same location), or smart calendar entries.

The identification of agents as described above can be explicit (e.g., designated as agents by their associated users or RWEs) or implicitly determined based on analysis of W4 data. As described above, such W4 data may be collected from messages, communications, and IOs previously obtained or processed from the W4COMN via a number of different communication channels and systems, including email and text-based communication channels, as well as any communication channel including audio data, including channels supporting telephony, Voice over Internet protocol (VoIP), and video communications such as video chat.

To determine implicit proxies, the W4 data can be plotted into a graph to determine what RWEs are relevant and how, and make probabilistic assumptions about the nature of relationships between RWEs based on this information. In one embodiment, associations are made for and between each RWE known to the W4COMN based on social, spatial, temporal, and logical data associated with each RWE. In a sense, the graphical drawing operation can be thought of as some form of comparison of social, spatial, temporal, and logical data of all RWEs retrieved to identify relationships between RWEs and other contextual similarities.

It should be noted that the determination of the implicit agent may be performed each time the delivery conditions are tested. This allows the appropriate agent to be dynamically determined at any time for any RWE. For example, in the week, a company vehicle or a company cellular phone may be considered a good proxy for the location of the recipient; but during weekends, a personal cell phone or personal vehicle may be considered a better proxy for the recipient than a working cell phone.

After identifying RWEs (including any proxy RWEs) that can be used to confirm the occurrence of a delivery condition, a delivery condition identifying operation 804 then identifies one or more data sources of data necessary to test the delivery condition. For example, if the delivery conditions have a location requirement, the recipient and the source of the data specifying the current location of the RWE will be identified. If the delivery condition has a time requirement, the recipient can be identified with a system clock or local time of the specified RWE. If the delivery condition has a condition related to a condition or state (e.g., a condition based on some identified current sensor reading or other condition such as current traffic conditions, current weather conditions, current speed, what the RWE is currently doing, current weather forecasts, occurrence of defined events, etc.), then the appropriate one or more data sources are identified that contain current information necessary to test these conditions so that messages can be delivered when it is determined that the current state of the RWE matches the specified state identified by the delivery condition.

For example, if the recipient is a user with a cellular telephone, the cellular telephone may be identified as a proxy for the recipient's current location. Identifying operation 804 will then identify the location of the data source from which the current location of the cellular telephone can be obtained. Such a data source may be maintained by, for example, a cellular service provider and accessible through its network. Alternatively, the cellular telephone may have a GPS locator so that the current location can be obtained from the cellular telephone itself. If the designated RWE is an enterprise, proxy RWEs may not be needed, but instead proxy IOs containing information about the enterprise, including its current location(s), can be identified.

Delivery condition identifying operation 804, as described above, also includes identifying how the delivery condition(s) are to be satisfied. For example, if a delivery condition requires that an RWE be located at a certain location, then a distance within a certain range between the RWE and the identified location will be considered "located" at that location in terms of the W4 COMN. Such a range may be predetermined by the operator of the W4COMN (e.g., "located" defined to be within 10 meters), provided or otherwise selected by the sender as part of the delivery conditions, or dynamically determined based on the accuracy required by the available data and location data or application suite or application.

If the delivery conditions have time requirements, then the sender has specified only a relative time (e.g., thursday 6:00am or christmas) instead of an absolute time (e.g., 11/8/2007, thursday, mountain standard time 6:00am), then location identification may be required. Unless otherwise indicated, the W4COMN may assume that all times are local to the sender or receiver, and when a future time is inaccurately specified, the sender may be prompted to give more information or the next possible match may be used.

If the delivery condition has a condition related to a condition or state, the appropriate condition is quantified so that it can be tested with data from the identified one or more data sources. Likewise, the exact condition may have been specified by the sender (e.g., "deliver if a wind greater than 40mph is detected in the home") or may need to be determined according to a less specific specification (e.g., "deliver if a strong wind is detected in the home"). For example, a delivery condition based on "heavy traffic" at a location will identify how heavy traffic is in terms of metrics monitored by and accessible from the identified traffic sensors. Thus, a threshold or range of values that is considered "heavy traffic" upon detection may be identified, or a definition retrieved. As another example, the delivery condition "when the recipient is playing a computer game" may be determined by identifying what software is a computer game and determining whether the software is running at any particular time. In addition, the quantification of the delivery conditions may be different for each instance of the delivery conditions, and may be determined independently based on the current context and the W4 data.

If the identifying operation 804 does not adequately identify any of the parameters, RWEs, and IOs as described above or identifies delivery conditions to a sufficient degree to determine when the delivery conditions are met, the sender may be notified and asked to provide additional clarification information. For example, in this case, the W4COMN may respond by prompting the sender of the message with a question such as: "do you say 'delivered to Debby' at grocery store refer to your wife Deborah or your sister Deborah? ". The prompt may include information derived from previous communications or other W4 data to help the sender confirm the proper identification of the recipient and delivery conditions.

Furthermore, in one embodiment, the delivery conditions may be specified very specifically by the sender when the message is created. For example, when it is detected that both the recipient's vehicle and cellular telephone are near a specified location, the sender may instruct the W4COMN to deliver a message to the recipient only via the recipient's cellular telephone, so that the sender can be confident that the message will not be delivered when, for example, the recipient is cycling past a gas station with their cellular telephone or when the sender is driving the recipient's vehicle past the location. This allows the sender to be very specific in defining the delivery conditions, which would be necessary if he wanted to do so or to properly deliver the message in the manner the sender intended.

After the delivery condition(s) and recipient(s) have been identified as described above, the delivery condition is tested in test operation 806. This would include examining the necessary data sources and comparing various data elements (e.g., current location, temperature, status, etc.) as needed to determine if the delivery conditions are met. In one embodiment, testing may include retrieving or requesting data from the identified data source. Test operation 806 may require only a simple comparison, for example, comparing two values, such as the current location, to determine if the values are within a specified range. The test operations 806 can be more complex, requiring complex calculations or simultaneous testing of multiple conditions related to many different RWEs, for example, and can include collecting and analyzing data from external sources.

Based on the results of test operation 806, a decision operation 808 determines whether the delivery condition(s) is satisfied. If the condition is not met, the delivery condition is retested by repeating test operation 806. With this repeated testing, the W4 data was monitored for the occurrence of delivery conditions. Retesting may be performed periodically on a fixed schedule or may be performed dynamically in response to external conditions, such as the receipt of new data on the W4 backbone in connection with RWEs implied by delivery conditions. The retesting may be conducted permanently until the condition is met, or until a predetermined maximum time period specified by the sender or the W4COMN is reached. If the message cannot be delivered before the maximum period of time is reached, the sender may be notified that the message was not delivered because the delivery condition was not met, e.g., "no message was delivered because Debby did not go to the grocery store within a specified period of time".

If decision operation 808 determines that the delivery condition is satisfied, then the message is sent to the recipient(s) in delivery operation 810. In one embodiment, if the identified recipient is a proxy RWE, the message can be delivered to the proxy RWE regardless of the identified conditions that satisfy the delivery conditions. For example, when a cell phone of a user associated with the identified email address is detected at the delivery condition location, an email may be sent to the email address. However, in this case, if the cellular phone does not have the email capability, it is likely that the user will not receive the email until later.

Alternatively, the method can select a communication channel and RWE for delivering the message based on conditions that trigger delivery. For example, if the message is an email and the delivery condition is that the recipient is at a location, upon detecting that the recipient's cellular telephone is at the location, the message may be reformatted for the recipient's cellular telephone, for example as an IM or SMS message, and sent to the cellular telephone via the cellular telephone network serving the cellular telephone. In such an embodiment, not only the identified recipient's proxy RWE, but all other proxies of the recipient (or other RWEs) are considered possible delivery routes to the recipient. This allows the W4 message delivery system to select the most appropriate delivery route/communication channel/agent combination when ultimately delivering a message, regardless of which delivery route/communication channel/agent was originally identified as the recipient or used by the sender to create and send a message to the W4 COMN.

Such a dynamic message delivery system allows for the delivery of messages with alternative but efficient means. For example, based on the W4 data, the user's best friend's cell phone may be selected as the user's (and the best friend's) agent, and the message to the user may be automatically delivered to the best friend's cell phone when the delivery conditions are met and it is determined that the user and the best friend's cell phone are co-located. In these cases, the message delivery method can quickly and efficiently deliver messages to devices that are completely unknown to the sender through a communication channel that is completely unknown to the sender. As another example, while traveling, if the user does not have access to email, an email message to the user may be automatically delivered to the user's colleagues.

Further, in one embodiment, the delivery channel and RWEs can be specified by the sender at the time the message is created. For example, the sender may instruct the W4COMN to only deliver a message to the recipient via the vehicle when it is detected that the vehicle is approaching a gas station, so that delivery of the message is not triggered when the recipient is cycling past the gas station with their cell phone. Alternatively, the sender may identify the W4COMN to dynamically select the communication route and recipient agent device so that when the delivery conditions occur, the message is delivered to the recipient in a manner that most likely causes it to reach the intended recipient user (rather than the user's agent).

As described above, the W4COMN network continuously contains and tracks updated information about senders and recipients, which are RWEs on the W4 COMN. Thus, messages sent by a first RWE sender may appear different when received by each RWE based on information known to the W4COMN about each RWE sender and recipient, based on information known to the W4COMN and used to augment each message passing through the W4COMN to each intended RWE recipient. Thus, for example, a sender of a message addressed to two recipients inviting them to have dinner at a japanese restaurant may type emails addressed to the first recipient and the second recipient, to mention: let us have a dinner at Nobu on thursday and night. Based on the user profile information known to the W4COMN, it may be known that the first recipient is a fan of sake. This may be based on the following: explicit user profile information obtained via communication related to sake or frequent visits to a website providing information related to sake, profile information or tags associated with user profiles on social networking sites known to the W4COMN, purchase information recorded by the W4COMN related to purchases of sake at wine stores, or search queries in which the first recipient has conducted a large number of searches or studies related to sake.

Accordingly, a message that would have been sent by a sender to a first recipient would have been received on the W4COMN, parsed to extract information about the subject matter of the message, and the identity of the intended recipient. Based on the parsed content and the intended recipient, the first recipient will receive an invitation at the Nobu dinner, which also includes information about the restaurant's exhaustive sake menu, including links to various types of sake provided as features in the menu, and a price, as a non-limiting example. Further, as another non-limiting example, a map of the location of the restaurant may also be included, as well as a link for obtaining directions from the recipient's home, assuming that the first recipient is a known RWE on the W4 COMN. Alternatively, the W4COMN, when analyzing information about the first recipient, can recognize that the time of the expected dinner meeting occurred during the time the first recipient was typically at work and thus provide directions from the first recipient's work location instead.

On the other hand, the second recipient may be a vegetarian or have other special dietary requirements. Similar to the description of the known information about the first recipient's preference for japanese sake, this information may also be known by the W4 COMN. Accordingly, the second recipient's message will not contain information about sake, but will contain information about vegetarian items available on the restaurant's menu, as well as a map from a different location to the restaurant, since the information about the second recipient known to the W4COMN (e.g., from the second recipient's calendar application) identifies the proposed dinner as occurring after the recipient's tennis lesson, and thus directions from the tennis court to the restaurant will be included in the augmented message.

As in the example above, the augmented message is created by information in the sender's profile, the recipient's corresponding profile, and the content of the message that are available to the W4 COMN. The augmentation of the message will preferably be done with the most up-to-date information about the sender and receiver known to the W4COMN at the time the message was created and expected to be sent. The message may be augmented based on the user's declared interests, which will determine how the augmented message is ultimately presented. Various users on the network identified as senders and recipients may cause messages to be augmented in a complex or simple manner based on information about the sender or recipient known to the system that augments the message. Thus, the intended recipient with a large amount of information (e.g., photos, literary interests, food interests, purchase history, calendar and location information, and a variety of different devices identified as being related to the individual) has a greater chance of receiving the augmented message than a person with less information available to the messaging network. However, to the extent that the system can derive any declared or implicit interest information from the user's past interactions with the network or network-accessible system, such information will be available as part of the augmentation process.

In addition, the final format in which the message is received may be personalized based on a variety of factors. Thus, as described above, a user RWE can interact with the W4COMN via a variety of devices. Further, the devices may be in different locations at different times. If the intended recipient is using their cell phone when the sender sends a message, the W4COMN may be made aware of this fact based on the usage information gathered by the cell phone or based on recognizing that the user is logged into the system via their cell phone instead of via their personal computer when the message is to be sent. Accordingly, another element of the context available for augmenting a message is the actual device to be used by the recipient to receive and view the augmented message. In other embodiments, the system will augment the message in different ways in each case where it is expected that the message will be received on the user's cell phone, via a TV, or via a web browser.

Thus, in its broadest sense, the present disclosure relates to augmenting messages sent by a sender to an intended recipient, the augmentation occurring via a communication network that contains or has access to information about the sender and the recipient (or only the recipient), such that each augmented message contains as rich content as possible, based on: parsing of message content, identification of various sender/receiver pairs of a message, modeling of social relationships between senders and receivers and various receivers, identification of other data about receivers that are known or accessible to the social structure or network, and utilizing such information to supplement the original message created by the sender with augmented information that may be of higher value to the recipient when it receives the augmented message.

In some cases, the message may be structured, such as a message that itself contains metadata. In this case, the parsing of the message content will utilize any metadata found in the message, which then serves as an additional information resource that can be used by the message expansion network to find additional content for creating an expanded message to be sent to the intended recipient.

Thus, while some of the examples above are given in the context of the W4COMN, it will be apparent from the disclosure herein that the disclosed systems and concepts may be used in networks other than the W4COMN, such as in social networks providing internal messaging or message boards, email systems capable of collecting information about intended recipients and senders, photo-sharing sites, or other communities of users connected by networks where user information may be extracted or obtained or stored, either explicitly or implicitly, for augmentation of messages. In a broader sense, the present disclosure relates to a form of message that can be parsed to extract message content as well as sender and recipient information. Further information about the intended recipient is then accessed and subsequently analyzed, and the results of the analysis are used to incorporate other information into the message in order to augment the message to include information that the sender did not otherwise send, but was augmented by the system based on stated or inferred interests or the recipient's requirements.

Continuing with one example above, once the message has been parsed and the recipient's information has been analyzed, the type of information that may be included in the augmented message is limited only by the potential size constraints of the message to be sent and/or the additional resources available to the network for use in the augmented message, such as, in the above example, photographs of restaurant content, reviews of restaurants from a variety of sources available on the network or throughout the world's web, links to related information (e.g., gas stations or parking lots near the restaurant), videos containing interior footage, or maps that may be adjusted based on known information about the location of the recipient or the subject of the message at the time of receipt or at the time of the intended meeting, as described above.

As another example, if the intended recipient has a physical need for information in one different format, such as blind, or is physically restricted in some way, such as having to take a wheelchair, the system may augment the message to include a text-to-speech version of the message content, or information relating to public transportation that can accommodate a wheelchair and travel a route adjacent to the intended meeting location.

The robustness of the augmentation of the message described herein may be comparable to the information available to the system for augmenting the message. Thus, as previously described, the more information about the intended recipient that is stored or available to the messaging system, the more resources are available to the messaging system to make decisions regarding which augmented content should be included in the message. However, regardless of the amount of information available to the system, messaging systems according to the present disclosure are expected to operate in such a way that: such that a set of rules is defined for parsing the message content and identifying the recipient. The parsed content is used to form a set of variables associated with the message that are used to extract or obtain information about the intended recipient. According to known constraint system methods, the obtained information is matched based on constraints designed into the system to map retrieved data about recipients and identify relationships between intended recipients and message content, to match augmented or enhanced content to message content, and to provide new messages containing content not present in the original message created by the sender.

For example, data parsed from the message may be analyzed and the data correlated with the parsed message and recipients and obtained by the network for use in using and creating the augmented message. The augmented message may be created by applying a stylesheet or template that identifies or parses out the message content against the available data to produce information that is highly relevant to the parsed message data. Thus, if the message contains information about a place, a map will be included. If the message relates to an event, for example, the route or directions may be included based on the location of the event and the location of the recipient (either at the time of receipt of the message or at the time of the proposed event, depending on the nature and scope of the information available about the recipient).

Thus, various schemes may be applied for entity classes and message types. The constraint and parsing process takes into account the type of message and the available data and through a series of iterations, creates an augmented message that preferably takes into account the condition of the sender or recipient and other factors that may affect what the augmented communication content should be.

In addition, conditions such as the type of device the intended recipient will use to view the message may be considered in creating the augmented message. The number of iterations of the processing variables and the applied constraints may vary according to design choice, however, it should be recognized that more robust message expansion will occur when message parsing is more granular and recipient data is more robust.

Fig. 9 illustrates an embodiment of a W4 enhanced engine 900 according to the present disclosure. (the terms "enhance" and "augment" are used interchangeably herein to mean changing a message in accordance with the systems and methods described herein). The W4 enhanced messaging 900 occurs during real-time data collection 902 by the W4COMN (which is continuous as described above). The process of W4 enhanced messaging 900 begins when a message request is received in step 904 for sending a message sent by a sender. Once this occurs, then in step 906, the message is parsed to extract the message content (which forms the W4 information object or IO) and W4 data relating to the RWEs and IOs associated with the message is correlated and retrieved. Subsequently, in step 908, the message is graphed and/or mapped to the retrieved W4 data to identify relationships between RWEs and IOs associated with the message. Accordingly, in step 910, the message is classified by the type of the message and thus matched to enhanced content regarding the message IO. In step 912, the delivery conditions are then analyzed to determine if the requirements of the W4COMN protocol have been met, and if so, the message continues to the next step; if not, then there is a feedback loop back to step 908 for further W4 data retrieval and identification. Then, in step 914, the message IO is combined with the enhanced content to prepare for delivery as a composite IO, which is an augmented message. A communication channel and/or format for delivering the composite IO is then selected 916 or when to send the message to the recipient RWE. The process ends at step 918 after delivery, or it may end when the augmentation message is sent.

FIG. 10A illustrates another embodiment of a system for delivering messages to any number of recipients over a network in an embodiment not implemented according to the W4 COMN. Enhanced messaging system 1000 includes a sender 1002, a messaging system 1004, and recipients 1006 (1-n). Messaging system 1004 has access to data source 1003. data source 1003 may be on the same network as messaging system 1004 or may be available on a global network (e.g., the world wide web). Data sources 1003 are sources of information about the sender and recipient of messages to be sent over the messaging system, as well as sources of information about the content of the messages. The data source 1003 may be a social networking site, a subscription service, a calendar system, a website, a music sharing site, or other network accessible source of information about the messaging system user and/or the message content from which data may be extracted and analyzed for forming augmented messages as described herein.

As described with reference to fig. 10A, a message and/or messages are sent from sender 1002 through messaging system 1004 for subsequent delivery to recipients 1006 (1-n). During this communication, the message is parsed in the manner described above to extract the message content and sender and recipient information or just the recipient information. Messaging system 1004 accesses data source 1003 for information related to the parsed message and utilizes the techniques described above to extract variables that can be associated with the obtained data to solve a series of constraints to create a new message with augmented content related to each recipient 1006(1-n) that is known or the resulting profile or characteristic or preference. Thus, creation of the enhanced message occurs within messaging system 1004.

Upon receipt of the message by messaging system 1004, the elements of the message are parsed and data relating to the recipient and the message is retrieved. The retrieved data is mapped to correlate between the appropriate recipient and the message content. Messaging system 1004 retrieves individual information about each recipient 1006(1-n) for augmentation of the message. Accordingly, a particular enhanced message augmented with data associated with each recipient 1006(1-n) is delivered to each recipient 1006 (1-n). Messaging system 1004 may also be able to access individual and/or personal information for each recipient 1006(1-n) that is provided to system 1004 by the recipient in the form of explicit preference data and social network affiliations.

Fig. 10B illustrates an alternative embodiment of the above-described enhanced messaging system 1000 that exists in fig. 10A. Messaging system 1004 interfaces with content server 1008. Content server 1008 is representative of the source or sources of additional types of content that may be used for incorporation into the enhanced message generated by messaging system 1004. By way of non-limiting example, the types of such content are commercial data, advertising data, audio data, video data, literary data, and all other types of monetizable content that may be entered into a digital message, such as advertising content or paid media placement that, in the context of the system described herein, would be highly targeted to recipients 1006(1-n) based on the context of the message, information about the recipients known or available to the network, and the sender 1002 and recipients 1006(1-n) or recipient and event relationships (as described above) described in the message.

A non-limiting example of a type of monetized content entered in an enhanced message prior to delivery requires sender 1002 to invite a group of recipients to play golf. One of the recipients 1006(1-n) has a strong interest in golf balls, which is derived from information found by data retrieved from a data source having information about that particular recipient. Accordingly, messaging system 1004 augments data related to golf activity before messages are delivered to recipients 1006(1-n) having derived interest in golf, such as: golf club coupons at local stores. The local store then pays the network provider for the insertion of such augmented messages.

Thus, as described above, the augmented content may take the form of advertising content or paid media placement that will be highly targeted to the recipients 1006(1-n) based on the context of the message, information about the recipients known or available to the network, and the sender-to-recipient or recipient-to-event relationships described in the message. Thus, the augmented message may contain content that a system administrator or operator providing the augmented messaging system may use to obtain revenue.

In an alternative embodiment, content server 1008 is part of enhanced messaging system 1000.

FIG. 11 shows a non-limiting example 1100 of an augmented message sent from a sender/sender 1002 to three recipients 1104, 1106, 1108. In this example, the message is a meeting invitation. The message is parsed and information and message type are determined for each of the three recipients 1104, 1106, 1108, each of which is detected using a different type of communication/receiving device, as described herein. Recipient 1104 is using a mobile device, recipient 1106 is using a television, and recipient 1108 is using a computer system. The message is augmented to include enhanced content specific to each of the recipients 1104, 1106, 1108 based on the associated personal data for each recipient. These three examples depict the flexibility of augmenting message delivery options in relation to alternative recipients using different devices.

In this example, sender 1002 issues a message related to a meeting invitation at destination Caf Bebo. The original message is augmented with recipient-specific content according to the data obtained relating to each recipient 1104, 1106, 1108. Each recipient 1104, 1106, 1108 receives a different augmented message that is appropriately modified according to the recipient's particular known or derived characteristics, preferences/offensiveness, location and/or profile (based on any available data, such as topical data, spatial data, temporal data, and/or social data) and communication channel and format associated with the type of receiving device, as described above.

The recipient 1104 receives the personalized enhanced message with augmented content in the form of digital imagery related to the meeting location, and a map of departures from the current location of the recipient 1104. For this example, the recipient 1104 receives a message augmented with a photograph of cafe Bebo and a directional guide from its current location, which may be derived, for example, from user profile data or from a sensed location of the recipient's 1104 mobile device.

A recipient 1106 watching a television or similar video device (which may include typical peripherals as will be understood by those of ordinary skill in the art) receives an augmented message containing video content and personalization related data. The recipient 1106 receives a different message augmented to include video content related to the meeting invitation and location (i.e., cafe Bebo). In addition, since the system has determined that recipient 1106 does not have a car, but is always cycling, based on data about recipient 1106, public transportation schedule and bicycle route information may be provided. In addition, since the system already knows that the recipient 1106 is a wine enthusiast, a link to a wine comment about the wine clickable on the cafe's menu may also be included in the personal augmented message for the recipient 1106.

The recipient 1108 logs onto a computer or PC (or similar device) and receives a different augmented message that relates to social and topical content related to the recipient 1108 of the system that is known to be a vegetarian. The personalized augmented message includes vegetarian dish menu options available at Caf Bebo extracted from the Caf Bebo's website, and also includes a link to a magazine review of provided vegetarian dishes. Receiver 1108 also receives video of imagery of cafe Bebo from sender Joe.

Thus, as a non-limiting example, the content included in each augmented message varies so as to be dependent upon and highly relevant to each recipient 1104, 1106, 1108 (for purposes of this example) and the communication device being used.

In other embodiments, it is possible that the augmented message may change over time because conditions relating to one or more recipients change after the message is created. Thus, for example, if a message is sent to a first recipient augmented based on a set of conditions at the time the message was delivered, but the message was not read by the first recipient, the system may record this fact and the first recipient changed location. The system may then reprocess the message based on the obtained new information about the changed location of the first recipient and send a new message or automatically replace the old message that was not read with a new message containing updated information. Thus, for example, if the original message contained directions to the intended meeting location based on the system's knowledge of the first recipient at location A, and the first recipient had changed location to location B before reading the first message, a second message may be created that contained directions from location B to the intended meeting location. Thus, by obtaining feedback about the message and the status of the message, a continuously improved message extension may be provided.

As shown in fig. 9A, step 960 is included to determine whether the status of the recipient of the message has changed after the message was sent or whether a message reply containing new information was sent. If so, the method returns to step 906 to create and send another augmented message in view of the new information of the message regarding the recipient's status.

Thus, as another non-limiting example, in another exemplary context, many social networking sites and professional relationship sites contain invitations to intended recipients that are not yet members of the sender's social network. Such invitations are often not responded to. In this case, if the invitation message is sent as part of the augmented messaging system described herein, the system may monitor the fact that the invitation has never been responded to, and send another augmented message containing new information that may further entice the recipient to change its decision not to respond to the invitation.

As another limiting example, the system may monitor replies containing information from the recipient to the sender and other recipients and augment the replies in a similar manner to incorporate additional content that may be relevant to the information added by the recipient (now the sender of the reply message). Thus, based on changes in the condition of the recipient, or changes in the content of the message due to additions made by the recipient, or based on changing information relative to the sender, the system may continually augment and resend or replace the augmented message as the condition of the sender and/or recipient changes.

Those skilled in the art will recognize that the methods and systems of the present disclosure may be implemented in many ways and thus are not limited by the exemplary embodiments and examples described above. In other words, the functional elements and individual functions performed by a single or multiple components, using various combinations of hardware and software or firmware, may be distributed across software applications at either the client or server level, or both. Thus, any number of the features of the different embodiments described herein may be combined into single or multiple embodiments, and alternate embodiments having fewer than, or more than, the features described herein, or having all of the features described herein, are possible. The functionality may also be distributed, in whole or in part, over a number of components, in a manner that is or becomes known. Thus, many software/hardware/firmware combinations are possible in implementing the functions, features, interfaces and preferences described herein. Additionally, the scope of the present disclosure covers conventionally known ways of performing the described features and functions and interfaces, as well as those variations and modifications that may be made to the hardware or software or firmware components described herein as would be understood by those skilled in the art now and hereafter.

Additionally, the embodiments of the methods presented and described in flow chart form in this disclosure are presented as examples to provide a more complete understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternate embodiments are contemplated in which the order of various operations is altered, and sub-operations described as part of larger operations are performed independently.

While various embodiments have been described for purposes of this disclosure, such embodiments should not be considered to limit the teachings of this disclosure to those embodiments. Various changes and modifications may be made to the elements and operations described above to achieve results that are still within the scope of the systems and processes described in this disclosure. Many other variations can be made which will readily suggest themselves to those skilled in the art and which are encompassed within the spirit of the disclosed disclosure.

Claims (12)

1. A method for delivering augmented messages, comprising:

receiving, at a computing device, a message from a sender for delivery over a network, the message comprising message content and an identity of an intended recipient;

analyzing, via the computing device, the message content to extract logical data based on logical data objects of the message content;

collecting recipient data available to the network by searching, via the computing device, social data, spatial data, temporal data, and logical data about intended recipients of the message and the extracted logical data;

correlating, via the computing device, the collected recipient data to refine the collected data into correlated data that correlates the extracted logical data with the intended recipient;

collecting, via the computing device, content available to the network relating to the affiliated data;

determining whether delivery conditions associated with the message are satisfied such that the steps of collecting recipient data, correlating, and collecting content are automatically repeated in the event that the delivery conditions are determined to not be satisfied;

determining, via the computing device, which of the collected content is to be inserted into the message;

inserting, via the computing device, the determined collected content into the message to form an enhanced message;

transmitting the enhanced message to the intended recipient over the network in accordance with the satisfied delivery condition; and

determining whether a condition of the intended recipient has changed since the enhanced message was transmitted to the intended recipient, such that the steps of analyzing, collecting recipient data, correlating, collecting content, determining whether delivery conditions are met, determining which of the collected content is to be inserted into a message, inserting and transmitting are automatically repeated, if it is determined that the condition has changed.

2. The method of claim 1, further comprising:

the condition of the intended recipient is analyzed and used as a factor in determining the collected content.

3. The method of claim 2, wherein the condition of the intended recipient is a location of the intended recipient at a time when the enhanced message is intended to be received.

4. The method of claim 2, wherein the condition of the intended recipient corresponds to a type of device being used by the intended recipient in anticipation of receiving the enhanced message.

5. The method of claim 2, wherein the condition of the intended recipient is an activity the intended recipient was engaged in when the enhanced message was intended to be received.

6. The method of claim 1, wherein the delivery condition corresponds to a delivery time of the enhanced message.

7. The method of claim 1, wherein the delivery condition corresponds to the intended recipient being at a delivery location of the enhanced message.

8. The method of claim 1, wherein the delivery condition corresponds to a type of device used by the intended recipient.

9. The method of claim 1, wherein upon determining that the delivery condition is not satisfied, feeding back the message to the sender for further data.

10. The method of claim 1, further comprising:

collecting sender data available to the network by searching social data, spatial data, temporal data, and logistical data about the sender, wherein correlating the collected recipient data further comprises correlating the collected sender data with the collected recipient data and the logistical data.

11. The method of claim 1, wherein the step of collecting content further comprises collecting third party content for entry into the enhanced message in exchange for payment to the third party.

12. An apparatus for delivering augmented messages, comprising:

means for receiving, at a computing device, a message from a sender for delivery over a network, the message comprising message content and an identity of an intended recipient;

means for analyzing, via the computing device, the message content to extract logical data based on logical data objects of the message content;

means for collecting recipient data available to the network by searching, via the computing device, social data, spatial data, temporal data, and logical data about intended recipients of the message and the extracted logical data;

means for correlating, via the computing device, the collected recipient data to refine the collected data into correlated data that correlates the extracted logical data with the intended recipient;

means for collecting, via the computing device, content available to the network relating to the correlated data;

means for determining whether a delivery condition associated with the message is satisfied such that the means for collecting recipient data, the means for correlating, and the means for collecting content are automatically repeated if it is determined that the delivery condition is not satisfied;

means for determining, via the computing device, which of the collected content is to be inserted into the message;

means for inserting, via the computing device, the determined collected content into the message to form an enhanced message;

means for transmitting the enhanced message to the intended recipient over the network in accordance with the delivery condition being satisfied; and

means for determining whether a condition of the intended recipient has changed since the enhanced message was transmitted to the intended recipient, such that in the event that it is determined that the condition has changed, means for analyzing, means for collecting recipient data, means for correlating, means for collecting content, means for determining whether delivery conditions are met, means for determining which of the collected content to insert into a message, means for inserting, and means for transmitting are automatically repeated.