HK1173544A - Learning system for the use of competing valuation models for real-time advertisement bidding - Google Patents

Description

Learning system for using competitive valuation models for real-time advertising offers

Cross Reference to Related Applications

This application claims the benefit of the following commonly owned U.S. provisional patent applications, which are incorporated herein by reference in their entirety: application No. 61/234,186, filed on 8/14/2009 and entitled "Real-Time filing System for Delivery of Advertifying".

Technical Field

The present invention relates to using historical and real-time data associated with digital media and using it to adjust pricing and placement of advertising media.

Background

Management of the presentation of advertisements to digital media users often features a batch mode optimization scheme in which advertising content is selected for presentation to a selected group of users, performance data is collected and analyzed, and then optimization steps are performed to achieve better future advertising performance. This process is then run iteratively in an optimization analysis sequence in an attempt to improve ad performance criteria (such as complete deals) through more informative ad user pairing and other techniques. However, this optimization framework is limited in several important respects. For example, in view of the digital media user growth brought about by widespread innovations (such as social networking), there is an overabundance of data relating to digital media usage that cannot be accommodated and analyzed by the preplanned batch-mode analytics that are currently modeling advertising performance in the industry. In addition, batch-mode ad analytics may force content groupings that do not correspond to actual and changing sequences of ad impressions that occur within a user's behavior or across a pool of users. Publishers of advertising content may thus be forced to unnecessarily utilize multiple advertising networks to distribute their advertisements based at least in part on the multiple optimization techniques and criteria used by the different advertising networks. This can create redundancy and limit the ability to rate the value of an ad's impression and its performance over time within the digital media user population.

What is needed, therefore, is a method and system for valuing the impression of an advertisement to a digital media user using automated analysis techniques that can use historical and real-time data relating to advertisement performance as part of a learning system to optimize advertisement selection and aid in the valuation of advertisement presentation.

Disclosure of Invention

In embodiments, the present invention may provide methods and systems for using a plurality of competing economic valuation models to predict an economic valuation for each advertisement layout of a plurality of advertisement layouts in response to receiving a request to lay out an advertisement. The economic valuation model can be based at least in part on real-time event data, historical event data, user data, third party business data, historical advertising impressions, advertiser data, advertising agency data, historical advertising performance data, and machine learning. Additionally, a computer program product based on the method and system of the present invention, when executed on one or more computers, can perform the step of evaluating each economic valuation produced by each of a plurality of competing economic valuation models to select one as a current valuation of an advertising layout.

In an embodiment, a computer program product based on the method and system of the present invention may perform the following steps when executed on one or more computers: a plurality of competing economic valuation models are deployed in response to receiving a request to place an advertisement to predict an economic valuation for each advertisement placement of a plurality of advertisement placements. In an embodiment, a request may be received from a publisher and a recommended bid amount may be automatically sent to the publisher. In another embodiment, a request may be received from a publisher and a bid equal to the recommended bid amount may be automatically offered on behalf of the publisher. Additionally, the bid amount of the recommendation may be associated with a recommendation time for the advertisement layout. In an embodiment, the recommended bid amount may be derived by analyzing a real-time bid log that may be associated with a real-time bid machine.

In an embodiment, a computer program product based on the method and system of the present invention may perform the following steps when executed on one or more computers: each economic valuation produced by each of the plurality of competing economic valuation models is evaluated to select one valuation as a first valuation of the advertising layout. Additionally, the computer program product may reevaluate each valuation produced by each of the plurality of competing economic valuation models to select one as the revised valuation for the advertisement layout. The revised valuation can be based at least in part on an analysis of an economic valuation model that can use real-time event data that may not be available at the time the first valuation is selected. Further, the computer program product may replace the first valuation with a second revised valuation for use in deriving a recommended valuation amount for the advertising layout.

In an embodiment, a computer program product based on the method and system of the present invention may perform the following steps when executed on one or more computers: a plurality of competing economic valuation models are deployed in response to receiving a request to place an advertisement to evaluate information relating to a plurality of available advertisement placements to predict an economic valuation for each of the plurality of advertisement placements. Additionally, the computer program product may evaluate each economic valuation produced by each of the plurality of competing economic valuation models to select one valuation as a future valuation of the advertising layout.

In an embodiment, a computer program product based on the method and system of the present invention may perform the following steps when executed on one or more computers: a plurality of competing economic valuation models are deployed in response to receiving a request to place an advertisement to evaluate information relating to a plurality of available advertisement placements to predict an economic valuation for each of the plurality of advertisement placements. Additionally, the computer program product may evaluate each economic valuation produced by each of the plurality of competing economic valuation models in real time to select one valuation as a future valuation of the advertising layout. In an embodiment, the future valuation may be based at least in part on simulation data describing future events. Additionally, the future event may be a stock market fluctuation. Further, simulation data describing future events may be derived from analysis of historical event data.

In an embodiment, a computer program product based on the method and system of the present invention may perform the following steps when executed on one or more computers: a plurality of competing real-time bidding algorithms involving a plurality of available advertising layouts are deployed to bid on an advertising layout in response to receiving a request to lay out an advertisement. The competing real-time quotation algorithm may use data from the real-time quotation log. Additionally, the computer program product may evaluate each offer algorithm to select a preferred algorithm.

In an embodiment, a computer program product based on the methods and systems of the present invention, when executed on one or more computers, may deploy a plurality of competing real-time bidding algorithms involving a plurality of available advertising layouts to bid on an advertising layout in response to receiving a request to lay out an advertisement. In addition, the computer program product may evaluate each offer recommendation generated by the competing real-time offer algorithm. In addition, the computer program product may reevaluate each offer recommendation generated by the competing real-time offer algorithm to select one as a revised offer recommendation. The revised offer recommendations may be based at least in part on a real-time offer algorithm that uses real-time event data that may not be available at the time the offer recommendation is selected. Further, the computer program product may replace the bid recommendation with a revised bid recommendation for use in deriving a recommended bid amount for the advertisement layout. The replacement may occur in real time with respect to receiving a request to place the advertisement.

While the invention has been described in connection with certain preferred embodiments, other embodiments will be understood and are encompassed by those of ordinary skill in the art.

Drawings

The invention and the following detailed description of certain embodiments thereof may be understood by reference to the following drawings:

FIG. 1A depicts a real-time bidding method and system for placing advertisements.

Fig. 1B depicts the execution of a real-time quotation system across multiple exchanges.

FIG. 2 depicts a learning method and system for optimizing offer management.

Fig. 3 depicts sample data fields that may be used to predict media success associated with key performance indicators.

FIG. 4 depicts training a plurality of algorithms related to an advertising campaign, wherein better performing algorithms may be detected.

Fig. 5A depicts the use of a differential segment for bid valuation.

FIG. 5B depicts a differential segment analysis of an ad campaign.

FIG. 5C depicts optimizing pricing through frequency analysis.

Fig. 5D depicts how pacing may be optimized by near cause (recency) analysis within a real-time bidding system.

Fig. 6 depicts the use of nano-segments for offer valuation.

Fig. 7 depicts sample integration of the real-time bidding method and system within the primary media supply chain.

FIG. 8A depicts a hypothetical situation study using a real-time bidding method and system.

Figure 8B depicts a second hypothetical situation study that compares two advertising campaigns using the real-time bidding method and system.

FIG. 9 depicts a simplified use case in the form of a flow chart summarizing key steps a user may take when using the real-time pricing method and system.

FIG. 10 depicts an exemplary embodiment of a user interface for a pixel provisioning system that may be associated with a real-time pricing system.

Fig. 11 depicts an exemplary embodiment of impression level data that may be associated with a real-time bidding system.

FIG. 12 depicts a hypothetical advertising campaign performance report.

FIG. 13 illustrates a bid valuation facility for purchasing real-time bids and valuations of an online advertising layout.

FIG. 14 illustrates a method for purchasing real-time bids and economic valuations for an online advertising layout.

FIG. 15 illustrates a method for determining a bid amount.

FIG. 16 illustrates a method of automatically offering bids for optimal placement of advertisements.

FIG. 17 illustrates facilities of an analytics platform that may be used to target bids for online advertising purchases, according to an embodiment of the present invention.

FIG. 18 illustrates a method for selecting and presenting to a user at least one of a plurality of available layouts based on economic valuations.

FIG. 19 illustrates a method for prioritizing available advertising layouts derived from economic valuations.

FIG. 20 illustrates a real-time facility for selecting alternative algorithms for predicting purchase price trends for offers for online advertising.

FIG. 21 illustrates a method for predicting performance of an advertising layout based on current market conditions.

FIG. 22 illustrates a method for determining preferences between a primary model and a secondary model for predicting economic valuations.

FIG. 23 illustrates a method for determining preferences between a primary model and a secondary model for predicting economic valuations.

FIG. 24 illustrates a method for selecting one valuation model among a plurality of competing valuation models in a real-time bid for an advertising layout.

FIG. 25 illustrates a method for replacing a first economic valuation model with a second economic valuation model to derive a recommended bid amount for an advertising layout.

FIG. 26 illustrates a method for evaluating a plurality of economic valuation models and selecting one valuation as a future valuation of an advertising layout.

FIG. 27 illustrates a method for evaluating multiple economic valuation models in real time and selecting one valuation as a future valuation of an advertising layout.

FIG. 28 illustrates a method for evaluating multiple bidding algorithms to select a preferred algorithm for placement of an advertisement.

FIG. 29 illustrates a method for replacing bid recommendations with modified bid recommendations for advertisement placement.

FIG. 30 illustrates a real-time facility for measuring the value of additional third party data.

FIG. 31 illustrates a method for advertisement rating with the ability to measure the value of additional third party data.

FIG. 32 illustrates a method for calculating a valuation of a third party data set and billing an advertiser for the partial valuation.

FIG. 33 illustrates a method for computing a valuation of a third party data set and calibrating bid amount recommendations paid by a publisher for placement of advertising content based at least in part on the valuations.

FIG. 34 depicts an embodiment of a data visualization that presents an advertisement performance summary by time of day versus day of the week.

FIG. 35 depicts an embodiment of a data visualization presenting advertisement performance summaries in terms of population density.

FIG. 36 depicts a data visualization embodiment presenting an advertising performance summary in terms of a geographic area of the United states.

FIG. 37 depicts a data visualization embodiment presenting an advertising performance summary in terms of personal revenue.

FIG. 38 depicts a data visualization embodiment presenting an advertising performance summary by gender.

Fig. 39 depicts affinity (affinity) indices for an advertising campaign by category.

FIG. 40 depicts a data visualization embodiment presenting a page visit summary by impression count.

Detailed Description

Referring to fig. 1A, a real-time bidding system 100A may be used to select and rate sponsored content purchasing opportunities, bid in real-time, and lay out sponsored content (such as advertisements) across multiple content delivery channels according to methods and systems as described herein. The real-time bidding facility may notify purchasing opportunities for placement of sponsored content across multiple advertisement ("ad") delivery channels. The real-time bidding facility may also enable the collection of data regarding advertisement performance and use of this data to provide active feedback to parties who want to place advertisements, and automatically adjust and target the advertisement delivery channel used to present sponsored content. The real-time bidding system 100A may facilitate selection of a particular advertisement type and associated cost of advertisement layout over time (and, for example, adjusted by layout time) for presentation in each layout opportunity. The real-time facility may facilitate valuation of advertisements using valuation algorithms and may also optimize the return on investment for the advertiser 104.

The real-time bidding system 100A may include and/or be further associated with one or more distribution service consumers, such as an ad agency 102 or advertiser 104, an ad network 108, an ad exchange 110 or publisher 112, an analytics facility 114, an ad tagging facility 118, an ad order sending and receiving facility 120, and an ad distribution service facility 122, an ad data distribution service facility 124, an ad display client facility 128, an ad performance data facility 130, a context organizer service facility 132, a data integration facility 134, and one or more databases that provide different types of data related to advertisements and/or ad performance. In an embodiment of the present invention, the real-time quotation system 100A may include analysis facilities that may include, at least in part, a learning machine facility 138, a valuation algorithm facility 140, a real-time quotation machine facility 142, a tracking machine facility 144, an impression/click/action log facility 148, and a real-time quotation log facility 150.

In embodiments, the one or more databases that provide data related to advertisements, advertisement performance, or advertisement layout context to the real-time bidding system 100A and to the learning machine facility 138 may include the agent database and/or the advertiser database 152. The agent database may include campaign descriptors and may describe channels, timelines, budget and other information relating to the use and distribution of advertisements, including historical information. The agent data 152 may also include activity and history logs that may include a layout for each advertisement shown to the user. The proxy data 152 may also include one or more of the following: an identifier for the user, web page context, time, payment price, shown advertisement message, and composite user action, or some other type of activity or historical log data. The advertiser database may include enterprise intelligence data or some other type of data that may describe dynamic and/or static marketing goals or that may describe the operation of the advertisers 104. In an example, the inventory excess quantity for a given product (that advertiser 104 has in its warehouse) may be described by advertiser data 152. In another example, the data may describe purchases that a consumer performs while interacting with advertiser 104.

In an embodiment, the one or more databases may include a historical event database. The historical event data 154 may be used to correlate the time of a user event with other events occurring, for example, in the area where the user is located. In an example, the response rate to certain types of advertisements may be related to stock market trends. Historical event data 154 may include, but is not limited to, weather data, event data, local news data, or some other type of data.

In an embodiment, the one or more databases may include a database of user data 158. User data 158 may include data that third parties may source and/or provide internally, which may contain personal link information about the advertisement recipients. This information may associate the user with preferences or other indicators that may be used to label, describe, or categorize the user.

In an embodiment, the one or more databases may include a real-time event database. The real-time event data 160 may include data that is similar to the historical data but more current. Real-time event data 160 may include, but is not limited to, data measured by the current time to seconds, minutes, hours, days, or some other time. In an example, if the learning machine facility 138 finds a correlation between the advertisement performance and the historical stock market index value, the real-time stock market index value may be used by the real-time ticker facility 142 to rate the advertisement.

In an embodiment, the one or more databases may include a context database that may provide context data 162 associated with the publisher, the publisher's content (e.g., the publisher's website), and so forth. The contextual data 162 may include, but is not limited to, keywords found within the advertisement, a URL associated with a prior layout of the advertisement, or some other type of contextual data 162 and may be stored as categorical metadata relating to the content of the publisher. In an example, such classification metadata may record that a first publisher's website is related to financial content while a second publisher's content is primarily related to sports.

In an embodiment, the one or more databases may also include a third party/business database. The third party/business database may include data 164 relating to consumer transactions, such as point of sale scanner data obtained from retail transactions or some other type of third party or business data.

In embodiments of the present invention, data from one or more databases may be shared with the analysis facility 114 of the real-time pricing system 100A through the data integration facility 134. In an example, the data integration facility 134 may provide data from one or more databases to an analysis facility of the real-time bidding system 100A for purposes of evaluating potential advertisements and/or advertisement layouts. For example, the data integration facility 134 may combine, merge, analyze, or integrate multiple data types (e.g., user data 158 and real-time event data 160) received from available databases. In embodiments, the context organizer may analyze the web content to determine whether the web page contains content about sports, finance, or some other topic. This information may be used as input to the analytics platform facility 114 to identify relevant publishers and/or web pages where advertisements will appear.

In an embodiment, the analysis facility of the real-time bidding system 100A may receive the advertisement request via the advertisement order sending and receiving facility 120. The ad request may come from the ad broker 102, the advertiser 104, the advertising network 108, the ad exchange 110, and the publisher 112, or some other party requesting ad content. For example, the tracker facility 144 may receive the advertisement request via the advertisement order transmission and reception facility 120 and provide a service that may include attaching an identifier (such as an advertisement tag using the advertisement tagging facility 118) to each advertisement order and generating an advertisement layout. This ad tracking functionality may enable the real-time bidding system 100A to track, collect, and analyze ad performance data 130. For example, tracking pixels may be used to mark online display advertisements. Once a pixel is serviced from the tracker utility 144, it may record the placement opportunity and the time and date of the opportunity. In another embodiment of the present invention, the tracker utility 144 may record the ID of the ad requestor (user) and other information that labels the user, including but not limited to Internet Protocol (IP) address, context of the ad and/or ad layout, history of the user, geographic location information of the user, social behavior, inferred demographics, or some other type of data. Ad impressions, user click-throughs, action logs, or some other type of data may be generated by tracker utility 144.

In an embodiment, the logged and other data types may be used by the learning machine facility 138 to refine and customize the targeting and valuation algorithms 140 as described herein. The learning machine facility 138 can generate rules regarding advertisements that perform well for a given customer and can optimize the content of the advertising campaign based on the generated rules. Additionally, in embodiments of the present invention, the learning machine facility 138 may be used to develop a targeting algorithm for the real-time ticker facility 142. The learning machine facility 138 may learn patterns including Internet Protocol (IP) addresses, context of advertisements and/or advertisement placements, URLs of advertisement placement websites, user history, user geolocation information, social behavior, inferred demographics, or any other characteristic of the user or that may link to the user, advertisement concepts, advertisement size, advertisement format, advertisement color, or any other characteristic of the advertisements, or some other type of data that may be used to target and value advertisements and advertisement placement opportunities, as well as other data. In embodiments of the present invention, a learning mode may be used to target advertisements. In addition, the learning machine facility 138 can be coupled as shown in FIG. 1 to one or more databases from which it can obtain additional data needed to further optimize the goal setting and/or valuation algorithm 140.

In embodiments of the present invention, an advertiser 104 may place an "order" with instructions that limit where and when an advertisement may be placed. The order from the advertiser 104 may be received by a learning machine facility of the platform or another unit. The advertiser 104 may specify a 'goodness' criterion for the success of the ad campaign. Additionally, the tracker device 144 may be used to measure 'goodness' criteria. Advertisers 104 may also provide historical data associated with 'orders' for revenue for bootstrap (bootstrap) analysis. Thus, based on data available from one or more databases and data provided by the advertiser 104, the learning machine facility 138 can develop customized targeting algorithms for advertisements. The targeting algorithm may calculate the expected value of the advertisement under certain conditions (e.g., using the real-time event data 160 as part of the modeling). The targeting algorithm may also seek to maximize a specified 'goodness' criterion. The targeting algorithm developed by the learning machine facility 138 may be received by a real-time ticker 142 that may wait for an opportunity to place an advertisement. In embodiments of the present invention, the real-time ticker facility 142 can also receive advertisement and/or bid requests via the advertisement order sending and receiving facility 120. The real-time ticker facility 142 can be viewed as a "real-time" facility in that it can reply to advertisement or bid requests associated with time constraints. The real-time ticker facility 142 can use a non-stateless method to calculate which advertising message will be shown while the user waits for a system decision. The real-time ticker facility 142 can use algorithms provided by the learning machine facility 138 to perform real-time calculations to dynamically estimate the optimal bid value. In an embodiment, the alternate real-time ticker facility 142 may have a stateless configuration for determining advertisements to be presented.

The real-time ticker facility 142 can mix historical and real-time data to generate a valuation algorithm for calculating a real-time bid value to be associated with an advertisement and/or advertisement placement opportunity. The real-time ticker facility 142 can calculate an expected value that combines information about: an Internet Protocol (IP) address, a context of an advertisement and/or advertisement layout, a history of a user, geographic location information of a user, social behavior, inferred demographics, or some other type of data. In embodiments, the real-time ticker facility 142 can use opportunistic algorithm updates by using the tracker 144 or advertising performance data to rank and prioritize algorithms based at least in part on the performance of each algorithm. The learning machine facility 138 may use and select from an open list of a plurality of competing algorithms in the machine learning facility and the real-time bidding facility. The real-time ticker 142 can use control system theory to control pricing and serving speed for a set of advertisements. In addition, the real-time quotation machine facility 142 can construct a user profile using the winning and losing quotation data. The real-time ticker 142 can also correlate the projected value to current events in the advertisement recipient's geography. The real-time ticker facility 142 can exchange advertisement buys across multiple exchanges, thus treating multiple exchanges as a single inventory source, selecting and buying advertisements based at least in part on valuations modeled by the real-time ticker system 100A.

In an embodiment, the real-time quotation system 100A may also include a real-time quotation log facility that may record quotation requests received and quotation responses sent by the real-time quotation machine facility 142. In embodiments of the present invention, the real-time quote log may record additional data related to the user. In an example, the additional data may include details of a website that the user may visit. These details can be used to derive user interests or browsing habits. In addition, the real-time bid log facility may record the arrival rates of advertisement placement opportunities from different advertisement channels. In an embodiment of the present invention, a real-time quote log facility may also be coupled to the learning machine facility 138.

In an embodiment, the real-time ticker 142 can dynamically determine an expected economic valuation for each layout of a plurality of potential layouts for an advertisement based at least in part on the valuation algorithm 140 associated with the learning machine facility 138. In response to receiving a request to place an advertisement, the real-time ticker facility 142 can dynamically determine an expected economic valuation for each of a plurality of potential layouts for the advertisement and can select and decide whether to present an available layout to one or more distribution service consumers based on the economic valuations.

In an embodiment, the real-time quotation machine 142 may include altering the model used to dynamically determine the economic valuation prior to processing the second request for the layout. The alteration of the model may be based at least in part on a valuation algorithm associated with the learning facility. In embodiments of the present invention, prior to selecting and presenting one or more available layouts, the behavior of the economic valuation model can be altered to produce a second set of valuations for each of the plurality of layouts.

In an embodiment, valuation algorithm 140 can evaluate performance information relating to each of a plurality of advertisement layouts. A dynamic variable economic valuation model can be used to determine the desired valuation. The valuation model can evaluate the value of the newspaper with respect to economic valuations for a plurality of layouts. The steps in bidding for multiple available layouts and/or multiple advertisements may be based on economic valuations. In an exemplary case, the real-time ticker facility 142 can employ the following sequence: in step 1, the real-time ticker 142 can use the valuation algorithm 140 to filter possible advertisements to be shown. At step 2, the real-time ticker facility 142 can check whether the filtered advertisements have residual budget funds and can remove from the list any advertisements that are listed as having no available budget funds. In step 3, the real-time ticker facility 142 can run an economic valuation algorithm for the advertisements to determine the economic value for each advertisement. At step 4, the real-time ticker facility 142 can adjust the economic value according to the opportunity cost of placing the advertisement. At step 5, the real-time ticker facility 142 can select the advertisement with the highest economic value after adjusting by the opportunity cost. At step 6, information about the first request (which may include information about the content of the publisher 112 of the request) may be used to update the dynamic algorithm before receiving and processing the second request. Finally, at step 7, the second advertisement may be processed in the same sequence as the first advertisement while the dynamic algorithm is updated before the third advertisement is placed. In an embodiment, multiple competitive valuation algorithms 140 can be used at each step to select advertisements to be presented. By tracking the ad performance of the final placement of ads, the competing algorithms may be evaluated to determine their relative performance and utility.

In an embodiment of the present invention, the competition algorithm may be tested by dividing the data portion into separate training and validation sets. Each algorithm may be trained on a training data set and then the predictability of each algorithm is verified (measured) against a validation data set. The predictability of each bidding algorithm may be evaluated against the validation set using metrics such as Receiver Operating Characteristic (ROC) regions, boost, accuracy/recall, return on advertising expense, other signal processing metrics, other machine learning metrics, other advertising metrics, or some other analysis algorithm, statistical technique, or tool. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention. The predictability of an algorithm can be measured in terms of how well the algorithm predicts how well showing a particular advertisement to a particular consumer in a particular context may affect the likelihood that the consumer engages in a desired action, such as purchasing one of the advertiser's products, ordering the advertiser's products, affecting the consumer's experience with the advertiser's products, visiting a web page, or taking some other kind of action that the advertiser has evaluated.

In embodiments of the present invention, cross-validation may be used to improve the algorithm evaluation metric. Cross-validation describes a method in which the training set-validation set procedure used to evaluate the competition algorithm and/or model is repeated multiple times by changing the training and validation data sets. Cross-validation techniques that may be used as part of the methods and systems described herein include, but are not limited to, repeated random subsampling validation, k-fold cross-validation, k × 2 cross-validation, leave-one-out cross-validation, or some other type of cross-validation technique.

In embodiments, the competition algorithm may be evaluated in real-time, in batch mode processing, or using some other periodic processing framework using the methods and systems as described herein. In embodiments, the competition algorithms may be evaluated online (such as using the internet or some other networked platform), or the competition algorithms may be evaluated offline and made available to the online facility after evaluation. In a sample embodiment, one algorithm may be strictly better than all other algorithms in its predictability, and it may be selected offline in the learning facility 138. In another sample embodiment, one algorithm from the set may be more predictive given a particular combination of variables, and more than one algorithm may be made available to the real-time bidding facility 142, and selecting the best performing algorithm may occur in real-time, for example, by examining the attributes of a particular layout request, and then determining which algorithm from the set of training algorithms is most predictive for that particular set of attributes.

In an embodiment, data corresponding to a valuation of an advertisement from the real-time bidding system 100A may be received by the advertisement distribution service 122 and targeted to a consumer of the valuation data, such as the ad agency 102, the advertiser 104, the advertising network 108, the ad exchange 110, the publisher 112, or some other type of consumer. In another embodiment of the present invention, the advertisement distribution service 122 may be an advertisement server. The advertisement distribution service 122 may distribute the output of the real-time bidding system 100A (such as the selected advertisement) to one or more advertisement servers. In an embodiment, the advertisement distribution service facility 122 may be coupled to a tracker facility 144. In another embodiment of the present invention, the advertisement distribution service 122 may be coupled to an advertisement display client 128. In embodiments, the advertising display client 128 may be a mobile phone, a PDA, a cellular phone, a computer, a communicator, a digital device, a digital display board, or some other type of device capable of presenting advertisements.

In embodiments, the advertisements received at the advertisement display client 128 may include interaction data; such as an offer on a pop-up movie ticket. A user of the ad display client 128 may interact with the ad and may perform an action, such as making a purchase, clicking on the ad, filling out a form, or performing some other type of user action. The user actions may be recorded by the advertisement performance data facility 130. In an embodiment, the advertisement performance data facility 130 may be coupled to one or more databases. In an example, the performance data facility may be coupled to the context database for updating the context database in real-time. In an embodiment, the updated information may be accessed by the real-time pricing system 100A for updating the valuation algorithm 140. In embodiments, the advertising performance data facility 130 may be coupled to one or more distribution service consumers.

Data corresponding to the valuations of the advertisements from the analytics platform facility 114 may also be received by the advertisement distribution service facility 122. In embodiments of the present invention, the advertisement distribution service 122 may use the valuation data to reorder/rearrange/reorganize one or more advertisements. In another embodiment, the advertisement distribution service 122 may use the rating data to rank advertisements based on predefined criteria. The predefined criteria may include time of day, location, etc.

The advertising data distribution service 124 may also provide the rating data to one or more consumers of the advertising rating data. In embodiments, the advertising data distribution service 124 may sell the rating data or may provide a subscription for the rating data to one or more consumers of the advertising rating data. In an embodiment, the advertisement distribution service facility 122 may provide output from the real-time bidding system 100A or from the learning machine facility 138 to one or more consumers of advertisement rating data. Consumers of advertising valuation data can include, without limitation, an advertising agency 102/advertiser 104, an advertising network 108, an advertising exchange 110, a publisher 112, or some other type of advertising valuation data consumer. In an example, the ad broker 102 may be a service enterprise dedicated to generating, planning, and manipulating advertisements for its customers. The ad broker 102 may be independent of the customer and may provide an external view into the effort to sell the customer's product or service. Additionally, the advertising agents 102 may be of different types, including without limitation limited to limited service advertising agents, expert advertising agents, in-house advertising agents, interactive agents, search engine agents, social media agents, healthcare communication agents, medical education agents, or some other type of agent. Additionally, in an example, the advertising network 108 may be an entity that may connect advertisers 104 to websites that may want their advertisements hosted. The advertising networks 108 may include, without limitation, vertical networks, blind networks, and target networks. The advertising network 108 may also be classified as primary and secondary networks. The primary advertising network may have a large number of advertisers 104 and publishers of their own, they may have high quality traffic, and they may serve advertisements and traffic to the secondary network. The secondary advertising network may have some advertisers 104 and publishers of their own, but their primary revenue sources may come from syndicated advertisements of other advertising networks. The network of the ad exchange 110 may include information related to ad inventory attributes such as ad impression price, number of advertisers 104 in a particular product or service category, old data on the highest and lowest bids for a particular slot, ad success (user clicks on ad impressions), etc. Advertisers 104 may be able to use this data as part of their decision making. For example, the stored information may depict a success rate for a particular publisher 112. Further, the advertiser 104 may have the option of selecting one or more models for conducting financial transactions. For example, cost per transaction pricing structures may be employed by advertisers 104. Similarly, in another example, the advertiser 104 may have the option of paying the cost per click. The ad exchange 110 may implement algorithms that may allow publishers 112 to price ad impressions in real time during bidding.

In an embodiment, the real-time bidding system 100A for advertising message delivery may be a component of machines intended for buying opportunities for laying out advertising messages across multiple delivery channels. The system may provide proactive feedback to automatically fine-tune and target the channels used to present the advertising messages and select what advertising messages will be shown in each layout opportunity and the associated costs over time. In an embodiment, the system may be comprised of interconnected machines including, but not limited to, (1) a learning machine facility 138, (2) a real-time ticker 142, and (3) a tracker 144. Two of the machines may generate a log that the learning machine facility 138 may use internally. In embodiments, the input to the system may come from real-time and non-real-time sources. Historical data may be combined with real-time data to fine-tune pricing and placement instructions for an advertising campaign.

In an embodiment, the real-time bidding system 100A for advertisement message placement may include external machines and services. External machines and services may include, but are not limited to, an agent 102, an advertiser 104, agent data 152 (such as campaign descriptors and historical logs), advertiser data 152, key performance indicators, historical event data 154, user data 158, context organizer service 132, real-time event data 160, advertisement distribution service 122, advertisement recipients, or some other type of external machine and/or service.

In embodiments, the agent and/or advertiser 104 may provide historical advertising data and may be a beneficiary of the real-time bidding system 100A.

In embodiments, the agent data 152 (such as campaign descriptors) may describe channels, times, budgets, and other information that may be allowed for disseminating advertisement messages.

In embodiments, the agent data 152 (such as activity and history logs) may describe a layout for each advertisement message show to the user, including one or more of: an identifier for the user, a channel, a time, a payment price, an advertisement message shown, and a user composite user action or some other type of activity or historical log data. The additional log may also record spontaneous user actions, such as user actions that are not directly traceable to ad impressions or some other type of spontaneous user action.

In embodiments, advertiser data 152 may be comprised of enterprise intelligence data or some other type of data that describes dynamic and/or static marketing goals. For example, the excess inventory amount of a given product that an advertiser 104 has in its warehouse may be described by this data.

In an embodiment, the key performance indicator may comprise a set of parameters expressing 'goodness' for each given user action. For example, product activation may be rated at $ X and product configuration may be rated at $ Y.

In embodiments, the historical event data 154 may be used by the real-time bidding system 100A to correlate the time of a user event with other events occurring in their area. For example, the response rate to certain types of advertisements may be related to stock market trends. Historical event data 154 may include, but is not limited to, weather data, event data, local news data, or some other type of data.

In an embodiment, user data 158 may include data provided by a third party that contains personal link information about the advertisement recipient. This information may show annotations or describe the user's user preferences or other indicators.

In embodiments, the context organizer service 132 may identify a context category for the media of the advertisement. For example, the context organizer may analyze the web content to determine whether the web page contains content about sports, finance, or some other topic. This information may be used as input to the learning system 138 to refine on which types of pages the advertisement will appear.

In an embodiment, the real-time event data 160 may include data that is similar to the historical data but more current. Real-time event data 160 may include, but is not limited to, data measured by the current time to seconds, minutes, hours, days, or some other time. For example, if the learning machine facility 138 finds a correlation between the performance of an advertisement and a historical stock market index value, the real-time stock market index value may be used by the real-time ticker 142 to rate the advertisement.

In embodiments, the advertisement distribution service 122 may include, but is not limited to, an advertising network 108, an ad exchange 110, a sell-side optimizer, or some other type of advertisement distribution service 122.

In an embodiment, the advertisement recipients may include individuals that receive the advertisement messages. The advertising content may be specifically requested ("pulled") as part of or attached to the content requested by the ad recipient, or may be "pushed" over a network, for example, by the ad distribution service 122. Some non-limiting examples of modes of receiving advertisements include the internet, mobile phone displays, radio transmissions, television transmissions, electronic billboards, printed media, and cinematographic projections.

In an embodiment, the real-time bidding system 100A for advertisement message placement may include internal machines and services. Internal machines and services may include, but are not limited to, a real-time quotation machine 142, a tracking machine 144, a real-time quotation log, an impression, click and action log, a learning machine facility 138, or some other type of internal machine and/or service.

In embodiments, the real-time bidding engine 142 may receive a bid request message from the advertisement distribution service 142. The real-time quotation machine 142 may be considered a "real-time" system in that it may respond to quotation requests associated with time constraints. The real-time ticker 142 can use a non-stateless approach to calculate which advertising message will be shown while the user waits for a system decision. The system can use algorithms provided by the learning machine facility 138 to perform real-time calculations to dynamically estimate the optimal reporting value. In an embodiment, the alternative system may have a stateless configuration for determining the advertisements to be presented.

In an embodiment, tracker 144 may provide a service that will attach a tracking ID to each advertisement. For example, online display advertisements may be tracked by pixels. Once a pixel gets serviced from the tracker 144, it can record the placement opportunity as well as the time and date; in addition, the machine may also record the user's ID and other information that identifies the user, including but not limited to an IP address, a geographic location, or some other type of data.

In an embodiment, the real-time quote log may record quote requests received and quote responses sent by the real-time quote machine 142. This log may contain additional data about which sites the user has visited, which can be used to derive user interests or browsing habits. In addition, this log may also record the arrival rates of advertising placement opportunities from different channels.

In an embodiment, the impressions, clicks, and action logs may be records that may be generated by a tracking system used by the learning machine facility 138.

In an embodiment, the learning machine facility 138 may be used to develop a targeting algorithm for the real-time ticker 142. The learning machine facility 138 can learn patterns (including social behaviors, inferred demographics, and other patterns) that can be used to target online advertisements.

In an example, an advertiser 104 may place an "order" with instructions that limit where and when an advertisement may be placed. The order may be received by the learning machine facility 138. The advertiser 104 may specify a 'goodness' criterion for campaign success. Such 'goodness' criteria may be measurable using the tracker 144. The advertiser 104 may provide historical data for bootstrapping the system. Based on the available data, the learning system 138 can develop custom targeting algorithms for advertisements. The algorithm may calculate the expected value of the advertisement given certain conditions and seek to maximize a specified 'goodness' criterion. The algorithm may be received by a real-time ticker 142 that may wait for an opportunity to place an advertisement. The bid request may be received by the real-time bid machine 142. The value of each bid request to each advertiser 104 may be evaluated using a received algorithm. A bid response may be sent for an advertisement having an enticing value. A lower value may be quoted if the estimate is appropriate. The bid response may request that the advertisement be placed at a particular price. The advertisement may be marked with a tracking system, such as pixels displayed in a browser. The tracker 144 may record ad impressions, user clicks, and user actions, and/or other data. The tracker logs can be sent to the learning system 138, which can use the 'goodness criteria' and decide which algorithms will be improved and further customize them. This process may be iterative. The system may also correlate the projected value to current events in the geographic area of the advertisement recipient.

In an embodiment, the real-time ticker 142 can dynamically update the targeting algorithm.

In an embodiment, the real-time ticker 142 can mix historical and real-time data to generate an algorithm for calculating a real-time ticker value.

In an embodiment, the real-time ticker 142 can calculate a projected value that combines information about: context of the ad layout, the user's history and geographic location information, and the ad itself or some other type of data.

In an embodiment, the real-time ticker 142 can use algorithms other than targeting "buckets".

In an embodiment, the real-time ticker 142 can use opportunistic algorithm updates by prioritizing the worst performing algorithms using the feedback of the tracker utility 144.

In an embodiment, the real-time quotation machine 142 may use an open list of a plurality of competing algorithms in the learning system 138 and the real-time quotation system 100A.

In an embodiment, the real-time ticker 142 can use control system theory to control pricing and serving speed for a set of advertisements.

In an embodiment, the real-time quotation machine 142 may construct a user profile using the winning and losing quotation data.

As shown in fig. 1B, in an embodiment, a real-time ticker can exchange ad bids across multiple exchanges 100B. Multiple exchanges are treated as a single inventory source.

Referring to FIG. 2, the analysis algorithms of the real-time bidding system may be used to optimize bid management, conversion, or some other type of advertising user interaction 200 associated with advertisements and advertisement impressions. In embodiments, a learning system embodied by the learning machine 138, for example, may generate rules regarding which advertisements perform well for a given customer and optimize content mixing for an advertising campaign based at least in part on the rules. In an example, a digital media user's behavior (such as an ad click-through, impression, web page visit, transaction or purchase, or third party data associated with the user) can be associated with and used by a learning system of a real-time bidding system. The real-time bidding system may use the output (e.g., rules and algorithms) of the learning system to pair the request for the advertisement with an advertisement selection that conforms to the rules and/or algorithms generated by the learning machine. The selected advertisement may be from an ad exchange, an inventory partner, or some other source of advertising content. The selected advertisement may then be associated with an advertisement tag as described herein and sent to a digital media user for presentation, such as on a web page. The ad tags may then be tracked and future impressions, click throughs, etc. recorded in a database associated with the real-time bidding system. The rules and algorithms may then be further optimized by the learning machine based at least in part on new interactions (or lack thereof) between the selected advertisement and the digital media user.

In an embodiment, a computer program product embodied in a computer-readable medium that, when executed on one or more computers, may dynamically determine an expected economic valuation for each of a plurality of potential layouts for an advertisement based at least in part on receiving a request to layout the advertisement for a publisher. In response to receiving a request to place an advertisement for a publisher, the methods and systems of the present invention may dynamically determine an expected economic valuation for each of a plurality of potential placements for the advertisement and/or advertisements and select and decide whether to present at least one of the plurality of advertisements and/or a plurality of available placements to the publisher based on the economic valuations.

In embodiments, the computer program implemented methods and systems may include altering the model for dynamically determining the economic valuation prior to processing the second request for the layout. The alteration of the model may be based at least in part on machine learning.

In an embodiment, prior to selecting and presenting at least one of the plurality of advertisements and/or the plurality of available layouts, the behavior of the economic valuation model can be altered to generate a second set of valuations for each of the plurality of layouts, wherein the selecting and presenting steps are based at least in part on the second set of valuations. The request for placement may be a time-limited request.

In an embodiment, the economic valuation model may evaluate performance information relating to each of a plurality of advertisement layouts.

In an embodiment, a dynamically variable economic valuation model can be used to determine the expected economic valuations. The dynamic variable economic valuation model can evaluate the value of the newspaper with respect to economic valuations for a plurality of layouts. The step of bidding for at least one of the plurality of advertisements and/or the plurality of available layouts may be based on the economic valuation.

Still referring to fig. 2, the real-time pricing system may include algorithms that match the description 200 above. Given the number of possible ads to be shown, the real-time bidding system may follow the following exemplary sequence: 1) all possible ads may be filtered to show using targeting rules, and the output of enumerated ads may be shown; 2) the system may check whether the potential advertisements have residual budget funds and may remove those advertisements from the list that have no available budget funds; 3) the system may run an economic valuation dynamic algorithm for the advertisements to determine an economic value for each advertisement; 4) value may be adjusted according to the cost of an opportunity to place an advertisement on a given site rather than an alternate site; 5) the advertisement with the highest value may be selected after being adjusted by the opportunity cost; 6) information about the first request (which may include information about the publisher content of the request) may be used to update the dynamic algorithm before receiving and processing the second request. This information may be used to determine whether a particular type of publisher content is frequently or infrequently available; and 7) the second advertisement may be processed in the same sequence as the first advertisement while the dynamic algorithm is updated before the third advertisement is placed.

In an embodiment, the dynamic algorithm may be similar to the algorithm used in an aircraft flight control system that adjusts for atmospheric conditions as atmospheric conditions change or an automobile cruise control system that dynamically adjusts throttle position as wind drag changes or the automobile climbs or descends a hill.

Referring to fig. 3, data relating to context, consumer (i.e., digital media user), and message/advertisement may be used to predict success of an advertisement based at least in part on specified key performance indicators 300. The context data may include data relating to: the media type, the time of day or week, or some other type of contextual data. Data related to a consumer or digital media user may include demographics, geographic data, and data related to consumer intent or behavior, or some other type of consumer data. The data relating to the message and/or advertisement may include data associated with the creative content, intent, or call to action embodied in the message/advertisement, or some other type of data of the message/advertisement.

As shown in fig. 4, the real-time bidding system may be used to generate ad campaign specific models and algorithms 400 that are continuously generated, tested, and run using data associated with campaign results (e.g., click-throughs, conversions, deals, etc.) as they become available in real-time. In an embodiment, multiple models may be tested using the preliminary data set to design a sample advertising campaign. Multiple models may be run against multiple training algorithms embodying a specified goal, such as a key performance indicator. Advertising content that performs well with respect to the algorithm may be maintained and presented to a plurality of digital media users. Additional data may be collected based at least in part on the interaction of the plurality of digital media users with the selected advertising content, and this data may be used to optimize the algorithm and select new or different advertising content for presentation to the plurality of digital media users.

Still referring to FIG. 4, in an embodiment, a computer program product embodied in a computer-readable medium, when executed on one or more computers, can deploy an economic valuation model 400 that can be refined through machine learning to evaluate information relating to a plurality of available layouts and/or a plurality of advertisements to predict an economic valuation for each of the plurality of layouts. At least one of a plurality of advertisements and/or a plurality of available layouts may be selected and presented to the publisher based at least in part on the economic valuation.

In embodiments, data may be taken from various formats (including but not limited to information not related to advertisements, such as successful market demographics, etc.). This may include a specific data stream, translating the data into a neutral format, a specific machine learning technique, or some other data type or technique. In embodiments, the learning system may perform auditing and/or supervision functions, including but not limited to optimizing methods and systems as described herein. In embodiments, a learning system may learn from multiple data sources and base optimization of the methods and systems as described herein at least in part on the multiple data sources.

In embodiments, the methods and systems as described herein may be used in an internet-based application, a mobile application, a fixed line application (e.g., wired media), or some other type of digital application.

In embodiments, the methods and systems as described herein may be used in a plurality of addressable advertising media (including, but not limited to, a set-top box, a digital billboard, a radio advertisement, or some other type of addressable advertising media).

Examples of machine learning algorithms may include, but are not limited to, naive bayes, bayesian networks, support vector machines, logistic regression, neural networks, and decision trees. These algorithms may be used to generate classifiers, which are algorithms that classify whether an advertisement is likely to be functional. In their basic form, they return a "yes" or "no" answer, and the score indicates the confidence strength of the classifier. When calibration techniques are applied, they return a probability estimate of the likelihood that the prediction will be accurate. They may also return what specific ads are most likely to be functional or which characteristics describe the ads most likely to be functional. These characteristics may include advertisement concept, advertisement size, advertisement color, advertisement text, or any other characteristic of the advertisement. In addition, they may also return what version of the advertiser's website is most likely to be functional or what characteristics describe the version of the advertiser's website that is most likely to be functional. These characteristics may include website concepts, products presented, colors, images, prices, text, or any other website characteristic. In an embodiment, the computer-implemented method of the present invention may include applying a plurality of algorithms to predict performance of an online advertising layout and tracking performance of the plurality of algorithms under a plurality of market conditions. Preferred performance conditions for the type of algorithm may be determined and market conditions tracked, and an algorithm for predicting performance of an advertising layout may be selected based at least in part on current market conditions. In an embodiment, the plurality of algorithms may include three algorithms.

In an embodiment, a computer program product embodied in a computer-readable medium, when executed on one or more computers, can predict an economic valuation for each of a plurality of available web publishable advertising layouts based in part on past performance and prices of similar advertising layouts using a master model. An economic valuation of each of the plurality of web publishable advertising layouts may be predicted by the second model, and the valuations produced by the primary model and the second model may be compared to determine a preference between the primary model and the second model. In an embodiment, the primary model may be an active model in response to a purchase request. The purchase request may be a time-limited purchase request.

In an embodiment, the second model may replace the primary model as an active model in response to the purchase request. The replacement may be based at least in part on a prediction that the second model will perform better than the primary model under current market conditions.

In embodiments, the computer-implemented method of the present invention may apply a plurality of algorithms to predict performance of an online advertising layout, track performance of the plurality of algorithms under a plurality of market conditions, and determine preferred performance conditions for the algorithm type. Market conditions may be tracked, and an algorithm for predicting performance of an advertising layout may be refined based at least in part on current market conditions.

In an embodiment, the computer-implemented method of the present invention may monitor a set of algorithms that each predicts a purchase price value for a set of advertisements and selects an optimal algorithm from the set of algorithms based at least in part on current market conditions.

Referring again to fig. 4, new data may be entered into the classification mechanism (shown as a funnel in fig. 4) (400). This data may be prepared for machine learning training by tagging each ad impression with an indicator that indicates whether the ad impression caused a click or action. An alternative machine learning algorithm can be trained on the labeled data. Portions of the annotated data may be saved for the testing phase. This test portion may be used to measure the predicted performance of each alternative algorithm. The most successful algorithm in predicting the outcome of the retrieved (hold-out) training data set may be forwarded to the real-time decision system.

In an embodiment, a computer program product embodied in a computer-readable medium that when executed on one or more computers may deploy a plurality of competing economic valuation models to predict an economic valuation for each advertisement layout of a plurality of advertisement layouts in response to receiving a request to lay out an advertisement for a publisher. The valuations produced by each of the plurality of competing economic valuation models can be evaluated to select one of the models for a current valuation of an advertising layout. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

In an embodiment, a computer program product embodied in a computer-readable medium that when executed on one or more computers may deploy a plurality of competitive economic valuation models to evaluate information relating to a plurality of available advertising layouts in response to receiving a request to lay out an advertisement. The economic valuation model can be used to predict an economic valuation for each advertisement layout of a plurality of advertisement layouts. The valuations produced by each of the plurality of competing economic valuation models can be evaluated to select one of the models for future valuations. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

In an embodiment, the data may be evaluated to determine whether it supports winning algorithms in the learning system. The incremental value of buying additional data can be determined and review and testing of the data samples can be used to determine whether the data increases the effectiveness of the prediction. For example, the system may use data derived from ad server logs combined with demographic information to derive a valuation model with a certain level of accuracy. Such a model may enable advertising that is below market prices for online advertising to benefit device manufacturers. Adding additional data sources (such as a list of consumers who have expressed their interest in buying a particular device) may increase the accuracy of the model, thus adding to the benefit of the device manufacturer. Declaring that the added receive benefit will be linked to the addition of a new data source, such data source may be assigned a value linked to the incremental benefit. While this example presents the case of online advertising, those skilled in the art will appreciate that the application can be generalized to advertising through different channels using different types of data sources and the following model: the model predicts economic value or pricing for advertisements.

As shown in FIGS. 5A and 5B, the advertising inventory may be divided into a plurality of segments or micro-segments (500, 502). The real-time bidding system may generate and continuously modify algorithms, such as by using a learning machine, based at least in part on the received performance (e.g., number of impressions or conversions associated with each advertisement) with respect to the advertisements in the inventory and micro-segments thereof. Based at least in part on the algorithms of the learning system, the real-time bidding system may generate a value that is considered "fair" with respect to the advertisement performance data. This bid value data, in turn, may be used to determine an average bid value to be associated with advertisements located in inventory. In embodiments, each differential segment may be associated with a rule, an algorithm, or a set of rules and/or algorithms, a payment price, and/or a budget. The rules may be used to buy an advertisement placement opportunity in a group of one or more opportunities. The size of the set of placement opportunities may be determined by the budget allocated to the rule. The rules may be sent to the seller of the advertisement placement opportunity through a server-to-server interface, through other electronic communication channels (including telephone and facsimile), through a paper-based order, through verbal communication, or any other means for communicating an order for buying an advertisement placement opportunity. FIG. 5C depicts the use of frequency analysis for pricing optimization 504 purposes. Fig. 5D depicts how pacing may be optimized by near cause analysis within the real-time quotation system 508. Referring now to fig. 6, a real-time bidding system may implement an automated analysis of ad inventory down to the nano-segment level (e.g., the bid value for each impression) in order to identify valuable segments (i.e., ads) of the otherwise low-value ad inventory 600. The real-time bidding system may generate and continuously modify algorithms based at least in part on received data regarding the performance of advertisements (e.g., number of impressions associated with each advertisement) in a nano-segment of the advertising inventory, for example, by using a learning machine. Based at least in part on the algorithms of the learning system, the real-time bidding system may generate a value for the bid that is considered "fair" with respect to the advertisement(s) in the nano-segment based at least in part on the performance data. In embodiments, the average bid price associated with a nano-segment may be adjusted based on other criteria (e.g., number of impressions associated with an advertisement). In embodiments, each nano-segment may be associated with a rule, an algorithm, or a set of rules and/or algorithms.

In an embodiment, a computer program product embodied in a computer-readable medium, when executed on one or more computers, may predict a purchase price for each of a plurality of available web publishable advertising layouts based, at least in part, on performance information and past bid prices for each of the plurality of advertising layouts. The purchase price for each of the plurality of advertisements may be tracked and predicted to determine pricing trends.

In an embodiment, the pricing trend may include predicting whether the valuation will change in the future.

In an embodiment, a computer program product embodied in a computer-readable medium that, when executed on one or more computers, may predict an economic valuation for each of a plurality of available web publishable advertising layouts based, at least in part, on performance information and past bid prices for each of the plurality of advertising layouts. Economic valuations for each of a plurality of advertisements can be tracked and predicted to determine pricing trends.

In an example, the system may present bids for buying advertisements in an auction, predicting the portion of them that are successful, and give it the advertisement for which the bid is sent. As the system operates, the portion of successful quotes may fall below the projected target. Such actions may occur with all or a subset of the available advertisements. The price trend prediction algorithm can estimate what corrections should be made to the bid price so that the successfully purchased portion of the advertisement becomes closer to the intended target and can eventually reach the intended target.

As shown in FIG. 7, the real-time bidding methods and systems as described herein may be integrated with, associated with, and/or linked 700 to a plurality of organizations and organization types, including but not limited to advertisers and advertising agencies. The real-time bidding system may use learning algorithms and techniques as described herein to perform bid-side optimization to optimize advertisement selection from a sell-side aggregator (such as a sell-side optimizer, advertising network, and/or exchange) that receives advertisements from content publishers. This may optimize the pairing of messages and advertisements available within the inventory with digital media users. The ad broker may include an internet-based advertising company, an ad seller (such as an organization that sells ad impressions displayed to digital media users), and/or an ad buyer. Advertisers and ad agencies may provide real-time bidding system ad campaign descriptors. The campaign descriptors may include, but are not limited to, channel, time, budget, or some other type of campaign descriptor data. In embodiments, the ad agent data may include a history log describing the layout and user impressions, conversions, etc. of each ad, including, but not limited to, an identifier associated with the user, a channel, a time, a payment price, an advertisement shown, a composite user action, or some other type of history data related to the ad and/or impression. The history log may also include data relating to spontaneous user actions. In embodiments, advertiser data utilized by the real-time bidding system may include, but is not limited to, metadata relating to the subject matter of the advertisement, such as the inventory level of the product that is the subject of the advertisement. Valuations, bid amounts, etc. can be optimized based on this and other metadata. Valuation, bid amount, etc. may be optimized based on the key performance indicators.

Fig. 8A and 8B depict hypothetical situation studies (800, 802) using a real-time bidding method and system. In embodiments, the learning system may use a training data set (such as a training data set derived from a prior retailer advertising campaign) to generate rules and algorithms as described herein. The training data set may include a record of prior impressions, transitions, actions, click-throughs, etc., performed by a plurality of digital media users and advertisements included in prior campaigns. The learning system may then identify a subset of advertising content from the prior campaign that is relatively more successful than other advertisements in the campaign and recommend it for future use based on a higher projected value of this advertising content.

In an embodiment, a computer program product embodied in a computer-readable medium, when executed on one or more computers, may deploy an economic valuation model to evaluate information relating to a plurality of available advertising layouts in response to receiving a request to lay out an advertisement. The economic valuation model may be used to predict economic valuations or pricing for bids for each of a plurality of advertisement layouts. A hypothetical regarding the market opportunity can be determined and the economic valuation model can be updated in response to the hypothetical market opportunity.

In an example, the system may discover data sets every few seconds or identify changes to a model that improves the accuracy of a valuation model used to predict the economic value of an advertisement. The system may have limitations on its ability to replace the valuation model as a whole at the same rate that new data is generated or changes to the model are made. It may therefore be beneficial to select which portions are less effective in providing an economic valuation. The opportunity update component can select what is the order and priority for replacing sections of the valuation model. Such prioritization may be based on the economic value of the section to be replaced versus the new section to be incorporated. The system can thus generate a prioritized instruction set as to what data or sections of the model will be added to the rating system and in what order to do so.

In embodiments, the methods and systems of the present invention may split an advertising campaign and compare performance from a first set of campaigns using the methods and systems as described herein with a second set of campaigns not using the methods and systems. The analysis comparison may show an elevation and charge (e.g., third party activity) based on the elevation between the first set and the second set.

In an example, the system may separate portions of the advertisement for generating a baseline sample to which the system is not applied, and thus may not give it revenue. Such a process may be automated. Such separation may be accomplished by random selection across the totality of available advertisements or to a small group of randomly selected users. The system may be used to place the remaining advertisements that do not belong to the baseline sample.

In an embodiment, when an ad campaign presents some goals that may be measured and the greater the revenue the better the campaign is judged to be, it indicates that the advertiser is believed to be willing to pay a premium for the ad campaign that is awarding increased revenue.

In embodiments, the pricing model may calculate the difference between the revenue generated from advertisements placed using the system and advertisements not placed using the system as on a baseline sample. The system gain is such a net difference. The price charged to the advertiser may be part of the system revenue.

Fig. 9 depicts a simplified flowchart that summarizes key steps 900 that may be involved when using the real-time quote method and system.

Fig. 10 depicts an exemplary embodiment of a user interface 1000 for a pixel provisioning system that may be associated with a real-time bidding system.

Fig. 11 depicts an exemplary embodiment of impression level data 1100 that may be associated with a real-time pricing system.

Fig. 12 depicts a hypothetical ad campaign performance report 1200.

FIG. 13 illustrates a bid valuation facility 1300 for purchasing real-time bids and valuations of an online advertising layout, in accordance with an embodiment of the present invention. The quote valuation facility 1300 may also include (among other facilities) a publisher facility 112, an analytics platform facility 114, an advertisement order sending and receiving facility 120, a context organizer service facility 132, a data integration facility 134, one or more databases providing different types of data for use by the analytics facility. In an embodiment of the present invention, the analytics platform facility 114 may include a learning machine facility 138, a valuation algorithm facility 140, a real-time quotation machine facility 142, a tracker machine facility 144, an impression/click/action log facility 148, and a real-time quotation log facility 150.

In an embodiment of the present invention, the learning machine 138 may be used to develop a targeting algorithm for the real-time ticker facility 142. The learning machine 138 may learn patterns (including social behaviors and inferred demographics, among other patterns) that may be used to target online advertisements. Additionally, the learning machine facility 138 can be coupled to one or more databases. In embodiments of the present invention, the one or more databases may include an ad agency/advertiser database 152. The ad agent data 152 may include campaign descriptors and may describe channels, times, budgets, and other information that may be allowed for disseminating ad messages. The ad agent data 152 may also include activity and history logs that may be a layout for each advertisement message to be shown to the user. The ad agent data 152 may include one or more of the following: an identifier for the user, a channel, a time, a payment price, an advertisement message shown, and a user composite user action or some other type of activity or historical log data. Additionally, advertiser data 152 may include business intelligence data or some other type of data that may describe dynamic and/or static marketing goals. In an example, the inventory excess amount of a given product that an advertiser 104 has in its warehouse may be described by advertiser data 152. Additionally, the one or more databases may include a historical events database. Historical event data 154 may be used to correlate the time of a user event with other events occurring in their area. In an example, the response rate to certain types of advertisements may be related to stock market trends. Historical event data 154 may include, but is not limited to, weather data, event data, local news data, or some other type of data. Additionally, the one or more data sets may include a user database. User data 158 may include data provided by a third party that may contain personal link information about the advertisement recipient. This information may provide the user with preferences or other indicators that may label or describe the user. Additionally, the one or more databases may include a real-time event database. The real-time event data 160 may include data that is similar to the historical data but more current. Real-time event data 160 may include, but is not limited to, data measured by the current time to seconds, minutes, hours, days, or some other time. In an example, if the learning machine facility 138 finds a correlation between the advertisement performance and the historical stock market index value, the real-time stock market index value may be used by the real-time ticker facility 142 to rate the advertisement. Additionally, the one or more databases can include a context database that can provide context data 162 associated with the publisher 112, the publisher's website, and the like. The one or more databases may also include third party/business databases.

Additionally, in embodiments of the present invention, the data integration facility 134 and the context organizer service facility 132 may be associated with the analytics platform facility 114 and one or more databases. The data integration facility 134 may facilitate integration of different types of data from one or more databases into the analysis platform facility 114. The context organizer service 132 may identify context categories for the media of the advertisement and/or publisher content, website, or other publisher advertising context. In an example, the context organizer may analyze the web content to determine whether the web page contains content about sports, finance, or some other topic. This information can be used as input to the learning machine facility to identify relevant publishers and/or web pages in which advertisements can appear. In another embodiment, the location of the advertisement on the publisher's 112 web page may be determined based on this information. In embodiments of the present invention, the context organizer service facility 132 may also be associated with the real-time ticker facility 142 and/or with one or more databases.

In embodiments of the present invention, the real-time ticker facility 142 can receive bid request messages from the publisher facility 112. The real-time quotation machine facility 142 can be considered a "real-time" facility in that it can reply to a quotation request associated with a time constraint, where the reply occurs substantially simultaneously with and/or in close temporal proximity to the request receipt. The real-time ticker facility 142 can use a non-stateless approach to calculate which advertising messages will be shown while the user waits for a system decision. The real-time ticker facility 142 can use algorithms provided by the learning machine 138 to perform real-time calculations to dynamically estimate the optimal bid value. In an embodiment, the alternate real-time ticker facility 142 may have a stateless configuration for determining advertisements to be presented.

Additionally, in embodiments of the present invention, the real-time ticker facility 142 can dynamically determine an expected economic valuation for each of a plurality of potential layouts for the advertisement based on receiving a request to lay out the advertisement for the publisher facility 112. In response to receiving a request to place an advertisement for the publisher facility 112, the real-time ticker facility 142 can dynamically determine an expected economic valuation for each of a plurality of potential layouts for the advertisement and can select and decide whether to present an available layout to the publisher facility 112 based on the economic valuations.

In an embodiment, the real-time ticker facility 142 can include altering the model for dynamically determining the economic valuation prior to processing the second request for the layout. The alteration of the model may be based at least in part on a machine learning facility. In embodiments of the present invention, prior to selecting and presenting at least one of the plurality of advertisements and/or the plurality of available layouts, the behavior of the economic valuation model can be altered to produce a second set of valuations for each of the plurality of layouts. In an embodiment, the step for selecting and presenting may be based on the second set of valuations. Additionally, in embodiments of the present invention, the request for placement may be a time-limited request. In addition, the economic valuation model can evaluate performance information relating to each of the plurality of advertisement layouts. A dynamic variable economic valuation model can also be used to determine the expected economic valuations. In embodiments of the invention, a dynamic variable economic valuation model may evaluate a value of a newspaper with respect to economic valuations for a plurality of layouts. The dynamic determination of the expected economic valuation for each of the plurality of potential layouts of the advertisement may be based at least in part on advertiser data 152, historical event data 154, user data 158, real-time event data 160, context data 162, and third party business data 164.

In an embodiment, the real-time ticker facility 142, in response to receiving a request to place an advertisement for a publisher 112, can dynamically determine an expected economic valuation for each of a plurality of potential placements for the advertisement. After the economic valuation model has been determined, the real-time ticker facility 142 can determine a bid amount based at least in part on an expected economic valuation for each of a plurality of potential layouts for the advertisement. The determination of the bid amount may include analyzing a real-time bid log. In another embodiment, the determination of the bid amount may include analytical modeling based at least in part on machine learning. The machine learning based analysis modeling may include analyzing historical log data summarizing at least one of: ad impressions, ad click throughs, and user actions taken in association with ad presentation. Additionally, in embodiments of the present invention, the determination of the bid amount may include analyzing data from the context organizer service 132.

In an embodiment of the present invention, the real-time ticker facility 142, in response to receiving a request to place an advertisement for a publisher 142, can dynamically determine a desired economic valuation for each of a plurality of potential placements for the advertisement. After the economic valuation model has been determined, the real-time ticker facility 142 can determine a bid amount based at least in part on an expected economic valuation for each of a plurality of potential layouts for the advertisement. The real-time ticker facility can then select an optimal layout for the advertisement from among a plurality of potential layouts. Additionally, the real-time ticker facility 142 can automatically offer bids for optimal placement of advertisements.

FIG. 14 illustrates a method 1400 for selecting and presenting at least one of a plurality of advertisements and/or a plurality of available layouts to a publisher based on economic valuations. The method begins at step 1402. At step 1404, in response to receiving a request to place an advertisement for a publisher, a desired economic valuation can be dynamically determined for each of a plurality of potential placements for the advertisement. At step 1408, at least one of a plurality of advertisements and/or a plurality of available layouts may then be selected and presented to the publisher based at least in part on the economic valuation. In an embodiment of the invention, the model for dynamically determining the economic valuation may be altered prior to processing the second request for the layout. In an embodiment, the model may be altered based at least in part on machine learning. In an embodiment of the invention, prior to the step of selecting and presenting, the behavior of the economic valuation model can be altered to produce a second set of valuations for each of the plurality of layouts. In an embodiment, the selecting step and the presenting step may be based on a second set of valuations used in place of the first valuation(s). In an embodiment, the request for placement may be a time-limited request. In an embodiment, an economic valuation model as described herein can evaluate performance information relating to each of a plurality of advertising layouts. A dynamic variable economic valuation model can be used to determine expected economic valuations and evaluate the valuations with respect to the economic valuations for the plurality of layouts. The expected economic valuation for each of the plurality of potential layouts for the advertisement may be based at least in part on advertiser data, historical event data, user data, real-time event data, context data, or third party business data. The method terminates at step 1410.

FIG. 15 illustrates a method 1500 for determining a bid amount according to an embodiment of the invention. The method begins at step 1502. At step 1504, in response to receiving a request to place an advertisement for a publisher, an expected economic valuation for each of a plurality of potential layouts for the advertisement may be dynamically determined. Then, at step 1508, an amount of the offer is determined based at least in part on the expected economic valuations for each of the plurality of potential layouts for the advertisement. In embodiments of the present invention, the determination of the bid amount may include real-time bid log analysis and/or analytical modeling based at least in part on machine learning. In an embodiment of the invention, the analytical modeling may include analyzing historical log data summarizing at least one of: ad impressions, ad click throughs, and user actions taken in association with ad presentation. In embodiments of the present invention, determining the bid amount may include analyzing data from a context organizer service.

FIG. 16 illustrates a method 1600 for automatically offering an optimal layout for an advertisement, where the optimal layout is selected based at least in part on a desired economic valuation. The method begins at step 1602. At step 1604, an expected economic valuation for each potential layout of the plurality of potential layouts for the advertisement is dynamically determined in response to receiving a request to layout the advertisement for the publisher. Then at step 1608, an amount of the offer is determined based at least in part on the expected economic valuation for each of the plurality of potential layouts for the advertisement. Additionally, at step 1610, an optimal layout for the advertisement is selected from among the plurality of potential layouts based at least in part on the bid amount. Finally, at step 1612, bids for optimal placement of the advertisement are automatically presented. The method terminates at step 1614.

Fig. 17 illustrates a real-time facility 1700 for targeting offers for online advertising purchases, according to an embodiment of the invention. The real-time facilities may include a learning machine facility 138 and a real-time ticker facility 142. In embodiments of the present invention, the real-time ticker facility 142 can receive bid request messages from the publisher facility 112. The real-time ticker facility 142 can be viewed as a "real-time" facility in that it can reply to a bid request associated with a time constraint. The real-time ticker facility 142 can use a targeting algorithm provided by the learning machine 138 to perform real-time calculations to dynamically estimate an optimal bid value.

Additionally, in embodiments of the present invention, the real-time ticker facility 142 can deploy an economic valuation model that can dynamically determine an economic valuation for each of one or more potential layouts for an advertisement (based on receiving a request to lay out the advertisement for the publisher facility 112). In response to receiving a request to place an advertisement for the publisher facility 112, the real-time ticker facility 142 can dynamically determine an economic valuation for each of one or more potential placements for the advertisement. After the economic valuations have been determined, the real-time ticker facility 142 can select and present at least one of a plurality of advertisements and/or a plurality of available layouts to the user based on the economic valuations. In embodiments, the selection and presentation to the publisher 112 may include a recommended bid amount for at least one of a plurality of advertisements and/or a plurality of available layouts. The bid amount may be associated with a time constraint. Additionally, in embodiments, refinement through machine learning may include comparing economic valuation models by retrospectively comparing how much the models reflect the actual economic performance of advertisements. In embodiments of the present invention, the economic valuation model may be based at least in part on ad agency data 152, real-time event data 160, historical event data 154, user data 158, third party business data 164, and context data 162. In an embodiment, the ad agent data 152 may include at least one campaign descriptor. In embodiments, the campaign descriptor may be historical log data, ad agent campaign budget data, and data indicating time constraints on ad placement.

In an embodiment, the learning machine facility 138 can receive an economic valuation model. The economic valuation model can be based at least in part on analyzing the real-time quote log data 150 from the real-time quote machine facility 142. The learning machine facility 138 can then refine the economic valuation model. The refinement may be based at least in part on analyzing the ad impression log. In an embodiment of the present invention, the refined economic valuation model can include a data integration step during which data to be used in the learning machine facility 138 can be transformed into a data format that can be read by the learning machine facility 138. The format may be a neutral format. Further, in an embodiment, refining the economic valuation model using a learning machine may be based at least in part on a machine learning algorithm. The machine learning algorithm may be based at least in part on a naive bayes analysis technique and a logistic regression analysis technique. In addition, the real-time ticker facility 142 can categorize each of the plurality of available advertising layouts using a refined economic valuation model. The classification may be data that indicates a probability that each available ad layout achieves an ad impression. The real-time ticker facility 142 can then prioritize the available ad layouts based at least in part on data indicative of a probability of achieving an ad impression. The real-time ticker facility 142 can then select and present at least one of a plurality of advertisements and/or a plurality of available layouts to the user based on the prioritization.

In an embodiment of the invention, the economic estimation model deployed by the real-time ticker facility 142 can be refined by a learning machine facility to evaluate information relating to one or more available layouts to predict an economic valuation for each of the one or more layouts. Additionally, in embodiments, the learning machine facility 138 may obtain different types of data for refining the economic valuation model. The different types of data may include, without limitation, agent data 152, which may include campaign descriptors and may describe channels, times, budgets, and other information that may be allowed for disseminating advertising messages. The agent data 152 may also include activity and history logs, which may be a layout for each advertisement message to be shown to the user. The proxy data 152 may also include one or more of the following: an identifier for the user, a channel, a time, a payment price, an advertisement message shown, and a user composite user action or some other type of activity or historical log data. In addition, the different types of data may include enterprise intelligence data or some other type of data that may describe dynamic and/or static marketing goals.

In embodiments of the present invention, the learning machine facility 138 may perform auditing and/or supervisory functions (including, but not limited to, optimizing methods and systems as described herein). In other embodiments of the present information, the learning system 138 can learn from multiple data sources and base optimization of the methods and systems as described herein at least in part on the multiple data sources. In embodiments, the methods and systems as described herein may be used in an internet-based application, a mobile application, a fixed line application (e.g., wired media), or some other type of digital application. In embodiments, the methods and systems as described herein may be used in one or more addressable advertising media (including, but not limited to, a set-top box, a digital billboard, a radio advertisement, or some other type of addressable advertising media).

Additionally, in embodiments of the present invention, the learning machine facility 138 may utilize various types of algorithms to refine the economic valuation model of the real-time ticker facility 142. Algorithms may include, without limitation, decision tree learning, association rule learning, artificial neural networks, genetic programming, inductive logic programming, support vector machines, clustering, bayesian networks, and reinforcement learning. In embodiments of the present invention, various types of algorithms may generate classifiers, which are algorithms that may classify whether an advertisement is likely to be functional. In their basic form, they may return a "yes" or "no" answer and/or a score that indicates the confidence strength of the classifier. When calibration techniques are applied, they can return a probability estimate of the likelihood that the prediction will be correct.

FIG. 18 illustrates a method 1800 for selecting and presenting to a user at least one of a plurality of available advertising layouts based on economic valuations. The method begins at step 1802. At step 1804, an economic valuation model can be deployed in response to receiving a request to place an advertisement for a publisher. The economic valuation model can be refined through machine learning to evaluate information relating to a plurality of available layouts and/or a plurality of advertisements to predict an economic valuation for each of the plurality of layouts. In embodiments, refinement through machine learning may include comparing economic valuation models by retrospectively comparing how much the models reflect the actual economic performance of advertisements. Additionally, the economic valuation model can be based at least in part on ad agency data, real-time event data, historical event data, user data, third party business data, and context data. Additionally, the ad agent data may include at least one campaign descriptor. Additionally, the campaign descriptors may be historical log data, ad agent campaign budget data, and ad agent campaign budget data. At step 1808, at least one of a plurality of advertisements and/or a plurality of available layouts may be selected and presented to the user based on the economic valuation. In embodiments, selecting and presenting to the publisher may include a recommended bid amount for at least one of a plurality of available layouts and/or a plurality of advertisements. Additionally, the bid amount may be associated with a time constraint. The method 1800 terminates at step 1810.

FIG. 19 illustrates a method 1900 for selecting prioritized layout opportunities from a plurality of available advertisement layouts based at least in part on an economic valuation model using real-time bid log data. The method 1900 begins at step 1902. At step 1904, an economic valuation model at the learning machine can be received. The economic valuation model can be based at least in part on analyzing a real-time quote log from a real-time quote machine. At step 1908, the economic valuation model can be refined using a learning machine. In an embodiment, the refinement may be based at least in part on analyzing an ad impression log. In addition, the refined economic valuation model can include a data integration step during which data to be used in the learning machine can be transformed into a data format that the learning machine can read. In an embodiment, the format may be a neutral format. Additionally, refining the economic valuation model using a learning machine can be based at least in part on a machine learning algorithm. The machine learning algorithm may be based at least in part on a naive bayes analysis technique. Additionally, the machine learning algorithm may be based at least in part on logistic regression analysis techniques. At step 1910, the refined economic valuation model can be used to classify each of a plurality of available advertising layouts. Each category may be summarized using data indicating the probability that each available ad layout achieves an ad impression. Additionally, at step 1912, the available advertising layouts may be prioritized based at least in part on the data. Further, at step 1914, at least one of a plurality of advertisements and/or a plurality of available layouts may be selected and presented to the user based on the prioritization. The method 1900 terminates at step 1918.

FIG. 20 illustrates a real-time facility 2000 for selecting an alternative algorithm for predicting purchase price trends for offers for online advertising, according to an embodiment of the present invention. The real-time facility 1700 may include a learning machine facility 138, a valuation algorithm facility 140, a real-time quotation machine facility 142, a plurality of data 2002, and a quotation request message 2004 from the publisher facility 112. In an embodiment of the present invention, the real-time ticker facility 142 can receive a bid request message 1704 from the publisher facility 112. The real-time ticker facility 142 can be viewed as a "real-time" facility in that it can reply to a bid request associated with a time constraint. The real-time ticker facility 142 can perform real-time calculations using targeting algorithms provided by the learning machine facility 138 to predict purchase price trends for offers for online advertising. In an embodiment of the present invention, the learning machine facility 138 may select an alternative algorithm based on performance of the current working algorithm for predicting purchase price trends for offers for online advertising. In another embodiment of the present invention, the learning machine facility 138 can select an alternative algorithm based on its performance for predicting purchase price trends for offers for online advertising. Additionally, in embodiments of the present invention, the learning machine facility 138 can obtain alternative algorithms from the valuation algorithms facility 140.

In an embodiment, the real-time ticker facility 142 can apply a plurality of algorithms to predict the performance of an online advertising layout. Once applied, the real-time ticker facility 142 can track the performance of multiple algorithms under a variety of market conditions. The real-time ticker facility 142 can then determine performance conditions for the algorithm type from the plurality of algorithms. The real-time ticker facility 142 can then track market conditions and can select an algorithm for predicting performance of an advertising layout based on current market conditions.

In an embodiment, at least one of the plurality of algorithms for predicting performance may include advertiser data 152. Advertiser data 152 may include business intelligence data or some other type of data that may describe dynamic and/or static marketing goals. In another embodiment of the present invention, at least one of the plurality of algorithms for predicting performance may include historical event data 154. Historical event data 154 may be used to correlate the time of a user event with the occurrence of other events in their area. In an example, the response rate to certain types of advertisements may be related to stock market trends. Historical event data 154 may include, but is not limited to, weather data, event data, local news data, or some other type of data. In yet another embodiment of the present invention, at least one of the plurality of algorithms for predicting performance may include user data 158. User data 158 may include data provided by a third party that may contain personal link information about the advertisement recipient. This information may provide the user with preferences or other indicators that may label or describe the user. In yet another embodiment of the present invention, at least one of the plurality of algorithms for predicting performance may include real-time event data 160. The real-time event data 160 may include data that is similar to the historical data but more current. Real-time event data 160 may include, but is not limited to, data measured by the current time to seconds, minutes, hours, days, or some other time. In yet another embodiment of the present invention, at least one of the plurality of algorithms for predicting performance may include context data 162. In yet another embodiment of the present invention, at least one of the plurality of algorithms for predicting performance may include third party business data.

Additionally, in embodiments of the present invention, the real-time ticker facility 142 can use a primary model for predicting an economic valuation of each of a plurality of available web publishable advertising layouts based at least in part on past performance and prices of similar advertising layouts. The real-time ticker facility 142 can also use a second model for predicting an economic valuation of each of the plurality of web-publishable advertising layouts. After using the primary model and the secondary model to predict the economic valuations, the real-time ticker facility 142 can compare the valuations generated by the primary model and the secondary model to determine preferences between the primary model and the secondary model. In embodiments of the present invention, comparing valuations may include retrospectively comparing the extent to which the model reflects the actual economic performance of the advertisement. Additionally, in embodiments of the present invention, the primary model may be an active model that responds to purchase requests. The purchase request may be a time-limited purchase request. In an embodiment of the invention, the second model may replace the primary model as an active model in response to the purchase request. Additionally, the replacement may be based on a prediction that the second model may perform better than the primary model under current market conditions. In embodiments of the present invention, the prediction may be based at least in part on machine learning, historical advertising performance data 130, historical event data, and real-time event data 160.

In another embodiment of the present invention, the real-time ticker facility 142 can use a primary model for predicting an economic valuation of each of a plurality of available mobile device advertising layouts based in part on past performance and prices of similar advertising layouts. The real-time ticker facility 142 can also use a second model for predicting an economic valuation of each of a plurality of mobile device advertising layouts. After using the primary model and the secondary model to predict the economic valuations, the real-time ticker facility 142 can compare the valuations generated by the primary model and the secondary model to determine preferences between the primary model and the secondary model. In embodiments of the present invention, comparing valuations may include retrospectively comparing the extent to which the model reflects the actual economic performance of the advertisement. Additionally, in embodiments of the present invention, the primary model may be an active model that responds to purchase requests. The purchase request may be a time-limited purchase request. In an embodiment of the invention, the second model may replace the primary model as an active model in response to the purchase request. Additionally, the replacement may be based on a prediction that the second model may perform better than the primary model under current market conditions.

In an embodiment of the present invention, the economic valuation model deployed by the real-time ticker facility 142 can be refined by the machine learning facility 138 to evaluate information relating to one or more available layouts to predict an economic valuation for each of the one or more layouts.

In embodiments, the learning machine facility 138 may obtain different types of data for refining the economic valuation model. The different types of data may include, but are not limited to, advertiser data 152, historical event data 154, user data 158, real-time event data 160, context data 162, and third party business data. The different types of data may have different formats and information (such as market demographics, etc.) that may not be directly related to the advertisement. In embodiments of the present invention, different types of data in different formats may be translated into a neutral format or some other data type that is specific to a format compatible with the learning machine facility 138 or suitable for the learning machine facility 138.

In embodiments, the learning machine facility 138 may utilize various types of algorithms to refine the economic valuation model of the real-time ticker facility 142. Algorithms may include, without limitation, decision tree learning, association rule learning, artificial neural networks, genetic programming, inductive logic programming, support vector machines, clustering, bayesian networks, and reinforcement learning.

FIG. 21 illustrates a method 2100 for predicting performance of an advertising layout based on current market conditions. The method begins at step 2102. At step 2104, a plurality of algorithms for predicting performance of an online advertising layout may be applied. In embodiments of the present invention, at least one of the plurality of algorithms for predicting performance may include advertiser data, historical event data, user data, real-time event data, context data, and third party business data, or some other type of data. The performance of the plurality of algorithms can then be tracked under various market conditions at step 2108. Additionally, at step 2110, performance for the algorithm type may be determined, and then market conditions may be tracked at step 2112. Finally, at step 2114, an algorithm for predicting performance of the advertising layout can be selected based on current market conditions. The method terminates at step 2118.

FIG. 22 illustrates a method 2200 for determining a preference between a primary model and a secondary model for predicting economic valuations in accordance with an embodiment of the invention. The method begins at step 2202. At step 2204, an economic valuation for each of the plurality of available web publishable advertising layouts may be predicted using the primary model. Economic valuation may be based in part on past performance and prices of similar advertising layouts. At step 2208, an economic valuation for each of the plurality of available web publishable advertising layouts may be predicted using the second model. Subsequently, at step 2210, the economic valuations using the primary model and the secondary model can be compared to determine a preference between the primary model and the secondary model. In embodiments of the present invention, comparing valuations may include retrospectively comparing the extent to which the model reflects the actual economic performance of the advertisement. Additionally, in embodiments of the present invention, the primary model may be an active model that responds to purchase requests. The purchase request may be a time-limited purchase request. In an embodiment of the invention, the second model may replace the primary model as an active model in response to the purchase request. Additionally, the replacement may be based on a prediction that the second model may perform better than the primary model under current market conditions. In embodiments of the present invention, the prediction may be based at least in part on machine learning, historical advertising performance data, historical event data, and real-time event data. The method terminates at step 2212.

Referring now to FIG. 23, a diagram illustrates a method 2300 for determining preferences between a primary model and a secondary model for predicting economic valuations, in accordance with another embodiment of the invention. The method starts at step 2302. At step 2304, an economic valuation for each of a plurality of available mobile device advertising layouts can be predicted using the primary model. Economic valuation may be based in part on past performance and prices of similar advertising layouts. At step 2308, an economic valuation for each of a plurality of available mobile device advertising layouts can be predicted using the second model. Subsequently, at step 2310, the economic valuations using the primary model and the secondary model can be compared to determine a preference between the primary model and the secondary model. In embodiments of the present invention, comparing valuations may include retrospectively comparing the extent to which the model reflects the actual economic performance of the advertisement. Additionally, in embodiments of the present invention, the primary model may be an active model that responds to purchase requests. The purchase request may be a time-limited purchase request. In an embodiment of the invention, the second model may replace the primary model as an active model in response to the purchase request. Additionally, the replacement may be based on a prediction that the second model may perform better than the primary model under current market conditions. The method terminates at step 2312.

Additionally, in embodiments of the present invention, the real-time ticker facility 142 can receive a request from the publisher facility 112 to place an advertisement. In response to this request, the real-time ticker facility 142 can deploy a plurality of competing economic valuation models to predict an economic valuation for each of a plurality of available advertising layouts. After deploying the plurality of economic valuation models, the real-time ticker facility 142 can evaluate each valuation produced by each of the plurality of competing economic valuation models to select one economic valuation model as the current valuation of the advertising layout.

In an embodiment of the invention, the economic valuation model may be based at least in part on the real-time event data 160. The real-time event data 160 may include data that is similar to the historical data but more current. Real-time event data 160 may include, but is not limited to, data measured by the current time to seconds, minutes, hours, days, or some other time. In another embodiment of the invention, the economic valuation model can be based at least in part on historical event data 154. Historical event data 154 may be used to correlate the time of a user event with the occurrence of other events in their area. In an example, the response rate to certain types of advertisements may be related to stock market trends. Historical event data 154 may include, but is not limited to, weather data, event data, local news data, or some other type of data. In yet another embodiment of the present invention, the economic valuation model may be based at least in part on the user data 158. User data 158 may include data provided by a third party that may contain personal link information about the advertisement recipient. This information may provide the user with preferences or other indicators that may label or describe the user. In yet another embodiment of the invention, the economic valuation model can be based at least in part on third party business data. In embodiments of the present invention, the third party commercial data may include financial data relating to historical advertising impressions. In yet another embodiment of the present invention, the economic valuation model may be based at least in part on the contextual data 162. In another embodiment of the invention, the economic valuation model can be based at least in part on advertiser data 152. Advertiser data 152 may include business intelligence data or some other type of data that may describe dynamic and/or static marketing goals. In yet another embodiment of the present invention, the economic valuation model may be based at least in part on the ad agency data 152. The ad agent data 152 may also include activity and history logs, which may be layouts for each advertisement message to be shown to the user. The ad agent data 152 may also include one or more of the following: an identifier for the user, a channel, a time, a payment price, an advertisement message shown, and a user composite user action or some other type of activity or historical log data. In yet another embodiment of the present invention, the economic valuation model can be based at least in part on historical advertisement performance data 130. In yet another embodiment of the present invention, the economic valuation model can be based at least in part on machine learning.

In an embodiment of the present invention, the economic valuation model deployed by the real-time ticker facility 142 can be refined by the machine learning facility 138 to evaluate information relating to one or more available layouts to predict an economic valuation for each of the one or more layouts.

In an embodiment of the present invention, after the real-time ticker facility 142 receives a request to place an advertisement from the publisher facility 112, the real-time ticker facility 142, in response to this request, can deploy a plurality of competing economic valuation models to predict an economic valuation for each of the plurality of advertisement placements. After deploying the plurality of economic valuation models, the real-time ticker facility 142 can evaluate each valuation produced by each of the plurality of competing economic valuation models to select one as the first valuation for the advertising layout. Upon selecting the first valuation, the real-time ticker facility 142 can re-evaluate each valuation produced by each of the plurality of competing economic valuation models to select one as the revised valuation for the advertising layout. In an embodiment of the invention, the revised valuation may be based at least in part on an analysis of an economic valuation model that uses real-time event data 160 that was not available at the time the first valuation was selected. The real-time ticker facility 142 can then replace the first valuation with a second revised valuation for use in deriving a recommended bid amount for the advertising layout. In embodiments of the invention, a request may be received from the publisher 112 and a recommended bid amount may be automatically sent to the publisher 112. In another embodiment of the invention, a request may be received from the publisher 112 and a bid equal to the recommended bid amount may be automatically offered on behalf of the publisher 112. In embodiments of the present invention, the recommendation bid amount may be associated with a recommendation time for an advertisement layout. In another embodiment of the present invention, the recommended bid amount may also be derived by analyzing a real-time bid log that may be associated with the real-time ticker facility 142. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

In another embodiment of the present invention, after the real-time ticker facility 142 receives a request from the publisher 112 to place an advertisement, the real-time ticker facility 142 can deploy a plurality of competing economic valuation models to evaluate information relating to a plurality of available advertisement placements. The real-time ticker facility 142 can deploy competitive economic valuation models to predict an economic valuation for each of a plurality of advertising layouts. After deploying the plurality of economic valuation models, the real-time ticker facility 142 can evaluate each valuation produced by each of the plurality of competing economic valuation models to select one valuation as a future valuation of the advertising layout. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

In another embodiment of the present invention, after the real-time ticker facility 142 receives a request to place an advertisement from the publisher facility 112, the real-time ticker facility 142 can deploy a plurality of competing economic valuation models to evaluate information relating to a plurality of available advertisement placements. The real-time ticker facility 142 can deploy competitive economic valuation models to predict an economic valuation for each of a plurality of advertising layouts. After deploying the plurality of economic valuation models, the real-time ticker facility 142 can evaluate each valuation produced by each of the plurality of competing economic valuation models in real-time to select one valuation as a future valuation of the advertising layout. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention. In embodiments of the present invention, the future valuation may be based at least in part on simulation data describing future events. In an embodiment of the invention, the future event may be a stock market fluctuation. Additionally, in embodiments of the present invention, simulation data describing future events may be derived from analysis of historical event data.

In embodiments of the present invention, after the real-time ticker facility 142 receives a request to place an advertisement from the publisher facility 112, the real-time ticker facility 142 can deploy a plurality of competing real-time bidding algorithms involving a plurality of available advertisement placements to bid on the advertisement placements. After deploying the plurality of competing real-time quotation algorithms, the real-time quotation engine facility 142 may evaluate each quotation algorithm to select a preferred algorithm. In embodiments of the present invention, the competitive real-time quotation algorithm may use data from the real-time quotation log. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

In another embodiment of the present invention, after the real-time ticker facility 142 receives a request to place an advertisement from the publisher facility 112, the real-time ticker facility 142 can deploy multiple competing real-time bidding algorithms involving multiple available advertisement placements. The real-time ticker facility 142 can deploy a plurality of competing real-time bidding algorithms to bid for an advertising layout. After deploying the plurality of competing real-time quotation algorithms, the real-time quotation engine facility 142 may evaluate each quotation recommendation generated by the competing real-time quotation algorithms. The real-time quotation machine facility 142 may re-evaluate each of the quotation recommendations generated by the competing real-time quotation algorithms to select one as a revised quotation recommendation. In embodiments of the present invention, the revised offer recommendations may be based at least in part on a real-time offer algorithm that uses real-time event data 160 that is not available at the time the offer recommendation is selected. The real-time ticker facility 142 can then replace the bid recommendations with revised bid recommendations for use in deriving a recommended bid amount for the advertising layout. In embodiments of the present invention, the replacement may occur in real-time with respect to receiving a request to place an advertisement.

Referring now to fig. 24, a diagram illustrates a method 2400 for selecting one valuation model among a plurality of competing valuation models in a real-time bid for an advertising layout, in accordance with an embodiment of the present invention. The method begins at step 2402. At step 2404, in response to receiving the request to place the advertisement, a plurality of competing economic valuation models can be deployed to predict an economic valuation for each advertisement placement of the plurality of advertisement placements. Each valuation generated by each of the plurality of competing economic valuation models can then be evaluated to select one of the valuation models as a current valuation of the advertisement layout at step 2408. In embodiments of the present invention, the economic valuation model may be based at least in part on real-time event data, historical event data, user data, context data, advertiser data, advertising agency data, historical advertising performance data, machine learning, and third party business data. In embodiments of the present invention, the third party commercial data may include financial data relating to historical advertising impressions. The method terminates at step 2410. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

FIG. 25 illustrates a method 2500 for replacing a first economic valuation model with a second economic valuation model to derive a recommended bid amount for an advertising layout. The method begins at step 2502. After step 2504, in response to receiving a request to place an advertisement, a plurality of competing economic valuation models can be deployed to predict an economic valuation for each advertisement placement of the plurality of advertisement placements. Subsequently, at step 2508, the valuations generated by each of the plurality of competing economic valuation models can be evaluated, and then a first valuation of an advertisement layout can be selected. Additionally, at step 2510, each valuation generated by each of the plurality of competing economic valuation models can be re-evaluated. One of the competing economic valuation models can then be selected as a revised valuation of the advertisement layout. The revised valuation can be based at least in part on an analysis of an economic valuation model that uses real-time event data that is not available at the time the first valuation is selected. Additionally, at step 2512, the first valuation can be replaced with a second revised valuation to be used in deriving a recommended bid amount for the advertising layout. In embodiments of the invention, a request may be received from a publisher and a recommended bid amount may be automatically sent to the publisher. In another embodiment of the invention, a request may be received from a publisher and a bid equal to the recommended bid amount may be automatically offered on behalf of the publisher. In yet another embodiment of the present invention, the recommendation bid amount may be associated with a recommendation time for an advertisement layout. Additionally, in another embodiment of the present invention, the recommended bid amount may also be derived by analyzing a real-time bid log associated with the real-time bid machine. The method terminates at step 2514. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

FIG. 26 illustrates a method 2600 of evaluating a plurality of economic valuation models and selecting one valuation as a future valuation of an advertising layout in accordance with one embodiment of the invention. The method begins at step 2602. At step 2604, in response to receiving a request to place an advertisement, a plurality of competing economic valuation models can be deployed. Information relating to a plurality of available advertising layouts may be evaluated to predict an economic valuation for each of the plurality of advertising layouts. Additionally, at step 2608, each valuation produced by each of the plurality of competing economic valuation models can be evaluated to select one valuation as a future valuation of the advertisement layout. The method terminates at step 2610. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

FIG. 27 illustrates a method 2700 for evaluating multiple economic valuation models in real time and selecting one valuation as a future valuation of an advertising layout in accordance with an embodiment of the invention. The method begins at step 2702. At step 2704, in response to receiving a request to place an advertisement, a plurality of competing economic valuation models can be deployed. Information relating to a plurality of available advertising layouts may be evaluated to predict an economic valuation for each of the plurality of advertising layouts. Each valuation generated by each of the plurality of competing economic valuation models can then be evaluated in real time to select one valuation as a future valuation of the advertising layout at step 2708. In embodiments of the present invention, the future valuation may be based at least in part on simulation data describing future events. In another embodiment of the invention, the future event may be a stock market fluctuation. In embodiments of the present invention, simulation data describing future events may be derived from an analysis of historical event data that may be selected based at least in part on context data relating to advertisements to be placed in an advertisement layout. The method terminates at step 2710. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

FIG. 28 illustrates a method 2800 for evaluating multiple bidding algorithms to select a preferred algorithm for placement of an advertisement, according to an embodiment of the present invention. The method begins at step 2802. At step 2804, in response to receiving a request to place an advertisement, a plurality of competing real-time bidding algorithms may be deployed. The bidding algorithm may be related to a plurality of available ad layouts to bid on an ad layout. Each quotation algorithm can then be evaluated to select a preferred algorithm at step 2808. The method terminates at step 2810. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

FIG. 29 illustrates a method 2900 for replacing bid recommendations with revised bid recommendations for advertisement placement, according to an embodiment of the invention. The method begins at step 2902. At step 2904, in response to receiving a request to lay out an advertisement, a plurality of competing real-time bidding algorithms relating to a plurality of available advertisement layouts may be deployed to bid on the advertisement layouts. At step 2908, each bid recommendation generated by the competing real-time bid algorithm may be evaluated. Additionally, at step 2910, each bid recommendation generated by the competing real-time bid algorithm may be re-evaluated to select one as a revised bid recommendation. In an embodiment, the revised offer recommendation is based at least in part on a real-time offer algorithm that uses real-time event data that is not available at the time the offer recommendation is selected. Then at step 2912, the bid recommendations may be replaced with revised bid recommendations for use in deriving a recommended bid amount for the advertisement layout. In embodiments of the present invention, the replacement may occur in real-time with respect to receiving a request to place an advertisement. The method terminates at step 2914. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

FIG. 30 is a diagram illustrating a real-time facility 3000 for measuring the value of additional third party data 164, according to an embodiment of the present invention. The real-time facility 2700 may include the learning machine facility 138, the valuation algorithm facility 140, the real-time quotation machine facility 142, the additional third party data set 3002, the quotation request message 3004 from the publisher facility 112, and the tracking facility 144. In an embodiment of the present invention, the real-time quotation machine facility 142 may receive a quotation request message 3004 from the publisher facility 112. The real-time ticker facility 142 can be viewed as a "real-time" facility in that it can reply to a bid request associated with a time constraint. The real-time ticker facility 142 can perform real-time calculations using the targeting algorithm provided by the learning machine facility 138. In an embodiment of the present invention, the real-time ticker facility 142 can deploy an economic valuation model to perform real-time calculations.

In an embodiment, the learning machine facility 138 can obtain a third party data set 3002 for use in refining the economic valuation model. In an embodiment of the present invention, the third party data set 2702 may include data related to advertising content users. In embodiments of the present invention, the data related to the advertising content user may include demographic data, transactional data, conversion data, or some other type of data. In another embodiment of the invention, the third party data set may include contextual data 162 relating to a plurality of available layouts and/or a plurality of advertisements. In embodiments of the present invention, the context data 162 may be derived from the context organizer service 132, which may be associated with the learning machine facility 138. In yet another embodiment of the invention, the third party data set 3010 may include financial data related to historical advertising impressions. Additionally, in embodiments of the present invention, the economic valuation model may be based at least in part on real-time event data, historical event data 154, user data 158, third party business data, advertiser data 152, and ad agency data 152.

In embodiments of the present invention, the real-time ticker facility 142 can receive the ad campaign data set and can split the ad campaign data set into a first ad campaign data set and a second ad campaign data set. Subsequently, the real-time ticker facility 142 can deploy an economic valuation model that can be refined through machine learning to evaluate information relating to a plurality of available layouts and/or a plurality of advertisements to predict an economic valuation for the layout of advertisement content from the first set of advertisement campaign data. In embodiments of the invention, machine learning may be based at least in part on third party data sets. Machine learning may be implemented by a learning machine facility 138. After refining the evaluation model, the real-time ticker facility 142 can place advertising content from the first and second sets of advertising campaign data within a plurality of available placements and/or a plurality of advertisements. Content from the first advertising campaign may be laid out based at least in part on the predicted economic valuation, and content from the second advertising campaign dataset may be laid out based on a method that is not dependent on the third party dataset. The real-time ticker facility 142 can also receive impression data from the tracker facility 144 that can relate to advertising content laid out from the first and second advertising campaign data sets. In embodiments of the present invention, impression data may include data regarding a user's interaction with advertising content. The real-time ticker facility 142 can then determine a value of the third-party dataset based at least in part on comparing impression data related to advertising content placed from the first and second ad campaign datasets.

Additionally, in embodiments of the present invention, the real-time ticker facility 142 can calculate a valuation of the third-party dataset 3002 based at least in part on comparing advertisement impression data related to advertisement content placed from the first and second advertisement campaign datasets. In embodiments of the present invention, the placement of advertising content from the first advertising campaign data set may be based at least in part on a machine learning algorithm that utilizes the third party data set 2710 to select an optimal advertising placement. The real-time ticker facility 142 can then bill the advertiser 104 for a partial valuation of the placement of the advertising content from the first advertising campaign data set. In embodiments of the present invention, calculating the valuation and billing the advertiser 104 may be performed automatically upon receiving a request for layout content from the advertiser 140. In another embodiment of the present invention, the valuation calculation can be a comparison of the performance of a plurality of competing valuation algorithms 140. In an embodiment of the present invention, comparing the performance of a plurality of competing valuation algorithms 140 may include using valuation algorithms 140 based at least in part on historical data. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

Additionally, in embodiments of the present invention, the real-time ticker facility 142 can calculate a valuation of the third-party data set 3010 based at least in part on comparing advertisement impression data related to advertisement content placed from the first and second advertisement campaign data sets. In embodiments of the present invention, the placement of advertising content from the first advertising campaign data set may be based at least in part on a machine learning algorithm that utilizes the third party data set 3010 to select an optimal advertising placement. The real-time ticker facility 142 can then calibrate a bid amount recommendation for the publisher 112 to pay for the placement of the advertising content based at least in part on the valuation. In an embodiment of the invention, the calibration may be iteratively adjusted to account for the real-time event data 160 and its impact on valuation.

FIG. 31 illustrates a method 3100 for advertisement rating with the ability to measure the value of additional third party data, in accordance with an embodiment of the invention. The method begins at step 3102. At step 3104, the ad campaign data set may be split into a first ad campaign data set and a second ad campaign data set. At step 3108, an economic valuation model, which can be refined through machine learning, can be deployed to evaluate information relating to a plurality of available layouts and/or a plurality of advertisements to predict an economic valuation for a layout of advertising content from a first set of advertising campaign data. In embodiments of the invention, machine learning may be based at least in part on third party data sets. At step 3110, advertising content from the first and second sets of advertising campaign data may be placed within a plurality of available layouts and/or a plurality of advertisements. In embodiments of the invention, content from a first advertising campaign may be laid out based at least in part on a predicted economic valuation, and content from a second advertising campaign data set may be laid out based on a method that is not dependent on a third party data set. Additionally, at step 3112, impression data may be received from the tracker facility relating to advertising content laid out from the first and second advertisement campaign data sets. In embodiments, the impression data may include data regarding a user's interaction with the advertising content. Subsequently, at step 3114, a value of the third-party data set may be determined based at least in part on comparing impression data relating to advertising content laid out from the first and second ad campaign data sets. In embodiments of the invention, the third party data set may include data relating to advertising content users, contextual data relating to a plurality of available layouts and/or a plurality of advertisements, or financial data relating to historical advertising impressions. In embodiments of the present invention, the data relating to the advertising content user may include demographic data, transactional data, or ad conversion data. In embodiments of the invention, the context data may be derived from a context organizer service associated with the machine learning facility. In embodiments of the present invention, the economic valuation model may be based at least in part on real-time event data, in part on historical event data, in part on user data, in part on third party business data, in part on advertiser data, or in part on ad agency data. The method terminates at step 3118.

FIG. 32 illustrates a method 3200 for calculating a valuation of a third party data set and billing an advertiser for the partial valuation, in accordance with an embodiment of the present invention. The method starts at step 3202. At step 3204, a valuation of the third party data set may be calculated based at least in part on comparing advertisement impression data relating to advertisement content placed from the first and second advertisement campaign data sets. In embodiments of the present invention, the placement of advertising content from the first advertising campaign data set may be based at least in part on a machine learning algorithm that utilizes third party data sets to select an optimal advertising placement. Subsequently, at step 3208, the advertiser may be billed with a partial rating for placement of the advertising content from the first advertising campaign data set. In embodiments of the present invention, calculating a valuation and billing an advertiser may be performed automatically upon receiving a request for layout content from the advertiser. In another embodiment of the invention, the calculation of the valuation may be a comparison of the performance of a plurality of competing valuation algorithms. In an embodiment of the invention, comparing the performance of the plurality of competing valuation algorithms may comprise using a valuation algorithm based at least in part on historical data. The method terminates at step 3210. It will be understood that general analytical methods, statistical techniques and tools for evaluating competitive algorithms and models, such as valuation models, and analytical methods, statistical techniques and tools known to those of ordinary skill in the art are intended to be encompassed by the present invention and can be used to evaluate competitive algorithms and valuation models in accordance with the methods and systems of the present invention.

FIG. 33 illustrates a method 3300 for computing a valuation of a third party data set and calibrating a bid amount recommendation for a publisher to pay for a layout of advertising content based at least in part on the valuation, in accordance with an embodiment of the invention. The method begins at step 3302. At step 3304, a valuation of the third party data set can be calculated based at least in part on comparing advertisement impression data relating to advertisement content placed from the first and second advertisement campaign data sets. In embodiments of the present invention, the placement of advertising content from the first advertising campaign data set may be based at least in part on a machine learning algorithm that utilizes third party data sets to select an optimal advertising placement. Then, at 3308, the bid amount recommendation for the publisher payment may be corrected based at least in part on the valuation of the layout for the advertising content. In an embodiment of the invention, the calibration may be iteratively adjusted to account for real-time event data and its impact on valuation. The method terminates at step 3310.

In an embodiment, the analysis output of the analysis platform 114 may be illustrated using data visualization techniques, including but not limited to the surface maps shown in fig. 34-38. The surface map may illustrate, for example, efficiency spots within the performance of an advertising campaign, where the height measure of the surface is indexed to the conversion value per ad impression of the average performance. In an embodiment, surface areas having a value greater than one (1) may indicate better average conversion values, while areas below one (1) may indicate insufficient performance. Confidence tests may be applied to account for lower volume cross-sectional views of the surface map and its associated data. FIG. 34 depicts a data visualization embodiment that presents an advertising performance summary by time of day versus day of the week. FIG. 35 depicts a data visualization embodiment that presents an advertising performance summary in terms of population density. FIG. 36 depicts a data visualization embodiment that presents an advertising performance summary in terms of geographic areas of the United states. FIG. 37 depicts a data visualization embodiment that presents an advertising performance summary in terms of personal revenue. FIG. 38 depicts a data visualization embodiment that presents an advertising performance summary by gender.

FIG. 39 illustrates affinity indices for an advertising campaign/brand by category. The method and system of the present invention may identify characteristics of consumers that are more likely to be interested in advertiser brands than the general population. The methods and systems may also identify characteristics of consumers that are less likely to be interested in advertiser brands than the general population. The characteristics of the more interested consumers are presented on the left side of the chart in fig. 39. The chart also shows an index that represents how likely those consumers are to subscribe to the advertiser brand than the general population. The right side of the chart presents the characteristics of consumers of less interest and shows an index that represents how less likely those consumers are to subscribe to the brand than the general population. An index, such as the one presented in fig. 39, may take into account the size of the sample and be expressed using a formula that incorporates the sample size and the range of uncertainty.

FIG. 40 depicts a data visualization embodiment presenting a page visit summary by impression count. The method and system of the present invention can identify conversion rates presented by different consumer groups. As shown in FIG. 40, each group may be defined by the number of advertisements shown to the consumer members of the group. The analytics platform 114 may analyze consumers who see a given number of advertisements and calculate conversion rates. The analytics platform 114 may only consider impressions that are shown to the consumer before the consumer performs an action. As an example, a consumer who has seen 3 advertisements prior to performing an action desired by the advertiser is a member of group 3. The other 10 members of group 3 may have seen 3 advertisements, but may not have performed any action deemed beneficial to the advertiser. The conversion rate for population 3 was 3/10=0.3 or 300,000 per million consumers. The analysis takes into account the sample size and is expressed using a formula that incorporates the sample size and the uncertainty range. The analysis was also fitted to a curve that most likely represents the behavior observed across all populations.

The methods and systems described herein may be performed in part or in whole by a machine executing computer software, program code, and/or instructions on a processor. The processor may be part of a server, a client, a network infrastructure, a mobile computing platform, a stationary computing platform, or other computing platform. The processor may be any kind of computing or processing device capable of executing program instructions, code, binary instructions, etc. The processor may be or include a signal processor, a digital processor, an embedded processor, a microprocessor, or any variant such as a co-processor (math co-processor, graphics co-processor, communications co-processor, etc.), etc., which may directly or indirectly facilitate the execution of program code or program instructions stored thereon. Further, a processor may be implemented to execute a number of programs, threads, and code. Threads may be executed concurrently to enhance performance of the processor and facilitate concurrent operation of applications. By way of implementation, the methods, program code, program instructions, etc. described herein may be implemented in one or more threads. Threads may spawn other threads that may have assigned their associated priorities; the processor may execute these threads based on priority or any other order based on instructions provided in the program code. The processor may include a memory that stores methods, code, instructions, and programs as described herein and elsewhere. The processor may access a storage medium through the interface that may store the methods, code, and instructions as described herein and elsewhere. A storage medium associated with the processor (the storage medium storing a method, program, code, program instructions, or other types of instructions capable of being executed by a computing or processing device) may include, but is not limited to, one or more of a CD-ROM, DVD, memory, hard disk, flash drive, RAM, ROM, cache, etc.

The processor may include one or more cores that may enhance the speed and performance of the multiprocessor. In embodiments, the processor may be a dual core processor, a quad core processor, other chip scale multiprocessors, etc., which may combine two or more separate cores (referred to as dies).

The methods and systems described herein may be deployed, in part or in whole, by a machine executing computer software on a server, client, firewall, gateway, hub, router, or other such computer and/or networking hardware. The software program may be associated with a server that may include file servers, print servers, domain servers, internet servers, intranet servers, and other variants (e.g., secondary servers, mainframe servers, distributed servers, etc.). A server may include one or more of a memory, a processor, a computer readable medium, a storage medium, ports (physical and virtual), a communication device, and interfaces capable of accessing other servers, clients, machines and devices through a wired or wireless medium, and the like. The methods, programs, or code as described herein and elsewhere may be executed by a server. Furthermore, other devices needed to perform the methods as described in the present application may be considered part of the infrastructure associated with the server.

The server may provide an interface with other devices, including but not limited to clients, other servers, printers, database servers, print servers, file servers, communication servers, distributed servers, etc. Further, such coupling and/or connection may facilitate remote execution of programs across a network. The networking of some or all of these devices may facilitate parallel processing of programs or methods at one or more locations without departing from the scope of the present invention. Further, any device attached to the server through the interface may include at least one storage medium capable of storing methods, programs, code, and/or instructions. The central repository may provide program instructions to be executed on different devices. In this embodiment, the remote store may serve as a storage medium for program code, instructions, and programs.

The software programs may be associated with clients that may include file clients, print clients, domain clients, internet clients, intranet clients, and other variants (such as secondary clients, host clients, distributed clients, etc.). The client may include one or more of a memory, processor, computer readable medium, storage medium, ports (physical and virtual), communication devices, and interfaces capable of accessing other clients, servers, machines and devices through wired or wireless media, and the like. The methods, programs, or code as described herein and elsewhere may be executed by a client, and in addition, other devices that are required to perform the methods as described in this application may be considered part of the infrastructure associated with the client.

Clients may provide interfaces with other devices, including, but not limited to, servers, other clients, printers, database servers, print servers, file servers, communication servers, distributed servers, and the like. Further, such coupling and/or connection may facilitate remote execution of programs across a network. The networking of some or all of these devices may facilitate parallel processing of programs or methods at one or more locations without departing from the scope of the present invention. Further, any device attached to the client through an interface may include at least one storage medium capable of storing methods, programs, applications, code, and/or instructions. The central repository may provide program instructions to be executed on different devices. In this embodiment, the remote store may serve as a storage medium for program code, instructions, and programs.

The methods and systems described herein may be deployed in part or in whole via a network infrastructure. The network infrastructure may include elements as are known in the art, such as computing devices, servers, routers, hubs, firewalls, clients, personal computers, communication devices, routing devices, and other active and passive devices, modules, and/or components. The computing and/or non-computing device(s) associated with the network infrastructure may include storage media such as flash memory, buffers, stacks, RAM, ROM, etc., among other components. The processes, methods, program code, instructions described herein and elsewhere may be performed by one or more network infrastructure elements.

The methods, program codes, and instructions described herein and elsewhere may be implemented on a cellular network having a plurality of cells. The cellular network may be a Frequency Division Multiple Access (FDMA) network or a Code Division Multiple Access (CDMA) network. The cellular network may include mobile devices, cell sites, base stations, repeaters, antennas, towers, and the like. The cellular network may be GSM, GPRS, 3G, EVDO, mesh, or other network type.

The methods, program codes, and instructions described herein and elsewhere may be implemented on or by a mobile device. The mobile device may include a navigation device, a cellular telephone, a mobile personal digital assistant, a laptop computer, a palmtop computer, a notebook computer, a pager, an e-book reader, a music player, and so forth. These devices may include storage media such as flash memory, buffers, RAM, ROM, and one or more computing devices, among other components. Computing devices associated with the mobile devices may be enabled to execute program code, methods, and instructions stored thereon. Alternatively, the mobile device may be configured to execute instructions in cooperation with other devices. The mobile device can communicate with base stations that interface with the server and are configured to execute program code. The mobile device may communicate over a peer-to-peer network, a mesh network, or other communication network. The program code may be stored on a storage medium associated with the server and executed by a computing device embedded within the server. A base station may include a computing device and a storage medium. The storage medium may store program code and instructions for execution by a computing device associated with the base station.

The computer software, program code, and/or instructions may be stored and/or accessed on a machine-readable medium, which may include: computer components, devices and recording media that hold data for computation for a certain time interval; semiconductor memory referred to as Random Access Memory (RAM); mass storage typically used for more permanent storage, such as optical disks, forms of magnetic storage (e.g., hard disks, tapes, drums, cards, and other types); processor registers, cache memory, volatile memory, non-volatile memory; optical storage (e.g., CD, DVD); removable media such as flash memory (e.g., US sticks or keys), floppy disk, magnetic tape, paper tape, punch cards, stand-alone RAM disks, Zip drives, removable mass storage, offline, etc.; other computer memory such as dynamic memory, static memory, read/write memory, alterable memory, read-only, random-access, sequential-access, location-addressable, file-addressable, content-addressable, network-attached memory, storage area networks, barcodes, magnetic ink, and the like.

The methods and systems described herein may transform a physical and/or intangible item from one state to another. The methods and systems described herein may also transform data representing physical and/or intangible items from one state to another.

The cells described and depicted herein (including in the flow charts and block diagrams throughout the drawings) are meant to represent logical boundaries between cells. However, in accordance with software or hardware engineering practices, the depicted elements and their functionality may be implemented on a machine as an overall software structure, as stand-alone software modules, or as modules that employ external routines, code, services, etc., or any combination of these, via a computer-executable medium having a processor capable of executing program instructions stored thereon, and all such implementations may be within the scope of the present disclosure. Examples of such machines may include, but are not limited to, personal digital assistants, laptop computers, personal computers, mobile phones, other handheld computing devices, medical devices, wired or wireless communication devices, transducers, chips, calculators, satellites, tablet PCs, electronic books, accessories, electronic devices, devices with artificial intelligence, computing devices, networking devices, servers, routers, and so forth. Additionally, the elements depicted in the flowchart and block diagrams, or any other logic components, may be implemented on a machine capable of executing program instructions. Accordingly, while the foregoing drawings and description set forth functional aspects of the disclosed systems, no particular software arrangement for implementing these functional aspects should be inferred from the descriptions unless explicitly stated or otherwise clear from the context. Similarly, it will be appreciated that the various steps identified and described above may be varied and the order of steps may be adapted to specific applications of the techniques disclosed herein. All such variations and modifications are intended to fall within the scope of the present disclosure. As such, unless a particular application requires or clearly dictates otherwise, or is otherwise clear from the context, the depiction and/or description of a sequence for various steps should not be construed as requiring a particular order of execution for those steps.

The above described methods and/or processes and steps thereof may be implemented in hardware, software, or any combination of hardware and software as appropriate for a particular application. The hardware may include general purpose computers and/or special purpose computing devices or specific computing devices or particular aspects or components of specific computing devices. The processes may be implemented in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices, as well as internal and/or external memory. The processes may also, or alternatively, be embodied in an application specific integrated circuit, a programmable gate array, programmable array logic, or any other device or combination of devices that may be configured to process electronic signals. It will also be appreciated that one or more of the processes may be implemented as computer executable code capable of being executed on a machine-readable medium.

Computer executable code may be produced using a structured programming language such as C, an object oriented programming language such as C + +, or any other high-level or low-level programming language (including assembly, hardware description, and database programming languages and techniques), which may be stored, compiled, or interpreted to run on one of the above-described devices, as well as on a processor, a heterogeneous combination of processor architectures, or a combination of different hardware and software, or any other machine capable of executing program instructions.

Thus, in one aspect, each of the methods described above, and combinations thereof, may be embodied in computer-executable code that, when executed on one or more computing devices, performs the steps thereof. In another aspect, the method may be embodied in a system that performs its steps and may be distributed over devices in many ways, or all functions may be integrated into a dedicated, stand-alone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may comprise any of the hardware and/or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.

While the present invention has been disclosed in conjunction with the preferred embodiments shown and described in detail, various modifications and improvements thereto will become readily apparent to those skilled in the art. Thus, the spirit and scope of the present invention should not be limited by the foregoing examples but should be understood in the broadest sense allowable by law.

All documents cited herein are incorporated herein by reference.

Claims (25)

1. A computer program product embodied in a computer readable medium that, when executed on one or more computers, performs the steps of:

in response to receiving a request to place an advertisement, deploying a plurality of competing economic valuation models to predict an economic valuation for each advertisement placement of the plurality of advertisement placements; and is

Evaluating each valuation produced by each of the plurality of competing economic valuation models to select one as a current valuation for an advertising layout.

2. The computer program product of claim 1, wherein the economic valuation model is based at least in part on real-time event data.

3. The computer program product of claim 1, wherein the economic valuation model is based at least in part on historical event data.

4. The computer program product of claim 1, wherein the economic valuation model is based at least in part on user data.

5. The computer program product of claim 1, wherein the economic valuation model is based at least in part on third party business data.

6. The computer program product of claim 1, wherein the third party commercial data comprises financial data relating to historical advertising impressions.

7. The computer program product of claim 1, wherein the economic valuation model is based at least in part on contextual data.

8. The computer program product of claim 1, wherein the economic valuation model is based at least in part on advertiser data.

9. The computer program product of claim 1, wherein the economic valuation model is based at least in part on ad agency data.

10. The computer program product of claim 1, wherein the economic valuation model is based at least in part on historical advertising performance data.

11. The computer program product of claim 1, wherein the economic valuation model is based at least in part on machine learning.

12. A computer program product embodied in a computer readable medium that, when executed on one or more computers, performs the steps of:

in response to receiving a request to place an advertisement, deploying a plurality of competing economic valuation models to predict an economic valuation for each of a plurality of combinations of advertisement placement and advertisements;

evaluating each valuation produced by each of the plurality of competing economic valuation models to select a first estimate that is a combination of an advertisement layout and an advertisement;

reevaluating each valuation produced by each model of the plurality of competing economic valuation models to select one as a revised valuation for the combination of the advertisement layout and the advertisement, wherein the revised valuation is based at least in part on an analysis of an economic valuation model using real-time event data that is not available at the time the first valuation is selected; and is

Replacing the first valuation with the second revised valuation for use in deriving a recommended bid amount for the combination of the advertisement layout and the advertisement.

13. Further comprising the computer program product of claim 12, wherein the request is received from a publisher and the recommended bid amount is automatically sent to the publisher.

14. Further comprising the computer program product of claim 12, wherein the request is received from a publisher and a bid equal to the recommended bid amount is automatically offered on behalf of the publisher.

15. Further comprising the computer program product of claim 12, wherein the recommendation bid amount is associated with a recommendation time for an advertising layout.

16. Further comprising the computer program product of claim 12, wherein the recommended bid amount is derived further by analyzing a real-time bid log associated with a real-time bidding machine.

17. A computer program product embodied in a computer readable medium that, when executed on one or more computers, performs the steps of:

in response to receiving a request to place an advertisement, deploying a plurality of competitive economic valuation models to evaluate information relating to a plurality of advertisement placements and a plurality of available combinations of a plurality of advertisements to predict an economic valuation for each combination of the plurality of advertisement placements and the plurality of advertisements; and is

Evaluating each valuation produced by each of the plurality of competing economic valuation models to select one valuation as a future valuation of a combination of an advertisement layout and an advertisement.

18. A computer program product embodied in a computer readable medium that, when executed on one or more computers, performs the steps of:

in response to receiving a request to place an advertisement, deploying a plurality of competitive economic valuation models to evaluate information relating to a plurality of available advertisement placements and combinations of a plurality of advertisements to predict an economic valuation for each combination of the plurality of advertisement placements and the plurality of advertisements; and is

Evaluating each valuation produced by each of the plurality of competing economic valuation models in real time to select one valuation as a future valuation for the combination of advertisement layout and advertisement.

19. Further comprising a computer program product according to claim 17, wherein the future valuation is based at least in part on simulation data describing future events.

20. Further comprising the computer program product of claim 12, wherein the future event is a stock market fluctuation.

21. Further comprising the computer program product of claim 12, wherein the simulation data describing future events is derived by analyzing historical event data selected based at least in part on context data relating to advertisements to be placed in the advertisement layout.

22. A computer program product embodied in a computer readable medium that, when executed on one or more computers, performs the steps of:

in response to receiving a request to place an advertisement, deploying a plurality of competing real-time bidding algorithms involving a combination of a plurality of available advertisement placements and a plurality of advertisements to bid on the advertisement placements; and is

Each offer algorithm is evaluated to select a preferred algorithm.

23. The computer program product of claim 22, wherein the competitive real-time quotation algorithm uses data from a real-time quotation log.

24. A computer program product embodied in a computer readable medium that, when executed on one or more computers, performs the steps of:

in response to receiving a request to place an advertisement, deploying a plurality of competing real-time bidding algorithms involving a combination of a plurality of available advertisement placements and a plurality of advertisements to bid on the advertisement placements;

evaluating each offer recommendation generated by the competing real-time offer algorithm;

reevaluating each offer recommendation generated by the competing real-time offer algorithm to select one as a revised offer recommendation, wherein the revised offer recommendation is based at least in part on a real-time offer algorithm that uses real-time event data that is not available at the time the offer recommendation is selected; and is

Replacing the bid recommendation with the revised bid recommendation for use in deriving a recommended bid amount for a combination of an advertisement layout and an advertisement.

25. The computer program product of claim 24, wherein the replacing occurs in real-time with respect to receiving the request to place an advertisement.