CN116569164A - System and method for intelligent categorization of content in a content management system - Google Patents

Abstract

According to embodiments, the systems and methods described herein may be used, for example, with a content management system to provide recommendations to categorize/classify content into user-defined categories, which in turn provides a content manager with an opportunity to easily place new content into an accurate category based on previously evaluated/categorized content. The recommendation system or tool may facilitate placement of content into related categories by automatic categorization/classification of newly created/edited content. Recommendation tools can be implemented and applied across different domains by generating feature vectors from content, creating clusters in feature space based on previously categorized content, and by computing new content recommendation categories from the clustered feature space distances.

Description

System and method for intelligently categorizing content in a content management system

Copyright Notice

A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

Priority Claim and Cross-Reference to Related Applications:

This application claims the benefit of priority to U.S. Provisional Patent Application No. 63/084,174, filed on September 28, 2020, entitled “SYSTEM AND METHOD FOR SMART CATEGORIZATION OF CONTENT IN A CONTENT MANAGEMENT SYSTEM”; and U.S. Patent Application No. 17/486,524, filed on September 27, 2021, entitled “SYSTEM AND METHOD FOR SMART CATEGORIZATION OF CONTENT IN A CONTENT MANAGEMENT SYSTEM”; and is related to U.S. Patent Application No. 16/657,395, filed on October 18, 2019, entitled “TECHNIQUES FOR RANKING CONTENT ITEM RECOMMENDATIONS”; which is a U.S. Provisional Patent Application No. 63/084,174, filed on September 28, 2020; and U.S. Patent Application No. 17/486,524, filed on September 27, 2021, entitled “SYSTEM AND METHOD FOR SMART CATEGORIZATION OF CONTENT IN A CONTENT MANAGEMENT SYSTEM” This application is a continuation-in-part application of and claims the benefit of priority to U.S. Patent Application No. 16/581,138, entitled “SMART CONTENT RECOMMENDATIONS FOR AUTHORS”, filed on October 18, 2018, and Indian Provisional Patent Application No. 201841039495, entitled “SMART CONTENT RECOMMENDATIONS FOR AUTHORS”; each of the above applications and their contents are incorporated herein by reference.

Technical Field

The present application relates generally to online business environments and the management and delivery of content data, and in particular to intelligent categorization/classification of content in content management systems.

Background Art

Generators and authors of original content for online publication and/or transmission may use a variety of different software-based tools and techniques to generate, edit, and store newly generated content.

In a content management system, various contents (e.g., documents, structured contents such as blogs, articles, press releases; and media documents such as images and videos) often need to be evaluated/categorized based on their contents. This categorization/classification occurs on a hierarchical set of categories or nodes. For example, a contract document for a real estate lease may be evaluated/categorized under Legal Documents->Real Estate->Contracts. The same document (or content) may also be in more than one categorization/classification at the same time. For example, the same contract document may exist under Active Contracts->Signed.

Categories are grouped under an organizational concept called a taxonomy. Organizations often have many taxonomies that reflect their business organization for content. When new documents or content items are added, or when new taxonomies emerge, or when significant changes occur in the content organization, the task of correctly classifying or reclassifying content falls to the end user (or content author). This can be a costly and error-prone endeavor as the volume of content and the number of taxonomies grow.

Summary of the invention

According to an embodiment, the systems and methods described herein may be used, for example, with a content management system to provide recommendations for categorizing/classifying content into user-defined categories, which in turn provides content managers with the opportunity to easily place new content into accurate categories based on previously evaluated/classified content.

Categorizing large amounts of content in an online manner is a complex task involving challenges such as single pass constraints on data and rapid response requirements. According to an embodiment, content users categorize similar content through logical clustering such as a hierarchical classification system tree and place similar content in the same node/category of the classification system tree. Over time, as the number of content entities and nodes in the classification system tree grows, similar content entities will find themselves residing in nodes side by side with each other. Given this state of content organization, computer algorithms can use content that resides in already evaluated/classified classification systems to determine where newly created/edited content may belong.

According to an embodiment, a recommendation system or tool may use artificial intelligence (AI) techniques to continuously learn from past data and help place content into relevant categories through automatic classification/categorization of newly created/edited content. Recommendation tools may be implemented and applied across different domains by generating feature vectors from content, creating clusters in feature space based on previously classified content, and recommending categories for new content through feature space distance calculations from clusters.

Aspects of the present disclosure relate to an artificial intelligence (AI) driven tool that is configured to be used as an intelligent digital assistant to recommend images, text content, and other related media content from a content repository. Certain embodiments may include a front-end software tool with a graphical user interface (GUI) to supplement the content creation interface for creating original media content (e.g., blog entries, online articles, etc.). In some cases, additional GUI screens and features may be incorporated into existing content creation software tools, for example, as software plug-ins. The intelligent digital content recommendation tool may communicate with multiple back-end services and content repositories, for example, to analyze text and/or visual input, extract keywords or topics from the input, classify and tag the input content, and store the classified/tagged content in one or more content repositories.

Additional techniques performed in various embodiments of the intelligent digital content recommendation tool (e.g., directly by the software tool and/or indirectly by calling a backend service) may include converting input text and/or images into vectors in a multidimensional vector space and comparing the input content with multiple repository contents to find multiple relevant content options within the content repository. This comparison may include a thorough and exhaustive deep search and/or a more efficient tag-based filtered search. Finally, relevant content items (e.g., images, audio and/or video clips, links to related articles, etc.) may be retrieved and presented to the content author for review and embedded into the original creative content.

Although the description herein mainly illustrates application to text content, according to various embodiments, the scheme can be extended to, for example, multimedia or other types of content such as images/videos through metadata extraction.

BRIEF DESCRIPTION OF THE DRAWINGS

A further understanding of the nature and advantages of embodiments according to the present disclosure may be realized by referring to the remainder of the specification in conjunction with the following drawings.

In the accompanying drawings, similar components and/or features may have the same reference number. In addition, various components of the same type may be distinguished by following the reference number with a dash and a second reference that distinguishes between the similar components. If only the first reference number is used in the specification, the description applies to any one of the similar components having the same first reference number, regardless of the second reference number.

1 is a diagram of an example computer system architecture including a data integration cloud platform in which certain embodiments of the present disclosure may be implemented.

2 is an example screen shot of a custom dashboard in a user interface for configuring, monitoring, and controlling a service instance, according to certain embodiments of the present disclosure.

FIG. 3 is an architecture diagram of a data integration cloud platform according to certain embodiments of the present disclosure.

4 is a diagram of an example computing environment configured to perform content classification and recommendation, according to certain embodiments of the present disclosure.

5 is another diagram of an example computing environment configured to perform content classification and recommendation, according to certain embodiments of the present disclosure.

6 is a flow diagram illustrating a process for generating feature vectors based on content resources within a content repository, according to certain embodiments of the present disclosure.

7 is an example image in which multiple image features are identified, in accordance with certain embodiments of the present disclosure.

FIG. 8 is an example of a text document illustrating a keyword extraction process according to certain embodiments of the present disclosure.

9-11 are diagrams illustrating a process of generating and storing image tags according to certain embodiments of the present disclosure.

12 is a flow diagram illustrating another process for comparing feature vectors and identifying related content within a content repository, according to certain embodiments of the present disclosure.

FIG. 13 is a diagram illustrating a technique for converting a submitted image into a feature vector in accordance with certain embodiments of the present disclosure.

14 is an illustrative vector space populated with feature vectors, in accordance with certain embodiments of the present disclosure.

FIG. 15 is a diagram illustrating deep feature space vector comparison according to certain embodiments of the present disclosure.

16-17 are diagrams illustrating filtered feature space vector comparisons according to certain embodiments of the present disclosure.

18 is a diagram representing a process of receiving and processing text input to identify related images or articles, according to certain embodiments of the present disclosure.

FIG. 19 is an example diagram illustrating comparison of extracted keywords with image tags according to certain embodiments of the present disclosure.

FIG. 20 is an example of keyword analysis in a 3D word vector space, according to certain embodiments of the present disclosure.

FIG. 21 is a diagram illustrating keyword-to-tag vector space analysis according to certain embodiments of the present disclosure.

FIG. 22 is a diagram illustrating an example of homonymous image labels according to certain embodiments of the present disclosure.

23-24 are diagrams illustrating exemplary disambiguation processes according to certain embodiments of the present disclosure.

25-28 are diagrams illustrating a process of comparing feature vectors and identifying related content within a content repository, according to certain embodiments of the present disclosure.

29-30 are example diagrams of text documents illustrating topic extraction processing according to certain embodiments of the present disclosure.

31-35 are diagrams illustrating a process of identifying relevant articles based on input text data according to certain embodiments of the present disclosure.

36 is a diagram of an example semantic text analyzer system in accordance with certain embodiments of the present disclosure.

37-38 are example user interface screens illustrating image recommendations provided to a user during content creation, according to certain embodiments of the present disclosure.

FIG39 depicts a simplified diagram of a distributed system for implementing certain embodiments according to the present disclosure.

40 is a simplified block diagram of one or more components of a system environment by which services provided by one or more components of the system may be provided as cloud services, in accordance with certain embodiments of the present disclosure.

FIG. 41 illustrates an exemplary computer system in which various embodiments may be implemented.

42 is a diagram of an example computing environment configured to evaluate and rank content items from a content repository in response to input content received from a user or client system, in accordance with certain embodiments of the present disclosure.

43 is a flow chart illustrating a process for identifying and ranking content items related to user content, in accordance with certain embodiments of the present disclosure.

FIG. 44 is an example screen shot of a content authoring user interface in accordance with certain embodiments of the present disclosure.

FIG. 45 illustrates an example table of matching sets of content items identified by a content recommendation system, according to certain embodiments of the present disclosure.

46 illustrates another example table of matching sets of content items including ranking scores, in accordance with certain embodiments of the present disclosure.

FIG. 47 is another example screen shot of a content authoring user interface in accordance with certain embodiments of the present disclosure.

FIG. 48 illustrates an example of a content management system environment according to an embodiment.

FIG. 49 illustrates an example use of a content management system for management and delivery of content data according to an embodiment.

FIG. 50 illustrates a flow chart of intelligent content classification according to an embodiment.

FIG. 51 illustrates a classification system creation flow chart according to an embodiment.

FIG. 52 illustrates a classification system modification flow chart according to an embodiment.

FIG53 illustrates a sample taxonomy tree with a category diagram, under an embodiment.

FIG. 54 further illustrates a sample taxonomy tree with a category diagram, according to an embodiment.

FIG. 55 further illustrates a sample taxonomy tree with a category diagram, according to an embodiment.

Figure 56 illustrates a configuration auto-classification threshold map according to an embodiment.

FIG. 57 illustrates a configuration for triggering reclassification (in bulk) of content in a repository graph, according to an embodiment.

FIG. 58 illustrates the problem of traditional clustering where clusters can be of arbitrary shapes, according to an embodiment.

FIG59 illustrates micro clustering according to an embodiment.

FIG. 60 illustrates a sample topic distribution for apparel customers, under an embodiment.

FIG. 61 illustrates visualization of cluster radii, according to an embodiment.

FIG. 62 illustrates a cluster representation according to an embodiment.

FIG63 illustrates the macroscopic stages of classification according to an embodiment.

Figure 64 illustrates the categorization once a higher level category is selected through the macro step.

Figure 65 is a diagram of how cluster weights decay over time, under an embodiment, and illustrates an example of a damped window model.

Figure 66 illustrates a graphical view of how shadow clustering may appear according to an embodiment.

Figure 67 illustrates a graphical view of how shadow clusters appear according to an embodiment.

FIG. 68 illustrates suggesting that a user create a new category from uncategorized content, according to an embodiment.

FIG. 69 is a flowchart of a method for intelligently categorizing content in a content management system according to an embodiment.

DETAILED DESCRIPTION

According to an embodiment, the present invention is illustrated in the figures of the accompanying drawings by way of example and not by way of limitation, and similar reference numerals indicate similar elements in the accompanying drawings. It should be noted that references to "an" or "one" or "some" embodiments in this disclosure do not necessarily refer to the same embodiments, and such references represent at least one. Although specific embodiments are discussed, it should be understood that specific embodiments are provided for illustrative purposes only. Those skilled in the relevant art will recognize that other components and configurations may be used without departing from the scope and spirit of the present invention.

In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the various embodiments and examples. However, it will be apparent that the various embodiments may be practiced without these specific details. For example, circuits, systems, algorithms, structures, techniques, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. The drawings and descriptions are not intended to be limiting.

Some examples (such as those disclosed with respect to the drawings in this disclosure) may be described as processes depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, sequence diagrams, or block diagrams. Although a sequence diagram or flow chart may describe an operation as a sequential process, many operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process terminates after its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or main function.

The processing described herein (such as the processing described with reference to the drawings in the present disclosure) can be implemented with software (e.g., code, program), hardware, or a combination thereof executed by one or more processing units (e.g., processor cores). The software can be stored in a memory (e.g., on a memory device, on a non-transitory computer-readable storage medium). In some examples, the processing depicted in the sequence diagrams and flow charts of this article can be implemented by any system disclosed herein. The specific series of processing steps in this disclosure are not intended to be restrictive. According to alternative examples, other step sequences can also be performed. For example, the alternative examples of the present disclosure can perform the steps outlined above in different orders. Moreover, the various steps shown in the figure can include multiple sub-steps, which can be performed in various orders suitable for the various steps. In addition, additional steps can be added or removed depending on the specific application. Those of ordinary skill in the art will recognize many variations, modifications, and substitutions.

In some examples, each process in the figures of the present disclosure may be performed by one or more processing units. A processing unit may include one or more processors, including a single-core or multi-core processor, one or more cores of a processor, or a combination thereof. In some examples, a processing unit may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some examples, custom circuits (such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs)) may be used to implement some or all of the processing units.

Certain embodiments described herein may be implemented as part of a data integration platform cloud (DIPC). In general, data integration involves combining data residing in different data sources and providing users with unified access and a unified view of the data. In many cases, such as merging a business entity with an existing legacy database, this process often occurs and becomes important. With the increasing amount of data and the ability to analyze data to provide useful results ("big data"), data integration is beginning to appear more and more frequently in enterprise software systems. For example, consider a web application in which users can query various types of travel information (e.g., weather, hotels, airlines, demographics, crime statistics, etc.). Enterprise applications do not necessarily require that all of these various data types be stored in a single database with a single schema, but can use unified views and virtual schemas in DIPC to combine many heterogeneous data sources so that they can be presented to users in a unified view.

DIPC is a cloud-based platform for data transformation, integration, replication and governance. It provides batch and real-time data movement between cloud and on-premise data sources while maintaining data consistency with default tolerances and elasticity. When these various data sources are combined into one or more data warehouses, DIPC can be used to connect to various data sources and prepare, transform, replicate, govern and/or monitor data from these various sources. DIPC can work with any type of data source and support any type of data in any format. DIPC can use Platform as a Service (PaaS) or Infrastructure as a Service (IaaS) architecture to provide cloud-based data integration for enterprises.

DIPC can provide many different utilities, including transferring entire data sources to new cloud-based deployments and allowing easy access to cloud databases from cloud platforms. Data can be streamed to the latest new data sources in real time and keep any number of distributed data sources synchronized. The load can be distributed between synchronized data sources so that they remain highly available to end users. The underlying data management system can be used to reduce the amount of data moved over the network to be deployed in database clouds, big data clouds, third-party clouds, etc. The drag-and-drop user interface can be used to execute reusable extraction, loading, and transformation (ELT) functions and templates. A real-time test environment can be created to perform reporting and data analysis on replicated data sources in the cloud so that data can remain highly available to end users. Using duplicate synchronized data sources, data migration can be performed with zero downtime. Synchronized data sources can also be used for seamless disaster recovery to maintain availability.

FIG. 1 illustrates a computer system architecture for integrating data from various existing platforms using DIPC according to some embodiments. The first data source 102 may include a cloud-based storage repository. The second data source 104 may include an on-premises data center. In order to provide unified access and views to the first data source 102 and the second data source 104, the DIPC 108 may use an existing high-performance ELT function library 106 to copy data 104 from the first data source 102 and the second data source. When the data is stored in the new cloud platform, the DIPC 108 may also extract, enrich and transform the data. Then, the DIPC 108 may provide access to any big data utility that resides in or can be accessed by the cloud platform. In some embodiments, the original data sources 102 and 104 may continue to provide access to customers, while the replicated data sources in the cloud platform may be used for testing, monitoring, governance and big data analysis. In some embodiments, data governance may be provided to profile, clean and govern data sources within an existing collection of custom dashboards in a user interface.

FIG. 2 illustrates one of the custom dashboards in the user interface that can be used to configure, monitor, and control service instances in the DIPC 108. A summary dashboard 202 may provide a control 204 that allows a user to create a service instance. Next, a series of progressive web forms may be presented to guide the user through the types of information used to create a service instance. In the first step, the user will be asked to provide a service name and description as well as an email address and a service version type. The user may also be asked to provide a cluster size that specifies the number of virtual machines used in the service. The service version type determines which applications are installed on the virtual machines. In the second step and the corresponding web form, the user may provide a running cloud database deployment to store the mode of the DIPC server. The same database may be used later to store data entities and perform integration tasks. In addition, a storage cloud may be specified and/or provisioned as a backup utility. The user may also provide credentials that may be used to access existing data sources used in data integration. In the third step, the provisioning information may be confirmed and the service instance may be created. The new service instance may then be displayed in the summary area 206 of the summary dashboard 202. From there, the user may access any information of any running data integration service instance.

FIG3 illustrates an architectural diagram of a DIPC according to some embodiments. The request may be received through a browser client 302, which may be implemented using a JavaScript Extension Toolkit (JET) component set. Alternatively or additionally, the system may receive the request through a DIPC agent 304 operating at a customer's on-premises data center 306. The DIPC agent 304 may include a data integrator agent 308 and a server for replicating services such as Oracle. Each of these agents 308, 310 can retrieve information from the on-premises data center 306 during normal operation and use connectivity services 312 to transmit data back to the DIPC.

Incoming requests can be passed through a login service 314, which can include load balancing or other utilities for routing requests through the DIPC. The login service 314 can use identity management services (such as identity cloud services 316) to provide security and identity management for the cloud platform as part of the integrated enterprise security structure. The identity cloud service 316 can manage the user identities of both the cloud deployment and on-premises applications described in this embodiment. In addition to the identity cloud service 316, the DIPC can also use the PaaS service manager (PSM) tool 318 to provide an interface to manage the life cycle of platform services in the cloud deployment. For example, the PSM tool 318 can be used to create and manage instances of data integration services in the cloud platform.

The DIPC may be implemented on a Web logic server 320 to build and deploy enterprise applications in a cloud environment. The DIPC may include a local repository 322 that stores data policies, design information, metadata, and audit data for information delivery through the DIPC. It may also include a monitoring service 324 to populate the local repository 322. A directory service 326 may include a collection of machine-readable open APIs to provide access to many SaaS and PaaS applications in a cloud deployment. The directory service 326 may also be available to a search application 338 that uses a distributed indexing service, such as Apache Connectivity services 328 and mediator services 330 can manage connections and provide transformation, validation, and routing logic for information passing through the DIPC. Information within the DIPC can be communicated using an event-driven architecture (EDA) and a corresponding message bus 332.

The DIPC may also include an orchestration service 334. The orchestration service 334 may enable automated tasks by calling REST endpoints, scripts, third-party automation frameworks, etc. These tasks may then be performed by the orchestration service 334 to provide DIPC functionality. The orchestration service 334 may use runtime services to import, transform, and store data. For example, the ELT runtime service 334 may execute a library of the above-mentioned ELT functions, while the replication runtime service 342 may replicate data from various data sources to the DIPC repository 316 where the cloud is deployed. In addition, the DIPC may include a code generation service 336 that provides automatic code generation for ELT functions and replication functions.

Smart Content – Intelligent content recommendations

As mentioned above, when a user creates/authors original media content (e.g., articles, newsletters, emails, blog posts, etc.), it is often useful to enhance the content of the creation with related additional content such as related images, audio/video clips, links to related articles or other content. However, in several aspects, searching for such additional content and embedding the additional content into the user's original creation content may be difficult. The initial difficulty may involve finding safe/reliable additional content from a trusted source and ensuring that the authorized user/author incorporates the content into their work. In addition, from any such safe and authorized content repository, it may be a manually intensive and inefficient process for the user/author to locate and incorporate/embed any related content in its original creation content.

Therefore, certain aspects described herein relate to intelligent digital content recommendation tools. In some embodiments, the intelligent digital content recommendation tool can be an artificial intelligence (AI) driven tool that is configured to process and analyze input content (e.g., text, images) from content authors in real time and recommend related images, additional text content, and/or other related media content (e.g., audio or video clips, graphics, social media posts, etc.) from one or more trusted content repositories. The intelligent digital content recommendation tool can communicate with multiple backend services and content repositories, for example, to analyze text and/or visual input, extract keywords or topics from the input, classify and tag the input content, and store the classified/tagged content in one or more content repositories.

Additional aspects described herein (each of which may be performed directly via an intelligent digital content recommendation tool executed on a client operated by a content author, and/or indirectly by calling various backend services) may include (a) receiving raw content in the form of text and/or images as input, (b) extracting keywords and/or topics from the raw content, (c) determining and storing associated keyword and/or topic tags for the raw content, (d) converting the raw content (e.g., input text and/or images) into a vector within a multi-dimensional vector space, (e) comparing such a vector to a plurality of other content vectors, each of which represents additional content in a content repository, in order to find and identify various potentially relevant additional content related to the original content input authored by the user/author, and finally (f) retrieving the identified additional content and presenting it to the author via the intelligent digital content recommendation tool. In some embodiments, each additional content item (e.g., an image, a link to a related article or web page, an audio or video file, a graphic, a social media post, etc.) can be displayed and/or thumbnailed by an intelligent digital content recommendation tool in a GUI-based tool that allows the user to drag and drop or otherwise place the additional content within the user's original creative content, including content positioning, formatting, resizing, etc.

Referring now to FIG. 4 , a block diagram illustrating the various components of a system 400 for intelligent content classification and recommendation is shown, including a client device 410, a content input processing and analysis service 420, a content recommendation engine 425, a content management system 435, and a content retrieval and embedding service 445. In addition, the system 400 includes one or more content repositories 440 storing content files/resources, and one or more vector spaces 430. As described in more detail below, a vector space may refer to a multidimensional data structure configured to store one or more feature vectors. In some embodiments, the recommendation engine 425, associated software components and services 420 and 445, the content management system 435, and the content repository 440 (which may store one or more data repositories or other data structures) may be implemented and stored as a back-end server system away from the front-end client device 410. Thus, the interaction between the client device 410 and the content recommendation engine 425 may be an Internet-based web browsing session or a client-server application session during which a user accesses content that may be originally authored, input via the client device 410, and receives content recommendations from the content recommendation engine 425 in the form of additional content that is retrieved from the content repository 440 and linked or embedded into the content authoring user interface at the client device 410. Additionally or alternatively, the content recommendation engine 425 and/or the content repository 440 and related services may be implemented as dedicated software components that execute on the client device 410.

The various computing infrastructure elements shown in this example (e.g., content recommendation engine 425, software components/services 420, 435, and 445, and content repository 440) may correspond to a high-level computer architecture created and maintained by an enterprise or organization that provides Internet-based services and/or content to various client devices 410. Content described herein (which may also be referred to as content resources and/or content files, content links, etc.) may be stored in one or more content repositories, retrieved and categorized by content recommendation engine 425, and provided to content authors at client devices 410. In various embodiments, content authors may input various different media types or file types of content at client devices 410 as original content, and similarly, content of various different media types or file types may be stored in content repository 440 and recommended for/embedded in a front-end user interface of client devices 410. These various different media types created by or recommended to content authors may include text (e.g., an authored letter, article, or blog), images (selected by or for the author), audio or video content assets, graphics, social media content (e.g., posts, messages, or tweets).

In some embodiments, the system 400 shown in FIG. 4 can be implemented as a multi-layer cloud-based system, wherein the upper user device 410 can request and receive access to network-based resources and services via the content processing/analysis component 420, and wherein the application server can be deployed and executed based on a basic resource set (e.g., cloud-based, SaaS, IaaS, PaaS, etc.) including hardware and/or software resources. In addition, although a cloud-based system can be used in some embodiments, in other examples, the system 400 can use an on-premises data center, a server farm, a distributed computing system, and various other non-cloud computing architectures. Some or all of the functionality described herein for the generation and storage of the content processing/analysis component 420, the content recommendation engine 425, the content management system 435, the content retrieval and embedding component 445, and the vector space 430 can be performed by a representative state transfer (REST) service and/or a web service, including a simple object access protocol (SOAP) web service or API, and/or web content disclosed via a hypertext transfer protocol (HTTP) or HTTP secure protocol. Thus, although not shown in FIG4 so as not to obscure the components shown with additional detail, the computing environment 400 may include additional client devices 410, one or more computer networks, one or more firewalls 435, proxy servers, and/or other intermediate network devices to facilitate interaction between the client devices 410, the content recommendation engine 425, and the backend content repository 440. Another embodiment of a similar system 500 is shown in more detail in FIG5.

Referring briefly to Fig. 5, another example diagram of a computing environment 500 is shown, which shows a data flow/data transformation diagram for performing content classification and recommendation. Therefore, the computing environment 500 shown in this example can correspond to a possible implementation of the computing environment 400 described above in Fig. 4. In Fig. 5, several diagram boxes shown represent specific data states or data transformations, rather than the structural hardware and/or software components described above in Fig. 4. Therefore, box 505 can represent input content data received via a user interface. Box 510 represents a set of keywords determined by system 400 based on input content 505. As discussed above, keywords 510 can be determined by input processing/analysis component 420 using one or more keyword extraction and/or topic modeling processes, and text feature vectors 515 can be generated based on the determined keywords 510.

Continuing with the example shown in FIG. 5 , a plurality of additional feature vectors 520 may be retrieved from the content repository 440. In this example, the additional feature vectors 520 may be selected from the content repository 440 by executing one or more neural network trained image models and providing the determined keywords 510 to the trained models. Based on the output of the trained models, the resulting feature vectors 520 may be further narrowed to exclude feature vectors having a feature vector probability less than z%, resulting in a subset of retrieved feature vectors 525. A feature space comparison 530 may then be performed between the test feature vector 515 and the subset of retrieved feature vectors 525. In some embodiments, and as shown in this example, a closest Euclidean distance calculation may be used to identify the retrieved feature vector 525 that is closest to the test feature 515. Based on the feature space comparison 530, one or more recommendations 530 may be determined, each recommendation 530 being determined based on an associated feature vector 525 having a threshold proximity to the test feature vector 515 and each recommendation 530 corresponding to an image in the content repository 440.

The components shown in system 400 for providing AI-based and feature vector analysis-based content recommendations and services to client devices 410 can be implemented in hardware, software, or a combination of hardware and software. For example, underlying system hardware or software components such as data storage devices, network resources, computing resources (e.g., servers), and various software components can be used to generate, deploy, and execute web services in data center 440. In some embodiments, web services can be different software components executed on the same (one or more) underlying computer servers, networks, data repositories, and/or in the same virtual machine. Some web-based content, computing infrastructure instances, and/or web services provided in content recommendation engine 425 can use dedicated hardware and/or software resources, while others can share underlying resources (e.g., shared clouds). In either case, some higher-level services (e.g., user applications) and users on client devices do not need to know the underlying resources used to support the services.

In such an embodiment, various application servers, database servers, and/or cloud storage systems, as well as other infrastructure components such as web caches, network components, etc. (not shown in this example), may include various hardware and/or software components (e.g., application programming interfaces (APIs), cloud resource managers, etc.) to provide and monitor the classification and vectorization of content resources, and to manage the underlying storage/server/network resources. The underlying resources of the content repository 440 may be stored within the content repository and/or cloud storage system, which may include, for example, a collection of non-volatile computer memory devices implemented as databases, file-based storage, etc., a collection of network hardware and software components (e.g., routers, firewalls, gateways, load balancers, etc.), a collection of host servers, and various software resources corresponding to various platforms, servers, middleware, and different versions of application software (such as storage of software images, installations, builds, templates, configuration files, etc.). The data center that houses the application servers of the recommendation engine 425, the vector space 430, and related services/components may also include additional resources, such as hypervisors, host operating systems, resource managers, and other cloud-based applications, as well as hardware and software infrastructure to support various Internet-based services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). In addition, the underlying hardware of the data center may be configured to support a variety of internal shared services, which may include, for example, security and identity services, integration services, repository services, enterprise management services, virus scanning services, backup and recovery services, notification services, file transfer services, etc.

As described above, according to the various embodiments described herein, many different types of computer architectures (cloud-based, web-based, hosted, multi-tier computing environments, distributed computing environments, etc.) can be used to provide web-based content recommendations from the content recommendation engine 425 to the client device 410 (which can be implemented via one or more content recommendation application servers). However, in some embodiments, a cloud computing platform can be used to provide certain advantageous features for the generation and management of web-based content. For example, in contrast to non-cloud-based embodiments with fixed architectures and limited hardware resources, a cloud computing platform can provide elasticity and scalability to quickly provision, configure, and deploy many different types of computing infrastructure instances. Moreover, public clouds, private clouds, and public-private hybrid cloud platforms can be used in various embodiments to take full advantage of the features and advantages of each different architecture.

In addition, as shown in this example, the system 400 also includes a content management system 435. In some embodiments, the content management system 435 may include a distributed storage processing system, one or more machine learning-based classification algorithms (and/or non-machine learning-based algorithms), and/or a storage architecture. As discussed in more detail below, in some embodiments, the content management system 435 may access content resources (e.g., web-based articles, images, audio files, video files, graphics, social media content, etc.) via one or more content repositories 440 (e.g., network-based document repositories, web-based content providers, etc.). For example, within the system 400, a dedicated JavaScript or other software component may be installed and run on one or more application servers, database servers, and/or cloud systems that store content objects or network-based content. These software components may be configured to retrieve content resources (e.g., articles, images, web pages, documents, etc.) and transmit them to the content management system 435 for analysis and classification. For example, each time a user within the operating organization of the system 400 imports or creates new content (such as an image or article), the software component can retrieve the content back to the content management system 435 for various processing and analysis described below (e.g., image processing, keyword extraction, topic analysis, etc.). In addition, although the content management system 435 is described as being implemented separately from the content recommendation engine 425 and the content repository 440 in this example, in other examples, the content management system 435 can be implemented locally with either of the content recommendation engine 425 and/or the storage devices storing the content repository 440, and therefore does not need to receive separate content transmissions from those devices, but can analyze and classify those content resources stored or provided by their respective systems.

One or more vector spaces 430 may also be generated and used to store feature vectors corresponding to different content items within the content repository 440, and feature vectors for the original creative content (e.g., received from the client device 410) may be compared to feature vectors for additional content items in the content repository 440. In some embodiments, multiple multidimensional feature spaces 430 may be implemented within the system 400, such as a first feature space 430a for the subject of the text input/article and a second feature space 430b for the image. In other embodiments, additional separate multidimensional feature spaces 430 may be generated for different types of content media (e.g., feature spaces for audio data/files, feature spaces for video data/files, feature spaces for graphics, feature spaces for social media content, etc.). As discussed below, a comparison algorithm may be used to determine the distance between vectors within the feature space. Thus, in the feature space of image feature vectors, an algorithm may be used to identify the image that is closest to the received input image, in the feature space of text feature vectors, an algorithm may be used to identify the text (e.g., article) that is closest to the received input text block, and so on. Additionally or alternatively, the comparison algorithm may use keywords/tags of the vector space to determine similarities between various media types.

In various embodiments, the system 400 can be implemented using one or more computing systems and/or networks. These computing systems may include one or more computers and/or servers, which may be general-purpose computers, dedicated server computers (e.g., desktop servers, UNIX servers, mid-range servers, mainframes, rack servers, etc.), server farms, server clusters, distributed servers, or any other suitable arrangement and/or combination of computing hardware. The content recommendation engine 425 may run an operating system and/or various additional server applications and/or middle-tier applications, including a hypertext transfer protocol (HTTP) server, a file transfer service (FTP) server, a common gateway interface (CGI) server, a Java server, a database server, and other computing systems. The content repository 440 may include a database server, such as those commercially available from Oracle, Microsoft, etc. Each component in the system 400 may be implemented using hardware, firmware, software, or a combination of hardware, firmware, and software.

In various embodiments, each component within the system 400 may include at least one memory, one or more processing units (e.g., (one or more) processors)) and/or storage devices. The (one or more) processing units may be appropriately implemented in hardware (e.g., integrated circuits), computer executable instructions, firmware, or a combination of hardware and instructions. In some examples, the various components of the system 400 may include several subsystems and/or modules. The subsystems and/or modules in the content recommendation engine 425 may be implemented in hardware, software executed on hardware (e.g., program code or instructions executable by a processor), or a combination thereof. In some examples, the software may be stored in a memory (e.g., a non-transitory computer-readable medium), on a memory device, or in some other physical memory, and may be executed by one or more processing units (e.g., one or more processors, one or more processor cores, one or more graphics processing units (GPUs), etc.). The computer executable instructions or firmware implementation of the (one or more) processing units may include computer executable instructions or machine executable instructions written in any suitable programming language, which may perform the various operations, functions, methods, and/or processes described herein. The memory can store program instructions that are loadable and executable on (one or more) processing units, as well as data generated during the execution of these programs. The memory can be volatile (such as random access memory (RAM)) and/or non-volatile (such as read-only memory (ROM), flash memory, etc.). The memory can be implemented using any type of persistent storage device (such as a computer-readable storage medium). In some examples, the computer-readable storage medium can be configured to protect the computer from electronic communications containing malicious code.

6 , there is shown a flow chart illustrating a process for generating feature vectors based on content resources in content repository 440 and storing the feature vectors in feature space 430. As described below, steps in this process may be performed by one or more components in computing environment 400, such as content management system 435, and various subsystems and subcomponents implemented therein.

In step 602, content resources may be retrieved from a content repository 440 or other data repository. As discussed above, individual content resources (which may also be referred to as content or content items) may correspond to data objects of any of a variety of content types: text items (e.g., text files, articles, emails, blog posts, etc.), images, audio files, video files, 2D or 3D graphic objects, social media data items, etc. In some embodiments, content items may be retrieved from a specific content repository 440, such as a proprietary data repository owned and operated by a specific trusted organization. Although the content repository 440 may be an external data source (such as an Internet web server or other remote data store), the system 400 that retrieves and vectorizes content from a local and/or privately controlled content repository 440 may achieve several technical advantages in the operation of the system 400, including ensuring that content from the repository 440 will be preserved and accessible when needed, and ensuring that users/authors are authorized to use and reproduce content from the repository 440. In some cases, the retrieval in step 602 (and subsequent steps 604 - 608 ) may be triggered in response to a new content item being stored in content repository 440 and/or a modification to an item in content repository 440 .

In step 604, the content items retrieved in step 602 may be parsed/analyzed, etc., to extract a set of project features or characteristics. The type of parsing, processing, feature extraction, and/or analysis performed in step 604 may depend on the type of content item. For image content items, artificial intelligence-based image classification tools may be used to identify specific image features and/or generate image tags. As shown in the example image of FIG. 7 , image analysis may identify multiple image features (e.g., smile, waitress, counter, vending machine, coffee cup, cake, hand, food, person, cafe, etc.), and each of these identified features may be used to tag the image. For text-based content items (such as blog posts, letters, emails, articles, etc.), the analysis performed in step 604 may include keyword extraction and processing tools (e.g., stemming, synonym retrieval, etc.), as shown in FIG. 8 . One or both types of analysis (e.g., extracting features from images as shown in FIG. 9 and extracting keywords/topics from text content as shown in FIG. 8 ) can be applied to the text content of FIG. 8 using analytics, machine learning algorithms, and/or artificial intelligence (AI) via a REST-based service or other web service (such as an AI-based cognitive image analysis service or a similar AI/REST cognitive text service). Similar techniques can be used in step 604 for other types of content items, such as video files, audio files, graphics, or social media posts, where, depending on the media type of the content item, a dedicated web service of the system 400 is used to extract and analyze specific features (e.g., words, objects in an image/video, facial expressions, etc.).

In step 606, after extracting/determining specific content features (e.g., visual objects, keywords, themes, etc.) from the content item, a feature vector can be generated based on the extracted/determined features. Using various transformation techniques, each set of features associated with the content item can be transformed into a vector that can be input into the common vector space 430. The transformation algorithm can output a predetermined vector format (e.g., 1x 4096-dimensional vector). Then, in step 608, the feature vector can be stored in one or more vector spaces 430 (e.g., a theme vector space 430a for text content, an image vector space 430b for image content, and/or a combined vector space for multiple content types). The vector space and the feature vectors stored therein can be generated and maintained by the content management system 435, the content recommendation engine 425, and/or other components of the system 400.

In some embodiments, a subset of the extracted/determined content features may also be saved as tags associated with the content item. An example process for generating and storing image tags based on an image and, in turn, retrieving an image based on the image tags is shown in FIGS. 9-11. Although these examples relate to image content items, similar tagging processes and/or keyword or topic extraction may be performed on text content items, audio/video content items, and the like. As shown in FIG. 9 , in step 901, an image may be created and/or uploaded to, for example, a content repository 440. In step 902, the image may be transferred from the content repository 440 to an artificial intelligence (AI)-based REST service configured to analyze the image and extract themes, topics, specific visual features, and the like. The AI REST service may determine one or more specific image tags based on the identified image features, and in step 903, the image tags may be transferred back to the content repository to be stored in or associated with the image in step 904. In FIG. 10 , a process identical to the process described in FIG. 9 is shown for generating and storing image tags based on an image. In addition, Figure 10 shows several illustrative features 1001 that can be implemented within the AI REST service in certain embodiments, including an image tag determination/retrieval component, an Apache MxNet component, and a cognitive image service. After determining one or more tags for a content item, the tags can be either stored back in the content repository 440 or stored in a separate storage location. For example, referring to the example image of Figure 7, twelve or more potential image features can be extracted from this single image, all of which can be combined into a feature vector. However; the AI REST service and/or the content repository 440 may determine that for content matching, the best practice is to tag the image with only some of the most common topics of the image (e.g., coffee, retail).

Referring now briefly to FIG. 11 , another example process is shown that relates to the process described in FIGS. 9 and 10 for generating and storing image tags based on images. In FIG. 11 , the reverse process is illustrated, where image tags are used to retrieve matching images from content repository 440. In step 1101, content authoring user interface 415 or other front-end interface may determine one or more content tags based on input received via the interface. In this example, a single content tag (“waitress”) is determined from the received user input, and in step 1102, the content tag is transmitted to a search API associated with content repository 440. The search API may be implemented within one or more separate layers of the computing system, including within content input processing/analysis component 420, content recommendation engine 425, and/or content management system 435. In step 1103, data identifying matching images determined by the search API may be transmitted back to content authoring user interface 415 to be integrated in the interface or otherwise presented to the user.

Thus, after steps 602-608 are completed for a plurality of content resources in content repository 440, one or more vector spaces 430 may be populated with vectors, where each vector corresponds to a content item in repository 440. Additionally, in some embodiments, a separate set of metadata tags may be generated for some or all of the content items and stored as separate objects from the vectors in vector space 430. Such tags may be stored in any of the data storage devices or components shown in FIG. 4, or in a separate data repository, and each tag may be associated with a content item in repository 440, a vector in vector space 430, or both.

Referring now to FIG. 12 , another flow chart is shown illustrating a second process for receiving original creative content from a user via a client device 410, extracting features and/or tags from the content in real time (or near real time) during the user's creative session, vectorizing the creative content (also in real time or near real time), and comparing the vectors of the original creative content to one or more existing vector spaces 430 in order to identify and retrieve related/associated content from one or more available content repositories 440. Steps in this process may also be performed by one or more components within the computing environment 400, such as a content recommendation engine 425 in collaboration with the client device 410, input processing/analysis component 420, and retrieval/embedding component 445, and the various subsystems and subcomponents implemented therein.

In step 1202, originally created content may be received from a user via client device 410. As discussed above, the originally created content may correspond to text typed by the user, a new image created or imported by the user, a new audio or video input recorded or imported by the user, a new graphic created by the user, etc. Thus, step 1202 may be similar to step 602 discussed above. However, while the content in step 602 may be previously created/stored content retrieved from repository 440, in step 1202, the content may be newly created content received via a user interface, such as a web-based text input control, an image importer control, an image creation control, an audio/video creation control, etc.

In step 1204, the content received in step 1202 (e.g., originally authored content) may be processed, for example, by the input processing/analysis component 420. Step 1204 may be similar or identical to step 604 discussed above with respect to parsing steps, processing steps, keyword/data feature extraction steps, etc. For example, for text input received in step 1202 (e.g., blog post, letter, email, article, etc.), the processing in step 1204 may include text parsing, identification of keywords, stemming, synonym analysis/retrieval, etc. In other examples, when an image is received in step 1202 (which may be uploaded by a user, imported from another system, and/or manually created or modified by a user via the content author user interface 415), step 1204 may include using an AI-based image classification tool as described above to identify specific image features and/or generate image tags. These analyses in step 1204 may be performed using analytics, machine learning algorithms, and/or AI (such as an AI-based cognitive image analysis service and/or an AI/REST cognitive text service) via a REST-based service or other web service. Similar techniques/services may be used in step 1204 for other types of content items, such as video files, audio files, graphics, or social media posts, where the dedicated web services of system 400 are used to extract and analyze specific features (e.g., words, objects in images/videos, facial expressions, etc.) depending on the media type of the content item. Step 1204 may also include any tagging process described herein for tagging text blocks, images, audio/video data, and/or other content with tags corresponding to any identified content themes, categories, or features.

In step 1206, one or more vectors compatible with one or more of the available vector spaces 430 may be generated based on the content received in step 1202. Step 1206 may be similar or identical to step 606 discussed above. As discussed above, vectors may be generated based on specific features within the content (and/or tags) identified in step 1204. The vector generation process in step 1206 may use one or more data transformation techniques, whereby a set of features associated with the originally authored content item may be transformed into a vector compatible with one of the public vector spaces 430. For example, FIG. 13 illustrates a technique by which, in step 1206, an image input received in step 1202 may be transformed into a feature vector (e.g., a 1x 4096 dimensional vector) in a predetermined vector format. As shown in FIG. 13, an image may be provided as an input to a model that is configured to extract and learn features in the image (e.g., using convolution, pooling, and other functions) and output a feature vector representing the input image. For example, as described in the 2014 paper “Visualizing and Understanding Convolutional Networks” by Matthew D. Zeiler and Rob Fergus of New York University, which is incorporated herein by reference, within a convolutional neural network, the initial layers of the neural network can detect simple features (such as linear edges) from an image, and more complex shapes and patterns in subsequent layers. For example, the first and/or second layers in a convolutional neural network can detect simple edges or patterns, while subsequent layers can detect actual complex objects (cups, flowers, dogs, etc.) present in the image. As an example, when a convolutional neural network is used to receive and process an image of a human face, the first layer can detect edges in various directions, the second layer can detect different parts of a given face (e.g., eyes, nose, etc.), and the third layer can obtain a feature map of the entire face.

In step 1208, the feature vector generated in step 1206 can be compared with the compatible feature vector space 430 (or space 430a-n) filled during the processes 602-608 discussed above. For example, an illustrative vector space filled with feature vectors corresponding to an image is shown in FIG. 14. In this example, each point can represent a vectorized image, and the circles in FIG. 14 (and the corresponding point colors) can indicate one of three example tags associated with the image. In this case, the image tags are "coffee", "mountain", and "bird", and it should be understood that these tags are not mutually exclusive (i.e., the image can be labeled with one, two, or all three tags). In addition, it should be understood that these tags and the layout of the multidimensional vector space in FIG. 14 are merely illustrative. In various embodiments, there is no limit on the number or type of tags that can be used, or on the dimensionality of the vector space 430.

To perform the vector space comparison in step 1208, the content recommendation engine 425 may calculate the Euclidean distance between the feature vector generated in step 1206 and each other feature vector stored in one or more vector spaces 430. Based on the calculated distance, the engine 425 may rank the feature vectors in ascending order of feature space distance, so that the smaller the distance between two feature vectors, the higher the ranking. Such a technique may allow the content recommendation engine 425 to determine a set of the highest ranked feature vectors within the (one or more) vector spaces 430, which are most similar to the feature vector generated in step 1206 in terms of features/characteristics/etc. based on the input received in step 1202. In some cases, a predetermined number (N) of the highest ranked feature vectors (e.g., 5 most similar articles, 10 most similar images, etc.) may be selected in step 1208, while in other cases, all feature vectors that meet a particular proximity threshold may be selected (e.g., distance between vectors < threshold (T)).

In some embodiments, the vector comparison in step 1208 may be a "deep feature space" comparison as shown in FIG. 15. In these embodiments, the feature vectors generated in step 1206 may be compared without regard to any tags or other metadata. In other words, in a deep feature comparison, the feature vector generated in step 1206 may be compared to every other feature stored in vector space 430. Although the deep feature comparison may be guaranteed to find the closest vector in vector space 430, such a comparison may require additional processing resources and/or additional time to return a vector result. This is especially true for large vector spaces that may include thousands or even millions of feature vectors, each of which represents a separate content object/resource stored in repository 440. For example, to calculate the Euclidean distance between two image feature vectors of size 1x4096, approximately 10,000 addition and multiplication instructions are required to be executed by system 400. Therefore, if there are 10,000 images in the repository, 10,000,000 operations must be performed.

Thus, in other embodiments, the vector comparison in step 1208 may be a "filtered feature space" comparison as shown in FIGS. 16-17. In a filtered feature space comparison, the vector space may first be filtered based on tags (and/or other characteristics, such as resource media type, creation date, etc.) to identify a subset of feature vectors within the vector space(s) 430 that have tags (and/or other characteristics) that match the tags (and/or other characteristics) of the feature vectors generated in step 1206. The feature vectors generated in step 1206 may then be compared to those feature vectors in the subset that have only matching tags/characteristics. Thus, while there may be the possibility of missing close feature vectors that are filtered out and not compared, a filtered feature space comparison may be performed faster and more efficiently than a deep space comparison.

As described above, step 1208 may include comparing the feature vector generated in step 1206 to a single vector space or to multiple vector spaces. In some embodiments, the feature vector generated in step 1206 may be compared to a corresponding type of vector space. For example, when text input is received in step 1202, the resulting feature vector may be compared to a vector within the topic vector space 430a, and when an image is received as input in step 1202, the resulting feature vector may be compared to a vector within the image vector space 430b, and so on. In some embodiments, it may be possible to compare feature vectors corresponding to one type of input to vector spaces containing different types of vectors (e.g., identifying image resources that are most closely related to text-based input, or vice versa). For example, FIG. 18 illustrates a process of receiving text input in step 1402 and retrieving similar images (e.g., the closest ones in the image vector space 430b) and similar articles (e.g., the closest ones in the topic vector space 430b) in step 1408.

For embodiments involving retrieval and/or comparison of tags associated with content resources, when the tag of a resource is related to but not fully matched with the corresponding tag/keyword/feature of another resource, problems may occur. An example of this potential problem is illustrated in Figure 19, wherein the keywords extracted from the text content resource of the original creation are compared with the set of image tags stored for the set of image content resources. In this example, the extracted keywords ("Everest", "Base Camp", "Summit", "Mountain" or "Himalaya") do not fully match the image tags ("Mountaineer", "Cappuccino" or "Macaw"). In some embodiments, word/phrase parsing and processing techniques such as stem, word definition and/or synonym retrieval and analysis can be used to detect the match between related but not matched terms. However, even for related keywords/tags, these technologies may fail. Therefore, in some embodiments, content processing/analysis component 420 and/or content recommendation engine 425 can perform word vector comparison to solve this problem. As shown in the example shown in FIG20, the keywords extracted from the text document in FIG19 can be analyzed in the 3D word vector space, and the distance between these keywords and each image label can be calculated. As shown in FIG21, the keyword-to-label vector space analysis performed in FIG20 can determine that the image label "Mountaineer" is close enough to the extracted keywords in the word vector space, so it should be considered an image label match for the filtered feature space comparison.

Another potential problem that may arise in embodiments of retrieving and/or comparing tags associated with content resources is caused by homonymous keywords and/or resource tags. Homographs or phrases (or homonyms) refer to words or phrases that have the same spelling and have different and unrelated meanings. An example of homonymous image tags is shown in Figure 22, where the first image is labeled with the word "Crane", which refers to a bird with long legs and a long neck, and the second image is labeled with the same word "Crane", which refers to a machine with an outstretched arm for moving heavy objects. In this case, the content processing/analysis component 420 and/or the content recommendation engine 425 can perform word sense disambiguation on the two image tags to determine which word sense each word Crane refers to. In this example, the word sense disambiguation process can initially retrieve the Wordnet database entry (or other definition data) associated with each tag, as shown in Figure 22 for two different "Crane" tags.

An illustrative word sense disambiguation process is shown in Figures 23-24. In this process, other keywords within the authored document and/or the specific context of the word "Crane" in the document (e.g., description, part of speech, tense, etc.) can be used by the content processing/analysis component 420 and/or the content recommendation engine 425 to determine the most likely meaning of the word Crane within the authored text document, thereby determining which "Crane" image tag is relevant to the authored text document. For example, now referring to Figure 23, an input text 2301 is shown from which multiple relevant keywords 2302 are extracted. The first extracted keyword ("Crane") can be compared to the image tags within the content repository 440, and in this example, two matching tags 2303 corresponding to two "Crane" tag images 2304a and 2304b within the content repository 440 have been identified.

As shown in FIG. 24 , to resolve the potential problem of word sense ambiguity, the disambiguation process can be continued by comparing one or more additional keywords 2302 extracted from the input content 2301 with other content tags of the two matching images 2304a and 2304b. In this example, the additional extracted keywords "mechanical", "machine", "lifting", and "construction" can be compared with the content tags and/or extracted features associated with each of the images 2304a and 2304b. As shown in FIG. 24 , these additional comparisons can disambiguate the initial keyword match for "Crane", so that the content recommendation system 425 does not return the avian crane image 2304a, but instead returns the construction crane image 2304b.

In other examples, similar disambiguation processing can be performed using image similarity. For example, the content processing/analysis component 420 and/or the content recommendation engine 425 can identify common image features between an image associated with the creative content (e.g., a drawn or creative image) and two different "Crane" images to determine which crane is a suitable related image. These disambiguation processes can also be combined in various ways, for example, comparing keywords extracted from the creative text document with visual features extracted from the image, etc. Therefore, in a text document involving the creation of a crane, if the boom and pulley can be visually identified in that image, then the related words "boom" and "pulley" may be visually matched with the lower crane image. Similarly, if the creative text document references a crane and contains the related words "beak" and "feathers", then if the beak and feathers can be visually identified in that image, then the "crane" keyword can be visually matched with the upper crane image.

Referring now to FIGS. 25-28 , an end-to-end example of performing the processing in FIG. 12 is shown, specifically for an embodiment of retrieving a collection of images related to an article authored by a user via the user interface 415 . Initially, at 2501 ( FIG. 25 ), a user enters the text of an article in the Demo Editor (AIditor) user interface. At 2502 , a plurality of keywords are extracted from the text of the article, and at 2503 , the AI Rest Service compares the extracted keywords with the image tags stored for the image library within the image content repository 440 . FIGS. 26 and 27 illustrate the same example processing as FIG. 25 , but with additional details about the operation of the AI Rest Service. As shown in FIG. 26 , using the techniques discussed above, the AI Rest Service identifies one or more tags (e.g., “mountaineer”) as being related to the authored article. Then, as shown in FIG. 27 , in some embodiments, different software services may be used to perform this step, such as a first cognitive text REST service for determining a collection of keywords from text input, and a second internal REST service for mapping keywords to image tags. Each of these services can be implemented within the content recommendation engine 425 and/or via an external service provider, and then, at step 2505 (shown in FIG. 28 ), the content recommendation engine 425 can transmit the determined (one or more) image tags to a search API associated with the image content repository 440. In some cases, the search API can be implemented in a cloud-based content center (e.g., Oracle Content Management (OCM)). In step 2506, the search API can retrieve a collection of related images based on tag matching, and in step 2507, the retrieved image (or a scaled-down version of the image) can be transmitted back and embedded in the user interface at 415 (at screen area 2810).

Although the examples shown in Figures 25-28 illustrate specific embodiments of a collection of related images of articles created by a user, it should be understood that the steps in Figure 12 can also be performed similarly to retrieve other types of content. For example, similar steps can be performed to retrieve articles (or other text documents) related to the text entered by the user via the user interface 415. In other embodiments, related content resources of other media types (e.g., audio files, video clips, graphics, social media posts, etc.) can also be retrieved. In addition, when a user imports/creates other types of input other than text into the user interface (e.g., drawn or uploaded images, spoken audio input, video input, etc.), similar steps can be performed to retrieve various different types of related content resources (e.g., related articles, images, videos, audio, social media, etc.) according to the configuration of the content recommendation engine 425 and/or user preferences.

For example, now referring to Figures 29-35, another example embodiment is shown, in which the processing steps of Figure 12 are performed to retrieve a collection of related articles (or other text content resources) based on the text input of the original creation received via the user interface 415 (e.g., a user's blog post, email, article, etc.). As shown in Figure 29, the user has created a new article by the user interface 415, and the AI-based REST service called by the content recommendation engine 425 has identified a collection of article topics. As shown in Figure 30, the identified article topics can be compared with the topics previously identified for the collection of articles in the article content repository 440. In these examples, Figure 29 is shown according to one embodiment, while Figure 30 shows another embodiment. Figure 29 is only a subset of Figure 30, and we can safely remove Figure 29 from the figure. Therefore, the article topic can be determined and stored as metadata or other associated data objects using techniques similar to the processing described above in Figure 6), wherein image features/labels are determined and stored in association with the image. Similarly, for each article stored in the repository 440, the article content repository 440 may have metadata or other associated storage, including article topics, dates, keywords, authors, publications, etc. In the example shown in FIG30, based on the matching of article topics, articles related to death on Mount Everest have been determined to be potentially relevant to the user's newly created article. FIGS. 31-35 illustrate an end-to-end process of using the system 400 to find articles related to the article input by the user, similar to the steps for finding related images shown in FIGS. 25-28. In step 3101 (FIG. 31), the user creates a new article via the user interface 415. In step 3102, the article text is transmitted to one or more software services (e.g., an AI-based REST service) by the content recommendation engine 425, and in step 3103 (FIG. 32), (one or more) software services use cognitive text service functionality to analyze the text of the article to determine one or more topics of the article. In step 3104, the determined article topic is transmitted back to the content recommendation engine 425, and in step 3105, the recommendation engine 425 sends both the article text and the identified topic to a separate API (e.g., in a cloud-based content center), where in step 3106, the article can be saved to a repository 440 for future reference and indexed based on the identified topic. Moreover, in step 3106 (FIG. 33), the existing repository 440 of articles can be searched via a search API based on a topic matching process (FIG. 34) to identify potentially relevant topics. Finally, in step 3107 (FIG. 35), the article identified as potentially relevant to the newly created article can be transmitted back (in its entirety or just a link) to be embedded in the user interface 415 (e.g., at user interface area 3510).

As illustrated in the above examples of Figures 29-33, some embodiments described herein may include identification of the subject of a newly created text document and/or a subject of a text document stored in a content repository 440, as well as comparison and identification of subject proximity and matching. In various embodiments described herein, various techniques including explicit semantic analysis may be used for text subject evaluation and subject "proximity" techniques. As shown in Figures 29-30, in some cases, such techniques may use large-scale data sources (e.g., Wikipedia) to provide fine-grained semantic representations of unrestricted natural language texts to represent the meaning of natural concepts derived from data sources in high-dimensional space. For example, text classification techniques may be used to explicitly represent the meaning of any text based on Wikipedia-based concepts. Semantic representations may be feature vectors of text fragments that are converted by topic modeling. Wikipedia (or another large-scale data source) may be used to include larger vocabularies (e.g., bags of words) into the system to cover a wide range of words. Wikipedia-based concepts may be titles of Wikipedia pages that are used as classes/categories when classifying given text fragments. Given a text snippet, the closest Wikipedia page title can be returned (e.g., "Mount Everest", "Stephen Hawking", "Car Accident", etc.), which can be used as the class/category of the text. The effectiveness of such techniques can be automatically evaluated by computing the semantic relatedness between snippets of natural language text.

In these text classification/relevance evaluation techniques, one of the advantages of using large-scale public available knowledge sources (e.g., Wikipedia or other encyclopedias) is that a large amount of highly organized human knowledge pre-encoded into public available resources can be accessed, which is in constant change/development. A semantic interpreter can be constructed based on Wikipedia and/or other sources using machine learning techniques, which maps segments of natural language text to weighted sequences of Wikipedia concepts sorted by their relevance to input. Therefore, the input text can be represented as a weighted vector of concepts, called an interpretation vector. Therefore, its meaning is interpreted according to the similarity of the text segment with many Wikipedia concepts. Then, the semantic relevance of the text can be calculated by comparing their vectors in a space defined by the concepts, such as using a cosine metric. Such semantic analysis can be explicit in the sense that obvious concepts can be based on human cognition. Because user input can be received in the form of plain text via user interface 415, conventional text classification algorithms can be used to rank the concepts represented by these articles according to their relevance to a given text segment. Therefore, online encyclopedias (e.g., Wikipedia) can be used directly without in-depth language understanding or pre-cataloged common sense knowledge. In some embodiments, each Wikipedia concept can be represented as an attribute vector of words that appear in the corresponding article. The terms "frequency inverted document frequency (TFIDF) scheme" can be used, for example, to assign weights to the entries of these vectors. These weights can quantify the strength of association between words and concepts. In order to speed up semantic interpretation, an inverted index can be used, which maps each word to a list of concepts in which the word appears. By removing those concepts whose weights of a given word are lower than a certain threshold, the inverted index can also be used to discard insignificant associations between words and concepts. A semantic interpreter can be implemented as a centroid-based classifier that can rank Wikipedia concepts by relevance based on received text fragments. For example, a semantic interpreter in a content recommendation engine can receive an input text fragment T and represent the fragment as a vector (e.g., using the TFIDF scheme). A semantic interpreter can iterate over text words, retrieve corresponding entries from an inverted index, and merge them into a weighted vector. The entries of the weighted vector may reflect the relevance of the corresponding concept to the text T. To calculate the semantic relevance of a pair of text segments, their vectors may be compared using, for example, the cosine metric.

In other examples, similar methods for generating features for text classification may include supervised learning tasks, in which words that appear in training documents can be used as features. Therefore, in some examples, Wikipedia concepts can be used to expand word bags. On the other hand, calculating the semantic relevance of a pair of texts is essentially a "one-time" task, so the word bag representation can be replaced by a concept-based representation. Evgeniy Gabrilovich and Shaul Markovitch of the Department of Computer Science and Technology at the Israel Institute of Technology describe these techniques and other related techniques in more detail in the paper "Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis", which is incorporated herein by reference for all purposes, as well as other related content discussed therein. Using the techniques and other techniques described in this paper, for a filtered subset of Wikipedia, for each article, there can be a concept as the title of the article. When the content recommendation engine 425 receives a text document via the user interface 415, the text can be summarized first. In these cases, after removing stop words and stemming the words, each unique word in the text can be weighted according to the frequency of occurrence and inversion frequency of the word in the article. Each word can then be compared to see in what Wikipedia article (concept) it appears, so the content recommendation engine 425 can create a concept vector for the word. The concept vectors of all words in the text document can be combined to form a weighted concept vector for the text document. The content recommendation engine 425 can then measure the similarity between each word concept vector and the text concept vector. All words above a certain threshold can then be selected as "keywords" for the document.

36, an example semantic text analyzer system 3600 is shown, illustrating techniques used by the analyzer system 3600 to perform semantic text summarization in certain embodiments described above. In various implementations, such a system 3600 may be combined with and/or separate from the content recommendation engine 425.

In some embodiments, the explicit semantic analysis used to calculate semantic relevance is reused to calculate a text summary for a given text document. More specifically, the text summary is derived based on word embeddings. In other words, compared to typical similarity measures (such as cosine on bag of words or edit distance on strings), the context of n-grams (e.g., words) is captured in order to determine semantic similarity.

A given text document may be an article, a web page, or other text for which a text summary is desired. As with the classification method described herein, text is not limited to written language, but may include other human-readable symbols, numbers, diagrams, tables, equations, formulas, and the like.

Text summarization methods using explicit semantic analysis generally operate as follows: (1) grammatical units (e.g., sentences or words) are extracted from a given text document using any known technique for identifying and extracting such units, (2) each extracted grammatical unit and the text document are represented as a weighted vector of a knowledge base concept; (3) the semantic relevance between the text document as a whole and each grammatical unit is calculated using the weighted vectors, and (4) one or more of the grammatical units that are most semantically relevant to the entire text document are selected to include in the text summary of the text document. In some cases, the representation of weighted vectors of knowledge base concepts can correspond to topic modeling, where each sentence or word can first be converted into a feature vector, and then feature differences/similarity can be calculated in a high-dimensional vector space. There can be a variety of methods for converting words into vectors, such as WORD2VEC or Latent Dirichlet Allocation.

Figure 36 illustrates a text summary using explicit semantic analysis. First, a text summarizer is constructed based on a knowledge base 3602. The knowledge base 3602 can be general or domain-specific. An example of a general knowledge base is a collection of encyclopedia articles, such as a collection of Wikipedia articles or other encyclopedias of text articles. However, the knowledge base 3602 can alternatively be domain-specific, such as a collection of text articles specific to a particular technical field, such as a collection of medical, scientific, engineering or financial articles.

Each article of knowledge base 3602 is represented as an attribute vector of n-grams (e.g., words) that appear in the article. Entries in the attribute vector are assigned weights. For example, the term "frequency inverted document frequency scoring scheme" can be used to use weights. The weights in the attribute vector of the article quantify the association strength between the n-grams (e.g., words) of the article and the article as a concept.

In some implementations, the term "inverse frequency document frequency scoring scheme" calculates a weight for a given n-gram t of a given article document d, as shown in the following equation:

Here, tf _t,d represents the frequency of n-gram t in document d. And df _t represents the document frequency of n-gram t in knowledge base 3602. M represents the total number of documents in the training set, L _d represents the length of document d in terms of terms, L _avg represents the average length of the training corpus, and K and b are free parameters. In some embodiments, k is approximately 1.5 and b is approximately 0.75.

The foregoing is an example of a term "frequency inverted document frequency scoring scheme" that can be used to weight the attributes of an attribute vector. Other statistical measures that reflect the importance of an attribute (e.g., an n-gram) to an article in the knowledge base 3602 can be used. For example, other TF/IDF variants that take into account anchor text (such as BM25F) can be used with certain types of knowledge bases (such as, for example, a knowledge base of web pages or other hyperlinked document collections).

The weighted inverted index builder computer 3604 builds a weighted inverted index 3606 from the property vectors representing the articles of the knowledge base 3602. The weighted inverted index 3606 maps each different n-gram represented in the set of property vectors to a concept vector of the concept (article) in which the n-gram appears. Each concept in the concept vector can be weighted according to the strength of the association between the concept and the n-gram to which the concept vector is mapped by the weighted inverted index 3606. In some embodiments, the indexer computer 3604 uses the inverted index 3606 to discard insignificant associations between n-grams and concepts by removing those concepts from the concept vector whose weight for a given n-gram is below a threshold.

To generate a text summary of a given text document 3610, grammatical units 3608 are extracted from the given text document 3610, and a semantic relevance between each grammatical unit and the given text document 3610 is calculated. A plurality of grammatical units having a high semantic relevance to the given text document 3610 are selected to be included in the text summary.

The number of grammatical units selected to be included in the text summary can vary based on various factors. One method is to select a predefined number of grammatical units. For example, the predefined number can be configured by the user of the system or learned through machine learning processing. Another method is to select all grammatical units with a certain degree of semantic relevance to a given text document 3610 that is higher than a predetermined threshold. The predefined threshold can be configured by the user of the system or learned through machine learning processing. Still another possible method is to determine the grammatical unit with the highest semantic relevance to a given text document 3610, and then select all other grammatical units in which the difference in semantic relevance between the grammatical unit and the given text document 3610 and the highest degree are lower than the predefined threshold. Select the grammatical unit with the highest degree and any other grammatical unit below the predefined threshold to be included in the text summary. Similarly, the predefined threshold can be configured by the user of the system or learned through machine learning processing.

In some embodiments, the grammatical unit with the highest or relatively high semantic relevance to a given text document 3610 is not always selected for inclusion in the text summary. For example, a first grammatical unit with a lower semantic relevance to a given text document 3610 than a second grammatical unit may be selected to be included in the text summary, while the second grammatical unit may not be selected to be included in the text summary if the first grammatical unit is not sufficiently dissimilar to grammatical units already selected to be included in the text summary. The degree of dissimilarity of grammatical units relative to existing text summaries can be measured in a variety of different ways, such as, for example, by using a lexical approach, a probabilistic approach, or a mixture of lexical and probabilistic approaches. Using a dissimilarity measure to select grammatical units to be included in a text summary can prevent multiple similar grammatical units from being included in the same text summary.

In some embodiments, other techniques for selecting multiple grammatical units to include in a text summary based on the semantic relevance of the units to a given text document 3610 and their dissimilarity to one or more other units may be used and are not limited to any particular technique. For example, given that the semantic relevance of multiple grammatical units to a given text document 3610 is above a threshold, the dissimilarity of each composite grammatical unit relative to the combination of multiple grammatical units may be measured, and multiple grammatical units that are least similar to each other may be selected to include them in the text summary. Therefore, overall, the grammatical units selected to be included in the text summary are highly semantically related to the text document, but are not similar to each other. This is a more useful text summary than a text summary containing highly semantically related but similar grammatical units, because similar grammatical units are more likely to be redundant with each other than dissimilar grammatical units in terms of the information conveyed by the grammatical units.

Another possibility is to compute a composite similarity/dissimilarity measure for grammatical units and then select grammatical units to include in the text summary based on their composite scores. For example, the composite measure can be a weighted average of the semantic relevance measure and the dissimilarity measure. For example, a possible composite measure computed as a weighted average is:

(a*similarity)+(b*dissimilarity)

Here, parameter similarity generally represents the semantic relevance of a grammatical unit to an input text 3610. For example, parameter similarity may be a similarity estimate 3620 calculated for a grammatical unit. Parameter dissimilarity represents a dissimilarity measure of the dissimilarity of a grammatical unit to a set of one or more grammatical units. For example, a set of one or more grammatical units may be a set of one or more grammatical units that have been selected to be included in a text summary. Parameter a represents the weight applied to the similarity measure in the weighted average. Parameter b represents the weight applied to the dissimilarity measure in the weighted average. The composite measure effectively balances the similarity measure and the dissimilarity measure with each other. They may be equally balanced with each other (e.g., a=0.5 and b=0.5). Alternatively, a greater weight may be given to the similarity measure (e.g., a=0.8 and b=0.2).

The grammatical unit extracted from a given text document may be a sentence, phrase, paragraph, word, n-gram or other grammatical unit. In the case where the grammatical unit 3608 extracted from a given text document 3610 is a word or n-gram, the process may be considered as keyword generation in contrast to text summarization.

The text summarizer 3612 accepts a text snippet. The text snippet is a given text document 3610 or a grammatical unit thereof. The text snippet is represented as an "input" vector of weighted attributes (e.g., words or n-grams) of the text snippet. Each weight in the input vector is for a corresponding attribute (e.g., word or n-gram) identified in the text snippet and represents the strength of association between the text snippet and the corresponding attribute. For example, the weights can be calculated according to a TF-IDF scheme, etc.

In some embodiments, the weights of the attributes in the input vector are calculated as follows:

Here, tf _t,d is the frequency of n-gram t in the text segment d. The parameters k, b, L _d, and L _avg are the same as before, but with respect to the knowledge base 3602 instead of the classification training set. In some embodiments, k is approximately 1.5 and b is approximately 0.75.

It should be noted that other weighting schemes are possible, and embodiments are not limited to any particular weighting scheme when forming the input vector.Forming the input vector may also include unit length normalization, such as described above with respect to the training data item vector.

The text summarizer 3612 iterates over the non-zero weighted attributes of the input vector formed based on the text segment, retrieves the attribute vector corresponding to the attribute from the weighted inverted index 3606, and merges the retrieved attribute vectors into a weighted vector representing the concept of the text segment. This weighted vector of the concept is hereinafter referred to as a "concept" vector.

The attribute vectors corresponding to the attributes of the input vector retrieved from the weighted inverted index 3606 are also vectors of weights. However, the weights in the attribute vectors quantify the strength of the association between the corresponding concept of the knowledge base 3602 and the attributes mapped to the attribute vectors by the inverted index 3606.

The text summarizer 3612 creates a concept vector for the text segment. A concept vector is a vector of weights. Each weight in the concept vector represents the strength of association between the corresponding concept of the knowledge base 3602 and the text segment. The concept weights in the concept vector are calculated by the text summarizer 3612 as the sum of values, and each attribute has a non-zero weighted value in the input vector. Each value of the sum for the attribute is calculated as the product of (a) the weight of the attribute in the input vector and (b) the weight of the concept in the attribute vector for the attribute. Each concept weight in the concept vector reflects the relevance of the concept to the text segment. In some embodiments, the concept vector is normalized. For example, the concept vector can be normalized for unit length or concept length (e.g., class length as above).

The text summarizer 3612 can generate a concept vector 3616 for the input text 3610 and a concept vector 3614 for each grammatical unit 3608. The vector comparator 3618 compares the concept vector 3614 generated for the grammatical unit with the concept vector 3616 generated for the input text 3610 using a similarity measure to generate a similarity estimate 3620. In some embodiments, a cosine similarity measure is used. Embodiments are not limited to any particular similarity measure, and any similarity measure capable of measuring the similarity between two non-zero vectors can be used.

The similarity estimate 3620 quantifies the semantic relatedness between the grammatical unit and the input text 3610 from which the grammatical unit was extracted. For example, the similarity estimate 3620 can be a value between 1 and 0, inclusive, where values closer to 1 represent higher semantic relatedness and values closer to 0 represent lower semantic relatedness.

A similarity estimate 3620 may be calculated for each grammar unit 3608. The similarity estimates 3620 generated for the grammar units 3608 may be used to select one or more of the grammar units 3608 to include in a text summary of the input text 3610 (or to select one or more keywords for generating keywords for the input text 3610).

The above-described techniques have various applications for text summarization to provide accurate text summaries of longer texts (such as, for example, news reports, blog posts, journal articles, web pages, etc.).

In any or all of the above embodiments, after one or more content resources (e.g., images, articles, etc.) have been identified as potentially relevant to the content currently being created by the user via the user interface 415, the relevant content resources are transmitted back to the content recommendation engine 425, where they can be retrieved, modified, and embedded into the user interface 415, for example, by the content retrieval/embedding component 445. Using the retrieval/embedding component 445, potentially relevant content resources can be provided to the user via the user interface 415 in a manner that the user can optionally include them in the content currently being created. Two example user interfaces are shown in Figures 37 and 38, in which image recommendations are provided to the user during content creation. In Figure 37, a media recommendation pane is shown, which includes images selected based on the text of the content currently being created by the user via the user interface. In Figure 38, visual feature analysis has been used to select a collection of images that may be relevant to the first image ("filename.JPG") selected by the user. Similar techniques and user interface screens can be used to allow users to select, drag and drop images, links to articles and other text documents, audio/video files, etc., and embed them into the content currently being created by the user.

39 depicts a simplified diagram of a distributed system 3900 in which the various examples discussed above may be implemented. In the example shown, the distributed system 3900 includes one or more client computing devices 3902, 3904, 3906, 3908 coupled to a server 3912 via one or more communication networks 3910. The client computing devices 3902, 3904, 3906, 3908 may be configured to run one or more applications.

In various embodiments, the server 3912 may be adapted to run one or more services or software applications that enable one or more operations associated with the content recommendation system 400. For example, a user may use a client computing device 3902, 3904, 3906, 3908 (e.g., corresponding to the content author device 410) to access one or more cloud-based services provided via the content recommendation engine 425.

In various examples, the server 3912 may also provide other services or software applications, and may include non-virtualized and virtualized environments. In some examples, these services may be provided to users of client computing devices 3902, 3904, 3906, 3908 as web-based services or cloud services, such as under a software as a service (SaaS) model. Users operating client computing devices 3902, 3904, 3906, 3908 may in turn utilize one or more client applications to interact with the server 3912 to utilize the services provided by these components.

In the configuration depicted in Figure 39, the server 3912 may include one or more components 3918, 3920, and 3922 that implement the functions performed by the server 3912. These components may include software components, hardware components, or a combination thereof that may be executed by one or more processors. It should be appreciated that a variety of different system configurations are possible that may differ from the example distributed system 3900.

Client computing devices 3902, 3904, 3906, 3908 may include various types of computing systems, such as portable handheld devices, such as smartphones and tablets; general-purpose computers, such as personal computers and laptops; workstation computers; wearable devices, such as head-mounted displays; gaming systems, such as handheld gaming devices, gaming consoles, and Internet-enabled gaming devices; thin clients; various messaging devices; sensors and other sensing devices; and the like. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, Linux or Linux-like operating systems), including various mobile operating systems (e.g., Microsoft Windows Windows Android ^TM 、 Palm ). The client device may be capable of executing a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use various communication protocols. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via this interface. Although FIG. 39 depicts only four client computing devices, any number of client computing devices may be supported.

The network(s) 3910 in the distributed system 3900 may be any type of network familiar to those skilled in the art, which may support data communications using any of a variety of available protocols, including, but not limited to, TCP/IP (Transmission Control Protocol/Internet Protocol), SNA (Systems Network Architecture), IPX (Internetwork Packet Exchange), AppleTalk, etc. Merely by way of example, the network(s) 3910 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., in any of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suites, and/or any other wireless protocol operating under the network) and/or any combination of these and/or other networks.

The server 3912 may be composed of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, The server 3912 may be composed of a plurality of servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable arrangement and/or combination. The server 3912 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices of the server. In various examples, the server 3912 may be suitable for running one or more services or software applications that perform the above operations.

Server 3912 can run an operating system, including any of the operating systems discussed above and any commercially available server operating system. Server 3912 can also run any of a variety of additional server applications and/or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Server, database server, etc. Examples of database servers include, but are not limited to, database servers commercially available from Oracle, Microsoft, Sybase, IBM (International Business Machines), etc.

In some embodiments, the server 3912 may include one or more applications to analyze and integrate data feeds and/or event updates received from users of the client computing devices 3902, 3904, 3906, 3908. As examples, the data feeds and/or event updates may include, but are not limited to, feeds or real-time updates received from one or more third-party information sources and continuous data streams, which may include real-time events related to sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. The server 3912 may also include one or more applications that display the data feeds and/or real-time events via one or more display devices of the client computing devices 3902, 3904, 3906, 3908.

The distributed system 3900 may also include one or more data repositories 3914, 3916. These data repositories may provide a mechanism for storing various types of information, such as the information described by the various examples discussed above. The data repositories 3914, 3916 may reside in a variety of locations. For example, a data repository used by a server 3912 may be local to the server 3912, or may be remote from the server 3912 and communicate with the server 3912 via a network-based or dedicated connection. The data repositories 3914, 3916 may be of different types. In some examples, the data repository used by the server 3912 may be a database, such as a relational database, such as a database provided by Databases provided by companies and other suppliers. One or more of these databases may be adapted to respond to SQL format commands to implement storage, updating, and retrieval of data in the database.

In some examples, the application may also use one or more of the data repositories 3914, 3916 to store application data. The data repositories used by the application may be of different types, such as, for example, a key-value store, an object store, or a general store backed by a file system.

In some examples, the cloud environment can provide one or more services, such as the services discussed above. Figure 40 is a simplified block diagram of one or more components of system environment 4000, in which these and other services can be provided as cloud services. In the example shown in Figure 40, cloud infrastructure system 4002 can provide one or more cloud services that can be requested by a user using one or more client computing devices 4004, 4006 and 4008. Cloud infrastructure system 4002 may include one or more computers and/or servers, which may include those computers and/or servers described above for server 3912 of Figure 39. The computers in cloud infrastructure system 4002 can be organized as general-purpose computers, dedicated server computers, server farms, server clusters or any other suitable arrangements and/or combinations.

The network(s) 4010 can facilitate communication and data exchange between clients 4004, 4006, and 4008 and cloud infrastructure system 4002. The network(s) 4010 can include one or more networks. The networks can be of the same or different types. The network(s) 4010 can support one or more communication protocols, including wired and/or wireless protocols, to facilitate communication.

The example depicted in Figure 40 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be appreciated that in other examples, cloud infrastructure system 4002 may have more or fewer components than those shown in Figure 40, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 40 depicts three client computing devices, any number of client computing devices may be supported in other examples.

The term "cloud service" is generally used to refer to services that are made available to users on demand by a service provider's system (e.g., cloud infrastructure system 4002) and via a communications network such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own internal servers and systems. The cloud service provider's system is managed by the cloud service provider. Customers can therefore take advantage of cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system can host applications, and users can order and use the applications on demand and self-service via the Internet without having to purchase infrastructure resources for executing the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, The company provides several cloud services such as middleware services, database services, Java cloud services, etc.

In various examples, cloud infrastructure system 4002 may provide one or more cloud services using different models such as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), and other models including hybrid service models. Cloud infrastructure system 4002 may include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

The SaaS model enables applications or software to be delivered to customers as a service over a communication network such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by the cloud infrastructure system 4002. Examples of SaaS services provided by the Company include, but are not limited to, services for human resources/capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, and others.

The IaaS model is often used to provide infrastructure resources (eg, servers, storage devices, hardware, and networking resources) to customers as cloud services to provide elastic computing and storage capabilities. The company provides a variety of IaaS services.

The PaaS model is typically used to provide platform and environment resources as a service, which enables customers to develop, run, and manage applications and services without having to purchase, build, or maintain such resources. Examples of PaaS services provided by the Company include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, and other services.

In some examples, resources in cloud infrastructure system 4002 can be shared by multiple users and dynamically reallocated based on demand. In addition, resources can be allocated to users in different time zones. For example, cloud infrastructure system 4002 can enable a first set of users in a first time zone to utilize resources of the cloud infrastructure system for a specified number of hours, and then enable the same resources to be reallocated to another set of users in a different time zone, thereby maximizing utilization of the resources.

The cloud infrastructure system 4002 can provide cloud services via different deployment models. In a public cloud model, the cloud infrastructure system 4002 can be owned by a third-party cloud service provider, and cloud services are provided to any general public customer, where the customer can be an individual or an enterprise. In certain other embodiments, under a private cloud model, the cloud infrastructure system 4002 can be operated within an organization (e.g., within an enterprise organization) and provide services to customers within the organization. For example, customers can be various departments of an enterprise, such as a human resources department, a payroll department, etc., or even individuals within an enterprise. In certain other embodiments, under a community cloud model, the cloud infrastructure system 4002 and the services provided can be shared by several organizations in a related community. Various other models can also be used, such as a mix of the models mentioned above.

The client computing devices 4004, 4006, 4008 may be devices similar to those described above with respect to the client computing devices 3902, 3904, 3906, 3908 of Figure 39. The client computing devices 4004, 4006, 4008 of Figure 40 may be configured to operate a client application, such as a web browser, a proprietary client application (e.g., Oracle Forms), or some other application, that may be used by a user of the client computing device to interact with the cloud infrastructure system 4002 to use services provided by the cloud infrastructure system 4002.

In various examples, the cloud infrastructure system 4002 may also provide "big data" and related computing and analysis services. The term "big data" is generally used to refer to very large data sets that can be stored and manipulated by analysts and researchers to visualize large amounts of data, detect trends, and/or otherwise interact with the data. The analysis that the cloud infrastructure system 4002 may perform may involve using, analyzing, and manipulating large data sets to detect and visualize various trends, behaviors, relationships, etc. within the data. Such analysis may be performed by one or more processors, possibly processing data in parallel, performing simulations using the data, etc. The data used for such analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and/or unstructured data (e.g., data blobs (binary large objects)).

40, cloud infrastructure system 4002 may include infrastructure resources 4030 for facilitating the provision of various cloud services provided by cloud infrastructure system 4002. Infrastructure resources 4030 may include, for example, processing resources, storage or memory resources, networking resources, and the like.

In some examples, to facilitate efficient provisioning of these resources to support the various cloud services provided by the cloud infrastructure system 4002 for different customers, the resources can be bundled into a collection of resources or resource modules (also referred to as a "pod"). Each resource module or cluster can include a pre-integrated and optimized combination of one or more types of resources. In some examples, different clusters can be pre-provisioned for different types of cloud services. For example, a first set of clusters can be provisioned for database services, and a second set of clusters can be provisioned for Java services, where the second set of clusters can include a different combination of resources than the clusters in the first set of clusters. For some services, resources allocated to provision the service can be shared between services.

The cloud infrastructure system 4002 itself may internally use services 4032 that are shared by different components of the cloud infrastructure system 4002 and facilitate the service provisioning of the cloud infrastructure system 4002. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, and the like.

In various examples, cloud infrastructure system 4002 may include multiple subsystems. These subsystems may be implemented in software or hardware or a combination thereof. As shown in FIG. 40 , a subsystem may include a user interface subsystem 4012 that enables a user or customer of cloud infrastructure system 4002 to interact with cloud infrastructure system 4002. User interface subsystem 4012 may include various interfaces, such as web interface 4014, online store interface 4016 (wherein cloud services provided by cloud infrastructure system 4002 are advertised and can be purchased by consumers), and other interfaces 4018. For example, a customer may use a client device to request (service request 4034) one or more services provided by cloud infrastructure system 4002 using one or more of interfaces 4014, 4016, and 4018. For example, a customer may access an online store, browse cloud services provided by cloud infrastructure system 4002, and place a subscription order for one or more services that the customer wishes to subscribe to in the services provided by cloud infrastructure system 4002. A service request may include information identifying the customer and one or more services that the customer wishes to subscribe to. For example, a customer may place an order for services such as those discussed above. As part of the order, the customer may provide information identifying, among other things, the amount of resources the customer requires and/or within what time frame.

In some examples, such as the example depicted in Figure 40, cloud infrastructure system 4002 may include an order management subsystem (OMS) 4020 configured to process new orders. As part of this processing, OMS 4020 may be configured to: generate an account for the customer (if not already created); receive billing and/or accounting information from the customer that will be used to bill the customer for providing the requested services to the customer; verify the customer information; upon verification, book the order for the customer; and orchestrate various workflows to, among other things, prepare the order for provision.

Once properly authenticated, OMS 4020 may call order provisioning subsystem (OPS) 4024, which is configured to provision resources for the order, including processing resources, memory resources, and networking resources. Provisioning may include allocating resources to the order, and configuring resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud service that the customer has ordered. For example, according to a workflow, OPS 4024 may be configured to determine the specific cloud service requested, and identify multiple clusters that may have been pre-configured for the specific cloud service. The number of clusters allocated for the order may depend on the size/quantity/level/scope of the requested service. For example, the number of clusters to be allocated may be determined based on the number of users supported by the service, the duration of the service being requested, and the like. The allocated clusters may then be customized for the specific requesting customer to provide the requested service.

Cloud infrastructure system 4002 can send a response or notification 4044 to the requesting customer to indicate when the requested service is ready. In some cases, information (e.g., a link) can be sent to the customer so that the customer can start using and utilizing the benefits of the requested service.

The cloud infrastructure system 4002 can provide services to multiple customers. For each customer, the cloud infrastructure system 4002 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the order, and providing the requested services to the customer. The cloud infrastructure system 4002 can also collect usage statistics about the customer's use of the subscribed services. For example, statistics can be collected for the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information can be used to bill the customer. Billing can be performed, for example, on a monthly cycle.

The cloud infrastructure system 4002 may provide services to multiple customers in parallel. The cloud infrastructure system 4002 may store information of these customers, including possibly proprietary information. In some examples, the cloud infrastructure system 4002 includes an identity management subsystem (IMS) 4028 that is configured to manage the information of the customers and provide separation of the managed information so that information associated with one customer is not accessible to another customer. The IMS 4028 may be configured to provide various security-related services, such as identity services, such as information access management, authentication and authorization services, services for managing customer identities and roles and related capabilities, and the like.

FIG. 41 illustrates an example of a computer system 4100 that can be used to implement the various examples discussed above. In some examples, the computer system 4100 can be used to implement any of the various servers and computer systems described above. As shown in FIG. 41 , the computer system 4100 includes various subsystems, including a processing subsystem 4104 that communicates with multiple other subsystems via a bus subsystem 4102. These other subsystems may include a processing acceleration unit 4106, an I/O subsystem 4108, a storage subsystem 4118, and a communication subsystem 4124. The storage subsystem 4118 may include a non-transient computer-readable storage medium 4122 and a system memory 4110.

The bus subsystem 4102 provides a mechanism for making the various components and subsystems of the computer system 4100 communicate with each other as desired. Although the bus subsystem 4102 is schematically shown as a single bus, the alternative example of the bus subsystem can utilize multiple buses. The bus subsystem 4102 can be any of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, and a local bus using any various bus architectures. For example, such architectures can include an industrial standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard, etc.

The processing subsystem 4104 controls the operation of the computer system 4100 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may include a single-core or multi-core processor. The processing resources of the computer system 4100 may be organized into one or more processing units 4132, 4134. The processing unit may include one or more processors (including single-core or multi-core processors), one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 4104 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing subsystem 4104 may be implemented using custom circuits, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

In some examples, the processing units in the processing subsystem 4104 can execute instructions stored in the system memory 4110 or on the computer-readable storage medium 4122. In various examples, the processing units can execute various programs or code instructions, and can maintain multiple concurrently executed programs or processes. At any given time, some or all of the program code to be executed may reside in the system memory 4110 and/or on the computer-readable storage medium 4122, including possibly on one or more storage devices. Through appropriate programming, the processing subsystem 4104 can provide the various functionalities described above. In the case where the computer system 4100 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

In some examples, a processing acceleration unit 4106 may be provided for performing customized processing or for offloading some of the processing performed by processing subsystem 4104 , thereby speeding up the overall processing performed by computer system 4100 .

The I/O subsystem 4108 may include devices and mechanisms for inputting information to the computer system 4100 and/or for outputting information from or via the computer system 4100. In general, use of the term "input device" is intended to include all possible types of devices and mechanisms for inputting information to the computer system 4100. User interface input devices may include, for example, keyboards, pointing devices such as a mouse or trackball, touch pads or touch screens incorporated into a display, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include software such as Microsoft Windows that enables a user to control and interact with input devices. Motion sensing and/or gesture recognition devices, Microsoft 360 game controller, a device that provides an interface for receiving input using gestures and spoken commands. The user interface input device may also include an eye gesture recognition device that detects eye activity from the user (e.g., a "wink" when taking a picture and/or making a menu selection) and converts the eye gesture into input to the input device. In addition, the user interface input device may include a voice recognition sensing device that enables the user to interact with the voice recognition system through voice commands.

Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or indicator rods, game pads and graphic tablets, and audio/video devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser rangefinders, and eye gaze tracking devices. In addition, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, position emission tomography, medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 4100 to a user or other computers. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as one utilizing a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio/video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

Storage subsystem 4118 provides a repository or data repository for storing information used by computer system 4100. Storage subsystem 4118 provides a tangible, non-transitory computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. Software (e.g., programs, code modules, instructions) that provide the above functionality when executed by processing subsystem 4104 can be stored in storage subsystem 4118. The software can be executed by one or more processing units of processing subsystem 4104. Storage subsystem 4118 can also provide a repository for storing data used in accordance with the present disclosure.

The storage subsystem 4118 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in Figure 41, the storage subsystem 4118 includes a system memory 4110 and a computer-readable storage medium 4122. The system memory 4110 may include multiple memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, a basic input/output system (BIOS) may be typically stored in ROM, which contains basic routines that help to transfer information between elements within the computer system 4100, such as during startup. RAM typically contains data and/or program modules currently operated and executed by the processing subsystem 4104. In some embodiments, the system memory 4110 may include multiple different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

As an example and not a limitation, as shown in FIG. 41 , system memory 4110 may load application programs 4112 being executed, which may include client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), etc., program data 4111, and operating system 4116. As an example, operating system 4116 may include various versions of Microsoft Apple and/or Linux operating system, various commercial or UNIX-like operating systems (including but not limited to various GNU/Linux operating systems, OS, etc.) and/or mobile operating systems such as iOS, Phone, OS, 10OS and OS operating system.

Computer-readable storage media 4122 may store programming and data structures that provide some example functionality. Computer-readable media 4122 may provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 4100. When executed by processing subsystem 4104, software (programs, code modules, instructions) that provide the above functionality may be stored in storage subsystem 4118. As an example, computer-readable storage media 4122 may include non-volatile memory, such as a hard drive, a disk drive, a hard drive such as a CD ROM, a DVD, (Blu-ray) discs or other optical media. Computer readable storage media 4122 may include, but are not limited to, Drive, flash memory card, universal serial bus (USB) flash drive, secure digital (SD) card, DVD disk, digital video tape, etc. The computer-readable storage medium 4122 may also include a solid-state drive (SSD) based on non-volatile memory (such as an SSD based on flash memory, an enterprise flash drive, a solid-state ROM, etc.), an SSD based on volatile memory (such as an SSD based on solid-state RAM, dynamic RAM, static RAM, DRAM, magnetoresistive RAM (MRAM) SSD), and a hybrid SSD using a combination of DRAM-based and flash memory-based SSDs. The computer-readable storage medium 4122 may provide storage for computer-readable instructions, data structures, program modules, and other data for the computer system 4100.

In some examples, storage subsystem 4118 may also include a computer-readable storage medium reader 4120, which may be further connected to computer-readable storage medium 4122. Reader 4120 may receive and be configured to read data from a memory device such as a disk, flash drive, or the like.

In some examples, computer system 4100 can support virtualization technology, including but not limited to virtualization of processing and memory resources. For example, computer system 4100 can provide support for executing one or more virtual machines. Computer system 4100 can execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine generally runs independently of other virtual machines. Memory, computing (e.g., processors, cores), I/O, and networking resources can be allocated to each virtual machine. Each virtual machine typically runs its own operating system, which can be the same or different from the operating system executed by other virtual machines executed by computer system 4100. Therefore, computer system 4100 can potentially run multiple operating systems simultaneously.

The communication subsystem 4124 provides an interface to other computer systems and networks. The communication subsystem 4124 serves as an interface for receiving data from the computer system 4100 and transmitting data to other systems. For example, the communication subsystem 4124 can enable the computer system 4100 to establish a communication channel to one or more client devices via the Internet for receiving information from the client device and sending information to the client computing device.

The communication subsystem 4124 can support both wired and/or wireless communication protocols. For example, in some examples, the communication subsystem 4124 may include a radio frequency (RF) transceiver component, a global positioning system (GPS) receiver component, and/or other components for accessing a wireless voice and/or data network (e.g., using cellular telephone technology, advanced data network technology (such as 3G, 4G or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 family of standards), or other mobile communication technologies, or any combination thereof). In some examples, in addition to or instead of a wireless interface, the communication subsystem 4124 can provide a wired network connection (e.g., Ethernet).

The communication subsystem 4124 may receive and transmit data in various forms. For example, in some examples, the communication subsystem 4124 may also receive incoming communications in the form of structured and/or unstructured data feeds 4126, event streams 4128, event updates 4130, etc. For example, the communication subsystem 4124 may be configured to receive (or send) data feeds 4126 in real time from users of a social media network and/or other communication services such as feeds, updates, web feeds such as Rich Site Summary (RSS) feeds, and/or real-time updates from one or more third-party information sources.

In some examples, communication subsystem 4124 may be configured to receive data that may be continuous or unbounded in nature without a clear end in the form of a continuous data stream, where the continuous data stream may include event stream 4128 of real-time events and/or event updates 4130. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

The communication subsystem 4124 may also be configured to output structured and/or unstructured data feeds 4126 , event streams 4128 , event updates 4130 , etc. to one or more databases that may communicate with one or more streaming data source computers coupled to the computer system 4100 .

Computer system 4100 can be one of various types, including a handheld portable device (e.g., Cellular phone, computing tablet, PDA), wearable device (e.g., head mounted display), personal computer, workstation, mainframe, kiosk, server rack, or any other data processing system.

Due to the ever-changing nature of computers and networks, the description of the computer system 4100 depicted in FIG. 41 is intended to be used as a specific example only. Many other configurations with more or fewer components than the system depicted in FIG. 41 are possible. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will appreciate other ways and/or methods of implementing various embodiments.

Smart Content-Result Ranking

For a content recommendation system configured to recommend specific content (content items) in response to input from a client or user, the process of evaluating content items and ranking them relative to each other plays an important role in the overall recommendation. For example, a user may enter one or more search terms into a search engine query interface, and the recommendation system may recommend one or more related content items (e.g., web pages, documents, images, etc.) that best match the input search terms. In other examples, a user may create original content such as emails, documents, online articles, blog posts, etc. using a content creation system, which may be configured to provide recommendations for related content items that may be related to the content created by the user. For example, as described above, a content recommendation system may recommend related images or links to web pages, etc. to a user, so that, if desired, the author may incorporate one or more of the recommended content items into the content being created by the user. Several examples of such techniques are described in the above sections. As illustrated by these examples, the ranking and recommendation of specific content items may be applied to many different use cases in response to receiving manually input user input and/or automatic input from other processes. Content items to be ranked and/or recommended may correspond to images, web pages, other media files, documents, digital objects, etc. that the recommendation system may use for searching. These content items may be stored in one or more repositories accessible to the recommendation system, which may be private or public repositories (eg, the Internet).

However, ranking content items is not a simple task. For example, consider a recommendation system that uses tag matching technology to perform recommendations. In such a system, content items that can be searched by the recommendation system are tagged, and the content items and tags can be stored in one or more repositories. Tagging can be performed by a content item tagging service/application. For a content item, one or more tags associated with the content item indicate the content contained in the content item. A value (sometimes also referred to as a tag probability) can also be associated with each tag, wherein the value provides a measure (e.g., probability) of the content indicated by the tag that appears in the content item. After receiving a user input (e.g., a search term/phrase, content created by a user) for which a recommendation of a content item is to be provided, the user input can be analyzed to identify one or more tags to be associated with the user input. The recommendation system can then use tag matching technology to identify a set of content items from the content items that can be searched, and the associated tags of the content items match the tags associated with the user input. The recommendation system can then use a certain ranking algorithm to rank the content items in the identified set and display the results to the user.

However, in certain use cases, the effectiveness of the ranking algorithm used by the recommender system may be limited and may not produce optimal results. For example, consider a situation where there are multiple tags associated with a user input. For example, if a user types the words "coffee" and "person" into an image search engine, the tags "coffee" and "person" may be associated with the user input. In some embodiments, the search terms themselves may be considered tags associated with the user input. For simplicity, assume that the set of content items available for search includes tagged images. Using these two tags, the recommender system can retrieve a set of matching content items from a set of content items (e.g., images), where the content item is considered to be a match if at least one tag associated with the content item matches the tag associated with the user input. Consider a scenario (Case 1) in which multiple matching content items are retrieved by the recommender system, and each content item has the tags "coffee" and "person" associated with it. One possible way to rank these retrieved content items is to (a) for each matching content item, add the values associated with the tags "coffee" and "person" for that content item, and then (b) rank the content items based on their associated sum. However, this presents a problem because for multiple matching content items, the values associated with the matching tags may add up to the same value, which is a very likely case because most tagging services normalize probabilities to a scale of one. For example, three matching images may have the following associated tag values: Image A (("coffee", 0.5), ("person", 0.5)), Image B (("coffee", 0.2), ("person", 0.8)), and Image C (("coffee", 0.7), ("person", 0.3)). As can be seen, the sum of the values of the matching tags for each of these matching images is one ("1"), so it is not possible to rank one of these images above another using conventional summing techniques. Thus, ranking images based solely on the sum of their matching tag values cannot be used to rank images.

Expanding on the above example, the set of matched images may also include images that match only one label (e.g., only "coffee" or only "person") and have the same associated label value. This again raises the problem of ranking images. For example, consider that three matching images may have the following associated label values: Image A (("coffee", 0.5)) and Image B (("person", 0.5)). Again, there is no way to rank one of these images above the other using conventional techniques.

The situation worsens further when there are more than two tags associated with the user input. For example, if a user types the words "coffee", "person", and "cafe" into an image search engine, the search tags associated with the user input are three: "coffee", "person", and "cafe". The recommendation system can retrieve a set of matching content items from a set of content items (e.g., images), where a content item is considered a match if at least one tag associated with the content item matches the tag associated with the user input. The number of matching tags for a matching content item can vary from only one matching tag to multiple matching tags (a maximum of three matching tags for the "coffee", "person", and "cafe" example). Also in this scenario, for multiple matching content items, the values associated with the matching tags can add up to the same value. For example, three matching images may have the following associated label values: Image A (("person", 0.8), ("coffee", 0.2)), Image B (("person", 0.2), ("coffee", 0.2), ("cafe", 0.6)), and Image C (("person", 0.5), ("cafe", 0.5)). As can be seen, the values of the matching labels for each of these matching images sum to one ("1"), and therefore, these images cannot be ranked one above the other using conventional summation techniques.

Thus, in many cases, simple tag matching techniques may not return optimal content recommendations for a variety of reasons. For example, some images (or other content items) in a repository may be tagged with only one or two content tags, while other images/content items may be tagged with a large number of tags, including dozens or even hundreds of tags for a single content item. In such cases, conventional tag matching techniques may over-recommend the heavily tagged content item (e.g., because it more frequently includes at least one tag that matches the input term) and/or may under-recommend such items (e.g., because even if one or more content tags are matched, most of its tags will still not match the input terms). Similarly, the input content provided by a user or client system may include only a few input terms (e.g., explicitly entered search terms or topics extracted from a larger input text), or may include a relatively large number of input terms, depending on the input data received. In such cases, conventional tag matching techniques may either fail to identify certain relevant content items in the repository (e.g., because too few input terms match the tags of the relevant content items), or may erroneously recommend less relevant content items (e.g., because the less relevant content items include one or more matching tags).

In certain embodiments, improved techniques for evaluating, ranking, and recommending tagged content items are described herein. In some embodiments, a content recommendation system may receive input content (such as a search query, newly created text input, etc.) from a client device. One or more tags may be included in or associated with the input content received from the client device, and/or may be determined and extracted from the input content based on preprocessing and analysis techniques performed on the input content. In addition, the content recommendation system may access a content repository storing multiple tagged content items (such as images, media content files, links to web pages, and/or other documents). In some cases, the content repository may store data identifying tagged content items, and for each tagged content item, associated tag information for each item may also be stored, wherein the tag information for the content item includes information identifying one or more tags associated with the content item and a tag value for each associated tag.

In response to receiving input data for which recommendations are to be made, a content recommendation system may retrieve a set of matching tagged content items from a content repository from a set of searchable content items, wherein if at least one content tag associated with the content item matches a tag associated with the input content, then the content item is considered a matching content item. Then, for each matching tagged content item retrieved from the content repository, the content recommendation system may calculate two scores: (1) a first score based on the number of tags associated with the content item that matches the tag associated with the input content (also referred to as a tag count score), and (2) a second score based on the tag value of each matching tag of the content item (also referred to as a tag value-based score or TVBS). The content recommendation system then calculates a final ranking score for each matching content item based on the first score and the second score of the matching content item. The final ranking score calculated for the set of matching content items is then used to generate a ranked list of matching content items. This ranked list is then used to identify a recommended subset of matching content items to be output to a user or client system.

Referring now to FIG. 42 , a block diagram of an illustrated computing environment 4200 is shown having a content recommendation system 4220 implemented to evaluate and rank content items from a content repository 4230 in response to input content received from a user or client system 4210, according to some embodiments. Various components and subsystems within the content recommendation system 4220 are also shown in this example, including a graphical user interface (GUI) 4215 through which the client system 4210 can interact with the content recommendation system 4220 to provide input content and receive data identifying a subset of recommended content items. In some embodiments, the GUI 4215 can be a GUI of a separate client application 4215 (e.g., a web browser application) that a user uses to create content. In this embodiment, the content recommendation system 4220 can receive content provided or created by a user from a client application. The content recommendation system 4220 can receive content using an application programming interface (API) that enables the client application and the content recommendation system 4220 to interact and exchange information with each other.

The embodiments depicted in Figure 42 are merely examples and are not intended to unduly limit the scope of the claimed embodiments. Those of ordinary skill in the art will recognize many possible variations, substitutions, and modifications. For example, in some embodiments, the content recommendation system 4220 may have more or fewer systems or subsystems than the systems or subsystems shown in Figure 42, may combine two or more systems, or may have different configurations or arrangements of the system. The content recommendation system 4220 may be implemented as one or more computing systems, including, in some embodiments, separate systems using independent computing and network infrastructures with dedicated and specialized hardware and software. Alternatively or additionally, one or more of these components and subsystems may be integrated into a single system that performs separate functionality. The various systems, subsystems, and components depicted in Figure 42 may be implemented in software (e.g., code, instructions, programs), hardware, or a combination of software executed by one or more processing units (e.g., processors, cores) of the corresponding system. The software may be stored on a non-transitory storage medium (e.g., on a memory device).

At a higher level, the content recommendation system 4220 is configured to receive user input content, and then make content item recommendations in response to and based on the user input content. Recommendations are made from a collection of content items that are available and accessible to the content recommendation system 4220 for recommendation. The collection of content items may include images, various types of documents, media content, digital objects, etc. Based on the tag information associated with the user input content and the tag information associated with the collection of content items, the content recommendation system 4220 is configured to use tag matching technology to identify a collection of matching content items for the user input content. The content recommendation system 4220 is then configured to use the innovative ranking technology described in the present disclosure to rank the content items in the collection of matching content items. Based on the ranking, the content recommendation system 4220 is then configured to identify and recommend a subset of matching content items to be output to a user or client system.

The content recommendation system 4220 includes a content tagger subsystem 4222 configured to receive or retrieve content items that can be recommended by the content recommendation system 4220. Content items may include, but are not limited to, images, web pages, documents, media files, etc. Content items may be received or retrieved from one or more content repositories 4230. The content repositories 4230 may include various public or private content repositories, such as libraries or databases, including image repositories, document repositories, local or wide area networks of web-based resources (e.g., the Internet), etc. One or more content repositories 4230 may be stored locally in the content recommendation system 4220, while other content repositories may be separate, remote from the content recommendation system 4220 and accessible to the content recommendation system 4220 via one or more computer networks.

In some embodiments, for each content item, the content tagger subsystem 4222 is configured to retrieve and analyze the content of the content item and identify one or more content tags to be associated with the content item. For each tag associated with the content item, the content tagger 4222 can also determine a tag value associated with the tag, wherein the value provides a measure (e.g., probability) of the content indicated by the tag appearing in the content item. The tag value of the tag can be a numerical measure representing how a specific content tag is applicable to that content item. One or more tags and corresponding tag values can be associated with the content item. For a content item with multiple associated tags and corresponding tag values, the tag value can represent the relative prominence of the image theme or topic indicated by the tag in that image. For example, a first tag associated with a content item with a relatively high tag value can indicate that the content or feature indicated by that first tag is particularly relevant and prominent in the content item. In contrast, a second tag associated with the same content item with a lower tag value can indicate that the content or feature indicated by the second tag is less prominent or popular in the content item relative to the content indicated by the first content tag. For example, an image content item may have two associated labels and values as follows: ("person", 0.8), ("coffee", 0.2)). This indicates that the image contains content related to coffee (e.g., a coffee cup) and content related to a person (e.g., a person drinking coffee), and that the person is displayed more prominently in the image than the depiction of the coffee (e.g., a large portion of the image may depict the person, while the coffee cup may cover a small portion of the image). Label values can be represented using different formats. For example, in some embodiments, a label value can be expressed as a floating point number between 0.0 and 1.0. In some embodiments, the sum of all label values for a label associated with a particular content item can be summed to a fixed and uniform value (e.g., adding up to 1).

In some embodiments, content tagger 4222 can use the services of a content tagging service to perform tagging tasks for content items, including identifying one or more tags to be associated with a content item and a tag value for each tag. In some embodiments, content tagger 4222 is implemented using one or more predictive machine learning models that have been trained to take a content item as input and predict a tag and associated tag value for the content item. In some embodiments, the tags can be selected from a set of preconfigured tags used to train the model. Various machine learning techniques using pretrained machine learning models and/or other artificial intelligence-based tools (including AI-based text or image classification systems, topic or feature extraction, and/or any other combination of the above techniques) can be used to determine the tags to be associated with the content item and the corresponding tag values.

In some embodiments, the content items retrieved from the content repository 4230 may already include associated content tags and tag values. When the retrieved content items do not include tag information, and/or when the content recommendation system 4220 is configured to determine additional tags for the content items, the content tagger 4222 can be used to update or generate new tags for the retrieved content items. The content tagger 4222 can use a variety of different techniques to generate tag information (e.g., one or more tags and associated tag values) for the content items. For example, the content tagger 4222 can use any or all of the previously described techniques to analyze the retrieved content items and determine content tags, such as parsing, processing, feature extraction, and/or other analysis techniques. The type of parsing, processing, feature extraction, and/or analysis can depend on the type of content item. For example, for text-based content items, such as blog posts, letters, emails, articles, documents, etc., the analysis can include keyword extraction and processing tools (e.g., stem extraction, synonym retrieval, etc.), topic analysis tools, etc. For content items that are images, an artificial intelligence-based image classification tool can be used to identify specific image features and/or generate image tags. For example, analysis of an image may identify multiple image features, and the image may be tagged with each of these identified features. One or both types of analysis (i.e., extracting tags from the image and extracting keywords/topics from the text content) may be performed via a REST-based service or other web service using analytics, machine learning algorithms, and/or artificial intelligence (AI)-based techniques (such as an AI-based cognitive image analysis service, or a similar AI/REST cognitive text service to be used for text content). Similar techniques may be used for other types of content items, such as video files, audio files, graphics, or social media posts, where specialized web services may be used to extract and analyze specific features (e.g., words, objects in an image/video, facial expressions, etc.), depending on the media type of the content item.

In some embodiments, content tagger 4222 may use one or more pre-trained models based on machine learning and/or artificial intelligence trained with training data to identify and extract content features to be used to determine tags and tag values for content items. For example, a model training system may generate one or more models that may be pre-trained using a machine learning algorithm based on a training data set including a training data set of previous input data (e.g., text input, images, etc.) and corresponding tags for the previous input data. In various embodiments, one or more different types of trained models may be used, including a classification system that performs supervised or semi-supervised learning techniques (such as a naive Bayes model, a decision tree model, a logistic regression model, or a deep learning model), or any other machine learning or artificial intelligence-based prediction system that can perform supervised or unsupervised learning techniques. For each machine learning model or model type, the trained model may be executed by one or more computing systems, during which the content item is provided as input to one or more models, and the output from the model may identify one or more tags as being associated with the content item, or the output of the model may be used to identify one or more tags to be associated with the content item. Thus, the content tagger 4222 can use a variety of different tools or techniques, such as, but not limited to, keyword extraction and processing (e.g., stemming, synonym retrieval, etc.), topic analysis, feature extraction from images, machine learning and AI-based modeling tools and text or image classification systems, and/or any other combination of the foregoing techniques to determine or generate tag information (e.g., one or more tags and associated tag values) for each content item that can be used for recommendation.

In some embodiments, content items that can be used for recommendation and their associated tag information (e.g., for each content item, one or more tags associated with the content item and the corresponding tag value) can be stored in data repository 4223. In some embodiments, content/tag information data repository 4223 can store data identifying content items retrieved from content repository 4230, which can include the items themselves (e.g., images, web pages, documents, media files, etc.), or additionally/alternatively, can include references to items (e.g., item identifiers, network addresses from which content items can be retrieved, descriptions of items, thumbnails of items, etc.). An example illustrating the types of data that can be stored in content/tag information data repository 4223 is shown in FIG. 45 and discussed in more detail below.

The content recommendation system 4220 includes a tag identifier subsystem 4221, which is configured to receive user input content from the device 4210 and determine one or more tags to be associated with the user content. In some embodiments, the user content received from the device 4210 may include associated tags. In some other embodiments, the tag identifier 4221 may be configured to process the input data to determine one or more tags to be associated with the input data. As an example, the tag identifier 4221 may use a data tagging service to identify a set of one or more tags to be associated with the input data. The tag identifier 4221 can then provide the tags associated with the input data (including user content in some embodiments) to the recommended content item identifier and ranking subsystem 4224 (which may be referred to as content item ranking device 4224 for brevity) for further processing.

In some embodiments, tag identifier 4221 can use various techniques used by content tagger 4222, and as described above, determine one or more tags to be associated with received user content. In some embodiments, tag identifier 4221 and content tagger 4222 can both use a superset of the same tags from which tags to be associated with user input and content items are determined. In some embodiments, tag identifier 4221 and content tagger 4222 can use the same data tagging service to identify tags to be associated with user content and content items, respectively. In yet other embodiments, tag identifier 4221 and content tagger 4222 subsystems can be implemented as a single subsystem that is configured to perform similar (or even identical) processing on content items received from repository 4230 and input content received from client system 4210.

As described, when content items are tagged by content tagger 4222, for each content item, one or more tags to be associated with the content item and the tag value of each tag are identified. With respect to tagging user content, in some embodiments, tag identifier 4221 is configured to determine only tags to be associated with user input without any associated tag values. In such embodiments, each tag is given an equal weight relative to the ranking performed by content item ranker 4224 based on the tags associated with the user input. In some other embodiments, both tags and associated tag values may be determined for user content and used by content item ranker 4224 to rank content item recommendations.

As described above, the user content received and processed by the tag identifier 4221 can appear in different forms. For example, the user content can include the content of a document authored by a user (e.g., an email, an article, a blog post, a document, a social media post, an image, etc.), content created or selected by a user (e.g., a multimedia file), etc. As another example, the user input can be a document accessed by a user (e.g., a web page). As yet another example, the user content can be a search term entered by a user (e.g., a browser-based search engine) for performing a search. In some embodiments, for example, for the search terms, the terms themselves can be used as tags.

As depicted in Figure 42 and as described above, content item ranker 4224 receives information identifying a set of one or more tags associated with user content from tag identifier 4221 as input. Based on this tag information of the user content and based on the content items that can be used for recommendation, content item ranker 4224 is configured to use tag matching technology to identify one or more content items that are most relevant and/or most related to the input content. In the case where multiple content items are identified as being related or relevant to the user input, content item ranker 4224 is also configured to use the innovative ranking technology described herein to rank the content items. More details related to various technologies used by content item ranker 4224 for scoring and ranking content items are described in more detail below. Content item ranker 4224 is configured to generate a ranking list of content items to be recommended to the user in response to the user input received for the user. The ranking list of content items is then provided to the recommendation selector subsystem 4225 for further processing.

Using the ranked list of content items received from the content item ranker 4224, the recommendation selector 4225 is configured to select one or more specific content items to be recommended to the user in response to the input content received from the client system 4210. In some scenarios, all the content items in the ranked list can be selected for recommendation. In some other scenarios, a subset of the ranked content items can be selected for recommendation, wherein the subset includes less than all the content items in the ranked list, and one or more content items included in the subset are selected based on the ranking of the content items in the ranked list. For example, the recommendation selector 4225 can select the highest ranked "X" (e.g., top 5, top 10, etc.) content items from the ranked list for recommendation, where X is a certain integer less than or equal to the number of ranked items. In some embodiments, the recommendation selector 4225 can select the content items to be included in the subset to be recommended to the user based on the scores associated with the content items in the ranked list. For example, only those content items with an associated score higher than a user-configurable threshold score can be selected to be recommended to the user.

Information identifying the content item selected for recommendation by the recommendation selector 4225 may then be transmitted from the content recommendation system 4220 to the user client device 4210 of the user. Information about the recommended content item may then be output to the user via the user client device. For example, information about the recommendation may be output via a GUI 4215 displayed on the user client device or via an application 4215 executed by the user client device. For example, if the user input corresponds to a search query entered by the user via a web page displayed by a browser executed by the user device, then information about the recommendation may also be output to the user via that web page or an additional web page displayed by the browser. In some embodiments, for each recommended content item, the information output to the user may include information identifying the content item (e.g., text information, thumbnails of images, etc.) and information for accessing the content item. For example, the information for accessing the content item may be in the form of a link (e.g., a URL), which, when the user selects the link (e.g., by a mouse click action), causes the corresponding content item to be accessed and displayed to the user via the user client device. In some embodiments, information identifying a content item and information for accessing the content item may be combined (eg, a thumbnail representation of a recommended image that both identifies the image content item and can be selected by a user to access the image itself).

In various embodiments, the content recommendation system 4220, including its associated hardware/software components 4221-4225 and services, can be implemented as a backend service remote from the front-end client device 4210. The interaction between the client device 4210 and the content recommendation system 4220 can be an Internet-based web browsing session, or a client-server application session, during which user access can enter user content (e.g., search terms, originally created content, etc.) via the client device 4210, and content item recommendations can be received from the content recommendation system 4220. Additionally or alternatively, the content recommendation system 4220 and/or the content repository 4230 and related services can be implemented as a dedicated software component that executes directly on the client device.

In some embodiments, the system 4200 shown in FIG. 42 can be implemented as a multi-layer cloud-based system, where the upper user device 4210 can request and receive access to network-based resources and services via a content recommendation system 4220 residing on a back-end application server deployed and executed on an underlying resource set (e.g., cloud-based, SaaS, IaaS, PaaS, etc.). Some or all of the functionality described herein for the content recommendation system 4220 can be executed or accessed by a representative state transfer (REST) service and/or a web service including a simple object access protocol (SOAP) web service or API and/or web content disclosed via a hypertext transfer protocol (HTTP) or HTTP security protocol. Therefore, although not shown in FIG. 42 to avoid obscuring the components shown with additional details, the computing environment 4200 can include additional client devices, one or more computer networks, one or more firewalls, proxy servers, routers, gateways, load balancers, and/or other intermediate network devices, thereby facilitating interaction between the client device 4210, the content recommendation system 4220, and the content repository 4230.

In various embodiments, the system depicted in computing environment 4200 may be implemented using one or more computing systems and/or networks, including specialized server computers (such as desktop servers, UNIX servers, midrange servers, mainframe computers, rack servers, etc.), server farms, server clusters, distributed servers, or any other suitable arrangement and/or combination of computing hardware. For example, content recommendation system 4220 may run an operating system and/or various other server applications and/or middle-tier applications, including a hypertext transfer protocol (HTTP) server, a file transfer service (FTP) server, a common gateway interface (CGI) server, a Java server, a database server, and other computing systems. Any or all components or subsystems within content recommendation system 4220 may include at least one memory, one or more processing units (e.g., (one or more) processors) and/or storage devices. The subsystems and/or modules in content recommendation system 4220 may be implemented in hardware, software executed on hardware (e.g., program code or instructions executable by a processor), or a combination thereof. In some examples, the software may be stored in a memory (e.g., a non-transitory computer-readable medium), on a memory device, or in some other physical memory, and may be executed by one or more processing units (e.g., one or more processors, one or more processor cores, one or more graphics processing units (GPUs), etc.). The computer executable instructions or firmware implementation of the (one or more) processing units may include computer executable instructions or machine executable instructions written in any suitable programming language, which may perform the various operations, functions, methods, and/or processes described herein. The memory may store program instructions that are loadable and executable on the (one or more) processing units, as well as data generated during the execution of these programs. The memory may be volatile (such as random access memory (RAM)) and/or non-volatile (such as read-only memory (ROM), flash memory, etc.). The memory may be implemented using any type of persistent storage device (such as a computer-readable storage medium). In some examples, the computer-readable storage medium may be configured to protect the computer from electronic communications containing malicious code.

Figure 43 depicts a simplified flowchart 4300 depicting a process for identifying and ranking content items related to user content performed by a content recommendation system according to certain embodiments. The process depicted in Figure 43 may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of a corresponding system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in Figure 43 and described below is intended to be illustrative and not restrictive. Although Figure 43 depicts various processing steps occurring in a particular order or sequence, this is not intended to be limiting. In certain alternative embodiments, the processing may be performed in a different order, or some steps may be performed in parallel. The processing depicted in Figure 43 may be performed by one or more systems depicted in Figure 42, such as by content recommendation system 4220. As an example, for the embodiment depicted in FIG42, the processing in 4302 and 4304 may be performed by tag identifier 4221, the processing in 4306 through 4316 may be performed by content item ranker 4224, and the processing in 4318 and 4320 may be performed by recommendation selector 4225. However, it should be understood that the techniques and functionality described in conjunction with FIG43 are not necessarily limited to implementation within the particular computing infrastructure shown in FIG42, but may be implemented using other compatible computing infrastructures described herein.

In 4302, the content recommendation system 4220 may receive input content from one or more users or client systems 4210. As discussed above with reference to FIG42, the input content may be received from the client device 4210 via a graphical user interface 4215 (e.g., a web-based GUI) provided by the content recommendation system 4220. In other examples, the input content may be received by a web server or backend service executed in the content recommendation system 4220 based on data transmitted by a front-end application (e.g., a mobile application) installed on the client device 4210.

In some embodiments, the input content received in step 4302 may correspond to a set of search terms or phrases entered by a user into a search engine user interface. In other embodiments, the input content may correspond to original content created by a user and entered into a dedicated user interface. For example, new original content may include online articles, newsletters, emails, blog posts, etc., and such content may be entered by a user via a software-based word processor tool, an email client application, a web development tool, etc. In yet other examples, the input content received in step 4302 may be an image, a graphic, an audio input, or any other text and/or multimedia content generated or selected by a user via client device 4310.

Referring briefly to FIG. 44 , an example user interface 4400 is shown that includes a user interface screen 4410 that allows a user to input original creative content. In this example, the user interface screen 4410 is generally labeled as a “content creation user interface,” but in various embodiments, the user interface 4410 may correspond to an interface of a word processor, an article designer or blog author, an email client application, or the like. In this example, the user interface 4410 includes a first text box 4411 in which a user may enter a title or subject of the creative content, and a second text box 4412 in which a user may enter the full text (e.g., an article, an email body, a document, etc.) as input content. In addition, the user interface 4410 includes a selectable button 4413 that allows a user to initiate a search for related content items (e.g., images, related articles, etc.) that may be incorporated into the newly created content. In some cases, the selection of button 4413 or a similar user interface component may initiate the process shown in FIG. 43 by first analyzing and transmitting the user content received via the user interface 4410 (e.g., the user content entered in 4411 and/or 4412) to the content recommendation system 4220. In other embodiments, background processing may run continuously (or periodically) within the front-end user interface to continuously (or periodically) analyze new text input received from the user (e.g., content entered by the user in 4411 and/or 4412) and re-initiate the processing of FIG. 43 in response to text updates so that content item recommendations may be continuously or periodically updated in real time.

Referring back to FIG. 43 , in 4304 , the content recommendation system 4220 determines one or more tags for the input content received in step 4302 . In some embodiments, the input content received in 4302 may already have tags associated with it or embedded in the input content, and the tag identifier 4221 may identify and extract a set of predetermined tags associated with the input content. If the input content received in step 4302 corresponds to a search term, the tag identifier 4221 may simply use the search term entered by the user as a tag (e.g., excluding certain words such as articles, conjunctions, prepositions, quantifiers, etc.). In the case of originally authored text content or any other untagged input content received by the content recommendation system 4220 , the tag identifier 4221 may be configured to analyze various features of the input content received in 4302 and determine one or more tags to be associated with the received input content based on the analysis. As previously described, the tag identifier 4221 may use a variety of different techniques to determine one or more tags to be associated with the input content received in 4302 .

Returning to the example user interface 4400 shown in FIG44 , for this example, the input received in 4302 may correspond to the text entered by the user in the subject/topic box 4411 (i.e., “Is coffee healthier than tea for you?”) and the text entered in box 4412. Based on analysis of the subject content in 4411 and the body text of the article authored in 4411, as well as any other input that may be provided, the content recommendation system 4220 may determine in 4304 that the tags “coffee,” “tea,” and “people” are associated with the user content.

In 4306, based on the tags determined for the input content in 4302, the content recommendation system 4220 identifies a set of matching content items from the set of content items available for recommendation using tag matching techniques, wherein if at least one tag associated with a content item matches the tag determined for the input content in 4304, the content item is considered a match and identified in 4306. The set of content items identified in 4306 may also be referred to as a matching set of content items, and includes content items that are candidates for recommendation to the user in response to the input content received in 4302. In various embodiments, data processing techniques such as stemming and synonym retrieval/comparison may be used as part of the matching process in 4306 to identify matching tags.

For example, in the embodiment depicted in FIG42 , a collection of content items available for recommendation and their associated tag information (e.g., tags associated with the content items and associated tag values) may be stored in content/tag information data repository 4223. As part of the processing in 4306, content recommendation system 4220 may compare the one or more tags determined in 4304 with tags associated with content items available for recommendation and identify content items from the collection that have at least one associated tag that matches the tag identified in 4304.

Continuing with the example of FIG. 44 and assuming that the tags "coffee", "person", and "tea" have been determined in 4304 for the user input content received in 4302, the example table 4500 in FIG. 45 shows a matching set of content items identified by the content recommendation system 4220 (e.g., by the content item sorter 4224) from a set of content items that can be used to make recommendations. As can be seen from table 4500, eight different image content items have been identified as having at least one associated tag that matches at least one of the tags "coffee", "person", or "tea". As shown in this example, an image recognizer 4501 is used to identify each matching image content item. The image description 4502 provided in FIG. 45 is a description of the content of the matched image, and has been provided in FIG. 45 so that the actual image does not have to be shown. Each matched image has one or more associated tags 4503, and a tag value 4504 is associated with each tag. In the example of Table 4500, the tag values are floating point values having a predetermined range (e.g., between 0.0 and 1.0), and the sum of the tag values for each content item equals the same total (e.g., 1.0). Such embodiments provide additional technical advantages, such as uniformity in the sum of the tag count scores, and TVBS can ensure that content items with more associated tags are not artificially over-ranked or over-recommended based on a larger number of associated tags. In addition, as discussed below, by having tag values between 0.0 and 1.0, when they are multiplied (e.g., in the use case described below), the resulting values will allow content items to be ranked within a particular group or bucket while ensuring that the highest ranked content item within one group is not ranked higher than the lowest ranked content item in the next higher group.

As can be seen from Figure 45, if at least one tag associated with a content item matches a tag associated with the input content, then the content item (in this example, an image) is considered a match. A matched content item may have associated tags that match one or more tags associated with the input content. A matched content item may also have other tags associated with it that are different from the tags used for the input content (e.g., Image_1, Image_3, etc.).

At 4308, a tag count score (a first score) is calculated for each matching content item identified in 4306 based on the number of tags associated with the content item that match the tags determined for the input content. In some scenarios, each matching tag of the content item is assigned a value of one, so the tag count score calculated for the content item in 4308 is equal to the number of tags of the content item that match the tags associated with the input content. For the example matching images identified in FIG. 45, table 4600 in FIG. 46 identifies a tag count score 4602 for each matching image identified in FIG. 45 (and FIG. 46). For example, the tag count score for Image_1 is "1" (one) because one tag associated with the image ("person") matches the tag associated with the input content. As another example, the tag count score for Image_2 is "2" (two) because two tags associated with the image ("coffee" and "person") match the image associated with the input content. As yet another example, the label count score for Image_8 is "1" (one) because one label associated with the image ("coffee") matches the label associated with the input content.

At 4310, the matching content items are grouped (bucketed) into groups or buckets based on the tag count scores calculated for the content items at 4308. In some embodiments, a group (or bucket) contains all content items with the same tag count score. Where each matching tag of a content item is assigned a value of one, each group or bucket includes content items with the same number of matching tags. The processing at 4310 may be optional and may not be performed in some embodiments.

Continuing with the example shown in FIG. 46 , the eight matching images can be grouped into two groups or buckets: a first group or bucket that includes content items with a tag count score of 1 (one) and a second group or bucket with a tag count score of 2 (two). The first group would include the images {Image_1, Image_3, Image_6, and Image_8}, and the second group would include the images {Image_2, Image_4, Image_5, and Image_7}. Notably, in this example, no matching images matched all three tags associated with the input content (“coffee”, “person”, “tea”).

At 4312, for each group identified at 4310, a tag value-based score (second score) is calculated for each candidate content item in that group. In some embodiments, the tag value-based score (TVBS) for a particular content item is calculated based on and using the tag values associated with the matching tags for that content item.

In some embodiments, the TVBS for a content item is calculated by multiplying the tag value associated with the tag of the content item, where the tag of the content item matches the tag associated with the input content. For example:

TVBS of Image_1 in Figure 46 = 0.93

TVBS of Image_2 in FIG46 = 0.5*0.5 = 0.25

TVBS of Image_3 in Figure 46 = 0.65

The TVBS of Image_4 in Figure 46 is 0.35*0.60=0.21, and so on.

Table 4600 shows the TVBS 4603 calculated for each matched image using the above technique. In some embodiments, a naive Bayes method is used to calculate the tag value-based score of the content item. For example, assuming that two tags tag ₁ and tag ₂ are determined for the input content, the tag value-based score (TVBS) of the image content item can be expressed as:

TVBS of Image _i = P(Image _i | tag ₁ , tag ₂ ) = the probability of Image _i given tags tag ₁ and tag ₂

Use this extension for 'n' number of tags:

P(Image _i | tag ₁ , tag ₂ , …, tag _n ) = the probability of Image _i given (tags tag ₁ and tag ₂ and … tag _n )

Assume that tags tag ₁ , tag ₂ , ..., tag _n are independent of each other:

TVBS of Image _i = TVBS _i =P (Image _i | tag ₁ , tag ₂ ..., tag _n )

=P(Image _i |tag ₁ )*P(Image _i |tag ₂ )*…*P(Image _n |tag _n )

Now, from the simple naive Bayes Bayes:

in:

P(Image _i |tag _t ) = probability of image _i and tag _t

P(Image _i ) = Each image is considered unique and this term can be discarded or ignored

P(tag _t ) = frequency of the tag in the set of content items (i.e., the number of content items (e.g., images) in the set of content items that are tagged with tag _t and are available for recommendation). Since this term is in the denominator, the less frequent a tag appears in the set of content items (i.e., the fewer content items have this associated tag), the higher the TVBS score of an image with that tag.

After expansion, the above formula becomes:

The numerator in equation 3 above is the product of the probabilities, which will result in a higher score when the probabilities are likely to be equal (the denominator will remain the same given that multiple images match the same label).

The following example illustrates the application of Equation 3 in calculating the TVBS of a content item. Assume that the tags determined for the input content are "person" and "coffee". Further assume that the set of content items (images) available for recommendation contains three images with the following tags and tag values:

Image A: (“person”, 0.5), (“coffee”, 0.5)

Image B: (“person”, 0.1), (“coffee”, 0.9)

Image C: (“person”, 0.8), (“coffee”, 0.2)

For image A:

P(person|image)=0.5 and P(coffee|image)=0.5

Applying Equation 3,

where “Frequency (person)” is the number of content items in the set of content items available for recommendation that have a “person” tag associated therewith, and

TVBS of image A = 0.028

Using a similar technique,

TVBS of image B = (0.1/3)*(0.9/3) = 0.01

TVBS of image C = (0.8/3)*(0.2/3) = 0.018

As can be seen from this example, in the case where the frequencies are the same, when the probabilities are likely to be equal, the TVBS is higher (if the same label is matched for multiple images, the denominator will remain the same).

According to the expansion of equation 3, the frequency of content items in the content item set with specific associated tags is taken into account for calculating TVBS (therefore, it will also affect the overall ranking score, as described below). The fewer the number of content items with specific associated tags (that is, the lower the frequency), the higher the TVBS score of the image with the specific tag. In some embodiments, it is desirable to increase the possibility that such content items with "scarce" or less frequent tags are ranked higher than content items with more frequent tags, thereby increasing the possibility that they are included in the list of content items recommended to the user. Therefore, the value of the TVBS of a content item is inversely proportional to the frequency of occurrence of specific tags that appear in the set of content items (that is, the number of content items with associated specific tags in the set).

At 4314, an overall ranking score is calculated for each matching content item identified at 4306 based on the tag count score calculated for the content item at 4308 and the TVBS calculated for the content item at 4312. In some embodiments, the overall ranking score for a candidate content item may be calculated as the sum of the tag count score (calculated at 4308) and the TVBS (calculated at 4312) calculated for the content item. That is, for an image content item Image _i :

Ranking score (Image _i ) = TagsCountScore _i + TVBS _i

For the example depicted in Figure 46, column 4604 indicates the overall ranking score calculated for each matched image by adding the label count score of that image (indicated in column 4602) and the TVBS of the image (indicated in column 4603) for each matched image. For example, the calculated overall ranking score is as follows:

Image_1: 1+0.93=1.93

Image_2: 2+0.25=2.25

Image_3: 1+0.65=1.65, and so on.

In step 4316, the content recommendation system 4220 (e.g., the content item ranker 4224 in the content recommendation system 4220) generates a ranked list of matching content items based on the overall ranking scores calculated for the content items in 4314. Thus, continuing with the example of FIGS. 44-46 , based on the overall ranking scores calculated for the matching images (in column 4604), the images may be ranked in order from highest to lowest as follows: (1) Image_2, (2) Image_5, (3) Image_4, (4) Image_7, (5) Image_6, (6) Image_1, (7) Image_8, (8) Image_3.

In certain embodiments where the tag value of a tag is within a range between 0.0 and 1.0, using the (TagCountScore+TVBS) method to calculate the overall ranking score ensures that content items with a higher number of associated tags that match tags associated with the input content are ranked higher than content items with a lower number of tag matches. For example, in the example of FIG. 46 , an image with a tag count score of 2 (corresponding to two tags associated with the image that match tags associated with the input content) will always have an overall ranking score and therefore be ranked higher than an image with a tag count score of 1 (corresponding to two tags associated with the image that match tags associated with the input content). This is because, given a tag value between zero and one, the TVBS of an image calculated by multiplying the tag values associated with the matched tags cannot exceed one. This also means that for a first group or bucket of content items corresponding to a first tag count score and a second group or bucket of content items corresponding to a second tag count score, if the first tag count score is higher than the second tag count score, then each content item in the first group will be ranked higher (due to the higher overall ranking score) than the content items in the second group. Therefore, in the example where three tags ("coffee", "people", and "tea") are determined for the input content, a content item with three content tags that match the tags of the input content will always be ranked before a content item with two matching content tags, and each content item with two matching content tags will always be ranked before a content item with one matching content tag, and so on. Within each group or bucket, the content items can be ranked based on their TVBS, which is conducive to higher and more equal parameters of the matching content tags. However, it should be understood that in other embodiments, in order to implement different content item ranking priorities and strategies, different equations or logic can be used to calculate the tag count score, TVBS, and overall ranking score.

In 4318, the content recommendation system 4220 (e.g., recommendation selector 4225) can use the ranking list generated in 4316 to select one or more content items to be recommended to the user. In some scenarios, all content items in the ranking list can be selected for recommendation. In some other scenarios, a subset of ranked content items can be selected for recommendation, wherein the subset includes less than all content items in the ranking list, and one or more content items included in the subset are selected based on the ranking of the content items in the ranking list. For example, the recommendation selector 4225 can select the highest ranking "X" (e.g., top 5, top 10, etc.) content items from the ranking list for recommendation, where X is a certain integer less than or equal to the number of ranked items in the list. In some embodiments, the recommendation selector 4225 can select the content items to be included in the subset to be recommended to the user based on the overall ranking score associated with the content items in the ranking list. For example, only those content items with an associated score higher than a user-configurable threshold score can be selected to be recommended to the user.

In 4320, the content recommendation system 4220 may transmit information related to the content item selected in 4318 to the user device. Since this information includes information related to the content item to be recommended to the user, this information may be referred to as recommendation information. The recommendation information transmitted in 4320 may also include ranking information (e.g., an overall ranking score associated with the selected content item). This information may be used on the user device to determine how information (e.g., order) about the recommended content items is displayed to the user via the user device. In some embodiments, as part of the recommendation information, the recommendation selector 4225 may transmit the content item itself, or certain information identifying the content item (e.g., a content item identifier and description, a thumbnail, a network path or download link, etc.) to the client device 4310 from which the input content is received in step 4302.

The information about the selected recommendation can then be output to the user via the user device. For example, the information about the recommendation can be output via a GUI 4215 displayed on a user client device or via an application 4215 executed by the user client device. For example, if the user input corresponds to a search query entered by the user via a web page displayed by a browser executed by the user device, the information about the recommendation can also be output to the user via a web page showing the search results or an additional web page displayed by the browser. In some embodiments, for each recommended content item, the information output to the user may include information identifying the content item (e.g., text information, thumbnails of images, etc.) and information for accessing the content item. For example, the information for accessing the content item can take the form of a link (e.g., a URL), and when the user selects the link (e.g., by a mouse click action), the link will cause the corresponding content item to be accessed and displayed to the user via the user client device. In some embodiments, the information identifying the content item and the information for accessing the content item (e.g., a thumbnail representation of a recommended image, which both identifies the image content item and can be selected by the user to access the image itself) can be combined.

For example, referring to FIG. 47 , an example user interface 4700 is shown, which corresponds to an update of the user interface screen 4400 from FIG. 44 that displays information related to recommended images. In this example, based on the title/subject 4711, the body text 4712, and/or any other input content, the content recommendation system 4220 has selected four top-ranked content item images from the ranked list of images to recommend to the user. Information related to these four top-ranked images is displayed in the order of ranking within a dedicated portion 4714 of the user interface 4700 to show content item recommendations. In some embodiments, thumbnail representations of the recommended images may be displayed in 4714. The user interface 4700 may support drag-and-drop functionality or other techniques to allow the user to incorporate one or more suggested images displayed in 4714 into the body 4712 of the created content.

The processing described in Figure 43 and described above is not intended to be restrictive. Various changes can be provided in different embodiments.For example, for the embodiment described in Figure 43 and described above, the processing in 4312,4314 and 4316 is performed for the content items of all matches identified in 4306.In some changes, the label count score calculated in 4308 can be used to filter out some content items from further processing.For example, when the content items have different label count scores, the content items with the lowest label count score (or some other threshold value) can be filtered out from the further processing in the flow chart.In some other embodiments, those content items with only the highest label count score can be selected for further processing, thereby filtering out other content items from further processing.For example, the content item ranker 4224 can only calculate the TVBS (as determined in step 4304) of the highest label score group, or can calculate the TVBS according to the order from the highest to the lowest label count score group, during this period, the calculation process can stop when reaching the candidate content items of threshold value number or the threshold label count score. Such filtering reduces the number of content items to be processed and may enable the overall recommendation operation to be performed faster and more efficiently using fewer processing resources (eg, processor, memory, network resources).

In the method depicted in FIG. 43 and described above, each tag associated with or determined for the input content in 4304 is assigned an equal weight relative to the ranking performed by the content recommendation system 4220. Based on this assumption, the tag score for each matching content item is determined as the number of its content tags that match the tags of the input content. Thus, relative to the determination of the tag count score, each matching content tag associated with the matching content item is equally evaluated/weighted. However, in other embodiments, different weights may be assigned to tags associated with the input content. For example, for two tags determined for the input content, by assigning a higher weight to one tag than another tag, one tag may be indicated as "more important" for the input content. For example, for the example depicted in FIGS. 44-47 and described above, in which the tags "people", "coffee", and "tea" are determined for the input content, instead of assigning equal importance to the three tags, the weights of the tags are as follows: people = 1, coffee = 2, and tea = 4. This weight may indicate the relative importance of the tags to the input content, for example, "tea" is more important than "coffee", and "coffee" is more important than "people". In some embodiments, the logic used to calculate the tag count score for each content item can be modified to take into account the different weights assigned to the tags of the input content. According to one such modified logic, the contribution of each matching tag for the content item is multiplied by the weight associated with the same tag for the input content. For example, using the (person=1, coffee=2, tea=4) weights for the input content, the tag count scores for the matched images in FIG. 45 would be as follows:

Image_1:

-TagCountScore (unweighted) = "people" tag match = 1

-TagCountScore (weighted) = "people" tag matching = 1*1 = 1

Image_2:

-TagCountScore (unweighted) = "coffee" and "people" tag matches = 1+1 = 2

-TagCountScore (weighted) = "coffee" and "people" tag matches = 2 (1) + 1 (1) = 3

Image_3:

-TagCountScore (unweighted) = "tea" tag match = 1

-TagCountScore (weighted) = "tea" tag matches = 4 (1) = 4

Image_4:

-TagCountScore (unweighted) = "coffee" and "people" tag matches = 1+1 = 2

-TagCountScore (weighted) = "coffee" and "people" tag matches = 2 (1) + 1 (1) = 3

Image_5:

-TagCountScore (unweighted) = "tea" and "person" tag matches = 1+1 = 2

-TagCountScore (weighted) = "tea" and "person" tag matches = 4 (1) + 1 (1) = 5

Image_6:

-TagCountScore (unweighted) = "coffee" tag match = 1

-TagCountScore (weighted) = "coffee" tag matches = 2 (1) = 2

Image_7:

-TagCountScore (unweighted) = "tea" and "person" tag matches = 1+1 = 2

-TagCountScore (weighted) = "tea" and "person" tag matches = 4 (1) + 1 (1) = 5

Image_8:

-TagCountScore (unweighted) = "coffee" tag match = 1

-TagCountScore (weighted) = "coffee" tag matches = 2 (1) = 2

Because the tag count scores are different, the grouping or bucketing of content items performed in 4310 will be different. Thus, content items with content tags that match both the "tea" and "person" tags will be grouped together and assigned a tag score of 5, while content items with only one content tag that matches "tea" will be grouped together and assigned a tag score of 4, and content items with content tags that match both the "coffee" and "person" tags will be grouped together and assigned a tag score of 3, and so on. Thus, in such embodiments, the overall ranking of candidate content items is affected not only by how many tags of the content items match the tags of the input content, but also by which specific tags of the content items match the input content tags and the weights assigned to the relative importance of those tags. In this example, images Image_5 and Image_7 will become the highest ranked overall content items based on their highest tag count scores of 5 (tea=4+person=1).

Smart Content - Smart Classification/Category

Categorizing large amounts of content in an online manner is a complex task involving challenges such as single pass constraints on data and rapid response requirements. According to an embodiment, content users categorize similar content through logical clustering such as a hierarchical classification system tree and place similar content in the same node/category of the classification system tree. Over time, as the number of content entities and nodes in the classification system tree grows, similar content entities will find themselves residing in nodes side by side with each other. Given this state of content organization, a computer algorithm (such as a classification engine) can use content that resides in an already evaluated/classified classification system to determine where newly created/edited content may belong.

According to an embodiment, the systems and methods described herein may be used, for example, with a content management system to provide recommendations for categorizing/classifying content into user-defined categories, which in turn provides content managers with the opportunity to easily place new content into accurate categories based on previously evaluated/classified content.

The recommendation system or tool may use artificial intelligence (AI) techniques to continually learn from past data and/or new input results from generated recommendations, and help place content into relevant categories through automatic categorization/classification of newly created/edited content.

By generating feature vectors from content, creating clusters in feature space based on previously categorized content, and recommending categories for new content by computing feature space distances from clusters, recommendation tools can be implemented and applied across different domains.

FIG. 48 illustrates an example use of a content management system environment according to an embodiment.

More specifically, FIG. 48 illustrates an exemplary content management system that may include a categorization engine for intelligently categorizing content at the content management system.

As shown in FIG. 48 , according to an embodiment, for each of a plurality of client devices 4800 , 4802 , and 4804 having a user interface 4801 , 4803 , 4805 and physical device hardware 4806 , 4807 , 4808 (e.g., CPU, memory), the client device may be provided with a content access application 4810 , 4811 , 4812 for execution thereon.

According to an embodiment, the client device may communicate 4862 with an application server 4830 , which includes physical computer hardware 4831 (eg, CPU, memory) and a content management system 4832 .

According to an embodiment, the content access application at the client device can communicate with the content management system via a network 4860 (e.g., the Internet or a cloud environment). The content access application can be configured to enable users 4850, 4852, 4854 to view, upload, modify, delete, or otherwise access content, such as, for example, content items 4820, 4822, 4824, at each client device. For example, new content can be added or uploaded to the content management system by a user interacting with the content access application on the associated client device. The content can be transferred to the content management system, for example, for tagging and storage.

According to an embodiment, a content management system may be or include a platform for integrating content that may be managed by multiple users or clients. According to an embodiment, a content management system may be configured to communicate with a content repository 4836 for storing content (or content items) 4840, and may deliver content to users via their client devices. According to an embodiment, a content repository may be a relational database management system (RDBMS), a file system, or other data storage repositories that a content management system may access. Content may include, for example, documents, files, emails, memos, images, videos, slideshows, conversations, and user profiles.

According to an embodiment, the content management system can be configured to associate metadata with the content. The metadata can include information about the content item, such as, for example, its title, author, publication date, historical data, such as, for example, who accessed the item and when, the storage location of the content, etc.

According to an embodiment, the metadata may be stored in a metadata database 4838. According to an embodiment, the content management system may be configured to communicate with the metadata database to access metadata stored therein, and to store metadata generated by the system in the metadata database.

According to an embodiment, the content management system may also be configured to communicate with a search index 4839. The search index may be configured to provide indexing and searching of content and data stored in the content repository and metadata database. According to an embodiment, the search index may be a relational database management system (RDBMS) or a search tool such as, for example, Oracle Secure Enterprise Search (Oracle SES).

According to an embodiment, the content management system may further include a content management application 4833 and a categorization engine 4834. The categorization engine may include an artificial intelligence/machine learning engine and library 4835, and a user interface 4836, which may be used, for example, to display output indicating content categorization recommendations and/or receive input indicating content categorization selections.

According to an embodiment, a classification engine can be used, for example, with a content management system to provide recommendations for categorizing/classifying content into defined (e.g., user-defined) categories, which in turn provides content managers with the opportunity to easily place new content into accurate categories based on previously evaluated/classified content. The classification engine can use artificial intelligence (AI) techniques and, for example, machine learning libraries to continuously learn from past data and assist in placing content into relevant categories by automatically categorizing/classifying newly created/edited content. Recommendation tools can be implemented and applied across different domains by generating feature vectors from content, creating clusters in feature space based on previously categorized content, and recommending categories for new content by calculating feature space distances from clusters. Additionally or alternatively, the engine can automatically categorize content via its calculations.

In general, according to an embodiment, the classification engine can be used to create new classification systems, modify existing classification systems, and/or categorize/classify existing and/or new content in batches based on various recommendations or suggestions associated with various confidence scores or confidence levels (such as high, medium, or low confidence).

For example, according to an embodiment, the system or user may ignore low confidence recommendations for categories/categories associated with a particular set of content, but may accept high/medium confidence recommendations for categories.

According to an embodiment, the categorization engine may display such recommendations or suggestions of categories/classifications associated with a particular set of content in a user interface for review and/or action by a content administrator.

For example, when processing a new collection of content, the system may suggest or recommend one or more categories/classifications for the new content based on previous categorization of various content, and the content administrator may then choose to accept or decline the assignment of these categories/classifications to the content.

According to an embodiment, such acceptance or rejection of the categorization suggestion by the user or content administrator may be stored in a database (such as a database of a content categorization engine). Such historical records may be used to improve future recommendations for content categorization.

According to an embodiment, classification can be performed as a two-stage process, including: a macro-level classification stage, in which the topic distribution and named entity distribution of the document can be matched with higher-level cluster nodes; and a micro-level classification stage, in which categories can be expanded and compared in feature space based on the evaluation of micro-clusters.

FIG. 49 illustrates an example use of a content management system for management and delivery of content data according to an embodiment.

According to an embodiment, a content management system (CMS) allows users to collaboratively manage digital content, including web content. An example content management system is Oracle Content Management (OCM), which can be provided and accessed via a cloud service. The content management system provides various features, such as, for example, storing, managing, and publishing associated digital content. Systems such as, for example, OCM allow for rapid development and publishing of content across multiple delivery channels.

According to an embodiment, a delivery channel can be any form of delivering content to consumers of the content. Exemplary delivery channels include websites, blogs, HTML emails, storyboards, mobile applications, etc. In order to provide rapid development of documents used by associated delivery channels, the system can include features such as, for example, out-of-the-box templates, drag-and-drop components, sample page layouts, and site themes that allow users to assemble content into publishable documents from predefined building blocks. The content management system can use these components to generate the markup language and code (collectively referred to herein as "code") of the document.

According to an embodiment, a server application program interface (API) contract (management API contract) can be provided that establishes interaction between a content management system and a business provider system (business provider). The server API contract operates as a contract between the content management system and the business provider (e.g., an online retailer) and enables communication and data exchange between the content management system and the business provider. According to an embodiment, based on interaction with an interactive feature, data and other metrics associated with a product can be retrieved from the business provider; or data associated with a request can be sent to the business provider to perform an action associated with a product.

According to an embodiment, a business provider system 4910 may be provided in the system to manage and deliver content data. The business provider system may include a product catalog 4911, an API, such as, for example, a server API 4912, and physical computer resources, such as, for example, a CPU, a memory 4913.

According to an embodiment, the content management system 4920 may include a content metadata database 4921 (e.g., an OCM content and metadata database), an API such as, for example, a server API 4922, and physical computer resources such as, for example, a CPU, a memory 4923.

According to an embodiment, a content administrator 4935 may interact with a management system 4930 in such a system to create/manage and deliver content data 4903. The management system may include or provide a user interface 4931, which may include or provide a content mapping configuration 4932. The management system may also include physical computer resources such as, for example, a CPU, memory 4933.

According to an embodiment, the server API contract (management API contract) 49049 can operate as a contract between a content management system and a business provider. For example, the server API contract can be configured to receive account information and credentials associated with a user account at a business provider.

According to an embodiment, the content management system can use the received account credentials to authenticate an administrative user (e.g., a content administrator) of the content management system at the business provider system via a server API (administration API); it provides the content management system with access to data stored at the business provider system. For example, the content management system can request the business provider via the server API to send data describing/defining a set of products offered for sale by the business provider. The business provider system can then send such data to the content management system via the server API. The data can include, for example, a list of products and data describing the list of products.

According to an embodiment, the content management system can publish content 4904 in multiple different channels, such as, for example, via a generated mobile application page 4905, or via a generated web page 4906.

According to an embodiment, end user 4945 can access such generated pages via client device 4940, for example via mobile application 4941 that displays generated mobile application page 4905, or via a web application (such as, for example, browser 4942) that accesses and displays generated web page 4906.

In accordance with an embodiment, publishable content (e.g., a web page or mobile application (app) page) may be built using a code library, such as, for example, a software development kit (SDK) 4907, which defines functional and visual content components, such as, for example, a "Buy" button.

Macro/micro classification processing

According to an embodiment, classification can be performed as a two-stage process, including: a macro-level classification stage, in which the topic distribution and named entity distribution of the document can be matched with higher-level cluster nodes; and a micro-level classification stage, in which categories can be expanded and compared in feature space based on the evaluation of micro-clusters.

FIG. 50 illustrates a flow chart of intelligent content classification according to an embodiment.

As shown in FIG. 50 , automatic classification (also referred to herein as “auto-classification”) of content into a known classification system (such as a domain-specific ontology) can be accomplished via an AI/ML system.

When new content is created 5000 (i.e., new content at a content management system, such as content generated therein, or content uploaded to a content management system), an engine such as a categorization engine as described above may automatically suggest 5010 during a prediction phase 5015 a category or categories into which the newly created content should be categorized.

According to an embodiment, after receiving one or more selections for categorizing the content, the content may be placed 5020 into such one or more selected categories.

Based on such selection, during the learning phase 5035, the clustering (eg, the clustering stored in a memory or library of the classification engine) may be updated 5030 accordingly. Thus, the prediction phase 5015 may be updated based on another newly created content 5000.

In other systems, such categorization is typically accomplished by analyzing large amounts of similar data in the public domain. However, according to an embodiment, the described automatic categorization of content in a user-defined category hierarchy requires a new approach; one that continuously observes previously assessed/categorized content, category metadata, and user behavior in response to categorization/classification suggestions.

According to an embodiment, unrated/categorized or refined rated/categorized content that enters a category hierarchy presents a special challenge because, on a cold start, the system will have very little information to go on to make recommendations. Over time, businesses using content management systems evolve, making old category hierarchies less relevant or even obsolete.

According to an embodiment, described herein is a system and tool for automatically classifying various content into categories based on clustering of related concepts using a micro-clustering scheme. This scheme outperforms classical clustering schemes. In addition, the tool can learn from user behavior and adapt itself over time by generating substantial productivity improvements for content authors. According to an embodiment, the system includes the following set of use cases:

Creating a new classification system

According to an embodiment, the disclosed systems and methods may be used to create a new taxonomy by, for example:

a. Receiving instructions instructing a user to create a taxonomy. In some embodiments, the taxonomy may include a hierarchical taxonomy structure, each node of which is a category.

b. Receiving instructions instructing a user to categorize/classify at least a portion of existing content items (e.g., content items already within the content management system) into appropriate categories by placing at least a portion of the existing content items (e.g., one at a time or in batches) into these taxonomies.

c. Receiving new content (eg, uploaded or created within a content management system).

d. The system automatically gets notified and starts learning from the above actions.

e. The system starts recommending categories that newly created content and existing uncategorized content may belong to (for individual content or in batches).

f. The user accepts or rejects the recommendations (one at a time or in batches), and the user action informs the system to learn from it and make better recommendations.

FIG. 51 illustrates a classification system creation flow chart according to an embodiment.

According to an embodiment, at step 5100, as described above, the system may receive instructions instructing a user to create one or more classification systems. When creating a classification system, the instructions may also include categories under the classification system. In some embodiments, the classification system may include a hierarchical classification structure, each node of which is a category.

According to an embodiment, at step 5105, the system may receive instructions instructing the user to categorize/classify at least a portion of existing content items (e.g., content items already in the content management system) into these classification systems by placing at least a portion of the existing content items (e.g., one at a time or in batches) into appropriate categories. This may also include categorization/classification of new content (e.g., content newly uploaded to the content management system). New content does not necessarily get categorized or classified.

According to an embodiment, at step 5110, the system is automatically notified and begins to learn from the above actions. The system begins to recommend categories to which newly created content and existing unclassified content may belong. This can be performed for individual/single content, or such suggestions can be performed for batch content classification.

Depending on the embodiment, at step 5115, the system may receive instructions instructing the user to accept or reject the generated suggestions (e.g., one at a time or in batches).

In this way, instructions indicating user actions inform the system to learn from and make better recommendations according to the embodiment at step 5120. Based on ML and AI, the system can refine the recommendations based on the recorded acceptance or rejection of the categorization suggestions.

Modifications within existing classification architecture

According to an embodiment, modifications within an existing classification architecture may be created using the following process.

a. Users create new categories or change the structure of the classification system or categorize content.

b. The system generates suggestions for existing unrated/categorized or newly added content (one at a time or in batches), including any newly added categories.

c. The user accepts or rejects the recommendations (one at a time or in batches), and the user's actions inform the system to make better recommendations.

FIG. 52 illustrates a classification system modification flow chart according to an embodiment.

According to an embodiment, at step 5200, as described above, the system may receive instructions instructing the user to add a new category or modify one or more existing classification systems. The system may then receive instructions instructing the user to classify existing content or new content into the newly created category (e.g., individually or in bulk).

According to an embodiment, at step 5205, the system may automatically begin generating suggestions and recommendations for categorizing new content as well as existing content including the newly created category.

Depending on the embodiment, at step 5210, the system may receive instructions instructing the user to accept or reject the generated suggestions (e.g., one at a time or in batches).

In this way, the instructions indicating the user's actions inform the system to learn from them and make better recommendations, according to the embodiment at step 5215. Based on ML and AI, the system can refine the recommendations based on the recorded acceptance or rejection of the categorization suggestions.

Bulk content classification

According to an embodiment, a batch content taxonomy may be created using the following process.

a. The system enables users to categorize/classify content collections into suggested categories based on confidence scores.

b. For categories, the system can place recommended content into three disjoint buckets (high, medium, low) based on the predicted confidence score for each content item.

c. Confidence score buckets provide a mechanism to group content item collections so that batch classification is easier to implement. For example, a user can accept all content classification suggestions for content in high confidence score buckets and reject all content classification suggestions for other content in low confidence score buckets.

d. In addition, users can skip the acceptance/rejection of recommendations through the global classification threshold configuration. If selected, the system can automatically assign content to categories with recommendation confidence scores above the user-configured threshold.

e. Typically, recommendations are generated through backend jobs that run automatically on a regular basis. In addition, the system allows repository administrators to manually trigger jobs, causing the recommendation system to immediately generate recommendations or categorize content into a taxonomy assigned to the repository.

FIG53 illustrates a sample taxonomy tree with a category diagram, under an embodiment.

As shown in Figure 53, according to an embodiment, as shown in the screenshot of the user interface 5300: (i) a sample taxonomy tree with categories, "View Category Suggestions" 5305 can take the user to the smart suggestion screen. The user interface 5300 is designed to help categorize content individually or in bulk.

According to an embodiment, each category in the screen displays the number of suggestions next to the category. For example, content 5315 may be displayed with multiple suggested categorizations 5320. Content suggestions are divided into three buckets (high, medium, low) based on the confidence score of that content to be placed in the category.

FIG. 54 further illustrates a sample taxonomy tree with a category diagram, according to an embodiment.

As shown in Figure 54, according to an embodiment, as shown in the screenshot of the user interface 5400: (i) a sample taxonomy tree with categories, "View Category Suggestions" can take the user to the smart suggestion screen. The user interface 5400 is designed to help categorize content individually or in batches.

54, according to an embodiment, in this example, content with high confidence 5405 and medium confidence 5410 is selected. An "assign to category" link 5415 enables the user to place all selected content into corresponding category(s).

According to an embodiment, as described above, based on the acceptance of suggestions that fall into high and medium confidence suggestions, the categorization engine may further refine subsequent categorization suggestions.

FIG. 55 further illustrates a sample taxonomy tree with a category diagram, according to an embodiment.

As shown in Figure 55, according to an embodiment, as shown in the screenshot of the user interface 5500: (i) a sample taxonomy tree with categories, "View Category Suggestions" can take the user to the smart suggestion screen. The user interface 5500 is designed to help categorize content individually or in batches.

As shown in Figure 55, according to an embodiment, again, the user can select content (in this example, select a low confidence 5505 score item). The user can then select the option to reject 5510 all low confidence score classification suggestions.

According to an embodiment, as described above, based on the rejection of suggestions that fall within the low confidence suggestions, the categorization engine may further refine subsequent categorization suggestions.

Figure 56 illustrates a configuration auto-classification threshold map according to an embodiment.

As shown in Figure 56, according to an embodiment, as shown in a screenshot of a user interface 5600. The user interface 5600 is designed to help/allow a user to set an automatic classification threshold 5605 for a confidence score.

According to an embodiment, in other words, when the system and method are configured to allow content items to be automatically classified or categorized, a threshold value can be set so that any content item with a confidence score exceeding the set threshold value can be automatically classified/categorized by the system without further user interaction. As shown in the figure, the configuration setting of the classification system level for controlling the automatic classification setting is between "medium" and "high" confidence scores. This means that content with a confidence score higher than this threshold value will be categorized/categorized into corresponding categories. In addition, those content items with confidence scores lower than the set threshold value can have the categorization/categorization option of being automatically discarded, or additionally or alternatively, these content items are provided to the user for input to make a decision.

FIG. 57 illustrates a configuration for triggering reclassification (in bulk) of content in a repository graph, according to an embodiment.

As shown in Figure 57, according to an embodiment, as shown in a screenshot of a user interface 5700. The user interface 5700 is designed to help/allow a user to trigger batch classification of documents within a target repository 5705, for example.

According to an embodiment, for example, the screenshots show that content items within an existing repository (e.g., at a content management system) can be reclassified via a background process. The background process can be run to generate new categorization/classification suggestions to be submitted to the user for decision. Alternatively, for example, the background process can be run to automatically classify content items based on, for example, a calculated confidence score.

Concept and solution

In various cases, classical clustering schemes do not work well from a content management system perspective, as it is difficult to categorize/classify content into the correct categories for a number of reasons, including but not limited to:

a. The amount of content in a content management system will grow dramatically over time;

b. Content may come from various departments, and domain-specific solutions may not work;

c. The categories/nodes in the classification system tree are not restricted, and new categories are frequently introduced;

d. The number of contents in a category can range from single digits to thousands, and the same content can be assigned to multiple categories;

e. The contents present in a category may not be semantically close to each other, and there may be multiple disjoint subsets of similar contents sharing the same category;

f. Considering the large amount of data, the nearest neighbor solution may be accurate but not scalable;

g. Due to the uneven distribution of content to categories and the lack of labeled data, traditional classification algorithms may not work in this scenario;

h. Traditional clustering schemes may not be applicable because clusters can grow to arbitrary sizes and shapes as more and more content is added, and reconstructing clusters requires passing the entire data points (content) present in the cluster, which is not allowed in terms of scalability.

FIG. 58 illustrates the problem of traditional clustering where clusters can be of arbitrary shapes, according to an embodiment.

According to an embodiment, data point N 5805 is shown in a cluster diagram that includes cluster 1 5800 and cluster 2 5810. From the depicted embodiment, it is clear that data point N 5805 should be part of cluster 1. However, in reality, the distance from N to the center of cluster 2 is actually less than the distance from N to the center of cluster 1. This means that cluster 2 appears to be more accurate.

According to an embodiment, taking into account accuracy and scalability, the described scheme provides a universal density-based single-pass micro-clustering that can discover clusters of any arbitrary shape (in terms of accuracy), reshape clusters without looking at past data (in terms of scalability), continuously learn and evolve from recent data, and correctly predict the categories of new content in batches, which in turn reduces the time and effort of classifying content into the correct category.

Adaptive Micro-Clustering

According to an embodiment, the described scheme includes two parts or aspects: (1) extracting features from documents and creating clusters - the learning phase; and (2) automatically classifying new content - the prediction phase.

1. Extract features from documents

According to an embodiment, extracting features from a document includes creating and updating clusters (micro clusters). In this step, the better the clusters are represented, the better the accuracy of the system. As described in Figure 59, a single cluster center alone may not represent the entire cluster well, and micro clusters can be additionally formed within the main cluster.

Figure 59 illustrates micro clustering according to an embodiment. In the same example as Figure 58 above, the larger cluster, Cluster 1 in Figure 58, is now represented by 3 points (e.g., micro clusters): Cluster 1 5900, Cluster 1'5901, and Cluster 1" 5902. At this time, the new data point N 5905 will be classified as belonging to Cluster 1 because the closest point to N is Cluster 1'5901, not Cluster 25910.

According to an embodiment, there are multiple steps to be performed to update clusters and split micro-clusters within clusters, as described below:

A. Generating feature vectors from text

According to an embodiment, the first step is to convert the original text document into a feature vector for further processing. The feature space of the text fragment is generated by a topic modeling system, which outputs sparse text features (e.g., 200k dimensions). Subsequently, the text features are projected to a dense feature space of lower dimensions (2048) using a random projection technique for rapid processing. When the content is sparse, the scheme works using the content model itself, such as name and description and other metadata for defining the classification system itself.

B. Extracting named entities from text

According to an embodiment, a named entity recognizer (NER) system performs the task of automatically extracting entities, such as names of people, organizations, customer products, countries, book titles, or music albums, among others.

According to an embodiment, through named entity recognition, the system can obtain key information from unstructured text and classify them into user-defined categories. For example, consider a document scenario from the pharmaceutical industry, where drug names, chemical elements are often mentioned; on the other hand, publisher documents contain book names, authors, fictional characters, etc. Such documents can be classified into their respective fields just by looking at the entities. Considering the named entity scores of documents with respect to the distribution of named entities in user-defined categories can make the classification task more efficient, scalable, and accurate.

C. Extracting topics from text

According to an embodiment, topic extraction automatically discovers keywords and key phrases from documents by identifying term frequencies and grouping similar word patterns. A user-defined taxonomy tree can be enlarged horizontally and vertically, and a statistical topic extraction model can be used for macro-level classification to select higher-order nodes in the tree, followed by feature-based micro-classifiers while suggesting categories for documents.

For example, an apparel customer might have a taxonomy tree hierarchy like “Women > Clothing > Ethnic > Sarees > Cotton > Chanderi” and “Home Linen > Bedding > Bed Cover > Cotton > Haiba” In this example, it is clear that to decide the macro level category, there is no need to compare each document at the leaf level (all documents belong to women’s clothing or bed linen), instead by looking at the topic distribution of articles in the first 3-4 nodes, the high level category can be selected and then the micro level (feature comparison) classifier can help suggest the type of saree (like chanderi or banarasi) or bed cover (haiba).

FIG. 60 illustrates a sample topic distribution for apparel customers, under an embodiment.

According to an embodiment, as shown, the topic distribution includes, for example, an apparel cluster 6001, a home linen cluster 6020, and a home decor cluster 6030. Example data points are shown around each cluster.

D. Creation of clusters

According to an embodiment, each category (node in a classification system tree) is assigned a cluster with it; wherein the cluster may include: a summarized feature space representation of feature vectors associated with the content present in the cluster, the distribution of named entities present in the documents in the cluster, and the distribution of topics extracted from the documents in the cluster.

According to an embodiment, clusters are created along with categories (initialized with feature vectors, named entities and topics extracted from category metadata).

E. Definition of Clustering

According to an embodiment, clusters may be defined using the following process. Cluster characteristics: According to an embodiment, clusters are defined based on centroid and radius. Centroid = mean of feature vectors of content present in the cluster, radius = average distance of member feature vectors to the centroid of the cluster. A smaller radius indicates that the content present in the cluster is closely related.

According to an embodiment, a list of named entities is attached to the clusters along with the frequencies. A distribution is created by selecting the top n named entities from each document belonging to the cluster.

According to an embodiment, similar to named entities, a topic distribution can be defined that identifies the most frequent topics attached to clusters. The topic distribution is also created by extracting the most relevant topics from each document.

FIG. 61 illustrates visualization of cluster radii, according to an embodiment.

According to an embodiment, Figure 61 depicts two exemplary clusters, Cluster 1 6100 and Cluster 2 6110. As shown, Cluster 1 has a smaller radius than Cluster 2. As described above, a smaller radius indicates that the content present in the cluster is closely related.

F. Update clustering features

According to an embodiment, as shown in FIG61 , clusters with high density and small radius (left) can achieve the best possible accuracy. To achieve this goal, as more and more content (e.g., new content) is added, clusters are split into micro-clusters. If new content causes the radius of a micro-cluster to expand, they do not need to be added to the micro-cluster immediately, but can only be identified as potential micro-clusters.

According to an embodiment, later, as the content grows, as more and more similar content is added, potential micro-clusters can grow into core micro-clusters, or they can be ignored as outliers through a damping factor over time. This micro-clustering scheme helps find disjoint subsets of related content in a category.

Cluster centroids and radii can be updated in an online manner (without looking at past data).

FIG. 62 illustrates a cluster representation according to an embodiment.

According to an embodiment, Figure 62 depicts three exemplary clusters, cluster 1 6200, cluster 2 6210, and cluster 3 6230. As shown, each cluster has a different radius, and each radius reflects the degree of closeness of the content items in each cluster. As described above, a smaller radius indicates that the content present in the cluster is closely related.

According to an embodiment, as content grows, potential micro-clusters can grow into core micro-clusters as more and more similar content is added, or they can be ignored as outliers over time through a damping factor, such as O1 6240 and O2 6241. This micro-clustering scheme helps find disjoint subsets of related content in a category.

According to an embodiment, when a new document D is placed in a cluster, the associated named entity distribution and topic distribution are updated for the cluster by extracting topics and named entities from D.

2. Categorize new content into relevant categories

According to an embodiment, classification is a two-stage process, as described below, including a macro-level classification stage and a micro-level classification stage:

A. Macro-level classification

According to an embodiment, as described above, each category node has a topic distribution and a named entity distribution associated therewith.When a document is uploaded, topics and named entities are first extracted from the document, and conditional probability scores are calculated for each high-level category in the taxonomy tree.

Figure 63 illustrates a macroscopic stage of classification according to an embodiment. More specifically, Figure 63 depicts an embodiment in which the topic distribution and named entity distribution of a document are matched with high-level cluster nodes, namely nodes 6300-6309.

According to an embodiment, the score function can be viewed as given a list of topics, e.g., "shirts", "cotton", etc. (and the weight of each topic for the document), what is the probability that the document will belong to the "women's clothing" category. Given a document D, which has n topics (e.g., topic-1, topic-2, ..., topic-n) and a weight for each topic (weight _topic-1 , weight _topic-2 ..., weight _topic-n ), the joint probability of the document with respect to category Catg-C can be calculated as:

Score(D|Catg-C)=(P(topic-1|Catg-C)*weight _topic-1 )

+(P(topic-2|Catg-C)*weight _topic-2 )

+...+(P(topic-n|Category-C)*weight _topic-n )

B. Micro-level classification

According to an embodiment, during this phase, feature comparisons are performed in order to classify the documents at a finer level, where each document exists in a child node of a macro-selected category.

Figure 64 illustrates the classification once a higher level category is selected through the macro step. According to an embodiment, in the micro step, the categories under the selected tree, the selected tree is clothing 6401--> women 6405, and the categories under the selected tree, which are categories 6410-6415, are expanded and compared in the feature space.

According to an embodiment, in order to categorize/classify new content into categories, the cosine similarity of the content and available micro-clusters may be calculated in feature space, and the category to which the most similar micro-cluster belongs may be recommended for the new article.

According to an embodiment, as new categories are introduced by, for example, content users, and more and more content is placed in the categories, the number of micro-clusters can reach very high values. Comparing new content to all existing micro-clusters in the high-dimensional feature space can be resource intensive and can hurt the performance of the system. This problem can be addressed by the damped window model scheme used in data stream clustering.

Damped window model (weight decays over time):

According to an embodiment, when using a damped window model (weight decays over time), each content is associated with a weight that depends on its arrival time. When new content arrives, it is assigned as high a weight as possible, which decreases over time (e.g., exponentially) according to an aging function. The aging function commonly used by the damped window model is an exponential decay function. The weight of a micro-cluster can be calculated as the sum of the weights of the content present therein. When comparing new content with existing micro-clusters, the micro-clusters can be first sorted (in descending order) based on the weights, and then only the top n micro-clusters are selected for feature similarity calculation. In this way, recently used dense micro-clusters can be given higher priority than less frequently used micro-clusters.

Figure 65 is a diagram of how cluster weights decay over time, under an embodiment, and illustrates an example of a damped window model.

According to an embodiment, as shown in Figure 65, three categories are shown, category 1 6510, category 2 6520, and category 3 6530. According to an embodiment, the damping function applied to the categories is:

where λ = 1

According to an embodiment, using the above equation, if Category 1 6500 has 100 articles added at t1, as shown in bold score box 6501, then Category 1 will have a score of 100 at t1, which will be reduced as shown. If no more articles are added to Category 1 before t5, then the score of 100 articles calculated by the damping function will be 6.25.

According to an embodiment, taking the above equation, if Category 2 6510 has 30 articles at t1, 20 articles at t2, and 10 articles are added at t4, as shown in bold score boxes 6511, 6512, and 6513, then Category 2 will have a score of 30 at t1, a score of 35 at t2, a score of 17.5 at t3, and a score of 17.75 at t4. If no more articles are added to Category 2 before t5, the score for 60 articles will be 9.375.

According to an embodiment, taking the above equation, if Category 3 6510 has 10 articles added at t2, 10 articles at t3, 10 articles at t4, and 10 articles added at t5, as shown in the bold score boxes 6521, 6522, 6523, and 6524 in the figure, then Category 3 will have a score of 0 at t1, a score of 10 at t2, a score of 15 at t3, a score of 17.5 at t4, and a score of 18.75 with 40 articles at t5.

Learn from user behavior

According to an embodiment, the user may choose to accept/reject the suggestions provided by the system regarding content classification/categorization. Depending on the user action, the system may add to the machine learning database, for example, in the following manner:

User accepts suggestion

According to an embodiment, recent acceptances strengthen existing signals. The content now becomes part of the category cluster and the content features are included in the cluster features. This is not a direct average update of the cluster features, the damping factor plays a major role in this update. The previous cluster features are partially dampened and then added to the recently placed content features to calculate the updated cluster features. This enables the system to give more weight to the recently added content when classifying the next batch. The frequencies of named entities and topics are also updated for the categories.

User rejects suggestion

According to an embodiment, when a user rejects a particular content classification/categorization, such rejection may ultimately lead to two situations:

i. The user rejects and places the article in a different category: This action will result in the same effect as accepting, the category where the user now places the content will consume the weighted content features, and the updated clustering features will start to continue recommending similar content with higher confidence.

ii. User rejects content and it remains "Unclassified": In this case, the system cannot know what to do with the content (or similar content) as it does not contribute to any cluster. To track such content, a category called "Shadow Cluster" is introduced. Shadow clusters contain rejected and unclassified content from all previously rejected suggestions. When new content arrives, if it has the highest similarity score with a shadow cluster, then the system does not suggest any category for that content.

Figure 66 illustrates a graphical view of how shadow clustering may occur, according to an embodiment.

According to an embodiment, as described above, cluster 1 6600, cluster 2 6610, and cluster 3 6620 may be populated with assigned/classified/categorized content items.

However, according to an embodiment, as described above, over time, shadow clusters 6640 may grow. The shape and size of these shadow clusters may grow indefinitely, and therefore, the system may eventually not suggest any categories for most newly created content. To avoid this, the same micro-clustering scheme can be applied to shadow clusters. Micro-clusters help group similar unclassified content within shadow clusters.

Figure 67 illustrates a graphical view of how shadow clustering may appear, according to an embodiment.

More specifically, FIG. 67 illustrates that shadow cluster 1 6700 , shadow cluster 2 6710 , and shadow cluster 3 6720 may be generated as micro clusters within a shadow cluster according to an embodiment.

According to an embodiment, as micro-clusters within shadow clusters grow, the recommendation system (e.g., categorization engine) may begin to generate suggestions for such new micro-clusters and provide these suggestions to the user. These uncategorized content may be formed into categories, and the names of the categories are suggested based on the most frequently used topics in the system. Users can form new categories from these content through the "Create New Category" option.

FIG. 68 illustrates suggesting that a user create a new category from uncategorized content, according to an embodiment.

According to an embodiment, shown in Figure 68 is an exemplary screenshot of a user interface 6800. In the depicted embodiment, uncategorized content 6805 is displayed. A suggestion to create a new category 6810 (e.g., in a classification system) for such uncategorized content may be presented to the user.

FIG. 69 is a flowchart of a method for intelligently categorizing content in a content management system according to an embodiment.

According to an embodiment, at step 6900, the method may provide one or more computers, including a processor, providing access to a content management system.

According to an embodiment, at step 6910, the method may provide, at one or more computers, a content classification engine that may access a classification system.

According to an embodiment, at step 6920, the method may generate a feature vector from the content at the content management system by a recommendation system of the content classification engine, which recommendation system may access the database of the AI/ML engine and the content classification engine, wherein the generation of the feature vector is based at least on an evaluation of content previously classified within the classification system.

According to an embodiment, at step 6930, the method may classify the new content into a classification system using the generated feature vector.

Although specific embodiments have been described, various modifications, changes, alternative constructions, and equivalents are possible. The embodiments described in this disclosure are not limited to operations within certain specific data processing environments, but can be freely operated within multiple data processing environments. In addition, although embodiments have been described using a series of specific transactions and steps, it is obvious to those skilled in the art that this is not intended to be limiting. Although some flow charts describe operations as sequential processing, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. Processing may have other steps not included in the figure. The various features and aspects of the above-mentioned embodiments can be used alone or in combination.

In addition, although the embodiments described in this disclosure have been described using a specific combination of hardware and software, it should be appreciated that other combinations of hardware and software are also possible. Some embodiments described herein can be implemented using only hardware or only software or a combination thereof. The various processes described herein can be implemented on the same processor or different processors in any combination.

Where a device, system, component or module is described as being configured to perform certain operations or functions, such configuration may be achieved, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations, such as by executing computer instructions or code, or a processor or core is programmed to execute code or instructions stored on a non-transitory storage medium, or any combination thereof to perform the operations. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

Specific details are given in this disclosure to provide a thorough understanding of the embodiments. However, embodiments may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the embodiments. This description provides only example embodiments and is not intended to limit the scope, applicability, or configuration of other embodiments. On the contrary, the previous description of the embodiments will provide an enabling description for implementing various embodiments for those skilled in the art. Various changes may be made to the functions and arrangements of the elements.

The description and drawings are therefore to be regarded as illustrative rather than restrictive. However, it will be apparent that additions, subtractions, deletions and other modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure. Thus, although specific embodiments have been described, these embodiments are not intended to be limiting; various modifications and equivalents are within the scope of the present disclosure.

The embodiments described herein can be implemented using one or more general or special-purpose digital computers, computing devices, machines or microprocessors, or other types of computers including one or more processors, memories and/or computer-readable storage media programmed according to the teachings of the present disclosure. Based on the teachings of the present disclosure, a skilled programmer can easily prepare appropriate software coding, as will be apparent to those skilled in the art of software.

According to some embodiments, the features described herein may be implemented in whole or in part in a cloud environment, as part of a cloud computing system or as a service of a cloud computing system that enables on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage devices, applications, and services), and may include, for example, characteristics as defined by the National Institute of Standards and Technology, such as, for example: on-demand self-service; extensive network access; resource aggregation; rapid elasticity; and measurement services. Example cloud deployment models may include: public cloud, private cloud, and hybrid cloud; and example cloud service models may include software as a service (SaaS), platform as a service (PaaS), database as a service (DBaaS), and infrastructure as a service (IaaS). According to embodiments, unless otherwise specified, clouds as used herein may include public clouds, private clouds, and hybrid cloud embodiments, as well as all cloud deployment models, including, but not limited to, cloud SaaS, cloud DBaaS, cloud PaaS, and cloud IaaS.

According to some embodiments, a computer program product may be provided, which is a non-transitory computer-readable storage medium (or media) having instructions stored thereon/in which the instructions may be used to program a computer to perform any of the processes described herein. Examples of such storage media may include, but are not limited to, hard disk drives, hard drives, fixed disks or other electromechanical data storage devices, floppy disks, optical disks, DVDs, CD-ROMs, microdrives and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems, or other types of storage media or devices suitable for non-transitory storage of instructions and/or data.

The foregoing description is provided for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described in order to best explain the principles of the present teachings and their practical application, thereby enabling others skilled in the art to understand the various embodiments and various modifications suitable for the particular use contemplated. It is intended that the scope be limited by the appended claims and their equivalents.

Claims (20)

1. A system for intelligently classifying content in a content management system, comprising: One or more computers, including processors, that provide access to the content management system; a content categorization engine provided at the one or more computers, the content categorization engine having access to a taxonomy; a recommendation system comprising a content categorization engine, the recommendation system generating feature vectors from content at a content management system, the recommendation system having access to a database of the content categorization engine; wherein the generation of feature vectors is based at least on an evaluation of previously classified content within the taxonomy; The generated feature vectors are used to classify new content into taxonomy. 2. The system of claim 1, wherein the recommendation system creates clusters in a feature space based on previously categorized content. 3. The system of claim 2, wherein the recommender system generates one or more recommendations for new content into a taxonomy by computing feature space distances from clusters. 4. The system of claim 1, Wherein the recommendation system is used to create a new classification system or modify a classification system. 5. The system of claim 1, wherein the content categorization engine's database includes a history of user acceptance of previous categorization recommendations; and Wherein the database of the content categorization engine includes a history of rejection of previous categorization recommendations by users. 6. The system of claim 5, Wherein the recommender system generates one or more recommendations for new content into the taxonomy based on the user's history of accepting previous categorization records and the user's history of rejecting previous categorization recommendations. 7. The system of claim 1, Wherein the recommender system generates recommendations for creating new categories within a taxonomy for a plurality of uncategorized content items. 8. A method for intelligently categorizing content in a content management system comprising: Provide one or more computers, including a processor, to provide access to the content management system; providing a content categorization engine at the one or more computers, the content categorization engine having access to the taxonomy; A feature vector is generated from content at a content management system by a recommender system comprising the content categorization engine, the recommender system having access to a database of the content categorization engine, wherein the generation of the feature vector is based at least on a review of previous categorizations within the taxonomy. Evaluation of the content of the class; Use the generated feature vectors to classify new content into a taxonomy. 9. The method of claim 8, further comprising: Clusters are created in the feature space by the recommender system based on previously classified content within the taxonomy. 10. The method of claim 9, further comprising: One or more recommendations for new content into a taxonomy are generated by the recommender system by computing feature space distances from clusters. 11. The method of claim 8, Wherein the recommendation system is used to create a new classification system or modify a classification system. 12. The method of claim 8, wherein the content categorization engine's database includes a history of user acceptance of previous categorization recommendations; and Wherein the database of the content categorization engine includes a history of rejection of previous categorization recommendations by users. 13. The method of claim 12, further comprising: One or more recommendations for new content into the taxonomy are generated by the recommender system based on the user's history of accepting previous categorization records and the user's history of rejecting previous categorization recommendations. 14. The method of claim 8, further comprising Recommendations for creating new categories within a taxonomy for a plurality of uncategorized content items are generated by the recommender system. 15. A non-transitory computer readable storage medium having instructions thereon which, when read and executed by a computer, cause the computer to perform steps comprising: providing a computer including one or more processors that provides access to the content management system; providing a content categorization engine at the one or more computers, the content categorization engine having access to the taxonomy; generating a feature vector from the content at the content management system by a recommender system comprising the content categorization engine, the recommender system having access to a database of the content categorization engine, wherein the generation of the feature vector is based at least on previous categorization within the taxonomy evaluation of content; Use the generated feature vectors to classify new content into a taxonomy. 16. The non-transitory computer readable storage medium of claim 15, said steps further comprising: Clusters are created in the feature space by the recommender system based on previously classified content within the taxonomy. 17. The non-transitory computer readable storage medium of claim 16, said steps further comprising: One or more recommendations for new content into a taxonomy are generated by the recommender system by computing feature space distances from clusters. 18. The non-transitory computer readable storage medium of claim 15, Wherein the recommendation system is used to create a new classification system or modify a classification system. 19. The non-transitory computer readable storage medium of claim 15, wherein the content categorization engine's database includes a history of user acceptance of previous categorization recommendations; and Wherein the database of the content categorization engine includes a history of rejection of previous categorization recommendations by users. 20. The non-transitory computer readable storage medium of claim 19, said steps further comprising: One or more recommendations for new content into the taxonomy are generated by the recommender system based on the user's history of accepting previous categorization records and the user's history of rejecting previous categorization recommendations.