KR102783347B1 - System of extracting and utilizing information based on artificial intelligence - Google Patents

Abstract

An artificial intelligence-based information extraction and utilization system is disclosed, including a URL collection unit that collects a Uniform Resource Locator (URL) of a homepage including information related to a target product; a webpage classification unit that classifies webpages including detailed information of the target product among webpages within the homepage having the collected URL; and an information extraction unit that extracts categories and/or values of information required for the target product from the webpages including the detailed information of the classified target product based on an artificial intelligence model and trains the artificial intelligence model based on the extracted data.

Description

{SYSTEM OF EXTRACTING AND UTILIZING INFORMATION BASED ON ARTIFICIAL INTELLIGENCE}

The present invention relates to an artificial intelligence-based information extraction and utilization system, and more specifically, to an artificial intelligence-based information extraction and utilization system that can easily extract necessary information about a desired product from a vast Internet-based information sea and produce and service various contents about the product by utilizing the extracted information.

The Internet, which is called an ocean of information, has a wide variety of information scattered everywhere, so it requires a lot of time and effort to obtain the information you want.

For example, if you want to obtain the information you want about a specific product, you have to access each and every website operated by the manufacturer or seller of that product, and even after accessing the website, you have to check the various web pages provided on the website to see if the information you want is there.

Therefore, the problem arises that it takes a lot of time and effort to obtain the desired information in an Internet environment.

Korean Patent Publication No. 10-2021-0042271 Korean Patent Publication No. 10-2021-0092979 Korean Patent Registration No. 10-2198904

Accordingly, the present invention aims to solve a technical problem of providing an artificial intelligence-based information extraction and utilization system that can easily extract necessary information about a desired product from a vast Internet-based information sea and produce and service various contents about the product by utilizing the extracted information.

According to one aspect of the present invention,

A URL collection unit that collects the Uniform Resource Locator (URL) of a homepage containing information related to the target product;

A web page classification unit that classifies web pages containing detailed information on the target product among web pages within the homepage having the collected URL; and

An information extraction unit that extracts categories and/or values of information required for the target product from a webpage containing detailed information of the classified target product based on an artificial intelligence model, and trains the artificial intelligence model based on the extracted data;

An artificial intelligence-based information extraction and utilization system including is provided.

The above artificial intelligence model can be trained to determine whether an image, a word (text), a text phrase including a group of words, a sentence, or a paragraph corresponds to the information required for the target product.

The above artificial intelligence model can be trained to determine categories and values of images, words (text), and text phrases including word clusters, sentences, and paragraphs, and their xpaths.

The above information extraction unit can extract information required for the target product by performing bottom-up judgment from a leaf node having a value in the DOM tree based on the extraction result of the artificial intelligence model.

The above information extraction unit can perform a judgment of the value or category of another node based on the judgment result of the value or category of an arbitrary node in the DOM tree based on the extraction result of the artificial intelligence model.

The above information extraction unit can determine that a node in the DOM tree is information required for the target product if the distance between the node and the information required for the target product is less than a preset threshold value and the node is not determined to be a boilerplate including a header and a footer.

The above web page classification unit can classify the web page based on at least one of the following elements: a DOM (Document Object Model) tree of the web page, CSS (Cascading Style Sheets) and style tags used for each web page, a structure-template of the web page, a structure-boilerplate of the web page, and images, words, phrases, and tables existing in the web page.

An artificial intelligence-based information extraction and utilization system according to an embodiment of the present invention can quickly and accurately extract and provide necessary information on a desired product by utilizing artificial intelligence capable of learning from a vast ocean of information based on the Internet.

In addition, the artificial intelligence-based information extraction and utilization system according to an embodiment of the present invention can provide application content for a target product by utilizing the extracted information, thereby increasing interest and favorability for the target product and further contributing to increased sales of the product.

Figure 1 is a block diagram of an artificial intelligence-based information extraction and utilization system according to one embodiment of the present invention.
FIGS. 2 to 4 are diagrams for explaining examples of learning of an artificial intelligence model in an information extraction unit of an artificial intelligence-based information extraction and utilization system according to one embodiment of the present invention.

The present invention can be modified in various ways and has various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention.

In describing the present invention, if it is judged that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, numbers (e.g., first, second, etc.) used in the description of this specification are merely identifiers for distinguishing one component from another.

In addition, throughout the specification, when a component is referred to as being "connected" or "connected" to another component, it should be understood that the component may be directly connected or directly connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated. In addition, throughout the specification, when it is said that a certain part "includes" a certain component, this does not mean that other components are excluded, but rather that other components can be included, unless otherwise specifically stated. In addition, terms such as "part" and "module" described in the specification mean a unit that processes at least one function or operation, and this means that it can be implemented by one or more hardware or software, or a combination of hardware and software.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a block diagram of an artificial intelligence-based information extraction and utilization system according to one embodiment of the present invention.

Referring to FIG. 1, an artificial intelligence-based information extraction and utilization system (10) according to one embodiment of the present invention may be configured to include a URL collection unit (11) that collects a Uniform Resource Locator (URL) of a homepage including information related to a target product, a webpage classification unit (12) that classifies a webpage including detailed information of a target product among webpages within the homepage having the collected URL, and an information extraction unit (13) that extracts a category and/or value of information required for a target product from a webpage including detailed information of the classified target product based on an artificial intelligence model and trains an artificial intelligence model based on the extracted data.

The URL collection unit (11) can collect the URL of a homepage that has information on the target product by inputting keywords and/or images, etc. related to the target product, and inputting the input keywords and/or images, etc. as search words on a portal site or social network service site to search the database of the portal site or social network service site.

As another example, the URL collection unit (11) can collect the URL of a homepage that has information on the target product through a database of homepages operated by associations (product associations, regional associations, etc.) related to the target product. For example, if the target product is wine, the URL collection unit (11) can collect the URL of a homepage (e.g., a specific producer's homepage) that includes information on wine from a homepage operated by a regional wine producer association.

As another example, the URL collection unit (11) may collect URLs by receiving a URL directly provided by the user.

The web page classification unit (12) can classify web pages that include detailed information on the target product among web pages within a homepage having collected URLs.

A homepage having collected URLs may be composed of multiple web pages, and among the multiple web pages, there may be web pages that include detailed information about the target product and web pages that are not related to detailed information about the target product and are used for other purposes. The web page classification unit (12) may classify web pages that are expected to include the required information among the multiple web pages that compose the homepage having collected URLs.

The web page classification unit (12) can classify web pages containing detailed information on target products within a homepage having URLs collected in various ways.

As a first example, the web page classification unit (12) can classify web pages based on structure tags, i.e., based on the Document Object Model (DOM) tree. The web page classification unit (12) identifies DOM tree elements and shapes (width, depth, number of total nodes, number of inner nodes, number of leaf nodes), depth of leaf nodes, tree edit distance, etc. for each web page, and classifies web pages having similar shapes based on the identified results, thereby classifying one of the web pages having similar shapes as a web page having detailed information on a product.

As a second example, the web page classification unit (12) can classify web pages based on structure-looking, that is, based on CSS (Cascading Style Sheets) and style tags used for each web page. The web page classification unit (12) determines the type of CSS used for each web page, the number of CSS used, and the location in the DOM tree, determines the type of style tag (<style> tag) used for each web page, the number of style tags used, and the location in the DOM tree, and classifies web pages using similar CSS/style tags based on the determined result, thereby classifying one of the web pages having similar appearances as a web page having detailed information on a product.

As a third example, the web page classification unit (12) can classify web pages based on structure-template (framework). The web page classification unit (12) can classify web pages by determining whether the web pages use a framework (e.g., Typo3, Bootstrap, Wordpress, etc.). In general, the layout and purpose of the web page can be identified based on which framework is used, so the web page classification unit (12) can classify web pages with detailed product information based on the framework used by the web page.

As a fourth example, the web page classification unit (12) can classify web pages based on a structure-boilerplate framework. The web page classification unit (12) can classify web pages with detailed product information by removing boilerplate (duplicate source codes, for example, head/footer, fixed-position advertisements, etc.) of web pages and identifying the remaining content of web pages from which the boilerplate has been removed.

As a fifth example, the web page classification unit (12) can extract images existing in the web page and classify the web page based on the resolution, size, or location of the image. The web page classification unit (12) can extract images through tags used in the web page, determine the image of the corresponding product among the images, and classify the web page having detailed information of the product based on the screen resolution, file resolution, file location, and properties of the node displaying the product image in the DOM tree of the product image. Here, the technique for determining the product image can be performed through the artificial intelligence model described below, and thus will be described in detail later. For example, the web page classification unit (12) can determine that the corresponding web page is not a web page having detailed information of the product if there is no product image among the extracted images, and can determine that the page having the largest screen resolution among the web pages having a product image is a web page having detailed information of the product.

As a sixth example, the web page classification unit (12) can classify the web page based on the words (text) existing in the web page. The web page classification unit (12) can extract the web page having the detailed product information by deleting all the <> tags from the code of the web page, leaving only the words displayed on the screen, and using the artificial intelligence model described below for the words of each node of the DOM tree to determine whether they correspond to product information. That is, if there is no word determined to be product information, the web page classification unit (12) can determine that the web page does not have detailed information on the target product, and if there is a word corresponding to 'category'-'product information' (e.g., 'price'-'100 won') among the words or 'product information' (e.g., 'today only') where no category exists, it can determine that the web page has detailed information on the target product.

As a seventh example, the web page classification unit (12) can classify web pages based on text phrases existing in the web pages. Here, the text phrases can be a group of words composed of words or combinations, sentences, or paragraphs composed of one or more sentences. The web page classification unit (12) can extract web pages having detailed product information by deleting all <> tags from the code of the web page and leaving only the text displayed on the screen, and then using the artificial intelligence model described below for the text of each node of the DOM tree to determine whether it corresponds to product information. That is, if there is no phrase determined as product information, the web page classification unit (12) can determine that the web page does not have detailed information on the target product, and if there is a relationship of 'category' - 'product information' (e.g., 'how to use' - 'mix thoroughly with water and drink') in the text or if there is text corresponding to 'product information' for which no category exists (e.g., 'keep in a high place out of reach of children'), it can be determined that the web page has detailed information on the target product.

As an eighth example, the web page classification unit (12) can classify the web page based on whether a table (text phrase) exists in the web page, etc. Since detailed information about a product on the web page is likely to exist in a table, the web page classification unit (12) can determine that the web page has detailed information about the target product by checking table-related tags (e.g., <table>, <div>, <ul>, <li>, <grid>, ...) and checking words or phrases written in the table.

As a ninth example, in the case of a webpage that only contains image files, the webpage classification unit (12) can determine whether the webpage has detailed information about the target product based on information extracted through the image using the artificial intelligence model described below.

The web page classification unit (12) can increase the accuracy of web page extraction by applying only one of the various techniques for extracting web pages having detailed information on target products as described above, or by applying two or more techniques in combination.

The information extraction unit (13) extracts categories and/or values of information required for a target product from a web page containing detailed information of the target product classified by the web page classification unit (12) based on an artificial intelligence model, and can train an artificial intelligence model based on the extracted data.

The information extraction unit (13) is an element for extracting information required for a target product from a web page having detailed information on the target product, and may include a learnable artificial intelligence model for extracting information required for the product.

Information required for a product extracted by an AI model can be classified into either quantitative information or qualitative information, and can have a category to which it belongs. That is, an AI model can be trained to extract quantitative information, qualitative information, and the category to which this information belongs as information required for the target product (target product information) from a classified web page.

For example, quantitative information among the target product information may include information specifying the price, ingredients, capacity, producer, seller, country of production, etc. Qualitative information among the target product information may include the producer's description of the product, expert evaluation of the product, etc. The categories to which this target quality information belongs may include price, directions for use, recipes, images, videos, history, seller contact information, etc.

Within the web code of a web page, the target product information can appear in the form of the value that the product information has within the DOM tree, and mainly two nodes can constitute one target product information. The target product information appears in the form of "xpath, category", "xpath, value of the corresponding category" within the web code, and can appear, for example, as "price": "100 won". Instead of two nodes, one node may exist without a category. In other words, the target product information can simply appear in the form of "xpath, value" within the web code, and an example of this can be a phrase such as "Selling at this price today only".

The artificial intelligence model equipped in the information extraction unit (13) can be trained to determine whether an image, word (text), or text phrase (a group of words, a sentence, a paragraph) included in a web page corresponds to target product information and extract it. For example, it can determine whether (or the probability) the word '100 won' corresponds to target product information in the category of 'price', whether the group of words 'so delicious' corresponds to a category describing the taste of the target product, and whether the paragraph 'produced for 100 years...' corresponds to a category of history of the producer of the target product.

The artificial intelligence model equipped in the information extraction unit (13) can be learned using learning data arbitrarily input by the initial user, and after the system as shown in Fig. 1 is configured, it can be updated and repeatedly learned using images, words, phrases, etc. input from web pages classified in the web page classification unit (12).

FIGS. 2 to 4 are diagrams for explaining examples of learning of an artificial intelligence model in an information extraction unit of an artificial intelligence-based information extraction and utilization system according to one embodiment of the present invention. In particular, FIGS. 2 and 3 are for explaining examples of learning of an artificial intelligence model for an image.

For example, the artificial intelligence model (131) can perform learning by receiving as learning data a frontal image of a wine or an image that can identify the label of the wine, as illustrated in FIG. 2.

Here, 'front image' and 'label image' may correspond to categories, and the artificial intelligence model (131) may be trained to distinguish images corresponding to the categories of 'front image' and 'label image' among the input learning data as target product images and classify them into each category, and to distinguish images that are not target product images.

That is, the artificial intelligence model (131) can distinguish an image corresponding to a desired product image as an image of a target product, that is, a wine product, as shown in Fig. 3, and distinguish the remaining images as not being images of wine products.

FIG. 4 illustrates an example of learning for words or phrases of an artificial intelligence model. Similar to the learning of the image described above, the artificial intelligence model (131) can perform learning by inputting words or phrases related to wine. For example, the artificial intelligence model (131) can be trained to input wine-related words included in categories corresponding to words such as 'name', 'price', 'date', 'ingredients', etc. as learning data and classify them into corresponding categories. In addition, the artificial intelligence model (131) can be trained to input wine-related phrases included in categories corresponding to phrases such as 'how to use', 'cautions', 'taste expression', etc. as learning data and classify them into corresponding categories.

By this, the artificial intelligence model (131) can classify words or phrases related to the target product, wine, and words or phrases that are not related, and classify words related to the target product into an appropriate category.

Furthermore, the artificial intelligence model (131) can learn xpaths for categories and values. For example, it can be learned to determine the xpath corresponding to the corresponding category and value by checking the xpaths of the word 'price' as a category and the word '100 won' as a corresponding value. In addition, the artificial intelligence model (131) can be learned to determine the xpath corresponding to the corresponding value without a category by checking the xpath for a value without a category.

The artificial intelligence model (131) learned in this way can extract and classify images, words, and phrases corresponding to target product information from web pages provided by the web page classification unit (12) within the system.

The information extraction unit (13) including the artificial intelligence model (131) can extract product information in the following manner by utilizing information obtained from the artificial intelligence model (131) and the web page classification unit (12).

The information extraction unit (13) can perform bottom-up judgment from a leaf node having a value in the DOM tree.

The information extraction unit (13) can be applied in a manner in which it determines whether the value (image, word or phrase) of a specific node is a category, a value for a specific category or a value without a category, and then determines the relationship between the determined node and the upper node based on the determined result.

Additionally, the information extraction unit (13) can determine the relationship with other nodes based on the judgment result for node A.

For example, if node A is a word and node A corresponds to a value, the information extraction unit (13) can determine whether node B is a category corresponding to node A, or if node A is a word and node A corresponds to a category, determine whether node B is a value corresponding to node A, or if node A is a word and node A corresponds to a category, determine whether node B, node C, and node D are all values corresponding to node A.

In addition, the information extraction unit (13) can determine whether node B is a category corresponding to node A if node A is a product image and node A corresponds to a value.

In addition, the information extraction unit (13) can determine whether node B is a category corresponding to node A if node A is a phrase and node A corresponds to a value.

In addition, the information extraction unit (13) can determine the relationship between two nodes even when node A is a word and node B is an image, text or phrase. When node A, which is a word, is a category, the information extraction unit (13) can determine whether node B, which is an image or a word phrase, is a value corresponding to node A.

Meanwhile, the information extraction unit (13) can perform correction of product information judgment.

Information extraction unit (13) By determining whether the relationship between the nodes that are matched with each other has the same parent node in the DOM tree, whether the same parent node is a root node, if node A', the immediate parent node of node A, and node B', the immediate parent node of node B, are different nodes, what is the tree distance between node A' and node B', whether the tree distance is less than a preset threshold, and whether the parent nodes of node A and node B are close, it is possible to determine how close the distance between node A and node B is.

Within the DOM tree, product information is likely to be very close to each other in terms of shape and will not be separated by a large distance. Accordingly, it is desirable that among the set of nodes determined by the target product information (category, value), the shortest distance between one node and another node is less than a predetermined threshold value.

Additionally, product information is more likely to be in nodes located inside the DOM tree rather than at the edges, with the ends of the DOM tree likely to be occupied by headers and footers.

In addition, it is highly likely that there is not only one webpage on the homepage that contains detailed information about the target product, and in the case of a node that displays only values without specifying a category, it is difficult to determine whether it is the product information you want to extract if you only judge it as a single webpage containing detailed information about the product. However, if there are multiple webpages with detailed information, the corresponding xpath will be repeated, which can be used to determine whether it is product information.

In other words, if the distance between nodes that are judged as information about the target product and those that are not judged as boilerplate such as headers and footers in the DOM tree is close (within a preset threshold), information that would not have been judged as product information from a local perspective can be corrected to product information from a global perspective.

In this way, the information extraction unit (13) can determine a node as product information if the node is not determined to be boilerplate such as a header or footer and is close to nodes determined to be information on the target product.

Meanwhile, the information extraction unit (13) can not only update the artificial intelligence model (131) for extracting target product information by additionally learning it using the extracted target product information, but can also create and provide application content or services using the additional artificial intelligence model.

For example, an additional artificial intelligence model provided in the information extraction unit (13) can extract key elements of product description expressions and produce content related thereto. The artificial intelligence model can be trained to extract some words that can be summarized/abbreviated among expressions listed in text, or to extract some expressions that can be summarized/abbreviated among visual expressions such as pictures and videos.

In addition, the AI model can automatically generate content related to product expressions. For example, the AI model can delete key words from product expressions listed in letters and place them in parentheses to create a subjective/objective choice or true/false selection quiz for the words in the corresponding space. In addition, based on the created quiz and its answers, it is possible to generate a virtual quiz and answers with similar format and explanation.

In addition, the AI model can be trained to perform score matching for product expressions. For example, the AI model can be implemented to learn 1,000 product expressions and ratings of critic A, and calculate the rating when a new product expression is input.

In addition, the AI model can be trained to express descriptions of new products by learning the product description style of each individual. For example, if the AI model learns 1,000 product description data of critic A and creates a model, it can be implemented to generate product descriptions that seem to have been written by critic A for new materials.

Additionally, the AI model can find product B with similar expressions to the description of product A, or find product C with expressions that match the description of product A. Additionally, the AI model can be implemented to generate sentences and paragraphs explaining why products A and B are similar and why products A and C match.

As described above, the AI-based information extraction and utilization system according to various embodiments of the present invention can quickly and accurately extract and provide necessary information on a desired product by utilizing AI capable of learning from a vast Internet-based information sea. In addition, the AI-based information extraction and utilization system according to various embodiments of the present invention can provide application content for a target product by utilizing the extracted information, thereby increasing interest and favorability for the target product and further contributing to increased sales of the product.

Although the present invention has been described above with reference to embodiments thereof, it will be readily understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

Claims (7)

A URL collection unit that collects a URL (Uniform Resource Locator) of a homepage including information related to a target product; a webpage classification unit that classifies a webpage including detailed information of the target product among webpages within the homepage having the collected URL; and an information extraction unit that extracts a category and/or value of information required for the target product from a webpage including detailed information of the classified target product based on an artificial intelligence model and trains the artificial intelligence model based on the extracted data.
The above artificial intelligence model is,
It is learned to determine whether an image, a word (text), and a text phrase containing a group of words, a sentence, or a paragraph corresponds to the information required for the target product, or it is learned to determine the category and value of an image, a word (text), and a text phrase containing a group of words, a sentence, or a paragraph, and its xpath.
The above information extraction unit,
Based on the extraction results of the above artificial intelligence model, bottom-up judgment is performed from leaf nodes having values in the DOM (Document Object Model) tree to extract the information required for the target product.
Based on the extraction results of the above artificial intelligence model, the value or category of node B, which is another node, is judged based on the value or category judgment result of node A, which is an arbitrary node in the DOM tree.
If node A is the above word and corresponds to the above value, then node B determines whether it is a category corresponding to node A.
If node A is the above word and falls into the above category, then node B is judged to be a value corresponding to node A.
If node A is the image of the product and corresponds to the value above, then node B determines whether it is a category corresponding to node A.
If node A is the above phrase and corresponds to a value, then node B determines whether it is a category corresponding to node A.
An artificial intelligence-based information extraction and utilization system characterized in that it determines how close the distance between nodes A and B is by determining whether the relationship between matched nodes has the same parent node in the DOM tree, whether the same parent node is a root node, and if node A', the immediate parent node of node A, and node B', the immediate parent node of node B, are different nodes, what the tree distance is between node A' and node B', and whether the tree distance is smaller than a preset threshold value. delete delete delete delete In claim 1, the information extraction unit,
An artificial intelligence-based information extraction and utilization system characterized in that if any node in a DOM tree is not judged to be a boilerplate containing a header and a footer and the distance from a node judged to be information required for the target product is less than a preset threshold value, the node is judged to be information required for the target product. In claim 1, the web page classification section,
An artificial intelligence-based information extraction and utilization system characterized in that it classifies the webpage based on at least one of the following elements: a DOM (Document Object Model) tree of the webpage, CSS (Cascading Style Sheets) and style tags used for each webpage, a structure-template of the webpage, a structure-boilerplate of the webpage, and images, words, phrases, and tables existing in the webpage.