TWI486798B - Method and device for classifying data - Google Patents

Description

Method and device for data classification

The present application relates to the field of data processing, and in particular, to a method and apparatus for classifying data.

In an e-commerce website, various product materials are usually stored in the form of texts, data sheets, and the like. There are tens of millions of commodity data that an e-commerce website needs to manage. Therefore, how to classify commodity data according to the information content described by it, and manage similar product data in a unified manner to reduce the management complexity of the system and reduce the system. The operational load is the first issue to be considered when operating an e-commerce website.

At present, in various e-commerce websites, clustering algorithms are usually used to classify various commodity materials, that is, according to a preset series of rules and conditions, the product data is divided into multiple categories through similarity analysis. In the prior art, the most commonly used clustering algorithm is a hierarchical clustering algorithm.

The so-called hierarchical clustering algorithm is a bottom-up strategy, which first treats each classified object as a single cluster, and then merges these clusters into higher-level clusters until all the classified objects are concentrated. In the same cluster, or, reach a certain termination condition.

In an e-commerce website, information about a product usually includes a variety of items, such as the identification, category, attributes, and the like of the item. The number of products set in the e-commerce website is tens of thousands. Correspondingly, the related information can be hundreds of thousands of items. For example, the category to which a product belongs is “mobile phone”, and one attribute of the product is “ "Mobile phone brand", and the value of this attribute can be "A brand" or "B brand" and so on. Therefore, the hierarchical clustering algorithm is used to classify the related data of various commodities in the e-commerce website, and the calculation amount is so large that the single machine cannot be completed, and the server cluster is required to perform unified calculation. Obviously, this will greatly waste system resources and consume a lot of computing time, so that the classification of related materials of goods can not be completed in a timely and effective manner, and the efficiency of the classification process of commodity-related data is reduced.

The embodiment of the present application provides a method and a device for classifying data, so as to improve the execution efficiency of the product-related data classification process.

The specific embodiment provided by the embodiment of the present application is as follows: a method for classifying materials, comprising: obtaining related materials of each commodity that need to be classified, and extracting a product title therein; respectively dividing the product titles into word segments, and determining each The weight of the word segmentation, wherein the weight of each word segment is used to indicate the historical frequency of occurrence of the word segment; for different commodities, the word segmentation sequence whose weight value matches the preset condition is separately selected; the word segment sequence selected for each commodity is compared, The related materials of the goods with the same word segmentation sequence are combined.

An apparatus for performing commodity classification, comprising: an extracting unit, configured to acquire related materials of each commodity that need to be classified, and extract a product title therein; and a dividing unit, configured to respectively divide the product titles into word segments, and Determining the weight of each word segment, wherein the weight of each word segment is used to indicate the historical frequency of occurrence of the word segment; the selecting unit is configured to separately select a word segmentation component segmentation sequence whose weight value meets the preset condition for different commodities; The sequence of word segments selected for each item is compared, and the related materials of the items having the same word segment sequence are combined.

In the embodiment of the present application, a certain type of commodity is identified by dividing and extracting the word segmentation sequence from the product title and the attribute information, and the products having the same word segmentation sequence are combined, thereby greatly reducing the commodity related data to be processed. The quantity can realize the classification of goods quickly and accurately in a short time, thereby effectively improving the execution efficiency of the product classification process, reducing the management complexity of the product-related data, and reducing the computational load of the system.

In order to improve the execution efficiency of the product-related data classification process and reduce the operational load of the system, in the embodiment of the present application, when the product-related data is classified, the related data of each commodity to be classified is obtained, and the product title is extracted; Each product title is divided into different words, and the weight of each word segment is determined, wherein the weight of each word segment is used to indicate the historical frequency of occurrence of the word segment; and the segmentation component word segmentation sequence whose weight value meets the preset condition is selected for different products. ; the sequence of word segments selected for each product is compared, and the related materials of the articles with the same word segment sequence are combined.

Wherein, when the products having the same word segment sequence are combined, the related materials of the products having the same word segment sequence are directly combined; or the specified attribute values of the products having the same word segment sequence are obtained, and the related materials of the products having the same attribute value are specified. Consolidate. In the following embodiments, the second case will be described as an example.

The preferred embodiments of the present application are described in detail below with reference to the accompanying drawings.

Referring to FIG. 1 , in the embodiment of the present application, the management apparatus for performing commodity classification includes an extracting unit 10, a dividing unit 11, a selecting unit 12, and a merging unit 13, wherein the extracting unit 10 is configured to obtain a classification that needs to be classified. The relevant information of each commodity, and extracting the product title therein; the dividing unit 11 is configured to divide the word segmentation of each product title separately, and determine the weight of each word segment, wherein the weight of each word segment is used to indicate the history of the word segmentation The frequency selection unit 12 is configured to separately select a word segmentation component word segment sequence whose weight value matches the preset condition for different commodities, and the merging unit 13 is configured to compare the word segment sequences selected for each commodity, and the segmentation sequence is the same. The relevant information of the goods is combined.

As shown in FIG. 1 , in the embodiment, the management apparatus further includes a processing unit 14 configured to separately set and store a corresponding product identification ID for each type of product obtained after the combination.

Based on the above principle, referring to FIG. 2, in the embodiment of the present application, the detailed process for the management device to classify all the product materials included in the e-commerce website is as follows: Step 200: Obtain relevant information of each item that needs to be classified, and Extract the product title and corresponding attribute information.

Usually, when a user publishes product information on an e-commerce website, various information such as a title, an attribute, and the like are filled in the product release page. The completed webpage is uploaded by the user's client to the web server. After receiving the webpage, the web server extracts the title information contained in the webpage. And the word segmentation is processed. For example, the title of a hair dryer is: ** Brand D3506 model hair dryer, obviously, the product title often contains keywords that can distinguish the product, therefore, the extraction of the product title is necessary.

The attribute information of a product often contains a detailed description of the product. For example, the information of a hair dryer is: time to market, color type, air outlet shape, market price, popularity index, and the like. In the embodiment of the present application, the attribute and the attribute value are all set in the form of an ID. For example, the color attribute of an item is green, which can be expressed as: attribute A: 2000, A is the ID of the color attribute, and 2000 is the ID of the green. In the present embodiment, the same degree of the product title and the attribute information is considered at the same time when the product combination is performed, and therefore, the item identification and the attribute information are simultaneously extracted in step 200. In an actual application, attribute information may also be extracted in the step of performing commodity consolidation, and step 200 is merely an example.

Step 210: segmentation of each product title, and determine the weight of each word segment, wherein the weight of the word segment is used to indicate the history frequency related information of the word segment; for example, the number of user history searches, or / and the number of merchant settings and the probability of distribution .

In this embodiment, the Hadoop distributed computing system (hadoop is a decentralized computing framework) is used to segment the product title and attribute information. For example, the product title "** brand D3506 model hair dryer" is divided into the following participles: "** brand", "D3506 model" and "hair dryer". In this embodiment, it is preferable to use a distributed Hadoop, that is, a computing cluster composed of a plurality of (for example, 50 to 300) machines to execute the hadoop program.

In order to improve the accuracy of the word segmentation, in the embodiment, preferably, after performing the word segmentation operation, the management device retains the core participle that can reflect the product brand and the product type according to the historical reference information in the database, for example, "** brand", "** style", etc., correspondingly, need to delete the excess word segmentation of the product classification without reference value, such as "authentic", "promotion", "special offer" and so on.

Step 220: Select a word segmentation sequence of the word segmentation whose value is in accordance with the preset condition for different commodities.

In this embodiment, the so-called preset condition is set to: select two word segments with the highest weight value in the product title, and select five word segments with the highest weight value from the attribute information. The above preset conditions are only examples. The selection method and the number of selected words can be set according to the actual application environment, and will not be described here.

Step 230: Compare the word segment sequences selected for each product, obtain the specified attribute values of the products with the same word segment sequence, and merge the related materials of the products with the same attribute value.

In this embodiment, the related materials of the commodity are merged, that is, the commodities are attributed to the same category, for example, the related materials of each commodity are stored in the same group of texts or data sheets, and in subsequent management, It is presented, published, modified, etc. as the same commodity.

In this embodiment, after the related materials of each product are classified according to steps 200 to 230, a product ID is set for each type of product for uniquely identifying the product. The actual test data indicates that the actual method can be used to The number of commodities in e-commerce websites covering hundreds of millions of commodities has been reduced to tens of millions of commodity categories, which greatly reduces the number of management objects of e-commerce websites, reduces the management complexity of commodity-related materials, and reduces The computing burden of the website.

After the above steps are performed, the weights of the word segments need to be re-adjusted for the result of the segmentation obtained in step 210. The operation of adjusting the weights may be performed immediately after the execution of step 210, or may be performed in steps 200 to 230. Then execute it later. Preferably, the weight of the word segmentation including the product model is required to be set, because the product model is composed of symbols such as digital letters, and the reference value in the product classification process is the largest, therefore, the word segmentation for the product model type, You need to set its weight value higher.

Based on the foregoing embodiment, in order to further improve the accuracy of the product classification result, in this embodiment, after performing the steps 200 to 230, the classification needs to be performed before setting the corresponding product IDs for the classified products. The result is further optimized. Referring to FIG. 3, the detailed detailed process is as follows: Step 300: Determine a word segmentation sequence for distinguishing each type of commodity according to the classification result.

The so-called word segmentation sequence is the iconic word segment combination of each type of product obtained after performing steps 200 to 230. For example, after the word segmentation, the product title and the attribute information include the participle "** brand", "** style The items of "red color", "DF0753" and "L code" belong to the same category, then the word segmentation sequence of the product is "** brand style red color DF0753L code".

Step 310: Calculate the similarity of the word segmentation sequences of any two types of commodities, respectively.

In this embodiment, the similarity of the word segmentation sequence of any two types of commodities is calculated by the following formula:

Where TD1 and TD2 are the word segmentation sequences of the two types of products respectively compared, for example, TD1=(word11, score11), (word12, score12), (word13, score13)

TD2=(word21, score21), (word22, score22), (word23, score23)

Word is a participle, and score is its weight.

Prop1 and prop2 are the primary attribute values corresponding to the two types of products that are compared respectively (primary attributes, that is, important attributes, such as the most important The attribute is the brand and model, and the color and weight are the general attributes. The primary attribute value represents a specific attribute, for example: the brand is a primary attribute, and the magnitude of the similarity is calculated by the cosine. The greater the similarity, the more similar the two commodities are.

λ is a coefficient of control weight. λ ₁ and λ ₂ are two different coefficients. When calculating similarity, it is indicated whether the title is important or the attribute is more important. When λ ₁ = 2, λ ₂ =1. It shows that the importance of the title is twice the importance of the attribute.

a, b are preset parameters, and n1 and n2 are respectively used to indicate the number of commodities included in each of the two types of commodities for which the similarity comparison is performed, and a and b are used to control the value of the similarity to indirectly control the two types of commodities. The possibility of merging, for example, when the number of commodities included in each of the two types of commodities is large, the value of the similarity can be adjusted by the values of a and b. The calculated similarity value becomes smaller, so that the chances of combining the two types of commodities become smaller.

For example, a=50, b=20 n1=100, n2==10, then, similarity=e ^{- λ 1*|TD1-TD2|} *e ^{- λ 2*} | ^prop1-prop2| *1/(1+ e^(50/20))=1/(1+e^2.5)=0.07585818 7%.

Step 320: Compare the similarity of the segmentation sequence of the obtained two types of products with the set threshold, and merge the two types of commodities whose similarity of the segmentation sequence reaches the set threshold.

For example, in step 310, the similarity of the word segmentation sequence of the two types of products is calculated to be 7%. If the threshold is set to 5, the similarity value is far less than the set threshold, which means that the two types of commodities cannot be merged.

The above steps 300-320 are performed because the two types of products with different word segmentation sequences may also be the same product, but the product title and attribute information set by the merchant are not completely the same. Therefore, by performing steps 300 to 320, The classification results obtained in steps 200 to 230 are optimized to make them more accurate. In practical applications, in order to further optimize the classification result, steps 300 to 320 may be performed in a plurality of iterations according to the set number of times, so that the product categories included in the finally obtained classification result are further reduced.

By adopting the above method, the product of tens of millions of products can be reduced, and further reduced to several million, and the whole process takes only a few hours, thereby greatly reducing the number of management objects of the e-commerce website and further reducing the management products thereof. The complexity of related materials has reduced the computational burden.

In summary, in the embodiment of the present application, a certain type of product is identified by dividing and extracting the word segmentation sequence from the product title and attribute information, and the related materials of the same word segmentation sequence are combined, thereby greatly reducing The quantity of commodity-related materials that need to be processed can quickly and accurately achieve product classification in a short period of time, thereby effectively improving the efficiency of the product classification process, reducing the management complexity of the product-related data, and reducing the system's Operational load.

Based on the above solution, in the embodiment of the present application, the optimization of the classification result is continued by the similarity between the word segmentation sequences, thereby further improving the accuracy of the classification result, and further reducing the number of commodity materials to be processed, and improving the number of products. The efficiency of the execution of the commodity classification process.

It is apparent that those skilled in the art can make various modifications and variations to the embodiments of the present application without departing from the spirit and scope of the application. Thus, the present embodiments of the present application are intended to cover such modifications and variations as the scope of the application and the scope of the invention.

10. . . Extraction unit

11. . . Division unit

12. . . Selection unit

13. . . Merging unit

14. . . Processing unit

1 is a functional structural diagram of a management device in an embodiment of the present application;

2 is a flow chart of classifying commodity data in the embodiment of the present application;

FIG. 3 is a flowchart of optimizing classification results in the embodiment of the present application.

Claims (13)

A method for classifying data, comprising: obtaining related materials of each commodity to be classified, and extracting a product title therein; classifying each product title separately, and determining weights of each word segment, wherein each The weight of the word segmentation is used to indicate the historical frequency of the word segmentation; for the different commodities, the word segmentation sequence that matches the pre-conditions is selected; and the segmentation sequence selected for each commodity is compared, and the segmentation sequence is the same. The relevant information of the goods is combined. The method according to claim 1, wherein after each word segmentation is divided into words, the weight of each word segment is adjusted according to the division result. The method of claim 1, wherein the merging the related materials of the same word segmentation comprises: directly merging related materials of the same word segmentation sequence; or acquiring the goods having the same word segmentation sequence Specify attribute values and merge related data for items with the same attribute value. The method of claim 1, wherein the method for calculating the similarity of the word segmentation sequence of any two types of products is performed by combining the related materials of the products having the same word segmentation sequence; The similarity of the word segmentation sequences of any two types of products obtained is compared with a set threshold, and the related data of the two types of commodities whose similarity of the segmentation sequence reaches the set threshold is combined. The method of claim 4, wherein, when calculating the similarity of the word segmentation sequence of the two types of commodities, the following formula is adopted: Among them, TD1 and TD2 are the word segmentation sequences of the two types of commodities respectively compared, and prop1 and prop2 are the main attribute values corresponding to the two types of commodities respectively compared, and λ ₁ and λ ₂ are preset control coefficients, and a and b are The preset parameters, n1 and n2, respectively, are used to indicate the number of items included in each of the two types of products for which the similarity comparison is performed. The method of claim 4, wherein, after combining the related data of the two types of products whose similarity of the word segment sequence reaches the set threshold, the iterative operation is performed according to the preset number of times. The method of claim 1, wherein the corresponding product identification ID is set and saved for each type of product obtained after the combination. An apparatus for classifying goods, comprising: an extracting unit, configured to acquire related materials of each item to be classified, and extract a product title therein; and a dividing unit configured to separately perform each product title Word segmentation, and Determining the weight of each word segment, wherein the weight of each word segment is used to indicate the historical frequency of occurrence of the word segment; the selecting unit is configured to separately select a word segmentation component segmentation sequence whose weight value meets a preset condition for different commodities; and a merging unit, It is used to compare the sequence of word segments selected for each item, and to combine the related materials of the items with the same word segment sequence. The device according to claim 8, wherein the dividing unit divides the segmentation of each product title, and adjusts the weight of each word segment according to the division result. The apparatus of claim 8, wherein the merging unit merges related materials of the same word segmentation product directly, or merges the related materials of the same word segmentation sequence directly; or obtains the same word segmentation sequence The specified attribute value of the item, and the related data of the item with the same attribute value is combined. The device of claim 8, wherein the method for classifying the products of the same word segmentation is combined, and calculating the similarity of the word segmentation sequence of any two types of products, and then obtaining any two of the obtained word segments. The similarity of the word segmentation sequence of the commodity is compared with the set threshold, and the related data of the two types of commodities whose similarity of the segmentation sequence reaches the set threshold is combined. The apparatus according to claim 11, wherein the merging unit combines the related data of the two types of commodities whose similarity of the segmentation sequence reaches the set threshold, and performs the iterative operation according to the preset number of times. a device as claimed in claim 8, 9 or 10, The method further includes: a processing unit, configured to separately set a corresponding product identification ID for each type of product obtained after the combination, and save the same.