WO2018014267A1 - Method and system for processing massive crowd feature data - Google Patents

Abstract

A method for processing massive crowd feature data, and a corresponding system. The method comprises: setting serial numbers of a crowd and a feature label (S101); establishing binary feature data corresponding to each feature label according to the serial numbers of the crowd (S102); converting the established binary feature data corresponding to each feature label into hexadecimal feature data (S103); and storing converted hexadecimal data corresponding to each feature label (S104). The method and system improve the processing efficiency of massive crowd feature data and save data storage space.

Description

Method and system for processing massive population characteristic data Technical field

The present invention relates to the field of big data processing technologies, and in particular, to a method and system for processing massive population feature data.

Background technique

The processing of the existing massive demographic data is usually stored in the form of rows and columns based on the individual individual audiences sorting corresponding feature tags of 0 or 1 identifiers. Although this processing method is simple, the label expression is complicated and difficult to implement. The traditional relation type stores a large amount of storage space in the form of row and column. At the same time, when performing query expression operation, the label position index table needs to be associated with the corresponding position. Then, logical expressions can be constructed to perform logical operations. The query processing process is complicated and time-consuming, resulting in low data processing efficiency for mass population features.

Summary of the invention

In view of this, the present invention provides a method and system for processing massive population feature data, and solves the technical problems of low data processing efficiency and large data storage space of the existing mass population features.

According to an embodiment of the present invention, a method for processing massive population feature data is provided, including: setting a population number and a feature tag; and establishing binary feature data corresponding to each feature tag according to the crowd number; corresponding to each feature tag to be established The binary feature data is converted into hexadecimal feature data; and the hexadecimal feature data corresponding to each feature tag converted is stored.

Preferably, the method for processing the mass demographic data further comprises: generating a query logic expression according to the feature tag of the selected query; performing logical operations on the hexadecimal feature data corresponding to the feature tag according to the generated logical expression to obtain a binary a string; and calculating the number of 1s in the obtained binary string, and using the calculated number as the number of audiences for the feature tag of the query.

Preferably, the generating a query logic expression according to the feature tag of the selected query comprises: generating a logical OR operation expression according to the feature tag of the selected query; generating a logical AND operation expression according to the feature tag of the selected query; and generating the logic Or an arithmetic expression and a logical AND operation expression to synthesize a query logical expression.

Preferably, the logical operation of the hexadecimal feature data corresponding to the feature tag according to the generated logical expression to obtain a binary string includes: grouping the generated logical expression; acquiring the feature tag of the group Corresponding hexadecimal feature data is converted into long type data; Boolean operation is performed on the long type data packet corresponding to each group feature tag; and the result of all grouping Boolean operations is converted into a binary string.

Preferably, the method for processing the mass demographic data further comprises: transmitting the calculated audience quantity of the feature tag to the query end.

According to another embodiment of the present invention, a processing system for mass demographic data is provided, including: a setting module for setting a crowd number and a feature tag; and a feature data establishing module for a crowd number set according to the setting module Establishing binary feature data corresponding to each feature tag; a conversion module, configured to convert binary feature data corresponding to each feature tag established by the feature data establishing module into hexadecimal feature data; and a storage module, configured to The hexadecimal feature data corresponding to each feature tag converted by the conversion module is stored.

Preferably, the processing system for the mass demographic data further includes: a query logic expression generating module, configured to generate a query logic expression according to the feature tag of the selected query; and a logic operation module, configured to generate according to the query logic expression The logical expression generated by the module performs a logical operation on the hexadecimal feature data corresponding to the feature tag to obtain a binary string; and a calculation module for calculating the number of 1 in the binary string obtained by the logical operation module, and The calculated quantity is the number of audiences for the feature tag of the query.

Preferably, the query logic expression generating module includes: a first generating unit, configured to generate a logical OR operation expression according to the feature tag of the selected query; and a second generating unit, configured to generate a logical AND operation according to the feature tag of the selected query And a synthesizing unit configured to synthesize the logical OR operation expression and the logical AND operation expression generated by the first generation unit and the second generation unit into a query logic expression.

Preferably, the logic operation module includes: a grouping unit, configured to group a logical expression generated by the query logic expression generating module; and a long type data converting unit, configured to acquire a feature label corresponding to the grouping unit group The hexadecimal feature data is converted into long type data; a grouping operation unit is configured to perform a Boolean operation on the long type data packet corresponding to each group feature tag converted by the long type data conversion unit; and a binary string conversion a unit for converting the result of all grouping Boolean operations of the grouping operation unit into a binary string.

Preferably, the processing system of the mass demographic data further includes a sending module, configured to send the audience number of the feature tag calculated by the computing module to the querying end.

The method and system for processing massive population feature data provided by the present invention establish binary feature data corresponding to each feature tag according to the population number, and convert the binary feature data corresponding to each feature tag into hexadecimal feature data. Storage, based on feature tags, efficiently realize distributed processing, storage and retrieval of massive population feature data, improve the processing efficiency of massive population feature data, and save data storage space.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly described below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those skilled in the art can obtain other drawings according to the drawings without any creative work.

FIG. 1 is a schematic flow chart of a method for processing mass population characteristic data according to an embodiment of the present invention.

2 is a schematic diagram of binary feature data in one embodiment of the present invention.

3 is a schematic diagram of hexadecimal feature data in an embodiment of the present invention.

FIG. 4 is a schematic flow chart of a method for processing mass population characteristic data according to another embodiment of the present invention.

FIG. 5 is a schematic flowchart of generating a query logic expression according to another embodiment of the present invention.

FIG. 6 is a schematic flowchart of logical processing of feature data in another embodiment of the present invention.

FIG. 7 is a schematic structural diagram of a processing system for mass population characteristic data according to still another embodiment of the present invention.

FIG. 8 is a schematic structural diagram of a processing system for mass population characteristic data according to still another embodiment of the present invention.

FIG. 9 is a schematic structural diagram of a query logic expression generating module according to still another embodiment of the present invention.

FIG. 10 is a schematic structural diagram of a logic operation module according to still another embodiment of the present invention.

detailed description

The technical solutions of the present invention are further described in detail below with reference to the accompanying drawings and specific embodiments. It is apparent that the described embodiments are only a part of the embodiments of the invention, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of the present invention.

In the description of the present invention, it is to be understood that the terms "first", "second" and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. In the description of the present invention, it should be noted that the terms "connected" and "connected" are to be understood broadly, and may be, for example, a fixed connection, a detachable connection, or an integral, unless otherwise explicitly defined and defined. Ground connection; it can be mechanical connection or electrical connection; it can be directly connected or indirectly connected through an intermediate medium. The specific meanings of the above terms in the present invention can be understood by those skilled in the art in light of the specific circumstances. Further, in the description of the present invention, the meaning of "a plurality" is two or more unless otherwise specified.

Any process or method description in the flowchart or otherwise described herein may be understood to include a Modules, segments or portions of code of one or more executable instructions for implementing steps of a particular logical function or process, and the scope of preferred embodiments of the invention includes additional implementations in which The order of discussion includes performing functions in a substantially simultaneous manner or in the reverse order, depending on the functionality involved, which should be understood by those skilled in the art to which the present invention pertains.

FIG. 1 is a schematic flow chart of a method for processing mass population characteristic data according to an embodiment of the present invention. As shown in the figure, the method for processing the mass demographic data includes:

Step S101: setting a crowd number and a feature tag.

In this embodiment, the mass population characteristic data may include identity information such as gender, age, occupation, household registration, or other data information. Referring to FIG. 2, after obtaining the mass population characteristic data, a population number and a feature tag are preset, and each individual group corresponds to a unique population number and is arranged in an uninterrupted order, and the feature tag can be based on the characteristics of the feature data. Practical applications require flexible settings, such as male, female, 25-30 years old, undergraduate, single, and staff.

Step S102: Establish binary feature data corresponding to each feature tag according to the crowd number.

2 is a schematic diagram of binary feature data in an embodiment of the present invention. As shown in the figure, the binary feature data is stored based on the feature tag setting, and the mass population feature data respectively identifies the feature tag in the form of 0 or 1, and 1 identifies the feature having the feature tag, and the 0 flag The feature tag does not have the feature tag, so that each feature tag stores the feature identifier of 0 or 1 in the order corresponding to the crowd number, and forms the audience dot matrix data of each feature tag in the form of binary data, which can fully utilize the binary efficient The computational features can realize the rapid analysis and processing of massive population feature data, and realize the distributed storage and retrieval of massive population feature data based on feature tags, support vertical cutting, facilitate the introduction of distributed computing and support clustering processing.

Step S103: Convert the binary feature data corresponding to each feature tag that is created into hexadecimal feature data.

FIG. 3 is a schematic diagram of hexadecimal feature data in an embodiment of the present invention. In this embodiment, according to the computer-based conversion rule, the binary feature data corresponding to each feature tag is divided into 64 units as a unit, and converted into hexadecimal feature data, as shown in the figure. The feature tag TagCode includes feature tags such as male, woman, married, bachelor, and student, and each of the feature tags forms an AudienceBool corresponding to the hexadecimal form of the crowd number sequence, and converts the binary data into hexadecimal data. , saving space for the storage of characteristic data of massive people.

Step S104: Store the hexadecimal feature data corresponding to each feature tag converted.

In the method for processing massive population feature data of the embodiment, the binary feature data corresponding to each feature tag is established according to the population number, and the binary feature data corresponding to each feature tag is converted into hexadecimal feature data. Storage, based on feature tags, efficiently realize distributed processing, storage and retrieval of massive population feature data, improve the processing efficiency of massive population feature data, and save data storage space.

FIG. 4 is a schematic flow chart of a method for processing mass population characteristic data according to another embodiment of the present invention. As shown in the figure, the method for processing the mass demographic data includes:

Step S201: setting a crowd number and a feature tag.

Step S202: Establish binary feature data corresponding to each feature tag according to the crowd number.

Step S203: Convert the binary feature data corresponding to each feature tag that is created into hexadecimal feature data.

Step S204: Store the hexadecimal feature data corresponding to each feature tag of the conversion.

Step S205: Generate a query logic expression according to the feature tag of the selected query.

In this embodiment, when the feature retrieval query needs to be performed on the mass demographic data stored in the foregoing method embodiment, it is first necessary to generate a query logic expression according to the actual retrieval query requirement. As shown in FIG. 5, the generating a query logic expression according to the feature tag of the selected query includes:

Step S301: Generate a logical OR operation expression according to the feature tag of the selected query.

Step S302: Generate a logical AND operation expression according to the feature tag of the selected query.

Step S303: synthesize the generated logical OR operation expression and the logical AND operation expression into a query logic expression.

For example, in this embodiment, a logical OR operation expression and a logical AND operation expression are generated according to the feature tag of the selected query, and the query logical expression (Tag1||Tag2..Tagn)&&(Tag1||Tag2. .Tagn), a logical expression that generates a specific feature query on the client.

Step S206: Perform a logical operation on the hexadecimal feature data corresponding to the feature tag according to the generated logical expression to obtain a binary string.

In this embodiment, referring to FIG. 6, the logic function is performed on the hexadecimal feature data corresponding to the feature tag according to the generated logical expression to obtain a binary string, including:

Step S401: Group the generated logical expressions.

Step S402: Acquire hexadecimal feature data corresponding to the feature tag of the packet and convert it into long type data.

Step S403: Perform a Boolean operation on the long type data packet corresponding to each group feature tag.

Step S404: Convert the result of all grouping Boolean operations into a binary string.

In this embodiment, after obtaining the query logic expression, the query logic expression is split and grouped to obtain an OR operation group (Tag1||Tag2||..Tagn) and (Tag1||Tag2||. .Tagn), then obtaining the hexadecimal feature data corresponding to the feature tag according to the selected query tag and converting it into long type data, and then performing a Boolean operation on the long type data packet corresponding to each group feature tag, and finally all The result of the grouping Boolean operation is converted to a binary string. In this embodiment, before performing the query operation process, the hexadecimal feature data corresponding to each feature tag is converted into long type data in the memory, and the long type data can facilitate Boolean operations, thereby improving a large number of people. The efficiency of feature data processing.

Step S207: Calculate the number of 1s in the acquired binary string, and use the calculated quantity as the audience number of the feature tag of the query.

In this embodiment, the number of 1s in the binary string is obtained by calculation, and the number of the audiences with the features of the specific feature tags in the mass population is obtained, thereby realizing efficient distributed computing processing and improving the processing and query of the mass data of the mass population. Work efficiency.

In some embodiments of the present invention, the method for processing the mass demographic data further includes: sending the calculated number of the audience of the feature tag to the query end, so that the user can directly obtain the query directly at the query end. As a result, the work efficiency of the mass data processing of the mass population is improved.

FIG. 7 is a schematic structural diagram of a processing system for mass population characteristic data according to still another embodiment of the present invention. As shown in the above, based on the foregoing method embodiment, the processing system 100 for mass demographic data includes a setting module 10, a feature data establishing module 20, a conversion module 30, and a storage module 40.

In this embodiment, the mass population characteristic data may include identity information such as gender, age, occupation, household registration, or other data information. After the mass demographic data is obtained, the setting module 10 pre-sets a population number and a feature tag, and each crowd individual corresponds to a unique crowd number and is arranged in an uninterrupted order, and the feature tag can be based on the characteristics of the feature data. And the actual application needs to be flexible, such as can be set to male, female, 25-30 years old, undergraduate, single, staff and so on.

The feature data establishing module 20 is configured to establish binary feature data corresponding to each feature tag according to the crowd number set by the setting module 10. The binary feature data is stored based on the feature tag setting, wherein the mass population feature data respectively identifies the feature tag in the form of 0 or 1, and 1 identifies a feature having the feature tag, and the 0 flag does not have the feature. The feature of the tag is such that each feature tag established by the feature data establishing module 20 stores the feature identifier of 0 or 1 in the order corresponding to the crowd number, and forms the dot matrix data of each feature tag in the form of binary data, which is sufficient Using binary efficient computing features, it can realize rapid analysis and processing of massive population feature data, and realize distributed storage of massive population feature data based on feature tags. And retrieval, support for vertical cutting, easy to introduce distributed computing and support clustering.

In this embodiment, the conversion module 30 divides the binary feature data corresponding to each feature tag established by the feature data establishing module 20 into 64 units as a unit according to a computer-based conversion rule, and converts it into a hexadecimal The feature data, for example, the feature tag TagCode includes feature tags such as male, woman, married, bachelor, and student, and each feature tag forms an AudienceBool corresponding to the feature identifier of the hexadecimal form of the crowd number sequence, and compresses the binary data. Converted to hexadecimal data, saving space for large populations of feature data storage.

In the processing system 100 of the mass demographic data of the present embodiment, the feature data establishing module 20 creates binary feature data corresponding to each feature tag according to the crowd number, and the conversion module 30 converts the binary feature data corresponding to each feature tag that is established. For the hexadecimal feature data, the storage module 40 stores it, and implements distributed processing, storage and retrieval of mass population feature data efficiently based on the feature tag, improves the processing efficiency of the mass population feature data, and saves data storage. space.

FIG. 8 is a schematic structural diagram of a processing system for mass population characteristic data according to still another embodiment of the present invention. As shown, the processing system 100 for mass demographic data includes a setup module 10, a feature data creation module 20, a conversion module 30, a storage module 40, a query logic expression generation module 50, a logic operation module 60, and a calculation module. 70.

In the embodiment, when the feature retrieval query is performed on the mass demographic data stored in the processing system 100 of the mass demographic data, the query logic expression generating module 50 generates a query according to the actual retrieval query feature tag requirement. Logical expression. As shown in FIG. 9, the query logic expression generating module 50 includes a first generating unit 501, a second generating unit 502, and a synthesizing unit 503.

The first generating unit 501 is configured to generate a logical OR operation expression according to the feature tag of the selected query; the second generating unit 502 is configured to generate a logical AND operation expression according to the feature tag of the selected query; The synthesizing unit 503 is configured to synthesize the logical OR operation expression and the logical AND operation expression generated by the first generation unit 501 and the second generation unit 502 into a query logic expression.

For example, the first generating unit 501 and the second generating unit 502 generate a logical OR operation expression and a logical AND operation expression according to the feature tag of the selected query, and the synthesizing unit 503 synthesizes the query logical expression (Tag1||Tag2.. Tagn)&&(Tag1||Tag2..Tagn), which generates a logical expression for a specific feature query on the client.

In this embodiment, the logic operation module 60 is configured to perform a logical operation on the hexadecimal feature data corresponding to the feature tag according to the logic expression generated by the query logic expression generation module 50 to obtain a binary character. string. Referring to FIG. 10, the logic operation module 60 includes a grouping unit 601, a long type data conversion unit 602, a grouping operation unit 603, and a binary string conversion unit 604.

The grouping unit 601 is configured to group the logical expressions generated by the query logic expression generating module, and the long type data converting unit 602 is configured to acquire the feature tags corresponding to the grouping unit 601 grouping. The hexadecimal feature data is converted into long type data; the grouping operation unit 603 is configured to perform a Boolean operation on the long type data packet corresponding to each group feature tag converted by the long type data conversion unit 602; The binary string conversion unit 604 is configured to convert the result of all grouping Boolean operations of the grouping operation unit 603 into a binary string.

For example, after the query logic expression generating module 50 obtains the query logic expression, the grouping unit 601 splits the query logic expression to obtain an OR operation group (Tag1||Tag2||.. Tagn) and (Tag1||Tag2||..Tagn), then the long type data conversion unit 602 obtains the hexadecimal feature data corresponding to the feature tag according to the selected query tag and converts it into long type data, The packet operation unit 603 performs a Boolean operation on the long type data packet corresponding to each packet feature tag, and finally the binary string conversion unit 604 converts the result of all the packet Boolean operations into a binary string. In this embodiment, before the packet operation unit 603 performs the query operation process, the hexadecimal feature data corresponding to each feature tag is converted into long type data in the memory, and the long type data can facilitate the Boolean operation. , improve the efficiency of large-scale population feature data processing.

In this embodiment, the calculation module 70 is configured to calculate the number of 1 in the binary string obtained by the logic operation module 60, and use the calculated quantity as the audience number of the feature tag of the query. The calculation module 70 obtains the number of the 1s in the binary string to obtain the number of the audiences with the characteristics of the specific feature tags in the mass population, realizes the efficient distributed computing process, and improves the processing and query of the mass population characteristic data. effectiveness.

In some embodiments of the present invention, the processing system 100 for the mass demographic data further includes a sending module, configured to send the audience number of the feature tag calculated by the calculating module 70 to the query end, so that the user is in the The query end can directly obtain the query result quickly, and improves the work efficiency of the mass data feature data processing.

It should be understood that portions of the invention may be implemented in hardware, software, firmware or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques well known in the art: having logic gates for implementing logic functions on data signals. Discrete logic circuit, ASIC with suitable combination logic gate, programmable gate array (PGA), Field Programmable Gate Array (FPGA), etc.

In the description of the specification, the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like are intended to mean the specific features described in connection with the embodiments or examples, A structure, material or feature is included in at least one embodiment or example of the invention. In the present specification, the schematic representation of the above terms does not necessarily mean the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples.

While the embodiments of the present invention have been shown and described, the embodiments of the invention may The scope of the invention is defined by the claims and their equivalents.

Claims (10)

A method for processing massive population feature data, comprising: Set the crowd number and feature label; Establishing binary feature data corresponding to each feature tag according to the crowd number; Converting binary feature data corresponding to each feature tag to hexadecimal feature data; The hexadecimal feature data corresponding to each feature tag converted is stored. The method for processing massive population feature data according to claim 1, further comprising: Generating a query logic expression according to the feature tag of the selected query; Obtaining a binary string by performing a logical operation on the hexadecimal feature data corresponding to the feature tag according to the generated logical expression; Calculate the number of 1s in the obtained binary string and use the calculated quantity as the audience number of the feature tag of the query. The method for processing a massive population feature data according to claim 2, wherein the generating a query logic expression according to the feature tag of the selected query comprises: Generating a logical OR operation expression according to the feature tag of the selected query; Generating logical AND operation expressions based on the feature tags of the selected query; The generated logical OR operation expression and the logical AND operation expression are combined into a query logic expression. The method for processing massive population feature data according to claim 2, wherein the logical operation of the hexadecimal feature data corresponding to the feature tag according to the generated logical expression to obtain a binary string comprises: Group the generated logical expressions; Obtaining hexadecimal feature data corresponding to the feature tag of the group and converting it into long type data; Boolean operations on long type data packets corresponding to each grouping feature tag; Converts the result of all grouped Boolean operations to a binary string. The method for processing massive population feature data according to claim 2, further comprising: The calculated audience number of the feature tag is sent to the query end. A processing system for massive population characteristic data, comprising: a setting module for setting a crowd number and a feature tag; a feature data establishing module, configured to establish binary feature data corresponding to each feature tag according to the crowd number set by the setting module; a conversion module, configured to convert binary feature data corresponding to each feature tag established by the feature data establishing module into hexadecimal feature data; And a storage module, configured to store the hexadecimal feature data corresponding to each feature tag converted by the conversion module. The system for processing massive population characteristic data according to claim 6, further comprising: Querying a logical expression generating module, configured to generate a query logic expression according to the feature tag of the selected query; a logic operation module, configured to perform a logic operation on the hexadecimal feature data corresponding to the feature tag according to the logic expression generated by the query logic expression generation module to obtain a binary character string; The calculation module is configured to calculate the number of 1s in the binary string obtained by the logic operation module, and use the calculated quantity as the audience number of the feature tag of the query. The system for processing massive population feature data according to claim 7, wherein the query logic expression generating module comprises: a first generating unit, configured to generate a logical OR operation expression according to the feature tag of the selected query; a second generating unit, configured to generate a logical AND operation expression according to the feature tag of the selected query; And a synthesizing unit, configured to synthesize the logical OR operation expression and the logical AND operation expression generated by the first generation unit and the second generation unit into a query logic expression. The processing system for mass population characteristic data according to claim 7, wherein the logic operation module comprises: a grouping unit, configured to group the logical expressions generated by the query logic expression generating module; a long type data conversion unit, configured to acquire hexadecimal feature data corresponding to the feature tag of the grouping unit group and convert the data into long type data; a grouping operation unit, configured to perform a Boolean operation on the long type data packet corresponding to each group feature tag converted by the long type data conversion unit; A binary string conversion unit for converting the result of all grouping Boolean operations of the grouping operation unit into a binary string. The system for processing mass demographic data according to claim 7, further comprising a sending module, configured to send the number of audiences of the feature tags calculated by the calculating module to the querying end.

Images