CN104679889A - Big data processing-oriented data storage method and device - Google Patents

Abstract

The invention provides a big data processing-oriented data storage method and a big data processing-oriented data storage device. The method comprises the following steps: setting entity objects, and establishing a hyper archive data table; establishing a corresponding relationship between each entity object and data rows in the hyper archive data table, and setting metadata corresponding to each column in the data rows; for a current entity object to be stored, searching for the data row corresponding to the current entity object in the hyper archive data table; storing all data related to the entity object into the found data row and in the column corresponding to the metadata in the data row in the hyper archive data table according to an attribute of the entity object. According to the scheme, big data storage efficiency can be improved.

Description

A data storage method and device for big data processing

technical field

The invention relates to the technical field of network communication, in particular to a data storage method and device for big data processing.

Background technique

With the continuous enrichment and improvement of data collection methods, more and more industry data has been accumulated. The scale of data has grown to the level of big data (hundreds of GB, TB, or even PB) that cannot be carried by the traditional software industry. In the big data scenario, how to store big data has become an important computing problem.

Currently, relational databases can be used to store big data. For example, multiple data with associated relationships are stored in different data tables of different databases, and the relationship between the data stored in the different data is recorded, so as to associate the various data.

It can be seen that the current practice of storing each data in different data tables of different databases using association relationships is loose, and the association relationship must be reflected by a relational database. For big data, this loose storage of data and use of associations The practice of relationally recording data in different data tables greatly reduces the efficiency of data storage, and further reduces the efficiency of subsequent search and maintenance.

Contents of the invention

The invention provides a data storage method and device for big data processing, which can improve data storage efficiency.

A data storage method oriented to big data processing, setting entity objects, and establishing a super file data table; establishing the corresponding relationship between each entity object and the data row in the super file data table, and setting the corresponding relationship of each column in the data row metadata;

For the current entity object to be stored, find the data row in the super file data table corresponding to the current entity object;

All data related to the entity object is stored in the data row in the found super archive data table according to its attributes, and is located in the corresponding metadata column in the data row.

The establishment of the corresponding relationship between each entity object and the data row in the super file data table includes: establishing the corresponding relationship between each entity object and a data row in the super file data table;

The storing all the data related to the entity object into the data row in the found super file data table includes: storing all the data related to the entity object into a row in the found super file data table.

Elements in the described super file data table include: data table (SupDocTable), key (SupDocKey), key component (SupDocKeyComponent), column family (SupDocFamily), GROUP column family (SupDocGroupFamily), MAP column family (SupDocMapFamily), column ( Any one or more of SupDocColumn).

When the elements in the super file data table include column clusters; set the subject field corresponding to each column cluster;

The storing all the data related to the entity object into the data row in the found super file data table includes: storing the data belonging to the same subject domain among all the data related to the entity object into the column cluster corresponding to the subject domain In , the data stored on each column in the column family is a simple data type or a complex data type of an aggregation unit.

The method further includes:

In the data row corresponding to each entity object, set the calculation result storage column;

Using the existing specified data in the data row corresponding to the first entity object in the super file data table, deduce and calculate the calculation result corresponding to the first entity object, and write the calculation result into the data corresponding to the first entity object The calculation result is stored in a row column.

The method further includes:

Set the summary aggregation calculation result storage table;

Using the data on all the rows of the super archive data table to perform summary aggregation calculation, and write the calculation results into the summary aggregation calculation result storage table.

A data storage device for big data processing, comprising:

The setting unit is used to set the entity object and establish the super file data table; establish the corresponding relationship between each entity object and the data row in the super file data table;

The search unit is used to search for the data row in the super file data table corresponding to the current entity object for the current entity object to be stored, and set the metadata corresponding to each column in the data row;

The storage execution unit is used to store all the data related to the entity object into the data row in the found super archive data table according to its attributes, and locate it on the column corresponding to the metadata in the data row.

The setting unit is used to set the elements in the super file data table including: data table (SupDocTable), key (SupDocKey), key component (SupDocKeyComponent), column family (SupDocFamily), GROUP column family (SupDocGroupFamily), MAP Any one or more of column family (SupDocMapFamily) and column (SupDocColumn).

The setting unit sets the subject field corresponding to each column cluster when setting the elements in the super file data table to include column clusters;

The storage execution unit stores the data belonging to the same subject domain among all the data related to the entity object into the column cluster corresponding to the subject domain, and the data stored on each column in the column cluster is a simple data type or an aggregation unit complex data types.

The device also includes: a first calculation unit, wherein the setting unit is further configured to set a calculation result storage column in the data row corresponding to each entity object; The existing designated data in the data row corresponding to the first entity object in the super file data table, deduce and calculate the calculation result corresponding to the first entity object, and write the calculation result into the data row corresponding to the first entity object The calculation result storage column in ;

and / or,

The device also includes: a second calculation unit, wherein the setting unit is further used to set a summary aggregation calculation result storage table; the second calculation unit is used to use all rows of the super file data table The data is aggregated and calculated, and the calculation result is written into the summary and aggregated calculation result storage table.

It can be seen that the embodiment of the present invention provides a data storage method and device oriented to big data processing, setting a super archive data table, and facing entity objects, one entity object has one super archive, and all data related to an entity object is stored in a centralized manner. The data row corresponding to the entity object in the super archive data table realizes the holographic storage of the data of an entity object and improves the storage efficiency.

Description of drawings

Fig. 1 is a flow chart of a data storage method oriented to big data processing in an embodiment of the present invention.

Fig. 2 is a flowchart of a data storage method oriented to big data processing in another embodiment of the present invention.

Fig. 3 is a schematic diagram of elements of a super archive data model in one embodiment of the present invention.

Fig. 4 is a schematic diagram of two MapReduce operations on the super archive data table in one embodiment of the present invention.

Fig. 5 is a schematic structural diagram of a data storage device for big data processing according to an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

One embodiment of the present invention proposes a data storage method for big data processing, referring to Fig. 1, the method includes:

Step 101: Set entity objects and create a super file data table.

Step 102: Establish the corresponding relationship between each entity object and the data row in the super file data table, and set the metadata corresponding to each column in the data row.

Step 103: For the current entity object to be stored, look up the data row in the super archive data table corresponding to the current entity object.

Step 104: Store all data related to the entity object in the data row in the found super file data table.

It can be seen that this embodiment of the present invention provides a data storage method oriented to big data processing, setting a super file data table, and facing entity objects, one entity object is one super file, and all data related to an entity object are stored in a centralized manner. The data row corresponding to the entity object in the super archive data table realizes the holographic storage of the data of an entity object and improves the storage efficiency.

In an embodiment of the present invention, each entity object may correspond to a data row in the super file data table.

When specifically setting the super file data table, in one embodiment of the present invention, the elements in the table include: data table (SupDocTable), key (SupDocKey), key component (SupDocKeyComponent), column cluster (SupDocFamily), GROUP column cluster Any one or more of (SupDocGroupFamily), MAP column family (SupDocMapFamily), column (SupDocColumn).

In one embodiment of the present invention, the elements in the super file data table include column clusters, and each column cluster corresponds to a subject field; at this time

The storing all the data related to the entity object into the data row in the found super file data table includes: storing the data belonging to the same subject domain among all the data related to the entity object into the column cluster corresponding to the subject domain In , the data stored on each column in the column family is a simple data type or a complex data type of an aggregation unit. This method can more effectively store the data belonging to the same subject of an entity object in a centralized manner.

In one embodiment of the present invention, since the data of the entity object in the super file data table is stored in a data row, it is convenient to perform subsequent calculation processing of the corresponding entity object using the data. for example,

The first type of computational processing consists of:

In the data row corresponding to each entity object, set the calculation result storage column;

Using the existing specified data in the data row corresponding to the first entity object in the super file data table, deduce and calculate the calculation result corresponding to the first entity object, and write the calculation result into the data corresponding to the first entity object The calculation result is stored in a row column.

The second calculation process includes:

Set the summary aggregation calculation result storage table;

Using the data on all the rows of the super archive data table to perform summary aggregation calculation, and write the calculation results into the summary aggregation calculation result storage table.

Another embodiment of the present invention proposes a data storage method for big data processing, referring to Fig. 2, including:

Step 201: Set entity objects.

Here, the entity object can be, for example, a person; a car; a case, etc.

Step 202: Create a super file data table, establish a corresponding relationship between each entity object and a data row in the super file data table, and set metadata corresponding to each column in the data row.

Here, referring to Fig. 3, the elements in the super file data table can include: data table (SupDocTable), key (SupDocKey), key component (SupDocKeyComponent), column family (SupDocFamily), GROUP column family (SupDocGroupFamily), MAP column family ( SupDocMapFamily), Column(SupDocColumn).

For example, through the processing of this step, the entity object "case A" corresponds to the first data row in the super file data table, which is recorded as data row 1, and each column element of data row 1 corresponds to a piece of metadata, such as data row 1 Including metadata "time", "location", "classification", "relationship of persons involved in the case", "target closing time", "case analysis", etc.

Step 203: Set column clusters in the data row, each column cluster includes more than one piece of metadata, and set the subject field corresponding to each column cluster.

For example, in a data row, metadata "time" and "location" can be set to belong to the same column family, and the subject field corresponding to the column family is basic information.

Step 204: In the data row corresponding to each entity object, set the calculation result storage column.

Step 205: Set up a table for storing summary and aggregation calculation results.

After this step is executed, the preset value processing before the actual storage of big data is completed.

Wherein, each element (ie column name) in each data row is also determined.

In one embodiment of the present invention, according to the super file data model element, the super file Schema definition language can be designed, and the content of the data table definition file can be mapped to an executable Apache HBaseSchema definition statement, so as to realize Shell commands and Web page data modeling tools, Thereby, the processing procedure of the above steps is realized.

Step 206: Denote the current entity object to be stored as entity object A, and search for the data row in the super archive data table corresponding to entity object A.

For example, entity object A corresponds to the second data row.

Step 207: Store all data related to the entity object in the data row in the found super-archive data table according to its attributes, and place it in the corresponding metadata column in the data row.

Step 208: When storing into the data row, store the data belonging to the same subject domain among all the data related to the entity object into the column family corresponding to the subject domain, and locate on the column corresponding to the metadata in the column family.

In this step, the data stored on each column in the column cluster is a simple data type or a complex data type of an aggregation unit (the complex type data structure is defined through json).

Step 209: When it is necessary to obtain a certain calculation result for the entity object 1, use the existing specified data in the data row corresponding to the entity object 1 in the super file data table to derive and calculate the calculation result 1 corresponding to the first entity object.

Step 210: Write the calculation result 1 into the calculation result storage column in the data row corresponding to the entity object 1.

Step 211: Use the data on all the rows of the super-archive data table to perform summary aggregation calculation, and write the calculation result 2 into the summary aggregation calculation result storage table.

The above two calculations in step 209 to step 211 can be implemented based on the MapReduce operation method. See Figure 4. Based on the MapReduce computing framework, the first type is the derivation calculation (MR-derivator) operation. On each single row of the entire table, the new information of the entity object is calculated using existing data and written into the row. A certain column; the second type is the aggregation calculation (MR-aggregator) operation, which performs summary aggregation calculations on all rows of the entire table, and writes the results to another data table (other super-archive tables or general-type tables).

The process shown in Figure 2 above can be implemented based on NOSQL column storage technology such as Apache HBase, and the column names in the column cluster are determined in advance.

An embodiment of the present invention also proposes a data storage device for big data processing, see FIG. 5, including:

The setting unit 501 is used to set the entity object and establish the super file data table; establish the corresponding relationship between each entity object and the data row in the super file data table;

A search unit 502, configured to search for the data row in the super file data table corresponding to the current entity object for the current entity object to be stored;

The storage execution unit 503 is configured to store all data related to the entity object into the data row in the super file data table found by the search unit.

In one embodiment of the present invention, the setting unit is used to set the elements in the super file data table including: data table (SupDocTable), key (SupDocKey), key component (SupDocKeyComponent), column family (SupDocFamily), Any one or more of GROUP column family (SupDocGroupFamily), MAP column family (SupDocMapFamily), column (SupDocColumn).

In one embodiment of the present invention, the setting unit sets the subject field corresponding to each column cluster when setting the elements in the super archive data table to include column clusters;

The storage execution unit stores the data belonging to the same subject domain among all the data related to the entity object into the column cluster corresponding to the subject domain, and the data stored on each column in the column cluster is a simple data type or an aggregation unit complex data types.

In an embodiment of the present invention, the device further includes: a first calculation unit, wherein the setting unit is further configured to set a calculation result storage column in the data row corresponding to each entity object; As calculated, it is used to deduce and calculate the calculation result corresponding to the first entity object by using the existing specified data in the data row corresponding to the first entity object in the super file data table, and write the calculation result into the The calculation result storage column in the data row corresponding to the first entity object.

In an embodiment of the present invention, the device further includes: a second calculation unit, wherein the setting unit is further used to set a summary aggregation calculation result storage table; the second calculation unit is used to use the super The data on all rows of the archive data table is aggregated and calculated, and the calculation results are written into the aggregated calculation result storage table.

Various embodiments of the present invention have at least the following beneficial effects:

1. Utilize the super file data table, and it is oriented to entity objects, one entity object is one super file, and all the data related to an entity object is centrally stored in the data row corresponding to the entity object in the super file data table, for example, in one row, In this way, the holographic storage of the data of an entity object is realized, and the storage efficiency is improved.

2. Since all the data of an entity object is centrally stored in the data row corresponding to the entity object in the super archive data table, it is necessary to show the user all the data of the entity object such as a case (the time when the case occurred) , place, person, case analysis, vehicle information, etc.) can realize the holographic display very conveniently, without the process of searching in different data tables of the relational database according to the association relationship between the pre-recorded data in the prior art, so , improving the efficiency of big data search and display.

3. According to the idea of "one entity object, one super file", the data model organizes and aggregates data centered on the entity object, displays the relevant information of each entity object in an all-round way, and supports dynamic expansion of attributes and multi-dimensional data processing.

4. Since the metadata corresponding to each column in the data row is set, the metadata management of the super file data table can be realized, and the super file Schema information can be displayed for data access users.

5. Through the defined two types of basic computing processing such as MapReduce processing, and the different arrangements and combinations of these two computing processing, pipeline-type big data processing and computing on super-archive datasets can be realized.

6. The data modeling method and data modeling tool enable the super archive data administrator to easily define and expand the data table Schema, as well as the version management of the Schema definition.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or sequence. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional same elements in the process, method, article or apparatus comprising said element.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (10)

1. towards a date storage method for large data processing, it is characterized in that, entity object is set, and set up super file data table; Set up the corresponding relation of data line in each entity object and described super file data table, and the metadata that in setting data row, each row is corresponding;

For the current entity object that need store, search the data line in the super file data table corresponding with described current entity object;

All data of being correlated with by entity object, and to be arranged on the row of these data line corresponding element data in the data line in the described super file data table found according to its property store.

2. method according to claim 1, it is characterized in that, the described corresponding relation setting up data line in each entity object and described super file data table comprises: the corresponding relation setting up a data line in each entity object and described super file data table;

Data line in the super file data table found described in described all data of being correlated with by entity object are stored into comprises: in a line in the super file data table found described in all data of being correlated with by entity object are stored into.

3. method according to claim 1, it is characterized in that, the element in described super file data table comprises: tables of data (SupDocTable), key (SupDocKey), key asembly (SupDocKeyComponent), row bunch (SupDocFamily), GROUP row bunch (SupDocGroupFamily), MAP row bunch (SupDocMapFamily), row (SupDocColumn) in any one or more.

4. method according to claim 3, is characterized in that, comprises further: in data line, arrange row bunch, and each row bunch comprises more than one described metadata, and arranges each row bunch corresponding subject area;

Data line in the super file data table found described in described all data of being correlated with by entity object are stored into comprises: the data belonging to same subject territory in all data of being correlated with by entity object are stored into the corresponding element data place in row corresponding to this subject area bunch, and the data that each row in row bunch store are the complex data type of simple data type or polymerized unit.

5., according to described method arbitrary in Claims 1-4, it is characterized in that, the method comprises further:

In the data line that each entity object is corresponding, set out result of calculation memory row;

Utilize the existing specific data in the data line of corresponding first instance object in described super file data table, derive and calculate the result of calculation of corresponding first instance object, this result of calculation is write the described result of calculation memory row in data line corresponding to described first instance object.

6., according to described method arbitrary in Claims 1-4, it is characterized in that, the method comprises further:

Setting gathers Aggregation computation result storage list;

Utilize the data on all row of described super file data table to carry out gathering Aggregation computation, described in result of calculation being write, gather Aggregation computation result storage list.

7. towards a data storage device for large data processing, it is characterized in that, comprising:

Setting unit, for arranging entity object, and sets up super file data table; Set up the corresponding relation of data line in each entity object and described super file data table, and the metadata that in setting data row, each row is corresponding;

Search unit, for for the current entity object that need store, search the data line in the super file data table corresponding with described current entity object;

Store performance element, for all data that entity object is correlated with according to its property store in the data line in the described super file data table found, and to be arranged on the row of these data line corresponding element data.

8. device according to claim 7, it is characterized in that, described setting unit, comprises for the element arranged in described super file data table: tables of data (SupDocTable), key (SupDocKey), key asembly (SupDocKeyComponent), row bunch (SupDocFamily), GROUP row bunch (SupDocGroupFamily), MAP row bunch (SupDocMapFamily), row (SupDocColumn) in any one or more.

9. device according to claim 8, is characterized in that, described setting unit arranges row bunch in data line, and each row bunch comprises more than one described metadata, and arranges each row bunch corresponding subject area;

The data belonging to same subject territory in all data that entity object is correlated with by described storage performance element are stored into the corresponding element data place in row corresponding to this subject area bunch, and the data that each row in row bunch store are the complex data type of simple data type or polymerized unit.

10., according to described device arbitrary in claim 7 to 9, it is characterized in that,

Described device also comprises: the first computing unit, and wherein, described setting unit, is further used for, in the data line that each entity object is corresponding, setting out result of calculation memory row; Described first calculate such, for utilizing the existing specific data in the data line of corresponding first instance object in described super file data table, derive and calculate the result of calculation of corresponding first instance object, this result of calculation is write the described result of calculation memory row in data line corresponding to described first instance object;

And/or,

Described device also comprises: the second computing unit, and wherein, described setting unit, is further used for setting and gathers Aggregation computation result storage list; Described second computing unit, the data on all row utilizing described super file data table carry out gathering Aggregation computation, gather Aggregation computation result storage list described in result of calculation being write.