CN105701096A

CN105701096A - Index generation method, data inquiry method, index generation device, data inquiry device and system

Info

Publication number: CN105701096A
Application number: CN201410685565.8A
Authority: CN
Inventors: 姚永刚; 王才平; 王周; 朱凯; 林育佳; 刘忠真; 吴泽君
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2014-11-25
Filing date: 2014-11-25
Publication date: 2016-06-22

Abstract

The invention discloses an index generation method, a data inquiry method, an index generation device, a data inquiry device and a system, and belongs to the field of data storage. The index generation method comprises the following steps of obtaining a data file; generating corresponding index information for each data record; and for each data record in the data file, storing the corresponding index information into an index file in a hierarchy index directory in a classified way according to the attribute information of the data record, wherein the hierarchy index directory includes n index files positioned in different directories, the n is greater than or equal to 1, and the index files in each directory correspond to one group of attribute information. The index information is dispersed into different directories of the hierarchy index directory according to the attribute information of the data record; the problem of low inquiry speed during the inquiry by using the existing index file when mass data is stored is solved; and the effect that the fast inquiry and processing can be provided even when mass data is stored is achieved.

Description

Index generation method, data query method, Apparatus and system

Technical field

The present invention relates to big data processing field, particularly to a kind of index generation method, data query method, Apparatus and system。

Background technology

Along with operation system function from strength to strength, in operation system, stored data also get more and more。

Current storage system typically sets up index file, such as a corresponding index file of data file, or, a corresponding index file of data base。When needs inquire about data record within the storage system, first according to search index in index key indexed file, this index describes target data record side-play amount in the data file, then target data record is extracted in the data file according to the index that inquires, and using this target data record as Query Result。

In the process realizing the present invention, inventor have found that prior art at least there is problems in that when storage has mass data, inquiry velocity when using current index file to inquire about is slower, it is impossible to meet quick search scene。

Summary of the invention

Inquiry velocity in order to solve when using current index file to inquire about is slower, it is impossible to the problem meeting the inquiry scene that requirement of real-time is higher, embodiments provides a kind of index generation method, data query method, Apparatus and system。Described technical scheme is as follows:

First aspect, embodiments provides a kind of index generation method, and described method includes:

Obtaining data file, described data file includes at least one data record；

Generating corresponding index information for every described data record, described index information is for indicating described data record storage position in described data file；

For the every data record in described data file, in the index file that corresponding described index information classification is stored to level index list by the attribute information according to described data record, described level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue。

Second aspect, embodiments provides a kind of data query method, and described method includes:

Receive inquiry request；

According to extracting index information at least one index file from level index list of described inquiry request, described level index list includes n the index file being positioned under different directories, n >=1, the index file one group of attribute information of correspondence under each catalogue；

Data record is obtained in the data file according to described index information。

The third aspect, embodiments provides a kind of index and generates device, and described device includes:

Data file acquisition module, is used for obtaining data file, and described data file includes at least one data record；

Index information generation module, for generating corresponding index information for the described data record of every in described data file acquisition module, described index information is for indicating described data record storage position in described data file；

Index information memory module, for for the every data record in described data file, in the index file that the described index information classification that corresponding described index information generation module is generated by the attribute information according to described data record stores to level index list, described level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue。

Fourth aspect, embodiments provides a kind of data query arrangement, and described device includes:

Inquiry request receiver module, is used for receiving inquiry request；

Index information extraction module, for at least one index file from level index list of the described inquiry request that receives according to described inquiry request receiver module extracts index information, described level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue；

Data record acquisition module, obtains data record in the data file for the described index information extracted according to described index information extraction module。

In sum, the index generation method that the embodiment of the present invention provides, by when index generates, obtain data file, corresponding index information is generated for every data record, for the every data record in data file, further according in the index file that corresponding index information classification is stored to level index list by the attribute information of data record, wherein, level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue；Solve when storage has mass data, the problem that inquiry velocity when using current index file to inquire about is slower；Even if having reached storage when having mass data, it is also possible to provided the effect of quick search process by multiple index files store according to different attribute information classification。

Accompanying drawing explanation

In order to be illustrated more clearly that the technical scheme in the embodiment of the present invention, below the accompanying drawing used required during embodiment is described is briefly described, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the premise not paying creative work, it is also possible to obtain other accompanying drawing according to these accompanying drawings。

Fig. 1 illustrates the method flow diagram of the index generation method that one embodiment of the invention provides；

Fig. 2 illustrates the method flow diagram of the data query method that one embodiment of the invention provides；

Fig. 3 A illustrates the method flow diagram of the data query method that one embodiment of the invention provides；

Fig. 3 B illustrates the structural representation of the index file that one embodiment of the invention provides；

Fig. 3 C illustrates the structural representation of the index file that another embodiment of the present invention provides；

Fig. 4 illustrates the structural representation of a kind of implementation environment involved by each embodiment of the present invention；

Fig. 5 illustrates the method flow diagram of the data query method that one embodiment of the invention provides；

Fig. 6 illustrates that the index that one embodiment of the invention provides generates the block diagram of device；

Fig. 7 illustrates that the index that one embodiment of the invention provides generates the block diagram of device；

Fig. 8 illustrates the block diagram of the data query arrangement that one embodiment of the invention provides；

Fig. 9 illustrates the block diagram of the data query arrangement that one embodiment of the invention provides；

Figure 10 illustrates that the data that one embodiment of the invention provides process the block diagram of end。

Detailed description of the invention

For making the object, technical solutions and advantages of the present invention clearly, below in conjunction with accompanying drawing, embodiment of the present invention is described further in detail。

Refer to Fig. 1, it illustrates the method flow diagram of the index generation method that one embodiment of the invention provides。The present embodiment is applied in storage system to illustrate with this index generation method, including:

Step 101, obtains data file, and this data file includes at least one data record。

Step 102, generates corresponding index information for every data record, and this index information is for indicating data record storage position in the data file。

Step 103, for the every data record in data file, in the index file that corresponding index information classification is stored to level index list by the attribute information according to data record, this level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue。

Refer to Fig. 2, it illustrates the method flow diagram of the data query method that one embodiment of the invention provides。The present embodiment can apply to illustrate in storage system with this data query method, including:

Step 201, receives inquiry request。

Storage system receives the inquiry request that client sends, and this inquiry request may include that the time range etc. belonging to service identification, data record。

Step 202, according to extracting index information at least one index file from level index list of inquiry request, this level index list includes n the index file being positioned under different directories, n >=1, the index file one group of attribute information of correspondence under each catalogue。

Step 203, obtains data record in the data file according to index information。

In sum, the data query method that the embodiment of the present invention provides, by when data query, receive inquiry request, according at least one index file from level index list of inquiry request extracts index information, obtain data record further according to index information in the data file, be stored in the index file different directories from classification and extract index information；Solve when storage has mass data, the problem that inquiry velocity when using current index file to inquire about is slower；Even if having reached storage when having mass data, it is also possible to provided the effect of quick search process by multiple index files store according to different attribute information classification。

Refer to Fig. 3 A, it illustrates the method flow diagram of the data query method that one embodiment of the invention provides。The present embodiment is applied in storage system to illustrate with this data query method, and the method includes:

Step 301, obtains data file, and this data file includes at least one data record。

In the process of the business of operation, in data, space in a newspaper can generate the data record of a rule。In data, data record can be reported storage system by space in a newspaper, storage system receive the data record that in data, space in a newspaper sends。Specifically:

Data record is made up of several fields。Difference according to business, the field that data record includes is also different。For the group chat monitoring business that business is instant social software, certain user has issued a chat message in chat group, then generating a data record, this data record includes: chat group's title, on call time, chat content, user account, the field such as user's pet name。

Wherein, in data, data record can be stored in data file by space in a newspaper in the form of text, and every data record occupies a line in the data file, and in data record, each field separates with list separator, and this list separator can be ‘ t '。

Such as, when a data record includes: time field, user's nickname field, chat group's name field, user account field time, a part of data record in a data file schematically as follows:

2014.11.1100:01:01 t Xiao Wang t travel information group t331121 t

2014.11.1200:01:11 t Xiao Zhang t ephemeris information group t331432 t

2014.11.1300:01:21 t little red t shopping information group t331254 t

2014.11.1400:01:22 t little beautiful t game exchange group t121121 t

In order to ensure the speed of inquiry, the data volume of each data file may remain in predefined size, and such as, the data record sum of each data file is less than 20,000；Or the data volume of each data file is less than 200MB etc.。That is, when in data, space in a newspaper data volume in current data file being detected reaches predetermined threshold value, a newly-built data file, and the data record received is stored in newly-built data file, and current data file is sent to storage system。

In a certain server that data file can be stored to storage system with load balancing mode by storage system。

In a kind of possible embodiment, under the in catalogue in a certain server that data file is stored to storage system with following file storage structure by storage system:

in/dcid/yyyymmdd/hh/filename。

Wherein, dcid is service identification, and yyyymmdd is date, and hh is hour, and filename is filename。

That is, data file can store to different catalogues according to the difference of attribute information, attribute information includes: the filename that time range corresponding to service identification corresponding to data file, data file, data file are corresponding。Adopt this storage organization, it is possible to improve the search efficiency of data file。

Step 302, generates corresponding index information for every data record, and this index information is for indicating data record storage position in the data file。

Storage system is after obtaining data file, it is possible to generate corresponding index information for the every data record in this data file, and this index information is for indicating data record storage position in the data file。

This step may include steps of:

First, for every data record, storage system extracts the index word segment value and this data record side-play amount in the data file that need to index in this data record。

Second, storage system generates an index information corresponding with data record according to index word segment value with data record side-play amount in the data file。

For data record A be " 2014.11.1100:01:01 t Xiao Wang t travel information group t331121 t "。This data record includes following field: date, hour, minute, second, user's pet name, chat group's title, user account。Wherein, when needing the field indexed for user's pet name。Now, data can be for the index information that this data record is set up:

Index word segment value " Xiao Wang ", data record A side-play amount " the 1234th row " in the data file。

When being other field when needing the field that indexes, other index information to set up process similar with aforesaid way, no longer repeat one by one。

Step 303, for the every data record in data file, according in the index file that corresponding index information classification is stored to level index list by the attribute information of data record。

After generating the index information that data file is corresponding, corresponding index information according to the attribute information of data record, can be classified in the index file stored to level index list by storage system。Wherein, this level index list level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue。

This step may include steps of:

The first, for every data record, the attribute information of data record is extracted。

The second, at least one index file in level index list determines corresponding with an attribute information index file。

3rd, index information corresponding for data record is stored in index file。

In a first step, storage system is after generating corresponding index information to every data record, it is possible to extract the attribute information of pieces of data record。Wherein, attribute information may include that at least one information in the filename of the data file corresponding to the delivery value of the cryptographic Hash of the index word segment value in the time range belonging to the service identification corresponding to data record, data record, the index field title in data record, data record and data record。Wherein, time range generally includes date and time information and hour minute information。

Such as, being " 2014.11.1100:01:01 t little Wang t travel information Qun t331121 t " for the data record C in data file B, the attribute information of this data record includes: date and time information " 2014.11.11 ", hour minute information " 00:01 ", index field title " user's pet name ", the filename " B " of data file, index word segment value the delivery value " MOD1 " of cryptographic Hash。

Wherein, MOD1 represents the cryptographic Hash delivery value relative to predetermined value of index word segment value " Xiao Wang "。This predetermined value can be 100, it is assumed that cryptographic Hash is 1108, then delivery value is 8。

In second step, storage system can determine index file by level index list, and this level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue。

Storage system according to the attribute information of data record, can divide some subdirectories under level index list, continues to divide some subdirectories, by that analogy under each subdirectory。

In a kind of possible embodiment, the index list structure of level index list may is that

/ index/dcid/yyyymmdd/hhmm/filename；Or,

/ index/dcid/yyyymmdd/hhmm/index_name/filename；Or,

/ index/dcid/index_name/yyyymmdd/hhmm/filename；Or,

/index/dcid/yyyymmdd/hhmm/index_name/mod/filename。

Wherein index is the root of level index list, dcid is the catalogue of the service identification Attribute transposition according to data record, yyyymmdd is the catalogue of the date Attribute transposition according to data record, hhmm is the catalogue of hour minute Attribute transposition according to data record, index_name is the catalogue that the index field name attribute according to data record divides, mod is the catalogue of the delivery value attribute division of the cryptographic Hash according to the index word segment value in data record, filename is the index file corresponding with the data file at data record place, this index file should with each Attribute Relative in catalogue。

It should be noted that the present invention does not limit at the sequencing of the index list structure of level index list by each catalogue of Attribute transposition。

It is "/index/dcid/yyyymmdd/hhmm/filename " for the index list structure of level index list, index is the root of level index list, dcid is the subdirectory under index, yyyymmdd is the subdirectory under dcid, hhmm is the subdirectory under yyyymmdd, and index file is then positioned under the catalogue of hhmm。When the data record C in data file B be " 2014.11.1100:01:01 t Xiao Wang t travel information group t331121 t " time, storage system is according to the date attribute " 2014.11.11 " of this data record, hour minute attribute " 00:01 ", service identification attribute " group chat monitoring ", it is determined that the index file that this data record is corresponding is "/index/ group chat monitoring/2014.11.11/00:01/B "。

In the 3rd step, index information corresponding for data record, after determining the index file corresponding with the attribute information of data record, is stored in this index file by storage system。

The index information of data record C can be stored in index file "/index/ group chat monitoring/2014.11.11/00:01/B " by storage system。

As a kind of possible embodiment, index file includes one-level index and secondary index, this one-level index is for storing the cryptographic Hash of index word segment value in pieces of data record, and the side-play amount in secondary index indexed file corresponding to the cryptographic Hash of each index word segment value；This secondary index is for storing data record side-play amount in the data file。

Fig. 3 B illustrates the structural representation of a kind of index file, as shown in Figure 3 B, indexed file structure is made up of one-level index and secondary index two parts, the wherein side-play amount offset1 in the cryptographic Hash key of the index word segment value of each data record of one-level index record and secondary index indexed file, secondary index have recorded data record side-play amount set Index in the data file, and wherein Index have recorded data record each side-play amount offset2 in the data file。Additionally, keycount represents the number of cryptographic Hash key in index file, len represents the data length of Index field。

Index information corresponding for data record can be stored in index file by storage system in the following manner:

1) cryptographic Hash of the index word segment value in data record being write according to predefined procedure in the one-level index of index file, predefined procedure includes: descending order, or, ascending order；

2) record data in the secondary index of the side-play amount write index file in data file；

3) in the side-play amount write one-level in the secondary index indexed file of write being indexed, and storage is associated with the cryptographic Hash of index word segment value。

As shown in Figure 3 C, when

Index information A is: index word segment value " Xiao Wang ", data record A side-play amount " the 1234th row " in the data file；

Index information B is: index word segment value " Xiao Li ", data record B side-play amount " the 1111st row " in the data file；

Index information C is: index word segment value " little beautiful ", data record C side-play amount " the 1222nd row " in the data file。

Assume the cryptographic Hash of " Xiao Wang " be 1108, the cryptographic Hash of " Xiao Li " be 1110, the cryptographic Hash of " little beautiful " be 1120, then storage system can by 1108,1110,1120 with in the descending or ascending one-level index being sequentially written in index file。In the present embodiment, during each cryptographic Hash is indexed by storage system by the ascending one-level being sequentially written in index file。

Data record A side-play amount " the 1234th row " in the data file, data record B side-play amount " the 1111st row " in the data file and data record C side-play amount " the 1222nd row " in the data file are respectively written in the secondary index of index file by storage system again with " the 1223rd row "。

Finally, during the side-play amount write one-level of the secondary index indexed file of write is indexed by storage system, and storage is associated with the cryptographic Hash of index word segment value。Namely, storage has the side-play amount " side-play amount A " in the secondary index indexed file of data record A side-play amount " the 1234th row " in the data file associate with cryptographic Hash 1108 and is stored in one-level index, storage has the side-play amount " side-play amount 2 " in the secondary index indexed file of data record B side-play amount " the 1111st row " in the data file to associate with cryptographic Hash 1110 to be stored in during one-level indexes, storage has data record C side-play amount " the 1222nd row " in the data file and the side-play amount " side-play amount 3 " in the secondary index indexed file of " the 1223rd row " to associate with cryptographic Hash 1120 to be stored in during one-level indexes。Wherein, a is the data length of the Index field of data record A, and b is the data length of the Index field of data record B, and c is the data length of the Index field of data record C。

In order to prevent repeating to be indexed data file setting up, after the index of the data record of complete paired data file is set up, storage system needs to be transferred under out catalogue by this data file from catalogue, and wherein, the bibliographic structure of out catalogue is identical with the bibliographic structure in catalogue。

Such as, when the bibliographic structure of in catalogue is: " in/dcid/yyyymmdd/hh/filename ", the bibliographic structure of corresponding out catalogue is: " out/dcid/yyyymmdd/hh/filename "。

Above-mentioned steps 301 to step 303 can individually be implemented as an index generation method。Corresponding data query method is as follows:

Step 304, receives inquiry request。

Storage system is after the index of the data record of complete paired data file is set up, it is possible to receive the inquiry request that user sends。

Step 305, extracts attribute information entrained in inquiry request and index word segment value。

Storage system is before extracting index information according to this inquiry request, it is necessary to extract the attribute information and index word segment value that carry in inquiry request, such that it is able to determine, according to this attribute information and this index word segment value, the index file required to look up。

The attribute information of this inquiry request may include that at least one information in the filename of the data file corresponding to the delivery value of the cryptographic Hash of the index word segment value in the time range belonging to the service identification corresponding to data record, data record, the index field title in data record, data record and data record。Such as, when the attribute information of inquiry request can be: service identification " group chat monitoring ", date and time information " 2014.11.11 ", index field title " user's pet name ", data file filename: the delivery value " MOD1 " of the cryptographic Hash of " B ", index word segment value。

Step 306, selects at least one index file corresponding with attribute information at least one index file from level index list。

Storage system is according to the attribute information extracted, it is possible to select the index file corresponding with attribute information from level index list。

Such as, when the data record C in data file B be " 2014.11.1100:01:01 t Xiao Wang t travel information group t331121 t " time, storage system is according to the date attribute " 2014.11.11 " of this data record, hour minute attribute " 00:01 ", service identification attribute " group chat monitoring ", it is determined that the index file that this data record is corresponding is "/index/ group chat monitoring/2014.11.11/00:01/B "。

Step 307, according to extracting index information in the cryptographic Hash indexed file of index word segment value。

Storage system is after determining index file, further according to the cryptographic Hash of index word segment value, extracts index information from index file, and wherein, the extracting method of index information is as follows:

1) use binary chop to search in one-level indexes, obtain the side-play amount in the secondary index indexed file corresponding to the cryptographic Hash of index word segment value。

2) inquire about according to the side-play amount in secondary index indexed file, obtain index information。

Storage system is in the one-level of the index file determined indexes, binary chop is used to search and determine index word segment value, and then determine the side-play amount in the secondary index indexed file corresponding to cryptographic Hash of the index word segment value associating storage with this index word segment value, inquire the index information in secondary index according to the side-play amount in this secondary index indexed file。

In indexing due to the one-level of indexed file, the cryptographic Hash of index word segment value is to arrange by descending order or ascending order, and binary chop therefore can be used quickly to determine the cryptographic Hash of index word segment value。

Step 308, obtains data record in the data file according to index information。

The index information that storage system is extracted includes data record side-play amount in the data file, this data record position in the data file can be obtained, thus obtaining this data record and returning to user according to this data record side-play amount in the data file。

In sum, the storage system that the embodiment of the present invention provides, by when index generates, obtaining data file, generate corresponding index information for every data record, for the every data record in data file, further according in the index file that corresponding index information classification is stored to level index list by the attribute information of data record, wherein, level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue；When data query, receive inquiry request, according at least one index file from level index list of inquiry request extracts index information, data record is obtained in the data file further according to index information, solve when storage has mass data, the problem that inquiry velocity when using current index file to inquire about is slower, even if having reached storage when having mass data, it is also possible to provided the effect of quick search process by multiple index files store according to different attribute information classification。

Additionally, the present embodiment is also by index file is divided into one-level index and secondary index, this one-level index stores the cryptographic Hash of the index word segment value in each data record, and the side-play amount in secondary index indexed file corresponding to the cryptographic Hash of each index word segment value, this secondary index storage data record side-play amount in the data file, thus after determining index file, it is possible to the cryptographic Hash according to index word segment value, quickly determine the data record position in data file。

Refer to Fig. 4, it illustrates the structural representation of a kind of implementation environment involved by each embodiment of the present invention, as shown in Figure 4, this implementation environment includes space in a newspaper 401 in data, server cluster 402, index generation end 403, data query service end 404, data query client 405 and controller 406。

In data, space in a newspaper 401 can include multiple business production machine (not shown), and business produces machine for producing data record reported data record in service operation process。In data, space in a newspaper 401 is connected with server cluster 402 by wireless network or cable network。

Server cluster 402 has been used for storage and the inquiry of data record, and send the server ip address information of data record correspondence storage, can including multiple server, server cluster 402 is connected with space in a newspaper in data 401, index generation end 403 and controller 406 by wireless network or cable network。As a kind of possible embodiment, each station server in server cluster 402 includes index and generates end 403 and data query service end 404。

Index generates end 403 and has been used for the reception of data record and the generation of index information and storage。

Data query service end 404 is for receiving inquiry request and completing the inquiry of data record。Data query service end 404 is connected with index generation end 403 and data query client 405 by wireless network or cable network。

Data query client 405 is for sending the server ip information of inquiry request storage corresponding to receiving data record。Data query client 405 is connected with controller 406 by wireless network or cable network。

Controller 406 is for storing the server ip address information of storage corresponding to sending data record。

Refer to Fig. 5, it illustrates the method flow diagram of the data query method that one embodiment of the invention provides, this data query method is applied in the storage system shown in Fig. 4, as it is shown in figure 5, the method includes:

Step 501, server cluster obtains data file, and this data file includes at least one data record。

In the process of the business of operation, in data, space in a newspaper can generate the data record of a rule。In data, data record can be reported server cluster by space in a newspaper, server cluster receive the data record that in data, space in a newspaper sends。Specifically:

2014.11.1100:01:01 t Xiao Wang t travel information group t331121 t

2014.11.1200:01:11 t Xiao Zhang t ephemeris information group t331432 t

2014.11.1300:01:21 t little red t shopping information group t331254 t

2014.11.1400:01:22 t little beautiful t game exchange group t121121 t

In order to ensure the speed of inquiry, the data volume of each data file may remain in predefined size, and such as, the data record sum of each data file is less than 20,000；Or the data volume of each data file is less than 200MB etc.。That is, when in data, space in a newspaper data volume in current data file being detected reaches predetermined threshold value, a newly-built data file, and the data record received is stored in newly-built data file, and current data file is sent to server cluster。

In a certain server that data file can be stored to server cluster with load balancing mode by server cluster。

In a kind of possible embodiment, under the in catalogue in a certain server that data file is stored to server cluster with following file storage structure by server cluster:

in/dcid/yyyymmdd/hh/filename。

Step 502, the IP address of the service identification of data file and the server storing this data file is sent to controller by server cluster。

Such as, when data file A is stored in server B, the service identification of data file A is " group chat monitoring ", then the service identification " group chat monitoring " of data file is associated with the IP address " 111.111.1.111 " of server B and sends to controller by server cluster。Wherein, the filename of each data file is unique。

Corresponding, the IP address information of the filename of controller association storage data file and the server of this data file of storage。

Step 503, it is that every data record generates corresponding index information that index generates end, and this index information is for indicating data record storage position in the data file。

Data file is stored in catalogue by server, index in this station server generates end can detect whether have data file under in catalogue, if detecting the presence of data file, then set up the level index list of this data file, and be that the every data record in this data file generates corresponding index information, this level index list is for storing the index information of every data record in this data file, and this index information is for indicating data record storage position in the data file。

Wherein, index generates end is that the index information that every record generates correspondence may include steps of:

First, for every data record, index generates end and extracts the index word segment value and this data record side-play amount in the data file that need to index in this data record。

Second, index generates end and generates an index information corresponding with data record according to index word segment value with data record side-play amount in the data file。

Step 504, for the every data record in data file, index generates end and is classified in the index file stored to level index list by corresponding index information according to the attribute information of data record。

After generating the index information that data file is corresponding, index generates end and according to the attribute information of data record, can be classified in the index file stored to level index list by corresponding index information。Wherein, this level index list level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue。

This step may include steps of:

3rd, index information corresponding for data record is stored in index file。

In a first step, index generates end after every data record is generated corresponding index information, it is possible to extract the attribute information of pieces of data record。Wherein, attribute information may include that at least one information in the filename of the data file corresponding to the delivery value of the cryptographic Hash of the index word segment value in the time range belonging to the service identification corresponding to data record, data record, the index field title in data record, data record and data record。Wherein, time range generally includes date and time information and hour minute information。

In second step, index generates end can determine index file by level index list, and this level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue。

Index generates end according to the attribute information of data record, can divide some subdirectories under level index list, continues to divide some subdirectories, by that analogy under each subdirectory。

/ index/dcid/yyyymmdd/hhmm/filename；Or,

/ index/dcid/yyyymmdd/hhmm/index_name/filename；Or,

/ index/dcid/index_name/yyyymmdd/hhmm/filename；Or,

/index/dcid/yyyymmdd/hhmm/index_name/mod/filename。

It is "/index/dcid/yyyymmdd/hhmm/filename " for the index list structure of level index list, index is the root of level index list, dcid is the subdirectory under index, yyyymmdd is the subdirectory under dcid, hhmm is the subdirectory under yyyymmdd, and index file is then positioned under the catalogue of hhmm。When the data record C in data file B be " 2014.11.1100:01:01 t Xiao Wang t travel information group t331121 t " time, index generates end according to the date attribute " 2014.11.11 " of this data record, hour minute attribute " 00:01 ", service identification attribute " group chat monitoring ", it is determined that the index file that this data record is corresponding is "/index/ group chat monitoring/2014.11.11/00:01/B "。

In the 3rd step, index generates end after determining the index file corresponding with the attribute information of data record, is stored in this index file by index information corresponding for data record。

Index generates end and can be stored in by the index information of data record C in index file "/index/ group chat monitoring/2014.11.11/00:01/B "。

Index generates end and can be stored in index file by index information corresponding for data record in the following manner:

As shown in Figure 3 C,

When index information A is: index word segment value " Xiao Wang ", data record A side-play amount " the 1234th row " in the data file；

Index information C is: index word segment value " little beautiful ", data record C side-play amount " the 1222nd row " " the 1223rd row " in the data file。

Assume the cryptographic Hash of " Xiao Wang " be 1108, the cryptographic Hash of " Xiao Li " be 1110, the cryptographic Hash of " little beautiful " be 1120, then index generate end can by 1108,1110,1120 with in the descending or ascending one-level index being sequentially written in index file。In the present embodiment, index generates end by each cryptographic Hash by the ascending one-level index being sequentially written in index file。

Index generates end and data record A side-play amount " the 1234th row " in the data file, data record B side-play amount " the 1111st row " in the data file and data record C side-play amount " the 1222nd row " in the data file and " the 1223rd row " is respectively written in the secondary index of index file。

Finally, index generates end and the side-play amount of the secondary index indexed file of write is write in one-level index, and associates storage with the cryptographic Hash of index word segment value。Namely, storage has the side-play amount " side-play amount 1 " in the secondary index indexed file of data record A side-play amount " the 1234th row " in the data file associate with cryptographic Hash 1108 and is stored in one-level index, storage has the side-play amount " side-play amount 2 " in the secondary index indexed file of data record B side-play amount " the 1111st row " in the data file to associate with cryptographic Hash 1110 to be stored in during one-level indexes, storage has data record C side-play amount " the 1222nd row " in the data file and the side-play amount " side-play amount 3 " in the secondary index indexed file of " the 1223rd row " to associate with cryptographic Hash 1120 to be stored in during one-level indexes。Wherein, a is the data length of the Index field of data record A, and b is the data length of the Index field of data record B, and c is the data length of the Index field of data record C。

In order to prevent repeating to be indexed data file setting up, after the index of the data record of complete paired data file is set up, index generates end to be needed to be transferred under out catalogue by this data file from catalogue, and wherein, the bibliographic structure of out catalogue is identical with the bibliographic structure in catalogue。

Above-mentioned steps 501 to step 504 can individually be implemented as an index generation method。Corresponding data query method is as follows:

Step 505, data query client receives inquiry request。

After the index of the data record of complete paired data file is set up, data query client can receive the inquiry request that user sends。

Step 506, the data query client service identification according to the data file carried in inquiry request, the IP address list of server corresponding to this service identification is obtained from controller。

The service identification of different pieces of information file is likely to identical, and these data files are likely stored in different servers, therefore, the data query client service identification according to the data file carried in inquiry request, it is possible to obtain the IP address list of server corresponding to this service identification from controller。Such as, have identical services mark " group chat monitoring " data file be respectively stored in IP address for " 111.111.1.111 ", IP address be " 111.111.2.111 ", IP address is in the server of " 111.111.3.111 ", the then data query client service identification " group chat monitoring " according to the data file carried in inquiry request, the IP address list obtained from controller is:

111.111.1.111

111.111.2.111

111.111.3.111

Step 507, data query client sends this inquiry request to the every station server in the IP address list obtained。

Step 508, data query service end extracts attribute information entrained in inquiry request and index word segment value。

After server receives inquiry request, the data query service end in this station server can extract the attribute information carried in this inquiry request and index word segment value, thus determining, according to this attribute information and this index word segment value, the index file required to look up。

Step 509, selects at least one index file corresponding with attribute information in data query service end at least one index file from level index list。

Data query service end is according to the attribute information extracted, it is possible to select the index file corresponding with attribute information from level index list。

Such as, when the data record C in data file B be " 2014.11.1100:01:01 t Xiao Wang t travel information group t331121 t " time, data query server is according to the date attribute " 2014.11.11 " of this data record, hour minute attribute " 00:01 ", service identification attribute " group chat monitoring ", it is determined that the index file that this data record is corresponding is "/index/ group chat monitoring/2014.11.11/00:01/B "。

Step 510, data query service end is according to extracting index information in the cryptographic Hash indexed file of index word segment value。

Data query service end is after determining index file, further according to the cryptographic Hash of index word segment value, extracts index information from index file, and wherein, the extracting method of index information is as follows:

Data query service end is in the one-level of the index file determined indexes, binary chop is used to search and determine index word segment value, and then determine the side-play amount in the secondary index indexed file corresponding to cryptographic Hash of the index word segment value associating storage with this index word segment value, inquire the index information in secondary index according to the side-play amount in this secondary index indexed file。

Step 511, data query service end obtains data record in the data file according to index information。

The index information that data query service end is extracted includes data record side-play amount in the data file, this data record position in the data file can be obtained, thus obtaining this data record and returning to user according to this data record side-play amount in the data file。

In sum, the data query method that the embodiment of the present invention provides, by when index generates, obtain data file, corresponding index information is generated for every data record, for the every data record in data file, further according in the index file that corresponding index information classification is stored to level index list by the attribute information of data record, wherein, level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue；When data query, receive inquiry request, according at least one index file from level index list of inquiry request extracts index information, data record is obtained in the data file further according to index information, solve when storage has mass data, the problem that inquiry velocity when using current index file to inquire about is slower；Even if having reached storage when having mass data, it is also possible to provided the effect of quick search process by multiple index files store according to different attribute information classification。

Additionally, the present invention is by after obtaining data file, send the IP address information of the server storing this data file to controller and correspond to the service identification of this data file, make when inquiring about data record, the server ip address storing this data file can be quickly determined, thus reaching to inquire about rapidly the effect of data record according to this machine address IP information。

Referring to Fig. 6, the index that it illustrates one embodiment of the invention provides generates the block diagram of device, this index generates device and can apply in storage system, this index generates device, including: data file acquisition module 610, index information generation module 620, index information memory module 630。

This data file acquisition module 610, is used for obtaining data file, and this data file includes at least one data record；

This index information generation module 620, for generating corresponding index information for the every data record in this data file acquisition module 610, this index information is for indicating data record storage position in the data file；

This index information memory module 630, for for the every data record in data file, in the index file that the index information classification that this corresponding index information generation module 620 is generated by the attribute information according to data record stores to level index list, level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue。

In sum, the index that the embodiment of the present invention provides generates device, by when index generates, obtain data file, corresponding index information is generated for every data record, for each data record in data file, further according in the index file that corresponding index information classification is stored to level index list by the attribute information of data record, wherein, level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue；Solve when storage has mass data, the problem that inquiry velocity when using current index file to inquire about is slower；Even if having reached storage when having mass data, it is also possible to provided the effect of quick search process by multiple index files store according to different attribute information classification。

Additionally, embodiment by being divided into one-level index and secondary index by index file, one-level index stores the cryptographic Hash of the index word segment value in each data record, and the side-play amount in secondary index indexed file corresponding to the cryptographic Hash of each index word segment value, secondary index storage data record side-play amount in the data file, thus after determining index file, can according to the cryptographic Hash of index word segment value, determine the data record position in data file, thus reaching faster to obtain the effect of data record。

Referring to Fig. 7, the index that it illustrates one embodiment of the invention provides generates the block diagram of device, this index generates device and can apply in storage system, this index generates device, including: data file acquisition module 710, index information generation module 720, index information memory module 730。

This data file acquisition module 710, is used for obtaining data file, and this data file includes at least one data record；

This index information generation module 720, for generating corresponding index information for the every data record in this data file acquisition module 710, this index information is for indicating data record storage position in the data file；

This index information memory module 730, for for the every data record in data file, in the index file that the index information classification that this corresponding index information generation module 720 is generated by the attribute information according to data record stores to level index list, level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue。

This attribute information includes: at least one information in the filename of the data file corresponding to the index field title in the service identification corresponding to data record, the time range belonging to data record, data record, the delivery value of cryptographic Hash of the index word segment value in data record and data record。

Optionally, this index information memory module 730, including: attribute information extraction unit 731, index file determine unit 732 and index information memory element 733。

This attribute information extraction unit 731, for for every data record, extracting the attribute information of data record；

This index file determines unit 732, for determining an index file corresponding with the attribute information that this attribute information extraction unit 731 extracts at least one index file in level index list；

This index information memory element 733, determines in the index file that unit 732 is determined for index information corresponding for data record is stored in this index file。

Optionally, this index file includes one-level index and secondary index；

This one-level index is for storing the cryptographic Hash of index word segment value in pieces of data record, and the side-play amount in secondary index indexed file corresponding to the cryptographic Hash of each index word segment value；

This secondary index is for storing data record side-play amount in the data file。

Optionally, this index information memory element 733, including: the first storing sub-units 733a, the second storing sub-units 733b and the three storing sub-units 733c。

This first storing sub-units 733a, in the cryptographic Hash of the index word segment value in data record being indexed according to the one-level of predefined procedure write index file, predefined procedure includes: descending order, or, ascending order；

This second storing sub-units 733b, for recording data in the secondary index of the side-play amount write index file in data file；

3rd storing sub-units 733c, in the side-play amount write one-level in the secondary index indexed file of write being indexed, and associates storage with the cryptographic Hash of index word segment value。

It should be understood that the index that above-described embodiment provides generates device when generating index, only it is illustrated with the division of above-mentioned each functional module, in practical application, as desired above-mentioned functions distribution can be completed by different functional modules, the internal structure being about to index generation device is divided into different functional modules, to complete all or part of function described above。It addition, the index that above-described embodiment provides generates device belongs to same design with index generation method embodiment, it implements process and refers to embodiment of the method, repeats no more here。

Referring to Fig. 8, it illustrates the block diagram of the data query arrangement that one embodiment of the invention provides, this data query arrangement can apply in storage system, this data query arrangement, including: inquiry request receiver module 810, index information extraction module 820 and data record acquisition module 830。

This inquiry request receiver module 810, is used for receiving inquiry request；

This index information extraction module 820, for at least one index file from level index list of the inquiry request that receives according to this inquiry request receiver module 810 extracts index information, level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue；

This data record acquisition module 830, obtains data record in the data file for the index information extracted according to this index information extraction module 820。

In sum, the data query arrangement that the embodiment of the present invention provides, by when data query, receive inquiry request, according at least one index file from level index list of inquiry request extracts index information, obtain data record further according to index information in the data file, be stored in the index file different directories from classification and extract index information；Solve when storage has mass data, the problem that inquiry velocity when using current index file to inquire about is slower；Even if having reached storage when having mass data, it is also possible to provided the effect of quick search process by multiple index files store according to different attribute information classification。

Referring to Fig. 9, it illustrates the block diagram of the data query arrangement that one embodiment of the invention provides, this data query arrangement can apply in storage system, this data query arrangement, including: inquiry request receiver module 910, index information extraction module 920 and data record acquisition module 930。

This inquiry request receiver module 910, is used for receiving inquiry request；

This index information extraction module 920, for at least one index file from level index list of the inquiry request that receives according to this inquiry request receiver module 910 extracts index information, level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue；

This data record acquisition module 930, obtains data record in the data file for the index information extracted according to this index information extraction module 920。

Optionally, this index information extraction module 930, including: this Query Information extraction unit 931, index file determine unit 932 and index information extraction unit 933。

This Query Information extraction unit 931, for extracting attribute information entrained in inquiry request and index word segment value；

This index file determines unit 932, for selecting at least one index file corresponding with the attribute information that this Query Information extraction unit 931 extracts at least one index file from level index list；

This index information extraction unit 933, determines extraction index information in the index file that unit 932 is determined for the cryptographic Hash according to index word segment value at this index file。

Optionally, this index file includes one-level index and secondary index；

Optionally, this index information extraction unit 933, including: secondary index determines that subelement 933a and index information determine subelement 933b。

This secondary index determines subelement 933a, for using binary chop to search in indexing in one-level, obtains the side-play amount in the secondary index indexed file corresponding to the cryptographic Hash of index word segment value；

This index information determines subelement 933b, inquires about for the side-play amount determined in the subelement 933a secondary index indexed file determined according to this secondary index, obtains index information。

It should be understood that the data query arrangement that above-described embodiment provides is when data query, only it is illustrated with the division of above-mentioned each functional module, in practical application, as desired above-mentioned functions distribution can be completed by different functional modules, it is divided into different functional modules, to complete all or part of function described above by the internal structure of data query arrangement。It addition, the data query arrangement that above-described embodiment provides belongs to same design with data query embodiment of the method, it implements process and refers to embodiment of the method, repeats no more here。

Refer to Figure 10, the data that it illustrates one embodiment of the invention provides process the structural representation of end。Described data process end 1000 and include CPU (CPU) 1001, include random access memory (RAM) 1002 and the system storage 1004 of read only memory (ROM) 1003, and the system bus 1005 of connection system memorizer 1004 and CPU 1001。Described data process the basic input/output (I/O system) 1006 that end 1000 also includes helping to transmit information between each device in computer, and for storing the mass-memory unit 1007 of operating system 1013, application program 1014 and other program modules 1015。

Described basic input/output 1006 includes the display 1008 for showing information and the input equipment 1009 of such as mouse, keyboard etc for user's input information。Wherein said display 1008 and input equipment 1009 all pass through to be connected to the IOC 1010 of system bus 1005 and are connected to CPU 1001。Described basic input/output 1006 can also include IOC 1010 for receiving and processing the input from other equipment multiple such as keyboard, mouse or electronic touch pens。Similarly, IOC 1010 also provides output to display screen, printer or other kinds of outut device。

Described mass-memory unit 1007 is connected to CPU 1001 by being connected to the bulk memory controller (not shown) of system bus 1005。Described mass-memory unit 1007 and the computer-readable medium being associated thereof process end 1000 for data and provide non-volatile memories。It is to say, described mass-memory unit 1007 can include the computer-readable medium (not shown) of such as hard disk or CD-ROM drive etc。

Without loss of generality, described computer-readable medium can include computer-readable storage medium and communication media。Computer-readable storage medium includes the volatibility of any method or technology realization for storing the information such as such as computer-readable instruction, data structure, program module or other data and medium non-volatile, removable and irremovable。Computer-readable storage medium includes RAM, ROM, EPROM, EEPROM, flash memory or its technology of other solid-state storage, CD-ROM, DVD or other optical storage, cartridge, tape, disk storage or other magnetic storage apparatus。Certainly, skilled person will appreciate that described computer-readable storage medium is not limited to above-mentioned several。Above-mentioned system storage 1004 and mass-memory unit 1007 may be collectively referred to as memorizer。

According to various embodiments of the present invention, described data process the remote computer operation that end 1000 can also be connected on network by networks such as such as the Internets。Namely data process end 1000 and can be connected to network 1012 by the NIU 1011 being connected on described system bus 1005, in other words, it is possible to use NIU 1011 is connected to other kinds of network or remote computer system (not shown)。

Described memorizer also includes one or more than one program, one or more than one program are stored in memorizer, and one or more than one program package are containing being used for carrying out the date storage method of embodiment of the present invention offer and/or the instruction of data query method。

The invention described above embodiment sequence number, just to describing, does not represent the quality of embodiment。

One of ordinary skill in the art will appreciate that all or part of step realizing above-described embodiment can be completed by hardware, can also be completed by the hardware that program carrys out instruction relevant, described program can be stored in a kind of computer-readable recording medium, storage medium mentioned above can be read only memory, disk or CD etc.。

The foregoing is only presently preferred embodiments of the present invention, not in order to limit the present invention, all within the spirit and principles in the present invention, any amendment of making, equivalent replacement, improvement etc., should be included within protection scope of the present invention。

Claims

1. an index generation method, it is characterised in that described method includes:

Obtaining data file, described data file includes at least one data record；

2. method according to claim 1, it is characterised in that described attribute information includes:

At least one information in the filename of the data file corresponding to the index field title in the service identification corresponding to data record, the time range belonging to data record, data record, the delivery value of cryptographic Hash of the index word segment value in data record and data record。

3. method according to claim 1 and 2, it is characterised in that described for every data record, classifies in the index file stored to level index list according to the attribute information of described data record by corresponding described index information, including:

For every data record, extract the attribute information of described data record；

At least one index file in described level index list is determined corresponding with a described attribute information index file；

Index information corresponding for described data record is stored in described index file。

4. method according to claim 3, it is characterised in that described index file includes one-level index and secondary index；

Described one-level index is for storing the cryptographic Hash of index word segment value in pieces of data record, and the side-play amount that secondary index corresponding to the cryptographic Hash of each described index word segment value is in described index file；

Described secondary index is for storing described data record side-play amount in described data file。

5. method according to claim 4, it is characterised in that described index information corresponding for described data record is stored in described index file, including:

The cryptographic Hash of the index word segment value in described data record being write according to predefined procedure in the one-level index of described index file, described predefined procedure includes: descending order, or, ascending order；

Described data record side-play amount in described data file is write in the secondary index of described index file；

The described secondary index side-play amount in described index file of write is write in described one-level index, and associates storage with the cryptographic Hash of described index word segment value。

6. a data query method, it is characterised in that described method includes:

Receive inquiry request；

7. method according to claim 6, it is characterised in that described attribute information includes:

8. the method according to claim 6 or 7, it is characterised in that extract index information in described at least one index file according to described inquiry request from level index list, including:

Extract attribute information entrained in described inquiry request and index word segment value；

At least one index file from described level index list selects at least one index file corresponding with described attribute information；

Cryptographic Hash according to described index word segment value extracts index information in described index file。

9. method according to claim 8, it is characterised in that described index file includes one-level index and secondary index；

10. method according to claim 9, it is characterised in that the described cryptographic Hash according to described index word segment value extracts index information in described index file, including:

Use binary chop to search in described one-level indexes, obtain the secondary index corresponding to cryptographic Hash of the described index word segment value side-play amount in described index file；

Inquire about according to described secondary index side-play amount in described index file, obtain described index information。

11. an index generates device, it is characterised in that described device includes:

Index information memory module, for for each data record in described data file, in the index file that the described index information classification that corresponding described index information generation module is generated by the attribute information according to described data record stores to level index list, described level index list includes n the index file being positioned under different directories, n >=1, the corresponding one group of attribute information of the index file under each catalogue。

12. device according to claim 11, it is characterised in that described attribute information includes:

13. the device according to claim 11 or 12, it is characterised in that described index information memory module, including:

Attribute information extraction unit, for for every data record, extracting the attribute information of described data record；

Index file determines unit, for determining an index file corresponding with the described attribute information that described attribute information extraction unit extracts at least one index file in described level index list；

Index information memory element, determines in the described index file that unit is determined for index information corresponding for described data record is stored in described index file。

14. device according to claim 13, it is characterised in that described index file includes one-level index and secondary index；

15. device according to claim 14, it is characterised in that described index information memory element, including:

First storing sub-units, in the cryptographic Hash of the index word segment value in described data record being indexed according to the one-level of the predefined procedure described index file of write, described predefined procedure includes: descending order, or, ascending order；

Second storing sub-units, for writing described data record side-play amount in described data file in the secondary index of described index file；

3rd storing sub-units, for the described secondary index side-play amount in described index file of write being write in described one-level index, and associates storage with the cryptographic Hash of described index word segment value。

16. a data query arrangement, it is characterised in that described device includes:

Inquiry request receiver module, is used for receiving inquiry request；

17. device according to claim 16, it is characterised in that described attribute information includes:

18. the device according to claim 16 or 17, it is characterised in that described index information extraction module, including:

Query Information extraction unit, for extracting attribute information entrained in described inquiry request and index word segment value；

Index file determines unit, for selecting at least one index file corresponding with the described attribute information that described Query Information extraction unit extracts at least one index file from described level index list；

Index information extraction unit, determines extraction index information in the described index file that unit is determined for the cryptographic Hash according to described index word segment value at described index file。

19. device according to claim 18, it is characterised in that described index file includes one-level index and secondary index；

20. device according to claim 19, it is characterised in that described index information extraction unit, including:

Secondary index determines subelement, for using binary chop to search in indexing in described one-level, obtains the secondary index corresponding to cryptographic Hash of the described index word segment value side-play amount in described index file；

Index information determines subelement, for determining that described secondary index that subelement the determines side-play amount in described index file is inquired about according to described secondary index, obtains described index information。