CN106203171A - Big data platform Security Index system and method - Google Patents

Abstract

A security indexing system for a big data platform, which receives search keyword information submitted by a client and performs a search based on the keyword in the search domain to identify documents matching the keyword, and the server provides documents that match the keyword, and the big data platform is secure The indexing system includes an index generation logic module that processes documents provided by the server to generate a corresponding set of inverted index files containing a feature index, a key server, a search request analysis module, a search engine pool, a search result generation module, a security index module, metadata engine module. The invention also discloses a big data platform security indexing method.

Description

Big data platform security index system and method

technical field

The invention relates to a system and method for a big data platform, in particular to a big data platform security index system and method.

Background technique

Big data platforms face many security risks in the process of storage, processing, and transmission. The best way at present is to encrypt the data to cut off the possibility of others prying into the privacy. Search engine technology has solved the problem of users efficiently locating the information they need in the ocean of information. But usually the search engine implementations of big data platforms are not secure. A common example is that the inverted index files are stored in plain text. Attackers steal the inverted index files through illegal means, and combined with language models and other technologies, high-quality Restore the contents of the file. Although the original file content is encrypted and stored on the server, it still causes privacy leaks.

In order to meet the security and privacy protection requirements of the big data platform, the index files of the search engine on the big data platform also need to be encrypted. But it is not easy to do it: first, the index is a large file, and the process of encryption and decryption is very time-consuming; second, the continuous update of the big data platform means that the index file will also be frequently modified, and these tasks must be It involves encryption and decryption; thirdly, the rapid growth of big data platforms also determines that the index files are bound to be very large, which will also cause the system to slow down; finally, after the index files are encrypted, they cannot directly respond to business requests, and the additional decryption work will Prolong the user's waiting time again.

Contents of the invention

In order to solve the above problems, the present invention provides a big data platform security indexing system and method.

A security indexing system for a big data platform, which receives search keyword information submitted by a client and performs a search based on the keyword in the search domain to identify documents matching the keyword, and the server provides documents that match the keyword, and the big data platform is secure The indexing system includes an index generation logic module, a key server, a search request analysis module, a search engine pool, and a search result generation module for processing the documents provided by the server to generate a corresponding inverted index file set containing feature indexes; The big data platform security indexing system also includes the security function of intelligently segmenting and encrypting the inverted index file set containing the feature index generated by the index generation logic module according to a single feature and storing it securely in ciphertext in the distributed file system Hadoop. Index module, the metadata engine module that manages the index segment file that described security index module generates; Described metadata engine module is the segment file that management described security index module generates and can locate the index engine of segment quickly according to feature, comprises a metadata indexing module that searches for a complete index of features in segments in a domain and a metadata indexing extension module that is indexed by the metadata indexing module and that indexes a secure index; the secure indexing module includes an inverted The index file set is written into the index cache and can support the cache module of index cache update, analyze the index cache data in the cache and generate persistence tasks on demand, and analyze the security index segments in the metadata engine to generate corresponding segment optimization The optimizer for the task.

Wherein, the optimizer includes an analyzer that checks the index cache data that needs to be persisted in the cache and generates a persistence task, and analyzes the status information of the security index segment recorded in the metadata engine to generate a segment optimization task, according to The task analyzed by the analyzer generates a task queue module of the task queue, and an executor for processing the tasks recorded in the task queue module.

A big data platform security indexing method, using a big data platform security index system to realize the security index, comprising the following steps:

s1. The user submits the keyword to be queried through the client, receives the keyword information submitted by the client through the big data platform security index system, and searches according to the keyword in the search domain to identify documents matching the keyword;

s2. The server provides documents matching keywords to the big data platform security indexing system;

s3. The index generation logic module in the secure index system of the big data platform processes the documents provided by the server and matches the keywords to generate a corresponding inverted index file set including feature indexes, the inverted files and Contains feature term and document extraction credential docID;

s4. The security index module in the security index system of the big data platform intelligently segments the inverted file set generated in s3 according to a single feature, and the size and specification of each segment are unified, and each segment is encrypted to encrypt The text form is stored in the distributed file system Hadoop; First, the cache module in the safe index module writes the inverted file set into the index cache and supports updating the index cache when new documents are encountered; secondly , analyzing the index cache data in the cache module through the optimizer in the security index module, and generating index persistence tasks and segment optimization tasks on demand;

s5. Manage the index segment files generated in s4 through the metadata engine module in the indexing system of the big data platform, and search and locate the file features according to the needs and quickly read the segment where the positioning features are located.

By adopting the big data platform security index system and Fangfa of the present invention, the intelligent encryption and encryption of multi-level indexes is realized, which does not affect the business response, saves time and effort, and at the same time, can realize the continuous optimization and update of the index, and improve the business response speed.

Description of drawings

Fig. 1 is a schematic diagram of the architecture of the big data platform security index system of the present invention;

Fig. 2 is a schematic structural diagram of a security index module in the big data platform security index system of the present invention;

Fig. 3 is a schematic diagram of document generation logic of the big data platform security indexing method of the present invention;

Fig. 4 is a schematic diagram of index segmentation and encrypted storage of the big data platform security index method of the present invention;

Fig. 5 is a multi-layer index schematic diagram of the big data platform security index method of the present invention;

Fig. 6 is a schematic flow chart of updating the index cache of the big data platform security indexing method of the present invention;

Fig. 7 is a schematic flow chart of a persistent index method of the big data platform security index method of the present invention;

FIG. 8 is a schematic diagram of a new feature persistence process of the persistent index method of the big data platform security index method of the present invention;

Fig. 9 is a schematic diagram of the old feature persistence process of the persistent index method of the big data platform security index method of the present invention;

Fig. 10 is a schematic flow chart of the segment optimization method of the big data platform security indexing method of the present invention.

detailed description

In order to better understand the present invention, the present invention will be described in detail below in conjunction with the accompanying drawings.

As shown in Figure 1, a security indexing system for a big data platform of the present invention receives the search keyword information submitted by the client and searches according to the keyword in the search field to identify documents matching the keyword, and the server provides documents that match the keyword documents, the big data platform security indexing system includes an index generation logic module for processing the documents provided by the server to generate a corresponding inverted index file set including a feature index, a key server, a search request analysis module, and a search engine pool, search result generation module; the big data platform security index system also includes processing the inverted index file set containing the feature index produced by the index generation logic module according to the single feature intelligent segmentation and encryption in ciphertext form for safe storage A security index module in the distributed file system hadoop, a metadata engine module that manages the index segment files generated by the security index module; the metadata engine module manages the segment files generated by the security index module and can according to the characteristics An indexing engine for quickly locating segments, including a metadata indexing module containing a full index of features in the segment in the search domain and a metadata indexing extension module indexed by the metadata indexing module and indexing a secure index; the secure indexing module includes The inverted index file set generated by the index generation logic module is written into the index cache and can support the cache module for updating the index cache, analyzing the index cache data in the cache and generating persistent tasks on demand and analyzing the metadata engine The secure index segment in generates the optimizer for the corresponding segment optimization task.

Wherein, as shown in Figure 2, the optimizer includes checking the index cache data that needs to be persisted in the cache and generating a persistence task and analyzing the status information of the security index segment recorded in the metadata engine to generate segment optimization An analyzer for tasks, a task queue module for generating a task queue based on the tasks analyzed by the analyzer, and an executor for processing tasks recorded in the task queue module.

A big data platform security indexing method, using a big data platform security index system to realize the security index, comprising the following steps:

s1. The user submits the keyword to be queried through the client, receives the keyword information submitted by the client through the big data platform security index system, and searches according to the keyword in the search domain to identify documents matching the keyword;

s2. The server provides documents matching keywords to the big data platform security indexing system;

s3. The index generation logic module in the secure index system of the big data platform processes the documents provided by the server and matches the keywords to generate a corresponding inverted index file set including feature indexes, the inverted files and Contains feature term and document extraction voucher docID; as shown in Figure 3, after document analysis is performed on the text data, the document is processed into a file containing feature term and document extraction voucher docID, and then these processed files are sorted, and Generate an inverted file; wherein, the characteristic term is not limited to a keyword, but can be a phrase, a number, a code or any similar value to be searched within the document;

s4. The security index module in the security index system of the big data platform intelligently segments the inverted file set generated in s3 according to a single feature, and the size and specification of each segment are unified, and each segment is encrypted to encrypt The text form is stored in the distributed file system Hadoop; First, the cache module in the safe index module writes the inverted file set into the index cache and supports updating the index cache when new documents are encountered; secondly , analyze the index cache data in the cache module through the optimizer in the security index module, and generate index persistence tasks and segment optimization tasks on demand; as shown in Figure 4, the index inverted file is processed Slices are intelligently segmented according to a single feature, and each segment is encrypted and stored in Hadoop in the form of ciphertext. The segmented encryption storage method makes disk read and write costs low and query efficiency stable;

s5, manage the index segment file generated in s4 through the metadata engine module in the indexing system of the big data platform, and quickly read the segment where the positioning feature is located according to the file characteristics of searching and positioning as required; as shown in Figure 5, the metadata The metadata index extension in the data engine indexes the encrypted security index in Hadoop, and the metadata index indexes the metadata index extension.

Wherein, as shown in Figure 6, updating the index through the cache module includes the following steps:

g1: start;

g2. Perform feature analysis on the document and extract document features;

g3. Generate an inverted file according to the characteristics of the document;

g4, generate an inverted file list;

g5. Determine whether the document feature already exists; if the feature already exists, go to g6; if the feature does not exist, go to g9;

g6. Locate the index segment according to the characteristics;

g7. Add new data to the index segment according to the size and specification of the index segment;

g8. Whether there are any remaining files in the inverted file; if there are remaining files, go to g4; if there are no remaining files, go to g11;

g9. According to the characteristics of the document, create a new index segment and write the file data, then go to g8;

g10, end.

As shown in Figure 7, through the cache module, an index persistence task is generated according to the index persistence requirements, including the following steps:

c1, start;

c2, analyzing the index cache data in the cache;

c3. Generate persistent tasks according to characteristics;

c4. Generate a feature list to be processed according to the features that need to be persisted;

c5. Determine whether the feature already exists, if so, go to c8; if not, go to c6;

c6. Persist new features;

c7, determine whether the feature list is empty; if so, go to c9; if not, go to c4;

c8. Persist the old features and transfer to c6;

c9, end.

As shown in Figure 8, the persistence of the new feature includes the following steps:

x1, start;

x2, Create a new empty segment template;

x3, write data data in the newly created segment;

x4. Update head data;

x5, encrypt the segment;

x6. Notify metadata engine.

As shown in Figure 9, the persistence of the old features includes the following steps:

l1, start;

l2. Locate the latest segment where the feature is located;

l3. Extract the file corresponding to the index segment;

l4. Decrypt files;

l5. Update files;

l6. Encrypted files;

l7. Store the encrypted file in Hadoop;

l8. Enter the countdown to delete the old segment;

l9. Notify the metadata engine.

As shown in Figure 10, the optimizer analyzes the index cache data, and generates segment optimization tasks including the following steps:

d1, start;

d2. Analyze security index;

d3, generating optimization tasks according to the analysis results in d2;

d4, judging whether the segment needs to be deleted, if so, go to d5; if not, go to d6;

d5, delete the expired segment;

d6, judge whether to split the segment, if so, go to d7; if not, go to d9;

d7, split the segment into multiple new segments;

d8. Enter the countdown to delete the old segment;

d9, judging whether to merge segments; if so, go to d10; if not, go to d12;

d10, merging segment;

d11. Enter the countdown to delete the old segment;

d12. Notify the metadata engine.

Briefly, a secure indexing system for a big data platform provided by the present invention is aimed at searching a set of documents (or files) within a search domain to find documents related to a user. Searching generally involves obtaining a set of keywords from a user to indicate a search and then identifying all documents within the search domain that match those keywords. The resulting set of candidate documents contains all potentially relevant documents from the search domain. A ranking algorithm can then be applied to the candidate documents to predict the relevance of the documents to the user. The candidate documents are then presented to the user, typically in descending order of predicted relevance. Embodiments of this type of search typically utilize an inverted index structure that associates keywords with documents.

The index generation logic module is responsible for processing the documents submitted by the server. In the present invention, the index generation logic output is described as an inverted file set composed of term and docID. A term is not limited to keywords, it can be a phrase, number, code, or any similar value to be searched within a document. The token (docID) refers to the credential for extracting the document, usually a string indicating the file path, and the server is required to attach the corresponding token when submitting the document. The secure index module can "smartly" divide large index files into small segments and store them in the distributed file system in ciphertext. The collection of inverted files submitted by the index generation logic module is taken over by the cache module and first written to the index cache, and then the metadata engine module is notified, which can effectively reduce the frequency of index updates, which is very important for providing stable and reliable search services. It is also the key to real-time search. The cache is expensive and very limited, so it will be persisted to Hadoop later, which is also one of the very important jobs of the optimizer.

The large index file is divided into a series of inverted files, which are filled in each segment according to certain rules, and then encrypted and stored in the distributed file system. Large index files do not really exist, and are completely replaced by well-designed segment file sets. First, segment files are designed to store only a single feature, which will be very useful for loading index segments in a targeted manner. In addition, uniform size and standardized Segment files also make encryption and decryption controllable.

The metadata engine module manages large-scale segment files, such as quickly locating the relevant segment files of a certain feature in a specific period of time on a topic, as well as the meta-information needed for optimizing the analyzer and performing segment optimization. It consists of a metadata index and a metadata index extension. In practical applications, the metadata engine module contains a metadata engine module that maintains the state information of the segment file and locates the meta information of the segment file associated with a certain feature under specific constraints. The status information of the segment file includes information required by the optimizer such as segment storage saturation, size, and time span of index features. Constraints for locating segments include information such as the hashtag at index time, the range of index time, and the latest segment.

The metadata indexing module is a complete index of features found in segment files in the search domain. It is structured to support various types of searches and can be used independently of metadata indexing extensions. A flag within the metadata engine indicates whether information is available in the metadata index extension, that is, the index associated with the requirement. This flag exists for each characteristic to provide control over how and when indexes in the Metadata Indexing extension are used. This design is necessary to take into account the response time and adapt to the high-speed and almost unlimited content growth of the big data platform. For example, the optimizer needs are far less urgent than search request analysis, and the timeliness requirements for stale data searches will be lower. They will belong to different indexes, and sometimes they may be split into multiple indexes to ensure that the response time of each type of demand can meet the requirements.

The essence of the metadata index extension is the container of the index, which actually records the data to be managed by the metadata engine, where the index split is based on the index size, topic label and time span. It can be said that the core of the entire architecture is an index tree. The metadata index indexes the metadata index extension, and the metadata index extension indexes the security index. The biggest difference is that the security index is designed as segmented encrypted storage. Because the B+ tree has the advantages of low disk read and write costs and stable query efficiency, usually the metadata index and metadata index extension adopt the B+ tree index implementation.

Optimizer It consists of analyzer, task queue module and executor. The analyzer generates a persistent task by checking the index cache data that needs to be persisted in the cache and analyzes the status information of the secure index segment recorded in the metadata engine module to generate a segment optimization task. The executor processes tasks in the task queue, including persistent index and secure index segment optimization tasks.

The cache module can be used in many ways to support updating index cache updates as well as to support index caches in persistent cache modules.

By adopting the big data platform security index system and Fangfa of the present invention, the intelligent encryption and encryption of multi-level indexes is realized, which does not affect the business response, saves time and effort, and at the same time, can realize the continuous optimization and update of the index, and improve the business response speed. Index inverted files are encrypted and stored in segments, which makes the disk read and write costs low, the query efficiency is stable, the encryption process is fast, and does not affect system stability; index cache update, old and new feature persistence, and segment optimization ensure the security of index files. Integrity and stability, avoid feature duplication and redundancy, and improve business response speed.

The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone familiar with the technical field within the technical scope disclosed in the present invention, according to the technical solution of the present invention Any equivalent replacement or change of the inventive concepts thereof shall fall within the protection scope of the present invention.

Claims (8)

1. A security indexing system for a big data platform, which receives search keyword information submitted by a client and searches according to the keyword in the search field to identify documents matching the keyword, and the server provides documents that match the keyword, and the big data The platform security index system includes an index generation logic module for processing the documents provided by the server to generate a corresponding inverted index file set containing feature indexes, a key server, a search request analysis module, a search engine pool, and a search result generation module, It is characterized in that, The big data platform security index system also includes intelligently segmenting and encrypting the inverted index file set containing the feature index generated by the index generation logic module according to a single feature, and encrypting it in ciphertext to securely store it in the distributed file system hadoop A security index module, a metadata engine module that manages index segment files generated by the security index module; The metadata engine module is an index engine that manages the segment files generated by the security index module and can quickly locate segments according to characteristics, including a metadata index module that includes a complete index of the features in the segments in the search domain and the metadata indexed by metadata The indexing module indexes and indexes the metadata indexing extension module of the security index; The secure index module includes a cache module that writes the inverted index file set generated by the index generation logic module into the index cache and supports index cache update, analyzes the index cache data in the cache, and generates persistent optimization task and the optimizer that analyzes the security index segment in the metadata engine and generates the corresponding segment optimization task. 2. The big data platform security indexing system according to claim 1, characterized in that, The optimizer includes an analyzer that checks the index cache data that needs to be persisted in the cache and generates a persistence task, and analyzes the status information of the secure index segment recorded in the metadata engine to generate a segment optimization task, according to the The task analyzed by the analyzer generates a task queue module of the task queue, and an executor for processing the tasks recorded in the task queue module. 3. a big data platform security indexing method, is characterized in that, described big data platform security indexing method adopts big data platform security indexing system to realize security indexing, comprises the steps: s1. The user submits the keyword to be queried through the client, receives the keyword information submitted by the client through the big data platform security index system, and searches according to the keyword in the search domain to identify documents matching the keyword; s2. The server provides documents matching keywords to the big data platform security indexing system; s3. The index generation logic module in the secure index system of the big data platform processes the documents provided by the server and matches the keywords to generate a corresponding inverted index file set including feature indexes, the inverted files and Contains feature term and document extraction credential docID; s4. The security index module in the security index system of the big data platform intelligently segments the inverted file set generated in s3 according to a single feature, and the size and specification of each segment are unified, and each segment is encrypted to encrypt The file format is stored in the distributed file system Hadoop; first, write the inverted file set into the index cache through the cache module in the security index module and support the index cache update at the same time; secondly, through the security index The optimizer in the module analyzes the index cache data in the cache module, and generates index persistence tasks and segment optimization tasks on demand; s5. Manage the index segment files generated in s4 through the metadata engine module in the indexing system of the big data platform, and search and locate the file features according to the needs and quickly read the segment where the positioning features are located. 4. The big data platform security indexing method according to claim 3, characterized in that, Updating the index through the cache module includes the following steps: g1: start; g2. Perform feature analysis on the document and extract document features; g3. Generate an inverted file according to the characteristics of the document; g4, generate an inverted file list; g5. Determine whether the document feature already exists; if the feature already exists, go to g6; if the feature does not exist, go to g9; g6. Locate the index segment according to the feature; g7. Add new data to the index segment according to the size and specification of the index segment; g8. Whether there are any remaining files in the inverted file; if there are remaining files, go to g4; if there are no remaining files, go to g11; g9. According to the characteristics of the document, create a new index segment and write the file data, then go to g8; g10, end. 5. The big data platform security indexing method according to claim 3, characterized in that, Through the cache module, index persistence tasks are generated according to the index persistence requirements, including the following steps: c1, start; c2, analyzing the index cache data in the cache; c3. Generate persistent tasks according to characteristics; c4. Generate a feature list to be processed according to the features that need to be persisted; c5. Determine whether the feature already exists, if so, go to c8; if not, go to c6; c6. Persist new features; c7, determine whether the feature list is empty; if so, go to c9; if not, go to c4; c8. Persist the old features and transfer to c6; c9, end. 6. The big data platform security indexing method according to claim 5, characterized in that, The persistence of the new feature comprises the following steps: x1, start; x2, Create a new empty segment template; x3, write data data in the newly created segment; x4. Update head data; x5, encrypt the segment; x6. Notify the metadata engine module. 7. The big data platform security indexing method according to claim 5, characterized in that, The persistence of the old features comprises the following steps: l1, start; l2. Locate the latest segment where the feature is located; l3. Extract the file corresponding to the index segment; l4. Decrypt files; l5. Update files; l6. Encrypted files; l7. Store the encrypted file in Hadoop; l8. Enter the countdown to delete the old segment; l9. Notify the metadata engine module. 8. The big data platform indexing method according to claim 3, characterized in that, The optimizer analyzes the index cache data, and generates segment optimization tasks including the following steps: d1, start; d2. Analyze security index; d3, generating optimization tasks according to the analysis results in d2; d4, judging whether the segment needs to be deleted, if so, go to d5; if not, go to d6; d5, delete the expired segment; d6, judge whether to split the segment, if so, go to d7; if not, go to d9; d7, split the segment into multiple new segments; d8. Enter the countdown to delete the old segment; d9, judging whether to merge segments; if so, go to d10; if not, go to d12; d10, merging segment; d11. Enter the countdown to delete the old segment; d12. Notify the metadata engine module.