CN103064933B - Data query method and system - Google Patents

Abstract

The present invention relates to a data query method and system. The method includes: receiving an input query request, the query request carrying a field to be queried and query words in the field; Query in the query to obtain the collector identification corresponding to the query word; generate a query command carrying the field and the query word according to the query request, and send the query command to the data collector corresponding to the collector identification, The data collector is used to query the local index table corresponding to the field carried by the query command in the data collector to obtain the data matching the query word carried in the query command; receive the data returned by the data collector the data, and form and output the query result of the query request according to the received data. The invention can improve the processing speed of data query, reduce the system resource occupation of the data collector and the load pressure of the data query server.

Description

Data query method and system

technical field

The invention relates to the technical field of computer networks, in particular to a data query method, a data query server, a data collector and a data query system.

Background technique

In the current era of extremely developed Internet, data collection and query systems have a wide range of uses. Various information technology (Information Technology, IT) systems, network equipment, and security equipment will generate a large number of logs and other data, and many of the log data require long-term archived and used for various audits and inquiries.

In the mass data collection and query system, the system architecture has two methods: distributed storage and centralized storage. No matter which method is used, it is faced with the demand for fast storage and fast query of massive log data.

An existing distributed data query system includes a data query server and multiple data collectors. The data collector is responsible for log collection (receiving, formatting, merging), storage and indexing. The data query server is a log query server. unified entrance. When the specified log needs to be queried, the data query server sends query commands to all data collectors, and after receiving the query results from all data collectors, the final query results are summarized. If there are many data collectors, the logs to be queried in one query only exist in a few data collectors, and the query operation is very frequent, this existing solution will increase the burden of all data collectors, including the power consumption of the data collectors and central processing unit (Central Processing Unit, CPU) resources, at the same time, in addition to querying, the data collector also needs to do the work of data receiving and storage. If the query operation is very frequent, it will also affect the collection performance of the data collector and reduce the system overall processing power.

The original log data of another existing distributed data query system is stored in a centralized manner, and each data collector is only responsible for log collection (receiving, formatting, merging) and reporting, and the log content is processed by the data collector. It is not saved locally, but reported to the data query server for storage. After the data query server receives the log reported by the data collector, it will be centrally stored in the database and indexed. When it is necessary to query the log, it can be directly queried in the database of the data query server. This way of centralized data storage makes log query operations limited to the database of the data query server and will not affect the data collector. However, since the log data is centrally stored in the database of the data query server, the data collector needs to report a large amount of log data. On the one hand, the load on the data query server is greatly increased; In this way, the number of data collectors that can be carried by a data query server is limited, and the processing capacity of the entire system cannot be very high.

Contents of the invention

The invention provides a data query method, a data query server, a data collector and a data query system, which can improve the processing speed of data query, reduce the system resource occupation of the data collector and the load pressure of the data query server, and improve the overall system efficiency. processing power.

In order to achieve the above object, the first aspect of the present invention provides a data query method, the method comprising:

Receiving an input query request, carrying the field to be queried and the query words in the field in the query request;

Obtain the collector identification corresponding to the query word from the centralized index table corresponding to the field, and store the corresponding relationship between the query word in the field and the collector ID in the centralized index table;

Generate a query command carrying the field and query words according to the query request, and send the query command to the data collector corresponding to the collector identification, so that the data collector can be used in the data collector In the local index table corresponding to the field carried by the query command in the query, the data matching the query word carried in the query command is obtained by querying;

The data returned by the data collector is received, and the query result of the query request is formed and output according to the received data.

In conjunction with the first aspect, in the first possible implementation manner of the first aspect, before obtaining the collector identifier corresponding to the query word from the centralized index table corresponding to the field, the method further includes:

For the field, establish a centralized index table corresponding to the field;

The establishment of the centralized index table corresponding to the field includes:

receiving the reporting index table of the field sent by each data collector, the reporting index table including the data in the data collector sending the reporting index table corresponding to the query words of the field;

In the centralized index table of the field, the corresponding relationship between the identifier of the data collector and the query word of the field in the report index table reported by the data collector is stored.

With reference to the first aspect, in the second possible implementation manner of the first aspect, the querying to obtain the collector identifier corresponding to the query word from the centralized index table corresponding to the field includes:

If the query request carries at least two fields to be queried, obtain query words of each field in the query request, and record the logical relationship between the query words of each field;

Querying from the centralized index table corresponding to each field to obtain the collector identification corresponding to the query word of each field;

According to the logical relationship between the query words in each field, the collector identifiers satisfying the logical relationship are obtained by screening the collector identifiers obtained from the query.

In a second aspect, the present invention also provides a data query method, the method comprising:

receiving a query command sent by the data query server, the query command including the field to be queried and the query word in the field carried in the query request received by the data query server;

Query from the local index table corresponding to the field to obtain the storage location of the data matching the query word in the query command, and store the query word in the field and the storage location of the data in the local index table corresponding relationship;

According to the storage location of the data, the data is acquired and sent to the data query server.

With reference to the second aspect, in the first possible implementation manner of the second aspect, before the query from the local index table corresponding to the field obtains the storage location of the data matching the query word in the query command ,Also includes:

For the field, establish a local index table corresponding to the field;

The establishment of the local index table corresponding to the field includes:

Obtain the data in the current data collector and the storage location of the data, the data includes the content of at least one field;

For each field, use the content of the data in the field as the query word of the data, establish the mapping relationship between the query word and the storage location, and form the local index table of the field in the current data collector .

With reference to the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, the data is stored in the local index table of the field in the current data collector After the corresponding relationship between the query word and the storage location of the data, it also includes:

extracting the query word from the local index table of the field, and performing deduplication processing on the query word to form the report index table of the field of the current data collector;

Send the report index table of the field to the data query server, so that the data query server can establish a centralized index table corresponding to the field.

With reference to the second aspect, in a third possible implementation manner of the second aspect, the query from the local index table corresponding to the field obtains the storage location of the data matching the query word in the query command, include:

If the query command carries at least two fields to be queried, obtain the query words of each field in the query command, and record the logical relationship between the query words of each field;

Querying from the local index table corresponding to each field to obtain the storage location of the data matched with the query word of each field in the query command;

According to the logical relationship between the query words in each field, the storage locations of the data satisfying the logical relationship are obtained by screening from the storage locations of the data obtained through the query.

In a third aspect, the present invention also provides a data query server, and the data query server includes:

A first receiving unit, configured to receive an input query request, the query request carrying the fields to be queried and query words in the fields;

The first query unit is configured to query from the centralized index table corresponding to the field to obtain the collector identifier corresponding to the query word carried in the query request received by the first receiving unit, and the centralized index table stores all The corresponding relationship between the query word in the above field and the collector identifier;

A first processing unit, configured to generate a query command carrying the field and a query word according to the query request, and send the query command to the data collector corresponding to the collector identifier obtained through query by the first query unit , for the data collector to search in the local index table corresponding to the field carried by the query command in the data collector to obtain data matching the query words carried in the query command;

The first output unit is configured to receive the data returned by the data collector, form and output the query result of the query request according to the received data.

With reference to the third aspect, in a first possible implementation manner of the third aspect, the data query server further includes:

The first indexing unit is configured to, for the field, establish a centralized index table corresponding to the field;

The first index unit includes:

The first receiving subunit is used to receive the report index table of the field sent by each data collector, and the report index table includes the data in the data collector that sends the report index table corresponding to the query of the field word;

The first index subunit is configured to store, in the centralized index table of the field, the corresponding relationship between the identifier of the data collector and the query word of the field in the report index table reported by the data collector.

With reference to the third aspect, in a second possible implementation manner of the third aspect, the first query unit includes:

The first parsing subunit is configured to obtain the query words of each field in the query request if the query request received by the first receiving unit carries at least two fields to be queried, and record the query words of each field Logical relationship between query terms;

The first query subunit is configured to query from the centralized index table corresponding to each field to obtain the collector identifier corresponding to the query word of each field obtained by the first parsing subunit;

The first filtering subunit is configured to filter and obtain from the collector identifier queried by the first query subunit according to the logical relationship between the query words in the fields obtained by the first parsing subunit The identifier of the collector that satisfies the logical relationship.

In a fourth aspect, the present invention also provides a data collector, which includes:

The second receiving unit is used to receive the query command sent by the data query server, the query command carries the field to be queried and the query word in the field;

The second query unit is configured to query from the local index table corresponding to the field to obtain the storage location of the data that matches the query word in the query command received by the second receiving unit, and the local index table stores all The corresponding relationship between the query word in the above field and the storage location of the data;

The second processing unit is configured to acquire the data and send it to the data query server according to the storage location of the data obtained through query by the second query unit.

With reference to the fourth aspect, in a first possible implementation manner of the fourth aspect, the data collector further includes:

The second indexing unit is configured to, for the field, establish a local index table corresponding to the field;

The second index unit includes:

An acquisition subunit, configured to acquire the data in the current data collector and the storage location of the data, the data includes the content of at least one field;

The second index subunit is used for each field acquired by the acquisition subunit, using the content of the data in the field as the query word of the data, and the local index of the field in the data collector In the table, the corresponding relationship between the query word of the data and the storage location of the data is stored.

With reference to the first possible implementation manner of the fourth aspect, in a second possible implementation manner of the fourth aspect, the second indexing unit further includes:

The third index subunit is used to extract the query word from the local index table of the field obtained by the second index subunit, perform deduplication processing on the query word, and form the current data collector The reporting index table of the field;

The sending subunit is configured to send the report index table of the field formed by the third index subunit to the data query server, so that the data query server can establish a centralized index table of the field.

With reference to the fourth aspect, in a third possible implementation manner of the fourth aspect, the second query unit includes:

The second parsing subunit is configured to obtain the query words of each field in the query command if the query command received by the second receiving unit carries at least two fields to be queried, and record the query words of each field Logical relationship between query terms;

The second query subunit is configured to query from the local index table corresponding to each field to obtain the storage location of the data that matches the query words of each field in the query command obtained by the second parsing subunit;

The second filtering subunit is configured to filter from the storage locations of the data queried by the second query subunit according to the logical relationship between the query words in the fields obtained by the second parsing subunit A storage location of data satisfying the logical relationship is obtained.

In the fifth aspect, the present invention also provides a data query system, the system comprising:

The data query server provided in the above third aspect and the data collector provided in the above fourth aspect.

The data query method, data query server, data collector and data query system provided by the embodiments of the present invention can effectively reduce the burden of the data collector by establishing a local index table and a centralized index table in the data collector and the data query server respectively. The system resource occupation allows the data collector to have more resources to improve the performance of the collection, thereby improving the overall processing capability of the system and improving the processing speed of data query.

Description of drawings

Fig. 1 is the architectural diagram of the data inquiry system that the embodiment of the present invention provides;

FIG. 2 is a signaling diagram of an index establishment process provided by Embodiment 1 of the present invention;

Fig. 3 is a flow chart of a data query method provided by Embodiment 1 of the present invention;

FIG. 4 is a flowchart of another data query method provided by Embodiment 1 of the present invention;

FIG. 5 is a schematic diagram of a data query system provided by Embodiment 2 of the present invention;

6 is a schematic diagram of a data query server and a data collector provided in Embodiment 2 of the present invention;

FIG. 7 is a schematic diagram of a data query server provided by Embodiment 3 of the present invention;

FIG. 8 is a schematic diagram of a data collector provided by Embodiment 3 of the present invention.

detailed description

The technical solutions of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

Fig. 1 is the architectural diagram of the data query system that the embodiment of the present invention provides, as shown in Fig. 1, the present invention adopts distributed structure, comprises a data query server 10 and a plurality of data collectors 20, and data collector 20 is responsible for The data query server 10 is a unified entrance for data query for the collection (including receiving, formatting, and merging), storage, and indexing of massive logs reported by the log source 30 .

The data query method provided by the present invention can be used for fast query of massive data, and log data is taken as an example in the following embodiments for illustration.

Embodiment one

Before querying log data, it is necessary to build an index on the data stored in the system in advance, usually during data storage, so that the system can query data based on the established index table.

In this embodiment, a local index table and a centralized index table are respectively established in the data collector and the data query server. The local index table is used to store the index of the log data in the current data collector. Its function is: when the query condition is given, the specific storage location of all logs that meet the conditions in the local data can be found. The centralized index table is used to store the query words of each field and the index of the collector identification. Its function is: when the query condition is given, it can find out which data collectors the data to be queried may be stored on. The centralized index table gives The identification information of the data collector that stores the data to be queried.

Figure 2 is a signaling diagram of the index establishment process provided by this embodiment, as shown in Figure 2, including:

Step S101, the data collector acquires the data currently in the data collector and the storage location of the data.

Optionally, the data stored in the data collector is the original log data reported by the log source. After the log source reports the original log data to the data collector, the data collector also needs to build a local index for the original log data.

The data collector formats and merges the original log data, and processes the original log data into the form of each record in the log table (that is, each row record in the log table). Each log table may have multiple fields, as follows As shown in Table 1, the log table includes fields such as field 1 and field 2, and the serial number indicates the storage location of the data.

Table 1

serial number field 1 content field 2 content ... 1 bbb fff ... 2 aaa ccc ... 3 aaa ggg ...

Step S102, for each field, the data collector uses the content of the data in the field as the query word of the data, establishes a mapping relationship between the query word and the storage location, and forms the field in the current data The local index table of the collector.

In the local index of the data collector, a local index table is established for each field of each record in the log table, and each index table corresponds to the content of a field in the specified log table and contains the specified field content The storage location information of the data in the log table. Among them, the field content is used as the query word of the corresponding data. The mapping relationship between the query word and the storage location can be expressed in the form of a table, but not limited to, as shown in Table 2 below, which is the local index table of field 1 in the data collector:

Table 2

field 1 content Location aaa 2 aaa 3

bbb 1

The specific location of the data on the data collector can be quickly found through the local index table of the corresponding field. For example, if you want a specific location of a certain content in field 1, you can quickly find the corresponding location according to the local index table of field 1.

Step S103 , the data collector extracts the query word from the local index table of the field, performs deduplication processing on the query word, and forms a report index table of the field of the current data collector.

The data collector extracts the reported reporting index table according to the local index table of the field. In the report index table, there is no specific original log location corresponding to each index content, but only the content of each data corresponding to the field, that is, the query word. Usually, before reporting, the query words will be deduplicated, so that the content of each field reported will not be repeated. For example, there are two aaa and one bbb in field 1 in Table 2. After deduplication processing, there is only one aaa and one bbb in the reporting index table, as shown in Table 3 below.

table 3

field 1 content aaa bbb ...

For the query words extracted from the newly added local index table, compare them with the reported report index table. If the query words are the same, they will not be added to the report index table, and only the newly added non-repeated report will be uploaded. direction chart.

Step S104, the data collector sends the report index table of the field to the data query server.

The data collector reports the reporting index table shown in Table 3 to the data query server. If there is an update in the local index table, only upload the newly added content in the report index table generated according to the updated local index table.

The data query server summarizes the reporting index tables sent by multiple data collectors, and establishes centralized index tables corresponding to each field in the data query server, specifically including:

Step S105, the data query server receives the report index table of the field sent by each data collector.

The report index table of the data collector includes the query words corresponding to the fields in the data in the data collector.

Step S106, establishing a mapping relationship between the query word and the collector identifier, and forming a centralized index table for the field.

For different fields, the data query server establishes centralized index tables for corresponding fields. For example, for field 1, the centralized index table of the data query server will store the query words contained in each data collector and the corresponding collector identifier, as shown in Table 4 below:

Table 4

Field 1 content Collector ID aaa Collector 1 aaa Collector 2 bbb Collector 1

As another example, for field 2, the centralized index table established by the data query server is shown in Table 5 below:

table 5

field 2 content Collector ID ccc Collector 1 ddd Collector 2 eee Collector 3

There is a centralized index table corresponding to each field in the data query server, and the reported index table reported by each data collector is summarized into the centralized index table of the data query server, that is, in the centralized index table of the field on the data query server storing the correspondence between the identifier of the data collector and the query words in this field in the reporting index table reported by the data collector.

After the index table is established, when a query request is received, the data to be queried can be found through the index table.

Fig. 3 is a flow chart of the data query method provided in this embodiment, as shown in Fig. 3, the data query method of the present invention includes:

Step S201, the data query server receives an input query request.

The user enters a query request by means of table search or expression search. For the form retrieval method, fixed fields will be given in the user interface, and the user can enter query words in the prompt boxes of multiple fields, and finally submit the query request to the data query server by pressing the submit button. For the method of expression retrieval, the user directly inputs the field to be queried and the query word of the field, and submits it to the data query server.

The query request received by the data query server carries the field to be queried and the query words of the field.

When the query request includes multiple fields, the query request received by the data query server also carries the logical relationship between the query words in each field. For example, the user inputs query words in different search fields, and the data query server receives these query words in each different search fields and the logical relationship between each query word.

For example, the input query request is: (field 1=aaa) AND (field 2=ccc), the query request includes two fields, field 1 and field 2, and the corresponding query words are aaa and ccc, and AND means The logical relationship between these two query words is the relationship of "and".

Step S202, the data query server queries the centralized index table corresponding to the field to obtain the collector identifier corresponding to the query word.

When field 1 is included in the query request, the query word matching the query word is obtained from the centralized index table of field 1, and the corresponding collector identifier is obtained, so as to obtain which data collector the query word is in.

For example, if the query request is: field 1=aaa, then the collector identifiers corresponding to aaa are obtained from the centralized index table shown in Table 4 as collector 1 and collector 3 .

For the case where the query request includes multiple fields, this step specifically includes:

Step S2021. Obtain the query terms of each field in the query request, and record the logical relationship between the query terms of each field carried in the query request.

Step S2022, querying the centralized index table corresponding to each field to obtain the identifier of the collector corresponding to the query word in each field.

Step S2023: According to the logical relationship between the query words in each field, filter the collector identifier obtained from the query in step S2022 to obtain the collector identifier satisfying the logical relationship.

For example, if the query request is: (field 1=aaa) AND (field 2=ccc), then the collectors corresponding to aaa are identified as collector 1 and collector 3 from the centralized index table shown in Table 4. The collector ID corresponding to ccc obtained from the centralized index table shown in 5 is collector 1, and the logical relationship between aaa and ccc is "AND", then only collector 1 can be selected as the collector ID satisfying the logical relationship.

Step S203, the data query server generates a query command carrying the fields and query words according to the query request, and sends the query command to the data collector corresponding to the collector identifier.

The query command generated by the data query server may be the same as the input query request, or may only include query words sent by the destination data collector. For example, if the query request is: (field 1=aaa) OR (field 2=ccc), then the collectors corresponding to aaa are identified as collector 1 and collector 3 by querying from the centralized index table shown in Table 4. The collector identifier corresponding to ccc obtained from the centralized index table shown in 5 is collector 1, the logical relationship between aaa and ccc is "or", and the query command sent to collector 1 is: (field 1=aaa) OR(field 2=ccc), the query command sent to the collector 3 is: field 2=ccc.

The data collector queries the local index table of the data collector to obtain the data matching the query word, and the specific process will be described in detail later with reference to FIG. 3 .

Step S204, the data query server receives the data returned by the data collector, forms and outputs a query result of the query request according to the received data.

The data query server summarizes the received data, which can be, but not limited to, output in the form of a table.

For example, if the query request is: (field 1=aaa) AND (field 2=ccc), the final output query results are shown in Table 6 below:

Table 6

serial number field 1 field 2 field 3 2 aaa ccc iii

Fig. 4 is a flow chart of another data query method provided in this embodiment. As shown in Fig. 4, the data query method of the present invention includes:

Step S301, the data collector receives the query command sent by the data query server.

The query command includes the field to be queried and the query words of the fields carried in the query request received by the data query server. Optionally, it may also include the query words of multiple fields and the query words between the query words. Logic.

Step S302, the data collector queries the local index table corresponding to the field to obtain the storage location of the data matching the query command.

For the case where the query command contains query words in multiple fields, this step specifically includes:

Step S3021, the data collector obtains the query words of each field in the query command received in step S301, and records the logical relationship between the query words of each field carried in the query command.

Step S3022, the data collector queries the local index table corresponding to each field to obtain the storage location of the data matching the query word in each field.

For example, if the query command is: (field 1=aaa) OR (field 2=ccc), then the storage location corresponding to the data obtained from the query in the local index table of field 1 is 2, and the data obtained in the local index table of field 2 The corresponding storage location is also 2.

Step S3023. According to the logical relationship between the query words in each field, the data collector screens the storage locations of the data obtained from the query to obtain the storage locations of the data satisfying the logical relationship.

The data collector screens the storage locations of the matched data according to the logical relationship between the query words to obtain the storage locations of the data satisfying the logical relationship.

Step S303, the data collector obtains the data according to the storage location of the data and sends it to the data query server.

The data collector acquires the corresponding data according to the storage location of the data obtained in step S302, for example: the data that can be obtained to satisfy the query command is:

serial number field 1 field 2 field 3 2 aaa ccc iii

The data collector sends the data to the data query server, so that the data query server summarizes and outputs the query results of the query request.

In the data query method provided by this embodiment, by establishing a local index table and a centralized index table in the data collector and the data query server respectively, when querying data, the collector identifier corresponding to the query word is found through the centralized index table of the corresponding field, In this way, the corresponding data can be obtained in the data collector corresponding to the collector identifier, which can effectively reduce the system resource occupation of the data query server and the data collector, so that the data collector can have more resources for improving the performance of the collection. Improve the processing speed of data query.

The above is a detailed description of the data query method provided by the present invention, and the data query system provided by the present invention will be described in detail below.

Embodiment two

FIG. 5 is a schematic diagram of a data query system provided in this embodiment. As shown in FIG. 5 , the data query system in this embodiment of the present invention includes: a data query server 10 and a data collector 20 . Among them, the data collector 20 is responsible for data collection, including receiving, formatting, merging, data storage and indexing of the stored data. The data query server 10 is used for unified management of the contents stored on multiple data collectors 20, and serves as a unified entrance for data query.

Figure 6 is a schematic diagram of the data query server 10 and the data collector 20 provided by this embodiment, as shown in Figure 6, the data query server 10 includes a first index unit 100, a first receiving unit 101, a first query unit 102, a A processing unit 103 and a first output unit 104 .

The data collector 20 includes a second indexing unit 200 , a second receiving unit 201 , a second querying unit 202 and a second processing unit 203 .

Before querying the log data, the data collector 20 and the data query server 10 need to build an index on the data stored in the system in advance, which is usually completed during data storage, so that the system can query the data according to the established index table. The data query server 10 uses the first indexing unit 100 to create a centralized index table for each of the fields. The data collector 20 utilizes the second index unit 200 to establish a local index table of fields.

The local index table is used to store the index of the log data in the current data collector. Its function is: when the query condition is given, the specific storage location of all logs that meet the conditions in the local data can be found. The centralized index table is used to store the index of the data to be queried and the identifier of the collector. Its function is: when the query condition is given, it can be found which data collectors the data to be queried may be stored on. The centralized index table gives the storage The identification information of the data collector whose data is to be queried.

Wherein, the second indexing unit 200 includes an acquiring subunit 2001 , a second indexing subunit 2002 , a third indexing subunit 2003 and a sending subunit 2004 .

The acquiring subunit 2001 is configured to acquire the data in the current data collector and the storage location of the data, and the data includes the content of at least one field.

Optionally, the data stored in the data collector is the original log data reported by the log source. After the log source reports the original log data to the data collector, the data collector also needs to build a local index for the original log data.

The data collector formats and merges the raw log data, and processes the raw log data into the form of each record in the log table (that is, each row in the log table). Each log table may have multiple fields, such as As shown in Table 1, the log table includes fields such as field 1 and field 2, and the serial number indicates the storage location of the data.

The second index subunit 2002 is used for each field, using the content of the data in the field as the query word of the data, establishing a mapping relationship between the query word and the storage location, forming the field in the The local index table of the current data collector.

In the local index of the data collector 20, a local index table is established for each field of each record in the log table, and each index table corresponds to a certain field content in the specified log table and contains the specified certain field The storage location information of the content data in the log table. Among them, the field content is used as the query word of the corresponding data. The mapping relationship between the query word and the storage location can be expressed in the form of a table, as shown in Table 2, but is not limited to.

The specific location of the data on the data collector can be quickly found through the local index table of the corresponding field. For example, if you want a specific location of a certain content in field 1, you can quickly find the corresponding location according to the local index table of field 1.

The third index subunit 2003 is configured to extract the query word from the local index table of the field, and perform deduplication processing on the query word to form the report index table of the field of the current data collector.

The data collector extracts the reported reporting index table according to the local index table of the field. In the report index table, there is no specific original log location corresponding to each index content, but only the content of each data corresponding to the field, that is, the query word. Usually, before reporting, the query words will be deduplicated, so that the content of each field reported will not be repeated. For example, there are two aaa and one bbb in field 1 in Table 2. After deduplication processing, there is only one aaa and one bbb in the reporting index table, as shown in Table 3.

For the query words extracted from the newly added local index table, compare them with the reported report index table. If the query words are the same, they will not be added to the report index table, and only the newly added non-repeated report will be uploaded. direction chart.

The sending subunit 2004 is configured to send the report index table of the field to the first index unit 100 of the data query server 10, so that the data query server can build a centralized index table of the field.

The first indexing unit 100 includes a first receiving subunit 1001 and a first indexing subunit 1002 .

The first receiving subunit 1001 is configured to receive the report index table of the field sent by each data collector. The report index table includes query words corresponding to the fields in which the data in the data collector sending the report index table corresponds.

The first index subunit 1002 is configured to establish a mapping relationship between the query word and the collector identifier to form a centralized index table for the field.

For different fields, the first indexing subunit 1002 establishes centralized index tables corresponding to the fields. For example, for field 1, the query words contained in each data collector and the corresponding collector identifiers are stored in the centralized index table, as shown in Table 4.

The first index subunit 1002 establishes a centralized index table corresponding to each field, and the reported index table reported by each data collector is summarized into the centralized index table of the data query server.

After the index tables are established in the data query server 10 and the data collector 20 respectively by the first index unit 100 and the second index unit 200, when a query request is received, the data to be queried can be found through the index tables.

The first receiving unit 101 is configured to receive an input query request.

The user enters a query request by means of table search or expression search. For the form retrieval method, fixed fields will be given in the user interface, and the user can enter query words in the prompt boxes of multiple fields, and finally submit the query request to the data query server by pressing the submit button. For the method of expression retrieval, the user directly inputs the field to be queried and the query word of the field, and submits it to the data query server.

The query request received by the first receiving unit 101 carries the field to be queried and the query word of the field.

When the query request includes multiple fields, the query request received by the first receiving unit 101 also includes a logical relationship between query words in each field. For example, the user inputs query words in different search fields, and the first receiving unit 101 receives these query words in different search fields and the logical relationship between the query words.

The first query unit 102 is configured to query from the centralized index table of the field to obtain the collector identifier corresponding to the query word carried in the query request received by the first receiving unit 101 .

When the query request includes field 1, the first query unit 102 obtains the query word that matches the query word from the centralized index table of field 1, and obtains the corresponding collector identifier, thereby obtaining the query word in which data collector.

For the case where the query request includes multiple fields, the first query unit 102 includes: a parsing subunit, a first query subunit and a first filtering subunit (not shown in the figure).

The first parsing subunit is used to obtain query words in each field in the query request, and record the logical relationship between the query words in each field.

The first query subunit is configured to query from the centralized index table corresponding to the field to obtain the collector identifier corresponding to the query word of each field obtained by the first parsing subunit.

The first filtering subunit is configured to filter the collector identification obtained from the query from the first query subunit according to the logical relationship between the query words in the fields obtained by the first parsing subunit to obtain the The collector identifier of the logical relationship.

The first processing unit 103 is configured to generate a query command carrying the field and the query word according to the query request, and send the query command to the data collector 20 corresponding to the collector identification obtained by the query obtained by the first query unit 102 .

The query command generated by the first processing unit 103 may be the same as the input query request, or may only include query words sent by the destination data collector. For example, if the query request is: (field 1=aaa) OR (field 2=ccc), then the first query unit 102 can obtain the collector identification corresponding to aaa from the centralized index table shown in Table 4 as collector 1 and collector 3, the first query unit 102 queries from the centralized index table shown in Table 5 to obtain the collector identifier corresponding to ccc as collector 1, the logical relationship between aaa and ccc is "or", and the first processing unit 103 generates The query command sent to the collector 1 is: (field 1=aaa) OR (field 2=ccc), and the query command generated by the first processing unit 103 and sent to the collector 3 is: field 2=ccc.

The second receiving unit 201 of the data collector 20 is used for receiving the query command sent by the data query server 10 .

The query command includes the field to be queried and the query words of the fields carried in the query request received by the data query server 10, and may include the query words of multiple fields and the logical relationship between the query words.

The second query unit 202 is configured to query the local index table corresponding to the field to obtain the storage location of the data matching the query word in the query command received by the second receiving unit 201 .

Optionally, for the case where the query command contains a query word in multiple fields, the second query unit 202 includes: a second parsing subunit, a second query subunit, and a second filtering subunit (not shown in the figure).

The second parsing subunit is used to obtain the query words of each field in the query command if the query command received by the second receiving unit 201 carries a plurality of fields to be queried, and record the query words carried in the query command The logical relationship between query words in each field.

The second query subunit is configured to query the local index table corresponding to each field to obtain the storage location of the data that matches the query word of each field in the query command acquired by the second parsing subunit.

The second filtering subunit is used to filter and obtain from the storage location of the data queried by the second query subunit according to the logical relationship between the query words in the fields obtained by the second parsing subunit The storage location of the data satisfying the logical relationship.

The second processing unit 203 is configured to obtain the data and send it to the data query server according to the storage location of the data obtained through query by the second query unit 202 .

The second processing unit 203 sends the data queried by the second query unit 202 to the first output unit 104 of the data query server 10, so as to output the query result of the query request.

The first output unit 104 is configured to receive the data returned by the second processing unit 203 of the data collector 20, form and output the query result of the query request according to the received data.

In the data query server, data collector and system provided by the embodiments of the present invention, the first index unit is used to establish a centralized index table in the data query server, and the second index unit is used to establish a local index table in the data collector, which can improve the efficiency of data query. processing speed. For the situation that the data to be queried only exists in a small number of data collectors, the present invention does not require every data collector to inquire, which reduces the burden on the data collectors. For the log system that requires frequent query, the solution provided by the embodiment of the present invention can effectively reduce the system resource occupation of the data collector, so that the data collector can have more resources to improve the performance of collection, thereby improving the overall processing of the system ability.

Embodiment Three

FIG. 7 is a schematic diagram of the data query server 10 provided in this embodiment. As shown in FIG. 7 , the data query server 10 includes: a network interface 71 , a processor 72 and a memory 73 . The system bus 74 is used to connect the network interface 71 , the processor 72 and the memory 73 .

The network interface 71 is used to communicate with the data collector 20 .

The memory 73 can be a permanent memory, such as a hard disk drive and a flash memory, and has software modules and device drivers in the memory 73 . The software modules can execute various functional modules of the above method of the present invention; the device driver can be a network and interface driver.

At startup, these software components are loaded into memory 73, then accessed by processor 72 and execute the following instructions:

Receiving an input query request, carrying the field to be queried and the query words in the field in the query request;

Obtain the collector identification corresponding to the query word from the centralized index table corresponding to the field, and store the corresponding relationship between the query word in the field and the collector ID in the centralized index table;

Generate a query command carrying the field and query words according to the query request, and send the query command to the data collector corresponding to the collector identification, so that the data collector can pass through the data collector The local index table corresponding to the field carried by the query command is queried to obtain data matching the query words carried in the query command;

The data returned by the data collector is received, and the query result of the query request is formed and output according to the received data.

The data query server in this embodiment finds the collector identifier corresponding to the query word through the centralized index table of the field, thereby obtaining corresponding data in the data collector corresponding to the collector identifier, which can effectively reduce the system resource occupation of the data query server. Improve the processing speed of data query.

Further, after the processor accesses the software components of the memory 73, it executes the instructions of the following process:

For the field, establish a centralized index table corresponding to the field;

The establishment of the centralized index table corresponding to the field includes:

receiving the reporting index table of the field sent by each data collector, the reporting index table including the data in the data collector sending the reporting index table corresponding to the query words of the field;

A mapping relationship between the query word and the collector identifier is established to form a centralized index table for the field.

The above instruction process is the process of the data query server establishing a centralized index table. By establishing the mapping relationship between the query word and the collector ID, in order to find the collector ID corresponding to the query word during data query, and get the matching from the corresponding data collector The data.

Further, after the processor accesses the software components of the memory 73, it executes the instructions of the following process:

When the query request carries at least two fields to be queried, obtain query words of each field in the query request, and record the logical relationship between the query words of each field;

Querying from the centralized index table corresponding to each field to obtain the collector identification corresponding to the query word of each field;

According to the logical relationship among the query words in each field, the collector identifiers satisfying the logical relationship are obtained by screening the collector identifiers obtained from the query.

The above instruction process is the process in which the data query server searches for the corresponding collector identifiers for the query words in multiple fields to be queried, which can avoid accessing data collectors that cannot fully satisfy the query request.

FIG. 8 is a schematic diagram of the data collector 20 provided in this embodiment. As shown in FIG. 8 , the data collector 20 includes: a network interface 81 , a processor 82 and a memory 83 . The system bus 84 is used to connect the network interface 81 , the processor 82 and the memory 83 .

The network interface 81 is used for communicating with the data query server 10 .

The memory 83 can be a permanent memory, such as a hard disk drive and a flash memory, and has software modules and device drivers in the memory 83 . The software modules can execute various functional modules of the above method of the present invention; the device driver can be a network and interface driver.

At startup, these software components are loaded into memory 83 and then accessed by processor 82 to execute the following instructions:

receiving a query command sent by the data query server, the query command including the field to be queried and the query word in the field carried in the query request received by the data query server;

Query the storage location of the data matched with the query command from the local index table corresponding to the field, and store the corresponding relationship between the query word in the field and the storage location of the data in the local index table;

According to the storage location of the data, the data is acquired and sent to the data query server.

The data collector of this embodiment finds the data corresponding to the query word through the local index table of the field, and provides it to the data query server, which can effectively reduce the system resource occupation of the data collector, so that the data collector can have more resources for Improve the performance of collection and improve the processing speed of data query.

Further, after the processor accesses the software components of the memory 83, it executes the instructions of the following process:

For the field, establish a local index table corresponding to the field;

The establishment of the local index table corresponding to the field includes:

Obtain the data in the current data collector and the storage location of the data, the data includes the content of at least one field;

For each field, use the content of the data in the field as the query word of the data, establish the mapping relationship between the query word and the storage location, and form the local index table of the field in the current data collector .

The above instruction process is the process for the data collector to establish a local index table. By establishing the mapping relationship between the query word and the storage location of the data, the data is obtained according to the storage location of the data corresponding to the query word.

Further, after the processor accesses the software components of the memory 83, it executes the instructions of the following process:

extracting the query word from the local index table of the field, and performing deduplication processing on the query word to form the report index table of the field of the current data collector;

Send the report index table of the field to the data query server, so that the data query server can establish a centralized index table corresponding to the field.

The above instruction process is the process in which the data collector establishes the reporting index table according to the local index table and sends it to the data query server, so that the data query server can establish a centralized index table.

Further, after the processor accesses the software components of the memory 83, it executes the instructions of the following process:

When at least two fields to be queried are carried in the query command, the query words of each field in the query command are obtained, and the logical relationship between the query words of the fields is recorded;

Querying from the local index table corresponding to each field to obtain the storage location of the data matching the query words of each field;

According to the logical relationship between the query words in each field, the storage locations of the data satisfying the logical relationship are obtained by screening from the storage locations of the data obtained through the query.

The above instruction process is the process in which the data collector finds the corresponding storage locations for the query words in multiple fields to be queried, which can avoid obtaining data that cannot fully satisfy the query command.

Professionals should further realize that the units and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the relationship between hardware and software Interchangeability. In the above description, the composition and steps of each example have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. Protection scope, within the spirit and principles of the present invention, any modification, equivalent replacement, improvement, etc., shall be included in the protection scope of the present invention.

Claims (11)

1. a data query method, it is characterised in that described method includes:

Data query server receives the inquiry request of user's input, carries the query word in the field to inquire about and described field in described inquiry request；

The inquiry from centralized indexes table corresponding to described field of described data query server obtains the harvester mark that described query word is corresponding, stores the corresponding relation of the query word in described field and harvester mark in described centralized indexes table；

Described data query server generates the querying command carrying described field and query word according to described inquiry request, and described querying command is sent to the data acquisition unit that described harvester mark is corresponding, in the local index table that the field carried in order to described data acquisition unit querying command described in described data acquisition unit is corresponding, inquiry obtains the data matched with the query word that carries in described querying command；

Described data query server receives the described data that described data acquisition unit returns, and forms the Query Result of described inquiry request according to the data received and exports；

Wherein, before the centralized indexes table that described data query server is corresponding from described field, inquiry obtains the harvester mark that described query word is corresponding, also include:

Described data query server, for described field, sets up the centralized indexes table that described field is corresponding；

Described set up the centralized indexes table that described field is corresponding, including:

What described data query server received the described field that each data acquisition unit sends reports concordance list, described in report concordance list include sending described in report the data in the data acquisition unit of concordance list corresponding to the query word of described field；

Described data query server is in the centralized indexes table of described field, and what the mark of storage data acquisition unit and described data acquisition unit reported reports the corresponding relation of the query word of this field in concordance list.

2. data query method according to claim 1, it is characterised in that the inquiry from centralized indexes table corresponding to described field of described data query server obtains the harvester mark that described query word is corresponding, including:

If carrying the field that at least two to be inquired about in described inquiry request, described data query server obtains the query word of each field in described inquiry request, and records the logical relation between the query word of the described each field carried in described inquiry request；

The inquiry from centralized indexes table corresponding to described each field of described data query server obtains the harvester mark that the query word of described each field is corresponding；

Described data query server is according to the logical relation between the query word of described each field, and from the described harvester mark that inquiry obtains, screening is met the harvester mark of described logical relation.

3. a data query method, it is characterised in that described method includes:

Data acquisition unit receives the querying command that data query server sends, and described querying command carries the query word in the field to inquire about and described field；

The inquiry from local index table corresponding to described field of described data acquisition unit obtains the storage position of the data matched with the query word in described querying command, stores the corresponding relation of the query word in described field and the storage position of described data in described local index table；

The described data acquisition unit storage position according to described data, obtains described Data Concurrent and gives described data query server；

Wherein, before the local index table that described data acquisition unit is corresponding from described field, inquiry obtains the storage position of data matched with the query word in described querying command, also include:

Described data acquisition unit, for described field, sets up the local index table that described field is corresponding；

Described set up the local index table that described field is corresponding, including:

Data acquisition unit obtains the storage position of the data in Current data acquisition device and described data, and described data include the content of at least one field；

Described data acquisition unit is for each field, using described data in the content of this field as the query word of described data, in the local index table of field described in described Current data acquisition device, store the corresponding relation of the query word of described data and the storage position of described data.

4. data query method according to claim 3, it is characterised in that in the described local index table of field described in described Current data acquisition device, after storing the corresponding relation of the query word of described data and the storage position of described data, also includes:

Described data acquisition unit extracts described query word from the local index table of this field, and described query word is carried out duplicate removal process, formed described Current data acquisition device described field report concordance list；

The concordance list that reports of described field is sent to described data query server by described data acquisition unit, sets up, in order to described data query server, the centralized indexes table that described field is corresponding.

5. data query method according to claim 3, it is characterised in that the inquiry from local index table corresponding to described field of described data acquisition unit obtains the storage position of the data matched with the query word in described querying command, including:

If carrying the field that at least two to be inquired about in described querying command, data acquisition unit obtains the query word of each field in described querying command, and records the logical relation between the query word of each field carried in described querying command；

The inquiry from local index table corresponding to described each field of described data acquisition unit obtains the storage position of the data matched with the query word of each field in described querying command；

Described data acquisition unit is according to the logical relation between the query word of described each field, and from the storage position inquiring about the described data obtained, screening is met the storage position of the data of described logical relation.

6. a data query server, it is characterised in that described data query server includes:

First receives unit, for receiving the inquiry request of user's input, carries the query word in the field to inquire about and described field in described inquiry request；

First query unit, for in the centralized indexes table corresponding from described field, inquiry obtains described first and receives the harvester mark that the query word that carries of described inquiry request that unit receives is corresponding, stores the corresponding relation of the query word in described field and harvester mark in described centralized indexes table；

First processing unit, for generating the querying command carrying described field and query word according to described inquiry request, and described querying command is sent to the data acquisition unit of the harvester mark correspondence that described first query unit inquiry obtains, in the local index table that the field carried in order to described data acquisition unit querying command described in described data acquisition unit is corresponding, inquiry obtains the data matched with the query word that carries in described querying command；

First output unit, for receiving the described data that described data acquisition unit returns, forms the Query Result of described inquiry request according to the data received and exports；

Described data query server also includes:

First indexing units, for for described field, setting up the centralized indexes table that described field is corresponding；

Described first indexing units includes:

First receives subelement, reports concordance list for what receive described field that each data acquisition unit sends, described in report concordance list include sending described in report the data in the data acquisition unit of concordance list corresponding to the query word of described field；

First index subelement, in the centralized indexes table of described field, what the mark of storage data acquisition unit and described data acquisition unit reported reports the corresponding relation of the query word of this field in concordance list.

7. data query server according to claim 6, it is characterised in that described first query unit includes:

First resolves subelement, if receiving in the described inquiry request that unit receives for described first and carrying the field that at least two to be inquired about, obtain the query word of each field in described inquiry request, and record the logical relation between the query word of the described each field carried in described inquiry request；

First inquiry subelement, obtains the harvester mark that the query word of described each field of described first parsing subelement acquisition is corresponding for inquiry the centralized indexes table corresponding from described each field；

First filters subelement, for the logical relation between the query word of the described each field obtained according to described first parsing subelement, inquire about from described first and the described harvester mark that subelement inquiry obtains is screened the harvester mark being met described logical relation.

8. a data acquisition unit, it is characterised in that including:

Second receives unit, and for receiving the querying command that data query server sends, described querying command carries the query word in the field to inquire about and described field；

Second query unit, obtain and the described second storage position receiving the data that the query word in the querying command that unit receives matches for inquiry in the local index table corresponding from described field, described local index table stores the corresponding relation of the query word in described field and the storage position of described data；

Second processing unit, the storage position of the described data for obtaining according to described second query unit inquiry, obtain described Data Concurrent and give described data query server；

Described data acquisition unit also includes:

Second indexing units, for for described field, setting up the local index table that described field is corresponding；

Described second indexing units includes:

Obtaining subelement, be used for the storage position of data and the described data obtaining in Current data acquisition device, described data include the content of at least one field；

Second index subelement, for each field obtained for described acquisition subelement, using described data in the content of this field as the query word of described data, in the local index table of field described in described data acquisition unit, store the corresponding relation of the query word of described data and the storage position of described data.

9. data acquisition unit according to claim 8, it is characterised in that described second indexing units also includes:

3rd index subelement, extracts described query word for indexing the local index table of this field that subelement obtains from described second, described query word is carried out duplicate removal process, formed described data acquisition unit described field report concordance list；

Sending subelement, the concordance list that reports for indexing the described field that subelement is formed by the described 3rd is sent to described data query server, sets up the centralized indexes table of described field in order to described data query server.

10. data acquisition unit according to claim 8, it is characterised in that described second query unit includes:

Second resolves subelement, if receiving in the described querying command that unit receives for described second and carrying the field that at least two to be inquired about, obtain the query word of each field in described querying command, and record the logical relation between the query word of each field carried in described querying command；

Second inquiry subelement, obtains resolving the storage position of the data that the query word of each field matches in the described querying command that subelement obtains with described second for inquiry the local index table corresponding from described each field；

Second filters subelement, for the logical relation between the query word of the described each field obtained according to described second parsing subelement, inquire about the storage position screening the data being met described logical relation the storage position of the described data that subelement inquiry obtains from described second.

11. a data query system, it is characterised in that described system includes:

Data acquisition unit as described in data query server as claimed in claims 6 or 7 and claim as arbitrary in claim 8～10.