CN112463832B - A query method, device and electronic device based on hbase-indexer - Google Patents

Abstract

The invention provides a query method and device based on a hbase-indexer, electronic equipment and a storage medium, belongs to the technical field of servers, and solves the technical problem that the hbase-indexer in the prior art cannot index data into an es searcher. The method comprises the following steps: the client configures parameters for a compatible es searcher which is used for client command input, and sets indexes for distinguishing the solr searcher and the es searcher; the server side confirms the type of the searcher according to the index, if the searcher is an es searcher, the searcher is switched to an es index logic, es connection is established according to the index, and the index is led into the es searcher; and adding query logic in an execution command of the hbase-indexer, wherein the query logic is used for performing combined query according to the parameters and the index when the client runs the combined query command, and returning a query result.

Description

A query method, device and electronic device based on hbase-indexer

technical field

The present invention relates to the technical field of servers, in particular to a query method, device, electronic device and storage medium based on hbase-indexer.

Background technique

Hbase-indexer is a software developed by NGDATA to store related HBase data in the lily subsystem to Solr (solr is a search server, an open source, Java-based information retrieval library). HBase data does not have a full-text retrieval function, and can only maintain high efficiency in the scenario of directly querying row keys, which is not compatible with the full-text retrieval function. Hbase-indexer was born for this situation, but it can only synchronize data to solr, only the synchronization module has no query module, and there is a lack of multi-tenant security mode. At present, many scenarios prefer to use ES (the full name is elasticsearch, which is a search server that provides a full-text search engine with distributed multi-user capabilities), because ES has more advantages than slor. Therefore, the function of Hbase-indexer is currently lacking.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a query method, device, electronic device and storage medium based on hbase-indexer, which alleviates the technical problem that hbase-indexer cannot query es and solr in the prior art.

A first aspect, the query method based on hbase-indexer provided by the present invention comprises the following steps:

The client configures parameters that are compatible with the es searcher passed in by the client command, and sets the index that distinguishes the solr searcher and the es searcher;

The server confirms the searcher type according to the index, and if it is an es searcher, it switches to the es indexing logic;

Create an es connection based on the index, and import the index into the es searcher;

Query logic is added to the execution command of hbase-indexer, and the query logic is used to perform a joint query according to parameters and indexes when the client runs the joint query command, and returns the query result.

Further, after the step of making the parameters passed in the client command compatible with the es searcher in the client and setting the index code to distinguish the solr searcher and the es searcher, it also includes:

The index verification process and the table permission verification process are added to the client.

Further, before the step of creating an es connection according to the index and importing the index into the es searcher, it also includes:

Use the security parameter in the index parameter to verify permission;

If so, pass the verification;

If not, an exception message is generated.

Further, after the step of using the es ticket for security authentication by the server, it further includes:

Add es-compatible parameters to the execution command of batch import history index.

Further, the query method based on hbase-indexer also includes:

Based on ambari (a web-based tool that supports provisioning, management, and monitoring of Apache Hadoop clusters), add service control scripts and custom components to the required configuration, so that clients can install hbase-indexer on the page, and start, Stop, restart, delete operations, and automatically modify parameters.

Further, the step of performing a joint query according to the query parameters and the index, and returning the query result, specifically includes:

receive query commands;

Get hbase-indexer index configuration;

Perform security authentication according to the index configuration and hbase table permissions;

If the authentication is passed, the corresponding query is performed according to the index type;

Get query results;

Get the row key of hbase according to the query result;

Determine the type of the result to be returned. If the result needs to be returned, use the hbase connection, and search for the result according to the row key and return to print. If only the row key needs to be returned, return the row key and print.

Further, before the step of performing the corresponding query according to the index type, the method further includes:

Determine whether the parameter of the query command is an index column and the query format is correct;

If so, convert the query command parameters into the query statement in the corresponding searcher.

In a second aspect, the present invention also provides a query device based on hbase-indexer, comprising:

Client management module, the client configures the parameters that are compatible with the es searcher passed in by the client command, and sets the index that distinguishes the solr searcher and the es searcher;

The server processing module, the server confirms the searcher type according to the index, if it is an es searcher, it switches to the es index logic, creates an es connection according to the index, and imports the index into the es searcher;

In the query module, query logic is added to the execution command of hbase-indexer, and the query logic is used for joint query according to parameters and indexes when the client runs the execution command, and returns the query result.

In a third aspect, the present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor implements the above method when executing the computer program A step of.

In a fourth aspect, the present invention also provides a computer-readable storage medium, where the computer-readable storage medium stores machine-executable instructions, and when the computer-executable instructions are invoked and run by a processor, the computer can run The instructions cause the processor to perform the method.

In the query method based on hbase-indexer provided by the present invention, the client configures the parameters that are compatible with the es searcher input by the client command, sets the index that distinguishes the solr searcher and the es searcher, and the server confirms according to the index The searcher type is switched to the corresponding index logic, a connection is created according to the index, and the index is imported into the searcher; query logic is added to the execution command of hbase-indexer, and the query logic is used when the client runs the execution command according to the parameters and The index performs joint query and returns the query results; completing the above steps makes up for the defect that hbase-indexer cannot index data into the es searcher in the existing solution, and adds the joint query logic corresponding to the searcher and hbase.

Correspondingly, the hbase-indexer-based query apparatus, electronic device, and computer-readable storage medium provided by the embodiments of the present invention also have the above technical effects.

Description of drawings

In order to illustrate the specific embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the specific embodiments or the prior art. Obviously, the accompanying drawings in the following description The drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

1 is a flowchart of a query method based on hbase-indexer provided by an embodiment of the present invention;

Fig. 2 is the query flow chart based on hbase-indexer provided by the embodiment of the present invention;

FIG. 3 is a schematic diagram of a query device based on hbase-indexer provided by an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of them. example. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The terms "comprising" and "having" mentioned in the embodiments of the present invention and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes other unlisted steps or units, or optionally also Include other steps or units inherent to these processes, methods, products or devices.

As shown in FIG. 1, the query method based on hbase-indexer provided by the embodiment of the present invention includes the following steps:

S11: The client configures the parameters that are compatible with the es searcher passed in by the client command, and sets the index that distinguishes the solr searcher and the es searcher;

S12: The server confirms the searcher type according to the index, and if it is an es searcher, switches to the es indexing logic;

S13: Create an es connection based on the index, and import the index into the es searcher;

S14: Add query logic to the execution command of hbase-indexer, where the query logic is used to perform a joint query according to parameters and indexes when the client runs the joint query command, and returns the query result.

In the query method based on hbase-indexer provided by the present invention, the client configures the parameters that are compatible with the es searcher input by the client command, sets the index that distinguishes the solr searcher and the es searcher, and the server confirms according to the index Searcher type and switch to the corresponding index logic, create a connection according to the index, and import the index into the searcher; add query logic to the execution command of hbase-indexer, and the query logic is used when the client runs the joint query command according to the parameters. Perform joint query with the index, and return the query result; completing the above steps makes up for the defect that hbase-indexer cannot index data into the es searcher in the existing solution, and adds the joint query logic corresponding to the searcher and hbase.

In a possible implementation manner, after the steps of making the parameters used for the client command input in the client compatible with the es searcher, and setting the index code to distinguish the solr searcher and the es searcher, the method further includes:

The index verification process and the table permission verification process are added to the client.

In a possible implementation manner, before the step of creating the es connection according to the index and importing the index into the es searcher, the method further includes:

Use the security parameters in the index parameter to verify whether you have permission. If so, create an es connection based on the index, and import the index into the es searcher. If not, generate an exception message.

Before the query, the identity and authority verification will be carried out. When the verification fails, it cannot be used to ensure the security of the system.

In a possible implementation manner, after the step of performing security authentication using the es ticket, the server further includes:

Add es-compatible parameters to the execution command of batch import history index.

After the index is created, historical data needs to be imported into es first. In the existing solution, historical records cannot be imported into es. By adding corresponding parameters, historical records can be smoothly imported into es.

In a possible implementation, the query method based on hbase-indexer further includes:

Based on ambari, add the required configuration for service control scripts and custom components, so that the client can install hbase-indexer on the page, and start, stop, restart, delete, and automatically modify parameters.

As shown in FIG. 2, in a possible implementation manner, the step of performing a joint query according to query parameters and indexes, and returning the query result, specifically includes:

S21: receive a query command;

S22: Get the hbase-indexer index configuration;

S23: Perform security authentication according to the index configuration and the authority of the hbase table, if the authentication is passed, execute step S24, and if the authentication fails, execute step S210;

S24: Perform a corresponding query according to the index type;

S25: get the query result;

S26: Obtain the row key of hbase according to the query result;

S27: Determine the type of the result to be returned, if the result needs to be returned, then go to step S28, if only the row key needs to be returned, go to step S29;

S28: Use hbase connection, and search results according to row keys and return to print;

S29: return row key and print;

S210: Generate abnormal information.

As shown in FIG. 3, in a possible implementation manner, before the step of performing the corresponding query according to the index type, the method further includes:

S240: Determine whether the parameter of the query command is an index column and the query format is correct, if so, execute step S241; if not, execute step S242;

S241: Convert the query command parameter into a query statement in a corresponding searcher.

S242: Send an error message to the client.

Before querying, the format of the search will be determined. When the format is incorrect, the query cannot be performed to ensure the validity of the query.

The specific implementation manner of the query method based on hbase-indexer provided by the embodiment of the present invention is as follows:

Since hbase-indexer is designed for solr, the parameters that can be passed in the client command are required by solr. If you switch to ES, you need to make the parameters passed by the client compatible with the needs of ES, and use "--connection-type-t" " to differentiate between Solr and ES indexers.

At the same time, in order to meet the needs of multi-tenant security scenarios, the client-side cached ticket is used. Before the index operation is executed, it is verified whether the cached ticket has the hbase table permission. If yes, go to the next step. If not, it will fail directly.

When creating an index, you need to add the parameters required for ES security, according to which you can verify the connection to ES, and you can control the number of replicas and shards of the index through parameters. Determine whether the index exists in ES, if it exists, it fails, and if it does not exist, it is created; when deleting the index, delete the index in ES.

According to the difference of the parameter "--connection-type-t", if it is ES, it will enter a parallel logic with solr, create an ES connection according to the parameters of the index, and replace all the processes imported into solr with the processes imported into ES. And add the row key of the execution error to the corresponding data structure.

In the security scenario, the connection on the hbase side uses the ticket corresponding to the authority used by the hbase server, and the connection with the ES uses the security parameters in the index parameter. If the ES non-secure mode and the security mode are converted, the client needs to update the parameters.

In the batch execution module, add the parameters required by ES to distinguish the indexer type, and modify the processing logic. The import method increases the import method of ES, and selects the import method of ES according to the "--connection-type-t" parameter.

According to the incoming parameters or from zookeeper (a distributed application coordination service that provides consistent services for distributed applications, configuration maintenance, domain name services, distributed synchronization, group services and other functions, the index parameters of hbase-indexer are maintained in The index configuration obtained in zookeeper) is used for security authentication of ES connection.

Add the query (ask) option to the execution operation of hbase-indexer and point to the executor of the query operation.

According to the incoming parameters, enter the query logic, if the indexer (index) exists, then obtain the configuration information on the zookeeper, determine whether the verification query parameters are index columns and whether the format is correct, and convert them into the corresponding searcher query. statement, if there are multiple query conditions, the row index is an AND relationship, and the column index is an OR relationship, and judge whether the query mode is the default full-text retrieval mode according to the passed parameters. If it is an exact query, the query relationship of the condition is Exact query.

Obtain the row key of hbase in the return result of the query corresponding to the searcher. If it is a columnar index, it needs to be converted. If the rowkey (an attribute in the xml configuration file used by hbase-indexer to create the index) is configured in the index, it is directly obtained. to this field. Judging from the parameters, if only the row key of hbase needs to be returned, then this is the end to return the row key. If you need to return results, you also need to use hbase connection, batch search based on row key and return print.

In security mode, use the ticket cached by the client to verify hbase permissions, and use the ES security configuration in the obtained index configuration to authenticate, create a connection and query.

Based on ambari, add service control scripts and the configuration required for various custom components, so that hbase-indexer can be installed only when hbase and ES or solr services are installed, and after installation, automatically modify hbase needs to be changed Configuration such as "hbase.replication=true" and so on.

In the case of enabling security, the relevant env configuration such as "-Djava.security.auth.login.config=***" can be automatically modified, and management operations such as opening, stopping, restarting and deleting can be performed through the page.

As shown in FIG. 3 , an embodiment of the present invention further provides a query device based on hbase-indexer, including:

Client management module 1, the client configures the parameters that are compatible with the es searcher passed in by the client command, and sets the index that distinguishes the solr searcher and the es searcher;

Server processing module 2, the server confirms the searcher type according to the index, if it is an es searcher, switches to the es index logic, creates an es connection according to the index, and imports the index into the es searcher;

The query module 3 adds query logic to the execution command of the hbase-indexer, and the query logic is used to perform a joint query according to the parameters and the index when the client runs the execution command, and returns the query result.

Corresponding to the above method, an embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program When implementing the steps of the above method.

Corresponding to the above method, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores machine-executable instructions, and when the computer-executable instructions are called and run by a processor, the The computer-executable instructions cause the processor to perform the steps of the above-described method.

The apparatus provided by the embodiment of the present invention may be specific hardware on the device or software or firmware installed on the device, or the like. The implementation principle and technical effects of the device provided by the embodiment of the present invention are the same as those of the foregoing method embodiment. For brief description, for the parts not mentioned in the device embodiment, reference may be made to the corresponding content in the foregoing method embodiment. Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the above method embodiments, which will not be repeated here.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may also be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality and possible implementations of apparatuses, methods and computer program products according to various embodiments of the present invention. operate. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

For another example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components may be combined or integrated into another system, or some features may be ignored. , or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in the embodiments provided by the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM for short), Random Access Memory (RAM for short), magnetic disk or CD, etc. that can store program codes medium.

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present invention, and are used to illustrate the technical solutions of the present invention, but not to limit them. The protection scope of the present invention is not limited thereto, although referring to the foregoing The embodiment has been described in detail the present invention, and those of ordinary skill in the art should understand: any person skilled in the art who is familiar with the technical field of the present invention can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present invention. Or changes can be easily conceived, or equivalent replacements are made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention. All should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (10)

1. a query method based on hbase-indexer, is characterized in that, is applied in the service system, comprises the following steps: The client configures parameters that are compatible with the es searcher passed in by the client command, and sets the index that distinguishes the solr searcher and the es searcher; The server confirms the searcher type according to the index, and if it is an es searcher, it switches to the es indexing logic; Create an es connection based on the index, and import the index into the es searcher; Query logic is added to the execution command of hbase-indexer, and the query logic is used to perform a joint query according to parameters and indexes when the client runs the joint query command, and returns the query result. 2. the query method based on hbase-indexer according to claim 1 is characterized in that, the parameter that is used for the compatible es searcher that the client command is passed in is configured at the client, and is set to distinguish between the solr searcher and the es searcher After the indexing steps, also include: The index verification process and the table permission verification process are added to the client. 3. the query method based on hbase-indexer according to claim 2, is characterized in that, described creating es connection according to index, before the step that index is imported into es searcher, also comprises: Use the security parameter in the index parameter to verify permission; If so, pass the verification; If not, an exception message is generated. 4. the query method based on hbase-indexer according to claim 1, is characterized in that, create es connection according to index, after the step that index is imported into es searcher, also comprises: Add es-compatible parameters to the execution command of batch import history index. 5. the query method based on hbase-indexer according to claim 1, is characterized in that, also comprises: Based on ambari, add the required configuration for service control scripts and custom components, so that the client can install hbase-indexer on the page, and start, stop, restart, delete, and automatically modify parameters. 6. the query method based on hbase-indexer according to claim 1, is characterized in that, according to parameter and index, carry out joint query, and the step of returning query result, specifically comprises: receive query commands; Get hbase-indexer index configuration; Perform security authentication according to the index configuration and hbase table permissions; If the authentication is passed, the corresponding query is performed according to the index type; Get query results; Get the row key of hbase according to the query result; Determine the type of the result to be returned. If the result needs to be returned, use the hbase connection, and search for the result according to the row key and return to print. If only the row key needs to be returned, return the row key and print. 7. The query method based on hbase-indexer according to claim 6, characterized in that, before the step of carrying out corresponding query according to the index type, further comprising: Determine whether the parameters of the query command are index columns and the query format is correct; If so, convert the query command parameters into the query statement in the corresponding searcher. 8. a query device based on hbase-indexer, is characterized in that, comprises: Client management module, the client configures the parameters that are compatible with the es searcher passed in by the client command, and sets the index that distinguishes the solr searcher and the es searcher; The server processing module, the server confirms the searcher type according to the index, if it is an es searcher, switches to the es index logic, creates an es connection according to the index, and imports the index into the es searcher; In the query module, query logic is added to the execution command of hbase-indexer, and the query logic is used for joint query according to parameters and indexes when the client runs the execution command, and returns the query result. 9. An electronic device, comprising a memory and a processor, wherein a computer program that can be run on the processor is stored in the memory, wherein the processor implements claim 1 when executing the computer program The steps of any one of to 7. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the computer-executable instructions to The processor executes the method of any one of claims 1 to 7.

Images