CN117874065A - Data acquisition method and device based on service database - Google Patents

Abstract

The present invention discloses a data acquisition method and device based on a business database. The method comprises: performing slot initialization and question initialization based on the business database to obtain a slot dictionary and a standard question list; performing entity search on the obtained query sentence according to the slot dictionary to obtain a preprocessed query sentence; determining the slot field and the corresponding slot in the preprocessed query sentence according to the slot dictionary; performing normalization processing on the slot field in the preprocessed query sentence according to the slot of the preprocessed query sentence to obtain a normalized query sentence; performing keyword recall and semantic recall on the normalized query sentence according to the standard question list, and obtaining an optimal intent according to the results of the keyword recall and the results of the semantic recall; obtaining a corresponding configuration SQL template based on the optimal intent; generating a query statement according to the configuration SQL template and the slot field of the preprocessed query sentence, executing the query statement, and obtaining query data.

Description

A data acquisition method and device based on business database

Technical Field

The invention relates to a data acquisition method and device based on a business database.

Background technique

With the rapid development of information technology, enterprises and organizations are accumulating more and more data, and databases contain a huge amount of important information. In order to obtain useful insights and answers from this data, business database question answering systems have emerged. These systems allow users to ask questions in natural language instead of directly writing complex SQL query statements, so they play a vital role in business decision-making, data analysis, and report generation.

Traditional database question-answering systems provide convenience in data retrieval and, based on precise keyword matching, in processing structured data.

Summary of the invention

In order to obtain more accurate query data based on natural language, an embodiment of the present invention provides a data acquisition method and device based on a business database.

In a first aspect, an embodiment of the present invention provides a data acquisition method based on a business database, the method comprising:

Slot initialization and question initialization are performed based on the business database to obtain a slot dictionary and a standard question list; the slot dictionary includes multiple slots and multiple slot fields corresponding to each slot;

Performing entity search on the acquired query sentence according to the slot dictionary to obtain a preprocessed query sentence;

Determine the slot field and the corresponding slot in the preprocessed query sentence according to the slot dictionary;

Normalizing the slot fields in the preprocessed query sentence according to the slots of the preprocessed query sentence to obtain a normalized query sentence;

According to the standard question list, the standardized query sentence is subjected to keyword recall and semantic recall, and the optimal intent is obtained according to the results of the keyword recall and the results of the semantic recall;

Acquire a corresponding configuration SQL template based on the optimal intent;

A query statement is generated according to the configured SQL template and the slot field of the pre-processed query statement, and the query statement is executed in the business database to obtain query data.

In one or some optional implementations of the embodiment of the present application, the slot initialization and question initialization based on the business database to obtain the slot dictionary and the standard question list include:

Obtaining SQL class slots and enumeration class slots in the business database;

Obtain all fields corresponding to each SQL class slot from the business database to form an SQL class slot dictionary;

Obtain all fields corresponding to each enumeration slot from the business database to form an enumeration slot dictionary;

Combining the SQL class slot dictionary with the enumeration slot dictionary to obtain a slot dictionary;

Question initialization is performed based on the business database to obtain a standard question list.

In one or some optional implementations of the embodiment of the present application, obtaining a list of standard questions based on the business database includes:

Obtaining a list of standard questions based on the business database;

Changing the SQL-type slots in the standard questions to fields that are not related to the business database;

The enumeration slot in the standard question is changed to any one of all values corresponding to the enumeration slot to obtain a standard question list.

In one or some optional implementations of the embodiments of the present application, the step of normalizing the slot field in the preprocessed query sentence according to the slot of the preprocessed query sentence to obtain a normalized query sentence includes:

The slot fields of the preprocessed query sentence belonging to the SQL slot dictionary are replaced with corresponding slots to obtain a replaced query sentence;

The slots in the replaced query sentence are changed to fields that are irrelevant to the business database.

In one or some optional implementations of the embodiment of the present application, performing pinyin correction and entity search on the acquired query sentence according to the slot dictionary to obtain the preprocessed query sentence includes:

Performing pinyin correction on the query sentence based on the slot dictionary to obtain a corrected query sentence;

Segmenting the corrected query sentence according to stop words to obtain multiple clauses;

Segment each clause to get a word list and generate an N-tuple list;

Unify the upper and lower case specifications of numbers and English letters in the N-tuple list;

According to the slot dictionary and the N-tuple list, a plurality of candidate fields and corresponding similarity scores of each N-tuple are obtained;

Taking the candidate field with the highest similarity score as the optimal candidate field of the corresponding N-tuple, and obtaining the optimal candidate field of each item in the N-tuple list;

Removing overlapping optimal candidate fields from the N-tuple list to obtain a final result of the clause;

The clauses in the error-corrected query sentence are replaced with the corresponding final results to obtain the preprocessed query sentence.

In one or some optional implementations of the embodiments of the present application, the standardized query sentence is subjected to keyword recall and semantic recall according to the standardized question list, and the optimal intent is obtained according to the results of the keyword recall and the results of the semantic recall, including:

Performing keyword search on the standardized query sentence in the standardized question list, and obtaining a plurality of retrieved query sentences arranged in descending order of similarity scores as the keyword recall result;

Extracting the canonical question list into a canonical question semantic list;

Performing semantic retrieval on the standardized query in the standardized question semantic list, obtaining multiple query retrieved sentences arranged in descending order of similarity scores as the result of the semantic recall; the result of the keyword recall and the result of the semantic recall respectively include multiple query sentences and corresponding similarity scores;

Calculating an average of similarity scores of all identical query sentences in the keyword recall result and the semantic recall result, and using the average as the similarity score of the corresponding query sentence;

Select the query with the highest similarity score as the candidate intent;

Calculate the edit distance between the candidate intent and the normalized query sentence, and determine whether the edit distance is less than a preset threshold:

If so, taking the candidate intention as the optimal intention;

If not, an error message indicating that a match could not be made is returned.

In one or some optional implementations of the embodiment of the present application, before returning an error message indicating that the match cannot be made, the following step is further included:

Obtaining a previous query sentence of the query sentence;

Determining a slot field and a corresponding slot in the previous query sentence according to the slot dictionary;

Determine whether all slots in the preprocessed query sentence exist in the slots of the previous query sentence:

If yes, the slot field of the normalized query sentence is replaced correspondingly into the previous query sentence, and the previous query sentence is used as the updated pre-processed query sentence;

If not, information indicating that the query sentence is not a follow-up question is returned.

In one or some optional implementations of the embodiments of the present application, after performing keyword recall and semantic recall on the standardized query sentence according to the standardized question list, and merging the results of the keyword recall and the results of the semantic recall to obtain the optimal intent, the method further includes:

Determine whether the slots of the preprocessed query sentence include all the slots required by the optimal intent:

If not, determine the missing slots according to the slots of the preprocessed query and all the slots required by the optimal intent, and return a request to supplement the missing slots;

The value of the corresponding missing slot is received to obtain the slot field of the optimized preprocessed query sentence.

In a second aspect, an embodiment of the present invention provides a data acquisition device based on a business database, the device comprising:

A configuration initialization module is configured to perform slot initialization and question initialization based on a business database to obtain a slot dictionary and a standard question list; the slot dictionary includes a plurality of slots and a plurality of slot fields corresponding to each slot;

A text preprocessing module, used to perform entity search on the acquired query sentence according to the slot dictionary to obtain a preprocessed query sentence;

An intention recognition module is used to determine the slot field and the corresponding slot in the preprocessed query sentence according to the slot dictionary; normalize the slot field in the preprocessed query sentence according to the slot of the preprocessed query sentence to obtain a normalized query sentence; perform keyword recall and semantic recall on the normalized query sentence according to the normalized question list, and obtain the optimal intention according to the results of the keyword recall and the results of the semantic recall;

The background query module is used to obtain the corresponding configuration SQL template based on the optimal intent; generate a query statement according to the configuration SQL template and the slot field of the preprocessed query question, and execute the query statement in the business database to obtain query data.

In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data acquisition method based on the business database as described above.

In a fourth aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data acquisition method based on the business database as described above is implemented.

The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

The data acquisition method based on the business database provided by the embodiment of the present invention, compared with the traditional database question-and-answer system that uses code to write the correspondence between intent and SQL statements in advance, this method converts natural language into intent and slot data, and then obtains SQL statements according to the configuration SQL template corresponding to the slot data intent. It only needs to define the corresponding configuration SQL template in advance. If new question-and-answer requirements arise, it only needs to add new intents and corresponding configuration SQL templates, and there is no need to repeatedly write codes. Compared with the NL2SQL technology that directly converts natural language into SQL language, it can effectively maintain high accuracy, explainability and performance, and provides technical personnel with an effective way to solve changing needs, reducing development costs and complexity, so that the business database question-and-answer system can better adapt to various business scenarios.

Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

FIG1 is a schematic diagram of steps for acquiring data based on a business database provided by an embodiment of the present invention;

FIG2 is a schematic diagram of the steps of slot initialization provided by an embodiment of the present invention;

FIG3 is a schematic diagram of the steps of question initialization provided by an embodiment of the present invention;

FIG4 is a schematic diagram of the steps of pinyin error correction provided by an embodiment of the present invention;

FIG5 is a schematic diagram of the steps of entity search provided by an embodiment of the present invention;

FIG6 is a schematic diagram of a process for obtaining an optimal intent according to a query sentence provided by an embodiment of the present invention;

FIG. 7 is a schematic diagram of the structure of a data acquisition device based on a business database provided in an embodiment of the present application.

Detailed ways

In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and/or combinations thereof.

It should also be understood that the term “and/or” used in the specification and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

The inventors found that in the prior art, although the traditional database question-and-answer system provides convenience in data retrieval, it still has obvious limitations. The system requires that the questions or intentions that users may ask must be clearly defined in the design phase before use, and these intentions must be mapped to specific SQL query templates. For each intention, code must be written, the input and output of each template must be defined, and the SQL statement must be completed using a slot-filling method to query the database data according to the preset intention. However, this type of system requires customized development and code modification to meet new requirements when dealing with ever-changing question-and-answer needs. The development cost and complexity are high, and the flexibility is insufficient.

Based on this, the inventors have made the present invention after further research and development, providing a data acquisition method and device based on a business database.

Embodiment 1

An embodiment of the present invention provides a data acquisition method based on a business database. As shown in FIG1 , the method includes:

S101: Perform slot initialization and question initialization based on the business database to obtain a slot dictionary and a standard question list; the slot dictionary includes multiple slots and multiple slot fields corresponding to each slot.

In an embodiment of the present application, in the above step S101, slot initialization and question initialization are performed based on the business database to obtain a slot dictionary and a standard question list, which mainly includes two parts: slot initialization and question initialization, which are used to obtain a slot dictionary and a standard question list respectively.

In the embodiment of the present application, the slots involved may include SQL slots and enumeration slots. The slot initialization step in the above step S101 includes: obtaining fields from the business database, dividing the obtained fields into SQL slots and enumeration slots; obtaining all fields corresponding to each SQL slot from the business database to form an SQL slot dictionary; obtaining all fields corresponding to each enumeration slot from the business database to form an enumeration slot dictionary; combining the SQL slot dictionary with the enumeration slot dictionary to obtain a slot dictionary.

In a specific embodiment, the slot initialization step is shown in Figure 2, which may specifically include: obtaining fields from the business database, forming an SQL-type slot dictionary and an enumeration-type slot dictionary based on the obtained fields, which correspond to the SQL-type slots and enumeration-type slots in Figure 2 respectively. After obtaining the slot dictionary, all fields in the slot dictionary need to be pre-processed with numbers and English uppercase and lowercase letters, and imported into the word segmentation system (such as jieba) and Elasticsearch to facilitate subsequent entity search steps. It is also necessary to convert all fields in the slot dictionary into pinyin to form a pinyin dictionary to facilitate the subsequent pinyin error correction steps.

In an embodiment of the present application, the question initialization step includes: obtaining a list of standard questions based on a business database; removing stop words and punctuation for each standard question in the list of standard questions; changing the SQL class slots in the standard questions to fields that are not related to the business database; changing the enumeration slots in the standard questions to any one of all values corresponding to the enumeration slots to obtain a list of standardized questions.

In a specific embodiment, the question initialization step is shown in Figure 3, which may specifically include: obtaining all question-related fields from the business database, manually processing them into a standard question list, i.e., the intent question library in Figure 3, removing stop words and punctuation therein, and normalizing the questions, that is, changing the SQL class slots in the standard questions to fields that are not related to the business database, and changing the enumeration slots in the standard questions to any one of all values corresponding to the enumeration slots to obtain a standardized question list, and then storing the standardized question list in two different storage engines, one stored in Elasticsearch for subsequent keyword recall, and the other converted into a semantic vector through a semantic model and stored in a vector retrieval engine (e.g., Milvus) for subsequent semantic recall.

In a specific embodiment, in order to enhance configurability, the SQL class slot of the field is obtained from the database by executing SQL statements, and the enumeration class slot is obtained through the JSON dictionary in the configuration. A specific example of executing SQL statements to obtain SQL class slots is:

Logo: station

Name: Power Station

Ranges:

Query library: A

Query statement: select distinct station_id,station_name from station

By executing the query statement of slot "power station" on business database A, all fields corresponding to the slot are obtained and exported to form a dictionary file named station_dict.txt. The dictionary format refers to the jieba custom dictionary format. The format of each line of data is: "word frequency part of speech", such as: "01 power station 1000station".

A specific example of executing SQL statements to obtain enumeration class slots is as follows:

Identifier: alerttype

Name: Alarm Type

Ranges:

Dictionary: {"warning": 1, "interrupt": 2, "alarm": 3}

The key in the enumeration slot dictionary is the slot field corresponding to the slot, and the following numeric value represents the parameter value corresponding to the Chinese name in the database table.

A specific example of getting the enumeration slot through the JSON dictionary in the configuration is:

Standard question: Does <plant> have <alarm type>?

Return answer:

Query library: A

Query statement: SELECT ad.rule_id FROM alert_data ad INNER JOIN(SELECTSUBSTRING_INDEX(group_concat(id ORDER BY collected_time DESC),',',1)mdid FROMalert_data WHERE alert_status!=1AND deleted_flag=1GROUP BY station_id,object_code,rule_name,alert_type)md ON ad.id=md.mdid AND ad.deleted_flag=1AND ad.continued_time IS NULL AND ad.alert_type IN(<alarm type>)ANDad.station_id IN(<power station>)

In the standard question, "<>" indicates the slot, which is the variable parameter in the question. <power station> is the power station slot in the slot dictionary, and all fields corresponding to any slot can be taken. The same is true for <alarm type>. The above example question indicates whether a specific power station has an alarm type, for example, whether power station 02 has an alarm? Or whether power station 03 has an early warning?

In an embodiment of the present application, in order to improve the accuracy of intent recognition, the step of changing the SQL-type slot in the standard question to a field that is not related to the business database can be executed. For example, since the two questions "Does <power station> have an alarm" and "Does the power station have an alarm" are different, the first sentence "<power station>" represents a variable parameter, that is, a specific power station under the slot is taken, and the second sentence "power station" is an immutable parameter, which represents the concept of power station, that is, all power stations. In order to improve the accuracy of intent recognition, it is necessary to perform a conversion to distinguish between variable parameters and immutable parameters, and convert variable parameters into other field words that do not overlap or conflict with the current business database, and immutable parameters are not converted. An example of a specific conversion is: "Does <power station> have an alarm" is normalized to "Does <apple> have an alarm", and "Does the power station have an alarm" remains unchanged as "Does the power station have an alarm", where "apple" is a field that is not related to the business database.

In the embodiment of the present application, in order to improve the accuracy of intent recognition, the enumeration slot in the standard question can be changed to any one of all the values corresponding to the enumeration slot. For example, "Does <power station> have <alarm type>" is normalized to "Does <power station> have an alarm".

In the embodiment of the present application, step S101 allows slots and questions to be flexibly defined by configuring parameters. This configurability enables the method to be flexibly configured in different scenarios and business requirements without modifying the code, and to adapt to different database structures, question sets and business rules.

S102: performing pinyin correction and entity search on the acquired query sentence according to the slot dictionary to obtain a pre-processed query sentence.

In an embodiment of the present application, in the above step S102, pinyin correction and entity search are performed on the obtained query sentence according to the slot dictionary to obtain a preprocessed query sentence, which includes two steps: pinyin correction and entity search. The corrected query sentence is obtained by pinyin correction, and the preprocessed query sentence is obtained according to the entity search.

The steps of pinyin correction in the above step S102 are shown in FIG4 , and may specifically include: a first step of segmenting the query sentence; a second step of concatenating the segmentation results to generate an n-gram string, wherein n is a preset maximum string length; a third step of performing pinyin correction on the string, converting all string into corresponding pinyin, and then querying a pinyin dictionary. If there is a single result, the query is directly corrected. If there are multiple results, the edit distance between each result and the string is calculated, and the result with the smallest edit distance is taken as the correction result, and the correction result is replaced with the original string; the steps from the first step to the third step are repeated, and the length of the string generated each time is reduced by one compared with the previous one, until the query sentence is corrected.

The entity search step in the above step S102 is shown in FIG5 , and specifically may include: segmenting the corrected query sentence according to stop words to obtain multiple clauses, wherein the stop words are words that are not very valuable in text analysis, such as “有”, “和”, “是”, etc.; segmenting each clause to obtain a word list, and concatenating them front and back to form a word string from 1-gram to n-gram, that is, an N-tuple list; preprocessing the N-tuple list to perform number and English case conversion; storing the slot dictionary and the N-tuple list in Elasticsearch, and using Elasticsearch to search to obtain multiple candidate fields and corresponding similarity scores for each N-tuple; using the candidate field with the highest similarity score as the optimal candidate field corresponding to the N-tuple, and obtaining the optimal candidate field for each item in the N-tuple list; mapping and connecting the optimal candidate field to the corresponding original field in the query sentence to facilitate subsequent processing; removing the overlapping optimal candidate fields in the N-tuple list to obtain the final result of the clause; replacing the clause in the corrected query sentence with the corresponding final result, and restoring it to obtain the preprocessed query sentence, that is, the sentence after the standardized entity in FIG5 .

In a specific embodiment, a specific example of the entity search process is:

Query sentence after error correction: Is there any alarm at the 10Kv smart substation B?

Division clause: 10Kv smart substation b, alarm

Segment the first clause: ['十', 'Kv', '智慧', '变電', 'b', '站']

N-tuple list: ['Smart Substation B', 'Kv Smart Substation', 'Ten Kv Smart Substation', 'Smart Substation', 'Ten Kv Wisdom', 'Kv Smart Substation B', 'Smart Substation B', 'Ten Kv Smart Substation B', 'Kv Wisdom', 'Ten Kv Smart Substation B', 'Kv Smart Substation B']

Preprocessing: ['smart substation b', 'kv smart substation b', 'kv smart substation', 'kv wisdom', '10kv smart substation b station', 'smart substation', 'smart substation b station', 'kv smart substation b station', '10kv smart substation b', '10kv wisdom', '10kv smart substation']

Candidate fields: {'10kv smart substation b@@10Kv smart substation b': {'smart 10kv substation b station': 21.14124}, 'smart substation b@@smart substation b': {'smart 10kv substation b station': 17.298553, 'smart b substation': 16.139145},,...}

Among them, the field ‘10kv smart substation b’ before the symbol “@@” is the preprocessed field sent to Elasticsearch for search, the field ‘10Kv smart substation b’ after the symbol “@@” is the original field before the preprocessing, ‘Smart 10kv substation b’ is the candidate field, and ‘21.14124’ is the similarity score of the candidate field.

Optimal candidate fields: {'Smart 10kv substation b': {'score': 23.621014, 'gram': '10kv smart substation b@@10Kv smart substation b'}, 'Smart b substation': {'score': 19.018782, 'gram': 'Smart substation b@@Smart substation b'}, ...}

Among them, 'Smart 10kv Substation b' is the optimal candidate field, 'score': 23.621014 represents the similarity score, 'gram': '10kv Smart Substation b@@十Kv Smart Substation b' represents the corresponding word segmentation, the field '10kv Smart Substation b' before the symbol "@@" is the preprocessed field sent to Elasticsearch for search, and the field '十Kv Smart Substation b' after the symbol "@@" is the original field before preprocessing.

Field restoration: {'ori':'10KV Smart Substation B', 'candidate':'Smart 10kV Substation B', 'score':23.621014}, {'ori':'Smart Substation B', 'candidate':'Smart B Substation', 'score':19.018782}, ...}

Among them, 'ori': '10KV smart substation B' represents the original field before preprocessing, 'candidate': 'Smart 10kV Substation B' is the optimal candidate field, and 'score': 23.621014 is the similarity score of the optimal candidate field.

Overlapping item processing: {'ori':'10KV smart substation B','candidate':'Smart 10kV substation B','score':23.621014}

Sentence restoration: Is there any alarm at the smart 10kV substation B?

After error correction, the query sentence "Is there any alarm at the 10KV smart substation B" can complete the entity search through the above steps and be converted into the standard "Is there any alarm at the smart 10kV substation B"

In the embodiment of the present application, the step of executing entity search is performed by uniformly converting numbers and English uppercase and lowercase letters of the query sentence after error correction, removing stop words and punctuation marks, ensuring that the input text is clean and standardized, solving problems such as missing words, extra words, and reversed order in the question sentence, and obtaining the preprocessed query sentence.

S103: Determine a slot field and a corresponding slot in the preprocessed query sentence according to the slot dictionary.

In the embodiment of the present application, in the above step S103, the slot dictionary is imported into the jieba word segmentation system, the jieba word segmentation system is used to process the preprocessed query sentence, the slot field in the preprocessed query sentence is obtained, and the slot corresponding to the slot field is determined.

S104: normalizing the slot fields in the preprocessed query sentence according to the slots of the preprocessed query sentence to obtain a normalized query sentence.

In the embodiment of the present application, in the above step S104, the slot field in the determined pre-processed query sentence is normalized, including: replacing the slot field of the pre-processed query sentence belonging to the SQL slot dictionary with the corresponding slot to obtain the replaced query sentence, and then changing the slot in the replaced query sentence to a field unrelated to the business database. Similar to the question initialization operation in step S101, the variable parameters are converted into words in other fields to obtain a normalized query sentence, so as to better perform intent recognition later.

In a specific embodiment, the jieba word segmentation system is used to process the pre-processed query sentence "Is there an alarm at the smart 10kV substation A?", and the slot field "Smart 10kV substation A" and the corresponding slot "<power station>" are obtained. After normalization, "Is there an alarm at <Apple>?" is obtained.

S105: According to the standard question list, the standardized query sentence is subjected to keyword recall and semantic recall, and the optimal intent is obtained according to the results of the keyword recall and the semantic recall.

In an embodiment of the present application, the above step S105 can realize the intent recognition of the standardized query sentence and obtain the optimal intent. The process of intent recognition may include two parts: keyword recall and semantic recall. Specifically, when performing keyword recall, a keyword search is performed on the standardized query sentence in the standard question list, that is, keyword recall is performed based on the standard question list in Elasticsearch, and a similarity score threshold is set to obtain multiple retrieved query sentences with similarity scores greater than the similarity score threshold, arranged in order from large to small, and the corresponding similarity scores as the result of keyword recall. When performing semantic recall, the standard question list is extracted as a standard question semantic list, that is, the semantic vector corresponding to the standard question list stored in the vector retrieval engine is converted into a continuous semantic vector through query embedding (QueryEmbedding) to obtain a standard question semantic list, which is stored in the vector retrieval engine; semantic retrieval is performed on the standardized query sentences in the standard question semantic list, that is, semantic recall is performed based on the standard question semantic list in the vector retrieval engine such as Milvus, and a similarity score threshold is also set to obtain multiple retrieved query sentences with similarity scores greater than the similarity score threshold, arranged in order from large to small, and the corresponding similarity scores as the result of semantic recall. Afterwards, the first n questions in the results of keyword recall are taken as the first candidate set, and the first m questions in the results of semantic recall are taken as the second candidate set. The first candidate set and the second candidate set are normalized respectively to ensure that the sum of the similarity scores of the n questions in the first candidate set and the m questions in the second candidate set is equal to 1; the average similarity score of the same query sentence in the first candidate set and the second candidate set is calculated according to the displacement identifier of the question (such as question ID), and the average value is used as the similarity score of the corresponding query sentence; the query sentence with the highest similarity score is selected as the candidate intent; the edit distance between the candidate intent and the standardized query sentence is calculated to determine whether the edit distance is less than a preset threshold: if so, the candidate intent is taken as the optimal intent; if not, an error message indicating that the match cannot be made is returned, that is, the query sentence is considered to be different from all the question intents in the standardized question list.

In the embodiment of the present application, the framework diagram of the above steps S102 to S105 is shown in Figure 6. First, step S102 performs pinyin error correction and entity search according to the query sentence to obtain the preprocessed query sentence, corresponding to the user sentence Query in Figure 6 after sentence preprocessing operations, including pinyin error correction and entity search. Then steps S103 and S104 are performed to determine the slot field and the corresponding slot in the preprocessed query sentence and perform normalization processing to obtain a normalized query sentence, corresponding to the element extraction and normalization conversion in Figure 6. Finally, step S105 is executed to obtain the optimal intent through keyword recall and semantic recall according to the normalized query sentence, corresponding to the keyword recall and semantic recall in Figure 6. Among them, keyword recall is ES (Elasticsearch) recall based on the ES text knowledge base and the intent question library, and the recall algorithm is BM25Search. The intent question library in the figure is the standard question list mentioned above. Semantic recall includes converting the standardized query into a semantic vector, and then performing Milvus recall by embedding the query to obtain a continuous semantic vector. The recall algorithm is Dense Vector Search. Next, the n questions with the highest similarity scores in the keyword recall results and the m questions with the highest similarity scores in the semantic recall results are taken and merged to obtain the TopK intent candidate set. After summing and averaging, the query with the highest similarity score is selected as the candidate intent. The candidate intent is the optimal intent obtained in Figure 6. Finally, the edit distance is judged. If the edit distance between the candidate intent and the standardized query is less than the preset threshold, the candidate intent is the optimal intent. If the edit distance between the candidate intent and the standardized query is not less than the preset threshold, the query is considered to be different from all the question intents in the standardized question list, and an error message indicating that the match cannot be made is returned.

In a specific embodiment, the steps of performing keyword recall and semantic recall on the standardized query sentence to obtain the optimal intent are as follows:

Normalization conversion: Does Apple have any warnings?

Keyword recall:

Ids: [16417929980417,16418097314507,16418046495959374,...]

Questions: ['Does <Apple> have any alerts', 'What alerts does <Apple> have', 'Which <Apple> devices have alerts', ...]

Scores: [0.2148, 0.2036, 0.1935, ...]

Semantic Recall:

Ids: [16417929980417, 16417929980459, 16417929985741, ...]

Questions: ['Does <Apple> have an alarm?', 'Does <Apple> have an alarm?', 'Does <Apple> have an alarm?', ...]

Scores: [0.2493, 0.2344, 0.196, ...]

Merge sort: {id:'16417929980417, question:'Does <Apple> have an alert?', 'score': 0.2701}

Best intention acquisition: Does Apple have any warnings?

Among them, Ids represents the question ID, Questions and Scores represent the questions in the corresponding recalled results and the corresponding similarity scores.

In an embodiment of the present application, if the edit distance between the candidate intent and the normalized query sentence is not less than a preset threshold, context processing can be performed to achieve continuous question and answer before returning an error message indicating that the match cannot be made. The context processing process specifically includes: obtaining the previous query sentence of the query sentence; determining the slot field and the corresponding slot in the previous query sentence according to the slot dictionary; determining whether the slots in the preprocessed query sentence all exist in the slots of the previous query sentence: if so, replacing the slot field of the normalized query sentence with the corresponding one in the previous query sentence, and using the previous query sentence as the updated preprocessed query sentence; if not, returning information indicating that the query sentence is not a follow-up question. Take the following T-1 time sentence and T time sentence for context processing as an example:

Question at T-1: Is there any alarm at the smart 10kV substation B?

Question at T time: Where is the smart 10kV substation A?

Updated pre-processed query question: Is there any alarm at the smart 10kV substation A?

Through the above context processing method, an updated pre-processed query sentence of the current question at time T is obtained to achieve continuous question answering.

In the embodiment of the present application, after obtaining the optimal intent, it is also possible to check whether the query sentence contains all the parameters required in the optimal intent by means of parameter verification and counter-questioning. The specific implementation process includes: judging whether the slots of the query sentence after preprocessing include all the slots required for the optimal intent: if not, determining the missing slots according to the slots of the query sentence after preprocessing and all the slots required for the optimal intent, and returning a request to supplement the missing slots; receiving the value of the corresponding missing slot, and obtaining the slot field of the optimized query sentence after preprocessing. For example, for the standardized query sentence "Does <Apple> have an alarm?", its intention is "Does <Power Station> have <Alarm Type>?", it is necessary to provide two necessary parameters, <Power Station> and <Alarm Type>, and the <Power Station> parameter can be obtained by entity recognition through the jieba word segmentation system. The <Alarm Type> parameter is matched by keywords, that is, the slot field in the enumeration slot <Alarm Type> is used to match the query sentence. If it is determined that the slots of the preprocessed query sentence do not include all the slots required for the optimal intent, a reverse question will be triggered, and a request to supplement the missing slot information will be returned. For example, when the query sentence is "Is there an alarm?", this method will recognize that the matching intent is "Does <power station> have <alarm type>?", detect that the '<power station>' parameter is missing, trigger the reverse question module, and ask "Which power station are you asking about?". Subsequently, it will wait for the next interaction of the user, receive the value of the corresponding missing slot, obtain the slot field of the optimized preprocessed query sentence, complete the supplement of the missing parameters, and successfully complete the question-answering process.

S106: Obtain a corresponding configuration SQL template based on the optimal intent.

S107: Generate a query statement according to the configured SQL template and the slot field of the pre-processed query statement, and execute the query statement in the business database to obtain query data.

In the embodiment of the present application, in the above steps S106 and S107, the corresponding configuration SQL template is obtained based on the optimal intent, the slot field in the preprocessed query is converted into a general parameter format, and the parameters are filled into the corresponding configuration SQL template to obtain a query statement. Then, the query statement is executed to retrieve the corresponding data, and the retrieved data is finally returned.

In a specific embodiment, the general parameter format is divided into two parts: "name" and "code", where "name" is used to record the slot field in the query sentence, and "code" records the parameter corresponding to the slot field. Example:

"args":{"name":{"station":["Smart 10kV Substation A"],"alerttype":["Alarm"]},

"code":{"station":["bdz001"],"alerttype":[3]}}

The corresponding generated query statement:

SELECT ad.rule_id FROM alert_data ad INNER JOIN(SELECT SUBSTRING_INDEX(group_concat(id ORDER BY collected_time DESC),',',1)mdid FROM alert_data WHERE alert_status!=1AND deleted_flag=1 GROUP BY station_id,object_code,rule_name,alert_type)md ON ad.id=md.mdid AND ad.deleted_flag=1ANDad.continued_time IS NULL AND ad.alert_type IN(3)AND ad.station_id IN("bdz001")

Compared with the traditional database question-and-answer system that uses code to write the correspondence between intent and SQL statements in advance, the data acquisition method based on the business database provided by the embodiment of the present invention converts natural language into intent and slot data, can accurately obtain the optimal intent corresponding to the query, and then obtain the SQL statement according to the configuration SQL template corresponding to the optimal intent. This method only needs to define the corresponding configuration SQL template in advance. If new question-and-answer requirements arise, only new intents and corresponding configuration SQL templates need to be added, and there is no need to repeatedly write code. It can effectively maintain high accuracy, explainability and performance, provide technical personnel with an effective way to solve changing needs, reduce development costs and complexity, and enable the business database question-and-answer system to better adapt to various business scenarios.

Embodiment 2

Based on the same inventive concept, an embodiment of the present invention further provides a data acquisition device based on a business database, as shown in FIG. 7 , the device includes:

A configuration initialization module 101 is configured to perform slot initialization and question initialization based on a business database, and obtain a slot dictionary and a standard question list; the slot dictionary includes a plurality of slots and a plurality of slot fields corresponding to each slot;

A text preprocessing module 102, configured to perform entity search on the acquired query sentence according to the slot dictionary to obtain a preprocessed query sentence;

The intention recognition module 103 is used to determine the slot field and the corresponding slot in the preprocessed query sentence according to the slot dictionary; normalize the slot field in the preprocessed query sentence according to the slot of the preprocessed query sentence to obtain a normalized query sentence; perform keyword recall and semantic recall on the normalized query sentence according to the normalized question list, and obtain the optimal intention according to the results of the keyword recall and the results of the semantic recall;

The background query module 104 is used to generate a query statement based on the optimal intent and the slot field of the pre-processed query sentence, and execute the query statement in the business database to obtain query data.

Embodiment 3

Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data acquisition method based on the business database as described in the above embodiment 1 is implemented.

Embodiment 4

Based on the same inventive concept, an embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data acquisition method based on the business database as described in the above-mentioned embodiment 1 is implemented.

Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

The present invention is described with reference to the flowchart and/or block diagram of the method, device (system), and computer program product according to the embodiment of the present invention. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the process and/or box in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (10)

1. A method for acquiring data based on a service database, comprising:

carrying out slot initialization and problem initialization based on a service database to obtain a slot dictionary and a specification problem list; the slot dictionary comprises a plurality of slots and a plurality of slot fields corresponding to each slot;

performing entity searching on the acquired inquiry question according to the slot dictionary to obtain a preprocessed inquiry question;

Determining a slot field and a corresponding slot in the query question after preprocessing according to the slot dictionary;

normalizing the slot fields in the preprocessed query question according to the slots of the preprocessed query question to obtain a normalized query question;

according to the standard problem list, carrying out keyword recall and semantic recall on the standardized query question sentence, and obtaining the optimal intention according to the keyword recall result and the semantic recall result;

acquiring a corresponding configuration SQL template based on the optimal intention;

generating a query sentence according to the configuration SQL template and the slot field of the preprocessed query question, and executing the query sentence in the service database to obtain query data.

2. The method of claim 1, wherein the performing slot initialization and problem initialization based on the service database to obtain a slot dictionary and a canonical problem list comprises:

acquiring SQL class slots and enumeration class slots in the service database;

acquiring all fields corresponding to each SQL class slot from the service database to form an SQL class slot dictionary;

Acquiring all fields corresponding to each enumeration slot from the service database to form an enumeration slot dictionary;

combining the SQL class slot dictionary with the enumeration slot dictionary to obtain a slot dictionary;

and initializing the problems based on the service database to obtain a standard problem list.

3. The method of claim 2, wherein the obtaining a list of canonical problems based on the business database comprises:

acquiring a standard problem list based on the service database;

changing SQL class slots in the standard problem into fields irrelevant to the service database;

and changing the enumeration slot position in the standard problem into any one of all values corresponding to the enumeration slot position to obtain a standard problem list.

4. The method of claim 2, wherein normalizing the slot fields in the preprocessed query question according to the slots of the preprocessed query question to obtain a normalized query question, comprises:

replacing the slot fields of the preprocessed query question with corresponding slots belonging to the SQL class slot dictionary to obtain a replaced query question;

and changing the slot positions in the replaced inquiry question into fields irrelevant to the service database.

5. The method of claim 1, wherein the performing keyword recall and semantic recall on the normalized query question according to the normalized question list, and obtaining the optimal intent according to the result of the keyword recall and the result of the semantic recall, comprises:

keyword retrieval is carried out on the normalized query question sentences in the standard question list, and a plurality of query question sentences with similarity scores which are sequentially arranged from large to small are obtained to serve as the recall result of the keywords;

extracting the standard problem list as a standard problem semantic list;

carrying out semantic retrieval on the normalized query question sentences in the standard question semantic list to obtain a plurality of query question sentences with similarity scores which are sequentially arranged from large to small as a result of the semantic recall; the keyword recall result and the semantic recall result respectively comprise a plurality of inquiry questions and corresponding similarity scores;

calculating the average value of similarity scores of all the same query questions in the keyword recall result and the semantic recall result, and taking the average value as the similarity score of the corresponding query questions;

Selecting the query question with the highest similarity score as a candidate intention;

calculating the editing distance between the candidate intention and the normalized query question, and judging whether the editing distance is smaller than a preset threshold value or not:

if yes, taking the candidate intention as an optimal intention;

if not, returning error information indicating that the matching is not possible.

6. The method of claim 5, further comprising, prior to returning the error information indicating the inability to match:

acquiring a previous query question of the query question;

determining a slot field and a corresponding slot in the previous inquiry question according to the slot dictionary;

judging whether slots in the query question after pretreatment are all in the slots of the previous query question:

if yes, correspondingly replacing the slot field of the normalized query question into a previous query question, and taking the previous query question as the updated preprocessed query question;

if not, returning information indicating that the inquiry question is not an additional inquiry.

7. The method of claim 1, wherein after keyword recall and semantic recall of the normalized query question according to the canonical question list, and merging the result of the keyword recall and the result of the semantic recall, obtaining an optimal intent, further comprising:

Judging whether the slots of the query question after preprocessing comprise all slots required by the optimal intention or not:

if not, determining missing slots according to the slots of the query question after the preprocessing and all slots required by the optimal intention, and returning a request for supplementing the missing slots;

and receiving the corresponding missing slot values to obtain the optimized slot field of the query question after preprocessing.

8. A data acquisition device based on a service database, comprising:

the configuration initialization module is used for carrying out slot initialization and problem initialization based on the service database to obtain a slot dictionary and a specification problem list; the slot dictionary comprises a plurality of slots and a plurality of slot fields corresponding to each slot;

the text preprocessing module is used for carrying out entity searching on the acquired inquiry question according to the slot dictionary to obtain a preprocessed inquiry question;

the intention recognition module is used for determining a slot field and a corresponding slot in the query question after pretreatment according to the slot dictionary; normalizing the slot fields in the preprocessed query question according to the slots of the preprocessed query question to obtain a normalized query question; according to the standard problem list, carrying out keyword recall and semantic recall on the standardized query question sentence, and obtaining the optimal intention according to the keyword recall result and the semantic recall result;

The background query module is used for acquiring a corresponding configuration SQL template based on the optimal intention; generating a query sentence according to the configuration SQL template and the slot field of the preprocessed query question, and executing the query sentence in the service database to obtain query data.

9. A computer readable storage medium having instructions stored therein which, when run on a terminal, cause the terminal to perform the service database based data acquisition method according to any one of claims 1 to 7.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing a method of service database based data acquisition as claimed in any one of claims 1 to 7 when the computer program is executed.