CN109271495B - Question-answer recognition effect detection method, device, equipment and readable storage medium - Google Patents

Abstract

Embodiments of the present disclosure provide a question and answer recognition effect detection method, device, equipment, and a readable storage medium. The question and answer recognition effect detection method includes: obtaining a hot word set including hot words according to the original data of user questions; sorting and screening the hot words in the hot word set to determine hot questions, and combining the hot words with user feedback and answers Information is associated to obtain the answer distribution associated with the hot word for the hot word, and obtain the hot word associated with the answer for the answer; detect whether the number of deduplicated answers associated with a hot word exceeds the first threshold and a Whether the number of hot words associated with the answer exceeds the second threshold; when it is detected that the number of deduplicated answers associated with a hot word exceeds the first threshold and/or the number of hot words associated with an answer exceeds the second threshold, determine In the case of abnormal matching between hot words and answers, the effect of question and answer recognition can be monitored in a centralized manner.

Description

Question answering recognition effect detection method, device, equipment and readable storage medium

technical field

The embodiments of the present disclosure relate to the field of computer technology, and in particular, to a question and answer recognition effect detection method, device, equipment and readable storage medium.

Background technique

In the construction process of the question-answer recognition system in the related art, a large amount of offline labeled data is required to check the recognition effect of the question-answer. The timeliness of outsourcing and crowdsourcing labeling for question and answer effect recognition is not good, and the cost of quality control and communication is high. The regression method using the verification set is affected by the arbitrariness and variability of user descriptions, and cannot effectively evaluate the actual user question-answering effect. Online bad case analysis (case-by-case) requires a lot of manpower for operations and takes a long time to analyze and conclude. Therefore, a large number of problems in online recognition of Q&A effects cannot be found and solved in a timely manner, which affects the Q&A user experience.

The method for discovering hot words in the question answering system in the related art monitors hot topics based on fluctuation trends of hot words and user feedback to discover potential business problems or question-answer matching problems. Operators browse, summarize and analyze hot topics after discovering them, and at the same time analyze the original question and answer logs of users under hot words one by one to determine the recognition effect. The main purpose of the solution of the related technology is to discover the problems that the users describe intensively, but there is no centralized monitoring of the recognition effect of the question and answer, so that the effect of the recognition of the question and answer of the long-tail question cannot be effectively paid attention to.

Therefore, there is an urgent need for a method that can quickly and effectively detect the effect of question and answer recognition.

Contents of the invention

In view of this, the first aspect of the present disclosure provides a method for detecting the effect of question and answer recognition, including:

Obtain a set of hot words including hot words according to the original data of user questions;

Sorting and screening the hot words in the hot word set to determine hot questions, and associating the hot words with user feedback and answer information, so as to obtain the distribution of answers associated with the hot words for the hot words, and obtain for the answers Hot words associated with the answer;

Detecting whether the number of deduplicated answers associated with a hot word exceeds a first threshold and whether the number of hot words associated with an answer exceeds a second threshold;

When it is detected that the number of deduplicated answers associated with a hot word exceeds a first threshold and/or the number of hot words associated with an answer exceeds a second threshold, it is determined that an abnormal case of matching between a hot word and an answer occurs.

The second aspect of the present disclosure provides a question and answer recognition effect detection system, including:

The hot word acquisition module is configured to acquire a hot word set including hot words according to the raw data of the user's question;

The question and answer association module is configured to sort and filter the hot words in the hot word set to determine hot questions, and associate the hot words with user feedback and answer information to obtain the hot words associated with the hot words Answer distribution, and obtain the hot words associated with the answer for the answer;

An anomaly detection module configured to detect whether the number of deduplicated answers associated with a hot word exceeds a first threshold and whether the number of hot words associated with an answer exceeds a second threshold;

An abnormality determination module configured to determine that when the abnormality detection module detects that the number of deduplicated answers associated with a hot word exceeds a first threshold and/or the number of hot words associated with an answer exceeds a second threshold, determine that An abnormal case of matching hot words and answers.

A third aspect of the present disclosure provides an electronic device, including a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to Implement the method as described in the first aspect.

A fourth aspect of the present disclosure provides a readable storage medium on which computer instructions are stored, and when the computer instructions are executed by a processor, the method described in the first aspect is implemented.

In the embodiment of the present disclosure, a hot word set including hot words is obtained according to the raw data of user questions; the hot words in the hot word set are sorted and screened to determine hot questions, and the hot words are combined with user feedback and The answer information is associated, to obtain the answer distribution associated with the hot word for the hot word, and obtain the hot word associated with the answer for the answer; detect whether the number of deduplicated answers associated with a hot word exceeds the first threshold and Whether the number of hot words associated with an answer exceeds the second threshold; when it is detected that the number of deduplicated answers associated with a hot word exceeds the first threshold and/or the number of hot words associated with an answer exceeds the second threshold, To determine the case of abnormal matching between hot words and answers, you can obtain hot word sets, and at the same time through the two-way correlation analysis of hot words and answers, combined with user feedback, to monitor the effect of question and answer recognition in a centralized manner, and can effectively pay attention to cases such as abnormal matching. The effect of question answering recognition on long-tail questions of the class.

These or other aspects of the present disclosure will be more clearly understood in the description of the following embodiments.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following will briefly introduce the accompanying drawings that need to be used in the description of the exemplary embodiments or related technologies. Obviously, the accompanying drawings in the following description These are some exemplary embodiments of the present disclosure, and those skilled in the art can also obtain other drawings based on these drawings without creative efforts.

FIG. 1 shows a flow chart of a method for detecting a question and answer recognition effect according to an embodiment of the present disclosure;

FIG. 2 shows a flow chart of a method for detecting the effect of question and answer recognition according to another embodiment of the present disclosure;

FIG. 3 shows an example of the flow chart of step S101 of the method for detecting the effect of question and answer recognition according to another embodiment of the present disclosure;

Fig. 4 shows a structural block diagram of a question and answer recognition effect detection device according to an embodiment of the present disclosure;

Fig. 5 shows a structural block diagram of a question and answer recognition effect detection device according to another embodiment of the present disclosure;

FIG. 6 shows a structural block diagram of the hot word acquisition module 401 in the question and answer recognition effect detection device according to an embodiment of the present disclosure;

Fig. 7 shows a structural block diagram of a device according to an embodiment of the present disclosure;

Fig. 8 is a schematic structural diagram of a computer system suitable for implementing the method for detecting the effect of question-and-answer recognition according to an embodiment of the present disclosure.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present disclosure, the following will clearly and completely describe the technical solutions in the exemplary embodiments of the present disclosure with reference to the accompanying drawings in the exemplary embodiments of the present disclosure.

In some processes described in the specification and claims of the present disclosure and the descriptions in the above drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that these operations may not be performed in the order in which they appear herein Execution or parallel execution, the serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. Additionally, these processes can include more or fewer operations, and these operations can be performed sequentially or in parallel. It should be noted that the descriptions of "first" and "second" in this article are used to distinguish different messages, devices, modules, etc. are different types.

The technical solutions in the exemplary embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the exemplary embodiments of the present disclosure. Obviously, the described exemplary embodiments are only part of the embodiments of the present disclosure, rather than Full examples. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present disclosure. .

Fig. 1 shows a flow chart of a method for detecting the effect of question and answer recognition according to an embodiment of the present disclosure. The method may include steps S101, S102, S103 and S104.

In step S101, a hot word set including hot words is obtained according to the raw data of the user's question.

In step S102, the hot words in the hot word set are sorted and screened to determine hot questions, and the hot words are associated with user feedback and answer information to obtain the answer distribution associated with the hot words for the hot words, and Obtain the hot words associated with the answer for the answer.

In step S103, it is detected whether the number of deduplicated answers associated with a hot word exceeds a first threshold and whether the number of hot words associated with an answer exceeds a second threshold.

In step S104, when it is detected that the number of deduplicated answers associated with a hot word exceeds the first threshold and/or the number of hot words associated with an answer exceeds the second threshold, it is determined that there is an abnormal matching between the hot word and the answer case.

In the embodiments of the present disclosure, a hot word is a vocabulary phenomenon, which reflects issues and things that people generally pay attention to in a period of time. Hot words have the characteristics of time and space, that is, they reflect the hot topics of a certain period in a certain group, and the main forms of expression are vocabulary and phrases.

In an embodiment of the present disclosure, the method of obtaining a hot word set including hot words according to the raw data of user questions may be to perform hot word mining. Based on the original data of user questions, hot word sets or candidate hot word sets can be obtained through operations such as new word discovery, phrase mining, and popularity calculation.

In one embodiment of the present disclosure, sorting and screening the hot words in the hot word set to determine hot questions includes: according to the time period distribution of hot words, the entry diversity distribution of hot words and the periodicity of hot words Analyze, sort and filter hot word results to determine hot questions. The time period distribution of the hot words may refer to the trend of the number of hot words in units of time periods such as 10 minutes.

In an embodiment of the present disclosure, associating hot words with user feedback and answer information includes: establishing an inverted index with hot words as a dimension, and associating hot words with user feedback and answer information. Therefore, after the acquisition of hot words (hot word mining) is completed, an inverted index can be established with the hot words as the dimension, and the hot words can be associated with user feedback and answer information to speed up the subsequent analysis process. In addition, the user feedback may include the user's evaluation of the question answering system and the like.

In an embodiment of the present disclosure, obtaining the distribution of answers associated with the hot word may include: analyzing the question and answer pairs associated with the hot word in units of a single hot word to obtain the answers associated with the hot word distribution of answers. It can be understood that there may be multiple pairs of questions and answers associated with a hot word, therefore, multiple pairs of questions and answers may be analyzed to obtain the distribution of answers associated with the hot word. For example, for a hot word with clear semantics, such as "credit card repayment", the corresponding answer distribution will revolve around the topic of the hot word.

In an embodiment of the present disclosure, obtaining the hot words associated with the answer includes: performing hot word clustering on user questions corresponding to the answer in units of a single answer to obtain the hot words associated with the answer. For example, the original question corresponding to a certain answer should focus on one or a small number of hot topics, and should not be related to multiple hot words exceeding a certain threshold.

In one embodiment of the present disclosure, when it is detected that the number of deduplicated answers associated with a hot word exceeds a first threshold and/or the number of hot words associated with an answer exceeds a second threshold, it is determined that a hot word and An abnormal case of answer matching refers to: no matter whether the number of deduplicated answers associated with a hot word exceeds the first threshold, or the number of hot words associated with an answer exceeds the second threshold, or the deduplication associated with a hot word In any one of the three situations where the number of repeated answers exceeds the first threshold and the number of hot words associated with an answer exceeds the second threshold, it can be determined that there is an abnormal case of matching between hot words and answers. In other words, if there are too many deduplicated answers associated with a hot word, the question associated with the hot word has an answer that is not related to the question; The corresponding original question revolves around too many topics (hot words). It can be considered that there are cases of abnormal matching between hot words and answers, or there are cases of abnormal matching between candidate hot words and answers for subsequent investigation. In the embodiments of the present disclosure, the abnormal case of hotword-answer matching can be considered as one of the bad cases. Those skilled in the art can understand that, according to actual needs, the first threshold and the second threshold can be any number. In addition, deduplication refers to the removal of duplicate answers, which can avoid statistical and computational distortion.

In the embodiment of the present disclosure, a hot word set including hot words is obtained according to the raw data of user questions; the hot words in the hot word set are sorted and screened to determine hot questions, and the hot words are combined with user feedback and answer information Carry out association, to obtain the answer distribution associated with the hot word for the hot word, and obtain the hot word associated with the answer for the answer; detect whether the number of deduplicated answers associated with a hot word exceeds the first threshold and an answer Whether the number of hot words associated exceeds the second threshold; when it is detected that the number of answers after deduplication associated with a hot word exceeds the first threshold and/or the number of hot words associated with an answer exceeds the second threshold, it is determined that there is a Abnormal cases of matching of hot words and answers can be obtained by obtaining hot word sets, and at the same time through the two-way correlation analysis of hot words and answers, combined with user feedback, the effect of question and answer recognition can be monitored in a centralized manner, and it can effectively pay attention to cases such as matching abnormal cases Question answering recognition performance for long-tail questions. The hot word mining scheme of related technologies can be completed by methods such as Apriori association rule mining and SegPhrase phrase discovery, which have high requirements for word segmentation accuracy or corpus quality. At the same time, the related technology solutions only stop at hot word mining, unlike In the solution of the embodiment of the present disclosure, the deep question answering system performs hot word evaluation and bad case discovery of question answer matching.

In embodiments of the present disclosure, long-tail questions may refer to those questions that do not appear frequently.

Fig. 2 shows a flow chart of a method for detecting the effect of question and answer recognition according to another embodiment of the present disclosure. The difference between the embodiment shown in FIG. 2 and the embodiment shown in FIG. 1 is that, besides steps S101 , S102 , S103 and S104 , FIG. 2 also includes steps S201 and S202 .

In step S201, it is detected whether the hot word is included in the deduplicated answer associated with the hot word.

In step S202, when it is detected that all the deduplicated answers associated with a hot word do not include the hot word, it is determined that a missing answer case occurs.

In one embodiment of the present disclosure, further analysis can be carried out in the case of abnormal matching between hot words and answers. If all the answer titles associated with a hot word do not contain the hot word string, it can be considered as There are missing answers in the answer library, so it can be considered that there is a missing answer case, or this case can be used as a candidate missing case for subsequent investigation. In embodiments of the present disclosure, the missing-answer case can be considered as one of the bad cases.

In the embodiment of the present disclosure, by obtaining the hot word set, at the same time, through the two-way association analysis of hot words and answers, combined with user feedback, the effect of question and answer recognition can be monitored intensively, and it can effectively pay attention to cases such as abnormal matching and missing answers The effect of question and answer recognition on long-tail questions such as .

Fig. 3 shows an example of a flow chart of step S101 of the question-answer recognition effect detection method according to another embodiment of the present disclosure. Step S101 includes steps S301, S302, S303 and S304.

In step S301, a preset new word discovery algorithm is used to calculate the original data of the user's question to obtain new words, wherein a new word is composed of characters.

In step S302, a word segmentation dictionary is generated according to the basic words in the preset basic thesaurus and the acquired new words, and the user question is segmented using the word segmentation dictionary to obtain a question segmentation result.

In step S303, the word segmentation result is calculated by a preset phrase discovery algorithm to obtain a phrase, wherein the phrase is composed of words obtained after word segmentation.

In step S304, the popularity of phrases is calculated according to a preset popularity algorithm, and phrases with popularity greater than a preset popularity threshold are determined as hot words.

In an embodiment of the present disclosure, the preset new word discovery algorithm may be an algorithm for acquiring new words in the related art, as long as the original data of the user's question can be calculated to acquire new words.

In one embodiment of the present disclosure, step S301 includes: using a preset new word discovery algorithm to calculate the degree of freedom and coagulation degree of the character string in the original data queried by the user, and using the preset new word threshold to calculate the character string Strings are restricted for new words. The character is used as a unit to calculate the degrees of freedom and cohesion, and by presetting the new word threshold, it can effectively identify the expression of new words and avoid bad cases in the process of discovering new words. The new word threshold may refer to the corresponding threshold set for the degree of freedom and solidification of the character string according to actual needs. In addition, the obtained new words may be potential new words in new questions, which may represent new products, new services, or new expressions. In addition, the discovered new words may be new words with high word granularity.

In one embodiment of the present disclosure, step S303 includes: using a preset phrase discovery algorithm to calculate the degrees of freedom and coagulation of words in the word segmentation result, and restricting the words by a preset phrase threshold to obtain phrases. Calculate the degree of freedom and cohesion of words as a unit, and through the preset phrase threshold, it can effectively identify the expression of phrases and avoid bad cases in the process of discovering phrases. The phrase threshold may refer to a corresponding threshold set for the degree of freedom and the degree of solidification of the word sequence according to actual needs.

In one embodiment of the present disclosure, when calculating the popularity of a phrase according to the preset popularity algorithm, a weighted analysis can be performed according to the information entropy and mutual information of the phrase, as well as the frequency of occurrence of the phrase, the appearance balance number of left and right neighbors, etc., to obtain The popularity of the phrase. In the embodiments of the present disclosure, information entropy is a concept used to measure the amount of information in information theory, which directly reflects the degree of ordering of the system. The more regular and orderly the system, the lower the information entropy; The higher the entropy is. Mutual information is a useful measure of information in information theory. It can be seen as the amount of information contained in a random variable about another random variable, or the uncertainty of a random variable that is reduced by knowing another random variable. sex. Left and right neighbors refer to the phrases to the left and right of the current phrase. According to the teachings of the embodiments of the present disclosure, those skilled in the art can understand that the popularity of a phrase can be calculated by a calculation method in the related art.

In the embodiment of the present disclosure, unlike the traditional schemes that use association mining and other methods to mine phrases, this scheme regards phrase aggregation as a special process of new word discovery, that is, a word is regarded as a unit to calculate the degree of freedom and cohesion, not only It can achieve phrase aggregation results, and can effectively identify emerging phrase expressions and avoid bad cases in the process of new word discovery. For example, in the process of discovering new words, the words "how to open" and "pass payment tool" were mistakenly mined. This scheme can synthesize the two words into an effective phrase "how to open a payment tool" through phrase mining (acquisition), so that Complete error correction.

Moreover, the Q&A recognition effect detection scheme according to the embodiments of the present disclosure combines the characteristics of the Q&A system, such as time fluctuations, user feedback, answer distribution, etc. for analysis, and can play a role in popularity recognition and long-tail question discovery.

The question and answer recognition effect detection device of the present disclosure will be described below with reference to FIG. 4 .

Fig. 4 shows a structural block diagram of a question and answer recognition effect detection device according to another embodiment of the present disclosure. The question and answer recognition effect detection device shown in FIG. 4 includes a hot word acquisition module 401 , a question and answer association module 402 , an anomaly detection module 403 and an anomaly determination module 404 .

The hot word acquisition module 401 is configured to acquire a hot word set including hot words according to the raw data of user questions.

The question-answer association module 402 is configured to sort and filter the hot words in the hot word set to determine hot questions, and associate the hot words with user feedback and answer information to obtain the answer distribution associated with the hot words , and obtain the hot words associated with the answer for the answer.

The anomaly detection module 403 is configured to detect whether the number of deduplicated answers associated with a hot word exceeds a first threshold and whether the number of hot words associated with an answer exceeds a second threshold.

The abnormality determination module 404 is configured to determine that a hot word occurs when the abnormality detection module 403 detects that the number of deduplicated answers associated with a hot word exceeds a first threshold and/or the number of hot words associated with an answer exceeds a second threshold. An abnormal case of word-answer matching.

In an embodiment of the present disclosure, the method of obtaining a hot word set including hot words according to the raw data of user questions may be to perform hot word mining. Based on the original data of user questions, hot word sets or candidate hot word sets can be obtained through operations such as new word discovery, phrase mining, and popularity calculation.

In an embodiment of the present disclosure, the question-answer association module 402 is further configured to: sort and filter hot word results according to the time period distribution of hot words, the entry diversity distribution of hot words, and the periodic analysis of hot words , to identify hot questions. The time period distribution of the hot words may refer to the trend of the number of hot words in units of time periods such as 10 minutes.

In an embodiment of the present disclosure, the question-answer associating module 402 is further configured to: build an inverted index with hot words as a dimension, and associate hot words with user feedback and answer information. Therefore, after the acquisition of hot words (hot word mining) is completed, an inverted index can be established with the hot words as the dimension, and the hot words can be associated with user feedback and answer information to speed up the subsequent analysis process. In addition, the user feedback may include the user's evaluation of the question answering system and the like.

In an embodiment of the present disclosure, the question-answer association module 402 is further configured to: analyze the question and answer pairs associated with the hot word in units of a single hot word to obtain the answer distribution associated with the hot word. It can be understood that there may be multiple pairs of questions and answers associated with a hot word, therefore, multiple pairs of questions and answers may be analyzed to obtain the distribution of answers associated with the hot word. For example, for a hot word with clear semantics, such as "credit card repayment", the corresponding answer distribution will revolve around the topic of the hot word.

In an embodiment of the present disclosure, the question-answer associating module 402 is further configured to: perform hotword clustering on the user's question corresponding to the answer in units of a single answer to obtain hotwords associated with the answer. For example, the original question corresponding to a certain answer should focus on one or a small number of hot topics, and should not be related to multiple hot words exceeding a certain threshold.

In one embodiment of the present disclosure, when it is detected that the number of deduplicated answers associated with a hot word exceeds a first threshold and/or the number of hot words associated with an answer exceeds a second threshold, it is determined that a hot word and An abnormal case of answer matching refers to: no matter whether the number of deduplicated answers associated with a hot word exceeds the first threshold, or the number of hot words associated with an answer exceeds the second threshold, or the deduplication associated with a hot word In any one of the three situations where the number of repeated answers exceeds the first threshold and the number of hot words associated with an answer exceeds the second threshold, it can be determined that there is an abnormal case of matching between hot words and answers. In other words, if there are too many deduplicated answers associated with a hot word, the question associated with the hot word has an answer that is not related to the question; The corresponding original question revolves around too many topics (hot words). It can be considered that there are cases of abnormal matching between hot words and answers, or there are cases of abnormal matching between candidate hot words and answers for subsequent investigation. In the embodiments of the present disclosure, the abnormal case of hotword-answer matching can be considered as one of the bad cases. Those skilled in the art can understand that, according to actual needs, the first threshold and the second threshold can be any number. In addition, deduplication refers to the removal of duplicate answers, which can avoid statistical and computational distortion.

In the embodiment of the present disclosure, the hot word acquisition module is configured to acquire a hot word set including hot words according to the raw data of user questions; the question-answer association module is configured to sort and filter hot words in the hot word set To determine hot questions, and associate hot words with user feedback and answer information, to obtain the answer distribution associated with the hot words for the hot words, and obtain the hot words associated with the answers for the answers; the anomaly detection module is configured In order to detect whether the number of deduplicated answers associated with a hot word exceeds the first threshold and whether the number of hot words associated with an answer exceeds the second threshold; the abnormality determination module is configured to detect an abnormality when the abnormality detection module detects a When the number of deduplicated answers associated with hot words exceeds the first threshold and/or the number of hot words associated with an answer exceeds the second threshold, it is determined that there is an abnormal case of matching between hot words and answers. By obtaining the hot word set, At the same time, through the two-way correlation analysis of hot words and answers, combined with user feedback, the Q&A recognition effect is centrally monitored, and it can effectively pay attention to the Q&A recognition effect of long-tail questions such as matching abnormal cases. The hot word mining scheme of related technologies can be completed by methods such as Apriori association rule mining and SegPhrase phrase discovery, which have high requirements for word segmentation accuracy or corpus quality. At the same time, the related technology solutions only stop at hot word mining, unlike In the solution of the embodiment of the present disclosure, the deep question answering system performs hot word evaluation and bad case discovery of question answer matching.

In embodiments of the present disclosure, long-tail questions may refer to those questions that do not appear frequently.

Fig. 5 shows a structural block diagram of a question and answer recognition effect detection device according to another embodiment of the present disclosure. The difference between the embodiment shown in FIG. 5 and the embodiment shown in FIG. 4 is that, in addition to including the hot word acquisition module 401, the question-and-answer association module 402, the anomaly detection module 403, and the anomaly determination module 404, FIG. 5 also includes missing steps A detection module 501 and a missing determination module 502 .

The missing detection module 501 is configured to detect whether the hot word is included in the deduplicated answer associated with the hot word.

The missing determination module 502 is configured to determine that a missing answer case occurs when the missing detection module 501 detects that the hot word is not included in all deduplicated answers associated with the hot word.

In one embodiment of the present disclosure, further analysis can be carried out in the case of abnormal matching between hot words and answers. If all the answer titles associated with a hot word do not contain the hot word string, it can be considered as There are missing answers in the answer library, so it can be considered that there is a missing answer case, or this case can be used as a candidate missing case for subsequent investigation. In embodiments of the present disclosure, the missing-answer case can be considered as one of the bad cases.

In the embodiment of the present disclosure, by obtaining the hot word set, at the same time, through the two-way association analysis of hot words and answers, combined with user feedback, the effect of question and answer recognition can be monitored intensively, and it can effectively pay attention to cases such as abnormal matching and missing answers The effect of question and answer recognition on long-tail questions such as .

Fig. 6 shows a structural block diagram of the hot word acquisition module 401 in the question and answer recognition effect detection device according to an embodiment of the present disclosure. The hot word acquisition module 401 includes a new word acquisition submodule 601 , a question segmentation submodule 602 , a phrase acquisition submodule 603 and a popularity calculation submodule 604 .

The new word acquisition sub-module 601 is configured to use a preset new word discovery algorithm to calculate the original data of the user's question to acquire new words, wherein a new word is composed of characters.

The question segmentation sub-module 602 is configured to generate a word segmentation dictionary based on the basic words in the preset basic thesaurus and the acquired new words, and use the word segmentation dictionary to segment the user's question to obtain a question segmentation result.

The phrase acquisition sub-module 603 is configured to calculate word segmentation results through a preset phrase discovery algorithm to acquire phrases, wherein the phrases are composed of words obtained after word segmentation.

The popularity calculation sub-module 604 is configured to calculate the popularity of phrases according to a preset popularity algorithm, and determine phrases with popularity greater than a preset popularity threshold as hot words.

In an embodiment of the present disclosure, the preset new word discovery algorithm may be an algorithm for acquiring new words in the related art, as long as the original data of the user's question can be calculated to acquire new words.

In one embodiment of the present disclosure, the new word acquisition sub-module 601 is configured to: use a preset new word discovery algorithm to calculate the degrees of freedom and solidification of character strings in the original data queried by the user, and use the preset Set the new word threshold to limit the string to get new words. The character is used as a unit to calculate the degrees of freedom and cohesion, and by presetting the new word threshold, it can effectively identify the expression of new words and avoid bad cases in the process of discovering new words. The new word threshold may refer to the corresponding threshold set for the degree of freedom and solidification of the character string according to actual needs. In addition, the obtained new words may be potential new words in new questions, which may represent new products, new services, or new expressions. In addition, the discovered new words may be new words with high word granularity.

In one embodiment of the present disclosure, the phrase acquisition sub-module 603 is further configured to: use a preset phrase discovery algorithm to calculate the degrees of freedom and solidification of words in the word segmentation results, and use the preset phrase threshold to calculate the degree of freedom and coagulation of words Limit to get phrases. Calculate the degree of freedom and cohesion of words as a unit, and through the preset phrase threshold, it can effectively identify the expression of phrases and avoid bad cases in the process of discovering phrases. The phrase threshold may refer to a corresponding threshold set for the degree of freedom and the degree of solidification of the word sequence according to actual needs.

In one embodiment of the present disclosure, when calculating the popularity of a phrase according to the preset popularity algorithm, a weighted analysis can be performed according to the information entropy and mutual information of the phrase, as well as the frequency of occurrence of the phrase, the appearance balance number of left and right neighbors, etc., to obtain The popularity of the phrase. In the embodiments of the present disclosure, information entropy is a concept used to measure the amount of information in information theory, which directly reflects the degree of ordering of the system. The more regular and orderly the system, the lower the information entropy; The higher the entropy is. Mutual information is a useful measure of information in information theory. It can be seen as the amount of information contained in a random variable about another random variable, or the uncertainty of a random variable that is reduced by knowing another random variable. sex. Left and right neighbors refer to the phrases to the left and right of the current phrase. According to the teachings of the embodiments of the present disclosure, those skilled in the art can understand that the popularity of a phrase can be calculated by a calculation method in the related art.

In the embodiment of the present disclosure, unlike the traditional schemes that use association mining and other methods to mine phrases, this scheme regards phrase aggregation as a special process of new word discovery, that is, a word is regarded as a unit to calculate the degree of freedom and cohesion, not only It can achieve phrase aggregation results, and can effectively identify emerging phrase expressions and avoid bad cases in the process of new word discovery. For example, in the process of discovering new words, the words "how to open" and "pass payment tool" were mistakenly mined. This scheme can synthesize the two words into an effective phrase "how to open a payment tool" through phrase mining (acquisition), so that Complete error correction.

Moreover, the Q&A recognition effect detection scheme according to the embodiments of the present disclosure combines the characteristics of the Q&A system, such as time fluctuations, user feedback, answer distribution, etc. for analysis, and can play a role in popularity recognition and long-tail question discovery.

The above has described the internal function and structure of the question and answer recognition effect detection device. In a possible design, the structure of the question and answer recognition effect detection device can be realized as a question and answer recognition effect detection device. As shown in FIG. 7, the processing device 700 can be It includes a processor 701 and a memory 702.

The memory 702 is used to store a program that supports the question and answer recognition effect detection device to execute the method for question and answer recognition effect detection in any of the above embodiments, and the processor 701 is configured to execute the program stored in the memory 702 .

The memory 702 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 701 .

The processor 701 is configured to execute all or part of the aforementioned method steps.

Wherein, the structure of the question and answer recognition effect detection device may further include a communication interface, which is used for the question and answer recognition effect detection device to communicate with other devices or communication networks.

Exemplary embodiments of the present disclosure also provide a computer storage medium for storing computer software instructions used by the question and answer recognition effect detection device, which includes instructions for executing the question and answer recognition effect detection method in any of the above embodiments. program.

Fig. 8 is a schematic structural diagram of a computer system suitable for implementing the method for detecting the effect of question-and-answer recognition according to an embodiment of the present disclosure.

As shown in FIG. 8 , a computer system 800 includes a central processing unit (CPU) 801 that can be programmed according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random-access memory (RAM) 803 Instead, various processes in the embodiment shown in FIG. 1 described above are executed. In RAM803, various programs and data necessary for the operation of system 800 are also stored. The CPU 801 , ROM 802 , and RAM 803 are connected to each other via a bus 804 . An input/output (I/O) interface 805 is also connected to the bus 804 .

The following components are connected to the I/O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc. and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 810 as necessary so that a computer program read therefrom is installed into the storage section 808 as necessary.

In particular, according to an embodiment of the present disclosure, the method described above with reference to FIG. 1 may be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on its readable medium, the computer program including program code for executing the data processing method of FIG. 1 . In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 809 and/or installed from a removable medium 811 .

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a roadmap or block diagram may represent a module, program segment, or part of code that contains one or more Executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units or modules involved in the embodiments described in the present disclosure may be implemented by means of software or hardware. The described units or modules may also be set in the processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

As another aspect, the present disclosure also provides a computer-readable storage medium. The computer-readable storage medium may be the computer-readable storage medium included in the device described in the above-mentioned embodiments; A computer-readable storage medium assembled in a device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the methods described in the present disclosure.

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principle. It should be understood by those skilled in the art that the scope of the invention involved in this disclosure is not limited to the technical solution formed by the specific combination of the above technical features, but also covers the technical solutions made by the above technical features without departing from the inventive concept. Other technical solutions formed by any combination of or equivalent features thereof. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

Claims (20)

1. A question-answer recognition effect detection method for a question-answer system is characterized by comprising the following steps:

acquiring a hot word set according to original data asked by a user;

sorting and screening the hotwords in the hotword set to determine a hot question, associating the hotwords with user feedback and answer information to obtain answer distribution associated with the hotwords for each hotword, and obtaining hotwords associated with the answer for each answer, wherein the user feedback comprises evaluation of a question-answer system by a user;

detecting whether the number of the duplicate-removed answers associated with one hotword exceeds a first threshold value and whether the number of the hotword associated with one answer exceeds a second threshold value;

when the fact that the number of the deduplicated answers related to a hotword exceeds a first threshold and/or the number of the hotword related to an answer exceeds a second threshold is detected, determining that an abnormal case of matching the hotword and the answers occurs in question-answering recognition of the question-answering system.

2. The method of claim 1, further comprising:

detecting whether the duplicate-removed answer associated with a hotword comprises the hotword;

and when all the deduplicated answers associated with a hotword are detected not to include the hotword, determining that the answer missing case occurs in the question-answer recognition of the question-answer system.

3. The method according to claim 1 or 2, wherein the obtaining of the set of hotwords from the raw data of the user's question comprises:

calculating original data of a user question through a preset new word finding algorithm to obtain a new word, wherein the new word takes characters as a constituent unit;

generating a word segmentation dictionary according to basic words in a preset basic word bank and the obtained new words, and segmenting a user question by using the word segmentation dictionary to obtain a question and word segmentation result;

calculating the word segmentation result through a preset phrase finding algorithm to obtain a phrase, wherein the phrase takes words obtained after word segmentation as a composition unit;

and calculating the heat of the phrase according to a preset heat algorithm, and determining the phrase with the heat larger than a preset heat threshold as a hotword.

4. The method according to claim 3, wherein the calculating the original data of the user's question by the preset new word finding algorithm to obtain the new word comprises:

the method comprises the steps of calculating the degree of freedom and the degree of solidification of a character string in original data of a user question through a preset new word discovery algorithm, and limiting the character string through a preset new word threshold value to obtain a new word.

5. The method according to claim 4, wherein the calculating the word segmentation result through a preset phrase finding algorithm to obtain a phrase comprises:

and calculating the degree of freedom and the degree of solidification of the words in the word segmentation result through a preset phrase finding algorithm, and limiting the words through a preset phrase threshold value to obtain phrases.

6. The method of claim 1, wherein the sorting and filtering the hotwords in the set of hotwords to determine a hot question comprises:

and sequencing and screening hot word results according to the time period distribution of the hot words, the inlet diversity distribution of the hot words and the periodic analysis of the hot words to determine the hot question.

7. The method of claim 1, wherein associating hotwords with user feedback and answer information comprises:

and establishing an inverted index by taking the hot words as dimensions, and associating the hot words with the user feedback and answer information.

8. The method of claim 1, wherein obtaining, for a hotword, a distribution of answers associated with the hotword comprises:

and analyzing the question and answer pairs associated with the hot words by taking a single hot word as a unit to obtain the answer distribution associated with the hot words.

9. The method of claim 1, wherein obtaining the hotword associated with the answer for the answer comprises:

and carrying out hot word clustering on the user question corresponding to the answer by taking a single answer as a unit so as to obtain a hot word associated with the answer.

10. A question-answer recognition effect detection apparatus for a question-answer system, comprising:

the hot word acquisition module is configured to acquire a hot word set according to original data of a user question;

a question-answer correlation module configured to rank and filter the hotwords in the hotword set to determine a hot question, and correlate the hotwords with user feedback and answer information to obtain, for each hotword, an answer distribution associated with the hotword, and obtain, for each answer, a hotword associated with the answer, the user feedback including a user evaluation on a question-answer system;

the anomaly detection module is configured to detect whether the number of the deduplicated answers associated with one hotword exceeds a first threshold and whether the number of the hotword associated with one answer exceeds a second threshold;

the anomaly determination module is configured to determine that an abnormal case of matching the hotwords and the answers is identified by the question-answering system when the anomaly detection module detects that the number of the deduplicated answers associated with one hotword exceeds a first threshold and/or the number of the hotwords associated with one answer exceeds a second threshold.

11. The apparatus of claim 10, further comprising:

a missing detection module configured to detect whether a deduplicated answer associated with a hotword includes the hotword;

the missing determination module is configured to determine that an answer missing case occurs in question and answer recognition of the question and answer system when the missing detection module detects that all deduplicated answers associated with one hotword do not include the hotword.

12. The apparatus according to claim 10 or 11, wherein the hotword obtaining module comprises:

the new word acquisition sub-module is configured to calculate original data asked by a user through a preset new word discovery algorithm to acquire a new word, wherein the new word takes characters as a constituent unit;

the question and word segmentation sub-module is configured to generate a word segmentation dictionary according to basic words in a preset basic word bank and the obtained new words, and perform word segmentation on the user questions by using the word segmentation dictionary to obtain a question and word segmentation result;

the phrase acquisition submodule is configured to calculate the word segmentation result through a preset phrase discovery algorithm so as to acquire a phrase, wherein the phrase takes words obtained after word segmentation as a component unit;

and the heat degree calculation sub-module is configured to calculate the heat degree of the phrase according to a preset heat degree algorithm and determine the phrase with the heat degree larger than a preset heat degree threshold value as a hot word.

13. The apparatus of claim 12, wherein the new word acquisition sub-module is further configured to:

the method comprises the steps of calculating the degree of freedom and the degree of solidification of a character string in original data of a user question through a preset new word finding algorithm, and limiting the character string through a preset new word threshold value to obtain a new word.

14. The apparatus of claim 13, wherein the phrase acquisition sub-module is further configured to:

and calculating the degree of freedom and the degree of solidification of the words in the word segmentation result through a preset phrase finding algorithm, and limiting the words through a preset phrase threshold value to obtain phrases.

15. The apparatus of claim 10, wherein the question-answer correlation module is further configured to:

and sequencing and screening hot word results according to the time period distribution of the hot words, the inlet diversity distribution of the hot words and the periodic analysis of the hot words so as to determine the hot question.

16. The apparatus of claim 10, wherein the question-answer correlation module is further configured to:

and establishing an inverted index by taking the hot words as dimensions, and associating the hot words with the user feedback and answer information.

17. The apparatus of claim 10, wherein the question-answer correlation module is further configured to:

and analyzing the question and answer pairs associated with the hot words by taking a single hot word as a unit to obtain the answer distribution associated with the hot words.

18. The apparatus of claim 10, wherein the question-answer correlation module is further configured to:

and performing hot word clustering on the user question corresponding to the answer by taking a single answer as a unit to obtain a hot word associated with the answer.

19. An electronic device comprising a memory and a processor; wherein the memory is to store one or more computer instructions, wherein the one or more computer instructions are to be executed by the processor to implement the method of any one of claims 1-9.

20. A readable storage medium having stored thereon computer instructions, which when executed by a processor, perform the method of any one of claims 1-9.

Images