WO2026002003A1 - Question-answering method and apparatus, and device and storage medium - Google Patents

Abstract

A question-answering method and apparatus, and a device and a storage medium. The method comprises: in response to question speech of a user having been received, acquiring question text identified from the question speech (410); on the basis of the question text, determining assistance information which matches the question text (420); sending the assistance information to a serving end (430); and receiving from the serving end at least one of response text or response speech for the question speech (440), wherein the response speech corresponds to the response text, and the response text is determined on the basis of the question text and the assistance information. In this way, a user-friendly service capability in a human-computer interaction scenario can be improved, and data security can be ensured.

Description

Methods, apparatus, devices, and storage media for question answering

This application claims priority to Chinese Patent Application No. 202410869789.8, filed on June 28, 2024, entitled "Method, Apparatus, Device and Storage Medium for Question Answering", the entire contents of which are incorporated herein by reference.

Technical Field

The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, and computer-readable storage media for question answering.

Background Technology

With the development of artificial intelligence technology, various application forms have emerged, including but not limited to text-based question answering, voice-based question answering, and text-to-image generation. Analysis of these applications reveals that the service capabilities of the models still need improvement during the interaction process.

Summary of the Invention

In a first aspect of this disclosure, a method for question answering is provided. The method includes: in response to receiving a user's question voice, acquiring question text identified from the question voice; determining auxiliary information matching the question text based on the question text; sending the auxiliary information to a server; and receiving from the server at least one of a response text or a response voice corresponding to the question voice, wherein the response voice corresponds to the response text, and the response text is determined based on the question text and the auxiliary information.

In a second aspect of this disclosure, a method for question answering is provided. The method includes: in response to receiving a user's question voice from a client, sending a question text identified from the question voice to the client; receiving auxiliary information from the client that matches the question text identified from the question voice; determining a response text based on the question text and the auxiliary information; and feeding back at least one of a response text or a response voice to the client, the response voice corresponding to the response text.

In a third aspect of this disclosure, an apparatus for question answering is provided. The apparatus includes: a question text acquisition module configured to acquire question text identified from a question voice in response to receiving a user's question voice; an auxiliary information determination module configured to determine auxiliary information matching the question text based on the question text; an auxiliary information sending module configured to send the auxiliary information to a server; and a response receiving module configured to receive at least one of a response text or a response voice from the server in response to the question voice, wherein the response voice corresponds to the response text, and the response text is determined based on the question text and the auxiliary information.

In a fourth aspect of this disclosure, an apparatus for question answering is provided. The apparatus includes: a question text sending module configured to send question text identified from the question voice to a client in response to receiving a user's question voice from a client; an auxiliary information receiving module configured to receive auxiliary information from the client that matches the question text identified from the question voice; a response text determining module configured to determine response text based on the question text and the auxiliary information; and a response feedback module configured to feed back at least one of response text or response voice to the client, wherein the response voice corresponds to the response text.

In a fifth aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the electronic device to perform either the method of the first aspect or the method of the second aspect.

In a sixth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores computer-executable instructions that, when executed by a processor, implement the method of the first aspect or the method of the second aspect.

In a seventh aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions, wherein when executed by a processor, the computer-executable instructions implement the method according to a first aspect or a second aspect of this disclosure.

It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

Attached Figure Description

The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

Figure 2 illustrates a flowchart of a signaling flow for question answering according to some embodiments of the present disclosure;

Figure 3 illustrates a schematic diagram of an example architecture for question answering according to some embodiments of the present disclosure;

Figure 4 shows a flowchart of a question-and-answer process according to some embodiments of the present disclosure;

Figure 5 shows a flowchart of a question-and-answer process according to some embodiments of the present disclosure;

Figure 6 shows an exemplary structural block diagram of a question-and-answer apparatus according to some embodiments of the present disclosure;

Figure 7 illustrates an exemplary structural block diagram of a question-and-answer apparatus according to some embodiments of the present disclosure; and

Figure 8 shows a block diagram of an electronic device that can implement one or more embodiments of the present disclosure.

Detailed Implementation

Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, relevant users should be informed of the type, scope of use, and usage scenarios of the information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and authorization should be obtained from the relevant users. Among them, relevant users may include any type of rights holder, such as individuals, enterprises, and groups.

For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly inform the user that the requested operation will require obtaining and using the user's information, thereby enabling the relevant user to choose whether to provide information to the software or hardware such as the electronic device, application, server, or storage medium that performs the operation of the technical solution disclosed herein based on the prompt message.

As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide information to the electronic device.

It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameter values until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values to determine the corresponding output.

Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In this example environment 100, an application 120 is installed on a client 110. A user 140 can interact with the application 120 via the client 110 and/or an attached device of the client 110. For example, the application 120 can capture the voice 145 of the user 140 via a voice capture device (e.g., a microphone) of the client 110.

In embodiments of this disclosure, application 120 can be any suitable application with question-and-answer functionality. For example, application 120 can utilize a digital assistant to provide question-and-answer services. This digital assistant supports text-based question-and-answer services, voice-based question-and-answer services, and content-based question-and-answer services in other modalities with user 140. In some embodiments, application 120 or its digital assistant can utilize machine learning model 160 (which may include one or more machine learning models, such as machine learning model 160-1, machine learning model 160-2, ..., machine learning model 160-N, etc., where N is a positive integer. For ease of description, one or more machine learning models are collectively referred to herein as machine learning model 160) to support interaction with user 140. For example, application 120 or its digital assistant can utilize one or more machine learning models 160 to provide question-and-answer services to user 140.

In environment 100, if application 120 is active, client 110 can present the user interface 150 of application 120. User interface 150 may include various pages that application 120 can provide, such as a question-and-answer page between the user and a digital assistant, etc. In some embodiments, client 110 may play voice 152 and present text 154 in user interface 150. Voice 152 may, for example, include voice 145 from user 140 or voice responses to voice 145.

Machine learning model 160 can be of different types. In some embodiments, one or more machine learning models 160 may be built based on a language model (LM). The machine learning model used is a content-generative model, capable of generating corresponding outputs based on model inputs. In some embodiments, the language model-based machine learning model can handle textual modal model inputs (e.g., natural language and/or machine language) and/or non-textual modal model inputs (e.g., images, speech, video, etc.), and can generate the desired output based on the model inputs and prompt words. Here, prompt words are used to guide the machine learning model to generate outputs that address the user needs indicated by the model inputs. In applications supporting user question answering, user 140's input can be provided to machine learning model 160 as at least a part of the model inputs (other parts may include prompt words). This user input is considered a question. Based on the model outputs, corresponding responses can be generated and provided to user 140.

In some embodiments, one or more machine learning models 160 may be speech-related models, including speech recognition (ASR) models and text-to-speech (TTS) models. The input to an ASR model is speech, and the output is text. The input to a TTS model is text, and the output is the corresponding speech.

In some embodiments, client 110 communicates with server 130 to provide services to application 120. As shown in FIG1, server 130 may invoke machine learning model 160 to support question-and-answer functionality between application 120 and user 140 based on the output of machine learning model 160. Client 110 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio/video players, digital cameras/camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, client 110 may also support any type of user-facing interface (such as "wearable" circuitry). Server 130 may be various types of computing systems/servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc. Server 130 may, for example, be implemented in a cloud environment.

It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

As mentioned earlier, with the development of artificial intelligence technology, various application forms have emerged, including but not limited to text-based question answering, voice-based question answering, and text-to-image generation. Analysis of these applications reveals that the service capabilities of the models still need improvement during the interaction process.

In view of the above, according to embodiments of the present disclosure, an improved solution for question answering is provided. According to the solution of the embodiments of the present disclosure, in response to receiving a user's question voice, question text identified from the question voice is obtained; based on the question text, auxiliary information matching the question text is determined; the auxiliary information is sent to a server; and at least one of a response text or a response voice for the question voice is received from the server, wherein the response voice corresponds to the response text, and the response text is determined based on the question text and the auxiliary information.

In this way, in a question-and-answer scenario, the client determines the auxiliary information that matches the question text based on the question text and sends the auxiliary information to the server. This allows for the generation of response text and response voice based on the question text and auxiliary information, which can improve the accuracy of the response content and thus enhance the service capabilities in the question-and-answer scenario. Furthermore, it avoids storing the auxiliary information on the server or in the cloud, which helps to ensure the data security of the auxiliary information.

The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

Figure 2 illustrates a flowchart of a signaling flow 200 for question answering according to some embodiments of the present disclosure. The signaling flow 200 relates to a client 110 and a server 130. Figure 3 illustrates a schematic diagram of an example architecture 300 for question answering according to some embodiments of the present disclosure. For ease of discussion, the signaling flow 200 will be described with reference to the environment of Figure 1 and in conjunction with the example architecture 300 shown in Figure 3.

In embodiments of this disclosure, as shown in signaling stream 200, client 110, in response to receiving a question voice from user 140, sends (202) a question voice to server 130. Server 130 receives (204) the question voice sent by client 110 and identifies question text from the question voice. Server 130 sends (208) the identified question text to client 110, and client 110 receives (210) the question text from server 130.

As shown in Figure 3, client 110 may include service module 310. Service module 310 may correspond to an application that supports voice question and answer. Service module 310 includes application voice module 311 and application messaging module 312. Server 130 may include access layer 340, voice module 350, and question and answer module 360. Access layer 340 includes network interface 341 and network interface 342. A first network connection may be established between network interface 341 of client 110 and server 130. A second network connection may be established between network interface 342 of client 110 and server 130. Alternatively or additionally, the first network connection and/or the second network connection may be a long connection conforming to the Transmission Control Protocol (TCP). Of course, the first network connection and the second network connection are not limited to long connections conforming to the TCP protocol; the first network connection and the second network connection may also be network connections conforming to other communication protocols. In specific implementation, the configuration can be selected according to actual needs.

The application voice module 311 can receive voice (e.g., voice 145) from a user (e.g., user 140). In some embodiments, the application voice module 311 can receive voice from user 140 (i.e., "user voice," sometimes referred to herein as "problem voice"). The application voice module 311 can send the user voice to the network interface 341 of the server 130 via a first network connection, and then send the user voice to the voice module 350 of the server 130 via the network interface 341.

The voice module 350 can perform ASR and/or TTS functions using the ASR model 371 and/or TTS model 372 in the voice service 370. The ASR model 371 and/or TTS model 372 are deployed on the server 130, or alternatively, the ASR model 371 and/or TTS model 372 can be deployed in the cloud. For example, the voice module 350 can send the user's voice to the ASR model 371 and obtain the ASR text corresponding to the user's voice (i.e., the question text converted from the question voice) from the ASR model 371.

The voice module 350 can send the acquired ASR text to the network interface 341 of the access layer 340. The access layer 340 can then send the acquired ASR text to the application voice module 311 of the client 110. Alternatively or additionally, the access layer 340 can send the ASR text to the client 110 via a long-lived connection between the network interface 341 and the client 110.

In some embodiments, the application voice module 311 of client 110 can also provide user voice to the ASR model 321 (sometimes referred to herein as the "first machine learning model") in the local model capability 320 of client 110, and use the ASR model 321 to perform speech recognition on the user voice to generate ASR text corresponding to the user voice. This can improve the speed of ASR text acquisition. Even with low network communication capabilities, it ensures that client 110 can effectively acquire the ASR text corresponding to the user voice, thereby ensuring that question-and-answer can proceed normally.

In some embodiments, the application speech module 311 may send the ASR text to the application messaging module 312 in response to receiving ASR text from the server 130 or from the ASR model 321. The application messaging module 312 may present the ASR text to the user 140. For example, the application messaging module 312 may present the ASR text (i.e., present the first question text after speech conversion) in the user interface.

In embodiments of this disclosure, as shown in signaling stream 200, client 110 determines (212) auxiliary information that matches the question text based on the question text identified from the question speech. Client 110 sends (214) the auxiliary information to server 130, and server 130 receives (216) the auxiliary information from client 110.

It is understandable that the problem text in the auxiliary information that the client 110 determines matches the problem text can be the ASR text received by the client 110 from the server 130, or the ASR text obtained by the client 110 from the ASR model 321 in the local model capability 320.

In some embodiments, client 110 may locally deploy an auxiliary information base 323, which may store at least one category of auxiliary information. Client 110 may determine auxiliary information matching the question text from the auxiliary information base 323. This auxiliary information may include various information relevant to the current user, which, in a question-and-answer scenario, can help improve the question-and-answer model 380's understanding of the user's intent and enhance question-and-answer capabilities and satisfaction. In some embodiments, the auxiliary information may be information provided in advance by the user. For example, at least one type of auxiliary information may include, but is not limited to, user-provided profiles, schedule information, knowledge bases of areas of interest, etc. In some embodiments, the auxiliary information may be summarized and recorded from historical question-and-answer records, etc. With the help of auxiliary information, the question-and-answer model can generate responses that are more satisfactory and more in line with the user's expectations.

It should be noted that the auxiliary information in the embodiments of this disclosure is acquired, stored, and used with the user's authorization. In the embodiments of this disclosure, various auxiliary information for the current user is stored locally on the client, and one or more pieces of auxiliary information are provided to the server only when needed, thus ensuring data security.

In some embodiments, client 110 provides the question text to information determination model 322 (sometimes referred to herein as a “second machine learning model”) in local model capability 320, using information determination model 322 to determine auxiliary information that matches the ASR text. Application speech module 311 may receive the auxiliary information fed back by information determination model 322. In some embodiments, information determination model 322 may be a text matching model to perform text matching between the ASR text corresponding to the question speech and the auxiliary information, thereby determining the best-matching auxiliary information.

Alternatively or additionally, the information determination model 322 can determine auxiliary information matching the ASR text from the auxiliary information base 323 for the user 140 based on the ASR text. Alternatively or additionally, the ASR model 321 or ASR model 371 can be configured to perform text recognition on the user's speech in a streaming manner, sequentially generating at least one text sequence of ASR text. The ASR model 321 or ASR model 371 can provide at least one text sequence of ASR text to the application speech module 311 in the generation order. The application speech module 311 can provide at least one text sequence of ASR text to the information determination model 322 in the generation order, and the information determination model 322 can determine matching auxiliary information based on all or part of the text sequence in the at least one text sequence.

For example, when adding auxiliary information to the auxiliary information database 323, the category corresponding to the auxiliary information can be determined, the auxiliary information can be added to the auxiliary information database 323, and the annotation used to label the category corresponding to the auxiliary information can also be added to the auxiliary information database 323. The information determination model 322 can determine the category corresponding to the ASR text based on at least one received text sequence. Then, the corresponding auxiliary information can be retrieved from the auxiliary information database 323 based on the determined category of the ASR text.

It should be understood that the information determination model 322 is not limited to determining auxiliary information matching the ASR text from the auxiliary information database 323; the information determination model 322 also obtains auxiliary information matching the ASR text from the network or remote devices. For example, the information determination model 322 can also acquire various information such as current events, news events, weather information, or historical knowledge that match the ASR text as auxiliary information matching the ASR text.

In some embodiments, after the application voice module 311 of the client 110 obtains the auxiliary information, it can send the auxiliary information to the network interface 341 of the server via the first network connection. The network interface 341 can provide the auxiliary information to the voice module 350.

In embodiments of this disclosure, as shown in signaling flow 200, server 130 determines (218) a response text based on the question text and auxiliary information. Server 130 sends (220) the response text to client 130, and client 110 receives the response text from server 130.

In some embodiments, server 130 may generate model input for question-answering model 380 (sometimes referred to herein as a "third machine learning model") based on question text and auxiliary information. Server 130 may then invoke question-answering model 380 to generate response text based on the model input.

Alternatively or additionally, the voice module 350 in the server 130 can also construct a sending message based on the acquired ASR text, and send the sending message and auxiliary information to the question-and-answer module 360. The question-and-answer module 360 can generate model input based on the sending message and auxiliary information, and can call the question-and-answer model 380 to generate response text based on the model input. The question-and-answer module 360 can receive the response text from the question-and-answer model 380, and can send the response text to the access layer 340. The network interface 342 of the access layer 340 can use a second network connection to send the response text to the application messaging module 312 of the client 110. In this way, ASR text and auxiliary messages can be transmitted between the voice module 350 and the question-and-answer model 360 of the server 130, which helps to simplify the interaction process between the server 130 and the client 110.

For example, suppose the auxiliary information base 323 stores a user's historical question-and-answer records. These records include question-and-answer record A, "What's the weather like today?", and the context record for record A, including question-and-answer record B, "What clothes should I wear?". Assuming the ASR text includes "What's the weather like today?", the information determination model 322 can use question-and-answer records A, B, and their contextual relationship as auxiliary information. The client 110 can upload this auxiliary information to the server 130. The question-and-answer model 380 can then concatenate the ASR text "What's the weather like today?" with the auxiliary information "'What's the weather like today?' and 'What clothes should I wear?'" to form the model input. The response text generated by the question-and-answer model 380 based on this input might include phrases like "Today is sunny, the temperature is XX, suitable for wearing a light coat," or "Today is cloudy, the temperature is XX, suitable for wearing warm clothes," etc. In this way, the question-and-answer model 380 can more accurately understand the user's questioning intent, reducing the number of questions asked and improving question-and-answer efficiency.

For example, suppose the auxiliary information base 323 may include user-provided schedule information. The user adds a schedule reminder in application 120, which may include "Traveling from city A to city B on [Date]". Suppose on [Date], user 140 asks (i.e., ASR text) "What's the weather like today?", information determination model 322 can use this schedule reminder as auxiliary information for the user. Question answering model 380 can concatenate the ASR text "What's the weather like today?" and the auxiliary information "Traveling from city A to city B on [Date]" as model input. The response text generated by question answering model 380 based on this model input might include "The weather in city A is [Weather Name] today, the weather in city B is [Weather Name] today," etc. Thus, although user 140's question is broad, question answering model 380 accurately understands user 140's intent, and the provided response text or voice is more accurate and better suited to user 140's actual needs.

Alternatively or additionally, the voice module 350 can also determine the context of the ASR text based on the ASR text, construct a message to be sent based on the ASR text, and send the message, auxiliary information, and the context of the ASR text to the question-answering module 360. The question-answering module 360 can generate model input for the question-answering model 380 based on the context of the message, auxiliary information, and ASR text. Then, the question-answering model 360 can provide the model input to the question-answering model 380, triggering the question-answering model 380 to generate response text based on the model input. The question-answering model 360 can receive the response text fed back by the question-answering model 380, send the response text to the access layer 340, and send the response text to the application messaging module 312 of the client 110 through the network interface 342. By adding the context of the ASR text, the accuracy of the response text can be improved.

In some embodiments, before receiving auxiliary information from the client 110, the server 130 may generate a first model input for the question model 380 (i.e., the third machine learning model) based on the ASR text. The server 130 may provide the first model input to the question model 380 so that the question-answering model 380 generates a response text for the first model input. Alternatively or additionally, the voice module 350 may receive ASR text from the ASR model 371, assuming that the voice module 350 has not yet received auxiliary information. The voice module 350 may send the ASR text to the question-answering module 360. The question-answering module 360 may generate a first model input based on the ASR text and provide the first model input to the question-answering model 380, triggering the question-answering model 380 to generate a response text for the first model input. In this way, even when effective auxiliary information cannot be obtained, the response text for the first model input can still be fed back to the client 110, ensuring the normal progress of question-answering. Even with a long delay in providing auxiliary information, the response text for the first model input can still be promptly fed back to the client 110, ensuring the timeliness of the response text feedback.

In some embodiments, after receiving auxiliary information from the client 110, the voice module 350 of the server 130 can send the auxiliary information to the question-and-answer module 360. The question-and-answer module 360 can generate a second model input for the question-and-answer model 380 based on the ASR text and auxiliary information. The question-and-answer module 360 can provide the second model input to the question-and-answer model 380, interrupting the question-and-answer model 380 from generating response text for the first model input, and triggering the question-and-answer model 380 to generate response text for the second model input. The question-and-answer model 380 can feed back the response text for the second model input to the application message module 312 of the client 110. In this way, when auxiliary information can be obtained in a timely manner, a more accurate response text for the user 140 can be obtained, thereby improving the quality of question-and-answer.

In the embodiments of this disclosure, as shown in signaling stream 200, server 130 generates response voice corresponding to response text, server 130 sends (226) response voice to client 110, and client 110 receives (228) response voice from server 130.

The response text here can be the response text input to the first model or the response text input to the second model. In some embodiments, the question-answering module 360 can receive the response text from the question-answering model 380 and send the response text to the speech module 350. The speech module 350 can provide the response text to the TTS model 372, triggering the TTS model to generate the response speech corresponding to the response text. The response speech here is also the speech converted from the response text, which can also be referred to as TTS speech.

In some embodiments, after receiving a TTS voice message, the voice module 350 can send the TTS voice message to the network interface 341 of the access layer 340, or send the TTS voice message to the application voice module 311 of the client 110 via a first network connection. It is understood that in practical applications, the server 130 can provide the client 110 with response text, response voice, or both, depending on the question-and-answer method of the application 120. For example, when the application 120 is configured to conduct text-based question-and-answer with the user 140, the server 130 can provide the client 110 with response text. When the application 120 is configured to conduct voice-based question-and-answer with the user 140, the server 130 can provide the client 110 with TTS voice. When the application 120 is configured to conduct both text and voice-based question-and-answer with the user 140, the server 130 can provide the client 110 with response text and TTS voice respectively.

In embodiments of this disclosure, as shown in signaling stream 200, client 110 plays (230) the response voice and presents (230) the response text. Here, the playback of the response voice and the presentation of the response text are synchronized.

In the embodiments of this disclosure, "synchronization" can refer to the simultaneous or nearly simultaneous timing of the playback of the response audio and the presentation of the response text. It should be understood that, depending on timing accuracy, processing performance, etc., a certain degree of error can be allowed in temporal synchronization. For example, if the time difference between the playback of the response audio and the presentation of the response text is less than a certain threshold, the playback of the response audio and the presentation of the response text can be considered synchronized. Furthermore, the use of "synchronization" in other embodiments of this disclosure follows a similar principle.

For example, client 110 can use application voice module 311 to play TTS voice messages and use application messaging module 312 to present response text on user interface 150 synchronously with the TTS voice playback. Alternatively or additionally, application messaging module 312 can present response text on user interface 150 synchronously with the TTS voice playback based on the correspondence between TTS voice messages and response text. Here, the correspondence between TTS voice messages and response text may simply be the voice portion corresponding to each text character or text string in the response text.

In summary, according to the embodiments of this disclosure, in a question-and-answer scenario, response text and response voice can be generated based on the question text and auxiliary information matching the question text. This can improve the service capabilities in the question-and-answer scenario and avoid storing auxiliary information on the server or cloud, which is beneficial to ensuring the data security of auxiliary information.

Figure 4 shows a flowchart of a question-and-answer process 400 according to some embodiments of the present disclosure. Process 400 can be implemented at client 110.

In box 410, client 110 responds to receiving the user's question voice and obtains the question text identified from the question voice.

In box 420, client 110 determines auxiliary information that matches the question text based on the question text.

In box 430, client 110 sends auxiliary information to server 130.

In box 440, client 110 receives from server 130 at least one of response text or response voice for a question voice, the response voice corresponding to the response text, and the response text being determined based on the question text and auxiliary information.

In some embodiments, process 400 is further configured to receive the question text identified from the question speech from server 130.

In some embodiments, process 400 is further configured to: provide question speech to a first machine learning model so that the first machine learning model performs speech recognition on the question speech; and receive question text as feedback from the first machine learning model.

In some embodiments, at least one category of auxiliary information is stored for the user in the auxiliary information base of the client 110, and the process 400 is further configured to: determine auxiliary information that matches the question text from the auxiliary information base based on the question text.

In some embodiments, process 400 is further configured to: provide question text to a second machine learning model so that the second machine learning model determines auxiliary information that matches the question text; and receive auxiliary information fed back by the second machine learning model.

In some embodiments, process 400 is further configured to: receive a response voice from server 130 via a first network connection between client 110 and server 130; and/or receive a response text from server 130 via a second network connection between client 110 and server 130.

In some embodiments, process 400 is further configured to: present response text while playing response voice at client 110, wherein the playback of response voice and the presentation of response text are synchronized.

Figure 5 shows a flowchart of a question-and-answer process 500 according to some embodiments of the present disclosure. Process 500 can be implemented at server 130.

In box 510, server 130 responds to receiving the user's question voice from client 110 by sending the question text identified from the question voice to client 110.

In box 520, server 130 receives auxiliary information from client 110 that matches the question text recognized from the question speech.

In box 530, server 130 determines the response text based on the question text and auxiliary information.

In box 540, server 130 sends at least one of response text or response voice to client 110, where response voice corresponds to response text.

In some embodiments, process 500 is further configured to: generate model input for a third machine learning model based on the question text and auxiliary information; and invoke the third machine learning model to generate response text based on the model input.

In some embodiments, process 500 is further configured to: generate a first model input for a third machine learning model based on the question text before receiving auxiliary information from client 110; and provide the first model input to the third machine learning model so that the third machine learning model generates a response text for the first model input.

In some embodiments, process 500 is further configured to: after receiving auxiliary information from client 110, generate a second model input for a third machine learning model based on the question text and the auxiliary information; and provide the second model input to the third machine learning model, interrupt the third machine learning model from generating a response text for the first model input, and trigger the third machine learning model to generate a response text for the second model input.

In some embodiments, process 500 is further configured to: send a response voice to client 110 via a first network connection between server 130 and client 110; and/or send a response text to client 110 via a second network connection between server 130 and client 110.

Embodiments of this disclosure also provide corresponding apparatus for implementing the methods or processes described above. Figure 6 shows an exemplary structural block diagram of a question-and-answer apparatus 600 according to some embodiments of this disclosure. The apparatus 600 may be implemented as or included in the client 110. The various modules/components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

As shown in Figure 6, the device 600 includes a question text acquisition module 610, an auxiliary information determination module 620, an auxiliary information sending module 630, and a response receiving module 640. The question text acquisition module 610 is configured to acquire question text identified from the question voice in response to receiving a user's question voice. The auxiliary information determination module 620 is configured to determine auxiliary information matching the question text based on the question text. The auxiliary information sending module 630 is configured to send the auxiliary information to a server. The response receiving module 640 is configured to receive at least one of a response text or a response voice from the server in response to the question voice, wherein the response voice corresponds to the response text, and the response text is determined based on the question text and the auxiliary information.

In some embodiments, the problem text acquisition module 610 is further configured to receive the problem text identified from the problem speech from the server.

In some embodiments, the question text acquisition module 610 is further configured to: provide question speech to a first machine learning model so that the first machine learning model performs speech recognition on the question speech; and receive question text as feedback from the first machine learning model.

In some embodiments, at least one category of auxiliary information is stored for the user in the auxiliary information library of the client, and the auxiliary information determination module 620 is further configured to: determine auxiliary information that matches the question text from the auxiliary information library based on the question text.

In some embodiments, the auxiliary information determination module 620 is further configured to: provide the question text to the second machine learning model so that the second machine learning model determines auxiliary information that matches the question text; and receive auxiliary information fed back by the second machine learning model.

In some embodiments, the response receiving module 640 is further configured to: receive response voice from the server via a first network connection between the client and the server; and/or receive response text from the server via a second network connection between the client and the server.

In some embodiments, the device 600 further includes a playback presentation module configured to present response text while playing response voice at the client, wherein the playback of the response voice and the presentation of the response text are synchronized.

Figure 7 illustrates an exemplary structural block diagram of a question-answering device 700 according to some embodiments of the present disclosure. The device 700 may be implemented as or included in the server 130. The various modules/components in the device 700 may be implemented by hardware, software, firmware, or any combination thereof.

As shown in Figure 7, the device 700 includes a question text sending module 710, an auxiliary information receiving module 720, a response text determining module 730, and a response feedback module 740. The question text sending module 710 is configured to send question text identified from the question voice received from the client. The auxiliary information receiving module 720 is configured to receive auxiliary information from the client that matches the question text identified from the question voice. The response text determining module 730 is configured to determine response text based on the question text and the auxiliary information. The response feedback module 740 is configured to provide feedback to the client with at least one of response text or response voice, where the response voice corresponds to the response text.

In some embodiments, the response text determination module 730 is further configured to: generate model input for a third machine learning model based on the question text and auxiliary information; and invoke the third machine learning model to generate response text based on the model input.

In some embodiments, the response text determination module 730 is further configured to: generate a first model input for a third machine learning model based on the question text before receiving auxiliary information from the client; and provide the first model input to the third machine learning model so that the third machine learning model generates a response text for the first model input.

In some embodiments, the response text determination module 730 is further configured to: after receiving auxiliary information from the client, generate a second model input for the third machine learning model based on the question text and the auxiliary information; and provide the second model input to the third machine learning model, interrupt the third machine learning model from generating a response text for the first model input, and trigger the third machine learning model to generate a response text for the second model input.

In some embodiments, the response feedback module 740 is further configured to: provide response voice to the client via a first network connection between the server and the client; and/or provide response text to the client via a second network connection between the server and the client.

The units and/or modules included in device 700 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and/or modules can be implemented using software and/or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and/or modules in device 700 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

It should be understood that one or more steps in the above methods can be performed by appropriate electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices may, for example, include the client 110 in Figure 1.

Figure 8 shows a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 800 shown in Figure 8 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 800 shown in Figure 8 can be used to implement the client 110 or server 130 of Figure 1, or to implement the device 600 of Figure 6 or the device 700 of Figure 7.

As shown in Figure 8, the electronic device 800 is in the form of a general-purpose electronic device. Components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage devices 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

Electronic device 800 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 830 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and/or data and can be accessed within electronic device 800.

Electronic device 800 may further include additional removable/non-removable, volatile/non-volatile storage media. Although not shown in FIG8, disk drives for reading or writing from removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading or writing from removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 820 may include computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

The communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 800 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

Input device 850 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 860 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 800 can also communicate with one or more external devices (not shown) via communication unit 840 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 800, or with any device that enables electronic device 800 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input/output (I/O) interface (not shown).

According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

Various aspects of this disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions/actions specified in one or more blocks of the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and/or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions/actions specified in one or more blocks of the flowchart and/or block diagram.

Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions/actions specified in one or more boxes of a flowchart and/or block diagram.

The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some, as newer, implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims (17)

A question-answering method, applied to a client, the method comprising: In response to receiving a user's voice question, obtain the question text identified from the voice question; Based on the question text, determine the auxiliary information that matches the question text; The auxiliary information is sent to the server; and The server receives at least one of a response text or a response voice in response to the question voice, wherein the response voice corresponds to the response text, and the response text is determined based on the question text and the auxiliary information. According to the method of claim 1, obtaining the question text recognized from the question speech includes: Receive the question text identified from the question speech from the server. According to the method of claim 1, obtaining the question text recognized from the question speech includes: The question speech is provided to a first machine learning model, so that the first machine learning model performs speech recognition on the question speech; and Receive the question text fed back by the first machine learning model. The method according to claim 1, wherein at least one category of auxiliary information is stored in the auxiliary information base of the client, and wherein determining the auxiliary information that matches the question text includes: Based on the question text, auxiliary information matching the question text is determined from the auxiliary information database. According to the method of claim 1, determining the auxiliary information matching the question text includes: The question text is provided to a second machine learning model so that the second machine learning model determines the auxiliary information that matches the question text; and Receive the auxiliary information fed back by the second machine learning model. According to the method of claim 1, receiving at least one of the response text or response voice from the server in response to the question voice includes: The client receives the response voice from the server via a first network connection between the client and the server; and/or The response text is received from the server via a second network connection between the client and the server. The method according to claim 6 further includes: The response text is displayed when the response voice is played at the client, and the playback of the response voice and the display of the response text are synchronized. A question-answering method, applied on a server-side, the method comprising: In response to receiving a user's voice question from the client, send the question text identified from the voice question to the client; Receive auxiliary information from the client that matches the question text identified from the question speech; Based on the question text and the auxiliary information, determine the response text; and The system provides the client with at least one of the response text or response voice, wherein the response voice corresponds to the response text. According to the method of claim 8, determining the response text based on the question text and the auxiliary information includes: Based on the question text and the auxiliary information, generate model input for the third machine learning model; and The third machine learning model is invoked to generate the response text based on the model input. The method according to claim 8 further includes: Before receiving the auxiliary information from the client, a first model input for the third machine learning model is generated based on the question text; and The first model input is provided to the third machine learning model so that the third machine learning model generates a response text for the first model input. According to the method of claim 10, determining the response text based on the question text and the auxiliary information includes: After receiving the assistance information from the client, a second model input for the third machine learning model is generated based on the question text and the assistance information; and The second model input is provided to the third machine learning model, interrupting the third machine learning model from generating a response text for the first model input, and triggering the third machine learning model to generate a response text for the second model input. The method of claim 8, wherein feeding back at least one of the response text or response voice to the client comprises: The server sends the response voice back to the client via a first network connection between the server and the client; and/or The response text is sent back to the client via a second network connection between the server and the client. An apparatus for question answering, comprising: The question text acquisition module is configured to acquire the question text identified from the question voice in response to receiving a user's question voice. The auxiliary information determination module is configured to determine auxiliary information that matches the question text based on the question text. The auxiliary information sending module is configured to send the auxiliary information to the server; and The response receiving module is configured to receive from the server at least one of a response text or a response voice in response to the question voice, wherein the response voice corresponds to the response text, and the response text is determined based on the question text and the auxiliary information. An apparatus for question answering, comprising: The question text sending module is configured to send question text identified from the question voice to the client in response to receiving a user's question voice from the client; An auxiliary information receiving module is configured to receive auxiliary information from the client that matches the question text recognized from the question speech; The response text determination module is configured to determine the response text based on the question text and the auxiliary information; and The response feedback module is configured to provide the client with at least one of the response text or response voice, wherein the response voice corresponds to the response text. An electronic device, comprising: At least one processor; and At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 7 or 8 to 12 when executed by the at least one processor. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 7 or 8 to 12. A computer program product comprising a computer program, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 7 or 8 to 12.