JP7689787B1 - Information processing device, information processing method, and program - Google Patents

Abstract

[Problem] To improve the efficiency of calls made by AI agents.
[Solution] An information processing device 2 includes: an acquisition unit 100 that acquires speech information regarding a target person's utterance, the speech information including first utterance information regarding a first utterance and second utterance information regarding a second utterance subsequent to the first utterance; a first response determination unit 102a that determines first response information regarding a response to at least a part of the utterance by inputting a first response determination instruction including the first utterance information into a large-scale language model; a second response determination unit 102b that determines second response information regarding another response to at least a part of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information into the large-scale language model before outputting the first response information to the target person when a specified condition regarding the utterance is satisfied; and an output unit 106 that outputs the first response information when the specified condition is not satisfied, and outputs at least one of the first response information and the second response information when the specified condition is satisfied.
[Selected Figure] Figure 1

Description

The present disclosure relates to an information processing device, an information processing method, and a program.

Conventionally, technology that uses voice recognition for phone calls is known. For example, Patent Document 1 describes technology that provides a means of identifying family members or acquaintances and a means of coordinating with the police for numbers with a history of misuse in a system in which a generative AI (Artificial Intelligence) answers calls from unregistered or withheld phone numbers and documents and transmits the purpose of the call after the call ends.

Patent No. 7550335

However, the technology described in Patent Document 1 does not sufficiently improve the efficiency of calls made by AI agents. For example, there is room for further study regarding reducing delays in responses to the target person.

The present disclosure provides an information processing device, an information processing method, and a program that can improve the efficiency of calls made by AI agents.

An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires speech information related to an utterance of a target person, the speech information including first utterance information related to a first utterance and second utterance information related to a second utterance subsequent to the first utterance; a first response determination unit that determines first response information related to a response to at least a part of the utterance by inputting a first response determination instruction including the first utterance information into a large-scale language model; a second response determination unit that determines second response information related to another response to at least a part of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information into the large-scale language model before outputting the first response information to the target person when a predetermined condition related to the utterance is satisfied; and an output unit that outputs the first response information when the predetermined condition is not satisfied and outputs at least one of the first response information and the second response information when the predetermined condition is satisfied.

An information processing method according to another aspect of the present disclosure includes an information processing device acquiring speech information related to an utterance of a target person, the speech information including first utterance information related to a first utterance and second utterance information related to a second utterance subsequent to the first utterance, determining first response information related to a response to at least a part of the utterance by inputting a first response determination instruction including the first utterance information into a large-scale language model, determining second response information related to another response to at least a part of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information into the large-scale language model before outputting the first response information to the target when a predetermined condition related to the utterance is satisfied, and outputting the first response information when the predetermined condition is not satisfied, and outputting at least one of the first response information and the second response information when the predetermined condition is satisfied.

A program according to another aspect of the present disclosure causes an information processing device to execute the following: acquire speech information regarding an utterance of a target person, the speech information including first utterance information regarding a first utterance and second utterance information regarding a second utterance subsequent to the first utterance; determine first response information regarding a response to at least a portion of the utterance by inputting a first response determination instruction including the first utterance information into a large-scale language model; determine second response information regarding another response to at least a portion of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information into the large-scale language model before outputting the first response information to the target when a predetermined condition regarding the utterance is satisfied; output the first response information when the predetermined condition is not satisfied, and output at least one of the first response information and the second response information when the predetermined condition is satisfied.

This disclosure makes it possible to make calls using AI agents more efficient.

FIG. 1 is a diagram for explaining an overview of a system 1 according to an embodiment of the present invention. FIG. 2 is a diagram for explaining an example of a functional configuration of the system 1 according to the present embodiment. 10A to 10C are diagrams for explaining an example of the operation of the information processing device 2 according to the present embodiment. 10A to 10C are diagrams for explaining an example of the operation of the information processing device 2 according to the present embodiment. 10A to 10C are diagrams for explaining an example of the operation of the information processing device 2 according to the present embodiment. FIG. 11 is a diagram for explaining another example of the operation of the information processing device 2 according to the present embodiment. FIG. 11 is a diagram for explaining another example of the operation of the information processing device 2 according to the present embodiment. FIG. 11 is a diagram for explaining another example of the operation of the information processing device 2 according to the present embodiment. FIG. 2 is a diagram for explaining an example of the hardware configuration of each device in the system 1 according to the present embodiment.

1. Overview In this embodiment, a situation is assumed in which a subject uses his/her own terminal device to talk to an AI agent. That is, the subject's terminal device generates voice information by receiving the subject's speech via a voice input device (e.g., a microphone, etc.), transmits the voice information to an information processing device (e.g., a server device, etc.), and acquires response information regarding a response by the AI agent. This response information is output by voice in the terminal device. This allows the subject to experience as if he/she is talking to the AI agent via the terminal device. One of the problems that the system 1 according to this embodiment (hereinafter simply referred to as "system 1") aims to solve is to make a call with an AI agent efficient under such circumstances.

An overview of system 1 will be described with reference to FIG. 1. System 1 includes a terminal device 3, an information processing device 2, and an LLM server device 4. The terminal device 3 is a device used by the target person. The information processing device 2 is a device that executes at least a part of the processing related to making calls by an AI agent more efficient. The LLM server device 4 is a device that provides services based on a large language model (LLM: Large Language Model, hereinafter referred to as "LLM").

The terminal device 3 generates first voice information by receiving the speech of the target person via the voice input device, and transmits it to the information processing device 2 (S1). The information processing device 2 acquires the first speech information based on the first voice information. The first speech information may include text data that is a transcription of the first voice information, or may include the first voice information itself (e.g., voice data, etc.).

Next, the information processing device 2 transmits a first response determination instruction including the first utterance information to the LLM server device 4 (S2). The first response determination instruction may include an instruction for determining first response information related to a response to at least a portion of the target person's utterance. The LLM server device 4 generates first response information based on the first response determination instruction and transmits the first response information to the information processing device 2 (S3).

The information processing device 2 does not output the first response information to the target person immediately after step S3. The information processing device 2 waits for the input of additional voice information from the terminal device 3 while retaining the first response information as a provisional response. Then, the information processing device 2 outputs the first response information (not shown) when a predetermined condition related to the target person's speech is not satisfied (in one example, when no additional voice information is input and it is determined that the speech will not continue). In contrast, the case where a predetermined condition related to the target person's speech is satisfied (in one example, when additional voice information is input and it is determined that the speech will continue) will be described below.

The terminal device 3 further generates second voice information by receiving the target person's speech via the voice input device, and transmits it to the information processing device 2 (S4). The information processing device 2 acquires second speech information based on the second voice information. The second speech information may include text data transcribed from the second voice information, similar to the first speech information, or may include the second voice information itself (e.g., voice data, etc.).

Next, the information processing device 2 transmits a second response determination instruction to the LLM server device 4, the second response determination instruction including the second utterance information, the first utterance information already acquired, and/or the first response information held as a provisional response (S5). The second response determination instruction may include an instruction for determining second response information related to a response to at least a portion of the target person's utterance. The LLM server device 4 generates second response information based on the second response determination instruction and transmits the second response information to the information processing device 2 (S6). The second response information can also be said to be information obtained by updating the first response information, which is a candidate for a provisional response, based on the second utterance information corresponding to additional input by the target person.

Next, the information processing device 2 outputs second response information (S7). The information processing device 2 may output the second response information after determining that there is no additional speech from the subject.

According to the system 1, the time interval that the subject feels as latency of the response can be suppressed. The system 1 acquires first response information, which is a provisional response, based on first speech information related to the subject's speech (see S3), and then outputs the first response information if a predetermined condition related to the speech is not met, and acquires second response information based on the first speech information and/or the first response information if the predetermined condition is met (see S6). According to this configuration, in one example, the response information is continuously updated while the user's speech continues, and the most recently updated response information is output promptly when the speech ends. That is, compared to the conventional technology in which the generation of a response is started after the subject's speech ends, for example, the latency felt by the subject can be suppressed. As a result, the efficiency of calls by AI agents is improved.

In this embodiment, for the sake of convenience, terms such as "update" and "provisional" may be used, but these terms do not limit the actual processing performed by a computer.

Furthermore, in the following, when there is no particular distinction between the first voice information and the second voice information, or when they are referred to collectively, they are referred to as "voice information". Similarly, when there is no particular distinction between the first utterance information and the second utterance information, or when they are referred to collectively, they are referred to as "utterance information". Similarly, when there is no particular distinction between the first response information and the second response information, or when they are referred to collectively, they are referred to as "response information". Similarly, when there is no particular distinction between the first response determination instruction and the second response determination instruction, or when they are referred to collectively, they are referred to as "response determination instructions".

Detailed aspects of system 1 are described below with reference to Figures 2 to 9.

2 Functional Configuration The functional configuration of the system 1 of this embodiment will be described with reference to Fig. 2. The system 1 includes an information processing device 2, a terminal device 3, an LLM server device 4, and a communication network 5. The information processing device 2, the terminal device 3, and the LLM server device 4 are configured to be able to communicate with each other via the communication network 5.

2.1 Information processing device 2
The information processing device 2 executes at least a part of the processing related to improving the efficiency of calls made by an AI agent. In one embodiment, the information processing device 2 is a server device in the case where the terminal device 3 is a client device. In one embodiment, the information processing device 2 is a cloud server device. Note that the information processing device 2 may be, for example, a device including one or more virtual or physical web server devices and one or more virtual or physical database server devices.

The information processing device 2 includes a control unit 10, a memory unit 12, a network interface unit 14, and a bus 16. The control unit 10, the memory unit 12, and the network interface unit 14 are electrically connected via the bus 16.

2.1.1 Control unit 10
The control unit 10 can function as an acquisition unit 100, a determination unit 102, a judgment unit 104, and an output unit 106 by executing various programs stored in the storage unit 12, which will be described later.

2.1.1.1 Acquisition unit 100
The acquiring unit 100 acquires speech information related to an utterance of a target person. The speech information includes first utterance information related to a first utterance and second utterance information related to a second utterance subsequent to the first utterance.

In one embodiment, the speech information may include text data transcribed from the target person's speech based on the audio information received by the information processing device 2 from the terminal device 3. That is, the first speech information may include text data transcribed from a first portion of the target person's speech, and the second speech information may include text data transcribed from a second portion of the target person's speech that follows the first portion. The acquisition unit 100 may transcribe the audio information using a voice recognition program selected based on the knowledge of a person skilled in the art.

In one example, if the subject's utterance is "Um, I'd like to make a reservation for two people at 7pm tonight...oh, my name is ____," the voice information can include voice data corresponding to this utterance, the first utterance information can include text data saying "Um, I'd like to make a reservation for two people at 7pm tonight," and the second utterance information can include text data saying "oh, my name is ____."

It should be noted that the extent of the target person's speech to be the first utterance and the extent to be the second utterance may be determined based on the determination result of the determination unit 104 described later. The determination unit 104 determines whether or not a break has occurred in the target person's speech. The acquisition unit 100 may acquire first utterance information and second utterance information by treating at least a portion of the target person's speech up to the point at which it is determined that a break has occurred as the first utterance and at least a portion of the utterance after the point at which it is determined that a break has occurred as the second utterance. In the above, an example was given of a case in which it was determined that a break occurred between "Um, I'd like to make a reservation for two people at 7pm tonight" (corresponding to the first utterance information) and "Oh, my name is ____" (corresponding to the second utterance information).

In one embodiment, the speech information may include the audio information itself. That is, the first speech information may include the audio information itself of the first utterance portion of the target person's speech, and the second speech information may include the audio information itself of the second utterance portion of the target person's speech that follows the first utterance.

2.1.1.2 Determination unit 102
The determination unit 102 includes a first response determination unit 102a, a second response determination unit 102b, and a backchannel determination unit 102c.

2.1.1.2.1 First response determining unit 102a
The first response determination unit 102a determines first response information regarding a response to at least a part of the target person's utterance by inputting a first response determination instruction including first utterance information to the LLM.

In one embodiment, inputting the first response determination instruction into the LLM may include transmitting an HTTP request including the first response determination instruction to the LLM server device 4. Determining the first response information may include obtaining an HTTP response to the HTTP request.

In one embodiment, the first response determination instruction may include, in addition to the first utterance information, for example, a system prompt and a conversation history between the subject and the AI agent (i.e., information regarding the subject's utterance prior to determining the first response information, and information regarding the response thereto by the information processing device 2), etc.

In one embodiment, the first response determination unit 102a inputs a first response determination instruction to the LLM when the determination unit 104 (described later) determines that a first division has occurred in the target person's speech. In one example, when the target person's speech is "Um, I'd like to make a reservation for two people at 7 p.m. tonight...oh, my name is ____," the determination unit 104 may determine that a first division has occurred between "I'd like to make a reservation for two people at 7 p.m. tonight" and "oh, my name is ____." The first response determination unit 102a may transmit a first response determination instruction to the LLM server device 4 at the time it is determined that a first division has occurred, and determine first response information.

In one embodiment, the first response determination unit 102a may further determine information for outputting the first response information by voice when determining the first response information. The information for outputting the response information by voice may be generated based on a voice generation program that can be arbitrarily selected by a person skilled in the art. Note that "when determining the response information" may be immediately before determining the response information, may be in parallel with the process of determining the response information, may be performed integrally and/or consecutively with the process of determining the response information, or may be immediately after the process of determining the response information.

In one embodiment, the storage unit 12 stores information for outputting each of one or more texts by voice, and when the first response information matches at least one of the one or more texts, determining information for outputting the first response information by voice includes acquiring information for outputting the matching text by voice from the storage unit 12, and when the first response information does not match any of the one or more texts, determining information for outputting the first response information by voice includes generating information for outputting the first response information by voice based on a predetermined voice generation program. That is, when the response information typically includes "common" words and/or phrases, the first response determination unit 102a may determine the voice of the words and/or phrases prepared in advance. In one example, when the storage unit 12 stores information for outputting the text "We have accepted your reservation" by voice, when the first response information includes the text "We have accepted your reservation", information for outputting that part of the first response information by voice is acquired from the storage unit 12. In another example, when the storage unit 12 stores information for outputting the text "Your reservation has been accepted" by voice as described above, if the first response information does not include the text "Your reservation has been accepted" (i.e., if information for outputting the corresponding voice cannot be obtained from the storage unit 12), the first response determination unit 102a generates information for outputting "Your reservation has been accepted" by voice using a predetermined voice generation program. With this configuration, a portion of the text included in the first response information can be output by a voice prepared in advance, thereby suppressing the latency felt by the subject.

2.1.1.2.2 Second response determining unit 102b
When a predetermined condition regarding the target's speech is satisfied (in one example, when it is determined that the target's speech will continue), the second response determination unit 102b determines second response information regarding another response to at least a part of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information to the LLM before outputting the first response information to the target.

In one embodiment, inputting the second response determination instruction to the LLM may include transmitting an HTTP request including the second response determination instruction to the LLM server device 4, similar to inputting the first response determination instruction to the LLM. Determining the second response information may include obtaining an HTTP response to the HTTP request.

In one embodiment, the second response determination instruction, like the first response determination instruction, may include, in addition to the first response information and the second utterance information, for example, a system prompt and a conversation history between the subject and the AI agent (i.e., information about the subject's utterance prior to determining the second response information, and information about the response thereto by the information processing device 2), etc.

As illustrated with reference to FIG. 1, the first response information may be a tentative response. If a predetermined condition regarding the target person's speech is satisfied, the second response determination unit 102b may determine the second response information before outputting the first response information to the target person (in one example, without outputting such a tentative response). In one embodiment, the period before the first response information is output to the target person may be the period before the first response information is output by voice or the like on the terminal device 3.

In one embodiment, when determining the second response information, the second response determination unit 102b may further determine information for outputting the second response information by voice.

In one embodiment, the storage unit 12 stores information for outputting each of the one or more texts by voice, and when the second response information is consistent with at least one of the one or more texts, determining information for outputting the second response information by voice includes obtaining information for outputting the matching text by voice from the storage unit 12, and when the second response information is not consistent with any of the one or more texts, determining information for outputting the second response information by voice includes generating information for outputting the second response information by voice based on a predetermined voice generation program.

In one embodiment, the second response determination unit 102b inputs a second response determination instruction to the LLM when the determination unit 104 determines that a second division has occurred in the utterance. In one example, when the target person's utterance is "Um, I'd like to make a reservation for two people at 7 p.m. tonight...oh, my name is ____," the determination unit 104 may determine that a first division has occurred between "I'd like to make a reservation for two people at 7 p.m. tonight" and "oh, my name is ____," as described above, and may further determine that a second division has occurred after "oh, my name is ____." The second response determination unit 102b may transmit a second response determination instruction to the LLM server device 4 at the time when it is determined that a second division has occurred, and determine second response information. Note that if a predetermined condition regarding the target person's speech is not satisfied after the first speech information is acquired (in one example, if it is determined that the target person's speech will not continue), the first response information may be output, and the second response determination unit 102b does not need to determine the second response information.

2.1.1.2.3 Backchannel determination unit 102c
The backchannel determining unit 102c determines backchannel information related to the backchannel corresponding to the first utterance information when determining the first response information. Note that "when determining the first response information" may be before the first response information is determined, may be during the process of determining the first response information, or may be after the first response information is determined.

In one embodiment, the backchannel determination unit 102c may determine the backchannel information based on whether the first utterance information includes information about a positive utterance by the target person or information about a negative utterance. In one example, if the first utterance information includes information about a positive utterance such as "Yes, it's possible," the backchannel determination unit 102c may determine the backchannel information including text data "Thank you." In another example, if the first utterance information includes information about a negative utterance such as "I'm sorry," the backchannel determination unit 102c may determine the backchannel information including text data "I understand." In this case, the backchannel determination unit 102c may refer to, for example, a list of words that may be included in a positive utterance and a list of words that may be included in a negative utterance, and determine the backchannel information corresponding to either list.

In one embodiment, the backchannel determination unit 102c may determine the backchannel information based on the format of the question included in the first utterance information. In one example, when the first utterance information includes a question that can be answered with YES/NO, such as "May I do ~?", or a question that can be answered in a closed format, such as "Which should I choose, A or B?", the backchannel determination unit 102c may determine the backchannel information including the text data "Thank you for checking." In another example, when the first utterance information includes a question that can be answered in an open format, such as "Please say it again," the backchannel determination unit 102c may determine the backchannel information including the text data "I understand." In this case, the backchannel determination unit 102c may determine the format of the question included in the first utterance information based on a natural language processing algorithm selected based on the knowledge of a person skilled in the art, and determine the backchannel information based on the determination result.

In one embodiment, when determining the backchannel information, the backchannel determination unit 102c may further determine information for outputting the backchannel information by voice.

In one embodiment, the storage unit 12 stores information for outputting each of the one or more texts by voice, and when the backchannel information matches at least one of the one or more texts, determining information for outputting the backchannel information by voice includes obtaining information for outputting the matching text by voice from the storage unit 12, and when the backchannel information does not match any of the one or more texts, determining information for outputting the backchannel information by voice includes generating information for outputting the backchannel information by voice based on a predetermined voice generation program.

In one embodiment, the backchannel determination unit 102c estimates the emotion of the target's speech based on the first speech information and/or the voice of the speech corresponding to the first speech information, and determines the backchannel information based on the estimated emotion. As an example, assume that the speech corresponding to the first speech information is a voice saying "Why can't you do this?" In this case, if it is estimated that the target is angry based on the voice, the backchannel determination unit 102c may determine the backchannel saying "I'm sorry." In contrast, if it is estimated that the target is not angry based on the voice, the backchannel determination unit 102c may determine the backchannel saying "Thank you for your question." With this configuration, a backchannel that is more natural for the target can be output. Note that the estimation of the emotion of the target's speech based on the first speech information and/or the voice of the speech corresponding to the first speech information can be performed based on an emotion estimation algorithm that can be arbitrarily selected by a person skilled in the art.

In one embodiment, the backchannel determination unit 102c determines the backchannel information by inputting a backchannel determination instruction including the first speech information to the LLM.

In one embodiment, inputting a backchannel determination instruction to the LLM may include transmitting an HTTP request including the backchannel determination instruction to the LLM server device 4, similar to inputting a response determination instruction to the LLM. Determining backchannel information may include obtaining an HTTP response to the HTTP request.

In one embodiment, the backchannel decision instruction, like the response decision instruction, may include, in addition to the first speech information, for example, a system prompt.

In one embodiment, when determining the second response information, the backchannel determination unit 102c determines other backchannel information related to another backchannel corresponding to the second utterance information. At this time, the backchannel determination unit 102c may determine the other backchannel information by inputting a backchannel determination instruction including the second utterance information to the LLM.

2.1.1.3 Determination unit 104
The determination unit 104 determines whether a first division has occurred in the utterance. The first utterance information includes information on the part of the utterance of the target person up to the first division.

In one embodiment, the determination unit 104 further determines whether a second division occurs after the first division of the utterance. The second utterance information includes information about the portion of the target person's utterance up to the second division.

In one embodiment, the determination unit 104 determines whether a break has occurred in the speech based on the audio continuity of the target's speech. In one example, the determination unit 104 sequentially determines whether the target is speaking based on each of a plurality of audio chunks (e.g., data obtained by dividing the audio information every 0.1 seconds) included in the audio information, and may determine that a break has occurred when it is determined that the target is not speaking in a predetermined number of consecutive audio chunks.

In one embodiment, the determination unit 104 determines whether a break has occurred in the speech based on the semantic continuity of the target person's speech. In one example, when the speech information includes text data obtained by transcribing audio information, the determination unit 104 may divide the text data into a portion related to a first theme and a portion related to a second theme, and determine that a break has occurred between them. For example, when the speech information includes text data such as "I would like to make a reservation for four people at 7 p.m. tomorrow night. Can I add all-you-can-drink to the course?", the determination unit 104 may divide the text data into a portion related to the first theme of "I would like to make a reservation for four people at 7 p.m. tomorrow night" (in this example, a question about whether a reservation can be made) and a portion related to the second theme of "Can I add all-you-can-drink to the course?" (in this example, a question about the content of the course), and determine that a break has occurred between them.

In one embodiment, the determination unit 104 further determines whether a predetermined time for outputting a response has elapsed after at least one of the first and second segments of the utterance. In one embodiment, the predetermined time is determined based on the backchannel information. In one example, the predetermined time is determined based on the playback time when the backchannel information is output by voice. For example, if the backchannel information includes text data such as "Understood," and it takes 0.8 seconds to output this by voice, the predetermined time may be 0.8 seconds (or a number of seconds obtained by adding about 0.1 seconds to this as a buffer).

2.1.1.4 Output Unit 106
The output unit 106 outputs the first response information when a predetermined condition regarding the target person's speech is not satisfied (for example, when the speech continues), and outputs at least one of the first response information and the second response information when the predetermined condition is satisfied (for example, when the speech ends). Outputting the response information includes transmitting information for outputting the response information to the terminal device 3.

In one embodiment, the output unit 106 controls the subject's terminal device 3 to output the response information by voice. At this time, the output unit 106 may transmit information for outputting a voice corresponding to the response information to the terminal device 3. The information for outputting a voice corresponding to the response information may be generated based on a voice generation program selected based on the knowledge of a person skilled in the art.

In one embodiment, the output unit 106 further outputs backchannel information. In one embodiment, the output unit 106 controls the terminal device 3 of the subject to output the backchannel information by voice. At this time, the output unit 106 may transmit information for outputting a voice corresponding to the backchannel information to the terminal device 3. The information for outputting a voice corresponding to the backchannel information may be generated based on a voice generation program selected based on the knowledge of a person skilled in the art.

In one embodiment, the output unit 106 outputs at least one of the first response information and the second response information after outputting the backchannel information. Outputting at least one of the first response information and the second response information after outputting the backchannel information includes controlling the terminal device 3 so that the backchannel information is output by voice on the terminal device 3 after the response information is output by voice on the terminal device 3.

In one embodiment, the output unit 106 outputs the second response information when the determination unit 104 determines that a predetermined time has elapsed after the second division.

2.1.2 Storage unit 12
The storage unit 12 stores various types of information for the operation of the information processing device 2. In one embodiment, the storage unit 12 stores a program executed by the control unit 10.

2.1.3 Network interface unit 14
The network interface unit 14 realizes communication with other devices via the communication network 5 .

2.2 Terminal Device The terminal device 3 is a communication device used by the subject. The terminal device 3 is, for example, a smartphone, a personal computer, a tablet terminal, a wearable terminal, etc. The terminal device 3 includes an input interface, an output interface, and a communication interface.

The input interface is an interface through which the terminal device 3 receives input from the subject. The input interface may be a touch panel, a microphone, a camera, a keyboard, a mouse, etc.

The output interface is an interface for transmitting information to the target person through images, audio, etc. Output interfaces include a display (which may also serve as a touch panel) and a speaker, etc.

The communication interface is an interface for realizing communication with other devices via the communication network 5. The communication interface may be a wireless communication interface or a wired communication interface.

The terminal device 3 may be able to access the services provided by the information processing device 2, for example, via a web browser, or may be able to access the services by installing dedicated software.

2.3 LLM server device 4
The LLM server device 4 is a device that provides services using the LLM. The LLM may be a deep learning model that has hundreds of millions of parameters and has learned data related to natural languages of hundreds of GB or more. The LLM is, for example, gpt-4o. In one example, the LLM server device 4 provides a service using the LLM via an API (Application Programming Interface).

In one embodiment, the LLM server device 4 accepts input of instructions (which may also be called prompts) from other devices and returns a response to the other devices according to the instructions. In one example, both the instructions and the response are text.

2.4 Communication Network 5
The communication network 5 realizes communication between each device included in the system 1. The communication network 5 realizes communication between each device based on, for example, the TCP/IP protocol.

3. Operation An example of the operation of the system 1 will now be described with reference to FIGS.

3 to 5, the operation of the system 1 according to the first embodiment will be described. In the first embodiment, a basic aspect of the system 1 will be exemplarily described.

3.1.1 Flowchart Fig. 3 is a flowchart for explaining the operation of the information processing device 2 according to the first embodiment. The flowchart in Fig. 3 describes steps S100 to S112, but in parallel with these processes, the information processing device 2 continuously and sequentially executes acquisition of voice chunks, determination of whether or not speech is being generated based on the voice chunks, and transcription of voice information. This process is an example of the acquisition unit 100 acquiring speech information related to the speech of the subject.

The information processing device 2 determines whether or not a break has occurred in the speech of the subject at that time (S100). In one example, the information processing device 2 performs this determination based on whether or not a predetermined number of audio chunks determined to be no speech occur consecutively at that time. This process is an example of the determination unit 104 determining whether or not a first break has occurred in the speech of the subject.

If there is no break in the speech (S100 NO), the information processing device 2 waits until there is a break in the speech (S102). As described above, even while waiting, the information processing device 2 continuously and sequentially acquires voice chunks, determines whether or not speech is being generated based on the voice chunks, and transcribes the voice information.

On the other hand, if a break has occurred in the speech (S100 YES), the information processing device 2 executes a response determination process. The response determination process at this point may include the following processes (1) to (3).
(1) Sending a first response determination instruction to the LLM server device 4, including first speech information (in one example, the transcribed text data obtained up to that point).
(2) Obtaining first response information from the LLM server device 4.
(3) Generating information for outputting a voice corresponding to the first response information.

The transcribed text data obtained up to this point is an example of information about the portion of the subject's speech up to the first segment.

The response determination process at this point is also an example of the first response determination unit 102a determining first response information regarding a response to at least a portion of the target person's utterance by inputting a first response determination instruction including first utterance information to the LLM.

The response determination process at this point is an example of the first response determination unit 102a inputting a first response determination instruction to the LLM when the determination unit 104 determines that a first break has occurred in the target person's speech.

Next, the information processing device 2 determines whether or not the speech has ended (S110). In one example, the information processing device 2 performs this determination based on whether or not a predetermined number of speech chunks that have been determined to be unspoken up until that point continue from the point in step S100.

When it is determined that the speech has ended (S110 YES), the information processing device 2 may output first response information, as described below. On the other hand, when it is determined that the speech has not ended, that is, when the target person's speech is continuing (S110 NO), the information processing device 2 waits until the next break in the speech occurs (S102). Then, when a break in the speech occurs in the subsequent loop (S100 YES), the information processing device 2 executes the response determination process again. The response determination process at this point may include the following processes (1) to (3).
(1) Sending a second response determination instruction to the LLM server device 4, the second response determination instruction including the first utterance information and/or the first response information and the second utterance information (in one example, the transcribed text data obtained up to that point).
(2) Obtaining second response information from the LLM server device 4.
(3) Generating information for outputting a voice corresponding to the second response information.

The second step S100 is an example of the determination unit 104 further determining whether or not a second segment has occurred after a first segment of the target person's speech.

The transcribed text data obtained up to this point is an example of information about the subject's speech up to the second segment.

This response determination process is also an example of the second response determination unit 102b determining second response information related to another response to at least a portion of the target's utterance when a predetermined condition related to the target's utterance is satisfied, by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information to the LLM before outputting the first response information to the target.

This response determination process is also an example of the second response determination unit 102b inputting a second response determination instruction to the LLM when the determination unit 104 determines that a second break has occurred in the target person's speech.

The information processing device 2 repeatedly executes the loop of steps S100 to S110 while the target person continues speaking. Then, in step S104, the response information is updated each time by the response determination process.

Step S110 immediately following step S100 for the second time is an example in which the determination unit 104 further determines whether or not a predetermined time for outputting a response has elapsed after the second division of the subject's speech.

After that, when it is determined that the speech has ended (S110 YES), the information processing device 2 transmits information for outputting a voice corresponding to the response information determined in the most recent response determination process of step S104 to the terminal device 3 (S112). The "response information determined in the most recent response determination process of step S104" may be, for example, the response information determined in the first response determination process if NO has never been determined in step S110, or the response information determined in the second response determination process if NO has only been determined once in step S110.

Transmitting information to the terminal device 3 for outputting a voice corresponding to the response information is an example of the output unit 106 outputting at least one of the first response information and the second response information.

In addition, outputting response information when it is determined that the speech has ended is an example of the output unit 106 outputting second response information when the determination unit 104 determines that a predetermined time has elapsed after the second division.

3.1.2 Table (when outputting first response information)
4 is a diagram for explaining the operation of the information processing device 2 according to the first embodiment from a different perspective. In the following explanation, "Th1" is a value corresponding to the length of time for determining that a break has occurred in an utterance, and "Th2" is a value corresponding to the length of time for determining that an utterance has ended.

The information processing device 2 sequentially acquires the voice chunks (voice chunks v ₁ to v _N+Th1+Th2+1 ) at each of the times t ₁ to t _N+Th1+Th2+1 , and executes vocalization determination for each of the voice chunks.

The information processing device 2 determines that the subject is speaking at time points t ₁ to t _N based on each of the N voice chunks v ₁ to v _N , and performs transcription. The text data obtained by transcribing the voice chunks v ₁ to v _N is an example of first utterance information related to the first utterance. This process corresponds to the process described in steps S100 NO to S102 in FIG. 3.

Next, the information processing device 2 determines that the target person does not speak from time t _N+1 to t N+Th1 based on Th1 voice chunks v _N+1 to v _{N+Th1 .} Then, the information processing device 2 determines that there is a break in the speech based on the fact that the target person did not speak in this time period, and executes a response determination process. In this way, the information processing device 2 determines the first response information. These processes correspond to the processes described in steps S100 YES to S104 in FIG. 3. The response determination process is continued from time t _N+Th1+1 to time t _N+Th1+Th2, which is a time period of Th2.

Next, the information processing device 2 determines that the target person has finished speaking based on the fact that the target person has not spoken during the Th2 time periods from time t _N+Th1+1 to time t _N+Th1+Th2 , and outputs the first response information. These processes correspond to the processes described in steps S110 YES to S112 in FIG. 3.

3.1.3 Table (when outputting second response information)
5 is a diagram for further explaining the operation of the information processing device 2 according to the first embodiment from a different perspective. The information processing device 2 sequentially acquires voice chunks (voice chunks v ₁ to v _M+Th1+Th2+1 ) at each of the times t ₁ to t _M+Th1+Th2+1 , and executes utterance determination for each voice chunk. Note that the operation from time t ₁ to t _N+Th1 is the same as the example in FIG. 4, and therefore will not be described below.

Fig. 4 describes an example in which it is determined that the target person is not speaking at time points tN+ _{Th1 +1} to tN _+Th1+Th2 . Fig. 5 describes an example in which the target person's speech is detected at time point _{tN+Th1+ k+1 between time points tN+Th1+1} to tN _{+Th1+Th2 (i.e., an example in which the target person resumes speaking at time point tN+Th1+k+ 1 after the information processing device 2 determines that a break has occurred in the speech at time point tN+Th1 +1 and before the first response information is output at time point tN+Th1+Th2} ).

The information processing device 2 determines that the subject is speaking at time points tN _+Th1+k+1 to _tM based on the voice chunks vN _+Th1+k+1 to _vM , and performs transcription. The text data obtained by transcribing the voice chunks vN _+Th1+k+1 to _vM is an example of second utterance information related to a second utterance that follows the first utterance. This process corresponds to the process described in steps S100 NO to S102 in FIG. 3.

Next, the information processing device 2 determines that the target person does not speak from time t _M+1 to t M+Th1 based on Th1 voice chunks v _M+1 to v _{M+Th1 .} Then, the information processing device 2 determines that a break has occurred in the speech based on the fact that the target person did not speak in this time period, and executes a response determination process. In this way, the information processing device 2 determines the second response information. These processes correspond to the processes described in steps S100 YES to S104 in FIG. 3. The response determination process is continued from time t _M+Th1+1 to time t _M+Th1+Th2, which is a time period of Th2.

Next, the information processing device 2 determines that the target person has finished speaking based on the fact that the target person has not spoken during the Th2 time periods from time t _+Th1+1 to time t _+Th1+Th2 , and outputs the second response information. These processes correspond to the processes described in steps S110 YES to S112 in FIG. 3.

6 to 8, the operation of the system 1 according to the second embodiment will be described. In the second embodiment, a mode of the system 1 when the information processing device 2 outputs backchannel information will be described as an example.

6 is a flowchart for explaining the operation of the information processing device 2 according to the sixth embodiment. The flowchart in Fig. 6 describes steps S200 to S212, but in parallel with these processes, the information processing device 2 continuously and sequentially acquires audio chunks, determines whether or not speech is being generated based on the audio chunks, and transcribes audio information.

Note that steps S200 to S204 and steps S210 to S212 in FIG. 6 may correspond to steps S100 to S104 and steps S110 to S112 in FIG. 3, respectively, so the following description will focus on steps S206 to S208 in FIG. 6.

While executing the response determination process (S204), the information processing device 2 may further execute a backchannel determination process (S206). The backchannel determination process may include the following processes (1) to (3).
(1) Transmitting a backchannel decision instruction to the LLM server device 4, including the first speech information (in one example, the transcribed text data obtained up to that point).
(2) Acquiring interjection information from the LLM server device 4.
(3) Generating information for outputting a voice corresponding to the backchannel information.
This backchannel determination process is an example of the backchannel determination unit 102c determining backchannel information related to the first utterance information when determining the first response information. Note that the backchannel determination process is not limited to only when determining the first response information, but may be executed for each loop of steps S200 to S210.

Next, the information processing device 2 outputs backchannel information (S208). That is, while executing the loop of steps S200 to S210 (processing repeated while the subject continues speaking), the information processing device 2 may output the backchannel information before outputting the response information in step S212. This allows the subject to experience a more natural conversation with the AI agent. Note that outputting the response information in step S212 after outputting the backchannel information in step S208 is an example of the output unit 106 outputting second response information after outputting the backchannel information.

In FIG. 6, step S204 and steps S206 to S208 are shown in series for convenience, but these processes may be executed in parallel. That is, the information processing device 2 may execute the backchannel determination process and output the backchannel information while executing the response determination process. With this configuration, the backchannel information is first output to the subject while the response determination process is being executed, so that the latency experienced by the subject can be further reduced.

3.2.2 Table (when outputting first response information)
FIG. 7 is a diagram for explaining the operation of the information processing device 2 according to the second embodiment from a different perspective. In FIG. 4, an example was explained in which the information processing device 2 executes a response determination process from time t N+Th1 ₊₁ to time t _N+Th1+ Th2, and does not output information during that time. In contrast, in the example of FIG. 7, the information processing device 2 determines that a break has occurred in the speech based on the fact that the target person did not speak from time t _N+1 to t _N+Th1 , and executes a response determination process and a backchannel determination process. As a result, the information processing device 2 determines the first response information and backchannel information, and outputs the backchannel information. These processes correspond to the processes explained in steps S200 YES to S208 of FIG. 6.

3.2.3 Table (when outputting second response information)
Fig. 8 is a diagram for further explaining the operation of the information processing device 2 according to the second embodiment from a different perspective. Fig. 5 has explained an example in which the information processing device 2 executes a response determination process from time tN+ _Th1+1 to time tN _+Th1+k and from time tM _+Th1+1 to time tM _+Th1+Th2 , and does not output information during those periods.

8, the information processing device 2 determines that a break has occurred in the speech based on the absence of speech from the target person at time points t _N+1 to t _N+Th1 , and executes a response determination process and a backchannel determination process. The information processing device 2 further outputs backchannel information at time points t _N+1 to t _N+Th1+k .

Furthermore, the information processing device 2 determines that a break has occurred in the speech based on the fact that the target person did not speak from time t _M+1 to t _M+Th1 , and executes a response determination process and a backchannel determination process. The information processing device 2 further outputs backchannel information from time t _M+1 to t _M+Th1+Th2 .

3.3 Specific Examples In the following, specific examples of the first response determination instruction, the first response information, the second response determination instruction, and the second response information will be described assuming that the utterance by the target person is "Um, I'd like to make a reservation for two people at 7pm tonight... Oh, my name is XX." Note that the first utterance information includes text data of "Um, I'd like to make a reservation for two people at 7pm tonight," and the second utterance information includes text data of "Oh, my name is XX."

In one example, the first response determination instruction may include the following text data (1) to (3).
(1) System prompt: "You are an intelligent AI agent who takes reservations at restaurants."
(2) Conversation history: "Subject: 'Hello?' → AI agent: 'Hello, this is XX (store name)'"
(3) Instructions including first utterance information: "The subject recently said, 'Um, I'd like to make a reservation for two people at 7pm tonight.' Please create a response to this."

In response to the first response determination instruction in this example, the LLM server device 4 may generate first response information including text data such as "This is a reservation for two people at 7 p.m. tonight," and transmit this to the information processing device 2.

In one example, the second response determination instruction may include the following text data (1) to (4).
(1) System prompt: "You are an intelligent AI agent who takes reservations at restaurants."
(2) Conversation history: "Subject: 'Hello?' → AI agent: 'Hello, this is XX (store name)' → Subject: 'Um, I'd like to make a reservation for two people at 7pm tonight.'"
(3) First response: "Your reservation is for two people at 7 p.m. tonight."
(4) Instructions including second utterance information: "The subject additionally uttered, 'Oh, my name is ____.' Please create a response based on the first response information."

In response to the second response determination instruction in this example, the LLM server device 4 may generate second response information including text data such as "This is a reservation for two people at 7pm tonight. Is your name Mr./Ms. XX correct?" and transmit it to the information processing device 2. Note that "creating a response based on the first response information" may mean that the text data included in the first response information is also included in the second response information, or that text data corresponding to the continuation of the text data included in the first response information is included in the second response information, or that the second response information is determined by referring to the first response information but not being bound by it.

9, an example of a hardware configuration in which the devices included in the above-described system 1 are realized by a computer 70 will be described. Note that the functions of each device can also be realized by dividing them into multiple devices.

As shown in FIG. 9, the computer 70 includes a processor 700, a storage device 702, an input I/F 704, a data I/F 706, a communication I/F 708, and a display device 710.

The processor 700 controls various processes in the computer 70 by executing programs stored in the storage device 702. For example, each functional unit of the control unit 10 of the information processing device 2 can be realized by the processor 700 executing a program stored in the storage device 702.

The storage device 702 is a storage medium such as a RAM (Random Access Memory). The RAM temporarily stores the program code of the program executed by the processor 700 and data required when the program is executed.

The storage device 702 may also be a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. The storage device 702 stores an operating system and various programs for implementing the above-mentioned configurations. The storage medium storing the various programs may be a non-transitory computer readable medium. In addition, the storage device 702 may store a table for registering various information and a DB for managing the table. Such programs and data are loaded into the storage device 702 as necessary and referenced by the processor 700.

The input I/F 704 is a device for receiving input from the subject. Specific examples of the input I/F 704 include a camera, a button, a microphone, a keyboard, a mouse, a touch panel, various sensors, a wearable device, and the like. The input I/F 704 may be connected to the computer 70 via an interface such as a Universal Serial Bus (USB).

The data I/F 706 is a device for inputting data from outside the computer 70. A specific example of the data I/F 706 is a drive device for reading data stored in various storage media. The data I/F 706 may be provided outside the computer 70. In that case, the data I/F 706 is connected to the computer 70 via an interface such as a USB.

The communication I/F 708 is a device for performing data communication via the communication network 5, either wired or wirelessly, with devices external to the computer 70. The communication I/F 708 may be provided external to the computer 70. In that case, the communication I/F 708 is connected to the computer 70 via an interface such as a USB.

The display device 710 is a device for displaying various types of information. Specific examples of the display device 710 include a liquid crystal display, an organic EL (Electro-Luminescence) display, a display of a wearable device, and the like. The display device 710 may be provided outside the computer 70. In that case, the display device 710 is connected to the computer 70 via, for example, a display cable. In addition, when a touch panel is used as the input I/F 704, the display device 710 can be configured as an integral part of the input I/F 704.

The components of the device included in the system 1 described in the above embodiment are assumed to be implemented in cooperation with other hardware by the processor 700 executing a program stored in the storage device 702. In other words, these components are assumed to be software or firmware, and also corresponding hardware, and in both concepts, they are also described as "functions," "means," "parts," "processing circuits," "units," or "modules," etc., and can be interpreted as each of these terms.

5. Modifications The above-described embodiments are intended to facilitate understanding of the present disclosure, and are not intended to limit and interpret the present disclosure. The configurations that the embodiments may have are not limited to those exemplified, and may be modified as appropriate. In addition, the configurations shown in different embodiments may be partially substituted or combined with each other.

The matters described in the above embodiment with the prefixes "first" and "second" can be understood as extending the relationship between "first" to "Nth" (where N is a natural number) based on the knowledge of those skilled in the art.

In one example, the acquisition unit 100 acquires speech information related to the target person's speech, and the acquired speech information may include first speech information related to a first utterance through Nth speech information related to an Nth utterance.

In one example, for natural numbers n = 2 to N, when a predetermined condition related to the utterance is satisfied, the nth response determination unit may determine the nth response information related to another response to at least a portion of the utterance by inputting an nth response determination instruction including at least one of the first utterance information to the nth-1st utterance information and the first response information to the nth-1st response information, and the nth utterance information, into the large-scale language model before outputting the nth-1st response information to the subject.

In one example, for natural numbers n = 2 to N, the output unit 104 may output at least one of the first response information to the n-1th response information when it is determined that the subject's speech has ended after the n-1th response information has been determined and before the nth response information has been determined, and may output at least one of the first response information to the nth response information when it is determined that the subject's speech has ended after the nth response information has been determined.

In the above embodiment, the LLM has been described as being hosted on the LLM server device 4, but this is not limited to the above. The LLM may be hosted on the information processing device 2 or on the terminal device 3.

In the above embodiment, the information processing device 2 is described as acquiring voice information from the terminal device 3 and executing a process for making calls by an AI agent more efficient based on the voice information, but this is not limited to the above. At least some of the functions described as being provided by the information processing device 2 in the above embodiment may be provided by the terminal device 3.

In the above embodiment, the information processing device 2 is described as communicating with the terminal device 3, but this is not limited to the above. There may be a predetermined intermediate server between the information processing device 2 and the terminal device 3.

In the above embodiment, an example has been described in which, when the backchannel information and the second response information are determined, if the second response information is output, the backchannel information is also output, but this is not limiting. For example, when the backchannel determination process and the response determination process related to the determination of the second response information are executed in parallel, whether or not to output the backchannel information can be determined based on the timing at which the response determination process is completed. In one example, if the response determination process is completed before the backchannel determination process is completed, the information processing device 2 can output the second response information without outputting the backchannel information. In contrast, if the response determination process is completed after the backchannel determination process is completed, the information processing device 2 can output the backchannel information during the execution of the response determination process, and then output the second response information. With this configuration, it is possible to avoid a situation in which the backchannel information is output first even though the response information has already been determined, and as a result, it may be possible to output the response information to the target person early.

6 Supplementary Note The wording in this embodiment can be understood as follows to the extent that no contradiction occurs.

In this embodiment, "executing a predetermined process based on the specified information" may mean any one of executing the predetermined process based on at least a part of the specified information, executing the predetermined process based on at least the specified information, and executing the predetermined process probabilistically based on the specified information. In other words, "executing a predetermined process based on the specified information" is not limited to executing the predetermined process based only on the specified information.

In this embodiment, "executing another process based on a specified process" may mean any of the following: executing the other process after the specified process is executed; executing the specified process and the other process consecutively; executing the other process based on information determined by the specified process; executing the other process on the condition that the specified process has been executed; and executing the other process by means of the specified process. Note that "executing another process by a specified process" may be understood in the same way as "executing another process based on a specified process".

In this embodiment, "specific information includes other information" may mean either that at least a portion of the specific information is the other information, or that the other information can be obtained based on the specific information.

In this embodiment, "a specified process includes another process" may mean either that at least a part of the specified process is the other process (i.e., that the other process is performed in the process of obtaining the result of the specified process) or that one aspect of the specified process is the other process.

In this embodiment, "a specific target corresponds to another target" may mean that the specific target and the other target are in a one-to-one relationship, that the other target is included in a specific set identified based on the specific target, or that the other target can be identified based on the specific target. Note that "a specific target corresponds to another target" is not limited to being managed, for example, on a database. In addition, "a specific target is associated with another target" may be understood in the same way as "a specific target corresponds to another target".

In this embodiment, "obtaining information" includes making the information processable in the control unit 10. "Obtaining information" may mean, for example, receiving the information from another device, obtaining the information through a specified process, reading the information from the storage unit 12, etc.

In this embodiment, "generating information" may mean either making the information obtained by the specified processing process processable in the control unit 10 or storing the information obtained by the specified processing in the storage unit 12.

In this embodiment, "determining information" may mean either selecting at least one piece of information from one or more pieces of information, or generating new information.

In this embodiment, "outputting information" may mean either transmitting the information to another device or outputting the information as audio or video.

7 Configuration Examples The present disclosure includes the following techniques.

[Appendix 1]
an acquisition unit that acquires speech information regarding an utterance of a target person, the speech information including first utterance information regarding a first utterance and second utterance information regarding a second utterance subsequent to the first utterance; a first response determination unit that determines first response information regarding a response to at least a part of the utterance by inputting a first response determination instruction including the first utterance information into a large-scale language model; a second response determination unit that determines second response information regarding another response to at least a part of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information into the large-scale language model before outputting the first response information to the target person when a predetermined condition regarding the utterance is satisfied; and an output unit that outputs the first response information when the predetermined condition is not satisfied, and outputs at least one of the first response information and the second response information when the predetermined condition is satisfied.

[Appendix 2]
a backchannel determining unit that determines backchannel information related to a backchannel corresponding to the first utterance information when determining the first response information, and the output unit further outputs the backchannel information.
2. The information processing device 2 according to claim 1.

[Appendix 3]
An information processing device 2 described in Appendix 2, in which the output unit 106 outputs interjection information and then first response information when a specified condition is not satisfied, and outputs the interjection information and then at least one of the first response information and the second response information when a specified condition is satisfied.

[Appendix 4]
The information processing device 2 according to claim 2 or 3, wherein the backchannel determining unit 102c determines the backchannel information by inputting a backchannel determination instruction including the first utterance information into a large-scale language model.

[Appendix 5]
The information processing device 2 described in Appendix 1 further includes a judgment unit 104 that judges whether a first division has occurred in the utterance, the first utterance information includes information regarding the part of the utterance up to the first division, and the first response determination unit 102a inputs a first response determination instruction to the large-scale language model when the judgment unit 104 judges that the first division has occurred in the utterance.

[Appendix 6]
The information processing device 2 described in Appendix 5, wherein the determination unit 104 further determines whether or not a second division has occurred after the first division of the utterance, the second utterance information includes information relating to the part of the utterance up to the second division, and the second response determination unit 102b inputs a second response determination instruction to the large-scale language model when the determination unit 104 determines that a second division has occurred in the utterance.

[Appendix 7]
The information processing device 2 described in Appendix 6, wherein the determination unit 104 further determines whether a predetermined time for outputting a response has elapsed after the second division of the utterance, and the output unit 106 outputs at least one of the first response information and the second response information when it is determined that the predetermined time has elapsed after the second division.

[Appendix 8]
An information processing device 2 as described in Appendix 7, further comprising an interjection determination unit 102c that determines interjection information regarding an interjection corresponding to the first utterance information when determining the first response information, wherein the output unit 106 further outputs the interjection information, and the specified time is determined based on the interjection information.

[Appendix 9]
The information processing device 2 described in any one of Appendices 1 to 8, wherein the output unit 106 controls the subject's terminal device to output the first response information by voice when a specified condition is not met, and controls the subject's terminal device to output at least one of the first response information and the second response information by voice when a specified condition is met.

[Appendix 10]
The information processing device 2 described in Appendix 9, wherein the first response determination unit 102a, when determining the first response information, further determines information for outputting the first response information by voice, and the second response determination unit 102b, when determining the second response information, further determines information for outputting the second response information by voice.

[Appendix 11]
and a storage unit (12) configured to store information for audio output of each of the one or more texts, wherein, when the first response information matches at least one of the one or more texts, determining information for audio output of the first response information includes acquiring information for audio output of the matching text from the storage unit, and when the first response information does not match any of the one or more texts, determining information for audio output of the first response information includes generating information for audio output of the first response information based on a predetermined voice generation program, when the second response information matches at least one of the one or more texts, determining information for audio output of the second response information includes acquiring information for audio output of the matching text from the storage unit, and when the second response information does not match any of the one or more texts, determining information for audio output of the second response information includes generating information for audio output of the second response information based on a predetermined voice generation program.

[Appendix 12]
an information processing method in which an information processing device 2 acquires speech information regarding an utterance of a target person, the speech information including first utterance information regarding a first utterance and second utterance information regarding a second utterance subsequent to the first utterance; determines first response information regarding a response to at least a portion of the utterance by inputting a first response determination instruction including the first utterance information into a large-scale language model; determines second response information regarding another response to at least a portion of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information into the large-scale language model before outputting the first response information to the target, when a predetermined condition regarding the utterance is satisfied; outputs the first response information when the predetermined condition is not satisfied, and outputs at least one of the first response information and the second response information when the predetermined condition is satisfied.

[Appendix 13]
A program that causes an information processing device (2) to execute the following steps: acquire speech information regarding an utterance of a target person, the speech information including first utterance information regarding a first utterance and second utterance information regarding a second utterance subsequent to the first utterance; determine first response information regarding a response to at least a portion of the utterance by inputting a first response determination instruction including the first utterance information into a large-scale language model; determine second response information regarding another response to at least a portion of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information into the large-scale language model before outputting the first response information to the target, when a predetermined condition regarding the utterance is satisfied; output the first response information when the predetermined condition is not satisfied, and output at least one of the first response information and the second response information when the predetermined condition is satisfied.

1...system, 2...information processing device, 3...terminal device, 4...LLM server device, 10...control unit, 12...storage unit, 70...computer, 100...acquisition unit, 102...determination unit, 102a...first response determination unit, 102b...second response determination unit, 102c...backchannel determination unit, 104...determination unit, 106...output unit, 700...processor

Claims (13)

an acquisition unit that acquires speech information related to an utterance of a target person, the speech information including first utterance information related to a first utterance and second utterance information related to a second utterance subsequent to the first utterance;
a first response determination unit that determines first response information related to a response to at least a part of the utterance by inputting a first response determination instruction including the first utterance information into a large-scale language model;
a second response determination unit that determines second response information related to another response to at least a part of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information into a large-scale language model before outputting the first response information to the target person when a predetermined condition related to the utterance is satisfied;
an output unit that outputs the first response information when the predetermined condition is not satisfied, and outputs at least one of the first response information and the second response information when the predetermined condition is satisfied;
An information processing device comprising: A backchannel determination unit that determines backchannel information related to a backchannel corresponding to the first utterance information when determining the first response information,
The output unit further outputs the backchannel information.
The information processing device according to claim 1 . When the predetermined condition is not satisfied, the output unit outputs the backchannel information and then outputs the first response information, and when the predetermined condition is satisfied, the output unit outputs the backchannel information and then outputs at least one of the first response information and the second response information.
The information processing device according to claim 2 . The backchannel determination unit determines the backchannel information by inputting a backchannel determination instruction including the first utterance information into a large-scale language model.
The information processing device according to claim 2 . a determination unit that determines whether a first division has occurred in the utterance,
the first utterance information includes information on a portion of the utterance up to the first break,
the first response determination unit inputs the first response determination instruction to a large-scale language model when the determination unit determines that the first break has occurred in the utterance.
The information processing device according to claim 1 . The determination unit further determines whether a second segment of the utterance occurs after the first segment of the utterance;
the second utterance information includes information on a portion of the utterance up to the second break,
the second response determination unit inputs an instruction to determine the second response to a large-scale language model when the determination unit determines that the second break has occurred in the utterance.
The information processing device according to claim 5 . The determination unit further determines whether or not a predetermined time for outputting a response has elapsed after the second division of the utterance;
the output unit outputs at least one of the first response information and the second response information when the determination unit determines that the predetermined time has elapsed after the second division.
The information processing device according to claim 6. A backchannel determination unit that determines backchannel information related to a backchannel corresponding to the first utterance information when determining the first response information,
The output unit further outputs the backchannel information,
The predetermined time is determined based on the backchannel information.
The information processing device according to claim 7. The output unit controls the terminal device of the subject to output the first response information by voice when the predetermined condition is not satisfied, and controls the terminal device of the subject to output at least one of the first response information and the second response information by voice when the predetermined condition is satisfied.
The information processing device according to claim 1 . The first response determination unit further determines information for outputting the first response information by voice when determining the first response information,
The second response determination unit further determines information for outputting the second response information by voice when determining the second response information.
The information processing device according to claim 9. a storage unit configured to store information for outputting each of the one or more texts by voice,
When the first response information is consistent with at least one of the one or more texts, determining information for outputting the first response information by voice includes obtaining information for outputting the consistent text by voice from the storage unit;
When the first response information does not match any of the one or more texts, determining information for outputting the first response information by voice includes generating information for outputting the first response information by voice based on a predetermined voice generation program;
When the second response information is consistent with at least one of the one or more texts, determining information for outputting the second response information by voice includes obtaining information for outputting the consistent text by voice from the storage unit;
When the second response information does not match any of the one or more texts, determining information for outputting the second response information by voice includes generating information for outputting the second response information by voice based on the predetermined voice generation program.
The information processing device according to claim 10. An information processing device,
acquiring speech information relating to an utterance by a target person, the speech information including first utterance information relating to a first utterance and second utterance information relating to a second utterance subsequent to the first utterance;
determining first response information related to a response to at least a portion of the utterance by inputting a first response determination instruction including the first utterance information into a large-scale language model;
determining second response information related to another response to at least a portion of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information into a large-scale language model before outputting the first response information to the target person when a predetermined condition related to the utterance is satisfied;
outputting the first response information when the predetermined condition is not satisfied, and outputting at least one of the first response information and the second response information when the predetermined condition is satisfied;
An information processing method. In the information processing device,
acquiring speech information relating to an utterance of a target person, the speech information including first utterance information relating to a first utterance and second utterance information relating to a second utterance subsequent to the first utterance;
determining first response information related to a response to at least a portion of the utterance by inputting a first response determination instruction including the first utterance information into a large-scale language model;
determining second response information related to another response to at least a portion of the utterance by inputting a second response determination instruction including the first utterance information and/or the first response information and the second utterance information into a large-scale language model before outputting the first response information to the target person when a predetermined condition related to the utterance is satisfied;
outputting the first response information when the predetermined condition is not satisfied, and outputting at least one of the first response information and the second response information when the predetermined condition is satisfied;
A program to execute.

Images