JP2025087599A - Method and system for generating answer using user feedback - Google Patents

Abstract

To provide a method and system for generating an answer using feedback of a user.SOLUTION: A method includes: generating answer candidates using a retriever of RAG (Retrieval Augmented Generation) for a first question from a user; generating a first answer to the first question from the user based on the answer candidates; providing a user interface to the user, the user interface including a function that allows the user to input a first feedback to the answer candidates; storing, in a storage, the first question, the first feedback input through the user interface, and information on answer candidates corresponding to the first feedback, in association with each other; selecting answer candidates to a second question from the user or a new user, using the first feedback and the information on the answer candidates corresponding to the first feedback stored in the storage; and generating a second answer to the second question using the selected candidates.SELECTED DRAWING: Figure 6

Description

The following description relates to a method and system for generating answers that utilizes user feedback.

Large Language Models (LLMs) are a type of artificial intelligence trained on large collections of text data to generate human-like responses to natural language input. They are language models consisting of artificial neural networks with a huge number of parameters (usually billions of weights or more). Such LLMs can be trained on substantial amounts of unlabeled text using self-supervised or semi-self-supervised learning.

RAG (Retrieval-Augmented Generation) is a technology used to complement the hallucinations of LLM and unlearned knowledge. Although LLM has the ability to generate responses by grasping general knowledge and context, it cannot answer questions other than those learned in the pre-learning process, and even if the content is learned, the answers generated by retrieving stored knowledge show considerable hallucinations.

To improve this, RAG is a method of inputting relevant knowledge into the LLM prompts and having the LLM answer based on this knowledge. Various LLMs have achieved significant performance improvements through this method.

However, RAG depends to a large extent on the performance of the retriever to answer the question. Naturally, the information required to answer the user's question must be given in advance. The problem of searching for documents that correspond to the answer from a wide range of documents, or of searching for an answer, has already been solved by ODQA (Open Domain Questions and Answers).
This is commonly known as the "Question Answering" problem.

Korean Patent No. 10-2551531

We provide a method and system for generating answers that utilizes user feedback.

The present invention provides an answer generation method for a computer device including at least one processor, the answer generation method including: generating, by the at least one processor, answer candidates for a first question of a user using a retriever of Retrieval Augmented Generation (RAG); generating, by the at least one processor, a first answer for the first question of the user based on the answer candidates; providing, by the at least one processor, a user interface including a function that enables the user to input a first feedback for the answer candidates; storing, by the at least one processor, information regarding the first question, the first feedback input through the user interface, and the answer candidates corresponding to the first feedback in a storage; and selecting, by the at least one processor, an answer candidate for the second question of the user or a new user using the first feedback and information regarding the answer candidates corresponding to the first feedback stored in the storage, and generating a second answer for the second question using the selected answer candidate.

According to one embodiment, the storing step includes a step of storing information about the first feedback and answer candidates corresponding to the first feedback in association with a sentence embedding value of the first question, and the generating step of the second answer includes a step of using information about the first feedback and answer candidates corresponding to the first feedback in association with a question retrieved from the storage through a sentence embedding value of the second question,
The method may further include a step of selecting answer candidates for the second question by adding answer candidates corresponding to the first feedback to a list of answer candidates generated for the second question, or removing answer candidates corresponding to the first feedback from the list.

According to another aspect, the step of generating the second answer may include a step of adding an answer candidate corresponding to the first feedback to a list of answer candidates generated for the second question when a first feedback associated with the question searched for in the storage is positive feedback; and a step of excluding the answer candidate corresponding to the first feedback from a list of answer candidates generated for the second question when the first feedback associated with the question searched for in the storage is negative feedback.

According to another aspect, the information about the answer candidates corresponding to the first feedback may include information about a location where the first feedback occurred in the answer candidates, and the step of selecting answer candidates for the second question may include generating new answer candidates including the location information and adding them to the list, or removing answer candidates including the location information from the list.

According to another aspect, the location may include a page of a document corresponding to the answer candidate or a location of content selected by the user in the answer candidate.

According to another aspect, the user interface can be characterized as including a function for displaying a list of answer candidates generated for the first question, a function for displaying content of an answer candidate selected from the list, and a function for inputting first feedback for the displayed answer candidate.

According to another aspect, the step of generating the first answer may include a step of generating a prompt based on the answer candidate; and a step of inputting the generated prompt into a Large Language Model (LLM) to generate the first answer.

According to another aspect, the user interface may further include a function for inputting a second feedback from the user for the first answer, and the answer generation method may further include a step of generating, by the at least one processor, a new prompt based on the negative feedback if negative feedback is input as the second feedback through the user interface; and a step of inputting, by the at least one processor, the new prompt into the LLM to generate a new first answer.

According to another aspect, the storing step may be characterized in that, when positive feedback is input as the second feedback through the user interface, an answer corresponding to the positive feedback is further associated with the first question and stored in the storage, and the generating step may be characterized in that an answer candidate for the second question is selected by further using the stored answer further associated with the first question.

A computer program is provided that is stored on a computer-readable recording medium and is coupled to a computer device to cause the computer device to execute the method.

A computer-readable recording medium is provided on which a program for causing a computer device to execute the method is recorded.

The present invention provides a computer device including at least one processor, the computer device being configured to execute computer device readable instructions by the at least one processor, the computer device being configured to generate answer candidates for a first question of a user using a RAG retriever, generate a first answer to the first question of the user based on the answer candidates, provide the user with a user interface including a function that allows the user to input first feedback for the answer candidates, store in a storage a link between information regarding the first question, the first feedback input through the user interface, and the answer candidates corresponding to the first feedback, the link between information regarding the first feedback, the answer candidates stored in the storage, and the answer candidates corresponding to the first feedback, for a second question of the user or a new user, the computer device being configured to select an answer candidate for the second question using the first feedback and the answer candidates corresponding to the first feedback stored in the storage, and generate a second answer to the second question using the selected answer candidate.

It is possible to provide a method and system for generating answers that utilizes user feedback.

FIG. 1 illustrates an example of a network environment according to one embodiment of the present invention. FIG. 2 is a block diagram illustrating an example computing device according to one embodiment of the invention. FIG. 3 is a diagram showing an example of a schematic configuration of an answer generating system according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of a user interface for inputting crew feedback from a user in an embodiment of the present invention. FIG. 5 is a diagram showing an example of a user interface for inputting answer feedback from a user in an embodiment of the present invention. FIG. 6 is a flowchart illustrating an example of a method for generating an answer in one embodiment of the present invention.

The following describes the embodiments in detail with reference to the attached drawings.

The response generation system according to the embodiment of the present invention may be embodied by at least one computer device. In this case, a computer program according to an embodiment of the present invention may be installed and run on the computer device embodying the response generation system, and the computer device may perform the response generation method according to the embodiment of the present invention under the control of the run computer program. The above-mentioned computer program may be stored in a computer-readable recording medium combined with the computer device to cause the computer to execute the response generation method.

Figure 1 is a diagram illustrating an example of a network environment according to one embodiment of the present invention. The network environment in Figure 1 shows an example including multiple electronic devices (110, 120, 130, 140), multiple servers (150, 160), and a network (170). Figure 1 is an example for explaining the invention, and the number of electronic devices and the number of servers are not limited to the same as in Figure 1.

The electronic devices (110, 120, 130, 140) may be fixed terminals or mobile terminals embodied in a computer system. Examples of the electronic devices (110, 120, 130, 140) include a smart phone, a mobile phone, a navigation system, a computer, a notebook computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a tablet PC, a game console, a wearable device, an IoT (internet of things) device, a VR (virtual reality) device, and an AR (augmented reality) device. As an example, FIG. 1 shows the shape of a smartphone as an example of the electronic device (110), but in the embodiment of the present invention, the electronic device (110) may substantially mean one of various physical computer systems that can communicate with other electronic devices (120, 130, 140) and/or servers (150, 160) through a network (170) using a wireless or wired communication method.

The communication method is not limited, and may include not only a communication method utilizing a communication network (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network, a satellite network, etc.) that the network 170 may include, but may also include short-range wireless communication between devices. For example, the network 170 may include one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, etc. In addition, the network 170 may include one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network.

Each of the servers (150, 160) may be embodied as a computer device or multiple computer devices that communicate with the multiple electronic devices (110, 120, 130, 140) through a network (170) to provide instructions, code, files, content, services, etc. For example, the server (150) may be a first computer device that provides the first and second services to the multiple electronic devices (110, 120, 130, 140) connected through the network (170).
The server (160) may be a system for providing a service, and the server (160) may also be a system for providing a second service to the plurality of electronic devices (110, 120, 130, 140) connected through the network (170). As a more specific example, the server (150) may provide a service (e.g., a search service, etc.) targeted by an application as a computer program installed and run on the plurality of electronic devices (110, 120, 130, 140) as a first service to the plurality of electronic devices (110, 120, 130, 140). As another example, the server (160) may provide a service for distributing files for installing and running the above-mentioned application to the plurality of electronic devices (110, 120, 130, 140) as a second service.

2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Each of the electronic devices (110, 120, 130, 140) and the servers (150, 160) may be implemented by a computer device (200) illustrated in FIG.

The computer device (200) may include a memory (210), a processor (220), a communication interface (230), and an input/output interface (240) as shown in FIG. 2. The memory (210) may include a computer readable recording medium such as a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the non-volatile mass storage device such as a ROM and a disk drive may be included in the computer device (200) as a separate permanent storage device distinct from the memory (210). The memory (210) may also store an operating system and at least one program code. These software components may be loaded into the memory (210) from a computer readable recording medium separate from the memory (210). Such separate computer readable recording medium may include a computer readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 210 through a communication interface 230 that is not a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computer device 200 based on a computer program that is installed by files received over the network 170.

The processor 220 may be configured to process computer program instructions by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor 220 via the memory 210 or the communications interface 230. For example, the processor 220 may
may be configured to execute instructions received according to program code stored in a storage device, such as memory 210.

The communication interface (230) may provide a function for the computer device (200) to communicate with other devices (e.g., the storage devices mentioned above) through the network (170). For example, requests, commands, data, files, etc. generated by the processor (220) of the computer device (200) according to program code stored in a storage device such as the memory (210) may be transmitted to other devices through the network (170) under the control of the communication interface (230). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (200) through the communication interface (230) of the computer device (200) via the network (170). The signals, commands, data, etc. received through the communication interface (230) may be transmitted to the processor (220) or memory (210), and the files, etc. may be stored in a storage medium (the permanent storage device mentioned above) that the computer device (200) may further include.

The input/output interface (240) may be a means for interfacing with an input/output device (250). For example, the input device may include devices such as a microphone, keyboard, or mouse, and the output device may include devices such as a display and a speaker. As another example, the input/output interface (240) may be a means for interfacing with a device that integrates input and output functions into one, such as a touch screen. The input/output device (250) may be configured as a single device together with the computer device (200).

Also, in other embodiments, the computing device (200) may include fewer or more components than those of FIG. 2. However, it is not necessary to explicitly illustrate most of the conventional components. For example, the computing device (200) may be embodied to include at least some of the input/output devices (250) described above, or may include a transceiver.
, and may further include other components such as a database.

3 is a diagram illustrating an example of a general view of an answer generation system according to an embodiment of the present invention. The answer generation system (300) according to the embodiment of FIG. 3 may include a retriever (310), a summarizer (320), a large language model (LLM) (330), and a storage (340) of RAG (Retrieval Augmented Generation), and may improve the performance of the retriever (310) by using user feedback.

When an input is input to the retriever (310) according to a user's question, the retriever (310) can search for answer candidates to the user's question based on the input. At this time, there are sparse search methods that search for answer candidates using keyword search, and dense search methods that search for answer candidates using an ensemble of invading vector search or cross encoder method. At this time, the answer candidates searched by the retriever (310) are called clues.

If all searched answer candidates are input as prompts into the LLM (330), the maximum token length limit may be exceeded. To solve this problem, the summarizer (320) receives clues as input and can summarize the clues focusing on the content related to the answer. In this case, the summary can use a generative summarization method or an extractive summarization method based on the LLM. The LLM for summarization may be the same as or different from the LLM (320) in FIG. 3. The summarizer (320) creates a prompt using the summarized clues and inputs it into the LLM (320), so that the final answer can be generated through the LLM (320) and then provided to the user.

At this time, the answer generation system (300) can receive two types of feedback from the user as input: clue feedback for the answer and answer feedback.

The clue feedback may be feedback for the clue (answer candidate) used to generate the prompt. As an example, the answer generation system (300) may provide the user with a function to check the contents of each clue through a list of clues used to generate the answer and select positive feedback or negative feedback for each clue. In this case, the answer generation system (300) may check the positive feedback or negative feedback for each clue selected by the user as the clue feedback for the corresponding clue.

The answer feedback may include feedback on the answer generated by the LLM (320). For example, the clue itself may be fine, but the LLM (320) may generate an answer that is unrelated to the clue. In this case, the answer generation system (300) may provide a function for checking the contents of each clue through the list of clues described above, and may provide a function for the user to input answer feedback when the answer is wrong even though the clue is not wrong.

The answer generation system (300) can store the user's feedback, including the cue feedback and/or the answer feedback, in the storage (340) and can generate answers for new questions of the user and/or new users using the feedback stored in the storage (340). As an example, the answer generation system (300) can use the cue feedback to control the retriever (310) to select more accurate cues and can use the answer feedback to control the summarizer (320) to generate more accurate prompts.

FIG. 4 is a diagram illustrating an example of a user interface for receiving clue feedback from a user in one embodiment of the present invention. FIG. 4 shows an example of a user interface (400) that the answer generation system (300) can provide to a user. The user interface (400) can include an area (410) for displaying question 1 and answer 1 when answer 1 to question 1 of a user is provided, an area (area of dotted line box (420)) for displaying a list of clues used to generate answer 1, and an area (430) for displaying the contents of a clue selected from the list of clues. The user interface (400) can also include a function (for example, button (440) and button (450)) for inputting positive feedback and/or negative feedback for the selected clue (luggage 1 in the embodiment of FIG. 4).

For example, the user can check the content of each crew from the list of crews used to generate answer 1 to see if the crew content is appropriate. At this time, the user can select button (440) to input positive feedback or select button (450) to input negative feedback for the currently displayed crew.

At this time, the answer generation system (300) can store in the storage (340) the sentence embedding value for the user's question (question 1 in the embodiment of FIG. 4), information on the selected clue (clue 1 in the embodiment of FIG. 4), and the user's clue feedback. According to an embodiment, the user interface (400) can provide a function for receiving positive feedback and/or negative feedback for each page of the clue or for a part of the content (for example, a specific text) selected by the user from the contents included in the clue. In this case, the answer generation system (300) can further store information on the location of the clue (for example, document information, page information, and/or selected content information) in the storage (340) as information on the clue.

When a new question is subsequently input by the user or a new user, the retriever (310) can search for clue feedback related to a similar question in the storage (340) through the sentence embedding value of the new question in the storage (340). At this time, the retriever (310) can select a more accurate clue by adding clues associated with positive feedback searched in the storage (340) to the list of clues or removing clues associated with negative feedback from the list of clues.

FIG. 5 is a diagram illustrating an example of a user interface for inputting answer feedback from a user in one embodiment of the present invention. FIG. 5 illustrates an example of a user interface (500) that the answer generation system (300) can provide to a user. The user interface (500) may include an area in which a question 1 input by a user and an answer 1 generated for the question 1 are displayed. In this case, the user interface (500) may provide a function for selecting affirmative or negative for the answer 1 as shown in a dotted box (510). In this case, if the user selects negative, a user interface (520) may be provided for the user to select a specific content from among contents prepared in advance as answer feedback or to input a new reason.

The answer generation system (300) can construct a new prompt based on the prompt used to generate answer 1 and the answer feedback input through the user interface (520). The answer generation system (300) can also input the new prompt to the LLM (330) to generate a new answer 2 and provide it to the user. In this case, the user can select either affirmation or negation for the new answer 2, and if negation is selected, the process of receiving answer feedback again and generating another new answer can be repeated. Alternatively, the answer generation system (300) can receive the answer desired by the user as feedback.

If the user selects affirmative, the answer generation system (300) may save the sentence embedding value for the user's question and the affirmatively selected answer in storage (340).

Thereafter, when a new question is input by the user or a new user, the answer generation system (300) can search for answer feedback related to a similar question from the storage (340) through the sentence embedding value of the new question. At this time, the retriever (310) can generate a more accurate answer by including answers related to the answer feedback searched from the storage (340) in a list of clues and having the summarizer (320) generate a prompt.

6 is a flowchart showing an example of an answer generation method according to an embodiment of the present invention. The answer generation method according to this embodiment can be executed by a computer device (200) that implements the answer generation system (300) described above. In this case, the processor (220) of the computer device (200) executes the operating system code or at least one
6. The method of the present invention can be implemented to execute control instructions provided by the code of one or more computer programs, in which the processor (220) can control the computer device (200) to execute steps (610 to 660) of the method of FIG. 6 according to the control instructions provided by the code stored in the computer device (200).

In step 610, the computer device 200 may generate answer candidates for the first question of the user using a retriever of the RAG. The retriever may correspond to the retriever 310 described above in FIG.

In step 620, the computer device 200 can generate a first answer to the user's first question based on the answer candidates. As an example, the computer device 200 can generate a prompt based on the answer candidates and input the generated prompt into a large language model (LLM) to generate the first answer.

In step 630, the computer device 200 may provide a user interface to the user including a function that allows the user to input a first feedback for the answer candidate. As an example, the user interface may include a function for displaying a list of answer candidates generated for the first question, a function for displaying the content of an answer candidate selected from the list, and a function for inputting a first feedback for the displayed answer candidate.

Also, according to an embodiment, the user interface may further include a function for inputting a second feedback for the first answer from the user. In this case, when negative feedback is input as the second feedback through the user interface, the computer device (200) may generate a new prompt based on the negative feedback and input the new prompt to the LLM to generate a new first answer. When the user also inputs negative feedback for the new first answer, the computer device (200) may again generate a new prompt based on the negative feedback and generate a new first answer.

Here, the first feedback can correspond to the crew feedback mentioned above, and the second feedback can correspond to the answer feedback.

In step (640), the computer device (200) may store information on the first question, the first feedback input through the user interface, and answer candidates corresponding to the first feedback in a storage in association with each other. Here, the storage may correspond to the above-mentioned storage (340). As an example, the computer device (200) may store information on the first feedback and answer candidates corresponding to the first feedback in association with a sentence embedding value of the first question.

Furthermore, when positive feedback is input as the second feedback through the user interface, the answer corresponding to the positive feedback can be further linked to the first question and stored in the storage.

In step (650), the computer device (200) may select an answer candidate for the second question of the user or the new user by using information on the first feedback stored in the storage and the answer candidate corresponding to the first feedback. As an example, the computer device (200) may select an answer candidate for the second question by adding an answer candidate corresponding to the first feedback to a list of answer candidates generated for the second question or removing an answer candidate corresponding to the first feedback from the list by using information on the first feedback associated with the question searched in the storage through the sentence embedding value of the second question and the answer candidate corresponding to the first feedback. As a more specific example, the computer device (200) may add an answer candidate corresponding to the first feedback to a list of answer candidates generated for the second question when the first feedback associated with the question searched in the storage is positive feedback. As another example, if the first feedback associated with the question searched for in the storage is negative feedback, the computer device (200) can exclude the answer candidate corresponding to the first feedback from the list of answer candidates generated for the second question.

Meanwhile, the information on the answer candidate corresponding to the first feedback may include information on the location where the first feedback occurred in the answer candidate. For example, the location where the first feedback occurred may include a page of a document corresponding to the answer candidate or a location of content selected by a user in the answer candidate. In this case, the computer device (200) may generate a new answer candidate including the content of the location and add it to the list of answer candidates, or may remove the answer candidate including the content of the location from the list of answer candidates.

In addition, if an answer corresponding to the positive feedback as the second feedback is further linked to the first question and stored in the storage, the computer device (200) can select an answer candidate for the second question by further using the answer further linked to the first question and stored in the storage. For example, the computer device (200) can add the answer corresponding to the positive feedback to a list of answer candidates generated for the second question.

In step (660), the computer device (200) can generate a second answer to the second question using the selected answer candidate. In this way, the computer device (200) can generate an answer through a more accurate answer candidate by selecting an answer candidate using the user's clue feedback (first feedback) and/or answer feedback (second feedback). Also, the computer device (200) can provide a more appropriate answer to the user by generating a new prompt through the answer feedback (second feedback).

In this way, according to the embodiments of the present invention, it is possible to provide a method and system for generating answers that utilizes user feedback.

The above-described systems or devices may be embodied using hardware components or a combination of hardware and software components. For example, the devices and components described in the embodiments may be embodied using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, the processing device may be described as being one in use, but a person skilled in the art may know that the processing device may include multiple processing elements and/or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may include computer programs, code, instructions, or any combination of one or more of these, capable of configuring or instructing a processing device to operate in a desired manner, either individually or collectively. Software and/or data may be implemented as a program or component of some type to be interpreted by a processing device or to provide instructions or data to a processing device.
The software may be embodied in a physical device, a virtual device, a computer storage medium or a device. The software may also be distributed over network coupled computer systems so that it is stored and executed in a distributed fashion. The software and data may be stored on one or more computer readable recording media.

The method according to the present embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The medium may continuously store a computer-executable program, or may temporarily store the program for execution or download. The medium may be a variety of recording or storage means in the form of a single or multiple pieces of hardware combined, but is not limited to a medium directly connected to any computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and media configured to store program instructions, including ROMs, RAMs, and flash memories. Examples of other media include recording media or storage media managed by app stores that distribute applications, and sites and servers that supply or distribute various other software. Examples of program instructions include machine code, such as that produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like.

As described above, the embodiments have been described by limited examples and drawings, but those having ordinary skill in the art can make various modifications and variations from the above description. For example, the described techniques can be performed in an order different from that described, and/or the components of the described systems, structures, devices, circuits, etc. can be combined or combined in a different manner from that described, or replaced or substituted by other components or equivalents, and still achieve suitable results.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the claims described below.

Claims (10)

1. A method of answer generation for a computing device including at least one processor, comprising:
generating, by the at least one processor, answer candidates for a first question of a user using a retriever of Retrieval Augmented Generation (RAG);
generating, by the at least one processor, a first answer to the user's first question based on the candidate answers;
providing, by the at least one processor, a user interface to the user, the user interface including a function that allows the user to input a first feedback for the answer candidate;
storing, by the at least one processor, information regarding the first question, the first feedback input through the user interface, and answer candidates corresponding to the first feedback in a storage device in association with each other;
selecting, by the at least one processor, a candidate answer to a second question of the user or a new user, using information on the first feedback stored in the storage and the candidate answers corresponding to the first feedback, and generating a second answer to the second question using the selected candidate answers.
Answer generation method. 2. The method of claim 1,
The storing step includes:
storing information about the first feedback and candidate answers corresponding to the first feedback in association with sentence embedding values of the first question;
The step of generating a second answer includes:
and selecting answer candidates for the second question by using information about a first feedback associated with a question searched for from the storage through a sentence-embedded value of the second question and an answer candidate corresponding to the first feedback, to a list of answer candidates generated for the second question, or by excluding the answer candidate corresponding to the first feedback from the list. 3. The method of claim 2,
The step of generating the second answer includes:
adding an answer candidate corresponding to the first feedback to a list of answer candidates generated for the second question when a first feedback associated with a question searched for in the storage is positive feedback; and excluding an answer candidate corresponding to the first feedback from a list of answer candidates generated for the second question when the first feedback associated with a question searched for in the storage is negative feedback. 3. The method of claim 2,
the information about the answer candidate corresponding to the first feedback includes information about a position in the answer candidate where the first feedback occurs;
The step of selecting answer candidates to the second question includes:
An answer generating method, comprising: generating a new answer candidate including the content of the location and adding it to the list; or removing an answer candidate including the content of the location from the list. 5. The method of claim 4,
The answer generating method, wherein the location includes a page of a document corresponding to the answer candidate or a location of content selected by the user in the answer candidate. 10. The method of claim 1 ,
The answer generating method, characterized in that the user interface includes a function for displaying a list of answer candidates generated for the first question, a function for displaying the content of an answer candidate selected from the list, and a function for inputting first feedback for the displayed answer candidate. 10. The method of claim 1 ,
The step of generating a first answer includes:
A method for generating an answer comprising: generating a prompt based on the answer candidates; and inputting the generated prompt into a Large Language Model (LLM) to generate the first answer. 8. The method of claim 7,
the user interface further includes a function for receiving, from the user, a second feedback on the first answer;
and if negative feedback is input as the second feedback via the user interface, generating a new prompt based on the negative feedback by the at least one processor; and inputting the new prompt into the LLM to generate a new first answer by the at least one processor. 8. The method of claim 7,
The storing step includes:
When positive feedback is input as the second feedback via the user interface, an answer corresponding to the positive feedback is further associated with the first question and stored in the storage;
The step of generating a second answer includes:
The answer generating method further comprises using an answer stored in association with the first question to select an answer candidate for the second question. at least one processor embodied to execute computer device readable instructions;
by said at least one processor,
Generate answer candidates for the user's first question using a RAG (Retrieval Augmented Generation) retriever,
generating a first answer to a first question of the user based on the candidate answers;
providing the user with a user interface including a function that allows the user to input a first feedback for the answer candidate;
storing information on the first question, the first feedback input through the user interface, and answer candidates corresponding to the first feedback in a storage device;
A computer device comprising: a computer that, for a second question of the user or a new user, selects a candidate answer to the second question using information about the first feedback stored in the storage and a candidate answer corresponding to the first feedback; and generates a second answer to the second question using the selected candidate answer.

Images