KR102800075B1 - Multilingual AI DOC System Building Method for Hindi English And Korean - Google Patents

Abstract

The present invention discloses a method for constructing a multilingual AI DOC system including Hindi, English, and Korean, which is designed to construct RAG data based on Hindi and English, train LLM through fine-tuning, design a natural language search question-answering system through the LangChain framework, and implement summary of main contents, Q&A of related contents, and search of specific contents when uploading documents by vectorizing commercial documents of public institutions and industries through the RAG-based question-answering system, and a method for operating the same.
The method for constructing a multilingual AI DOC system including Hindi, English, and Korean of the present invention comprises: a first step in which Hindi-based learning data, English-based learning data, and Korean-based learning data are collected and processed to train an on-premise-based language model, sLLM (smaller Large Language Model), which understands Hindi, English, and Korean; a second step in which a Hindi learning dataset is constructed using commercial documents of public institutions and industries in Hindi, and the sLLM of the first step is constructed using a RAG (Retrieval-Augmented Generation) framework that vector embeds the Hindi learning dataset and stores it in a vector database; and a third step in which fine-tuning is performed on the sLLM of the second step.

Description

{Multilingual AI DOC System Building Method for Hindi English And Korean}

The present invention relates to a method for constructing a multilingual AI DOC system including Hindi, English, and Korean, and a method for operating the same, and more specifically, to a method for constructing a multilingual AI DOC system including Hindi, English, and Korean, and a method for operating the same, which is designed to construct RAG data based on Hindi and English, train LLM by fine-tuning, design a natural language search question-answering system through the LangChain framework, and provide customized information to users by vectorizing commercial documents of public institutions and industries through the RAG-based question-answering system and implementing summaries of main contents, Q&As of related contents, and searches for specific contents when uploading documents.

Recently, India is being evaluated positively and creating an environment for the advancement of Korean small and medium-sized enterprises as a promising next-generation market with the largest population and an annual GDP growth rate of 7%. India has a vast market with 650 million Hindi speakers and 1.3 billion English speakers. Due to the nature of India, Hindi is used by about 50% of the total population, but since Hindi and English are the official languages used in administration, Indians who do not use Hindi have difficulty in handling government administration.

To overcome these issues, more research is needed on multilingual AI that can reduce administrative inconvenience for Indians who use local languages, and increase efficiency in the Indian market and economy by standardizing.

Republic of Korea Patent Publication No. 10-2431383 Republic of Korea Patent Publication No. 10-2384641

The present invention is to solve the above problems, and provides a method for constructing a multilingual AI DOC system including Hindi, English, and Korean, which is designed to construct RAG data based on Hindi and English, train LLM through fine-tuning, design a natural language search question-answering system through the LangChain framework, and implement summarized main contents, Q&A on related contents, and search for specific contents when uploading documents by vectorizing commercial documents of public institutions and industries through the RAG-based question-answering system, and provide customized information to users, and a method for operating the same.

The method for constructing a multilingual AI DOC system including Hindi, English, and Korean of the present invention to achieve the above-described purpose includes: a first step in which Hindi-based learning data, English-based learning data, and Korean-based learning data are collected and processed to train an on-premise-based language model, sLLM (smaller Large Language Model), which understands Hindi, English, and Korean; a second step in which a Hindi learning dataset is constructed using commercial documents of public institutions and industries in Hindi, and the sLLM of the first step is constructed using a RAG (Retrieval-Augmented Generation) framework that vector embeds the Hindi learning dataset and stores it in a vector database; and a third step in which fine-tuning is performed on the sLLM of the second step.

The second step is a step 2-1 in which a Hindi document QA dataset is constructed in the structure of queries, contexts, and answers from commercial documents of public institutions and industries in Hindi, a step 2-2 in which a search model is trained using the queries and contexts and the search model is evaluated, a step 2-3 in which the search model is retrained through hard negatives if the target performance is not achieved in the evaluation of the second step, a step 2-4 in which the context based on the search model trained in the second step or retrained in the second step is vectorized and a vector database is constructed, a dataset for learning a generative model is constructed using the vector database of the second step, and the generative model is trained and evaluated, a step 2-6 in which expert feedback data is collected and an interactive language model reflecting the preferences of Hindi users and experts is trained if the target performance is not achieved in the evaluation of the second step, and the search model trained in the second step or retrained in the second step and the search model trained in the second step or retrained in the second step and the search model trained in the second step or retrained in the second step are trained in the second step or It includes step 2-7, where a RAG framework based on the generative model retrained in step 2-6 is built and evaluated, and step 2-8, where error cases are analyzed and additional learning is performed when the target performance is not achieved in the evaluation in step 2-7.

The commercial documents of the public institutions and industries of the above-mentioned Step 2-1 are selected from one of the following fields: the first field is the environment field, the second field is the education field, and the third field is the energy field. If the commercial document is selected from the first field, the second field and the third field are excluded. In this case, if the first field and the second field are included in the selected commercial document and the first field accounts for less than 80% of the selected commercial document, the commercial document that includes the first field and the second field is excluded from the Hindi document QA dataset. If the first field accounts for a large proportion of all commercial documents, it is adopted as the dataset. However, if the first field accounts for a small proportion of all commercial documents, it is not adopted as the dataset. This allows for increasing the specialization of the terminology of the dataset.

The evaluations of the above steps 2-2, 2-5 and 2-7 are characterized in that they are carried out by measuring the input data recognition rate, malfunction rate, response speed and answer accuracy.

The above malfunction rate is expressed by the following equation 1:

(Formula 1)

Malfunction rate = Σ((number of malfunctions / number of questions) / number of repetitions)) × 100

The above response speed is expressed by Equation 2 below.

(Formula 2)

Response speed = Σ{(time to output answer - time to input question)} / number of questions

Here, the evaluation criteria are input data recognition rate of 95% or higher, malfunction rate of 5% or lower, response speed of 3 seconds or lower, and answer accuracy of 88% or lower. If the criteria above are met, it is evaluated as good, and if the criteria above are not met, re-learning is performed.

The above evaluation criteria vary depending on the number of questions, and are set as the first evaluation criterion when the number of questions is 50 or more, and are set as the second evaluation criterion when the number of questions is less than 50, and the first evaluation criterion is an input data recognition rate of 93% or more, a malfunction rate of 7% or less, a response speed of 5 seconds or less, and an answer accuracy of 85% or less. If the above criteria are met, it is evaluated as good, and if the criteria are not met, relearning is performed, and the second evaluation criterion is an input data recognition rate of 95% or more, a malfunction rate of 5% or less, a response speed of 3 seconds or less, and an answer accuracy of 88% or less. If the above criteria are met, it is evaluated as good, and if the criteria are not met, relearning is performed. When the number of questions is less than 50, the criteria are set higher than when the number of questions is 50 or more, and errors arising from deviations for a small number of samples can be reduced.

The third step includes a third step in which a question query is input into the search model to calculate question embedding and document embedding, a third step in which a related document is searched by calculating relevance using Top-K using the question embedding and document embedding, and a third step in which the RAG framework-based sLLM is fine-tuned so that an appropriate answer and summary are generated by inputting the question and related document into the generative model.

In addition, the operating method of the multilingual AI DOC system including Hindi, English, and Korean of the present invention for achieving the above-described purpose includes: a first step in which a question is input and a document file to be questioned is uploaded; a second step in which the text of the question and the document file are chunked and vectorized to calculate question embedding and document vector embedding; a third step in which the question embedding and the document embedding are input into a search model to search related documents; a fourth step in which the question and the related document are input into a generation model to generate appropriate answers and summaries; and a fifth step in which the result of the generation model is input into sLLM to produce a final result; wherein one and two or more of the steps 1 to 5 use artificial intelligence and Glaude using AICC (AI Contact Center).

As described above, the present invention has the effect of increasing work process efficiency through knowledge retrieval and document summarization and drafting using artificial intelligence.

In addition, the present invention has the effect of improving national satisfaction and quality of life by improving work productivity in the public and private sectors, strengthening knowledge sharing and collaboration, and providing better decision-making and services through artificial intelligence.

Figure 1 is a conceptual diagram of a multilingual AI DOC system including Hindi, English, and Korean of the present invention.
FIG. 2 is a flowchart of a method for constructing a multilingual AI DOC system including Hindi, English, and Korean of the present invention.
Figure 3 is a structural diagram of the RAG framework of the present invention.
FIG. 4 is a detailed flowchart of the second step of the method for constructing a multilingual AI DOC system including Hindi, English, and Korean of the present invention.
FIG. 5 is a detailed flowchart of the third step of the method for constructing a multilingual AI DOC system including Hindi, English, and Korean of the present invention.
FIG. 6 is a flowchart of an operation method of a multilingual AI DOC system including Hindi, English, and Korean of the present invention.

In various embodiments of the present disclosure, the expressions such as “includes,” “may include,” etc., indicate the presence of the disclosed corresponding function, operation, or component, etc., and do not limit one or more additional functions, operations, or components, etc. In addition, in various embodiments of the present disclosure, it should be understood that the terms such as “includes,” “have,” etc., are intended to specify the presence of a feature, a number, a step, an operation, a component, a part, or a combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In various embodiments of the present disclosure, the expression "or" and the like includes any and all combinations of the words listed together. For example, "A or B" may include A, may include B, or may include both A and B.

The expressions “first,” “second,” “first,” or “second,” etc., used in various embodiments of the present disclosure can modify various components of the various embodiments, but do not limit the components. For example, the expressions do not limit the order and/or importance of the components. The expressions can be used to distinguish one component from another component. For example, the first user device and the second user device are both user devices, and represent different user devices. For example, without departing from the scope of the various embodiments of the present disclosure, the first component can be referred to as the second component, and similarly, the second component can also be referred to as the first component.

When it is said that a component is "connected" or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that there may also be other new components between the component and the other component. On the other hand, when it is said that a component is "directly connected" or "directly connected" to another component, it should be understood that no other new components exist between the component and the other component.

In the embodiments of the present disclosure, terms such as "module," "unit," "part," etc. are terms used to refer to components that perform at least one function or operation, and these components may be implemented as hardware or software, or as a combination of hardware and software. In addition, a plurality of "modules," "units," "parts," etc. may be integrated into at least one module or chip and implemented as at least one processor, except in cases where each of them needs to be implemented as individual specific hardware.

The terms used in the various embodiments of the present disclosure are only used to describe specific embodiments and are not intended to limit the various embodiments of the present disclosure. The singular expression includes the plural expression unless the context clearly indicates otherwise.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present disclosure belong.

Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined in various embodiments of the present disclosure.

Hereinafter, a multilingual AI DOC system (10) including Hindi, English, and Korean according to the present invention will be described with reference to the drawings. FIG. 1 is a conceptual diagram of a multilingual AI DOC system (10) including Hindi, English, and Korean according to the present invention. Referring to FIG. 1, a multilingual AI DOC system (10) including Hindi, English, and Korean according to an embodiment of the present invention may be configured to include a Retrieval-Augmented Generation (RAG) module and a smaller Large Language Model (sLLM) module. The multilingual AI DOC system (10) including Hindi, English, and Korean according to the present invention may include one or more processors, one or more memories, one or more storages, and one or more communication interfaces, which may be connected to each other through a bus. In addition, the multilingual AI DOC system (10) including Hindi, English, and Korean may include hardware such as an input device and an output device. In addition, the multilingual AI DOC system (10) including Hindi, English, and Korean may be equipped with various software including an operating system capable of running a program. In addition, the multilingual AI DOC system (10) including Hindi, English, and Korean provides a description of the Hindi document (DOC, Document) entered for a question, writing tips, a summary, and translation services into English and Korean.

The RAG module (100) collects Hindi-based document data and stores it by vector embedding. The RAG module (100) includes a document collection unit that collects documents written in Hindi, a vector embedding unit that chunks the collected documents and embeds texts into vectors, a vector database where vector-embedded data is stored, an ETL unit that extracts, transforms, and loads internal information of a public institution or company, and an internal data storage unit where the loaded internal information is stored.

The sLLM module (200) learns Hindi-based learning data, English-based learning data, and Korean-based learning data to understand Hindi, English, and Korean. The sLLM module (200) includes a Hindi learning unit that learns Hindi-based learning data, an English learning unit that learns English-based learning data, a Korean learning unit that learns Korean-based learning data, and an answer generation unit that generates a final answer to a question using data generated from the generation model of the RAG module (100).

Hereinafter, a method for constructing a multilingual AI DOC system (10) including Hindi, English, and Korean of the present invention will be described with reference to the drawings. FIG. 2 is a flowchart of a method for constructing a multilingual AI DOC system (10) including Hindi, English, and Korean of the present invention, and FIG. 3 is a structural diagram of a RAG framework of the present invention. Referring to FIG. 2, a method for constructing a multilingual AI DOC system (10) including Hindi, English, and Korean of the present invention can be configured to include the following three steps. In summary, the first step (S10) is a step in which sLLM is trained with basic language learning data so that sLLM can understand Hindi, English, and Korean. The second stage (S20) is a stage to complement the sLLM of the first stage (S10) by vector embedding the Hindi learning dataset with commercial documents from public institutions and industries in Hindi for the sLLM learned in Hindi, English, and Korean based on the RAG framework. The sLLM learned for the Hindi learning dataset with commercial documents from public institutions and industries in Hindi through the RAG framework can be said to be an sLLM specialized for Hindi-related documents. The third stage (S30) is a stage to improve the accuracy of the sLLM through fine-tuning that adjusts the parameters by evaluating the search results of the RAG search model and generation model for the sLLM of the second stage (S20) built based on the RAG framework.

Step 1 (S10) is a step in which Hindi-based learning data, English-based learning data, and Korean-based learning data are collected and processed to train an on-premise language model, sLLM (smaller Large Language Model), which understands Hindi, English, and Korean. Here, sLLM (smaller Large Language Model) is a language model that basically performs the same function as LLM, but the size of the model is relatively smaller than that of LLM, and the number of model parameters can be reduced and the accuracy can be improved through fine-tuning. Since sLLM has a relatively small data size, it is generally preferable to install it on-premise, that is, in-house, for security reasons. Operating sLLM on-premise has the advantage of protecting sLLM from external intrusions. In the present invention, a large language model that understands Hindi, English, and Korean is required. To this end, Hindi-based learning data, English-based learning data, and Korean-based learning data are collected and processed to train a large language model. In the present invention, sLLM can be built with the LangChain framework. The LangChain framework is an open source framework that supports the development of applications using the language model of generative AI. LangChain consists of the Model I/O, Retrieval, Chains, Agents, Memory, and Callbacks modules, and allows various language processing tasks to be performed using LLM. LangChain supports various LLM models such as OpenAI and Hugging Face, and users can select and use a model that suits their needs. The core concept of LangChain is to chain the execution of LLM prompts and the execution of external sources, and various tasks such as translation, summary, question answering, text generation, and natural language inference can be performed using LLM. First, as a translation service, it can distribute and manage translation models and provides translation models that support various languages. Second, as a summary service, it can distribute and manage summary models, providing summary models for various topics. Third, as a question answering service, it can distribute and manage question answering models and provides question answering models for various fields. Fourth, as a text generation service, it can distribute and manage text generation models and provides models that can generate texts in various formats. Fifth, as a natural language inference service, it can distribute and manage natural language inference models and provides models that can perform various natural language inference tasks.

The second step (S20) is a step in which a Hindi learning dataset is constructed with commercial documents from public institutions and industries in Hindi, and the Hindi learning dataset is vector-embedded and stored in a vector database to construct the sLLM of the first step (S10) with a RAG (Retrieval-Augmented Generation) framework. Since the present invention implements a summary of main contents, Q&A of related contents, search for specific contents, etc. when a Hindi document is uploaded, a Hindi learning dataset is constructed with commercial documents from public institutions and industries written in Hindi, and the Hindi learning dataset is vectorized and embedded to construct an sLLM based on the RAG framework. Fig. 4 is a detailed flowchart of the second step of a method for constructing a multilingual AI DOC system (10) including Hindi, English, and Korean of the present invention. Referring to FIG. 3, the second step (S20) of the method for constructing a multilingual AI DOC system (10) including Hindi, English, and Korean of the present invention may be configured to include the following eight steps. In the present invention, the RAG framework is used to complement the shortcomings of LLM. The RAG model is a model that performs text generation work, searches for information from given source data, and performs a process of generating a desired text by utilizing the information. The data processing process for using RAG divides the original data into small pieces in chunk units, converts text data into a vector, which is a number, and stores it in a vector database through embedding. Here, the vector database is a database that can store, index, and search data such as texts and images in vector form, and if an algorithm is used, it is possible to search for approximate nearest neighboring similar items through hashing, quantization, or graph-based search, so it operates efficiently and scalably for a large data set of high-dimensional vectors.

Step 2-1 (S21) is the step of constructing a Hindi document QA dataset with the structure of query, context, and answer. A RAG model is constructed with documents in Hindi. To this end, a Hindi learning dataset is constructed with commercial documents from public institutions and industries in Hindi, and this is constructed as a Hindi document QA dataset with the structure of query, context, and answer.

Step 2-2 (S22) is the step where the search model is learned and evaluated using the query and context. In the RAG framework, a search model and a generative model are used. The search model learns to embed questions and documents, to make the embeddings of documents related to the question close, and to make the embeddings of unrelated documents far apart, and searches for documents related to the question in a vector database based on the search model. In addition, the generative model is a language model based on a decoder or encoder-decoder, which generates texts that fit the given context, and additionally learns specific domain data based on the previously learned generative model to generate an appropriate answer to the question. Step 2-2 (S22) trains the search model using the query and context of the Hindi learning dataset, and evaluates whether the learning was performed properly. Here, the evaluations of Steps 2-2 (S22), 2-5 (S25), and 2-7 (S27) are performed by input data recognition rate, malfunction rate, response speed, and answer accuracy. The input data recognition rate evaluates the accuracy of the input data, and the malfunction rate measures errors when asking questions, including hallucinations, failures in masking sensitive information, no response, and non-response. In addition, the response speed measures the speed at which answers are output when a question is input. In addition, the answer accuracy checks the accuracy of the answer by asking questions about a set of questions/answers. The input data recognition rate is calculated by inputting a question, conducting a conversation test, checking the output intent, and comparing the result with the correct answer to calculate Precision and Recall, and calculating the average response rate by repeating 4 times each for a total of 100 times with a list of 25 questions, and checking whether the intent inference rate is 90% or higher based on the results. The malfunction rate is calculated by inputting a question to check whether there is a malfunction, and calculating the malfunction rate by repeating 4 times each for a total of 100 times with a list of 25 questions. The malfunction criteria are hallucination, failure to mask sensitive information, forced termination, no response, etc., and the malfunction rate is calculated by the following Equation 1 and checking whether the malfunction rate is 5% or less.

(Formula 1)

Malfunction rate = Σ((number of malfunctions / number of questions) / number of repetitions)) × 100

The response speed is checked by checking the source code, which is the code that outputs the time when the question is input and the code that outputs the time when the answer is output. The time required from the time when the question is input to the time when the answer is output is calculated by inputting the question and using the output log. The average response time is calculated by repeating 4 times for each of the 25 question lists, for a total of 100 times. The response speed is calculated by the following formula 2, and it is checked whether the response speed is less than 3 seconds.

(Formula 2)

Response speed = Σ{(time to output answer - time to input question)} / number of questions

Answer accuracy is checked by checking the question content and the answer to the question, and the Benchmark LLM Evaluation is performed to evaluate whether the intent of the derived sentence matches the LLM and interactive verification. That is, the question is input into the provided solution, and GPT-4 checks whether the output result matches the answer to the question to calculate the accuracy of the answer. This is performed four times for each of 25 sample questions, the average accuracy is calculated, and it is confirmed that the answer accuracy is 88% or higher. The evaluation criteria are input data recognition rate of 95% or higher, malfunction rate of 5% or lower, response speed of 3 seconds or less, and answer accuracy of 88% or lower. If the above criteria are met, it is evaluated as good, and if the above criteria are not met, relearning is performed.

Step 2-3 (S23) is the step where the search model is retrained through hard negatives when the target performance is not achieved in the evaluation of Step 2-2 (S22). Hard negatives refer to data samples that appear similar but actually belong to different categories. These samples are selected to help the search model learn more accurate features. In particular, hard negative samples are encoded similarly to existing samples but actually belong to different classes, which induces the search model to develop stronger discrimination power. One way to utilize hard negative samples is hard negative mining (HNM). This method actively finds hard negative samples during the model learning process and uses them as learning data, thereby making the model's discrimination boundary clearer, especially through a contrastive loss function. In step 2-3 (S23), if the evaluation results for the search model are less than 95% for input data recognition rate, more than 5% for malfunction rate, more than 3 seconds for response speed, and less than 88% for answer accuracy, the search model is retrained using additional methodologies such as the hard negative method described above.

Step 2-4 (S24) is the step where the context based on the search model learned in Step 2-2 (S22) or re-learned in Step 2-3 (S23) is vectorized and a vector database is built. In Step 2-2 (S22), a search model is learned using queries and contexts, and if the search model is evaluated and the evaluation criteria are met, Step 2-2 (S22) is terminated and the learned search model-based context is vectorized and a vector database is built, or if the evaluation criteria are not met in Step 2-2 (S22), the re-learned search model-based context is vectorized through hard negatives in Step 2-3 (S23) to build a vector database. That is, the context is vectorized and embedded in the learned or re-learned search model and stored as a vector database.

Step 2-5 (S25) is a step in which a dataset for learning a generative model is constructed with queries, searched contexts, and answers based on the search model learned in Step 2-2 (S22) or re-learned in Step 2-3 (S23), and the generative model is learned and evaluated. In Step 2-1 (S21), if the learning of the search model for the Hindi document QA dataset is evaluated through queries and contexts among the queries, contexts, and answers in Step 2-4 (S24), and a database for the search model is built through re-learning, a learning dataset for the generative model is constructed based on the learned or re-learned search model-based queries, searched contexts, and answers, and the generative model is learned and evaluated. That is, the learned or re-learned search model-based queries and searched contexts are input into the generative model to generate answers, and a dataset is constructed with the learned search model-based queries, searched contexts, and answers, and the generative model is learned through this. The evaluation of the learned generative model is performed in the same way as the search model evaluation, including input data recognition rate, malfunction rate, response speed, and answer accuracy. The input data recognition rate evaluates the accuracy of the input data, and the malfunction rate measures errors when asking questions, including hallucination, failure to mask sensitive information, no response, and no response. In addition, the response speed measures the speed of answer output when a question is input. In addition, the answer accuracy checks the accuracy of the answer by asking questions about a set of determined questions/answers. The evaluation is performed in the same way as Step 2-2 (S22). In addition, the learned search model-based queries, searched contexts, and answers of the generative model are vector embedded and stored in a vector database.

Step 2-6 (S26) is the step where expert feedback data is collected when the target performance is not achieved in the evaluation of Step 2-5 (S25) and the interactive language model that reflects the preferences of Hindi users and experts is trained. In Step 2-6 (S26), if the evaluation results of the generative model are less than 95% for input data recognition rate, more than 5% for malfunction rate, more than 3 seconds for response speed, and less than 88% for answer accuracy, the generative model is retrained as an interactive language model that reflects the preferences of Hindi users and experts.

Step 2-7 (S27) is a step in which a RAG framework based on the search model learned in Step 2-2 (S22) or re-learned in Step 2-3 (S23) and the generative model learned in Step 2-5 (S25) or re-learned in Step 2-6 (S26) is constructed and evaluated. If the generative model learned and evaluated in Step 2-5 (S25) satisfies the evaluation criteria, a RAG framework is constructed using the search model learned in Step 2-2 (S22) or re-learned in Step 2-3 (S23) and the generative model of Step 2-5 (S25). In addition, if the generative model is learned and evaluated in Step 2-5 (S25) and does not meet the evaluation criteria, the RAG framework is built using the search model learned in Step 2-2 (S22) or re-learned in Step 2-3 (S23) and the generative model re-learned with the conversational language model reflecting the preferences of Hindi users and experts in Step 2-6 (S26). In other words, once the search model and generative model are learned, the RAG framework based on the search model and generative model is built and evaluated. The evaluation is conducted by input data recognition rate, malfunction rate, response speed, and answer accuracy. The input data recognition rate evaluates the accuracy of the input data, and the malfunction rate measures the number of errors when asking questions, and errors include hallucination, failure to mask sensitive information, no response, and no response. In addition, the response speed measures the output speed of the answer when a question is input. In addition, the answer accuracy checks the accuracy of the answer by asking questions about a set question/answer set.

Step 2-8 (S28) is the step where error cases are analyzed and additional learning is performed when the target performance is not achieved in the evaluation of Step 2-7 (S27). In Step 2-8 (S26), if the evaluation results for the search model and the generation model are less than 95% for input data recognition rate, more than 5% for malfunction rate, more than 3 seconds for response speed, and less than 88% for answer accuracy, the error cases are deeply analyzed and the search model and the generation model are retrained with data that corrects the error cases.

The third step is a step in which fine-tuning is performed on sLLM. FIG. 5 is a detailed flowchart of the third step of the method for constructing a multilingual AI DOC system (10) including Hindi, English, and Korean of the present invention. Referring to FIG. 5, the third step of the method for constructing a multilingual AI DOC system (10) including Hindi, English, and Korean of the present invention may be configured to include the following three steps. Fine-tuning refers to a task of performing additional learning using a specific dataset on an already trained large-scale language model in order to secure high suitability for a specific task or domain. Fine-tuning is classified into full fine-tuning and repurposing, and the present invention can use the method of full fine-tuning. Full fine-tuning refers to a task of fine-tuning the entire pre-learned model including all model parameters. In this method, all layers and parameters of the pre-learned model are updated and optimized to adjust them to the requirements of the target task. This method is generally suitable when there is a large difference between the task and the pre-trained model, or when the model's flexibility and adaptability to the task are required.

Step 3-1 (S31) is the step where the query of the question is input into the search model and the question embedding and document embedding are calculated. In order to fine-tune sLLM based on the RAG framework, it is necessary to investigate whether the question embedding and document embedding of the search model are highly related. To this end, the query is input into the search model and the vector embedding of the question embedding and document embedding is calculated.

Step 3-2 (S32) is the step where related documents are searched by calculating the relevance with Top-K using question embedding and document embedding. The parameters of the large language model are the settings used to control and adjust the output of the model, and these parameters play an important role in determining the way the large language model generates results according to user requests and the characteristics of the results. The main parameters include Temperature, Top-P, and Top-K, and by adjusting them appropriately, results with desired characteristics can be generated. Temperature is a parameter that controls the output diversity of the large language model, and has a value between 0 and 1, with lower values generating more reliable and predictable outputs and higher values generating more diverse and creative outputs. Therefore, it may be appropriate to use low Temperature when searching for results from fact-based information data, and high Temperature when requesting results from diverse and random results. Next, Top-P is also called nuclear sampling, and is a method of selecting the next token based on the cumulative probability distribution. Top-P is a method of selecting the next word by considering only the words whose cumulative sum of probabilities is P%, which filters out words with low probabilities and improves the quality of the answer. Next, Top-K is a parameter that limits the candidate words when the large language model generates results. Top-K limits the number of most likely next tokens to consider at each stage, and the model selects from among the K tokens with the highest probability ranking. In other words, a low K value generates more focused and predictable outputs, but can limit the creativity of the model. On the other hand, a high K value allows for more diverse outputs, but can sometimes generate irrelevant results. Top-K helps maintain a certain level of diversity while preventing the model from becoming too unpredictable. Top-K, similar to Temperature, is a parameter that can control how diverse the large language model will produce results. In the present invention, Top-K can be used to adjust the K value so that the most relevant documents can be searched by specifying the K value in the results from the search model through the question embedding and document embedding. In the present invention, the K value can be set to 10 so that 10 similar results are produced.

Step 3-3 (S33) is the step where sLLM based on the RAG framework is fine-tuned so that questions and related documents are input into the generative model to generate appropriate answers and summaries. The parameters of sLLM can be adjusted so that questions and related documents are input into the generative model to generate appropriate answers and summaries. One or more of Temperature, Top-P, and Top-K can be selected and used.

Hereinafter, the operation of the multilingual AI DOC system (10) including Hindi, English and Korean of the present invention will be described with reference to the drawings. FIG. 6 is a flowchart of the operation method of the multilingual AI DOC system (10) including Hindi, English and Korean of the present invention. Referring to FIG. 6, the operation method of the multilingual AI DOC system (10) including Hindi, English and Korean of the present invention can be configured to include the following five steps.

Step 1 (R10) is the step where a question is input and a document file to be asked is uploaded. Since the present invention is a method of receiving a question about a document written in Hindi and providing an answer in Hindi, English, and Korean, the document to be asked can be uploaded as a document file and the question can be entered in Hindi, English, or Korean. A Hindi document file with an extension such as txt, pdf, or doc is input into this system through an input device. The document can be a commercial document of a public institution or an industrial company, such as various reports, internal regulations, proposals, or plans. The user can inquire about a question related to a Hindi document and receive an answer about the writing method, summary, and important matters of the document.

The second stage (R20) is the stage where the text of the question and document files is chunked and vectorized to calculate the question embedding and document embedding. When the document and question are input, the text of the document and question are chunked and vectorized for each word, and the question embedding and document embedding are calculated through this.

Step 3 (R30) is the step where question embedding and document embedding are input into the search model to search for related documents. Once the question embedding and document embedding are calculated, they are input into the search model to search for related documents. Related documents are searched in order of high relevance through vector embedding.

Step 4 (R40) is the step where questions and related documents are input into the generation model to generate appropriate answers and summaries. Once relevant documents are retrieved from the search model, questions and related documents are input into the generation model to generate answers and summaries.

Step 5 (R50) is the step where the result of the generative model is input into the sLLM prompt and the final result is produced. When the answer and summary, which are the results of the generative model, are generated, they are input into the sLLM prompt and synthesized with the contents of the existing database to produce the final result. In the present invention, the RAG framework can be used to overcome the problems of illusion and recency, which are the shortcomings of the conventional large language model. The final result can be provided in the language desired by the user, such as Hindi, English, and Korean. In addition, the top 10 cases can be extracted and recommended according to the similarity. In the present invention, for inquiries about Hindi documents, a business document summary and report are written, and specifically, main content summary, related content Q&A, and specific content search are implemented. In addition, in the present invention, translations for Hindi and English can be performed through LangChain's Indic-English encoder and English-indic decoder, and translations for English and Korean can be performed through LangChain's translation model.

The system described above may be implemented using hardware components, software components, and/or a combination of hardware components and software components. For example, the system and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing unit may execute an operating system (OS) and one or more software applications running on the OS. In addition, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit is sometimes described as being used alone, but those skilled in the art will appreciate that the processing unit may include multiple processing elements and/or multiple types of processing elements. For example, the processing device may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

The software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and/or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal waves, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over network-connected computer systems, and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known to and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, flash memories, etc. Examples of the program commands include not only machine language codes generated by a compiler but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

Although the embodiments have been described above by way of limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, appropriate results can be achieved even if the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or are replaced or substituted by other components or equivalents.

Therefore, other implementations, other embodiments, and equivalents to the claims are also included in the scope of the claims described below.

10: Multilingual AI DOC System
100 : RAG module
200 : sLLM module

Claims (6)

delete The first stage is where Hindi-based training data, English-based training data, and Korean-based training data are collected and processed to train the sLLM (smaller Large Language Model), an on-premise language model that understands Hindi, English, and Korean;
The learned sLLM of the first stage is constructed as a Hindi learning dataset using commercial documents of public institutions and industries in Hindi, and the sLLM is constructed as a RAG (Retrieval-Augmented Generation) framework by vector embedding the Hindi learning dataset and storing it in a vector database; and
The third stage, in which fine tuning of the sLLM of the second stage is performed; includes Hindi, English and Korean;
The second step is a step 2-1 in which a Hindi document QA dataset is constructed with the structure of a query, context, and answer, a step 2-2 in which a search model is trained using the query and context and the search model is evaluated, a step 2-3 in which the search model is retrained through hard negatives if the target performance is not achieved in the evaluation of the second step, a step 2-4 in which the context based on the search model trained in the second step or retrained in the second step is vectorized and a vector database is constructed, a step 2-5 in which a dataset for training a generative model is constructed with the query, searched context, and answer based on the search model trained in the second step or retrained in the second step, and the generative model is trained and evaluated, a step 2-6 in which expert feedback data is collected and trained as an interactive language model reflecting the preferences of Hindi users and experts if the target performance is not achieved in the evaluation of the second step, and a search model trained in the second step or retrained in the second step and the search model trained in the second step or retrained in the second step and the search model trained in the second step or retrained in the second step are A method for building a multilingual AI DOC system including Hindi, English, and Korean, comprising a step 2-7 in which a RAG framework based on a generative model learned or relearned in steps 2-6 is built and evaluated, and a step 2-8 in which error cases are analyzed and additional learning is performed when the target performance is not achieved in the evaluation in step 2-7. In claim 2,
The evaluations of the above steps 2-2, 2-5 and 2-7 are conducted by measuring the input data recognition rate, malfunction rate, response speed and answer accuracy, and if the above evaluation criteria are met, it is evaluated as good, and if the above evaluation criteria are not met, re-learning is performed. A method for constructing a multilingual AI DOC system including Hindi, English and Korean. In claim 3,
The criteria for the above evaluation are linked to the number of questions and the evaluation criteria are different.
If the number of questions is 50 or more, it is set as the first evaluation criterion, and if the number of questions is less than 50, it is set as the second evaluation criterion.
The above first evaluation criteria are an input data recognition rate of 93% or higher, a malfunction rate of 7% or lower, a response speed of 5 seconds or lower, and an answer accuracy of 85% or lower. If the above criteria are met, it is evaluated as good. If the above criteria are not met, re-learning is performed.
The above second evaluation criteria are an input data recognition rate of 95% or higher, a malfunction rate of 5% or lower, a response speed of 3 seconds or lower, and an answer accuracy of 88% or lower. If the above criteria are met, it is evaluated as good, and if the above criteria are not met, re-learning is performed. This is a method for building a multilingual AI DOC system including Hindi, English, and Korean. In claim 4,
The third step is a method for building a multilingual AI DOC system including Hindi, English, and Korean, including a step 3-1 in which a question query is input into the search model to calculate question embedding and document embedding, a step 3-2 in which a relevance is calculated using Top-K using the question embedding and document embedding to search related documents, and a step 3-3 in which the RAG framework-based sLLM is fine-tuned so that an appropriate answer and summary are generated by inputting the question and related documents into the generation model. delete