CN118520882A - Medical long text question answering method and device, electronic equipment and storage medium - Google Patents

Abstract

The application discloses a medical long text question answering method, a device, electronic equipment and a storage medium, wherein the method comprises the following steps: multiplying the first expansion coefficient by the original fundamental frequency to obtain a target fundamental frequency; multiplying the second expansion coefficient by the original scaling factor to obtain a target scaling factor; in RoPE coding layers, coding the first embedded data of the semantic unit based on the target basic frequency and the target scaling factor to obtain second embedded data of the semantic unit; dividing second embedded data of a plurality of semantic units into a plurality of groups according to a preset grouping strategy; and inputting the group into an attention layer of the large language model, performing attention calculation on the current group to obtain a calculation result corresponding to the group, and combining the calculation results corresponding to the group to obtain an attention result output by the attention layer. The application is beneficial to improving the reasoning effect of the large language model on the long text question-answering data.

Description

Medical long text question answering method, device, electronic device and storage medium

Technical Field

The present application relates to the field of Internet technology, and in particular to a method, device, electronic device and storage medium for answering long medical texts.

Background Art

Large language models (LLMs) are usually trained on context data in a predefined window size. When the length of the context content exceeds the pre-training length, the performance of the large language model usually declines significantly.

However, in the medical field, data is usually in the form of long text, such as case reports, diagnosis and treatment plans, etc., and in practical applications, it is usually necessary to perform document summaries, question and answer, etc. The limited context length support of text makes large language models lack adaptive attention results in the medical field. Therefore, how to obtain the attention results of large language models is an urgent problem to be solved.

Summary of the invention

The embodiments of the present application provide a medical long text question answering method, device, electronic device and storage medium to solve the technical problem of how to obtain the attention results of a large language model.

In a first aspect, an embodiment of the present application provides a medical long text question answering method, which is applied to an electronic device, wherein the electronic device stores a large language model, and the medical long text question answering method includes:

Acquire medical text data, extract question and answer data from the medical text data, and concatenate multiple question and answer data to obtain long text question and answer data with a target length;

Extracting semantic units from the long text question-answering data, inputting first embedding data of the semantic units into the RoPE coding layer of the large language model, and obtaining the original base frequency and the original scaling factor of the attention logarithm of the RoPE coding layer, wherein the first embedding data is embedding data carrying semantic information of the semantic unit;

Generate a first expansion coefficient according to the target length, the maximum length of the text processed by the large language model in the pre-training stage, and a preset first expansion coefficient generation model, and multiply the first expansion coefficient by the original basic frequency to obtain a target basic frequency;

Obtaining a position index of the semantic unit, generating a second expansion factor according to the position index, the maximum length, and a preset second expansion factor generation model, and multiplying the second expansion factor by the original scaling factor to obtain a target scaling factor;

In the RoPE coding layer, based on the target base frequency and the target scaling factor, the first embedded data of the semantic unit is encoded to obtain second embedded data of the semantic unit, where the second embedded data is embedded data carrying the semantic information and relative position information of the semantic unit;

According to a preset grouping strategy, the second embedded data of the plurality of semantic units are divided into a plurality of groups, wherein the grouping strategy is that the current group contains a preset number of the second embedded data of the semantic units in the previous or next group;

The group is input into the attention layer of the large language model, attention calculation is performed on the current group to obtain the calculation result corresponding to the group, and the calculation results corresponding to the group are merged to obtain the attention result output by the attention layer.

In a possible implementation manner of the first aspect, acquiring medical text data, extracting question and answer data from the medical text data, and concatenating a plurality of the question and answer data to obtain long text question and answer data having a text length of a target length includes:

Acquire the medical text data, extract each question sentence from the medical text data, and combine the question sentence and the answer information corresponding to the question sentence into a question-answer pair;

Multiple question-answer pairs are combined into question-answer data, and multiple question-answer data are concatenated to obtain the long text question-answer data whose text length is the target length.

In a possible implementation manner of the first aspect, dividing the second embedded data of the plurality of semantic units into a plurality of groups according to a preset grouping strategy includes:

Divide the context length of the long text question-answering data by the maximum length of the pre-trained large language model and round up to determine the number of groups;

The second embedded data of a plurality of the semantic units are divided into a plurality of the groups according to the grouping number and the grouping strategy.

In a possible implementation of the first aspect, after inputting the group into the attention layer of the large language model, performing attention calculation on the current group to obtain a calculation result corresponding to the group, and merging the calculation results corresponding to the group to obtain an attention result output by the attention layer, the medical long text question answering method includes:

Select full parameter training or Lora fine-tuning to train the large language model to obtain the trained large language model.

In a possible implementation manner of the first aspect, the first expansion coefficient generation model is:

;

Wherein, x is the first expansion coefficient, is the target length, and c is the maximum length of the text processed by the large language model in the pre-training stage.

In a possible implementation manner of the first aspect, the second expansion coefficient generation model is:

;

Among them, y is the second expansion coefficient, c is the maximum length of the text processed by the large language model in the pre-training stage, m is the position index of the semantic unit, and z is a preset constant.

In a possible implementation manner of the first aspect, the preset number includes one of 1/2 of the packet size, 1/3 of the packet size, 1/5 of the packet size, 1/6 of the packet size, or a combination thereof.

In a second aspect, an embodiment of the present application provides a medical long text question-answering device, which is applied to an electronic device, wherein the electronic device stores a large language model, including:

A first acquisition module is used to acquire medical text data, extract question and answer data from the medical text data, and concatenate multiple question and answer data to obtain long text question and answer data with a target length;

A second acquisition module is used to extract semantic units from the long text question-answering data, input first embedding data of the semantic units into the RoPE coding layer of the large language model, and obtain the original base frequency and the original scaling factor of the attention logarithm of the RoPE coding layer, wherein the first embedding data is the embedding data carrying the semantic information of the semantic unit;

A first generating module, configured to generate a model according to the target length, a maximum length of text processed by the large language model in a pre-training phase, and a preset first expansion coefficient, generate a first expansion coefficient, and multiply the first expansion coefficient by the original base frequency to obtain a target base frequency;

A second generation module is used to obtain a position index of the semantic unit, generate a second expansion factor according to the position index, the maximum length and a preset second expansion factor generation model, and multiply the second expansion factor by the original scaling factor to obtain a target scaling factor;

A coding module, configured to encode, in the RoPE coding layer, the first embedded data of the semantic unit based on the target base frequency and the target scaling factor to obtain second embedded data of the semantic unit, wherein the second embedded data is embedded data carrying the semantic information and relative position information of the semantic unit;

A grouping module, used for dividing the second embedded data of the plurality of semantic units into a plurality of groups according to a preset grouping strategy, wherein the grouping strategy is that the current group contains a preset number of the second embedded data of the semantic units in the previous or next group;

The output module is used to input the group into the attention layer of the large language model, perform attention calculation on the current group, obtain the calculation result corresponding to the group, merge the calculation results corresponding to the group, and obtain the attention result output by the attention layer.

In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the medical long text question and answer method in the first aspect when executing the computer program.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the medical long text question and answer method in the above-mentioned first aspect is implemented.

In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the medical long text question and answer method described in the first aspect above.

The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, the grouping is input into the attention layer of the large language model, and attention calculation is performed on the current group to obtain the calculation result corresponding to the group, and the calculation results corresponding to the group are merged to obtain the attention result output by the attention layer. Therefore, the electronic device can obtain the attention result of the large language model, which is beneficial to improving the efficiency of obtaining the attention result. On the other hand, the attention result can help the large language model understand the relative positions and relationships of the different semantic units in the long text question and answer data, which is beneficial to improving the reasoning effect of the large language model on the long text question and answer data.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

FIG1 is an application scenario diagram of the medical long text question answering method provided in an embodiment of the present application;

FIG2 is a schematic diagram of a flow chart of a medical long text question answering method provided in an embodiment of the present application;

FIG3 is a flow chart of obtaining long text question and answer data provided by an embodiment of the present application;

FIG4 is a schematic block diagram of a medical long text question-answering device provided in an embodiment of the present application;

FIG5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or combinations thereof.

It should also be understood that the term “and/or” used in the specification and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations/steps, nor must they be executed in the order described. For example, some operations/steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

The medical long text question-and-answer method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR)/virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

For example, the electronic device may be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a TV set top box (STB), a customer premises equipment (CPE) and/or other devices for communicating on a wireless system and a next-generation communication system, such as a mobile terminal in a 5G network or a mobile terminal in a future evolved Public Land Mobile Network (PLMN) network, etc.

Please refer to FIG1 , which is an application scenario diagram of the medical long text question answering method provided in an embodiment of the present application, and is described in detail as follows:

Please refer to FIG1 , which is an application scenario diagram of the medical text question answering method provided in an embodiment of the present application, and is described in detail as follows:

The electronic device acquires medical text data, extracts question and answer data from the medical text data, and concatenates a plurality of the question and answer data to obtain long text question and answer data having a text length of a target length.

Among them, medical text data can come from one database or multiple databases.

For the sake of explanation, the medical text data comes from three databases as an example, which is described in detail as follows:

The three databases include database A, database B and database C. The electronic device is connected to database A, database B and database C respectively, and obtains medical text data from database A, database B and database C.

In an embodiment of the present application, the electronic device can be connected to different databases at the same time to obtain medical text data from different databases.

Please refer to Figure 2, which is a flow chart of the medical long text question and answer method provided in an embodiment of the present application. The method can be applied to an electronic device that stores a large language model.

As shown in FIG2 , the medical long text question answering method provided in the embodiment of the present application includes the following steps, which are described in detail as follows:

S201, obtaining medical text data, extracting question and answer data from the medical text data, and concatenating a plurality of the question and answer data to obtain long text question and answer data having a text length of a target length;

Among them, the long text question and answer data is the question and answer data of long text.

There are many types of long text question and answer data. For the sake of explanation, the long text question and answer data include the following types:

Drug development and clinical trial data: During the drug development process, researchers conduct a large number of clinical trials to evaluate the safety and effectiveness of drugs. These trials generate a large amount of text data, such as trial protocols, trial records, and result reports.

Medical education and training data: Medical schools, medical institutions, etc. often organize various medical education and training activities, such as lectures, seminars, online courses, etc. These activities will generate a large amount of teaching materials and learning data, many of which are in the form of long texts.

In the embodiments of the present application, long text question and answer data in the medical field is of great significance for promoting the progress of medical research.

S202, extracting a semantic unit from the long text question-answering data, inputting first embedding data of the semantic unit into the RoPE coding layer of the large language model, and obtaining an original base frequency and an original scaling factor of the attention logarithm of the RoPE coding layer, wherein the first embedding data is embedding data carrying semantic information of the semantic unit;

Among them, the RoPE (Rotary Position Embedding) encoding layer, which means rotational position encoding layer in Chinese, is a position encoding method used in the Transformer architecture.

S203, generating a first expansion coefficient according to the target length, the maximum length of the text processed by the large language model in the pre-training stage, and a preset first expansion coefficient generation model, and multiplying the first expansion coefficient by the original base frequency to obtain a target base frequency;

Wherein, the first expansion coefficient generation model is:

;

Wherein, x is the first expansion coefficient, is the target length, and c is the maximum length of the text processed by the large language model in the pre-training stage.

Among them, when the target length is greater than the maximum length, the first expansion coefficient is greater than 1, and the first expansion coefficient is multiplied by the original basic frequency to obtain the target basic frequency. At this time, the target basic frequency will be greater than the original basic frequency, so the technical effect of amplifying the original basic frequency can be achieved.

S204, obtaining a position index of the semantic unit, generating a second expansion factor according to the position index, the maximum length and a preset second expansion factor generation model, and multiplying the second expansion factor by the original scaling factor to obtain a target scaling factor;

Wherein, the second expansion coefficient generation model is:

;

Among them, y is the second expansion coefficient, c is the maximum length of the text processed by the large language model in the pre-training stage, m is the position index of the semantic unit, and z is a preset constant.

When the position index is greater than the maximum length, the second expansion factor will be greater than 1. The second expansion factor is multiplied by the original scaling factor to obtain the target scaling factor. At this time, the target scaling factor will be greater than the original scaling factor, thereby achieving the technical effect of amplifying the original scaling factor.

The value of z is set by the user or by the system default and is not restricted here.

For ease of explanation, an example is given below:

For example, if z is 1, the second expansion coefficient generation model is:

;

For example, if z is 2, the second expansion coefficient generation model is:

;

In the embodiment of the present application, as the tasks of the large language model change, the preset constants can be readjusted to obtain the best performance.

S205, in the RoPE coding layer, based on the target base frequency and the target scaling factor, encode the first embedded data of the semantic unit to obtain second embedded data of the semantic unit, where the second embedded data is embedded data carrying the semantic information and relative position information of the semantic unit;

Exemplarily, in the RoPE coding layer, encoding the first embedded data of the semantic unit based on the target base frequency and the target scaling factor to obtain the second embedded data of the semantic unit includes:

In the RoPE coding layer, based on an extension mode of RoPE coding, the first embedded data of the semantic unit is encoded to obtain second embedded data of the semantic unit, wherein the extension mode of RoPE coding includes the target base frequency and the target scaling factor.

For ease of explanation, the extension method using RoPE encoding is as follows:

;

Among them, m represents the position index of the Token, b is the original base frequency of RoPE, d is the dimension of the embedding vector, j is the ordinal number of the dimension, and t is the original scaling factor of the attention logarithm.

Wherein, x is the first expansion coefficient, and y is the second expansion coefficient.

in, This is the target fundamental frequency.

in, is the target scaling factor.

when When increasing, will also increase because and is a constant.

when When increasing, Will decrease.

Therefore, when When increasing, will approach 1, will approach .

Among them, the target base frequency amplifies the original base frequency, and the target scaling factor amplifies the original scaling factor. Therefore, the extended method of RoPE encoding can enhance the sensitivity and expression ability of the large language model to the relative position information of the semantic unit, as detailed as follows:

Enhanced position sensitivity: When the original base frequency increases, the difference between different position indices becomes more obvious in terms of rotation angle, which means that the model can more easily distinguish the relative position information of different semantic units.

Improve model expressiveness: When the target scaling factor increases, even a small position change may result in a large rotation angle change, which helps the model capture more subtle position dependencies.

Flexibility: The target base frequency and the target scaling factor can be used to flexibly control the properties of RoPE encoding to adapt to different application scenarios. For example, when processing long sequences, the target base frequency and the target scaling factor can help improve the modeling ability of large language models for long-range dependencies.

In an embodiment of the present application, the target fundamental frequency and the target scaling factor can enhance the sensitivity and expressiveness of RoPE encoding in representing position information, enabling the large language model to better capture positional dependencies in sequence data and improve its performance in various application scenarios.

S206, dividing the second embedded data of the plurality of semantic units into a plurality of groups according to a preset grouping strategy, wherein the grouping strategy is that the current group contains a preset number of the second embedded data of the semantic units in the previous or next group;

The step of dividing the second embedded data of the plurality of semantic units into a plurality of groups according to a preset grouping strategy includes:

Divide the context length of the long text question-answering data by the maximum length of the pre-trained large language model and round up to determine the number of groups;

The second embedded data of a plurality of the semantic units are divided into a plurality of the groups according to the grouping number and the grouping strategy.

The preset number includes one of 1/2 of the group size, 1/3 of the group size, 1/5 of the group size, 1/6 of the group size, or a combination thereof.

For the sake of illustration, an example is given below:

For example, let N be the number of characters contained in each group. Now let's extend the text processing capability of the large language model to 10K. In order to save computing resources, the input long text question and answer data is first divided into several groups for intra-group calculations. The number of groups is calculated as follows: 10000/4096 = 2.441, which is rounded up to 3. Because the group must be an integer, rounding down will result in information loss.

In order to ensure the fluidity and connectivity of information after grouping, a preset number of the second embedded data of the semantic units are reserved at the beginning and end of adjacent groups, the preset number is N/2, and the grouping is as follows:

.

in, A collection of groups.

in, Indicates that the 0th Token to the Nth Token are a group.

in, is the target length. For the last group.

in, The previous group of the last group.

For the sake of illustration, an example is given below:

For example, when N is 4096, in order to ensure information interaction between adjacent groups, the content of the first group is: the 1st semantic unit to the 4096th semantic unit.

The second grouping partially overlaps with the first grouping, and the second grouping is: the 2048th semantic unit to the 6144th semantic unit.

At this time, the semantic units from 2048 to 4096 exist in both the first group and the second group, and the subsequent groupings are also carried out in this way.

S207, input the grouping into the attention layer of the large language model, perform attention calculation on the current grouping to obtain the calculation result corresponding to the grouping, merge the calculation results corresponding to the grouping to obtain the attention result output by the attention layer.

Wherein, after inputting the group into the attention layer of the large language model, performing attention calculation on the current group to obtain the calculation result corresponding to the group, merging the calculation results corresponding to the group to obtain the attention result output by the attention layer, the medical long text question answering method includes:

Select full parameter training or Lora fine-tuning to train the large language model to obtain the trained large language model.

For the sake of illustration, an example is given below:

For example, the attention result is passed to the fully connected layer of the large language model, and the predicted answer output by the fully connected layer based on the attention result is obtained. According to the comparison result between the predicted answer and the preset answer of the question and answer data, the loss value of the loss function is calculated. When the loss value is greater than the preset value, the model parameters of the large language model are updated through full parameter training or Lora fine-tuning. Based on the question and answer data, the large language model after the model parameters are updated is trained until the loss value is less than the preset value, and then the training of the large language model is stopped to obtain the trained large language model.

Among them, the attention results can help the large language model understand the relative positions and relationships of the different semantic units in the long text question and answer data, which is conducive to improving the reliability and rationality of the predicted answers output by the fully connected layer based on the attention results, and further helps to improve the reasoning effect of the large language model on the long text question and answer data.

After the large language model is trained by selecting the full parameter training or Lora fine-tuning method to obtain the trained large language model, the medical long text question answering method includes:

connecting to a medical question-answering system, and obtaining a medical conversation in the medical question-answering system;

In the medical conversation, a current question is obtained, a feature vector of the current question is extracted, the feature vector is input into the trained large language model, and a current answer output by the trained large language model based on the feature vector is obtained.

The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, the grouping is input into the attention layer of the large language model, and attention calculation is performed on the current group to obtain the calculation result corresponding to the group, and the calculation results corresponding to the group are merged to obtain the attention result output by the attention layer. Therefore, the electronic device can obtain the attention result of the large language model, which is beneficial to improving the efficiency of obtaining the attention result. On the other hand, the attention result can help the large language model understand the relative positions and relationships of the different semantic units in the long text question and answer data, which is beneficial to improving the reasoning effect of the large language model on the long text question and answer data.

Please refer to FIG. 3 , which is a flow chart of obtaining long text question and answer data provided by an embodiment of the present application, and is described in detail as follows:

S301, acquiring the medical text data, extracting each question sentence from the medical text data, and combining the question sentence and the answer information corresponding to the question sentence into a question-answer pair;

S302, composing a plurality of the question-answer pairs into question-answer data, and concatenating the plurality of the question-answer data to obtain the long text question-answer data whose text length is the target length.

For ease of explanation, an example is given below:

For example, question-answering data A in the medical field: the number of question-answer pairs is 5,000;

Medical question-answering data B: The number of question-answer pairs is 3,000;

Concatenate question-and-answer data A and question-and-answer data B to obtain long text question-and-answer data with a text length of the target length. The number of question-and-answer pairs in the long text question-and-answer data is 8,000.

The long text question-answering data in the medical field ensures that each question-answer pair maintains its original content and merges each question-answer pair into a larger long text question-answering data. In this way, the long text question-answering data will contain a variety of questions and answers from the medical field, forming a rich and diverse question-answering dataset.

In the embodiment of the present application, the long text question and answer data provides rich contextual information for the large language model, enabling the large language model to more accurately understand and infer the intention and logical relationship behind the question. By being exposed to a large number of question and answer pairs, the large language model can learn how to generate logical answers in different situations, thereby improving the accuracy and relevance of the large language model's answers.

Corresponding to the medical long text question and answer method described in the above embodiment, please refer to Figure 4, which is a schematic block diagram of the medical long text question and answer device provided in the embodiment of the present application. The medical long text question and answer device 400 shown in Figure 4 can be applied to the electronic device in the application scenario diagram shown in Figure 1. Taking the electronic device as an example, the medical long text question and answer device 400 shown in Figure 4 is explained in detail. The medical long text question and answer device 400 may include a first acquisition module 401, a second acquisition module 402, a first generation module 403, a second generation module 404, an encoding module 405, a grouping module 406, and an output module 407.

The first acquisition module 401 is used to acquire medical text data, extract question and answer data from the medical text data, and concatenate multiple question and answer data to obtain long text question and answer data with a target length;

A second acquisition module 402 is used to extract semantic units from the long text question-answering data, input first embedding data of the semantic units into the RoPE coding layer of the large language model, and obtain the original base frequency and the original scaling factor of the attention logarithm of the RoPE coding layer, wherein the first embedding data is the embedding data carrying the semantic information of the semantic unit;

A first generating module 403 is used to generate a model according to the target length, the maximum length of the text processed by the large language model in the pre-training stage, and a preset first expansion coefficient, generate a first expansion coefficient, and multiply the first expansion coefficient by the original basic frequency to obtain a target basic frequency;

A second generation module 404 is used to obtain a position index of the semantic unit, generate a second expansion factor according to the position index, the maximum length and a preset second expansion factor generation model, and multiply the second expansion factor by the original scaling factor to obtain a target scaling factor;

An encoding module 405 is used to encode the first embedded data of the semantic unit in the RoPE encoding layer based on the target base frequency and the target scaling factor to obtain second embedded data of the semantic unit, where the second embedded data is embedded data carrying the semantic information and relative position information of the semantic unit;

A grouping module 406, configured to group the second embedded data of the plurality of semantic units into a plurality of groups according to a preset grouping strategy, wherein the grouping strategy is that the current group contains a preset number of the second embedded data of the semantic units in the previous or next group;

The output module 407 is used to input the group into the attention layer of the large language model, perform attention calculation on the current group, obtain the calculation result corresponding to the group, merge the calculation results corresponding to the group, and obtain the attention result output by the attention layer.

It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, the grouping is input into the attention layer of the large language model, and attention calculation is performed on the current group to obtain the calculation result corresponding to the group, and the calculation results corresponding to the group are merged to obtain the attention result output by the attention layer. Therefore, the electronic device can obtain the attention result of the large language model, which is beneficial to improving the efficiency of obtaining the attention result. On the other hand, the attention result can help the large language model understand the relative positions and relationships of the different semantic units in the long text question and answer data, which is beneficial to improving the reasoning effect of the large language model on the long text question and answer data.

Please refer to FIG. 5 , which is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

As shown in Figure 5, the electronic device 2 of Figure 5 includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, and when the processor 20 executes the computer program 22, the steps in any of the above-mentioned method embodiments are implemented.

The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that FIG5 is merely an example of the electronic device 2 and does not limit the electronic device 2. The electronic device 2 may include more or fewer components than shown in the figure, or may combine certain components, or may include different components, such as input and output devices, network access devices, etc.

The processor 20 is used to run the computer program 22 stored in the memory 21, and implements the following steps when executing the computer program 22:

Acquire medical text data, extract question and answer data from the medical text data, and concatenate multiple question and answer data to obtain long text question and answer data with a target length;

Extracting semantic units from the long text question-answering data, inputting first embedding data of the semantic units into the RoPE coding layer of the large language model, and obtaining the original base frequency and the original scaling factor of the attention logarithm of the RoPE coding layer, wherein the first embedding data is embedding data carrying semantic information of the semantic unit;

Generate a first expansion coefficient according to the target length, the maximum length of the text processed by the large language model in the pre-training stage, and a preset first expansion coefficient generation model, and multiply the first expansion coefficient by the original basic frequency to obtain a target basic frequency;

Obtaining a position index of the semantic unit, generating a second expansion factor according to the position index, the maximum length, and a preset second expansion factor generation model, and multiplying the second expansion factor by the original scaling factor to obtain a target scaling factor;

In the RoPE coding layer, based on the target basic frequency and the target scaling factor, the first embedded data of the semantic unit is encoded to obtain the second embedded data of the semantic unit, where the second embedded data is embedded data carrying the semantic information and relative position information of the semantic unit; according to a preset grouping strategy, the second embedded data of multiple semantic units are divided into multiple groups, and the grouping strategy is that the current group contains a preset number of the second embedded data of the semantic units in the previous or next group; the group is input into the attention layer of the large language model, and the attention calculation is performed on the current group to obtain the calculation result corresponding to the group, and the calculation results corresponding to the groups are merged to obtain the attention result output by the attention layer.

In some embodiments, the processor 20 is configured to implement:

Acquire the medical text data, extract each question sentence from the medical text data, and combine the question sentence and the answer information corresponding to the question sentence into a question-answer pair;

Multiple question-answer pairs are combined into question-answer data, and multiple question-answer data are concatenated to obtain the long text question-answer data whose text length is the target length.

In some embodiments, the processor 20 is configured to implement:

Divide the context length of the long text question-answering data by the maximum length of the pre-trained large language model and round up to determine the number of groups;

The second embedded data of a plurality of the semantic units are divided into a plurality of the groups according to the grouping number and the grouping strategy.

In some embodiments, the processor 20 is configured to implement:

Select full parameter training or Lora fine-tuning to train the large language model to obtain the trained large language model.

In some embodiments, the processor 20 is configured to implement:

The first expansion coefficient generation model is:

;

Wherein, x is the first expansion coefficient, is the target length, and c is the maximum length of the text processed by the large language model in the pre-training stage.

In some embodiments, the processor 20 is configured to implement:

The second expansion coefficient generation model is:

;

Among them, y is the second expansion coefficient, c is the maximum length of the text processed by the large language model in the pre-training stage, m is the position index of the semantic unit, and z is a preset constant.

In some embodiments, the processor 20 is configured to implement:

The preset number includes one of 1/2 of the group size, 1/3 of the group size, 1/5 of the group size, 1/6 of the group size, or a combination thereof.

The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 2. Further, the memory 21 may also include both an internal storage unit and an external storage device of the electronic device 2. The memory 21 is used to store an operating system, an application program, a boot loader, data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is to be output.

It should be noted that the information interaction, execution process, etc. between the above-mentioned devices/units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

The computer-readable storage medium stores program codes, which can be called by a processor to execute the medical long text question-answering method described in the above method embodiment.

The computer-readable storage medium has a storage space for program codes.

The program code includes the code of any step in the medical long text question answering method described in the above method embodiment.

For example, the program code is called by the processor and can execute the following steps:

Acquire medical text data, extract question and answer data from the medical text data, and concatenate multiple question and answer data to obtain long text question and answer data with a target length;

Extracting semantic units from the long text question-answering data, inputting first embedding data of the semantic units into the RoPE coding layer of the large language model, and obtaining the original base frequency and the original scaling factor of the attention logarithm of the RoPE coding layer, wherein the first embedding data is embedding data carrying semantic information of the semantic unit;

Generate a first expansion coefficient according to the target length, the maximum length of the text processed by the large language model in the pre-training stage, and a preset first expansion coefficient generation model, and multiply the first expansion coefficient by the original basic frequency to obtain a target basic frequency;

Obtaining a position index of the semantic unit, generating a second expansion factor according to the position index, the maximum length, and a preset second expansion factor generation model, and multiplying the second expansion factor by the original scaling factor to obtain a target scaling factor;

In the RoPE coding layer, based on the target base frequency and the target scaling factor, the first embedded data of the semantic unit is encoded to obtain second embedded data of the semantic unit, where the second embedded data is embedded data carrying the semantic information and relative position information of the semantic unit;

According to a preset grouping strategy, the second embedded data of the plurality of semantic units are divided into a plurality of groups, wherein the grouping strategy is that the current group contains a preset number of the second embedded data of the semantic units in the previous or next group;

The group is input into the attention layer of the large language model, attention calculation is performed on the current group to obtain the calculation result corresponding to the group, and the calculation results corresponding to the group are merged to obtain the attention result output by the attention layer.

The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

Among them, the computer-readable storage medium can also be an external storage device of the medical long-text question-and-answer device or electronic device, for example, a plug-in hard disk equipped on the medical long-text question-and-answer device or electronic device, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), a non-volatile computer-readable medium (non-transitory computer-readable storage medium), etc.

Since the computer program stored in the computer-readable storage medium can execute any one of the medical long-text question-and-answer methods provided in the embodiments of the present application, the computer-readable storage medium can achieve the beneficial effects that can be achieved by any one of the medical long-text question-and-answer methods provided in the embodiments of the present application. Please see the previous embodiments for details and will not be repeated here.

An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned medical long text question and answer method.

The computer program product is loaded into the electronic device and can execute the following steps:

Acquire medical text data, extract question and answer data from the medical text data, and concatenate multiple question and answer data to obtain long text question and answer data with a target length;

Extracting semantic units from the long text question-answering data, inputting first embedding data of the semantic units into the RoPE coding layer of the large language model, and obtaining the original base frequency and the original scaling factor of the attention logarithm of the RoPE coding layer, wherein the first embedding data is embedding data carrying semantic information of the semantic unit;

Generate a first expansion coefficient according to the target length, the maximum length of the text processed by the large language model in the pre-training stage, and a preset first expansion coefficient generation model, and multiply the first expansion coefficient by the original basic frequency to obtain a target basic frequency;

Obtaining a position index of the semantic unit, generating a second expansion factor according to the position index, the maximum length, and a preset second expansion factor generation model, and multiplying the second expansion factor by the original scaling factor to obtain a target scaling factor;

In the RoPE coding layer, based on the target base frequency and the target scaling factor, the first embedded data of the semantic unit is encoded to obtain second embedded data of the semantic unit, where the second embedded data is embedded data carrying the semantic information and relative position information of the semantic unit;

According to a preset grouping strategy, the second embedded data of the plurality of semantic units are divided into a plurality of groups, wherein the grouping strategy is that the current group contains a preset number of the second embedded data of the semantic units in the previous or next group;

The group is input into the attention layer of the large language model, attention calculation is performed on the current group to obtain the calculation result corresponding to the group, and the calculation results corresponding to the group are merged to obtain the attention result output by the attention layer.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

Based on such understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims (10)

1. A medical long text question-answering method, characterized by being applied to an electronic device, the electronic device storing a large language model, the medical long text question-answering method comprising:

Acquiring medical text data, extracting question-answer data from the medical text data, and performing splicing processing on a plurality of question-answer data to obtain long text question-answer data with a text length being a target length;

extracting a semantic unit from the long text question-answering data, inputting first embedded data of the semantic unit into a RoPE coding layer of the large language model, and acquiring an original basic frequency and an original scaling factor of an attention logarithm of the RoPE coding layer, wherein the first embedded data is embedded data carrying semantic information of the semantic unit;

generating a model according to the target length, the maximum length of the text processed by the large language model in a pre-training stage and a preset first expansion coefficient, generating a first expansion coefficient, and multiplying the first expansion coefficient by the original basic frequency to obtain a target basic frequency;

acquiring a position index of the semantic unit, generating a model according to the position index, the maximum length and a preset second expansion coefficient, generating a second expansion coefficient, and multiplying the second expansion coefficient by the original scaling factor to obtain a target scaling factor;

In the RoPE coding layer, the first embedded data of the semantic unit is coded based on the target basic frequency and the target scaling factor to obtain second embedded data of the semantic unit, wherein the second embedded data is the embedded data carrying the semantic information and the relative position information of the semantic unit;

Dividing the second embedded data of the semantic units into a plurality of groups according to a preset grouping strategy, wherein the grouping strategy is that the current group contains the second embedded data of the semantic units with preset quantity in the previous or next group;

and inputting the group into the attention layer of the large language model, performing attention calculation on the current group to obtain a calculation result corresponding to the group, and merging the calculation results corresponding to the group to obtain an attention result output by the attention layer.

2. The method for medical long text question-answering according to claim 1, wherein the steps of obtaining medical text data, extracting question-answering data from the medical text data, and performing a splicing process on a plurality of question-answering data to obtain long text question-answering data with a text length being a target length include:

Acquiring the medical text data, extracting each question from the medical text data, and combining the question and answer information corresponding to the question into an answer pair;

And combining the question-answer pairs into question-answer data, and splicing the question-answer data to obtain the long text question-answer data with the text length being the target length.

3. The medical long text question-answering method according to claim 1, wherein the dividing the second embedded data of the plurality of semantic units into a plurality of groups according to a preset grouping policy includes:

dividing the context length of the long text question-answering data by the maximum length of the large language model pre-training and rounding up to determine the grouping number;

Dividing the second embedded data of the plurality of the semantic units into a plurality of the packets according to the number of the packets and the grouping policy.

4. The medical long text question-answering method according to claim 1, wherein after the group is input into the attention layer of the large language model, performing attention calculation on the current group to obtain a calculation result corresponding to the group, merging the calculation results corresponding to the group to obtain an attention result output by the attention layer, the medical long text question-answering method comprises:

And selecting a full-parameter training or Lora fine tuning mode, and training the large language model to obtain the trained large language model.

5. The medical long text question-answering method according to claim 1, wherein the first expansion coefficient generation model is:

；

wherein x is the first expansion coefficient, the And c is the maximum length of the text processed by the large language model in the pre-training stage for the target length.

6. The medical long text question-answering method according to claim 1, wherein the second expansion coefficient generation model is:

；

The y is the second expansion coefficient, c is the maximum length of the text processed by the large language model in the pre-training stage, m is the position index of the semantic unit, and z is a preset constant.

7. The medical long text question-answering method according to any one of claims 1 to 6, wherein the preset number includes one of 1/2 of a packet size, 1/3 of a packet size, 1/5 of a packet size, 1/6 of a packet size, or a combination thereof.

8. A medical long text question-answering apparatus, characterized by being applied to an electronic device storing a large language model, comprising:

the first acquisition module is used for acquiring medical text data, extracting question-answer data from the medical text data, and performing splicing processing on a plurality of question-answer data to obtain long text question-answer data with a text length being a target length;

The second acquisition module is used for extracting a semantic unit from the long text question-answer data, inputting first embedded data of the semantic unit into a RoPE coding layer of the large language model, and acquiring an original basic frequency and an original scaling factor of an attention log of the RoPE coding layer, wherein the first embedded data is embedded data carrying semantic information of the semantic unit;

The first generation module is used for generating a model according to the target length, the maximum length of the text processed by the large language model in the pre-training stage and a preset first expansion coefficient, generating a first expansion coefficient, and multiplying the first expansion coefficient by the original basic frequency to obtain a target basic frequency;

The second generation module is used for acquiring the position index of the semantic unit, generating a model according to the position index, the maximum length and a preset second expansion coefficient, generating a second expansion coefficient, and multiplying the second expansion coefficient by the original scaling factor to obtain a target scaling factor;

The encoding module is used for encoding the first embedded data of the semantic unit in the RoPE encoding layer based on the target basic frequency and the target scaling factor to obtain second embedded data of the semantic unit, wherein the second embedded data is the embedded data carrying the semantic information and the relative position information of the semantic unit;

the grouping module is used for dividing the second embedded data of the semantic units into a plurality of groups according to a preset grouping strategy, wherein the grouping strategy is that the current group contains the second embedded data of the semantic units with the preset quantity in the previous or next group;

And the output module is used for inputting the group into the attention layer of the large language model, carrying out attention calculation on the current group to obtain a calculation result corresponding to the group, and combining the calculation results corresponding to the group to obtain an attention result output by the attention layer.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the medical long text question-answering method according to any one of claims 1 to 7 when the computer program is executed by the processor.

10. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the medical long text question-answering method according to any one of claims 1 to 7.