TWI866860B - A method and device for generating summary master-slave dialogue information - Google Patents

Abstract

The present invention provides a method and device for generating summary master-slave dialogue information. The method can be applied to the technical field of processing master-slave dialogue, and includes the following steps: obtaining dialogue information; performing dialogue role labeling to form role dialogue sentences with role labels; filtering the role dialogue information; performing feature extraction on the filtered role dialogue information to generate a feature vector of the role dialogue; obtaining a category message prediction probability distribution vector corresponding to a plurality of categories by processing the feature vector of the role dialogue; determining a category summary sentence by comparing the category message prediction probability value of the individual role dialogue sentence in each of the plurality of categories with the category message probability threshold value of each of the categories to determine that the individual role dialogue sentence belongs to the category message of the category; and generating a category summary message. The method provided by the present invention can improve the accuracy of summary information generation, avoid the problem of missing important information caused by extracting the role dialogue information into summary information of each category, and make the calculation faster and save calculation resources. For example, in the scenario of a long-term care institution, the dialogue information between the resident attendant and the cared person can be effectively summarized to facilitate subsequent care services.

Description

A method and device for generating summary master-slave dialogue information

The present invention belongs to the field of summary information generation technology for processing dialogues, and in particular to a summary information generation method and device for master-slave dialogues.

Conversation summaries play an important role in many application scenarios, especially in master-slave interactive dialogue scenarios. For example, in long-term care institutions in Taiwan, the key to completing the dialogue summary task is to summarize the needs summary and solution summary from the dialogue between the resident attendant (master) and the cared-for case (slave). The summarized information helps provide relevant care services to cases with similar problems, and is also helpful in assisting long-term care institutions to make appropriate solutions and provide reasonable care strategies.

At present, the method for generating dialogue summaries is mainly based on the generative method of Seq2Seq neural network model. This method is generally generated recursively from scratch, so there is a problem of low decoding efficiency, and it is not good to effectively reuse the input text. There is a problem that the generated demand summary and solution summary are of poor quality and it is easy to miss the key information in the care dialogue. Furthermore, there is currently no method to form a summary specifically for the characteristics of the master-slave dialogue situation. Therefore, we need to develop a more effective dialogue summary generation method, especially for use in the case of master-slave interactive dialogue, to improve the quality, accuracy and speed of dialogue summaries, and better serve Taiwan's long-term care institutions.

The application example of the present invention provides a method and related device for generating summary information of a master-slave dialogue, aiming to solve the problems of insufficient quality, accuracy and speed of generating summary information of a master-slave dialogue in the prior art.

One aspect of the present invention provides a summary message generation method, which includes the following steps: Acquire dialogue information, the dialogue information includes a plurality of dialogue sentences; Perform dialogue role tagging, compare and identify a dialogue sentence containing a master keyword, and tag the role of the voiceprint feature data corresponding to the dialogue sentence as a master, and tag the role of the voiceprint feature data corresponding to the remaining dialogue sentences as a slave, so that each dialogue sentence forms a role dialogue sentence with a role tag, and all of these role dialogue sentences constitute role dialogue information; Filter the role dialogue information to identify each role dialogue sentence in the role dialogue information as belonging to the master or the slave; Extract features from each character dialogue sentence in the filtered character dialogue information to generate a feature vector of the character dialogue; By inputting the feature vector of the character dialogue into a multi-branch sequence annotation model for processing, a category message prediction probability distribution vector corresponding to a plurality of categories is obtained, each category message prediction probability distribution vector includes a category message prediction probability value corresponding to each character dialogue sentence, and each category message prediction probability value is used to characterize the possibility that each character dialogue sentence belongs to the category; Determine the category summary statement by comparing the predicted probability value of the category message of each of the multiple categories of the individual character dialogue statement with the category message probability threshold of each category. If it is greater than (or equal to) the category message probability threshold, determine that the individual character dialogue statement belongs to the category message of the category; and Generate category summary messages by splicing the character dialogue statements belonging to each category to generate multiple category summary messages.

The step of marking the dialogue role further includes: Comparing and identifying the pre-stored voiceprint feature data of one of the master or the slave with the speech voice in the dialogue information, if it matches the master after the comparison and identification, it is marked as the master, if it does not match the master after the comparison and identification, it is marked as the slave; or if it matches the slave after the comparison and identification, it is marked as the slave, if it does not match the slave after the comparison and identification, it is marked as the master.

It also includes the dialogue between the non-master and the non-slave subordinate characters, and the step of filtering the character dialogue information includes: Deciding to delete or retain the character dialogue sentences between the non-master and the non-slave subordinate characters.

The step of filtering the character dialogue information includes: Calculating the ratio of the number of character dialogue sentences of the subordinate character to the number of character dialogue sentences of the master or slave to which it is subordinate, and comparing them through a predetermined threshold to determine whether to delete or retain.

Among them, the step of determining the category summary sentence includes: when the role dialogue sentence of the master party is determined to belong to a certain category, the role dialogue sentence of the slave party that appears immediately thereafter is directly attributed to the category message of the same category without the probability valve comparison operation, or when multiple role dialogue sentences of the slave party appear immediately thereafter, only part of the sentences are selected for the probability valve comparison judgment, and if the result also belongs to the category message of the category, the remaining unselected part of the sentences are directly attributed to the category message of the same category, and the probability valve comparison judgment is no longer performed.

Another aspect of the present invention provides a device for generating master-slave dialogue information, which can be operated on a computer or mobile device, and the device includes: A speech sound source conversion module, which converts the dialogue sentence sound source into a text file; A dialogue role marking module, which compares and identifies a role dialogue sentence containing a master keyword, and marks the role of the voiceprint feature data corresponding to the role dialogue sentence as a master, and marks the role of the voiceprint feature data corresponding to the rest of the role dialogue sentences as a slave, so that each role dialogue sentence forms a role dialogue sentence with a role mark, and all these role dialogue sentences constitute role dialogue information; A filtering module filters the character dialogue information to identify whether each character dialogue sentence in each character dialogue information belongs to the master or the slave; A feature extraction module extracts features from each character dialogue sentence in the filtered character dialogue information to generate a feature vector of the character dialogue; A probability distribution vector prediction module processes the feature vector of the character dialogue to obtain a category message prediction probability distribution vector corresponding to a plurality of categories, each category message prediction probability distribution vector includes a category message prediction probability value corresponding to each character dialogue sentence, and each category message prediction probability value is used to characterize the possibility that each character dialogue sentence belongs to the category; The category summary statement determination module compares the predicted probability value of the category message of each of the multiple categories of the individual character dialogue statement with the category message probability threshold of each category. If the predicted probability value is greater than (or equal to) the category message probability threshold, the individual character dialogue statement is determined to belong to the category message of the category; and the category summary message generation module splices the character dialogue statements belonging to each category to generate multiple category summary messages.

The dialogue role marking module further includes comparing and identifying the pre-stored voiceprint feature data of one of the master or the slave with the speech voice in the dialogue information. If the voiceprint feature data matches the master after the comparison and identification, the master is marked as the master. If the voiceprint feature data does not match the master after the comparison and identification, the slave is marked as the slave. Or if the voiceprint feature data matches the slave after the comparison and identification, the slave is marked as the slave. If the voiceprint feature data does not match the slave after the comparison and identification, the master is marked as the slave.

It also includes the dialogue between the subordinate characters who are not the master and the slave, and the filtering module performs the step of filtering the role dialogue information including: Decide whether to delete or retain the role dialogue sentences between the subordinate characters who are not the master and the slave.

The filtering module calculates the ratio of the number of dialogue sentences of the subordinate character to the number of dialogue sentences of the master or slave to which the subordinate character belongs, and compares them with a predetermined threshold value to determine whether to delete or retain.

Among them, the category summary sentence determination module is set to, when the role dialogue sentence of the master party is determined to belong to a certain category, the role dialogue sentence of the slave party that appears immediately thereafter is directly attributed to the category message of the same category without the probability valve comparison operation, or if multiple role dialogue sentences of the slave party appear immediately thereafter, only some of the sentences are selected for the probability valve comparison judgment, and if the result also belongs to the category message of the category, then the remaining unselected part of the sentences are directly attributed to the category message of the same category, and the probability valve comparison judgment is no longer performed.

The device for generating summary master-slave dialogue information further includes: Database, including: Voice database, storing or being used to store voice data of at least one of the master and the slave; and Role determiner keyword database, storing keywords related to the role of the master or the names of the master and/or the slave as keywords.

The dialogue role marking module further includes a role determiner to perform keyword comparison and identification.

Another aspect of the present invention provides a computer device, including: a memory, a transceiver, a processor, and a bus system; wherein the memory is used to store programs; the processor is used to execute the programs in the memory, including executing the methods of the above aspects; the bus system is used to connect the memory and the processor so that the memory and the processor communicate. Another aspect of the present invention provides a computer-readable storage medium, in which instructions are stored, and when the instructions are run on the computer, the computer executes the methods of the above aspects.

Another aspect of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.

From the above technical methods, it can be seen that the application examples of the present invention have significant advantages in the context of master-slave dialogue, such as long-term care. The present invention provides a summary message generation method and related devices for master-slave dialogue, which are particularly suitable for communication between resident attendants or social workers (master) and cared-for cases (slave).

In the long-term care field, it can effectively extract key information from the dialogue between residents or social workers and cared cases, such as service quality (main party), health status (slave party), and other service needs reported by the case (slave party), and generate a structured summary. This is crucial to improving the communication efficiency of long-term care services, recording and tracking case status, and providing personalized care recommendations.

In summary, the application example of the present invention uses the above method to effectively generate summary messages of various categories, which is particularly suitable for scenarios of master-slave dialogues, such as dialogue summary generation in long-term care settings, thereby improving the quality, accuracy and speed of dialogue summaries and increasing practicality.

In the embodiments of the present invention, the term "or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects before and after are in an "or" relationship.

In the embodiments of the present invention, the term "plurality" refers to two or more, and other quantifiers are similar.

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making progressive improvements are within the scope of protection of the present invention.

The method and the device are based on the same concept. Since the principles of the method and the device for solving problems are similar, the implementation of the device and the method can refer to each other, and the repetitions will not be repeated.

The application example of the present invention provides a summary message generation method and related device combined with master-slave dialogue markers, aiming to solve the problems of insufficient quality, accuracy and speed of summary messages generated based on dialogue messages in the prior art, and is particularly suitable for the scenario of master-slave dialogue.

In dialogue situations, there are often master-slave dialogues, in addition to the aforementioned dialogues between home attendants or social workers (master) and the cared-for clients (slave), such as the dialogues between doctors (master) and patients (slave), or the dialogues between accountants/lawyers (master) and their clients (slave). In most master-slave dialogue situations, the master mainly leads or controls the dialogue, and the slave follows or is guided by the master. In most master-slave dialogue situations, the master often guides the dialogue with questions, and the slave is often the one who answers the questions. In most master-slave dialogue situations, the master is usually the one who has professional knowledge or is the resource provider, and the slave is usually the one who assists or cooperates with the master.

Please refer to FIG. 1, which is a device 100 for generating master-slave dialogue information of the present invention. The device 100 for generating master-slave dialogue information is a computer program product or a computer program. The computer program product or the computer program includes computer instructions. The computer instructions are stored in a computer-readable storage medium and can be operated on a computer. In a broad sense, the computer includes a computer or a mobile device. The device 100 for generating master-slave dialogue information of the present invention includes: a sentence sound source conversion module 1, a dialogue role labeling module 2, a filtering module 3, a feature extraction module 4, a probability distribution vector prediction module 5, a category summary sentence determination module 6, a category summary message generation module 7, and a database 8. The speech sound source conversion module 1 and the database 8 can be internally installed or externally connected. The external connection means that the other components (modules) are different programs in the same execution hardware, or are not in the same hardware, or are connected through a network cloud.

Among them, the dialogue role labeling module 2 further includes a speech recognition matcher 21 and an optionally set role determiner 22, the filtering module 3 further includes a content matcher 31, the feature extraction module 4 further includes a text summary layer 41, a feature embedding layer 42 and a feature fusion layer 43, the probability distribution vector prediction module 5 further includes a multi-branch sequence annotation model 51, wherein the multi-branch sequence annotation model 51 further includes M (M≧2) category information annotation modules 511, and the database 8 stores a speech database 81 and a role determiner keyword library 82.

Please refer to FIG. 2 , which is a flow chart of the master-slave dialogue information generating method of the present invention. The method includes the steps described below, and the devices involved can be referred to in conjunction with FIG. 1 .

Step S1: Obtaining dialogue information: This step is mainly for the two parties of the master-slave dialogue to obtain the content of the dialogue. The source of the acquisition may be the recording of the on-site dialogue converted into a text file after processing, or it may be an online dialogue through a computer or mobile device, phone, etc. This situation may also include the form of voice dialogue, or the dialogue itself is conducted in text. If the source is voice, when converted into a text file, the conversion method can be manual typing or automatic conversion using a computer program. For example, Figure 3 shows a dialogue between a home attendant or social worker (master) A and a cared case (slave) B in the present invention, and Figure 4 shows a dialogue between a doctor (master) A and a patient (slave) B in the present invention. Both are (may have been converted into) text files. Through this step S1, dialogue information including N dialogue sentences can be collected, where N is an integer greater than 1. In one embodiment of the present invention, the automatic conversion method using a computer program is to convert the sentence sound source into a text file by connecting a sentence sound source conversion module 1 that can be optionally built-in or externally connected.

Step S2: Dialogue role labeling: For the N dialogue sentences collected in the aforementioned step S1, label the role to which the dialogue belongs for each individual dialogue sentence, so that each dialogue sentence has a role label, that is, label whether each sentence is issued by the main party or the slave party in the dialogue. When the source of the acquired dialogue information is a recording of a live dialogue, the marking action can be performed while it is being processed and converted into a text file. For example, each individual dialogue sentence can be marked with dialogue roles while it is being typed manually. If it is a method of automatically converting using a computer program, it can be marked through voice technology, such as voice characteristics (such as different audio frequencies between men and women, or voiceprint matching technology, etc.), and when the sentence sound source is converted into a text file, it can be marked through the dialogue role marking module 2 of the present invention, so that it can be automatically marked as coming from two different people, A and B, and then manually defined (or changed) which of A and B is the main party and which is the follower. When the source of the acquired conversation information is an online conversation through a computer or mobile device, in this case, the sentences from the master's specific computer or mobile device (which may be a specific computer or mobile device or different microphones under the same device) can be directly attributed to the master, and the sentences from the slave's specific device can be attributed to the slave. Taking the characteristics of this embodiment as an example, although it is a live conversation between the master and the slave, the master first wears a specific portable microphone and connects it to the recording device (wired or wireless), while the slave uses the live recording device to collect the sound separately in a wide-angle manner. In one embodiment of the present invention, a sentence sound source conversion module 1 can be selected to be built-in or externally connected to convert the sentence sound source into a text file.

In one embodiment of the present invention, the method of dialogue role labeling can be performed by the dialogue role labeling module 2, which automatically labels the speech source as coming from two different persons, A and B, when the speech source is converted into a text file, and then manually defines (or changes) which of A and B is the main party and which is the subordinate party. Alternatively, a voice database 81 is stored in the database 8 (which can be stored in a storage device of a hardware device, such as a hard disk, etc.), and the voice database 81 stores voice recognition data (such as voiceprint feature data) of known interlocutors (which can be both the master and the slave), or in particular voice recognition data of the master (such as a resident attendant or a social worker). When the sentence sound source conversion module 1 converts the sentence sound source into a text file, the voice recognition comparator 21 of the dialogue role marking module 2 automatically captures the master's voice recognition data, and after comparison and recognition, the one that meets the requirements is marked as the master, and the other party in the dialogue (or the one that does not meet the requirements) is marked as the slave. In this way, the dialogue role marking module 2 can be used to automatically mark the dialogue roles, so as to form role dialogue information in which each individual dialogue sentence in the N dialogue sentences is marked with the dialogue role mark. In another embodiment of the present invention, the voice recognition and matching device 21 of the dialogue role marking module 2 automatically captures the voice recognition data of the master and the slave, and marks them as the master or the slave after matching and recognition.

It is worth noting that although in most situations of master-slave dialogue, there are usually only two people, the master and the slave, in the dialogue, it is still possible that one of the master or the slave, or both the master and the slave, has a subordinate role to participate in or support the dialogue, or just occasionally insert sentences unrelated to the main dialogue during the dialogue. For example, in the dialogue between the resident attendant or social worker (master) and the cared case (slave), the resident attendant or social worker (master) may have his assistant or supervisor, and the cared case (slave) may have his accompanying family members. In the dialogue between the doctor (master) and the patient (slave), the doctor (master) may have an assisting nurse, and the patient (slave) may have his accompanying family members. However, it is understandable that through the above method, the sentences in which the subordinate role appears can also be marked. For example, the voice database in the aforementioned database not only stores the voice recognition data of the host's resident attendant or social worker, but also stores the voice recognition data of the host's assistant or supervisor. In one embodiment, the master's sentence is marked as A, and the slave's sentence is marked as B; and the sentence in which the master's subordinate role appears is marked as A1, and the sentence in which the subordinate role appears is marked as B1.

In one embodiment, the above method is used to identify and mark the sentences of the master as A, the sentences of the subordinate character of the master as A1, and the sentences of the subordinate as B. The speech sources of other people for whom no voice recognition data exists are automatically set as the subordinate character of the subordinate and marked as B1.

In another embodiment of the present invention, the method of dialogue role labeling can be performed by (optionally) setting a role determiner 22 in the dialogue role labeling module 2 to perform role determination. The role determiner 22 captures a predetermined number of sentences of the master and the slave at the beginning of the dialogue, wherein the predetermined number is greater than or equal to 2 (for example, the first 3 sentences of each, which are usually self-introduction sentences, or the first 6 sentences existing in the same recording file). The role determination device 22 can be supplemented by keywords stored in the role determination device keyword library 82 of the database 8, such as "residential attendant", "social worker", "assistant", "inspector", "doctor", "nurse", etc., or "service agency name (social welfare, long-term care, hospital, pediatrics)", etc., or the names of the principal and/or subordinate parties (including surname and/or given name), etc., so that the role determination can be performed and the role label of the sentence can be given. The role determination device 22 can operate independently or assistively with the aforementioned method to perform role determination and labeling according to the situation. For example, in an independently operated embodiment, the method of dialogue role labeling can first be performed by the role determiner 22 to perform role determination. In the case where the source of the dialogue information obtained is the aforementioned online dialogue through a computer or mobile device, the speech of a specific computer or mobile device source (which may be a specific computer or mobile device or different microphones under the same device) determined as the master party can be directly marked as the master party through keywords as described above, and the speech of the other party can be marked as the slave party. In another embodiment of auxiliary operation, the method of dialogue role labeling can first be performed by the role determiner 22 to perform role determination, and directly determine the sentence source of the main party through keywords as described above, and directly capture the main party voice recognition data, and mark the matching one as the main party after comparison and recognition, so as to reduce the number of calculations and resources. In another embodiment, the role determiner 22 may be used to perform role determination, compare and identify a dialogue sentence containing a master keyword, and mark the role of the voice print feature data corresponding to the dialogue sentence as the master (the corresponding master voice print feature data may be stored in the voice database 81 in the database 8 at the same time), and mark the role of the voice print feature data corresponding to the remaining dialogue sentences as the slave (the corresponding slave voice print feature data may be stored in the voice database 81 in the database 8 at the same time), and form role dialogue information with role marks for each dialogue sentence. The actual application and judgment results of the above-mentioned embodiments can be seen from the two sentences before the dialogue process in FIG. 3 , or the one sentence before the dialogue process in FIG. 4 .

Step S3: Filtering the role dialogue information, and deciding to delete or retain the dialogue of the subordinate role: As mentioned above, although in most master-slave dialogue situations, the dialogue is usually conducted by only the master and the slave, it is still possible that the master or the slave, or both the master and the slave, have other subordinate roles participating in the dialogue to support the dialogue or simply inserting sentences unrelated to the main dialogue during the dialogue. In the present invention, the filtering method can be performed by the filtering module 3 to decide to delete or retain the dialogue of the subordinate role, so as to distinguish whether each dialogue sentence belongs to the master or the slave. The filtering method of the filtering module 3 can be set by the user to delete all the dialogues of the subordinate characters, or to keep all of them, or the user can choose to delete or keep them each time the information summary is performed. In another embodiment, the filtering module 3 filters based on the relative proportion of the sentences appearing in the subordinate characters, that is, in the entire dialogue, the ratio of the number of sentences appearing in the subordinate characters to the number of sentences appearing in the master (slave) party to which it is attached, and determines whether to delete (less than or equal to the valve value) or keep (greater than or equal to the valve value) through a valve value. In another embodiment, the filtering module 3 determines whether to delete (less than or equal to the valve value) or keep (greater than or equal to the valve value) based on the number of sentences appearing in the subordinate characters. The method of the above embodiment is particularly effective in filtering out situations where the auxiliary character merely inserts sentences irrelevant to the main dialogue or the information of the entire dialogue is not very important.

In another embodiment, the filtering module 3 first compares the content of the sentences in which the subordinate role appears through the content matcher 31, that is, the sentences in which the subordinate role appears are compared with the content of the sentences of the main (slave) party to which it is attached that appear before and after the sentences, determine the content similarity ratio, and determine whether to delete (greater than or equal to the threshold) or retain (less than or equal to the threshold) through a threshold. This embodiment is particularly helpful in the case where the subordinate role is a subordinate servant, such as the accompanying family member of the cared case (slave) or the accompanying family member of the patient (slave). Because the servant or its subordinate role may just repeat the words spoken by the other person, especially in the long-term care situation, when the cared case (slave) is an older person. In this embodiment, a more detailed embodiment is that the filtering module 3 can delete the sentences with similar contents of the subordinate party through the results of the feedback of the content matcher 31, and retain the corresponding sentences of the subordinate role. This reflects that the sentences of the subordinate role are richer and more sufficient (more content) or longer than the subordinate party to which it is subordinate. Especially in the long-term care situation, when the cared-for case (subject) is an older person, the subordinate role (such as his children) is the main provider of the dialogue content, and the older subordinate party is just a simple agreement out of habit. Or to explain it in another way, the filtering module 3 can also replace the sentence content of the subordinate party with the sentence content of the subordinate role; or in other words, the sentence of the subordinate role is determined to belong to the subordinate party.

In the case where there is no subordinate role participating in the dialogue (i.e., there is no subordinate role's statement), in this step, the role labeling of the dialogue is completed directly using the above step S2 to determine whether each dialogue statement belongs to the master or the slave.

It is understood that the valve values in the above embodiments can be adjusted and set through experience or optimized through training. In addition, when the above decision is to delete or retain, the above "equal to the valve value" will of course only be selected to be set to one of deletion or retention.

Step S4: Extracting features from character dialogue information and generating feature vectors of character dialogue: In one embodiment of the present invention, feature extraction module 4 is used to extract features from character dialogue and generate dialogue feature vectors. These feature vectors include N (no deletion after the aforementioned filtering; if D dialogue sentences are deleted after the aforementioned filtering, then of course the remaining (N-D). The same situation will not be specifically described below.) dialogue sentence feature values, and correspond to the corresponding dialogue sentences. In the feature extraction process, special attention is paid to distinguishing the sentence features of "master" and "slave".

In an embodiment of the present invention, the feature extraction module 4 includes: a text summary layer 41, a feature embedding layer 42 and a feature fusion layer 43. Among them, the text summary layer 41 can be implemented using a BERT model or a BERTSUM model. In one embodiment of the invention, the feature extraction module 4 performs feature extraction in the following manner: taking the character dialogue message as the input of the text summary layer 41 in the feature extraction module 4, performing latent semantic feature extraction on the character dialogue message through the text summary layer 41, and generating a latent semantic vector; taking the character dialogue message as the input of the feature embedding layer 42 in the feature extraction module 4, performing feature mapping on the character dialogue message through the feature embedding layer 42, and generating a character embedding feature vector; taking the latent semantic vector and the character embedding feature vector as the input of the feature fusion layer 43, performing feature fusion, and generating a dialogue feature vector. In the text summary layer 41, the feature extraction module 4 extracts the latent semantic features of the role dialogue information. This step aims to capture the deep meaning and subtle differences in the dialogue, especially in identifying the language characteristics of the dominant and the guided in the dialogue. The role embedding feature vector not only reflects the language characteristics of each role in the dialogue, but also includes the dynamics of the master-slave relationship in the dialogue. The final dialogue feature vectors generated integrate the language details and the dynamics of the master-slave relationship in the dialogue, providing a rich information basis for subsequent dialogue analysis and summary generation.

In this way, the feature extraction module 4 is able to effectively identify and extract key semantics and role dynamics in master-slave dialogues, thus laying a solid foundation for generating high-quality dialogue summaries.

In one implementation of the present invention, the feature extraction module 4 is further optimized to handle the complex dynamics in the master-slave dialogue. This process first involves concatenating the latent semantic vector with the role embedding feature vector to generate a feature concatenation vector. This step combines the deep semantic content and role features in the dialogue, especially in identifying the characteristics of the dominant and the guided in the dialogue.

Next, the feature concatenation vector is input into the self-attention mechanism layer in the feature extraction model 4. Here, through the feature fusion of the self-attention mechanism, the module is able to more accurately identify the key information and master-slave relationships in the conversation. This process generates the final conversation feature vectors, which integrate the linguistic details of the conversation and the dynamics of the master-slave relationship, providing a rich information basis for generating high-quality conversation summaries.

Through these optimizations, the feature extraction module 4 and the category summary message generation module 7 are able to process the master-slave dialogue more effectively and provide more accurate and comprehensive dialogue analysis and summary.

Step S5: Processing by the multi-branch sequence annotation model 51 to obtain M class message prediction probability distribution vectors: In this step, the dialogue feature vector is used as the input of the multi-branch sequence annotation model 51 in the probability distribution vector prediction module 5, and the dialogue feature vector is processed by the M (M≧2) class message annotation modules 511 in the multi-branch sequence annotation model 51 to obtain M (M≧2) class message prediction probability distribution vectors. Among them, each class message prediction probability distribution vector includes N class message prediction probability values, and each class message prediction probability value is used to represent the possibility that the dialogue sentence belongs to the class message of the class.

Specifically, as mentioned above, in most situations of master-slave dialogues, the master mainly leads or controls the conduct of the dialogue, and the slave follows or is guided by the master to conduct the dialogue. Therefore, the M categories may mainly be the information that the master wants to obtain in the dialogue. For example, please refer to Figure 3, which is a dialogue example of a master-slave dialogue information generation method of the present invention, and the device judgment or summary result. This dialogue example is a long-term care field. For example, in a long-term care system, the dialogue between a resident attendant or social worker (master, role A in the dialogue) and a cared-for case (slave, role B in the dialogue) may have 3 (i.e., M=3) categories, as shown in the system judgment or summary column in the figure, including (master) service quality, (slave) health status, and (slave) other service needs. For another example, please refer to FIG. 4 , which is a dialogue example of another master-slave dialogue information generation method of the present invention, and a device judgment or summary result. This dialogue example is a dialogue between a doctor (master) and a patient (slave) in a hospital setting. For example, there may be 4 (i.e., M=4) categories, as shown in the system judgment or summary field, including (slave) symptom description, (master) diagnosis, (master) cause of disease judgment, and (master) medical instructions.

In one embodiment of the present invention, the multi-branch sequence annotation model in the probability distribution vector prediction module 5 is composed of multiple (i.e., M) parallel category information annotation modules 511, which have the same model structure and are composed of a fully connected layer and a compression function layer. The probability distribution vector prediction module 5 performs the following operations: inputting the conversation feature vector into the fully connected layer (Fully Connected Layer) of each of the M category message annotation modules 511, performing fully connected processing on the conversation feature vector through the fully connected layers of the M category message annotation modules 511, and generating M fully connected feature vectors, wherein each fully connected feature vector includes N fully connected feature values; inputting the M fully connected feature vectors into the compression function layer of each of the M category message annotation modules 511. In this step, the recognition and analysis of the master-slave conversation features are further enhanced. The M fully connected feature vectors are then compressed by the compression function layer of the M category information annotation module 511 to generate M category information prediction probability distribution vectors.

Step S6: Determine the category summary sentences: In this step, the category summary sentence determination module 6 determines M groups of category summary sentences from the N dialogue sentences according to the N category message prediction probability values corresponding to each category message prediction probability distribution vector in the M category message prediction probability distribution vectors, wherein the M groups of category summary sentences correspond to the M category messages. These category summary sentences correspond to the M category messages, respectively, including the leading and guided dynamics in the dialogue.

In detail, the category summary sentence determination module 6 can perform the following operations: obtain a first category message probability threshold; use the N dialogue sentences corresponding to the N dialogue sentences whose predicted probability values of the first category messages are greater than or equal to the first category message probability threshold as first category summary sentences; and obtain a second category message probability threshold; use the N dialogue sentences corresponding to the N dialogue sentences whose predicted probability values of the second category messages are greater than or equal to the second category message probability threshold as second category summary sentences. For example, in Figure 3, according to the predicted probability value of each dialogue sentence relative to the above three categories of messages, namely, (master) service quality, (slave) health status, and (slave) other service requirements, the predicted probability value of the message in the category is compared with the probability threshold of the category message. If it is greater than (or equal to) the probability threshold of the category message, it can be determined that the dialogue sentence belongs to the category message. For example, the dialogue sentence of the master A in the dialogue "It's okay, I want to understand the status of our resident service staff. Is there anything that needs to be strengthened?" corresponds to the predicted probability value of the category message of "(master) service quality" category, which is 0.85, which is greater than the probability threshold of this category message (0.8), so it can be determined that this dialogue sentence belongs to the category message of (master) service quality. On the contrary, the conversation sentence of the master A in the dialogue "Grandma's current movement condition does not seem to have improved?" corresponds to the category message prediction probability value (0.2) of the "(master) service quality" category, which is less than the probability threshold value (0.8) of this category message, so it can be judged that this dialogue sentence does not belong to the category message of this (master) service quality; but the category message prediction probability value 0.9 of this sentence corresponding to the "(slave) health status" category is greater than the probability threshold value (0.7) of this category message, so it can be determined that this dialogue sentence belongs to the category message of this (slave) health status. By analogy, based on the predicted probability value of the category message of each dialogue sentence in each category message, and then compared with the probability threshold value of the category message, it can be determined to which category message the dialogue sentence belongs. The results are shown in the system judgment or summary column in Figures 3 and 4, and will not be repeated here. It can be understood that the valve values in the above-mentioned embodiments can be adjusted and set through experience or optimized by training. In addition, when determining whether a dialogue sentence is determined to belong to a certain category of information, the above-mentioned "equal to the probability threshold value of the certain category of information" can of course also be set to determine that it does not belong to the probability threshold value of the certain category of information. In this process, special attention is paid to sentences whose predicted probability values of category messages are higher than the corresponding probability threshold values of the corresponding category information. This includes salient expressions from both the leading and the led parties in the conversation, thus ensuring that the category summary sentences accurately reflect the leading and subordinate relationships in the conversation.

It is worth noting that in the present invention, since the sentences of the "master" and "slave" relationship are marked and distinguished, as mentioned above, in the dialogue of the master-slave situation, in most of the master-slave dialogue situations, the master mainly leads or controls the dialogue, and the slave follows or is guided by the master to conduct the dialogue. Therefore, in an embodiment of the present invention, if a sentence of the master is determined to belong to a category message of a certain category, the sentence of the slave that appears immediately afterwards can be directly attributed to the category message of the category, which can save computing resources. Or if multiple sentences of the slave appear immediately afterwards, only some sentences are selected for computational judgment. If the result also belongs to the category message of the category, the remaining unselected sentences are directly attributed to the category message of the category, and no computational judgment is performed to save computing resources. Or if multiple slave statements appear immediately afterwards, and some of the statements are selected for calculation and judgment, if the result belongs to the category message of "another category", then the predicted probability value of the category message of the aforementioned master statement in this "another category" is checked. If its value (for example, 0.68) is close to the probability threshold value of this "another category" (for example, 0.7) (for example, the absolute value of the difference (0.7-0.68=0.02) or the percentage of the difference (0.02/0.7=2.8%) can be compared with the set valve value to judge), then the statement that should belong to this master is corrected to "another category".

Step S7: Generate category summary message: In this step, the category summary message generation module 7 is used to splice each group of category summary sentences in the M groups of category summary sentences to generate M category summary messages. These summary messages will reflect the master-slave dynamics in the dialogue and provide a concise and comprehensive overview of the dialogue.

That is, the category summary message generation module 7 splices the first category summary sentences to generate the first category summary message, and splices the second category summary sentences to generate the second category summary message. For example, in FIG3, the system determines that the dialogue party A is the long-term care supervisor of the master party and B is the daughter of the client party through the first two sentences of dialogue, and determines that the following four sentences of dialogue are of the category of "(master party) service quality", so the summary is attributed to the category of "(master party) service quality". In one embodiment, the category name of the category to which the dialogue summary belongs (such as "(master party) service quality" here) is further added to the summary result to increase the readability of the summary result. These summary messages will reflect the dynamics of the conversation and provide a concise, easy-to-read and comprehensive overview of the conversation. This process ensures that the conversation summary not only contains the key content of the conversation, but also accurately captures the dynamics of the conversation, thereby providing users with a deep and comprehensive understanding of the conversation. These summary sentences reflect the main expressions and positions of the dominant party in the conversation, as well as the important responses and views of the guided party in the conversation.

Through this method, the example of the present invention not only improves the quality of the dialogue summary, but also can accurately capture and reflect the master-slave relationship in the dialogue, making the summary information more targeted and practical. It is also applicable to other scenarios with master-slave dialogue relationships that require analysis of dialogue dynamics, such as: communication between supervisors and employees, dialogues between teachers and students, etc.

Another aspect of the present invention provides a computer device, including: a memory, a transceiver, a processor and a bus system; wherein the memory is used to store programs; the processor is used to execute the programs in the memory, including executing the methods of the above aspects; the bus system is used to connect the memory and the processor so that the memory and the processor communicate. Another aspect of the present invention provides a computer-readable storage medium, in which instructions are stored, and when the instructions are run on the computer, the computer executes the methods of the above aspects.

Another aspect of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.

In the long-term care field, it can effectively extract key information from the conversation between the resident attendant or social worker and the case, such as care needs, emotional support, case feedback, etc., and generate a structured summary. This is crucial to improving the communication efficiency of long-term care services, recording and tracking case status, and providing personalized care recommendations.

In summary, the application example of the present invention effectively generates summary messages of various categories, and is particularly suitable for generating summaries of conversations between residents or social workers and cases in long-term care settings, thereby improving the quality and practicality of the conversation summaries.

The above contents are further detailed descriptions of the present invention in combination with specific/preferred implementations, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those with ordinary knowledge in this field, variations and improvements can be made without departing from the concept of the present invention, and these are all within the scope of protection of the present invention.

100: Device for generating master-slave dialogue information 1: Sentence sound source conversion module 2: Dialogue role labeling module 3: Filtering module 4: Feature extraction module 5: Probability distribution vector prediction module 6: Category summary sentence determination module 7: Category summary message generation module 8: Database 21: Voice recognition matcher 22: Role determiner 31: Content matcher 41: Text summary layer 42: Feature embedding layer 43: Feature fusion layer 51: Multi-branch sequence annotation model 511: Category message annotation module 81: Voice database 82: Role determiner keyword library S1-S7: Steps

FIG. 1 is a device diagram of the master-slave dialogue information generation method of the embodiment of the present invention; FIG. 2 is a flow chart of the master-slave dialogue information generation method of the embodiment of the present invention; FIG. 3 is a dialogue example of the master-slave dialogue information generation method of the embodiment of the present invention, and the device judgment or summary result; FIG. 4 is another dialogue example of the master-slave dialogue information generation method of the embodiment of the present invention, and the device judgment or summary result.

S1-S7: Steps

Claims (12)

A method for generating summary master-slave dialogue information, the method comprising: Acquiring dialogue information, the dialogue information comprising a plurality of dialogue sentences; Performing dialogue role tagging, comparing and identifying a dialogue sentence containing a master keyword, and tagging the role of the voiceprint feature data corresponding to the dialogue sentence as a master, and the role of the voiceprint feature data corresponding to the remaining dialogue sentences as a slave, so that each dialogue sentence forms a role dialogue sentence with a role tag, and all the role dialogue sentences constitute role dialogue information; Filtering the role dialogue information to identify each role dialogue sentence in the role dialogue information as belonging to the master or the slave; Extract features from each character dialogue sentence in the filtered character dialogue information to generate a feature vector of the character dialogue; Input the feature vector of the character dialogue into a multi-branch sequence annotation model for processing to obtain a category message prediction probability distribution vector corresponding to a plurality of categories, each category message prediction probability distribution vector includes a category message prediction probability value corresponding to each character dialogue sentence, and each category message prediction probability value is used to characterize the possibility that each character dialogue sentence belongs to the category; Determine the category summary sentence by comparing the category message prediction probability value of each category of the individual character dialogue sentence in the plurality of categories with the category message probability threshold value of each category, if it is greater than (or equal to) the category message probability threshold value, then determine that the individual character dialogue sentence belongs to the category message of the category; and Generate a category summary message, and concatenate the character dialogue sentences belonging to each category to generate multiple category summary messages. The method for generating summary master-slave dialogue information as described in claim 1, wherein the step of marking dialogue roles further includes: Comparing and identifying the pre-stored voiceprint feature data of one of the master or the slave with the speech voice in the dialogue information, and if the comparison and identification is consistent with the master, marking it as the master, and if the comparison and identification is inconsistent with the master, marking it as the slave; or if the comparison and identification is consistent with the slave, marking it as the slave, and if the comparison and identification is inconsistent with the slave, marking it as the master. The method for generating summary master-slave dialogue information as described in claim 1 further includes dialogues between subordinate characters that are not the master and the slave, and the step of filtering the role dialogue information includes: Deciding to delete or retain the role dialogue sentences of the subordinate characters that are not the master and the slave. The method for generating summary master-slave dialogue information as described in claim 3, wherein the step of filtering the role dialogue information includes: Calculating the ratio of the number of role dialogue sentences of the subordinate role to the number of role dialogue sentences of the master or the slave to which it is subordinate, and comparing them through a predetermined threshold to determine whether to delete or retain. A method for generating summary master-slave dialogue information as described in claim 1, wherein the step of determining the category summary sentence includes: when the role dialogue sentence of the master party is determined to belong to a certain category, the role dialogue sentence of the slave party that appears immediately thereafter is directly attributed to the category message of the same category without the probability valve comparison operation, or when multiple role dialogue sentences of the slave party appear immediately thereafter, only part of the sentences are selected for the probability valve comparison judgment, and if the result also belongs to the category message of the category, the remaining unselected part of the sentences are directly attributed to the category message of the same category, and the probability valve comparison judgment is no longer performed. A device for generating master-slave dialogue information can be operated on a computer or mobile device, and the device includes: A sentence sound source conversion module, which converts the dialogue sentence sound source into a text file; A dialogue role marking module, which compares and identifies a dialogue sentence containing a master keyword, and marks the role of the voiceprint feature data corresponding to the dialogue sentence as a master, and marks the role of the voiceprint feature data corresponding to the rest of the dialogue sentences as a slave, so that each dialogue sentence forms a role dialogue sentence with a role mark, and all the role dialogue sentences constitute role dialogue information; A filtering module, which filters the role dialogue information to identify whether each role dialogue sentence in the role dialogue information belongs to the master or the slave; The feature extraction module extracts features from each character dialogue sentence in the filtered character dialogue information to generate a feature vector of the character dialogue; The probability distribution vector prediction module processes the feature vector of the character dialogue to obtain a category message prediction probability distribution vector corresponding to a plurality of categories, each category message prediction probability distribution vector includes a category message prediction probability value corresponding to each character dialogue sentence, and each category message prediction probability value is used to characterize the possibility that each character dialogue sentence belongs to the category; The category summary statement determination module compares the predicted probability value of the category message of each of the multiple categories of the individual character dialogue statement with the category message probability threshold of each category. If the predicted probability value is greater than (or equal to) the category message probability threshold, the individual character dialogue statement is determined to belong to the category message of the category; and the category summary message generation module splices the character dialogue statements belonging to each category to generate multiple category summary messages. The device for generating summary master-slave dialogue information as described in claim 6, wherein the dialogue role labeling module further includes comparing and identifying the pre-stored voiceprint feature data of one of the master or the slave with the voice of each sentence in the dialogue information, and if it matches the master after the comparison and identification, it is marked as the master, and if it does not match the master after the comparison and identification, it is marked as the slave; or if it matches the slave after the comparison and identification, it is marked as the slave, and if it does not match the slave after the comparison and identification, it is marked as the master. The device for generating summary master-slave dialogue information as described in claim 6 further includes dialogues between subordinate characters that are not the master and the slave, and the filtering module performs the step of filtering the role dialogue information including: Deciding to delete or retain the role dialogue sentences of the subordinate characters that are not the master and the slave. A device for generating summary master-slave dialogue information as described in claim 8, wherein the filtering module calculates the ratio of the number of role dialogue sentences of the subordinate role to the number of role dialogue sentences of the master or the slave to which it is subordinate, and compares them through a predetermined threshold to decide whether to delete or retain. A device for generating summary master-slave dialogue information as described in claim 6, wherein the category summary sentence determination module is configured such that when the role dialogue sentence of the master party is determined to belong to a certain category, the role dialogue sentence of the slave party that appears immediately thereafter is directly attributed to the category message of the same category without undergoing the probability valve comparison operation, or if multiple role dialogue sentences of the slave party appear immediately thereafter, only some of the sentences are selected for the probability valve comparison judgment, and if the result also belongs to the category message of the category, the remaining unselected partial sentences are directly attributed to the category message of the same category without undergoing the probability valve comparison judgment. The device for generating summary master-slave dialogue information as described in claim 6 further includes a database, including: a voice database storing or being used to store voice data of at least one of the master and the slave; and a role determiner keyword database storing keywords related to the role of the master or the names of the master and/or the slave as keywords. The device for generating summary master-slave dialogue information as described in claim 6, wherein the dialogue role labeling module further includes a role determiner to perform comparison and identification of the keyword.