CN118430538A - Error correction multi-mode model construction method, system, equipment and medium - Google Patents

Abstract

The application discloses an error correction multi-mode model construction method, a system, equipment and a medium, which are used for acquiring audio information; performing voice separation processing and transcription processing on the audio information to obtain a processing result; and combining the processing result with a corresponding real result to obtain a fine tuning sample, inputting a preparation model, adjusting the preparation model based on the fine tuning sample to obtain an error correction multi-modal model, wherein the preparation model comprises a linear layer and a large-scale language model in a frozen state, and the projection layer aligns a voice encoder for human voice separation with the large-scale voice model in the frozen state. Based on the multi-mode large model, two mode information are simultaneously utilized, the strong zero-shot capability is utilized, a more complex conference scene is adapted, two modules of a conference transcription system are optimized, and the accuracy of conference transcription is improved.

Description

A method, system, device and medium for constructing an error correction multimodal model

Technical Field

The present application relates to the field of computer technology, and in particular to a method, system, device and medium for constructing an error correction multimodal model.

Background technique

The main task of the error correction multimodal model construction system is to convert the voice content in the meeting into text form, so that participants can check and record it. When the number of people in the meeting gradually increases and the environment becomes more and more noisy, the existing meeting quality inspection system will face unprecedented challenges. For example, in a large online meeting, dozens or even hundreds of participants speak at the same time, their voices are intertwined, and the background is also mixed with the sound of keyboard tapping, paper shuffling, and occasional doorbells. The amount of information that the system needs to process increases dramatically, and the clarity of the voice signal also drops significantly. This causes the two modules of voice separation and transcription to face huge difficulties in error correction.

The main task of the voice separation module is to separate the voice signals of different people to ensure that everyone's speech can be accurately recognized. However, in a noisy environment, the voice signals of different people often overlap and interfere, making separation extremely difficult. In addition, since everyone's voice characteristics are different, the separation module also needs to have strong recognition capabilities to cope with various complex voice situations. The transcription module is responsible for converting the separated voice signals into text. However, due to the quality of the voice signal and the possible errors in the separation module, the transcription module is also prone to errors in the transcription process. These errors may include recognition errors, missing words, extra words, etc., which seriously affect the accuracy of the transcription results.

Since the voice separation and transcription modules involve two different modalities (speech and text), their error correction schemes are usually separated. This means that when optimizing these two modules, only one of them can be improved, and the mutual influence between them cannot be considered at the same time. This leads to the limitation of the overall performance improvement of the error correction multimodal model construction, which in turn affects the accuracy of the error correction multimodal model construction.

Summary of the invention

Based on the above problems, the present application provides a method, system, device and medium for constructing an error-correcting multimodal model to improve the accuracy of conference transcription.

To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

The first aspect of the present application provides a method for constructing an error correction multimodal model, comprising:

Get audio information;

Performing voice separation and transcription processing on the audio information to obtain a processing result;

The processing result is combined with the corresponding real result to obtain a fine-tuning sample, which is input into a preparation model. The preparation model is adjusted based on the fine-tuning sample to obtain an error-correcting multimodal model, wherein the preparation model includes a linear layer and a large language model in a frozen state, and the projection layer aligns the speech encoder for voice separation with the large speech model in a frozen state.

In a possible implementation, performing voice separation and transcription processing on the audio information to obtain a processing result includes:

Inputting the audio information into a silence detection module, and removing the silence part in the audio information;

The audio information obtained after the elimination is respectively input into a speaker-based automatic speech recognition module and a human voice separation module to obtain processing results. The automatic speech recognition module is used to convert the audio frames of the non-silent part of the audio information into text information using an acoustic model and a language model, and the human voice separation module is used to assign audio frames to different speakers according to speaker characteristics.

In a possible implementation, the processing result includes a first result obtained by processing an automatic speech recognition module, and the first result includes a text unit obtained by analyzing and converting each frame of audio, and the text unit is a phoneme or a part of a word.

In a possible implementation, the processing result includes a second result obtained by processing an automatic speech recognition module, where the second result is obtained by allocating audio frames based on the timbre characteristics and speech speed characteristics of the speaker.

In a possible implementation, the combining of the processing result with the corresponding real result to obtain a fine-tuning sample, before inputting the fine-tuning sample into a preparation model, further includes:

Obtaining a feature label of the processing result;

The step of combining the processing result with the corresponding real result to obtain a fine-tuning sample includes:

The feature label, the processing result, and a true result corresponding to the processing result are combined to obtain a fine-tuning sample.

In a possible implementation, the combining of the processing result with the corresponding real result to obtain a fine-tuning sample, before inputting the fine-tuning sample into a preparation model, further includes:

Obtaining a feature label of the processing result;

The step of combining the processing result with the corresponding real result to obtain a fine-tuning sample and inputting the fine-tuning sample into a preparation model comprises:

Combining the feature labels and the processing results to obtain fine-tuning samples, and inputting the samples into the projection layer in the preparation model;

The feature labels are added to the fine-tuned samples combed by the projection layer and input into the frozen large speech model in the preparation model.

The second aspect of the present application provides a system for constructing an error correction multimodal model, comprising:

A first acquisition unit, used to acquire audio information;

A processing unit, used for performing voice separation and transcription processing on the audio information to obtain a processing result;

The input unit is used to combine the processing result with the corresponding real result to obtain a fine-tuning sample, input the fine-tuning sample into a preparation model, adjust the preparation model based on the fine-tuning sample, and obtain an error-correcting multimodal model. The preparation model includes a linear layer and a large language model in a frozen state, and the projection layer aligns the speech encoder used for voice separation with the large speech model in a frozen state.

In a possible implementation, the system further includes:

A second acquisition unit, used to acquire a feature label of the processing result;

The step of combining the processing result with the corresponding real result to obtain a fine-tuning sample and inputting the fine-tuning sample into a preparation model comprises:

Combining the feature labels and the processing results to obtain fine-tuning samples, and inputting the samples into the projection layer in the preparation model;

The feature labels are added to the fine-tuned samples combed by the projection layer and input into the frozen large speech model in the preparation model.

The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the error correction multimodal model construction method described in the first aspect is implemented.

The fourth aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the error correction multimodal model construction method as described in the first aspect above.

Compared with the prior art, this application has the following beneficial effects:

Combine the speech encoder for voice separation with the large language model to achieve multimodal processing from audio to text. This fusion can not only improve the depth of understanding of speech signals, but also use the rich contextual knowledge of the language model for more accurate transcription and error correction. The speech encoder for voice separation is aligned with the frozen LLM through the projection layer to ensure that the audio features can be effectively combined with the text processing capabilities of the LLM. This alignment method not only improves the efficiency of the model in processing audio data, but also enables the model to more accurately recognize the voice content. Compared with the traditional solution in which the error correction method is often only performed on a single module, this application is based on a multimodal large model that uses two modal information at the same time, and uses its powerful ability to predict or classify new tasks or unseen categories without direct training for specific tasks or categories, to adapt to more complex meeting scenarios and improve the accuracy of meeting transcription.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a flow chart of a method for constructing an error correction multimodal model provided in an embodiment of the present application;

FIG2 is a schematic diagram of a process for constructing an error correction multimodal model provided in an embodiment of the present application;

FIG3 is a hybrid encryption process provided by an embodiment of the present application;

FIG4 is a structural diagram of an error correction multimodal model building system provided in an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

To facilitate understanding of the technical solutions provided by the embodiments of the present application, the technical terms involved in the embodiments of the present application will be explained below.

Zero-shot capability, also known as zero-shot learning, means that the model can still make accurate predictions or inferences about new categories or tasks even if it has not seen or trained the same categories or tasks as the test phase during the training phase. This capability is achieved by advanced deep learning models and transfer learning methods.

To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.

Refer to Figure 1, which is a flow chart of the serial output training scheme provided by the present application, which includes a silence detection (Vad) module, a silence detection (Vad) module, and a voice separation (Diarization) module. Frame-level diarization with SOT (FD-SOT): This refers to a voice separation method based on frame-level serial output training. It processes audio frames to identify different speakers. Multi-talker: It means that there are multiple speakers communicating at the same time. Speaker-attributed ASR: This is an automatic speech recognition system based on speaker features, which can transcribe according to the voice characteristics of each speaker. W1, W2, W3/W4, W5: These represent different audio channels or streams, which can represent recordings of different speakers or different audio sources. Monaural transcription: Monophonic transcription means that the audio signal is processed from a single channel (not stereo or multi-channel). Oracle VAD: An idealized voice activity detector for accurately detecting the beginning and end of speech. Long-term audio: refers to audio recorded for a long time, which may be recordings of meetings or other continuous activities. Spkr 1 & Spkr 2: represent two speakers respectively (or more, but only Spkr 1 and Spkr2 are marked in the figure). Frame-level: refers to the unit of processing and analysis is the audio frame. Speaker Profile: speaker characteristics, used to distinguish different speakers. Diarization: voice separation technology, used to divide mixed speech signals into speech segments of different speakers. Speaker-attributed transcription refers to speaker attribute transcription, that is, associating or attributing the transcribed text with different speakers in the meeting. In a meeting with multiple participants, this transcription method can distinguish and identify the speeches of different speakers, making the transcription results clearer and more organized.

The first process: frame-level diarization with SOT (FD-SOT)

In this process, the error correction multimodal model building system processes multi-speaker audio signals by serial output training. First, the silence detection (Vad module) receives the original audio signal and detects the silent part. After removing the silent part, the audio signal is passed to the speaker-based automatic speech recognition (ASR) module. The ASR module converts the audio frames of the non-silent part into text using the acoustic model and language model. Here, each frame of audio is analyzed and converted into the corresponding text unit, which may be a phoneme or part of a word. Subsequently, the human voice separation (Diarization module) operates at the frame level. It receives the text frames output by the ASR module and tries to assign these frames to different speakers based on speaker characteristics (such as timbre, speaking speed, etc.). This process involves analyzing the acoustic features of the frames to distinguish the speech of different speakers. Finally, the system outputs a sequence of audio frames that have been transcribed into text and labeled with speaker information.

The second process: Word-level Diarization with SOT (WD-SOT)

Similar to the first process, the second process also contains silence detection (Vad module) and speaker-based automatic speech recognition (ASR) modules. However, in the voice separation (Diarization module) stage, the second process operates at a finer granularity. It receives the text output by the ASR module and performs disambiguation at the word level. This means that the Diarization module not only analyzes the acoustic features of each word, but also considers the contextual information and speaker characteristics of the word to more accurately determine the speaker to which each word belongs. Through word-based disambiguation, the system can more accurately identify the speech of different speakers and mark the speaker information of each word in the transcription results. This is especially important for application scenarios that require accurate distinction between speakers (such as meeting records, multi-person conversation analysis, etc.).

First, the diversity of online conference access methods means that participants may access the conference through different devices, platforms or networks, which may lead to differences in the quality and format of the audio signal. The non-fixed participants mean that the participants in each meeting may be different, and their voice characteristics, accents, speaking speed, etc. may also be different. The non-fixed environment means that the meeting may take place in a variety of noisy or quiet environments, and background noise, echo and other problems may also interfere with the construction of the error correction multimodal model.

Since all these complicated scenes are concentrated in one meeting, the amount of information that the error correction multimodal model construction system needs to process is huge and complex. This leads to unsatisfactory results of error correction multimodal model construction, which is mainly reflected in the following aspects:

When the number of participants increases and the environment becomes noisy, the conference quality inspection system will indeed face huge challenges. In such a complex scenario, the error correction work of the two modules of voice separation and transcription becomes particularly critical. Moreover, these two modules involve two modes, voice and text, respectively, which makes their optimization solutions often independent and lack holistic consideration.

Traditional error correction methods often only target a single module, such as optimizing the voice separation module to improve the accuracy of speech recognition, or optimizing the transcription module to improve the quality of text output. However, this method ignores the intrinsic connection and mutual influence between the two modules, resulting in limited improvement in overall performance.

To solve this problem, this application can use the powerful modality alignment capability of MLLM (text, speech) to jointly correct the two modules. MLLM can process data in both text and speech modalities at the same time, and realize cross-modal information fusion and error correction by capturing the intrinsic relationship between them.

It should be noted that the error correction multimodal model construction method, system, device and medium provided in the present application can be applied to the field of computer technology. The above is only an example and does not limit the application field of the error correction multimodal model construction method, system, device and medium provided in the present application. In addition, the embodiments of the present application may not limit the execution subject of the error correction multimodal model construction. For example, the error correction multimodal model construction method of the embodiment of the present application can be applied to data processing devices such as terminal devices or servers. Among them, the terminal device can be an electronic device such as a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc. The server can be a stand-alone server, a cloud server, or a cluster server composed of multiple servers.

In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The following is an example of an error correction multimodal model construction method provided by the present application. See Figure 2, which is a flow chart of an error correction multimodal model construction method provided by an embodiment of the present application. It includes:

S101. Acquire audio information.

In actual application scenarios, a multi-speaker, mono audio input source can be used to provide audio information including acoustic features. This input source may include various conference recordings, interviews, lectures, etc.

S102: performing voice separation processing and transcription processing on the audio information to obtain a processing result.

In a possible implementation, performing voice separation and transcription processing on the audio information to obtain a processing result includes:

The audio information is input into a silence detection module, and the silence part in the audio information is removed; the audio information obtained after the removal is respectively input into a speaker-based automatic speech recognition module and a human voice separation module to obtain a processing result, wherein the automatic speech recognition module is used to convert the audio frames of the non-silent part in the audio information into text information by using an acoustic model and a language model, and the human voice separation module is used to assign the audio frames to different speakers according to speaker characteristics.

ASR (Automatic Speech Recognition) technology can be used to extract speech features from these audios and convert these features into time series data. ASR technology can recognize the speech content in the audio and convert it into text form, which constitutes the predicted results (Predicted Transcriptions).

In a possible implementation, the processing result includes a first result obtained by processing an automatic speech recognition module, and the first result includes a text unit obtained by analyzing and converting each frame of audio, and the text unit is a phoneme or a part of a word.

That is, an ASR model (such as a deep learning model) is used to analyze each frame of audio and convert it into corresponding text units. These text units can be parts of phonemes or words, depending on the training objectives and design of the ASR model.

In a possible implementation, the processing result includes a second result obtained by processing an automatic speech recognition module, where the second result is obtained by allocating audio frames based on the timbre characteristics and speech speed characteristics of the speaker.

By combining the allocation and recognition of audio frames with the speaker's timbre and speech rate characteristics, the second result is finally obtained. This second result not only contains the recognition result of the audio content (i.e., text), but also contains the speaker information corresponding to each text. This enables the conference transcription system to more accurately identify the content of different speakers and conduct subsequent analysis and processing.

S103. Combining the processing result with the corresponding real result to obtain a fine-tuning sample, inputting the fine-tuning sample into a preparation model, adjusting the preparation model based on the fine-tuning sample to obtain an error-correcting multimodal model, wherein the preparation model includes a linear layer and a large language model in a frozen state, and the projection layer aligns the speech encoder for voice separation with the large speech model in a frozen state.

In one possible implementation, the frozen speech model LLM is a pre-trained large language model with powerful language understanding and generation capabilities. Since the LLM has been trained with a large amount of data, it can be frozen, that is, its parameters are no longer updated, and it can be directly used in new tasks. In the construction of the error correction multimodal model, the language understanding ability of the LLM can be used to correct errors in the transcribed text. By inputting the encoded speech features into the LLM, a text representation corresponding to the speech content can be obtained, thereby realizing error correction of the transcribed text.

The role of the projection layer is to align the output of the speech encoder with the input of the LLM. This usually involves converting the output of the speech encoder into a format or dimension suitable for processing by the LLM. While the parameters of the LLM are frozen, the parameters of the projection layer are trainable. By training the projection layer, it can be enabled to effectively pass the output of the speech encoder to the LLM.

The trained projection layer is integrated with the speech encoder and LLM to form a complete error-correcting multimodal model. This model is capable of processing both speech and text information. Error correction mechanisms can be added to the model to use the language processing capabilities of the LLM to correct errors in the transcription. This can be achieved by comparing the text generated by the LLM with the original transcription and identifying and correcting the differences.

Due to the limitations of ASR technology, these prediction results may contain errors or inaccuracies. Therefore, these prediction results can be verified and corrected. This can be done manually, that is, manually listening to the audio content and checking it against the prediction results. The verified and corrected text is the true result (TrueTranscriptions).

Thus, prediction result-true result sample pairs are formed. The prediction results in these sample pairs represent the current performance of ASR technology, while the true results provide an accurate reference standard. Subsequently, these sample pairs are used to fine-tune the multimodal large model. The fine-tuning process is to improve the accuracy of transcription by adjusting the parameters of the model so that it can better identify and correct errors in transcription. The fine-tuned model will have stronger generalization capabilities and can better adapt to complex transcription environments.

Finally, the fine-tuned multimodal large model can be integrated into the error-correcting multimodal model building system to receive conference audio in real time and transcribe it. By leveraging the powerful capabilities of the fine-tuned model, the system can accurately recognize the speech content and correct possible errors, thereby providing high-quality transcription services.

In the whole process, obtaining the predicted result-true result sample pair is a key step. It not only provides strong data support for fine-tuning the model, but also helps evaluate and improve the performance of ASR technology. By continuously optimizing this process, the accuracy and reliability of the error correction multimodal model construction system can be continuously improved.

In a possible implementation, the combining of the processing result with the corresponding real result to obtain a fine-tuning sample, before inputting the fine-tuning sample into a preparation model, further includes:

Obtaining a feature label of the processing result;

Combining the feature labels and the processing results to obtain fine-tuning samples, and inputting the samples into the projection layer in the preparation model;

The feature labels are added to the fine-tuned samples combed by the projection layer and input into the frozen large speech model in the preparation model.

In a possible implementation, the combining of the processing result with the corresponding real result to obtain a fine-tuning sample, before inputting the fine-tuning sample into a preparation model, further includes:

Obtaining a feature label of the processing result;

The step of combining the processing result with the corresponding real result to obtain a fine-tuning sample includes:

The feature label, the processing result, and a true result corresponding to the processing result are combined to obtain a fine-tuning sample.

Different from the above implementation, in this implementation, the feature label is added to the processing result and the sample composed of the processing result to form a sample with the feature label, that is, a fine-tuning sample.

In the application of large language models, in order to enhance the performance of the model in different contexts and conditions, specific labels or indicators are often added to the input data (i.e. prompt). These labels can include language, dialect, gender, emotion, style, etc., to help the model better understand and generate text that meets the requirements.

As for language and dialect labels, their role is to tell the model the language type of the current input text. Since different languages have different grammatical rules and vocabularies, correct language and dialect labels are crucial to ensure that the model generates accurate and appropriate responses. For example, if the model is prompted to generate text in Chinese, but the input content is English, the model may produce confusing responses because it does not know which language's rules should be followed.

Gender labels are equally important, especially in text involving character dialogue or descriptions. Gender labels can help the model understand which pronouns should be used (such as "he" or "she") and other gender-related vocabulary and expressions. This is crucial for generating natural and accurate text.

In addition, labels such as sentiment and style can also help the model better adapt to different application scenarios. For example, when generating advertising copy, using style labels such as "formal" or "humorous" can guide the model to generate text that conforms to a specific style. Similarly, in sentiment analysis, using sentiment labels such as "positive" or "negative" can help the model more accurately identify the emotional tendencies in the text.

In summary, by adding specific labels or indicators to the input content, we can help large language models better understand and generate text that meets the requirements, thereby improving the accuracy and reliability of the model in various application scenarios. It should be noted that the feature labels of this application are not limited to the several labels illustrated above.

In one possible implementation, necessary interfaces can be developed to enable the error correction multimodal model to be seamlessly integrated with the error correction multimodal model building system. In the error correction multimodal model building system, conference audio is received in real time and transcribed and corrected using the error correction multimodal model. The corrected text can be displayed to the conference participants in real time or saved as a conference record.

In summary, the embodiments provided in this application have the following beneficial effects:

1. First, the solution realizes multimodal processing from audio to text by combining a voice-separated speech encoder with a powerful large language model (LLM). This fusion not only improves the depth of understanding of speech signals, but also uses the rich contextual knowledge of the language model for more accurate transcription and error correction.

2. A projection layer is introduced to transform the output of the speech encoder so that it matches the expected input format of the LLM. This step is critical because it ensures effective information transfer between the two different modalities, allowing speech features to directly "talk" to the language model.

3. Use the comparison between the predicted results and the actual transcription results as samples to directly fine-tune the entire multimodal model, rather than relying solely on the ASR (automatic speech recognition) loss function. This approach can more directly optimize end-to-end performance, especially for error correction tasks, because you can directly adjust for transcription errors.

4. Leverage the generalization ability of LLM: The pre-trained LLM has strong zero-shot learning ability, which means that even if the error type does not appear in the fine-tuning dataset, the model is likely to correctly identify and correct it through generalization ability. This is especially important for dealing with diverse transcription scenarios.

5. Before inputting information into LLM, by adding metadata tags such as language, dialect, gender, etc. in the prompt, additional contextual information is provided to the model, which helps the model make more accurate inferences in a specific language or dialect environment.

The above are some specific implementations of the error correction multimodal model construction method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding system for error correction multimodal model construction. The system provided in the embodiment of the present application will be introduced from the perspective of functional modularization. Figure 4 is a structural diagram of an error correction multimodal model construction system provided in the embodiment of the present application.

The system comprises:

A first acquisition unit 110, configured to acquire audio information;

The processing unit 111 is used to perform voice separation and transcription processing on the audio information to obtain a processing result;

The input unit 112 is used to combine the processing result with the corresponding real result to obtain a fine-tuning sample, input the fine-tuning sample into the preparation model, adjust the preparation model based on the fine-tuning sample to obtain an error-correcting multimodal model, the preparation model includes a linear layer and a large language model in a frozen state, and the projection layer aligns the speech encoder used for voice separation with the large speech model in a frozen state.

In a possible implementation, the system further includes:

A second acquisition unit, used to acquire a feature label of the processing result;

The step of combining the processing result with the corresponding real result to obtain a fine-tuning sample and inputting the fine-tuning sample into a preparation model comprises:

Combining the feature labels and the processing results to obtain fine-tuning samples, and inputting the samples into the projection layer in the preparation model;

The feature labels are added to the fine-tuned samples combed by the projection layer and input into the frozen large speech model in the preparation model.

The embodiments of the present application also provide corresponding devices and computer storage media for implementing the error correction multimodal model construction method provided in the embodiments of the present application.

Among them, the device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes, so that the device executes the error correction multimodal model construction method described in any embodiment of the present application.

The computer storage medium stores codes, and when the codes are executed, the device executing the codes implements the error correction multimodal model construction method described in any embodiment of the present application.

It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And/or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and/or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character "/" generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

It should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

It should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. The method for constructing the error correction multi-mode model is characterized by comprising the following steps of:

Acquiring audio information;

Performing voice separation processing and transcription processing on the audio information to obtain a processing result;

And combining the processing result with a corresponding real result to obtain a fine adjustment sample, inputting a preparation model, adjusting the preparation model based on the fine adjustment sample to obtain an error correction multi-modal model, wherein the preparation model comprises a linear layer and a large-scale language model in a frozen state, and the projection layer aligns a voice encoder for separating human voice with the large-scale voice model in the frozen state.

2. The method according to claim 1, wherein the performing the human voice separation processing and the transcription processing on the audio information to obtain a processing result includes:

inputting the audio information into a silence detection module, and removing a silence part in the audio information;

The audio information obtained after the elimination is respectively input into an automatic speech recognition module and a voice separation module based on a speaker to obtain a processing result, wherein the automatic speech recognition module is used for converting an audio frame of a non-mute part in the audio information into text information by utilizing an acoustic model and a language model, and the voice separation module is used for distributing the audio frame to different speakers according to the characteristics of the speaker.

3. The method of claim 2, wherein the processing results include a first result processed by an automatic speech recognition module, the first result including text units that are part of a phoneme or word that each frame of audio was analyzed and converted.

4. The method of claim 2, wherein the processing results include a second result of processing by the automatic speech recognition module, the second result being obtained by assigning audio frames based on a timbre characteristic and a speech rate characteristic of the speaker.

5. The method of claim 1, wherein combining the processing result with the corresponding real result results yields a fine-tuned sample, and further comprising, prior to inputting into the preliminary model:

acquiring a characteristic label of the processing result;

Combining the processing result with the corresponding real result to obtain a fine tuning sample, including:

and combining the feature tag, the processing result and a real result corresponding to the processing result to obtain a fine tuning sample.

6. The method of claim 1, wherein combining the processing result with the corresponding real result results yields a fine-tuned sample, and further comprising, prior to inputting into the preliminary model:

acquiring a characteristic label of the processing result;

And combining the processing result with a corresponding real result to obtain a fine adjustment sample, and inputting the fine adjustment sample into a preparation model, wherein the fine adjustment sample comprises:

combining the feature tag and the processing result to obtain a fine adjustment sample, and inputting the fine adjustment sample into a projection layer in a preparation model;

And adding the characteristic labels into the fine tuning samples which are carded by the projection layer, and inputting the characteristic labels into a frozen large voice model in the preparation model.

7. An error correction multimodal model building system, the system comprising:

The first acquisition unit is used for acquiring the audio information;

The processing unit is used for performing voice separation processing and transcription processing on the audio information to obtain a processing result;

The input unit is used for combining the processing result with a corresponding real result to obtain a fine adjustment sample, inputting a preparation model, adjusting the preparation model based on the fine adjustment sample to obtain an error correction multi-modal model, wherein the preparation model comprises a linear layer and a large language model in a frozen state, and the projection layer aligns a voice encoder for separating human voice with the large voice model in the frozen state.

8. The system of claim 7, wherein the system further comprises:

the second acquisition unit is used for acquiring the characteristic label of the processing result;

And combining the processing result with a corresponding real result to obtain a fine adjustment sample, and inputting the fine adjustment sample into a preparation model, wherein the fine adjustment sample comprises:

combining the feature tag and the processing result to obtain a fine adjustment sample, and inputting the fine adjustment sample into a projection layer in a preparation model;

And adding the characteristic labels into the fine tuning samples which are carded by the projection layer, and inputting the characteristic labels into a frozen large voice model in the preparation model.

9. An electronic device, comprising: memory, a processor, and a computer program stored on the memory and executable on the processor, which when executed, implements the error correction multi-modal model construction method according to any one of claims 1-6.

10. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein instructions, which when run on a terminal device, cause the terminal device to perform the error correction multi-modality model construction method according to any of claims 1-6.