RU2841235C1 - Methods and servers for training model to detect speaker change - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: result is achieved by the following steps: obtaining a data set for training punctuation, comprising first input data including audio data and text data representing speech, and a first label including a sequence of control tokens, training the model using the punctuation training data set to form the trained punctuation model, obtaining the data set for the speaker changing training data, containing the second input data including the second audio data and the second text data, and a second label including a second sequence of control tokens, additional training of the trained punctuation model using the data set for training a speaker change, forming a speaker change model, obtaining working text data and corresponding working audio data, and generating a second working sequence of tokens based on the working audio data and the working text data using the speaker change model.

EFFECT: high accuracy of speech recognition and speaker diarization.

16 cl, 6 dwg

Description

Field of technology to which the invention relates

[01] This technology relates to audio and text processing, in particular to methods and servers for training a model to detect speaker change.

State of the art

[02] Translating speech in a video from the original language to another language can involve a great deal of labor to dub the audio fragments of the original video to be translated. Dubbing technologies play an important role in making content accessible to a global audience, as they help present content in different languages. In general, dubbing involves combining additional or supplementary recordings (dubbed speech) with the originally recorded speech to form a finished audio track for a video.

[03] Traditional voice-over is implemented on the basis of neural networks that are capable of recognizing speech, converting speech into text, dividing the recognized text into separate fragments, assigning phrases to the appropriate speakers, translating the fragments into the target language and generating dubbed audio, which is then superimposed on the original video material.

[04] The dubbed speech may differ from the originally recorded speech in that its start and end points may not coincide with the start and end points of the originally recorded speech. As a result, the translated audio is out of sync and unattractive to viewers. The quality of the dubbed speech synchronization is also reduced due to the change of speakers in the originally recorded speech.

[05] Speaker Change Detection (SCD) in voice-over is a technology-driven process that involves identifying and marking changes in speakers or actors in audio and/or video recordings. Speaker change detection solutions help synchronize dubbed voices with the original video material and/or maintain accurate lip-sync by recognizing when one character's speech ends and another character's speech begins.

[06] A number of methods for detecting speaker change have been proposed in known technical solutions.

[07] US Patent Application 20220254351 A1, “Method and system for correcting speaker diarization using speaker change detection based on text,” published on August 11, 2022 and owned by Works Mobile Japan Corp, discloses a method and system for correcting speaker diarization using speaker change detection based on text. The method for correcting speaker diarization may include speaker diarization for an input audio stream, recognizing speech contained in the input audio stream, converting the speech to text, detecting a speaker change based on the received text, and correcting the speaker diarization based on the detected speaker change.

[08] In the paper “Speech Recognition and Multi-Speaker Diarization of Long Conversations” by Mao et al., published on arxiv.org on November 5, 2020, a method is presented to separately train Automatic Speech Recognition (ASR) and Speaker Diarization (SD) models when the speech fragment boundaries are known. In addition, this paper presents an attention-shift decoding algorithm and data augmentation methods, which, when combined with pre-training of the models, are claimed to improve the performance of Automatic Speech Recognition and Speaker Diarization.

Disclosure of invention

[09] The developers of this technology have identified a number of technical shortcomings that are characteristic of existing voice-over services. The purpose of this technology is to eliminate at least some of the shortcomings of known technical solutions.

[10] In at least some embodiments, methods and a system for providing operation of a voice-over engine are implemented. In general, the voice-over engine is a computer module used in the voice-over process, during which synchronized audio tracks are generated in a target language that differs from the source language. For example, the target audio track can be synchronized with the lip movement and time marks of the original audio in the source language. In some embodiments, the voice-over engine can perform voice morphing to change the pitch, timbre and rhythm of the dubbed audio in accordance with the original audio. The voice-over engine can be used in real time, such as in live broadcasts and video conferences where instant voice-over is required, and can also be used in the post-production stage for pre-recorded content.

[11] In particular, in accordance with the first broad aspect of the present technology, a method for training a model is implemented. The method is implemented on the server. The method comprises obtaining a data set for punctuation training, comprising first input data including audio data and text data representing the speech of a speaker, and a first label including a sequence of control tokens that comprises (1) a control text token indicating a word, and (2) a control punctuation token indicating a punctuation mark and following the control text token, training a model, using the data set for punctuation training, to form a working sequence of tokens based on a combination of working audio data and working text data in order to form a trained punctuation model, obtaining a data set for speaker change training, comprising second input data including second audio data and second text data representing the speech of several speakers, and a second label including a second sequence of control tokens that comprises (1) a second control text token indicating a second word, (2) a second control punctuation token indicating to a second punctuation mark, and (3) a speaker change control token indicating a speaker change and following the second punctuation control token, further training the trained punctuation model using the speaker change training dataset, generating a second working token sequence based on a combination of working audio data and working text data in order to generate a speaker change model, obtaining a working dataset containing working text data and working audio data, generating, using the speaker change model, a second working token sequence containing a working text token, a working punctuation token following the working text token, and a speaker change working token following the working punctuation token, based on a combination of working audio data and working text data, and generating synthesized audio content based on the second working token sequence.

[12] In some embodiments of the method, the speaker change model comprises a submodel for audio data, a submodel for text and a third submodel, and the formation of the second working sequence of tokens involves forming, using the submodel for audio data, a working sequence of vector representations of audio data based on working audio data, forming, using the submodel for text, a working sequence of vector representations of text based on working text data, forming concatenated intermediate output data using the working sequence of vector representations of audio data and the working sequence of vector representations of text, and forming, using the third submodel, a second working sequence of tokens using the concatenated intermediate output data.

[13] In some embodiments of the method, the submodel for audio data is a WavLM model.

[14] In some embodiments of the method, the submodel for the text is the mT5 model.

[15] In some embodiments of the method, the third submodel is a transformer model.

[16] In some embodiments of the method, it further provides for generating, using a speech-to-text conversion model, text data based on audio data, generating, using a speech-to-text conversion model, second text data based on second audio data, and generating, using a speech-to-text conversion model, working text data based on working audio data.

[17] In some embodiments of the method, the control punctuation token follows immediately after the control text token, the speaker change control token follows immediately after the second control punctuation token, the working punctuation token follows immediately after the working text token, and the working speaker change token follows immediately after the working punctuation token.

[18] In accordance with the second broad aspect of the present technology, a method for retraining a pre-trained model is implemented. The method is implemented on the server. The method involves obtaining a data set for training a speaker change, comprising second input data including second audio data and second text data representing the speech of several speakers, and a second label including a second sequence of control tokens, which contains (1) a second control text token indicating a second word, (2) a second control punctuation token indicating a second punctuation mark, and (3) a speaker change control token indicating a speaker change and following the second control punctuation token in the second sequence of control tokens, further training the pre-trained model using the data set for training a speaker change, forming a second working sequence of tokens based on a combination of working audio data and working text data in order to form a speaker change model, wherein in order to form a working sequence of tokens based on a combination of working audio data and working text data, the pre-trained model was trained based on the data set for training punctuation, comprising first input data including audio data and text data representing the speech of a speaker, and a first label including a sequence of control tokens that comprises (1) a control text token indicating a word and (2) a control punctuation token indicating a punctuation mark and following the control text token, obtaining a working data set containing the working text data and the working audio data, generating, using a speaker change model, a second working token sequence comprising a working text token, a working punctuation token following the working text token, and a speaker change working token following the working punctuation token, based on a combination of the working audio data and the working text data, and generating synthesized audio content based on the second working token sequence.

[19] In some embodiments of the method, the control punctuation token follows immediately after the control text token, the speaker change control token follows immediately after the second control punctuation token, the working punctuation token follows immediately after the working text token, and the working speaker change token follows immediately after the working punctuation token.

[20] According to the third broad aspect of the present technology, a server for training the model is implemented. The server comprises at least one processor and at least one physical machine-readable memory storing executable instructions, the execution of which by at least one processor on the server ensures the receipt of a data set for punctuation training, containing first input data, including audio data and text data representing the speech of a speaker, and a first label, including a sequence of control tokens, which contains (1) a control text token indicating a word, and (2) a control punctuation token indicating a punctuation mark and following the control text token, training a model, using the data set for punctuation training, to form a working sequence of tokens based on a combination of working audio data and working text data for the purpose of forming a trained punctuation model, obtaining a data set for training a speaker change, containing second input data, including second audio data and second text data representing the speech of several speakers, and a second label, including a second sequence of control tokens that comprises (1) a second control text token indicating a second word, (2) a second control punctuation token indicating a second punctuation mark, and (3) a speaker change control token indicating a speaker change and following the second control punctuation token, further training the trained punctuation model using the speaker change training dataset, generating a second working token sequence based on a combination of working audio data and working text data in order to generate a speaker change model, obtaining a working dataset containing working text data and working audio data, generating, using the speaker change model, a second working token sequence comprising a working text token, a working punctuation token following the working text token, and a speaker change working token following the working punctuation token based on a combination of working audio data and working text data, and generating synthesized audio content based on the second working sequence tokens.

[21] In some embodiments of the server, the speaker change model comprises a submodel for audio data, a submodel for text, and a third submodel, and generating the second working sequence of tokens involves generating, using the submodel for audio data, a working sequence of vector representations of audio data based on working audio data, generating, using the submodel for text, a working sequence of vector representations of text based on working text data, generating concatenated intermediate output data using the working sequence of vector representations of audio data and the working sequence of vector representations of text, and generating, using the third submodel, a second working sequence of tokens using the concatenated intermediate output data.

[22] In some server implementations, the submodel for audio data is the WavLM model.

[23] In some server implementations, the submodel for text is the mT5 model.

[24] In some server implementations, the third submodel is a transformer model.

[25] In some embodiments of the server, at least one processor further provides, on the server, generating, using a speech-to-text conversion model, text data based on audio data, generating, using a speech-to-text conversion model, second text data based on second audio data, and generating, using a speech-to-text conversion model, working text data based on working audio data.

[26] In some server implementations, the control punctuation token immediately follows the control text token, the speaker change control token immediately follows the second control punctuation token, the working punctuation token immediately follows the working text token, and the working speaker change token immediately follows the working punctuation token.

[27] As used herein, a "server" is a computer program running on appropriate hardware that is capable of receiving requests (e.g., from client devices) over a network and of executing or causing these requests to be executed. Such hardware may be implemented as a single physical computer or a single physical computer system, which is not essential to the present technology. As used herein, the term "server" does not imply that any particular task or all tasks (e.g., received commands or requests) are received, executed or started on the same server (i.e., by the same software and/or hardware), but rather that any number of software or hardware may participate in receiving, transmitting, executing or causing the execution of any tasks or requests or the results of any tasks or requests, and that all of these software and hardware may represent a single server or multiple servers, with the term "at least one server" covering both of these cases.

[28] In the context of this description, a "client device" is any computer hardware capable of running software suitable for performing a given task. Thus, non-limiting examples of client devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, as well as network equipment such as routers, switches and gateways. It should be noted, however, that a device acting as a client in this context does not lose the ability to act as a server for other client devices. The use of the expression "client device" does not exclude the use of multiple client devices to receive, transmit, perform or initiate the performance of any tasks or requests, the results of any tasks or requests, or the steps of any method described in this document.

[29] As used herein, the term "database" means any structured collection of data, regardless of its specific structure, database management software, and computer hardware for storing, using, or otherwise facilitating the use of such data. A database may reside on the same hardware as the process that stores or uses the information stored in the database, or on separate hardware such as a dedicated server or group of servers.

[30] In the context of this description, the expression "information" includes information of any kind or type capable of being stored in a database. Thus, information includes, but is not limited to, audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (reviews, comments, questions, messages, etc.), documents, spreadsheets, word lists, etc.

[31] In the context of this description, the expression "component" means software (in the context of specific hardware) necessary and sufficient to perform the specific functions referred to.

[32] In the context of this description, the expression "computer storage medium" is intended to refer to media of any kind and type, including RAM, ROM, disks (CD-ROM, DVD, floppy disks, hard disks, etc.), USB drives, solid state drives, tape drives, etc.

[33] As used herein, the numerals “first,” “second,” “third,” etc., serve merely to distinguish between the nouns to which they refer, and do not describe any particular relationships between those nouns. Thus, it should be understood that, for example, the terms “first server” and “third server” do not imply the existence of any particular order, type, chronology, hierarchy, or ranking of servers, and the use of these terms (in and of themselves) does not necessarily imply the presence of any “second server” in any particular situation. Furthermore, as occurs in other contexts herein, references to a “first” element and a “second” element do not exclude the possibility that those two elements may in fact be the same element. For example, in some cases the “first” and “second” servers may represent the same software and/or hardware, and in other cases, different software and/or hardware.

[34] Each embodiment of the present technology relates to at least one of the above-mentioned objectives and/or one of the above-mentioned aspects, but not necessarily to all of them. It should be understood that some aspects of the present technology associated with an attempt to achieve the above-mentioned objective may not correspond to this objective and/or may correspond to other objectives not explicitly mentioned herein.

[35] Additional and/or alternative features, aspects and advantages of embodiments of the present technology are contained in the following description, in the attached drawings and in the claims.

Brief description of the drawings

[36] For a more complete understanding of the present technology, as well as its aspects and additional features, reference should be made to the following description, which should be used in conjunction with the accompanying drawings.

[37] Fig. 1 shows a system suitable for implementing non-limiting embodiments of the present technology.

[38] Fig. 2 illustrates a processing pipeline implemented by the server shown in Fig. 1 for generating modified audio based on original audio from a video file, in accordance with at least some embodiments of the present technology.

[39] Fig. 3 schematically shows the architecture of the speaker change detection model implemented by the server shown in Fig. 1.

[40] Fig. 4 shows the first iteration of training performed in the process of punctuation training of the speaker change detection model shown in Fig. 3, and the second iteration of training performed on the server shown in Fig. 1 in the process of further training of the speaker change detection model shown in Fig. 3.

[41] Fig. 5 shows a working iteration of training performed on the server shown in Fig. 1, during the operation of the speaker change detection model shown in Fig. 3.

[42] Fig. 6 is a block diagram of a method implemented on the server shown in Fig. 1 in at least some non-limiting embodiments of the present technology.

Implementation of the invention

[43] Fig. 1 is a schematic representation of a system 100 suitable for implementing non-limiting embodiments of the present technology. It should be understood that the illustrated system 100 is an illustrative embodiment of the present technology. Accordingly, the following description of the system is merely a description of examples illustrating the present technology. This description is not intended to define the scope or limits of the present technology. In some cases, supposedly useful examples of modifications to the system 100 are provided below. They are intended to facilitate understanding and also do not define the scope or limits of the present technology. The presented list of modifications is not exhaustive, and one skilled in the art will understand that other modifications are possible. In addition, the absence of specific examples of modifications does not mean that modifications are impossible and/or that the description contains the only possible embodiment of a particular element of the present technology. One skilled in the art will understand that this is not necessary. It should also be understood that in some cases, the system 100 provides simple embodiments of the present technology and that such embodiments are provided to facilitate understanding. Those skilled in the art should appreciate that various implementations of this technology may be significantly more complex.

[44] In general, the system 100 is capable of providing electronic voice services to a user 102 of an electronic device 104. For example, the system 100 is capable of receiving a video file with audio in a first language, generating audio in a second language, and providing the video file with audio in the second language to the user. Below, a description of at least some elements of the system 100 is given, while it should be understood that without departing from the essence and scope of the present technology, the system 100 may include elements not shown in Fig. 1.

Communication network

[45] The electronic device 104 is connected to the communication network 110 to provide communication with the server 112. In particular, the electronic device 104 can be connected to the server 112 via the communication network 110 to provide online services to the user 102, for example, based on video streaming mechanisms. The communication network 110 provides, among other things, data exchange between the electronic device 104 and the server 112 in the form of one or more data packets.

[46] In some non-limiting embodiments of the present technology, the communication network 110 is implemented as the Internet. In other non-limiting embodiments of the present technology, the communication network 110 may be implemented differently, such as any global communication network, local communication network, private communication network, etc. The manner in which the communication channel (not specifically indicated) between the electronic device 104 and the communication network 110 is implemented depends, among other things, on the manner in which the electronic device 104 is implemented.

[47] For example, in embodiments of the present technology where the electronic device 104 is implemented as a wireless communication device (such as a smartphone), the communication channel may be implemented as a wireless communication channel (such as, among other things, a 3G communication network channel, a 4G communication network channel, a Wi-Fi ^® (Wireless Fidelity) channel, a Bluetooth ^® channel, etc.), but is not limited to this. In examples where the electronic device 104 is implemented as a laptop, the communication channel may be either wireless (e.g., Wi-Fi ^® , Bluetooth ^® , etc.) or wired (e.g., based on an Ethernet connection).

Electronic device

[48] The system 100 includes an electronic device 104 associated with a user 102. Accordingly, the electronic device 104 is in some cases referred to as a “client device,” “end user device,” “client electronic device,” or simply a “device.” It should be noted that no specific mode of operation is offered or provided for the electronic device 104 associated with the user 102, such as the need to log in, the need to register, etc.

[49] There are no particular limitations on the implementation of the electronic device 104, and examples of the implementation of the electronic device 104 may include personal computers (desktop computers, laptops, netbooks, etc.), wireless communication devices (smartphones, cell phones, tablets, etc.) and network equipment (e.g., routers, switches and gateways). The electronic device 104 comprises hardware, software and/or firmware (or a combination thereof) that are known in the art and ensure the execution of a browser application.

[50] In general, a browser application is designed to provide the user 102 with access to one or more network resources, such as web pages. There are no particular restrictions on the implementation of a browser application. One example of a browser application is the Yandex ^™ browser.

[51] The user 102 may use a browser application to access a video streaming platform that broadcasts video content. For example, the electronic device 104 may generate a request indicating the video content that the user 102 wants to view. In some embodiments, the request from the electronic device 104 may also indicate the desired language of the audio that accompanies the video content. In addition, the electronic device 104 may receive a response (not shown) to the request to play the video content and audio in the selected language for the user 102. Typically, the transmission of the request and the receipt of the response by the electronic device 104 may be carried out via the communication network 110. The content of the request and the response may depend, among other things, on whether the video and audio content is transmitted in a live broadcast mode.

Database

[52] The system 100 also comprises a database 150 that is connected to the server 112 and is capable of storing data extracted or otherwise received or generated by the server 112. In general, the database 150 can receive from the server 112 data that has been extracted, received or otherwise generated by the server 112 during processing for temporary and/or permanent storage, and also transmit the stored data to the server 112 for use. It is assumed that, without deviating from the spirit and scope of the present technology, the database 150 can be divided into several distributed databases.

[53] The database 150 is capable of storing data to ensure the operation of the streaming video transmission mechanisms of the server 112. In this regard, the database 150 can store a plurality of digital content elements, including, but not limited to, video and audio files representing multimedia content consumed by the user 102. Examples of digital content elements may include, in particular, digital video recordings, digital movies, digital audio recordings, digital music, website materials, social network materials, etc.

[54] As described below, the database 150 is capable of storing data for training and fine-tuning (i.e., fine-tuning) one or more machine learning models to generate sequences of output tokens, in particular text tokens, punctuation tokens, and speaker change tokens.

Server

[55] The system 100 also includes a server 112, which may be implemented as a conventional computer server. In the presented non-limiting embodiments of the present technology, the server 112 is a single server. In other non-limiting embodiments of the present technology, the functions of the server 112 may be distributed among multiple servers. Without departing from the spirit and scope of the present technology, the server 112 may include one or more processors, one or more physical storage devices, machine-readable instructions and/or additional hardware elements, additional software components, and/or combinations thereof to implement various functions of the server 112.

[56] In general, the server 112 may operate under the control and/or management of a video service provider (not shown), such as, for example, the operator of the Yandex ^™ video streaming platform. It is assumed that the streaming video service provider and the browser application provider may be the same provider. For example, the browser application (in particular, the Yandex ^™ browser) and the streaming video transmission mechanisms (in particular, the Yandex ^™ streaming video transmission mechanisms) may be provided, controlled and/or managed by the same operator or the same organization.

[57] As indicated above, a video streaming engine (not shown) is located on the server 112. In general, the video streaming engine is implemented as a plurality of computer procedures that are used to provide the user 102 with video content with accompanying audio content in one or more languages.

[58] The developers of this technology took into account that a large volume of multimedia content broadcast on the Internet is initially created in English, while many users do not speak English or do not speak English at a level sufficient to properly perceive content in English. In order to ensure the availability of such multimedia content to a wide range of users, traditional solutions either offer subtitles in different languages or dub the audio content in different languages. The developers have developed methods and systems that provide video dubbing services, thanks to which the audio content in one or more languages is generated by the server 112 without human intervention.

[59] In some embodiments, server 112 is capable of providing electronic automatic voice-over services in cases where the original audio content is translated into one or more languages, played in a male or female voice, and can be superimposed on the original video content for playback to user 102.

[60] As described in more detail below, automatic voice-over services can be implemented based on several neural networks that are capable of (1) recognizing speech (converting audio into text), splitting the recognized text into individual fragments (e.g., phrases), translating the fragments into the target language, and generating dubbed content that is superimposed on the original video content. The system can additionally determine the gender of the speaker and synthesize a voice with the corresponding characteristics.

[61] Fig. 2 shows a processing pipeline 200 implemented on a server 112 in some embodiments of the present technology. A first procedure of the processing pipeline 200 is performed by a speech recognition module 202 with respect to source audio content. In general, the first procedure serves to obtain source audio content 251 and generate speech recognition data 252.

[62] The speech recognition data 252 may be the result of automatically recognizing speech or other audio from a video content item. The speech recognition data 252 may include a plurality of generated character strings, wherein each individual generated character string represents a word, phrase, or set of characters spoken by one or more speakers in the video content item. Each generated character string in the speech recognition data 252 may be associated with time information that characterizes a particular time when the generated character string is spoken in the video. For example, the time information for the phrase "good morning" may include the time when the word "good" is spoken and the time when the word "morning" is spoken. The time information may relate to the start and end times of speech (such as timestamps) for each generated character string, or to the start time and duration of speech for each generated character string.

[63] In some embodiments, the speech recognition module 202 is capable of further processing the original audio content to remove non-speech-related noise, such as music sounds, therefrom. In further embodiments, the speech recognition module 202 is capable of inserting punctuation marks and/or further splitting words into word fragments.

[64] In some embodiments, the speech recognition module 202 may include a speech-to-text (STT) model. The speech-to-text model may be a machine learning model, such as a neural network (NN) model. Non-limiting examples of neural network models that may be used to implement the present technology include a recurrent neural network model and a long short-term memory (LSTM) neural network model. In additional embodiments, the machine learning model may be implemented as a transformer model.

[65] In at least some embodiments, an Automatic Speech Recognition (ASR) model is implemented. The ASR model may be implemented as a combination of a seq2seq model and a Convolutional Neural Network (CNN). The seq2seq model may be implemented as a VGG Transformer model. Additionally or optionally, a Connectionist Temporal Classification (CTC) Transformer may also be used for intermediate recognition (e.g., for fragment recognition). In at least some embodiments, the ASR model may be implemented similar to the ASR model disclosed in patent US11145305 of the same applicant, the contents of which are incorporated herein by reference in their entirety.

[66] As described in more detail below, the speech recognition module 202 is capable of detecting a speaker change. Speaker change detection is a technology-oriented process that involves identifying and marking a change in speakers or actors in an audio and/or video recording. Recognizing the moments when some characters end their speech and when other characters begin to speak allows the speech recognition module 202 to provide additional information to other elements of the processing pipeline 200 and thereby improve the process of generating duplicate content.

[67] The second procedure in the processing pipeline 200 includes finding in the speech recognition data 252 phrases spoken by the corresponding speakers in the original audio content 251. In other words, the second procedure may include determining the speaker who generated the corresponding portion of the speech recognition data 252. According to some non-limiting embodiments of the present technology, the second procedure may be performed by a diarization model 301. In general, the diarization model 301 is capable of determining when two consecutive phrases in the speech recognition data 252 are spoken by the same speaker and when they are spoken by different speakers. To do this, the diarization model 301 is capable of taking into account the speaker change tokens generated by the speech recognition module 202.

[68] In particular, when receiving two phrases and a speaker change token indicating a speaker change probability of 0, the diarization model 301 may determine that both phrases were spoken by the same speaker. Conversely, if the probability indicated by the speaker change token between the two phrases exceeds a predetermined probability threshold (e.g., 0.75, 0.85, or 0.95), the diarization model 301 may determine that the two phrases were spoken by different speakers.

[69] In various non-limiting embodiments of the present technology, the diarization model 301 may be implemented as a neural network with an Emphasized Channel Attention, Propagation and Aggregation in Time Delay Neural Network (ECAPA-TDNN) architecture. In a specific non-limiting example, the diarization model 301 may be implemented as described at huggingface.co/speechbrain/spkrec-ecapa-voxceleb. To ensure that the diarization model 301 takes speaker change tokens into account, a clustering algorithm may be additionally applied to the output of the diarization model 301.

[70] In some non-limiting embodiments of the present technology, after applying the diarization model 301, before translating the speech recognition data 252, the server 112 is configured to concatenate the phrases spoken in the original audio content 251 by each speaker to group the phrases in the speech recognition data 252 by speaker.

[71] After applying the diarization model 301 and grouping the phrases in the speech recognition data 252 by speakers, the translation module 204 performs the third procedure of the processing pipeline 200 with respect to the speech recognition data 252. In general, the third procedure serves to obtain the speech recognition data 252 and generate the translation data 253.

[72] The translation data 253 represents a translation of the character strings from the speech recognition data 252 (presented in the original language of the source audio content) into a second language. The translation module 204 may also receive information about the gender of the person speaking in the source audio content. In some cases, the translation module 204 may generate different translation data 253, in particular, depending on whether the speaker is a man or a woman.

[73] It is contemplated that the translation module 204 is capable of employing a plurality of translation models to translate character strings from a source language into one or more other languages.

[74] In this case, no special restrictions are imposed on the method of implementing a particular translation model in the translation module 204. In one embodiment, the translation model can be implemented as a statistical machine translation (SMT) model trained to translate a phrase from a first language into a second language.

[75] In general, statistical machine translation is concerned with the automatic transformation of phrases in one natural language (e.g., French) into another natural language (e.g., English). The first language is called the source language and the second language is called the target language. This process can be viewed as stochastic. There are many variants of statistical machine translation, which differ in the way the translation is modeled. Some solutions are based on string-to-string transformations, others on tree-to-string transformations, and still others use tree-to-tree transformation models. The performance of a statistical machine translation model is assessed using bilingual (parallel) corpora (original-translation pairs) and/or monolingual corpora (examples of target sentences). It is assumed that the server 112 is capable of generating a translation function by training a statistical machine translation model on matched text corpora representing the corresponding language pair. It is assumed that, without deviating from the essence and scope of the present technology, the translation model can be implemented as an encoder-decoder model.

[76] In some embodiments, the translation model may be represented as a neural network with a transformer architecture. The ability of transformer architectures to take into account the broad context introduced by long short-term memory networks and then by the attention mechanism may be of practical importance to the translation model. It is envisaged that additional features, such as the gender of the speaker, may be taken into account in the translation model.

[77] The fourth procedure from the processing pipeline 200 is performed by the speech synthesis module 206 in relation to the translation data 253. In general, the fourth procedure serves to obtain the translation data 253 and generate the speech synthesis data 254.

[78] The speech synthesis data 254 includes translated audio content generated, in particular, based on the translation data 253, and representing phrases spoken in one (or several) other languages. In addition, based on the speech synthesis data 254, the server 112 is capable of generating output audio content 208, containing voiced translated character strings from the translation data 253. Also, the server 112 is capable of providing playback of the output audio content 208 on the electronic device 104 for presentation to the user 102.

[79] According to some non-limiting embodiments of the present technology, the speech synthesis module 206 may include an acoustic model 214 and a vocoder 212. According to some non-limiting embodiments of the present technology, the acoustic model 214 and the vocoder 212 may be implemented as deep neural network models. It should be noted that the acoustic model 214 may generate mel spectrograms, for example, based on phonemes, and the vocoder 212 may synthesize time-varying sound waves, in particular, based on mel spectrograms from a text-to-spectrogram model.

[80] In some embodiments of the present technology, the acoustic model may be implemented as the well-known Tacotron 2 model. In general, the Tacotron 2 model is a sequence-to-sequence recurrent feature prediction network with one or more attention mechanisms that predicts a sequence of frames of a mel spectrogram from an input sequence of symbols. In other embodiments, the vocoder may be implemented as the well-known HiFi-GAN model. In general, the HiFi-GAN architecture includes one generator and two discriminators: multi-scale and multi-period. The generator is a fully convolutional neural network. It uses the mel spectrogram as input and increases its resolution using transposed convolutions until the length of the output sequence matches the temporal resolution of the raw audio signals. After each transposed convolution, a Multi-Receptive Field Fusion (MRF) module is applied. In other embodiments, the vocoder may be implemented similar to the model described in patent application US 2022084499 "Method and server for a text-to-speech processing" ("Method and server for converting text to speech") of the same applicant, published March 17, 2022, the contents of which are incorporated herein by reference in their entirety.

[81] In some embodiments of the present technology, the speech synthesis module 206 may be implemented similar to the speech synthesis module described in patent application RU 2022134630 (“Method and server for generating modified audio for video”) of the same applicant, filed on December 27, 2022, the contents of which are fully incorporated herein by reference.

[82] In some embodiments, the speech synthesis module 206 may also receive information about the gender of the person speaking in the original audio content. In some cases, the speech synthesis module 206 may generate different speech synthesis data 254, in particular, depending on whether the speaker is a man or a woman.

[83] It is obvious that the speech synthesis module 206 may also receive information about the change of speakers between different fragments of the original audio content 251, output by the diarization model 301 before translating the speech recognition data 252. Thus, in some cases, the translation module 204 may generate portions of the speech synthesis data 254, in particular, depending on who this speech is associated with - with the first (current) speaker or the second (other) speaker.

Speaker change detection model

[84] Fig. 3 shows a non-limiting example of a speaker change detection model 300 used on server 112 as part of speech recognition module 202 in at least one embodiment of the present technology.

[85] In general, the speaker change detection model 300 is capable of receiving audio data 350 and corresponding text data 360 as input data and generating in response output data in the form of a sequence of tokens 310 containing information characterizing a speaker change in the content represented by the audio data 350 and the corresponding text data 360.

[86] The speaker change detection model 300 includes an audio data submodel 302 and a text submodel 304, to which the audio data 350 and the text data 360 are transmitted, respectively. In particular, the server 112 is capable of dividing the audio data 350 into a sequence of audio data fragments 355 and transmitting this sequence as input to the audio data submodel 302. In addition, the server 112 is capable of dividing the text data 360 into a sequence of text data fragments 365 and transmitting this sequence as input to the text submodel 304.

[87] In the embodiment shown in Fig. 3, the server 112 is capable of generating a sequence 355 containing 6340 fragments of audio data and a sequence 365 containing 512 fragments of text data. It is contemplated that the total number of fragments in the sequence 355 and the sequence 365 may depend on, among other things, a particular embodiment of the submodel 302 for audio data, a particular embodiment of the submodel 304 for text, the amount of audio data 350 and/or the amount of text data 360.

[88] In this embodiment, the architecture of the submodel 302 for audio data may be implemented similar to the architecture of the WavLM model disclosed in the article by Sanyuan Chen et al., “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing,” published July 2022, the contents of which are incorporated herein by reference in their entirety. The WavLM model, which is based on the transformer model, comprises a convolutional feature encoder and a transformer encoder. The convolutional encoder consists of seven temporal convolution blocks, followed by a normalization layer and a GELU activation function layer. In one embodiment, the temporal convolutions have 512 channels with steps of (5, 2, 2, 2, 2, 2, 2) and kernel widths of (10, 3, 3, 3, 3, 2, 2), so that the output represents audio fragments of approximately 25 ms duration, followed by 20 ms steps. The output in the convolutional representation is masked as the input to the transformer. The transformer has a layer of relative position vector representation based on a convolution with a kernel size of 128 and 16 groups at the bottom. A controlled relative position offset can be used to improve the model, which is encoded depending on the offset between the "key" and the "query" in the self-attention mechanism of the transformer.

[89] In this embodiment, the architecture of the text submodel 304 may be implemented similar to the architecture of the mT5 model disclosed in Linting Xue et al., “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer,” published March 2021, the contents of which are incorporated herein by reference in their entirety.

[90] The speaker change detection model 300 is capable of concatenating the vector representations of the output data of the submodel 302 for audio data with the vector representations of the output data of the submodel 304 for text in order to form concatenated intermediate output data 306 of the speaker change detection model 300. It can be stated that the speaker change detection model 300 is capable of simultaneously processing the audio data 350 and the text data 360 and forming the concatenated intermediate output data 306 based on both (1) the vector representations of the audio data and (2) the vector representations of the text.

[91] In the embodiment shown in Fig. 3, the server 112 is configured to generate concatenated intermediate output data 306 by concatenating 6340 vector representations of audio data and 512 vector representations of text. It is contemplated that the total number of vector representations of audio data and text to be concatenated may depend on, among other things, a particular embodiment of the submodel 302 for audio data, a particular embodiment of the submodel 304 for text, the amount of audio data 350, and/or the amount of text data 360.

[92] Next, the concatenated intermediate output data 306 is passed to a third submodel 308, which is capable of processing the concatenated intermediate output data 306 and generating a sequence of tokens 310 in response. In this embodiment, the architecture of the third submodel 308 is implemented similar to the architecture of the transformer submodel. For example, the third submodel 308 can be implemented as a transformer model capable of receiving concatenated vector representations of audio data and text (e.g., the concatenated intermediate output data 306) and generating a sequence of tokens (e.g., the sequence of tokens 310).

[93] In the context of the present technology, the speaker change detection model 300 is capable of generating a token sequence 310 that comprises at least three types of tokens, namely (1) word tokens, (2) punctuation tokens, and (3) speaker change tokens. In the example shown in Fig. 3, the token sequence 310 comprises a word token 312, a punctuation token 320, and a speaker change token 330. It should be noted that the subsequence of tokens containing the dictionary token 312, the punctuation token 320 and the speaker change token 330 may indicate that the word associated with the dictionary token 312 is spoken by the first speaker and this word is the last word in the phrase, since the punctuation token 320 represents a period, and the next word in the recording is spoken by the second (different) speaker. Below, with reference to Figs. 4 and 5, it is described how the server 112 is able to train and use the speaker change detection model 300 to generate, respectively, punctuation tokens and speaker change tokens in the sequence of tokens 310.

Education

[94] In some embodiments of the present technology, the server 112 may implement at least two stages of training the speaker change detection model 300. In the first stage, the server 112 may train a machine learning algorithm using a large amount of data. Then, in the second stage, the server 112 may retrain the pre-trained machine learning algorithm using additional data to tune the algorithm for a specific task. In general, training the machine learning algorithm in the first stage involves forming templates of the machine learning algorithm based on the data, and in the next stage, retraining is performed to optimize performance. Retraining may be used to eliminate the effect of overfitting, tune the model for new tasks, and/or adjust the parameters of the model to improve the results. This is an iterative process that may be performed by the server 112 to improve the accuracy and efficiency of the model.

[95] In particular, initial training enables the construction of a machine learning model. The initial training stage may begin with the collection and preparation of a base dataset containing input data (features) and corresponding target output data. The model is then trained by the server 112, which identifies patterns, relationships, and statistical representations in this dataset, making predictions, or performing classification.

[96] During the initial training phase, server 112 may split the initial data set into a training set and a validation set. Server 112 may also select a model architecture and initialize its hyperparameters (e.g., learning rate, batch size, etc.). The training data set is then fed to the model with the selected architecture, which iteratively tunes the internal parameters of the model architecture to minimize the discrepancy between the model's predictions and target values (labels). Server 112 may then calculate one or more performance metrics to evaluate the performance of the model on the validation set.

[97] In the retraining stage, the pre-trained model is tuned to a specific task or specific conditions. If the pre-trained model demonstrates satisfactory results on the training data, but does not cope with previously unknown data (overfitting effect), then regularization and/or hyperparameter tuning methods can be used in the retraining stage to improve generalization.

[98] In the presence of, in particular, one or more pre-trained models, retraining may involve training one or another pre-trained model, taken as a base variant, on a new related task. For example, server 112 may receive a pre-trained model from database 150 and implement a retraining stage to adjust this pre-trained model to solve a new related task. The developers of this technology have found that applying the principles of transfer learning to a pre-trained model instead of implementing the initial training stage of the model allows saving time and computing resources.

[99] Fig. 4 shows a first iteration 400 of training and a second iteration 450 of training that the server 112 may perform, respectively, in the initial training stage of the model 300'' and the retraining stage of the pre-trained model 300'. The first iteration 400 of training and the second iteration 450 of training are described sequentially below.

Punctuation training

[100] In the first iteration 400 of training, the server 112 may receive a first training set 402 containing training input data 404 and a label 406. The training input data 404 includes input audio data 408 and input text data 410. The input audio data 408 represents a recording of the speaker's speech, and the input text data 410 is a corresponding text representing the speaker's speech. It should be noted that such text contains words spoken by the speaker, without any punctuation marks. The label 406 contains a target sequence of tokens 420, which contains word tokens representing the words spoken by the speaker, and punctuation tokens representing the control positions of punctuation marks among these words. In one embodiment, the positions of the punctuation tokens in relation to the text may be specified by human evaluators. In another embodiment, the text may initially contain punctuation marks, but they are subsequently removed when generating training data. In particular, the target token sequence 420 comprises a word token 422 followed by a punctuation token 424, which may indicate that the word token 422 represents the last word in a phrase spoken by a speaker.

[101] The server 112 is capable of transmitting training input data 404 to the model 300'' (e.g., the input audio data 408 can be transmitted to the submodel for audio data, and the input text data 410 can be transmitted to the submodel for text). The model 300'' is capable of generating a predicted token sequence, which is compared with the target token sequence 420 from the label 406. Then, the model 300'' is adjusted based on the results of the comparison of the predicted token sequence and the target token sequence 420. After a large number of training iterations performed similar to the first training iteration 400, the server 112 is capable of performing initial training of the model 300'' and obtaining a trained model 300' as a result. The trained model 300' is trained to predict the position of punctuation tokens in the output token sequence in order to determine the places in the text version of the speaker's speech where punctuation marks should be inserted to supplement the text version of the speaker's speech. It should be noted that a text version of the speaker's speech supplemented in this way can contribute to increasing the efficiency of one or more elements of the voice-over system.

[102] The developers of the present technology took into account that large data sets exist for punctuation training. The data sets for punctuation training may include video and/or audio files with corresponding subtitles. In some embodiments, when forming the data set for punctuation training, the server 112 is capable of performing one or more filtering operations. It is assumed that the server 112 is capable of receiving the data set for punctuation training from the database 150. In one embodiment, the data set for punctuation training may include approximately one million hours of (actual) audio data with corresponding text data containing punctuation marks. It can be argued that in the process of punctuation training, the model is trained to predict punctuation indicators after corresponding words.

Additional training for changing the announcer

[103] In the second iteration 450 of training, the server 112 may receive a second training data set 452 comprising training input data 454 and a label 456. The training input data 454 includes input audio data 458 and input text data 460. The input audio data 458 represents a recording of speech by one or more speakers, and the input text data 460 represents a corresponding text representing the speech of one or more speakers. It should be noted that such text comprises words spoken by one or more speakers, without any punctuation marks. The label 456 comprises a target sequence of tokens 470 that comprises word tokens representing words spoken by one or more speakers, punctuation tokens representing control positions of punctuation marks among words, and speaker change tokens representing control indications of places at which the current speaker stops speaking and another speaker starts speaking. In particular, the target sequence of tokens 470 comprises a word token 472 that is first followed by a punctuation token 474 that may indicate that the word token 422 represents the last word in a phrase spoken by the speaker, and then by a speaker change token 476 that may indicate that the word token 422 represents the last word in a phrase spoken by the current speaker before another speaker starts speaking a new phrase.

[104] The server 112 is capable of transmitting the training input data 454 to the pre-trained model 300' (e.g., the audio input data 458 can be transmitted to the sub-model for audio data, and the text input data 460 can be transmitted to the sub-model for text). The pre-trained model 300' is capable of generating a predicted token sequence, which is compared with the target token sequence 470 from the label 456. The pre-trained model 300' is then adjusted based on the results of the comparison of the predicted token sequence and the target token sequence 470. After a large number of training iterations performed similar to the first training iteration 400, the server 112 is capable of retraining the pre-trained model 300' and obtaining as a result the speaker change detection model 300. The speaker change detection model 300 is trained to predict the position of punctuation tokens in the output token sequence in order to determine the locations in the text version of the speaker's speech where punctuation marks should be inserted to supplement the text version of the speaker's speech, and to predict the position of speaker change tokens in the output token sequence in order to determine the locations in the text version of the speaker's speech where the speaker change occurs to further supplement the text version of the speaker's speech. It should be noted that the text version of the speaker's speech supplemented in this way can contribute to increasing the efficiency of one or more elements of the voice-over system.

[105] The developers of the present technology have found that there are much fewer large data sets for speaker change training than for punctuation training. Speaker change training data sets may also include video and/or audio files with corresponding subtitles. In some embodiments, when generating a speaker change training data set, the server 112 is capable of performing one or more filtering operations. It is assumed that the server 112 is capable of receiving a speaker change training data set from the database 150. In one embodiment, the speaker change training data set may include approximately one hundred hours of (actual) audio data with corresponding text data containing punctuation marks. It can be argued that during the speaker change training process, the model is trained to predict punctuation marks indications, as well as speaker change indications after corresponding words.

[106] Without being bound by any particular theory, the developers took into account that performing additional training on a model previously trained to solve the punctuation mark detection task allows the model to use punctuation tokens as “hints” when solving the speaker change detection task. In other words, the speaker change detection model 300 in some sense “understands” that a speaker change is unlikely within a phrase (i.e., after a dictionary token) and/or is comparatively more likely after the end of a phrase (i.e., after a punctuation token representing a period). It can also be argued that the speaker change detection model 300 is trained to “understand” that the probability that the next token (to be generated) will be a speaker change token is comparatively higher if the last token (to be generated) is a punctuation token rather than a dictionary token.

Job

[107] Fig. 5 shows a working iteration 500 of the speaker change detection model 300 executed on the server 112 to generate a sequence of tokens 310. The server 112 is capable of receiving a working set of data 502. The working set of data 502 comprises audio data 350 and corresponding text data 360. In some embodiments, the working set of data 502 may include pre-recorded data obtained from the database 150. In other embodiments, the working set of data 502 may be a set of data received by the server 112 in real time, such as during live broadcasts and/or video conferences where instant dubbing is required. In further embodiments, a video file may be received along with the audio data 350, and the server 112 is capable of extracting the corresponding text data 360 from the audio data 350 using known techniques.

[108] As mentioned earlier, the server 112 is capable of dividing the audio data 350 into a sequence of audio data fragments and transmitting them as input data to the audio data submodel 302 of the speaker change detection model 300. In addition, the server 112 is capable of dividing the text data 360 into a sequence of text data fragments and transmitting them as input data to the text submodel 304 of the speaker change detection model 300. In response, the speaker change detection model 300 may concatenate the intermediate vector representations generated by the audio submodel 302 and the text submodel 304, after which the concatenated intermediate output data of the speaker change detection model 300 is further processed by the output submodel 308 to generate a sequence of output tokens 310 comprising a word token 312, followed first by a punctuation token 320 and then by a speaker change token 330.

[109] Then, as mentioned above, in some non-limiting embodiments of the present technology, the server 112 is configured to transmit a sequence of output tokens 310 comprising a word token 312 followed first by a punctuation token 320 and then by a speaker change token 330 to a diarization model 301 to identify phrases spoken by different speakers in the sequence of output tokens 310. Then, using the output of the diarization model 301, the server 112 is configured to group phrases from the speech recognition data 252 by the speakers who speak those phrases. Thereafter, as described in detail above, the server 112 is able to (1) translate the phrases grouped in this manner using the translation module 204, thereby obtaining the translation data 253, and (2) generate, using the speech synthesis module 206, for each translated phrase from the translation data 253 associated with the current speaker, a corresponding portion of the speech synthesis data 254, as indicated above.

[110] Taking into account the above-described architecture and the examples given, it is possible to implement a method for training such a model as the speaker change detection model 300. Fig. 6 shows a block diagram of a method 600 in accordance with some non-limiting embodiments of the present technology. The method 600 can be implemented on the server 112.

[111] As indicated above, the server 112 is capable of performing at least two stages of training the speaker change detection model 300. In the first stage, which is described in detail above with reference to Fig. 4 with respect to the first iteration 400 of training, the server 112 is capable of training a machine learning algorithm based on a large amount of data to generate predictive punctuation tokens for a given sequence of text tokens. Then, in the second stage, which is described in detail above with reference to Fig. 4 with respect to the second iteration 450 of training, the server 112 is capable of further training the pre-trained machine learning algorithm based on additional data in order to adjust this algorithm to predict the positions of speaker change tokens in a given sequence of text tokens.

Step 602: obtaining a punctuation training dataset comprising first input data including audio data and text data representing the speaker's speech, and a first label including a sequence of control tokens.

[112] The method 600 begins with step 602, in which the server 112 is configured to receive a first training data set 402 for training the model 300'' to determine the positions of punctuation tokens in a given sequence of text tokens. As described in detail above with reference to Fig. 4, the first training data set 402 comprises training input data 404 and a label 406. According to some non-limiting embodiments of the present technology, the training input data 404 comprises input audio data 408 and input text data 410. The input audio data 408 represents a recording of the speaker's speech, and the input text data 410 represents the corresponding text representing the speaker's speech.

[113] The label 406 contains a target sequence of tokens 420, which contains word tokens representing words spoken by the speaker, and punctuation tokens representing control positions of punctuation marks among words. In one embodiment, the positions of the punctuation tokens with respect to the text may be specified by human evaluators.

[114] The next step in method 600 is step 604.

Step 604: training a model using the punctuation training dataset to generate a working sequence of tokens based on a combination of working audio data and working text data to generate a trained punctuation model.

[115] In step 604, according to some non-limiting embodiments of the present technology, the server 112 is configured to provide training input data 404 to the model 300'' (e.g., the audio input data 408 may be provided to the audio submodel and the text input data 410 may be provided to the text submodel). In response, the model 300'' is configured to generate a predicted token sequence that is compared to a target token sequence 420 from the label 406. The server 112 is then configured to tune the model 300'' by optimizing the discrepancy between the predicted token sequence and the target token sequence 420 (which may be expressed by a loss function, for example). After a plurality of training iterations performed similar to the training iteration 400, the server 112 is configured to perform initial training of the model 300'' and obtain a trained model 300' as a result.

[116] Thus, the trained model 300' is trained to predict the position of punctuation tokens in the output token sequence in order to determine the places in the text version of the speaker's speech where punctuation marks should be inserted to complement the text version of the speaker's speech. It should be noted that a text version of the speaker's speech supplemented in this way can contribute to increasing the efficiency of one or more elements of the voice-over system.

[117] The next step in method 600 is step 606.

Step 606: obtaining a speaker change training dataset comprising a second input data including second audio data and second text data representing speech of multiple speakers, and a second label including a second sequence of control tokens.

[118] In step 606, according to some non-limiting embodiments of the present technology, the server 112 is configured to receive a second training data set 452 for training the trained model 300' to predict the positions of speaker change tokens in a sequence of text tokens. As described in detail above with reference to FIG. 4, the second training data set 452 comprises training input data 454 and a label 456. According to some non-limiting embodiments of the present technology, the training input data 454 comprises audio input data 458 and text input data 460. The audio input data 458 represents a recording of speech by one or more speakers, and the text input data 460 represents a corresponding text representing the speech by one or more speakers. It should be noted that such text comprises words spoken by one or more speakers, without any punctuation marks. Label 456 contains a target sequence of tokens 470 that contains (1) word tokens representing words spoken by one or more speakers, (2) punctuation tokens representing control positions of punctuation marks among words, and (3) speaker change tokens representing control indications of places at which the current speaker stops speaking and another speaker begins speaking.

[119] The next step in method 600 is step 608.

Step 608: further train the trained punctuation model using the speaker change training dataset to generate a second working token sequence based on a combination of the working audio data and the working text data to form a speaker change model.

[120] At step 608, in accordance with some non-limiting embodiments of the present technology, the server 112 is configured to further train the trained model 300' to determine the positions of the speaker change tokens using the second training data set obtained at step 606.

[121] To this end, in accordance with some non-limiting embodiments of the present technology, the server 112 is configured to transmit training input data 454 to the pre-trained model 300' (e.g., the audio input data 458 may be transmitted to the audio sub-model 302 and the text input data 460 may be transmitted to the text sub-model 304). In response, the pre-trained model 300' may generate a predicted token sequence that the server 112 may compare with a target token sequence 470 from the label 456. The server 113 may then adjust the pre-trained model 300' by minimizing the discrepancy between the predicted token sequence and the target token sequence 470 (which may be expressed, for example, by an appropriate loss function value).

[122] Thus, after a large number of training iterations performed similar to training iteration 400, the server 112 is able to perform further training of the pre-trained model 300' and obtain as a result a speaker change detection model 300.

[123] The speaker change detection model 300 is now trained to predict the position of punctuation tokens in the output token sequence to determine the locations in the text version of the speaker's speech where punctuation marks should be inserted to supplement the text version of the speaker's speech, and to predict the position of speaker change tokens in the output token sequence to determine the locations in the text version of the speaker's speech where a speaker change occurs to further supplement the text version of the speaker's speech. It should be noted that a text version of the speaker's speech supplemented in this way may contribute to improving the efficiency of one or more elements of the voice-over system.

[124] The next step in method 600 is step 610.

Step 610: Obtain a working dataset containing working text data and working audio data.

[125] At step 610, in accordance with some non-limiting embodiments of the present technology, the server 112 is able to receive a working set of data 502. As described in detail above with reference to Fig. 5, the working set of data 502 comprises audio data 350 and corresponding text data 360.

[126] The next step in method 600 is step 612.

Step 612: Generating a second working token sequence using the speaker change model based on the combination of working audio data and working text data.

[127] The server 112 is then able to apply the speaker change detection model 300 to the working data set 502. To do this, in accordance with some non-limiting embodiments of the present technology, the server 112 is able to split the audio data 350 into a sequence of audio data fragments and pass them as input to the audio data submodel 302 of the speaker change detection model 300. In addition, the server 112 is able to split the text data 360 into a sequence of text data fragments and pass them as input to the text submodel 304 of the speaker change detection model 300.

[128] In response, the speaker change detection model 300 is configured to concatenate the intermediate vector representations generated by the audio submodel 302 and the text submodel 304, after which the concatenated intermediate output data of the speaker change detection model 300 is further processed by the output submodel 308 to generate a sequence of output tokens 310 comprising a word token 312 followed first by a punctuation token 320 and then by a speaker change token 330.

[129] The next step in method 600 is step 614.

Step 614: using the diarization model, determining the working text tokens associated with the corresponding speakers in the second working token sequence for subsequent translation and speech synthesis.

[130] Next, after determining the positions of the punctuation tokens and the speaker change tokens in the working data set 502, the server 112 is able to determine the belonging of phrases to speakers in the text data 360 from the working data set 502. In other words, the server 112 is able to determine when two consecutive phrases from the text data 360 are pronounced by the same speaker, and when they are pronounced by different speakers. To this end, in accordance with some non-limiting embodiments of the present technology, the server 112 is able to apply the diarization model 301 to the sequence of output tokens 310, as described above.

[131] In some non-limiting embodiments of the present technology, the server 112 may then (1) group (e.g., by concatenation) the phrases that have been identified by the diarization model 301 as being associated with a given speaker, and (2) transmit the thus grouped output tokens of the speaker change detection model 300 first to the translation module 204 to provide translation, and then to the speech synthesis module 206 to generate the corresponding portions of speech synthesis data 254.

[132] In some non-limiting embodiments of the present technology, the translation module 204 is configured to generate a portion of translation data 253 for each group determined by the diarization model 301 from the sequence of output tokens 310. Similarly, in some non-limiting embodiments of the present technology, the speech synthesis module 206 is configured to generate a portion of speech synthesis data 254 for each group associated with a corresponding speaker determined by the diarization model 301 from the sequence of output tokens 310, which may include, for example, generating the portion of speech synthesis data 254 using a voice corresponding to the voice of the speaker who speaks the source phrase in the audio data 350 of the payload data 502. In other words, after the tokens from the sequence of output tokens 310 are grouped by the diarization model 301, the server 112 is configured to translate and synthesize the output tokens 310 using the modules, respectively, on a group basis. 204, 206 translation and speech synthesis.

[133] Thus, in some non-limiting embodiments of the present technology, the translation and speech synthesis modules 204, 206 may use not the actual speaker change tokens generated by the speaker change detection model 300, such as the speaker change token 330, but only phrases grouped by speakers determined by the diarization model 301 using the speaker change tokens.

[134] This completes the implementation of method 600.

[135] Possible changes and improvements to the above-described embodiments of the present technology may be obvious to a person skilled in the art. The preceding description is provided for illustrative purposes only and is not intended to limit the scope of the invention. The scope of protection of the present technology is determined solely by the scope of the appended claims.

Claims (49)

1. A method of training a model, implemented on the server, providing for: - obtaining a data set for punctuation training, comprising first input data including audio data and text data representing the speech of a speaker, and a first label including a sequence of control tokens containing a control text token indicating a word and a control punctuation token indicating a punctuation mark and following the control text token; - training the model, using the punctuation training dataset, to generate a working sequence of tokens based on a combination of working audio data and working text data in order to generate a trained punctuation model; - obtaining a data set for speaker change training, comprising second input data including second audio data and second text data representing speech of multiple speakers, and a second label including a second sequence of control tokens containing a second control text token indicating a second word, a second control punctuation token indicating a second punctuation mark, and a speaker change control token indicating a speaker change and following the second control punctuation token, - further training of the trained punctuation model, using the data set for training speaker change, forming a second working sequence of tokens based on a combination of working audio data and working text data in order to form a speaker change model; - obtaining a working data set containing working text data and working audio data; - generating, using the speaker change model, a second working token sequence comprising a working text token, a working punctuation token following the working text token, and a working speaker change token following the working punctuation token, based on a combination of working audio data and working text data; and - formation of synthesized audio content based on the second working sequence of tokens. 2. The method according to item 1, wherein the speaker change model comprises a submodel for audio data, a submodel for text and a third submodel, and the formation of the second working sequence of tokens provides for: - formation, using a submodel for audio data, of a working sequence of vector representations of audio data based on working audio data; - formation, using a submodel for text, of a working sequence of vector representations of text based on working text data; - formation of concatenated intermediate output data using a working sequence of vector representations of audio data and a working sequence of vector representations of text; and - formation, using the third submodel, of the second working sequence of tokens using concatenated intermediate output data. 3. The method according to claim 2, wherein the submodel for audio data is a WavLM model. 4. The method according to claim 2, wherein the submodel for the text is the mT5 model. 5. The method according to claim 2, wherein the third submodel is a transformer model. 6. The method according to paragraph 1, additionally providing for: - formation, using a speech-to-text conversion model, of text data based on audio data; - generating, using a speech-to-text conversion model, second text data based on the second audio data; and - formation, using a speech-to-text conversion model, of working text data based on working audio data. 7. The method according to claim 1, wherein the control punctuation token immediately follows the control text token, the speaker change control token immediately follows the second control punctuation token, the working punctuation token immediately follows the working text token, and the working speaker change token immediately follows the working punctuation token. 8. A method for further training a pre-trained model, implemented on the server, which provides for: - obtaining a data set for speaker change training, comprising second input data including second audio data and second text data representing speech of multiple speakers, and a second label including a second sequence of control tokens containing a second control text token indicating a second word, a second control punctuation token indicating a second punctuation mark, and a speaker change control token indicating a speaker change and following the second control punctuation token in the second sequence of control tokens; - further training of the pre-trained model, using the data set for training speaker change, forming a second working sequence of tokens based on a combination of working audio data and working text data in order to form a speaker change model, wherein in order to form a working sequence of tokens based on a combination of working audio data and working text data, the pre-trained model was trained based on a data set for training punctuation, containing first input data, including audio data and text data representing the speaker's speech, and a first label, including a sequence of control tokens, which contains a control text token indicating a word, and a control punctuation token indicating a punctuation mark and following the control text token; - obtaining a working data set containing working text data and working audio data; - generating, using the speaker change model, a second working token sequence comprising a working text token, a working punctuation token following the working text token, and a working speaker change token following the working punctuation token, based on a combination of working audio data and working text data; and - formation of synthesized audio content based on the second working sequence of tokens. 9. The method according to claim 8, wherein the control punctuation token immediately follows the control text token, the speaker change control token immediately follows the second control punctuation token, the working punctuation token immediately follows the working text token, and the working speaker change token immediately follows the working punctuation token. 10. A server for training a model, comprising at least one processor and at least one physical machine-readable memory storing executable commands, the execution of which by at least one processor on the server ensures: - obtaining a data set for punctuation training, comprising first input data including audio data and text data representing the speech of a speaker, and a first label including a sequence of control tokens containing a control text token indicating a word and a control punctuation token indicating a punctuation mark and following the control text token; - training the model, using the punctuation training dataset, to generate a working sequence of tokens based on a combination of working audio data and working text data in order to generate a trained punctuation model; - obtaining a data set for speaker change training, comprising second input data including second audio data and second text data representing speech of multiple speakers, and a second label including a second sequence of control tokens containing a second control text token indicating a second word, a second control punctuation token indicating a second punctuation mark, and a speaker change control token indicating a speaker change and following the second control punctuation token; - further training of the trained punctuation model, using the data set for training speaker change, forming a second working sequence of tokens based on a combination of working audio data and working text data in order to form a speaker change model; - obtaining a working data set containing working text data and working audio data; - generating, using the speaker change model, a second working token sequence comprising a working text token, a working punctuation token following the working text token, and a working speaker change token following the working punctuation token, based on a combination of working audio data and working text data; and - formation of synthesized audio content based on the second working sequence of tokens. 11. The server according to item 10, in which the speaker change model contains a submodel for audio data, a submodel for text and a third submodel, and the formation of the second working sequence of tokens provides for: - formation, using a submodel for audio data, of a working sequence of vector representations of audio data based on working audio data; - formation, using a submodel for text, of a working sequence of vector representations of text based on working text data; - formation of concatenated intermediate output data using a working sequence of vector representations of audio data and a working sequence of vector representations of text; and - formation, using the third submodel, of the second working sequence of tokens using concatenated intermediate output data. 12. The server of claim 11, wherein the submodel for audio data is the WavLM model. 13. The server of claim 11, wherein the submodel for the text is the mT5 model. 14. The server according to claim 11, wherein the third submodel is a transformer model. 15. The server according to item 10, wherein at least one processor additionally provides on the server: - formation, using a speech-to-text conversion model, of text data based on audio data; - generating, using a speech-to-text conversion model, second text data based on the second audio data; and - formation, using a speech-to-text conversion model, of working text data based on working audio data. 16. The server of claim 10, wherein the control punctuation token immediately follows the control text token, the speaker change control token immediately follows the second control punctuation token, the working punctuation token immediately follows the working text token, and the speaker change working token immediately follows the working punctuation token.