RU2546064C1 - Distributed system and method of language translation - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: distributed network translation system provides the access to the distributed network of professional translators and machine translation systems (MT) which together perform translation in real time. The system consists of cloud servers, a user interface, a segmentation module which performs splitting of the initial file into a set of segments, a module of preliminary translation, translation memory base, morphological dictionaries, a module of glossaries, a module of the creation of the accompanying data forming a data set for reference for performers and for individual control of MT systems for translation of this document, a module of word-by-word alignment, a module of MT processing selecting the best version of translation, a module of identification of multiple performers recommended for work on the document translation, a module collecting data on all actions executed by performers in the web interface, a module which compiles final translations of each separate segment into the final file with translation.

EFFECT: increase of speed, efficiency and accuracy of translation.

13 cl, 3 dwg

Description

FIELD OF THE INVENTION

[001] The present invention describes a system and method for language translation of a text in general, and in particular a system for joint translation.

BACKGROUND

[002] The collection and exchange of information for any scientific, commercial, political or social purpose often requires a quick and effective translation of the text, so that a lot of knowledge and ideas become useful on a global scale. Computer programs that automatically translate from one language to another (“machine translation programs”) can, in principle, satisfy this need, and such programs have been developed and continue to be developed for many languages. For a formal presentation style in deeply studied languages (as opposed to an informal, idiomatic or colloquial style), such machine translation programs demonstrate a fairly adequate translation quality.

[003] For more difficult or less studied languages (for example, Arabic), however, existing machine translation programs do not work well even for formal communication (for example, Modern Standard Arabic) and they are especially weak in case of informal, conversational and idiomatic communication . Similarly, where a more accurate translation is required, machine translation alone is not enough even for well-studied languages (for example, English, French, Spanish, German and other languages).

[004] Professional translators, in principle, can provide high-quality translations for difficult languages and informal communications, but Internet applications require constant availability and prompt response, which cannot be guaranteed if existing approaches to organizing the work of professional translators are used.

[005] In light of the foregoing, a method and system is needed that can ensure the efficient use of translation memory databases while working on the translation of large teams of professional translators.

SUMMARY OF THE INVENTION

[006] The invention consists in the following:

[007] The present invention provides a system and method for translating from a language of a source file. The system consists of a web server that receives and processes the data of the source file for translation, a database for storing translated text, processed source files and glossary terms, a segmentation module designed to split the source file into many segments, a processing module that searches for corresponding segments existing data, to find complete and / or partially matching segments from previously translated texts, machine translation module for generating machine translation egmenta, module search terms used to find the segment in terms of glossary and a user interface that is available for multiple users at the same time, the mapping of machine translation, full and partial matches from the translation memory, glossary of terms, and provides the ability to perform professional translation of the original file. The system and method can be represented as executable code (software), hardware, or a combination thereof.

[008] In another aspect of the present invention, for each segment stored in the database, the processing module searches for exact or partial correspondence with previously translated sentences, searches for terms from the glossary and machine translations of the sentence. In the implementation of the system, many users can get simultaneous access to the user interface of the system, and the transfers made by each user are transferred from the user interface to the server and stored in the database. The user interface may also display previously executed segments by other users that fully or partially coincide with the segment being translated.

BRIEF DESCRIPTION OF GRAPHIC SCHEMES

[009] The implementation of the invention will be described hereinafter in accordance with the accompanying graphic diagrams, which are presented to illustrate the essence of the invention and in no way limit the scope of the invention. The following graphic schemes are attached to the application:

[010] Fig. 1 is a data flow diagram illustrating a distributed language translation system implemented in accordance with the present invention.

[011] Fig. 2 is a data flow diagram illustrating an automatic pre-translation method used in the pre-translation module 118, which in turn is part of a distributed language translation system implemented in accordance with the present invention.

[012] Fig. 3 schematically illustrates the interaction of an integration layer of external information systems and a distributed language translation system implemented in accordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[013] In the following detailed description of an embodiment of the invention, numerous implementation details are set forth to provide a clear understanding of the present invention. However, to a person skilled in the art, it will be apparent how the present invention can be used, both with and without implementation details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the understanding of the features of the present invention.

[014] Furthermore, it will be clear from the foregoing that the invention is not limited to the foregoing implementation. Numerous possible modifications, changes, variations and replacements preserving the essence and form of the present invention will be apparent to those skilled in the subject field.

[015] The present invention is directed to providing a system and method for fast, efficient and more reliable language translation through a distributed network language translation system.

[016] A distributed language translation network system is a distributed network of professional translators and machine translation systems that interact through the program and user interfaces of the system and together perform real-time translation of texts for which the use of exclusively machine translation or traditionally organized professional translation is not enough, including translation of dynamic communications and other texts created in various information environments.

[017] In the implementation of the present invention, the system is a cloud-based platform accessible via the Internet, professional translators have access to the system through a user interface that can be opened in a web browser, the interface has separate windows for managing a translation project and for simultaneous translation and editing of text by several executors in real time. The distributed network language translation system provides tools for aggregating the resources of a large number of translators, with different access modes, with various professional skills, both professional translators and computer-based machine translation systems, for efficiently performing high-quality translations in real time.

[018] In the implementation of the present invention, the translation is performed by breaking the source text into segments, performing terminology searches on the segment and searching for matches in the translation memory database, then simultaneously sending each segment for translation to several machine translation systems. Moreover, each data source (glossary, translation memory, machine translation system) has its own quality rating, metric of coincidence and / or own confidence in the output, calculated individually for each segment. Moreover, the search results by terminology and translation memory databases are used for additional individual tuning of machine translation systems, and partial matches with the source segment found in the translation memory database are used to substitute in the final machine translation the part of the segment that matches. Then, taking into account the rating of each machine translation system and the values of metrics for each segment, both its own for each machine translation system, and external (taking into account the smoothness of the text as a whole, and internal factors, such as the number of fragments / phrases, of which translation is collected, whether terminology is found from the glossary in the segment, etc.), a machine translation option with the best values of automatic metrics is selected.

[019] The distributed network translation system is based on translation memory technology, designed to increase the translation efficiency, consisting of a storage of parallel segments and a search subsystem that allows you to perform both searches with conditions that apply only to the original language, and searches with conditions that apply both at the same time and original language and translation language. The translation memory stores previously translated segments in the database, so that in the future it would not be necessary to carry out the translation again from scratch. Thus, one of the main functions of the translation memory database is to search for previously translated segments that are similar to the newly translated segment, as well as to compare the segments of their translations, as well as individual phrases and words in these segments.

[020] Figure 1 shows a diagram of a distributed network translation system, which is an example implementation in accordance with the present invention. As shown in Figure 1, a distributed network translation system 100 serves many translation customers 102 who want to receive translation of the source files 104. Many translation customers 102 connect to the remote web server 112 via the Internet 110 using the user interface 106 and a web browser. The source file 104 is uploaded to the server 112 via the Internet 110. After the source file 104 is uploaded to the web server 112, the segmentation module 114 processes the source file 104 and splits it into text into multiple segments. A plurality of segments, each of which contains a part of the source text of the file 104, is then processed by the preliminary translation module 118, which finds for each source segment the resources corresponding to it from the database of linguistic resources 116, namely, it finds completely matching segments in the translation memory, finds partially matching segments in the memory translations, finds the glossary terms found in the segment, and also for each word of the segment determines the corresponding entries in the morphological frequency dictionary . The database of linguistic resources 116 consists of a morphological frequency dictionary, glossaries, and a translation memory database, with previously translated segments, supplemented with document metadata that included these segments, as well as data on the history of work on each segment, i.e. what kind of contractor has completed what stage of the workflow for this segment, what changes has he made to the text of the translation, as well as records of all the actions performed by him in the system interface during the work on this segment. Thus, in the pre-translation module 118, for each segment found in the translation memory, a document is defined to which it belongs, who exactly performed the segment translation (which machine translation system was used, who was the translator, who is the editor, who is the proofreader, etc. .). As well as quantitative assessments of the quality of work received by each of the listed artists on this document (if an expert analysis of the quality of the translation was carried out). Further, based on pre-configured pre-translation rules applied at the level of each segment, which can be additionally changed in each specific translation project, preliminary automatic translation is generated in the pre-translation module 118 and a file containing for each segment completely and partially matching segments from the translation memory, glossary terms, as well as machine translation options generated by various machine translation systems. The rules of preliminary translation are applied at the level of each individual segment and determine which of the translation options will be used by default, as well as which stages of work with this segment should be performed by professional performers, depending on the default translation option and automatic assessment of its quality and the necessary amount of improvements.

[021] In an example implementation of the present invention, the translation memory is a system for storing and searching parallel segments (sentences, phrases or sentence fragments) - a pair of source text and translation text. The translation memory is used to help the translator in the translation of the text and accumulates segments already translated earlier, in order to avoid their repeated translation from scratch in the future. This function is performed by searching the translation memory database for a newly translated segment of fully and partially matching previously translated segments. To establish the degree of coincidence of previously translated segments with the newly translated segment, a correspondence metric is used, which reflects the degree of coincidence of the text of the newly translated segment with the text of the original translated earlier and stored in the segment database.

[022] Many performers (translators) 108 are connected to the platform 100 via the web interface 106. Segments resulting from the operation of the text segmentation module 114 are transmitted for processing to the preliminary translation module 118, which generates a preliminary translation of the source file 104 (which may including contain the best version of machine translation from a variety of options received from available machine translation systems), a set of professional operations that must be performed for each segment, as well as Boron recommended professional translators / editors / proofreaders, indicating for each of them in its rating and preference metrics of its attraction to work on this particular document. The web user interface 106 contains various interface windows for managing the project and for direct work on the translation and editing of the text of the translation, while it is possible to simultaneously edit one document by many translators / editors / proofreaders / reviewers, etc. Using the appropriate web interface 108, the translator can view the translation of the source file 104, taking into account the preliminary translation performed by the preliminary translation module 118, as well as translations made or edited by other translators / editors / proofreaders / reviewers, while the translation memory is automatically updated as you type translators translating new segments and when editing previously performed automatic or professional translations and changes and new additions are selected zhayutsya automatically in real-time search results.

[023] Translations of segments that are entered or edited by professional translators 108 after generating their preliminary translation in the preliminary translation module 118 (different segments can go through different stages of professional work, which is also determined as a result of the operation of module 118), are automatically checked after they are saved by the appropriate the contractor for the correct translation of the terminology in accordance with the glossary and the correctness of the passage of other automatic quality control rules, configured for this file. Based on the results of these checks, a special flag can be set for a segment that signals that a potential error has been automatically fixed, its description is provided, in addition, for each segment, the performer in the web interface sees the history of work on it - who made changes at what stages of work and what changes were made, in addition, comments made by previous performers in this segment are visible, in which performers can include a justification of the reasons for which the translation option was chosen, not passing automatic verification. Each segment of the document must go through all the stages of the workflow defined at the document level, with the exception of cases where as a result of the work of the preliminary translation module 118 some of the stages for individual segments can be skipped.

[024] Figure 2 is a diagram illustrating the operation of a pre-translation module that is part of a distributed network translation system implemented in accordance with the present invention. In block 202, the input file 104 is downloaded through the web-based user interface of the network distributed system or received through access to the system program interfaces (APIs). The source file 104 may have either a textual or binary representation; then the source file 104 is processed by a software filter corresponding to the file format to extract the contents of the file as text. At block 204, the text content of the source file 104 is segmented by a software parser compatible with the Segmentation Rule Exchange (SRX) language. The resulting segments are then stored in the database along with a specially generated XML file with binary inserts containing the information necessary for the subsequent assembly of the translated file while maintaining the formatting of the original file 104.

[025] The process of breaking text into segments can be visualized as follows: the cursor moves through the text, one character at a time. At each cursor position, the rules are checked, consisting of a template for the preceding text and a template for the subsequent text, the rules are checked in accordance with the order specified for them, to first check whether the text preceding the current cursor position matches one of the patterns for the previous text, and then the text of the following cursor after the current cursor position is checked for the template associated with the given template for the subsequent text. If the text matches both patterns, then either the cursor moves to the next position without inserting the segment break border if these patterns match the exclusion rule, or the segment break border is inserted if the patterns match the break rule.

[026] In block 206, for each segment of the original document 104 stored in the database, a search is made for full and partial matches for previously translated segments stored in the translation memory database 116. Terms are also searched from glossaries included in this segment. In addition, for each word of the segment, links to its corresponding entries in the morphological frequency dictionary are established. For each fully or partially matching segment found in the translation memory 116, information is also loaded on the document into which it is included, the linguistic resources used in the translation of the document (glossaries, translation memory databases, machine translation systems). This also includes information on the executors who worked on the document and on each specific segment of the document, their professional assessments of quality and performance metrics and the amount of corrections made at each stage of the workflow for each of the document segments. Accordingly, they are used as metrics obtained for a particular found segment. , and the average values of metrics for the document as a whole.

[027] At the moment when each new translated (parallel) segment is added to the index of the translation memory database 116, the added segment is divided into separate words, and then each individual word is searched by morphological frequency dictionaries containing all valid forms of each word, indicating which of the forms of this word is basic, as well as various additional metadata that describe both the concept as a whole (for example, part of speech or the metric of frequency of use), and each of the forms (gender, case, etc.). Thus, a link to a corresponding entry in the dictionary is attributed to each word. Several possible entries can correspond to one word, in which case all of them are assigned to it. This information is included in the multi-parameter index, which allows subsequently to find, for example, all segments in which the given word is used, in any of its valid forms. The segment translation text is analyzed in a similar way, and the obtained information is also included in the index, which also allows us to find segments in the original where the given word is used in any of its admissible forms, and some other word is used in the translation in any of its admissible forms, or vice versa, the given word is not found in the translation in any of its admissible forms; to perform such queries it is not necessary to search all possible forms of the word, just specify a link to the basic form y and the desired values of other metadata. Also, on the basis of such an analysis, the word-for-word correspondence of the text of the original and the translation is built, i.e. for each word of the original there is a translation word corresponding to it (in this case, individual words, both in the text of the original and in the text of the translation, may not have correspondences, i.e., they have an empty set). For each word, a search is also performed on all the glossaries of this client and the public glossaries of 208, links to the corresponding entries in the glossary are also stored in the search index, if for the next word the terms are in the glossaries of several words, the entry of the remaining words in the segment is checked from a glossary term (the check is also performed at the level of the presence of a word in any of the acceptable forms). In addition, words whose frequency of use indicated in the dictionary are below a given fixed threshold value are marked as low-frequency. The threshold value is selected empirically for each language individually.

[028] In block 206, for each segment, a set of parallel segments (translation options) is found found from the translation memory, each of which is assigned a metric of the degree of similarity of the parallel segment from the translation memory to the original segment. As described above, in the process of placing a new parallel segment in the index of the translation memory database, each word of the segment is analyzed using morphological dictionaries. When searching for parallel segments similar to the original segment, the word used in different forms in the text of the original segment and in the found parallel segment is considered as the same word, respectively, when calculating the metric of similarity of segments, the metric decreases minimally in case of differences in the forms of the word . The segment similarity metric is much stronger when there are words in the original segment that are not in the parallel segment and vice versa, when there are words in the parallel segment that are not in the original segment.

[029] In block 208, the text of the source segment is divided into separate words and then a search is performed for each word in the morphological frequency dictionary, as a result of which the word is assigned a link to the possible basic forms of the word and the corresponding metadata as common to the basic form (for example, part speech), and the corresponding form in which the word is used in the original segment (case, declension, etc.). The dictionary contains for each basic form of the word data on the frequency of use of the word. Words of the original segment that have a frequency of use (which is defined on the common language corpus) below the threshold are marked as low-frequency. For each word, a search is also performed on all the glossaries of this client and the public glossaries of 208, links to the corresponding entries in the glossary are also stored in the search index, if for the next word the terms are in the glossaries of several words, the entry of the remaining words in the segment is checked from a glossary term (the check is also performed at the level of the presence of a word in any of the acceptable forms) and if such a term appears in the entire segment, this information is also stored in ind KSE, including marked the exact words included in the term. Then, based on the rules for extracting terminology and aggregated data collected for all segments of the document, a list of terms that are candidates for inclusion in the glossary for this project is determined. Two types of criteria are used to select candidate terms: linguistic and statistical. Linguistic criteria contain rules defining permissible and unacceptable combinations of words (in parts of speech, for example), as well as a set of stop words that are not included in the terms. For example, a linguistic rule may determine that a phrase consisting of two nouns is valid for extraction. Statistical criteria specify the minimum frequency of occurrences of terms in the text at which the terms are extracted, while different linguistic rules may correspond to different minimum frequencies of occurrences, in addition, the frequency may be changed individually by the translation project manager depending on the project stage, terminology coverage by existing glossaries, etc. d. As a result, a set of candidate terms is determined for inclusion in the glossary, which is stored in the database in block 218. The project manager can then appoint the person responsible for the translation of these terms, the translator views the selected terms and translates them in block 220 via the web interface 106. B in this interface, the translator sees the options for translating these terms found in the client’s translation memory (if terms are found there), the options for translating terms offered in other public glossaries and general accessible translation memory databases, he also sees the context of the use of these terms in the source text. The translated and verified terms are transmitted to block 208 and are ultimately included in the aggregated sets of linguistic resources generated in block 214.

[030] As described previously, in the process of adding a parallel segment to the translation memory database 116, information about terms found in the segment from the glossary and low-frequency words is added to the index. In block 210, a search is performed on the translation memory index of 116 parallel segments in which low-frequency words and glossary terms found in the original segment are encountered, and thus a set of similar parallel segments in which the same low-frequency words and glossary terms are found is determined. For each parallel segment included in this set, the terminology matching metric is calculated from the point of view of machine translation:

Where index i denotes the initial segment, index j denotes a similar segment from the constructed set, f is the frequency of use of the corresponding word from the morphological frequency dictionary. The constants C ₁ , C ₂ , C ₃ , f ₀ are selected empirically for each language in order to maximize correlation with professional estimates of segment similarity and maximize string metrics of machine translation quality, based on texts filtered in accordance with the above calculation algorithm MT_terminology_match _{i, j} . MT_terminology_index _{i, j is} calculated using the same formula as MT_terminology_index _i , with the difference that only words and terms found in the original of both segments are included in the calculation. In addition, for each pair of segments i and j, a set of words matching in them is stored, including with links to glossary terms. Words are considered coincident if their basic forms coincide, that is, words used in various forms are considered coincident and are included in the calculation. The terms of the glossary gtossary_entries _i and glossary_entries _j are taken into account upon their inclusion in the original text, whether the translation of the terms in the parallel segment matches the translation given in the glossary does not matter for the calculation of this metric.

For each pair of segments selected in this way, the metric of coincidence of segments as a whole is also calculated using the same formula as the metric used to assess the degree of coincidence of segments from the translation memory.

To select the best professional performers (translators, editors, proofreaders, etc.) to work on the text, the metric of terminology coincidence is also calculated from the point of view of professional translation:

Where matching_glossary_entries _i are glossary terms found in a parallel segment for which the corresponding translation in the segment text matches the translation given in the glossary. different_glossary_entries _i are glossary terms found in a parallel segment, for which the corresponding translation in the segment text differs from the translation given in the glossary, these types of terms are included in the calculation of the metric only if at least one of the following conditions is true: the glossary term contains more than one word or the term glossary is a low-frequency word.

[031] For each segment similar to the source selected in block 210 and each partially matching segment from the translation memory found in block 206, we determine the documents to which they relate and the stages of the workflow used for these documents: machine translation systems , personalities of the executors working on the document (translators, editors, proofreaders, etc.), their performance and string metrics, modifications made at each stage of the work process. Primary data on the performance of performers (translators, editors, proofreaders, etc.) are collected in real time in the web user interface 106, the interface collects, saves and then sends data to the server about all user actions, about all data entered from keyboard, all mouse clicks on interface elements and compound events, such as entering a segment translation editing field, exiting a segment translation editing field, substituting text from the translation memory, glossary, or from machine translation. Metrics characterizing the volume of changes made to the text are also calculated. Two types of metrics are calculated: metrics calculated solely based on a comparison of the source and final lines (for example, Levenshtein distance), and metrics that take into account the actions performed by users during text editing (data entered from the keyboard, mouse clicks on interface elements and compound events ) When calculating the time spent on translating or editing a segment translation, periods of inactivity are also taken into account when the web interface lost focus or when the time interval between two consecutive actions exceeds a predetermined threshold.

[032] Block 210 is repeated for each segment of the source text, resulting in a set of similar parallel segments from the translation memory 116. In this case, one parallel segment from the translation memory 116 can be included in the number of similar segments for several segments of the source text.

Where MT_terminology_document_match _j is the similarity metric of the parallel segment j from the translation memory 116 with the source document, i is the index of the source text segments with a positive value MT_terminology_match _{i, j is} the similarity metric with this parallel segment j, N is the number of different segments of the source document with a positive value this metric (two document segments identical in terms of text, located at different places in the document, are considered different segments in this case).

A similar formula is used to calculate Human_terminology_match _j

[033] In block 212 for each document for which we found similar segments in blocks 210 and 206, we can thus calculate the metric of similarity of the document as a whole with our original document.

Where TM_match_metric _j is the metric of similarity between segments, used to find and rank matches with the translation memory database described in paragraph

[028], normalized to take values in the interval (0,1], m is the previously translated document, words_total _m is the number of words in the document m. C ₄ , C ₅ are constants predetermined empirically.

[034] For each previously translated document 106, we can determine the glossaries and translation memory databases that were explicitly selected when translating this document by the translation customer or project manager. In block 214, we form four sets of linguistic resources for the source document:

(1) Glossaries and translation memory databases selected explicitly for the document by the customer or project manager;

(2) Priority-ordered set of resources for customizing translation models of machine translation systems:

1. a set of terms from the glossaries explicitly selected for this document;

2. a set of parallel segments from translation memory databases, explicitly selected for this document;

3. for each document translated earlier, for which the value of the metric MT_document_similarity _m exceeds an empirically determined threshold value (resources are sorted according to the value of the metric MT_document_similarity _m , the resources of documents with a higher metric value get a higher priority), we add the following data to the set :

a. a set of terms from the glossaries explicitly selected for this document, only terms containing at least two words are included;

b. parallel segments from this document;

c. a set of parallel segments from translation memory databases, explicitly selected for this document;

(3) A priority set of resources for customizing language models (for the language into which the translation is performed) of machine translation systems:

1. a set of parallel segments from translation memory databases, explicitly selected for this document;

2. for all documents for which the value of the Human_document_similarity _m metric exceeds an empirically determined threshold value (resources are sorted according to the value of the Human_document_similarity _m metric, the resources of documents with a higher metric value receive a higher priority), we add the following data to the set:

a. parallel segments from this document;

b. a set of parallel segments from translation memory databases, explicitly selected for this document;

(4) A set of segments of the source document, each of which has an ordered set of the following data:

1. segments that partially overlap with the source segment from the translation memory — only segments included in the data set are included here (2), segments are ordered according to the values of the segment matching metric used to search the translation memory databases;

2. similar segments with a positive value of the Human_terminology_match_strict _{i, j} metric — only segments included in the data set are included here (2), segments are ordered according to the values of the segment matching metric used to search the translation memory databases;

for each parallel segment included in this set, the word-by-word alignment of the original and segment translation is also included in the set;

[035] In block 226, the data set (1) can be uploaded to a separate file, which can be uploaded from the system via the web interface and used to translate the source file in any external environment. If further work on the translation of the document is performed by the executors through the web interface of the system, this data is stored in the database 106 and subsequently displayed as prompts to the executors (translators, editors, proofreaders, etc.) when working on each relevant segment.

[036] In block 216, data sets (2) and (3) are added to the statistical and language-based machine translation systems to individually configure machine translation, with a higher priority in the data hierarchy than the general data used to train the systems. Data is added with a higher priority in the data hierarchy than general data, in addition, each of the above subsets of data is assigned an individual priority based on the data sorting procedure described above. If necessary, after adding these resources, pre-training or retraining of the machine translation system is automatically performed.

[037] In block 222, a preliminary machine translation of each segment of the source text is performed by all available machine translation systems that are preselected for this source document. For each machine translation system that can be individually configured, it is individually configured in block 216 using the data sets (2) and (3) created in block 214. Each segment sent to the machine translation of each of the machine translation systems is accompanied by a data set (4) created for this segment in block 214. The data set (4) is used as follows: we take each parallel segment from the set (4) from the number of partially matching segments and determine the set of words (substrings) matching with the source segment, for each matching set of words we determine using the word alignment constructed by adding a parallel segment in the translation memory the translation corresponding to them in the given parallel segment. Then we consider possible combinations of various substrings in order to obtain the greatest coverage of the text of the source segment, while only substrings consisting of several words or containing low-frequency words are added to the base substrings (obtained on the basis of one parallel segment). As a result, we get a set of many options for covering the source segment with substrings with translations from parallel segments partially matching it from the translation memory.

Then we take a set of segments with a positive value of the metric Human_terminology_match_strict _{i, j} from the data set (4) and extract from them, using the word-by-word alignment constructed when adding a parallel segment to the translation memory, the translation options for low-frequency words and terms from the glossary (for each word and the term can be extracted with several translation options with different valid word forms). After that, these translation options are added as substrings of the segment with translation into those coverage variants in which these words and terms are not yet included in any of the substrings of the segment with translation.

Thus, for each source segment we get a set of many options for covering the source segment with substrings containing the translation of this substring. For each variant of covering a segment with substrings, we determine the percentage of coincidence, which is calculated as the ratio of the number of characters in the segment covered by substrings to the total number of characters in the segment.

Each such variant of marking a segment with substrings with translation is then sent to each of the available machine translation systems. For each segment markup variant received at the input, each machine translation system generates a segment translation and the accompanying metric of confidence of the system as a translation, if the system supports the calculation of such a metric. For each version of the translation, regardless of the machine translation system that created it, a single metric is also calculated that characterizes the degree of smoothness of the text of the translation from the point of view of the target language (for example, a metric based on the calculation of perplexity - the degree of uncertainty in the probabilistic model of the language) can be used, for This uses a statistical model of the translation language and is individually configured in block 216. After which we exclude machine translation options that contain incorrect terminology (those checked against the terminology of glossaries, explicitly selected for this project). Then, for each machine translation system, we select the translation with the highest metric of self-confidence as the translation of the machine translation system itself, if this system generates it, or the translation with the best value of our own metric for text smoothness based on perplexity. Metric values are stored in the database along with the machine translation options themselves.

[038] In the implementation of the present invention, during the work of the translator (as well as the editor, proofreader, etc.) on translating or editing a segment via the web user interface 106, all of its actions in this web interface, including pressing keyboard keys, mouse clicks on interface elements, use of office shortcuts, etc., are recorded and sent to the server, where they are saved to the database. Based on this source data, individual performance and quality metrics are calculated asynchronously, such as the time taken to translate (as well as edit or proofread) the segment, as well as the number of significant changes made to the segment translation text (for example, replacing one word with another, especially for low-frequency words) and insignificant changes (changes in the endings of words for their coordination, etc.). Changes are recorded at each stage of the workflow with respect to the translation option that was after the previous stage of the workflow (for example, changes made by the editor after the translator, or by translators after the machine translation system). In addition, for individual segments of the translated document, a professional assessment of the quality of the translation can be selectively conducted. This professional quality assessment is based on the error classification methodology, in which a linguistic quality control specialist performs a rigorous analysis of the translation of each of the segments included in the sample (usually a small one) and fixes for each segment the types and criticality of the errors found, in accordance with the approved classification rules mistakes. Both types of metrics are used together to assign a reputation score to an executor (translator, editor or proofreader).

For each segment in which the machine translation was edited, we calculate the number of the following events: (1) significant changes in the translation of the terminology (rephrasing) - when one or more words are replaced by other words, (2) a change in the order of words, (3) coordination words (changing the endings of words, especially in languages with complex matching rules and rich morphology, etc.). Then we calculate for each machine translation system the expected number of changes, as a tabular function of the segment length and the self-confidence metrics of the machine translation system as a translation or an external metric for the smoothness of the translation based on the calculation of the text persplectivity for the corresponding statistical model of the translation language:

For each artist, we also have data on all segments on which he worked previously, and in particular on all segments for which he performed machine translation editing. We can calculate the amount of changes made by him of each type and the total time spent on editing the segment. Next, we can calculate for each executor and each machine translation system a set of constants t ₁ , t ₂ , t ₃ that provide the best linear interpolation on the collected set of primary data:

,

,

[039] Based on the values of the machine translation metrics calculated for each segment translation in block 222, the length of the source segment and the table functions for the expected number of changes of each type, which we determined earlier for each machine translation system, we calculate the expected number of changes for machine translation each segment. After that, we choose the machine translation option with the least expected number of changes (changes of different types are included with different weights corresponding to the values of the constants t ₁ , t ₂ , t ₃ averaged over the entire database). If several translation options get close values of this metric, then the option with the highest percentage of text coverage of the substring segments with the translation is selected from them. The selected translation option, together with the expected number of changes of each type, is stored in the database and included in the aggregated data set generated in block 226.

[040] For each parallel segment from the translation memory included in the set (3), designed to individually configure the statistical language model, created in block 214, we have a complete set of data on the workflow that took place when translating the document: who performed the translation (or editing machine translation), who performed editing and proofreading, etc. We can also install documents containing these segments, professional quality assessments made by linguistic quality control specialists for samples from the text of documents, automatically collected metrics for the number of changes of different types (change in the translation of the term, change in the word order, word matching) made by the editors in the text of the translation and proofreaders after the translator (or machine translation editor), as well as the time they spent on changing each segment. In block 230, we then calculate a weighted translation quality metric for each artist (translator, editor, proofreader, etc.) who participated in the translation of documents included in set (3):

Where LQA_metric _m is a standardized professional assessment of the quality of translation, calculated for a subset of a sample from document m, which includes only professionally evaluated segments in which this artist participated.

Based on the calculated value of the LQA_total metric, we exclude from further consideration performers for whom the metric has a value below an empirically determined threshold. This threshold mainly depends on the translation quality requirements set for the project by the project manager at the stage of its creation and initial configuration. Then the results of the LQA_total metric are clustered, after which the performers within each cluster are ordered according to the value of the Weight_total metric. Thus, in block 230, a sorted list of preferred performers is generated to work on the translation of this source document.

For each translator, we also determine the expected time for editing a machine translation of this source document. The expected time for editing a machine translation is calculated based on the values of the constants t ₁ , t ₂ , t ₃ for each translator and each machine translation system and the machine translation options selected for each of the segments in block 224 and the values of the expected number of corrections of each type assigned to this machine segment translation.

[041] The project manager reviews the set of recommended executors for the project (translators, editors, proofreaders, etc.), the expected time and complexity of editing machine translation, and project statistics, including the number of different matches with the translation memory, the number of occurrences of terms from the glossary, and also the amount of data and similarity metrics calculated for the data sets generated in block 214. The project manager also has data on the availability of recommended artists in the web interface of the system, according to loaded from the project management system of the transfer, and makes the final decision on the composition of the executors, after which it initiates sending invitations to participate in the project to the set of selected executors in the web interface of the system. This decision can also be made automatically, based on real-time data on the availability of performers, the projected workload of each of the performers for the other projects it has adopted, and the required deadline for the translation of this project.

[042] After receiving a notification of an invitation to participate in a translation project, each relevant artist (translator, editor, proofreader, etc.) confirms or rejects this invitation. After confirming participation, the designated contractor in block 232 enters the system and through the web user interface 106 starts work on the translation of the document. The system supports two types of projects: sequential, when each subsequent stage of the workflow begins only after the previous stage of the workflow on the entire document is completely completed, and parallel, when the stages of the workflow are performed sequentially at the level of each individual segment, while the document can simultaneously segments present at various stages of the workflow.

[043] After all segments of the document pass all the specified stages of the workflow and the translation of the document is completed, the translation goes to block 128, in which the final translation file is generated. The translated document 130 has the same text or binary file format as the original file. The translated document is formed from the translation text of each segment and the metadata file created when parsing the source file in the segmentation module 114.

[044] The translated document 130 is then automatically transmitted to the translation customer via the web user interface from which the customer downloads the translated file. If the source files came from a certain external information system, then the translated files can be placed into the same system using the software APIs of the interfaces contained in the integration layer 302.

[045] Fig. 3 is a schematic illustration of a platform representing an implementation of the distributed network translation system described in the present invention. According to Fig. 3, the platform consists of three layers: an integration layer 302, a translation platform 304, and additional modules 306. An integration layer 302 provides loading into the system the source files for translation 102 and converting the source files into a set of text segments. Integration layer 302 is represented in the implementation of the system if external information systems, such as content management systems, document management systems, or general purpose portals, are used as sources of source files. The translation platform layer 304 includes a web user interface, server-side data processing modules, and a data storage layer. The web-based user interface provides an interface for access to the system for translators, project managers, editors, proofreaders, linguistic quality control specialists, terminology specialists, representatives of the translation customer, etc. The web user interface contains separate interface windows for setting up and managing the project and for working directly on translations and editing a document (with the possibility of simultaneous work on the translation of many artists). Server-side data processing includes the conversion of source files to various formats, translation validation, preliminary translation of documents, etc. The layer of additional modules 306 includes in ABBYY Lingvo ™ dictionaries and other dictionaries, machine translation systems used for preliminary translation of documents with subsequent editing, an airliner that creates XML files from parallel documents of arbitrary formats and layout, suitable for export to translation memory, as well as a module spelling and punctuation control. Other necessary subsystems can also be integrated with the system via the integration bus with software APIs.

Claims (13)

1. Network distributed language translation system for translating source files, consisting of:
cloud-accessible servers available on the network, simultaneously available to many artists and multiple translation customers connecting to servers via the Internet;
a user interface that allows many translation customers to upload source files for translation into the language translation system and receive information on the terminology translation options offered by the performers, as well as view and approve glossaries for the project;
a segmentation module that splits the source file into many logical segments and sends these segments for translation to the preliminary translation module;
translation memory database that stores previously translated logical segments and allows you to search for segments similar to a given segment;
morphological dictionaries containing information about the various acceptable forms of words and metadata that correspond to both the concept as a whole and individual word forms, including the frequency of use of words, and also allow us to establish a correspondence between the words in the text of the original and the translation of the segment and the corresponding entries in the dictionary ;
a glossary module that allows you to store glossary terms and find their occurrences in the text of the original and translation segments, including for terms consisting of several words;
a module for creating related data that searches for segments matching the source segment in the translation memory, calculates the degree of similarity and the degree of similarity of the terminology used in them, and generates a data set for prompting the performers and for customizing machine translation systems to translate this document;
a word-by-word alignment module that establishes a correspondence between the words of the original and the translation of a parallel segment;
a machine translation processing module that selects the best machine translation of a segment from the many options generated by each machine translation system and aggregates these translations for all document segments;
a module for determining the set of artists recommended for work on the translation of the document, based on the history of the work they performed earlier and professional quality assessments set for documents similar to the source file for translation;
a module that collects data on all actions performed by executors in the web interface of work on the translation of documents;
module that performs assembly of the final translations of each individual segment into the final file with the translation. 2. The system of claim 1, wherein the indexed logical segments and their translation are stored in the translation memory together with the metadata corresponding to the segment. 3. The system according to claim 2, in which the metadata stored together with the indexed logical segments includes data on the time spent working with the segment, the executors who performed the translation, the number of changes of various types made to the translation by various executors, as well as professional quality assessments assigned to this logical segment by linguistic quality control specialists. 4. The system of claim 1, wherein the logical segments can be phrases, sentences, parts of sentences, or idiomatic expressions. 5. The system according to claim 1, in which parallel logical segments from the translation memory are mapped to the original logical segment in accordance with a similarity metric of segments, which is a metric of coincidence of two lines, taking into account the possibility of having several different valid word forms for the same the words. 6. The system according to claim 1, in which the user interface in real time displays the translation of logical segments performed by many executors working on the translation of the document at the moment. 7. The system of claim 1, wherein the translation can be performed by both professional translators and machine translation systems. 8. The system according to claim 1, in which the performers are selected from the group including translators, editors, proofreaders. 9. The method of translating the source file from the original language into the target language, consisting of the following steps:
receiving a request to translate the source file in a distributed network system;
splitting the source file into many logical segments;
search for similar segments and segments with similar terminology in the database of linguistic resources, consisting of previously translated parallel logical segments, dictionaries and glossaries, with the calculation of the quality metric for the logical segments found in the translation memory;
sending multiple requests for the translation of each segment to several machine translation systems located in a distributed network, with the transfer to each machine translation system of a data set accompanying the source document for individual setting of the machine translation system, as well as the transfer for each segment of the set of translation options of part of the text for machine translation translation of parts for which there is no translation option, and assembly of the final version of machine translation;
create a set of translation options based on segments similar to the source segment, in which parts of the text of the source segment are marked with translation options from similar segments, and parts that do not have a translation option are translated in accordance with the glossaries, dictionaries and phrases generated by machine translation systems from translation memory databases;
collecting a variety of segment translation options from various machine translation systems;
choosing the best machine translation option from the number of options received;
selection of the most suitable professional performers registered in the system, based on the predicted level of quality of their work, which is calculated on the basis of previously translated documents similar to this source document;
sending invitations to selected artists to participate in the project;
providing multiple artists with a web-based user interface for simultaneous multi-user translation work in real time;
assembly translations of individual segments into the resulting translated file. 10. The method according to p. 9, in which the source file can have either text or binary format. 11. The method according to p. 9, in which a professional assessment of the quality of the translation is calculated by fixing the type and criticality of errors in the selection of segments of the translated document, the translation of the document can be performed either from scratch by the translator or by editing a preliminary machine translation. 12. The method according to p. 9, in which the best version of machine translation is selected both among the basic translation options, and translation options generated by applying an advanced technology for working with translation memory, in which a set of translation options is created with text markup of the source segment in parts of parallel segments from the translation memory, indicating the translation option for this part, with the substitution by the machine translation system for unallocated parts from the glossary, dictionary or set of phrases generated by the system machine translation from translation memory. 13. The method of claim 9, wherein the translated segments of the source document are stored in the translation memory for future use.

Images