CN111539196A - Text duplicate checking method and device, text management system and electronic equipment - Google Patents

Abstract

The embodiment of the application provides a text duplicate checking method, a text duplicate checking device, a text management system and electronic equipment, wherein the text duplicate checking method comprises the following steps: acquiring a plurality of parts of duplicate texts to be checked; performing word division on each duplicate text to be checked according to a word database, and determining the word frequency of each word in the duplicate text to be checked; determining word vectors and vector weights of the word vectors in each text to be checked according to the words and the word frequencies corresponding to the words; and determining the repetition rate between any two texts to be checked according to the vector weight. According to the text duplication checking method, text words are divided through the word database, words appearing in the word database and word frequency of the words in the text are determined, vector weights of the words are calculated according to the words and word vectors obtained according to the corresponding word frequency, and then the duplication rate of any two texts is determined through comparison of the vector weights, so that the duplication rate comparison of any two texts can be performed efficiently and accurately.

Description

Method, device, text management system and electronic device for text duplication checking

technical field

The present application relates to the field of computer technology, and in particular, the present application relates to a text duplication checking method, device, text management system and electronic device.

Background technique

In the process of business management, bidding is a relatively frequent activity. In order to choose a partner that is more in line with the interests of all parties, it is necessary to read and review the bidding materials provided by the bidder, and weigh the optimal partner accordingly. However, a tenderer often corresponds to many bidders, and the number of bids received is large, and each bid has a lot of content, so it is difficult to find a suitable bidder in a short period of time. Reading and screening bids is a tedious task that requires a lot of manpower and material resources, and because the bids are all for the same project theme, there is usually a situation where the content is relatively similar. According to the selection of business partners, it will be a time-consuming and labor-intensive work. Therefore, people usually use some text-checking techniques.

In the prior art, segmentation technology and semantic analysis technology are usually used, and the longest string matching principle is used to calculate the repetition rate of characters in the document. However, the semantic analysis technology requires high-performance hardware devices to support the semantic recognition of bid documents, which often has the problem of excessive computing resources. Moreover, the technology is relatively closed and the scope of application and promotion is narrow. However, based on the longest string matching principle, many keywords are often missed, and the accuracy of document duplication checking is not high enough.

SUMMARY OF THE INVENTION

Aiming at the shortcomings of the existing methods, the present application proposes a method, device, text management system and electronic equipment for text duplication checking to solve the technical problem of low efficiency or accuracy of text duplication checking in the prior art.

In a first aspect, an embodiment of the present application provides a method for checking text duplicates, including:

Get a number of duplicate texts to be checked;

According to the word database, word division is performed for each text to be checked, and the word frequency of each word in the text to be checked is determined;

According to the word and the word frequency corresponding to the word, determine the word vector and the vector weight of the word vector in each text to be checked;

According to the vector weight, the repetition rate between any two texts to be checked is determined.

In some implementations of the first aspect, according to the word database, words are divided for each repeated text to be checked, and the word frequency of each word in the repeated text to be checked is determined, including:

According to the preset semantic strategy, the text to be checked is divided into several words;

If the word exists in the word database, the word is determined to be the first type of word;

If the word does not exist in the word database, the word is determined to be the second type of word;

Determine the word frequency of each first-class word in the text to be checked.

In combination with the first aspect and the above implementation manner, in some implementation manners of the first aspect, the word frequency of each word in the text to be checked is determined, including:

Determine the total number of words in the duplicate text to be checked, and the number of repetitions of each word;

According to the total number of words and the number of repetitions of each word, the word frequency of each word in the text to be checked is determined.

In combination with the first aspect and the above implementations, in some implementations of the first aspect, the word vector and the vector weight of the word vector in each text to be checked are determined according to the word and the word frequency corresponding to the word, including:

According to all the words of the repeated text to be checked, a text vector space is formed;

Determine the word vector of each word according to each word of the duplicate text to be checked and the word frequency corresponding to each word;

According to the text vector space and the word vector of each word, determine the weight of the word vector in the text vector space.

In combination with the first aspect and the above implementation manner, in some implementation manners of the first aspect, the similarity of any two duplicate texts to be checked is determined according to the vector weight, including:

Determine all the same words in the two duplicate texts to be checked;

According to the vector weights of the same words in the two documents to be checked, the coincidence rate of the same words in the two documents to be checked is determined;

According to the coincidence rate of all the same words, the repetition rate of any two duplicate texts to be checked is determined.

In a second aspect, the application provides a text checking device, including:

The acquisition module is used to acquire several duplicate texts to be checked;

The word segmentation module is used to divide words for each text to be checked according to the word database, and determine the word frequency of each word in the text to be checked;

The vector module is used to determine the word vector and the vector weight of the word vector in each text to be checked according to the word and the word frequency corresponding to the word;

The duplicate checking module is used to determine the repetition rate of any two duplicate texts to be checked according to the vector weight.

In a third aspect, the present application provides a text management method, including:

Obtain authentication information and determine the authentication result;

According to the authentication result, process the text corresponding to the authentication information;

According to the method for text duplication checking as described in the first aspect of the present application, the text is checked for duplication.

In a fourth aspect, the application provides a text management system, including:

The account management device is used to obtain the identity verification information and determine the identity verification result;

a file transmission device, processing the text corresponding to the identity verification information according to the identity verification result;

A text duplication checking device is used to check text for duplication according to the method for text duplication checking described in the first aspect of the present application.

In a fifth aspect, the present application provides an electronic device, including:

processor;

a memory, electrically connected to the processor;

At least one program, stored in the memory and configured to be executed by the processor, is configured to: implement the method for text duplication checking as described in the first aspect of the present application.

In a sixth aspect, the present application provides a computer-readable storage medium, where the computer storage medium is used to store computer instructions, and when the computer instructions are executed on a computer, the above-mentioned text duplication checking as described in the first aspect of the present application is implemented. Methods.

The beneficial technical effects brought by the technical solutions provided in the embodiments of the present application are:

The text duplicate checking method provided by this application divides the text words through the word database, determines each word in the text that appears in the word database and the word frequency of the word, and calculates the vector weight of the word according to the word vector obtained by the word and the corresponding word frequency, and then Through the comparison of vector weights, the similarity between two texts is determined. This method divides words accurately and efficiently, so it can efficiently and more accurately compare the repetition rate of any two texts.

Additional aspects and advantages of the present application will be set forth in part in the following description, which will become apparent from the following description, or may be learned by practice of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a schematic flowchart of a text management method provided by an embodiment of the present application;

2 is a schematic structural framework diagram of a text management system provided by an embodiment of the present application;

3 is a schematic flowchart of a method for text duplication checking provided by an embodiment of the present application;

4 is a schematic flowchart of a method for performing word division on each to-be-checked text according to a word database provided by the embodiment of the present application, and determining the word frequency of each word in the to-be-checked text;

5 is a schematic flowchart of a method for determining a word vector and a vector weight of the word vector in each text to be checked according to the word and the word frequency corresponding to the word according to the embodiment of the present application;

6 is a schematic flowchart of a method for determining the repetition rate of duplicate texts to be checked in pairs according to vector weights provided by an embodiment of the present application;

7 is a schematic structural framework diagram of a text checking device provided by an embodiment of the present application;

FIG. 8 is a schematic structural framework diagram of an electronic device according to an embodiment of the present application.

Detailed ways

The application is described in detail below, and examples of embodiments of the application are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar components or components having the same or similar functions throughout. Also, detailed descriptions of known technologies are omitted if they are not necessary for illustrating features of the present application. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, but not to be construed as a limitation on the present application.

It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It should also be understood that terms, such as those defined in a general dictionary, should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as herein, should not be interpreted in idealistic or overly formal meaning to explain.

It will be understood by those skilled in the art that the singular forms "a", "an", "the" and "the" as used herein can include the plural forms as well, unless expressly stated otherwise. It should be further understood that the word "comprising" used in the specification of this application refers to the presence of stated features, integers, steps, operations, elements and/or components, but does not preclude the presence or addition of one or more other features, Integers, steps, operations, elements, components and/or groups thereof. It should be understood that the term "and/or" as used herein includes all or any element and all combinations of one or more of the associated listed items.

In the experience activities, enterprises often need to cooperate with other enterprises to engage in a certain work, so bidding management is one of the necessary activities for enterprises. However, using the duplication checking method provided in the prior art, either high-performance hardware is required to provide sufficient accurate duplication checking effect and duplication checking efficiency, or the duplication checking accuracy rate is insufficient, which is difficult to be used as an application reference.

In order to solve the above problems, the present application provides a method, device, document management system and electronic device for checking text duplicates.

First, the embodiments of the present application provide a text management method from an overall perspective, and the text management method can be used to manage bidding documents or texts. The schematic flowchart of the text management method is shown in FIG. 1 , and includes at least the following step:

S100: Acquire authentication information, and determine an authentication result.

S200: According to the identity verification result, process the text corresponding to the identity verification information.

S300 : Check the text for duplication according to the method for checking the text for duplication described in this application.

The method for text duplication checking described in this application will be described in detail later.

Based on the same inventive concept, the embodiments of the present application correspondingly provide a text management system for implementing the above-mentioned text management method. Specifically, as shown in FIG. 2 , the text management system at least includes the following devices: Management device, file transfer device and text check device. The account management device is used for acquiring identity verification information and determining the identity verification result. The file transmission device is used for processing the text corresponding to the identity verification information according to the identity verification result. The text duplication checking device is used to check the text for duplication according to the method for text duplication checking as described in this application.

Specifically, the account management device is mainly responsible for implementing the account management functions of the text management system provided by the embodiments of the present application, and these account management functions include account registration, account management, account login and other functions. For example, account registration means collecting the bidder's input information, and generating the bidder's account after approval. Account management means entering the account information of the bidders who have successfully registered into the database for management. Account login means to verify whether the current login account is a registered account. After the verification is successful, you can enter the bidder operation interface. Of course, through the account management device, tenderers and bidders corresponding to different system rights can also be distinguished.

Correspondingly, the file transmission device is responsible for realizing the file or text transmission function of the system, including the text uploading and text downloading functions. Text upload means permitting bidders to upload text to the designated directory, while text download means permitting the tenderee or bidder to download the uploaded document. That is, bidders who successfully log in to the text management system through the account management device can upload their own bid documents or download bid documents and other documents corresponding to the bidder's authority.

The text duplication checking device is responsible for realizing the function of text duplication checking, and searching for relevant texts in the text management system. For example, the search scope is the text uploaded by all bidders in a certain bidding activity, and the duplication checking result is in the form of the similarity of any two texts. given. The detailed check method can be found in the following introduction.

In addition, the text management system provided by the embodiment of the present application further includes a text reading device, and the text reading device is a document reader using a local text editing tool, such as a Word program, by enabling a silent Word process, to Handle operations such as reading and editing of text, this system can handle files in .doc and .docx formats. The text reading device can decode the text completed by the existing Word program of the machine, thereby reducing the program load.

Through the overall design of the text management system, the design of each entity and its attributes has been completed, and the attributes of the enterprise bidding database can be determined, which can guide the completion of the construction of the bidding database. The text management system provided by the embodiment of the present application mainly includes three main functional devices, which are presented as a modular design structure. The construction of the entire text management system is clear in thinking, clear and concise in structure, and can conveniently manage bidding activities and ensure bidding and bidding activities. The text analysis of the activity is carried out efficiently.

The technical solution of the text duplication checking of the present application and how the technical solution of the present application solves the high-efficiency text duplication checking will be described in detail below with specific embodiments.

An embodiment of an aspect of the present application provides a method for checking text duplication. As shown in FIG. 3 , the method for checking text duplication at least includes the following steps:

S310: Acquire a number of duplicate texts to be checked.

S320: Divide each word for repeated text to be checked according to the word database, and determine the word frequency of each word in the repeated text to be checked.

S330: Determine the word vector and the vector weight of the word vector in each text to be checked according to the word and the word frequency corresponding to the word.

S340: Determine the repetition rate between any two duplicate texts to be checked according to the vector weight.

The text duplicate checking method provided by this application divides the text words through the word database, determines each word in the text that appears in the word database and the word frequency of the word, and calculates the vector weight of the word according to the word vector obtained by the word and the corresponding word frequency, and then Through the comparison of vector weights, the repetition rate between two texts is determined. This method divides words accurately and efficiently, so it can efficiently and more accurately compare the repetition rate of any two articles.

Feasible, in an implementation manner of the above-mentioned embodiment of the present application, S310: according to the word database, perform word division on each repeated text to be checked, and determine the word frequency of each word in the repeated text to be checked, as shown in FIG. 4 . This step specifically includes:

S311: Divide the text to be checked into several words according to a preset semantic strategy.

S312: If the word exists in the word database, determine that the word is a word of the first type; if the word does not exist in the word database, determine that the word is a word of the second type.

S313: Determine the word frequency of each first-type word in the text to be checked.

Common word segmentation techniques include word-based full-text indexing or word-based full-text indexing. Word-based full-text indexing improves the accuracy of word segmentation and extracts keywords as index items, so the accuracy of duplicate checking is greatly improved. However, due to the particularity of the Chinese language, there may be situations where a single character is a word, or there may be three or more characters that constitute a word and other words. Therefore, the preset semantic strategy is to perform text scanning and word segmentation according to the semantic division rules determined by the implementation. Some preset semantic strategies use a single character as a word, while others use two characters as a word, and so on.

Since the bidding document is a text file generated for a certain project subject, there are a lot of standardized words in this text file, and there are a lot of repeated words in each text in the professional statement of the procurement plan or engineering construction plan. Therefore, the text duplication checking method provided by the present application uses the word database to correct the words divided by the preset semantic strategy, and the word segmentation efficiency is improved by this method, thereby improving the text duplication checking efficiency and accuracy. The word database is an industry dictionary. This word database includes enough keywords that have been counted by enterprises in the long-term bidding work, and the word database can be continuously updated and iterated, and can also be updated by manual editing. iterate.

When the above S310 is executed, after the text to be repeated is divided into words according to a preset semantic strategy, the text to be repeated exists in the form of many scattered words, and there will be a large number of the same words among these words, and some words are not. It is the word needed in this bidding activity. For example, for the content of "our company", through the preset semantic strategy, it may be divided into two words with independent meanings, "me" and "company", and in the word database The words "I" and "si" do not exist, so these two words are divided into the second category of words. Obviously, the system running the method of text duplication will change the preset semantic strategy, and use "our company" as a separate word, and if the word "our company" exists in the word database, then "our company" is the first A class of words, and count the number of times this word appears in the entire text, that is, word frequency.

Through the word division of different preset semantic strategies, different text words and word frequencies corresponding to the words can be obtained. In order to improve the efficiency of word division, it is possible to check and count the semantic strategy types of word division in the word database to be used, and use the semantic strategy with a larger proportion as the preset semantic strategy to avoid repeated attempts to preset semantic strategies.

Feasible, in a specific implementation of the above embodiment, the step of determining the word frequency of each word in the duplicate text to be checked in S320 specifically includes: determining the total number of words in the duplicate text to be checked, and the frequency of each word. repeat times. According to the total number of words and the number of repetitions of each word, the word frequency of each word in the text to be checked is determined.

After dividing the text to be checked by words to form a data file composed of many words, the total number of words contained in the text to be checked can be counted. There will be a large number of repeated words in these words, so each word will exist correspondingly. a number of repetitions. In order to more objectively count the frequency of words appearing in texts and compare them between texts, a relative word frequency can be obtained by comparing the number of repetitions of words with the total number of words. The length of the tenders is different, which leads to the problem that the repeated results are quite different.

Feasible, in an implementation manner of the above embodiment of the present application, S330: According to the word and the word frequency corresponding to the word, determine the word vector and the vector weight of the word vector in each text to be checked, as shown in Figure 5, This step specifically includes:

S331: Construct a text vector space according to all words in the duplicate text to be checked.

S332: Determine the word vector of each word according to each word of the repeated text to be checked and the word frequency corresponding to each word.

S333: Determine the weight of the word vector in the text vector space according to the text vector space and the word vector of each word.

In order to obtain the repetition rate between different texts statistically, it is necessary to process the words and the word frequencies corresponding to the words. Vectorize. Through the SVM algorithm, the text to be checked is converted into a specific vector space, and each vector in the vector space is composed of a word and the word frequency corresponding to the word. In the foregoing steps, S331 and S332 do not need to distinguish the sequence. Then, the weight of each vector in the vector space is calculated by the weight calculation method. The commonly used weight calculation method can be performed by the TF-IDF (Term Frequency-Inverse Document Frequency, term frequency inverse document frequency) function.

Feasible, in an implementation manner of the above embodiment of the present application, S340: Determine the repetition rate of any two duplicate texts to be checked according to the vector weight, as shown in FIG. 6 , the step specifically includes:

S341: Determine all the same words in the two duplicate texts to be checked.

S342: Determine the coincidence rate of the same word in the two documents to be checked according to the vector weights of the same word in the two documents to be checked.

S343: Determine the repetition rate of any two duplicate texts to be checked according to the coincidence rate of all the same words.

Through the foregoing method steps, after the system determines the weight of each word vector in each text vector space, the system can specifically calculate the similarity between a certain text and each text except this text according to the cosine algorithm . The detailed calculation process can be as follows:

First, the segmented text is specialized by SVM model into feature vector V(d)=(t ₁ ,ω ₁ (d);...;t _n ,ω _s (d)), where t _i (i=1 ,2,...,s) is a list of different terms, s is a positive integer, ω _i (d) is the weight of t _i in the text vector space d, which is generally defined as t _i in the text vector space d The function of the frequency of occurrence tf _i (d) in , the formula form is: ω _i (d)=ψ(tf _i (d)). Specifically, ω _i (d) is calculated by formula (1) and formula (2). Then, the similarity of the two duplicate texts to be checked is obtained by the cosine similarity formula (3), that is, the degree of repetition of the two duplicate texts to be checked is obtained, that is, the repetition rate.

N in formula (1) and formula (2) is the number of all texts, and n _i is the number of texts containing the entry t _i .

余弦相似度公式(3)，d_i表示第一篇文本对应的文本向量空间，d_j表示第二篇文本对应的文本向量空间，m是文本向量空间中的词条数目。

Cosine similarity formula (3), d _i represents the text vector space corresponding to the first text, d _j represents the text vector space corresponding to the second text, and m is the number of entries in the text vector space.

Based on the same inventive concept, another embodiment of the present application provides a text duplication checking apparatus 10. As shown in FIG. 7, the text duplication checking apparatus 10 specifically includes an acquisition module 11, a word segmentation module 12, a vector module 13 and a search Heavy module 14.

Wherein, the obtaining module 11 is used to obtain several copies of duplicate texts to be checked.

The word segmentation module 12 is configured to perform word division on each text to be repeated according to the word database, and determine the word frequency of each word in the text to be repeated.

The vector module 13 is configured to determine the word vector and the vector weight of the word vector in each text to be checked according to the word and the word frequency corresponding to the word.

The duplicate checking module 14 is configured to determine the repetition rate of any two duplicate texts to be checked according to the vector weight.

The text checking device provided by the present application divides the text words through the word database, determines each word in the text that appears in the word database and the word frequency of the word, and calculates the vector weight of the word according to the word vector obtained by the word and the corresponding word frequency, and then Through the comparison of vector weights, the repetition rate between two texts can be determined, which can efficiently and more accurately compare the repetition rate of any two articles.

Feasible, the word segmentation module 12 performs word division on each repeated text to be checked according to the word database, and determines the word frequency of each word in the repeated text to be checked, including: dividing the repeated text to be checked into several parts according to a preset semantic strategy. words. If the word exists in the word database, the word is determined to be a first-class word. If the word does not exist in the word database, the word is determined to be a second-class word. Determine the word frequency of each first-class word in the text to be checked.

It is feasible that the word segmentation module 12 determines the word frequency of each word in the text to be repeated, including: determining the total number of words in the text to be repeated, and the number of repetitions of each word. According to the total number of words and the number of repetitions of each word, the word frequency of each word in the text to be checked is determined.

It is feasible that the vector module 13 determines the word vector and the vector weight of the word vector in each text to be checked according to the word and the word frequency corresponding to the word, including: constructing a text vector space according to all the words of the text to be checked. According to each word of the duplicate text to be checked and the word frequency corresponding to each word, the word vector of each word is determined. According to the text vector space and the word vector of each word, determine the weight of the word vector in the text vector space.

It is feasible that the duplicate checking module 14 determines the repetition rate of any two duplicate texts to be checked according to the vector weight, including: determining all the same words in the two duplicate texts to be checked. According to the vector weights of the same words in the two documents to be checked, the coincidence rate of the same words in the two documents to be checked is determined. According to the coincidence rate of all the same words, the repetition rate of the two duplicate texts to be checked is determined.

Based on the same inventive concept, an embodiment of the present application provides an electronic device, which includes: a memory and a processor.

The memory is electrically connected to the processor.

At least one computer program, stored in the memory, is used to implement any text duplication checking method provided by the embodiments of the present application/various optional methods for implementing the text duplication checking methods provided by the embodiments of the present application when executed by the processor implementation.

Those skilled in the art may understand that the electronic device provided by the embodiments of the present application may be specially designed and manufactured for the required purpose, or may also include known devices in a general-purpose computer. These devices have computer programs stored in them that are selectively activated or reconfigured. Such a computer program may be stored in a device (eg, computer) readable medium or in any type of medium suitable for storing electronic instructions and separately coupled to a bus.

Compared with the prior art, the electronic device provided by the present application can compare the repetition rate of any two articles with high efficiency and accuracy.

The present application provides an electronic device in an optional embodiment. As shown in FIG. 8 , the electronic device 1000 shown in FIG. 8 includes: a processor 1001 and a memory 1003 . Wherein, the processor 1001 and the memory 1003 are electrically connected, such as through a bus 1002 .

The processor 1001 may be a Central Processing Unit (CPU), a general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or an FPGA (Field-Programmable Gate Array). , field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logical blocks, modules and circuits described in connection with this disclosure. The processor 1001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

The bus 1002 may include a path to communicate information between the aforementioned components. The bus 1002 may be a PCI (Peripheral Component Interconnect, Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture, Extended Industry Standard Architecture) bus or the like. The bus 1002 can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in FIG. 8, but it does not mean that there is only one bus or one type of bus.

The memory 1003 can be a ROM (Read-Only Memory, read only memory) or other types of static storage devices that can store static information and instructions, a RAM (random access memory, random access memory) or other types that can store information and instructions It can also be an EEPROM (Electrically Erasable Programmable Read Only Memory, Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read-Only Memory, CD Read Only Memory) or other CD storage, CD storage (including compressed compact disc, laser disc, compact disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer any other medium, but not limited to this.

Optionally, the electronic device 1000 may also include a transceiver 1004 . The transceiver 1004 may be used for the reception and transmission of signals. Transceiver 1004 may allow electronic device 1000 to communicate wirelessly or by wire with other devices to exchange data. It should be noted that, in practical applications, the transceiver 1004 is not limited to one.

Optionally, the electronic device 1000 may further include an input unit 1005 . The input unit 1005 may be used to receive input numbers, characters, images and/or sound information, or generate key signal input related to user settings and function control of the electronic device 1000 . The input unit 1005 may include, but is not limited to, one or more of a touch screen, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), trackball, mouse, joystick, camera, pickup, and the like.

Optionally, the electronic device 1000 may further include an output unit 1006 . The output unit 1006 may be used to output or display the information processed by the processor 1001 . The output unit 1006 may include, but is not limited to, one or more of a display device, a speaker, a vibration device, and the like.

Although FIG. 8 shows the electronic device 1000 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

Optionally, the memory 1003 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 1001 . The processor 1001 is configured to execute the application program code stored in the memory 1003, so as to implement any method for checking text duplicates provided by the embodiments of the present application.

Based on the same inventive concept, the embodiments of the present application provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, any one of the embodiments of the present application is implemented. Method for Checking Text Duplication/Various optional implementation manners for implementing the method for checking text duplication provided in the embodiments of the present application.

The embodiments of the present application provide a computer-readable storage medium. Compared with the prior art, the computer-readable storage medium provided by the present application can efficiently and more accurately perform the analysis of any two articles through the computer program stored therein. Comparison of repetition rates.

Those skilled in the art can understand that various operations, methods, steps, measures, and solutions in the process discussed in this application may be alternated, modified, combined or deleted. Further, other steps, measures, and solutions in various operations, methods, and processes that have been discussed in this application may also be alternated, modified, rearranged, decomposed, combined, or deleted. Further, steps, measures and solutions in the prior art with various operations, methods, and processes disclosed in this application may also be alternated, modified, rearranged, decomposed, combined or deleted.

The terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may expressly or implicitly include one or more of that feature. In the description of this application, unless stated otherwise, "plurality" means two or more.

In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it may be a fixed connection or a detachable connection Connection, or integral connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood in specific situations.

In the description of this specification, the particular features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples.

It should be understood that although the various steps in the flowchart of the accompanying drawings are sequentially shown in the order indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order and may be performed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution sequence is also It does not have to be performed sequentially, but may be performed alternately or alternately with other steps or at least a portion of sub-steps or stages of other steps.

The above are only part of the embodiments of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, several improvements and modifications can also be made. It should be regarded as the protection scope of this application.

Claims (10)

1. a method for text checking, is characterized in that, comprises: Get a number of duplicate texts to be checked; According to the word database, word division is performed on each of the text to be checked, and the word frequency of each word in the text to be checked is determined; According to the word and the word frequency corresponding to the word, determine the word vector and the vector weight of the word vector in each of the texts to be checked; According to the vector weight, the repetition rate between any two of the to-be-checked duplicate texts is determined. 2. The method for checking duplicate texts according to claim 1, wherein, according to a word database, each of the duplicate texts to be checked is divided into words, and it is determined that each word is in the duplicate texts to be checked. word frequencies, including: According to a preset semantic strategy, the to-be-checked text is divided into several words; If the word exists in the word database, determining that the word is a first-class word; If the word does not exist in the word database, determining that the word is a second type of word; Determine the word frequency of each of the first type of words in the to-be-checked text. 3. The method for text duplication checking according to claim 1, wherein the determining the word frequency of each word in the text to be checked for duplication comprises: Determine the total number of words in the duplicate text to be checked, and the number of repetitions of each word; According to the total number of words and the number of repetitions of each word, the word frequency of each word in the text to be checked is determined. 4 . The method for checking text duplicates according to claim 1 , wherein, according to the word and the word frequency corresponding to the word, the word vector and the word vector are determined in each of the texts to be checked for duplicates. 5 . vector weights in , including: According to all the words of the repeated text to be checked, a text vector space is formed; Determine the word vector of each of the words according to each word of the to-be-checked text and the word frequency corresponding to each of the words; According to the text vector space and the word vector of each of the words, the weight of the word vector in the text vector space is determined. 5. The method for text duplication checking according to claim 1, wherein, according to the vector weight, determining the repetition rate of any two of the duplicate texts to be checked, comprising: Determine all the same words in the two duplicate texts to be checked; According to the vector weights of the same words in the two documents to be checked, the coincidence rate of the same words in the two documents to be checked is determined; According to the coincidence rate of all the same words, the repetition rate of any two duplicate texts to be checked is determined. 6. A text checking device, characterized in that, comprising: The acquisition module is used to acquire several duplicate texts to be checked; A word segmentation module, configured to perform word division on each of the repeated texts to be checked according to a word database, and determine the word frequency of each word in the repeated texts to be checked; a vector module, configured to determine the word vector and the vector weight of the word vector in each of the texts to be checked according to the word and the word frequency corresponding to the word; A duplicate checking module, configured to determine the repetition rate of any two of the duplicate texts to be checked according to the vector weight. 7. A method for text management, comprising: Obtain authentication information and determine the authentication result; According to the identity verification result, process the text corresponding to the identity verification information; According to the method for checking text duplication according to any one of claims 1 to 5, the text is checked for duplication. 8. A text management system, comprising: The account management device is used to obtain the identity verification information and determine the identity verification result; a file transmission device, configured to process the text corresponding to the identity verification information according to the identity verification result; A text duplication checking device is used for checking the text for duplication according to the method for checking text duplication according to any one of claims 1 to 5. 9. An electronic device, characterized in that, comprising: processor; a memory, electrically connected to the processor; at least one program, stored in the memory and configured to be executed by the processor, the at least one program being configured to: implement the text duplication check according to any one of claims 1 to 5 method. 10 . A computer-readable storage medium, wherein the computer storage medium is used to store computer instructions, and when the computer instructions are executed on a computer, the computer instructions according to any one of claims 1 to 5 are implemented. 11 . A method of text duplication checking.

Images