CN111460810A - Crowd-sourced task spot check method and device, computer equipment and storage medium - Google Patents

Abstract

The present application discloses a method, device, computer equipment and storage medium for sampling inspection of crowdsourcing tasks, including obtaining, for each historical crowdsourcing task, each response object participating in the historical crowdsourcing task, and the corresponding response of each response object Answer; analyze and process the answer to obtain the spot check word, and extract the spot check keyword from the spot check word segment, and store the spot check keyword in the preset answer thesaurus; Word segmentation, counting the number of times that the word segmentation hits the sampling keyword in the preset answer thesaurus, as the basic times corresponding to each answer, and determines the reliability corresponding to each answer object according to the basic times corresponding to each answer value; determine the sampling object according to the reliability value, and check the response answer corresponding to the sampling object. The application can extract the keywords of sampling inspection, strengthen the pertinence of sampling inspection, and improve the efficiency of sampling inspection.

Description

Sampling method, device, computer equipment and storage medium for crowdsourcing tasks

technical field

The present application relates to the technical field of data processing, and in particular, to a sampling inspection method, device, computer equipment and storage medium for crowdsourcing tasks.

Background technique

With the rapid development of network technology, in order to obtain more creative information or solve some cross-domain problems efficiently and conveniently, some companies or institutions often issue crowdsourcing tasks to Internet objects through the Internet, and solve these problems through crowdsourcing tasks. question.

Crowdsourcing tasks refers to the practice of a company or institution outsourcing work tasks that were previously performed by employees to an unspecified (and usually large) mass network on a free and voluntary basis. The employees on the crowdsourcing platform are divided into two categories: those who post tasks on the platform are called task publishers, and those who complete tasks are called responders. The task publisher publishes the task on the platform, and the respondent gets a certain reward for completing the task. The way crowdsourced tasks work can help task issuers get a lot of free objects to solve real-world problems by leveraging the intelligence of those objects.

At present, due to the uncertainty of the field of expertise and professional level of the respondent, it is necessary to spot-check the correctness of the collected answers to the crowdsourcing tasks. When there are many answers, the inspection will take a long time. The current practice is to randomly select a preset number of answers from all the answers for inspection by random sampling, and carry out the crowdsourcing task according to the inspection results. Evaluation, this method of random sampling is less targeted, which makes the evaluation of the effect of crowdsourcing tasks unsatisfactory, resulting in the problem of low sampling efficiency of crowdsourcing tasks. How to select crowdsourcing tasks in a targeted manner and improve the sampling inspection of crowdsourcing tasks Efficiency has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present application is to propose a method for sampling inspection of crowdsourcing tasks, so as to solve the problem that the random sampling inspection method in the prior art is weak in pertinence, resulting in low sampling inspection efficiency of crowdsourcing tasks.

In order to solve the above-mentioned technical problems, the embodiment of the present application provides a sampling method for crowdsourcing tasks, including:

For each historical crowdsourcing task, obtain each response object participating in the historical crowdsourcing task, and the response answer corresponding to each response object;

Analyzing and processing the response answer, obtaining the word-sampling check, and extracting the keyword for spot-checking from the word-sampling check, and storing the keyword for spot-checking into a preset answer thesaurus;

For the sampling word segmentation in each response answer of each answering object, count the number of times the sampling word segmentation hits the sampling keyword in the preset answer thesaurus, as the number of times corresponding to each answer answer. base times;

According to the basic times corresponding to each response answer, determine the reliability value corresponding to each response object;

According to the order of the reliability values from small to large, a preset number of response objects are selected as sampling objects, and the response answers corresponding to the sampling objects are checked.

Further, performing the analysis and processing on the response answer, and obtaining the word segmentation by sampling inspection includes:

Using a dynamic programming algorithm, word segmentation is performed on the response answer to obtain an initial word segmentation;

Filter processing is performed on the initial word segmentation, and synonymous replacement is performed on the initial word segmentation after the filtering processing to obtain the sampling word segmentation.

Further, the extraction of sampling keywords from the sampling word segmentation, and storing the sampling keywords into a preset answer thesaurus includes:

For the same historical crowdsourcing task, obtain all the sampling word segments corresponding to the historical crowdsourcing task, and count the number of occurrences of each of the sampling word segmentations, to obtain the sampling word frequency corresponding to the sampling word segmentation;

The sampling word frequency of the sampling inspection word frequency is greater than the preset word frequency, as the sampling inspection keyword, and the sampling inspection keyword is stored in the preset answer thesaurus.

Further, performing synonymous replacement on the initial word segmentation after filtering, and obtaining the sampling word segmentation includes:

By means of named entity recognition, synonymous substitution is performed on the initial participle to obtain the sampling participle.

Further, performing synonymous replacement on the initial participle by means of named entity recognition includes:

Get the preset standard vocabulary dictionary;

For each of the initial participles, by means of traversal, perform named entity recognition on the initial participles and each vocabulary in the standard vocabulary dictionary, respectively, to obtain an entity recognition result;

If the entity recognition result is that the same named entity exists, the initial participle and standard vocabulary corresponding to the recognition result are acquired, and the standard participle is used to replace the initial participle.

Further, in the described sampling word segmentation in each response answer for the each response object, in a preset answer thesaurus, the number of times the sampling word segmentation hits the sampling keyword is used as each response. The base times corresponding to the answers include:

Obtain each historical crowdsourcing task that the respondent participates in as a reference task;

For each of the reference tasks, the sampling keyword is obtained from the preset answer thesaurus, and the number of times that the sampling word corresponding to the response answer of the answering object hits the sampling keyword is counted as the basic number of times.

Further, according to the basic times corresponding to each response answer, determining the reliability value corresponding to each response object includes: according to the basic times, count the sum M of the sampling keywords corresponding to each of the reference tasks. , and count the sum N of the number of hits of the response object for each of the reference tasks, where M and N are both positive integers, and N is less than or equal to M;

The formula δ=N/M is used for calculation to obtain the reliability value δ.

In order to solve the above-mentioned technical problems, a technical solution adopted by the present invention is to provide a sampling device for crowdsourcing tasks, including:

an obtaining module, used for obtaining the response answer of each response object corresponding to the historical crowdsourcing task for each historical crowdsourcing task;

The parsing module is used to parse and process the response answer, obtain the word-sampling check, and extract the keyword of the spot-checking of the word-sampling check, and store the keyword of the spot-checking into a preset answer thesaurus;

The statistical module is used to count the number of times that the sampling word hits the sampling keyword in the preset answer thesaurus for the sampling word segmentation in each response answer of the each response object, as The basic times corresponding to each answer;

A determination module, for determining the reliability value corresponding to each response object according to the basic times corresponding to each response answer;

The selection module is configured to select a preset number of response objects in the order of the reliability values from small to large as sampling objects, and to check the response answers corresponding to the sampling objects.

In order to solve the above-mentioned technical problem, a technical solution adopted by the present invention is to provide a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors can realize the above Any one of the sampling plans described.

In order to solve the above-mentioned technical problem, a technical solution adopted in the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned items is realized. the described sampling plan.

A method for sampling inspection of crowdsourcing tasks in the above scheme is to obtain the response answer of each response object corresponding to the historical crowdsourcing task for each historical crowdsourcing task, parse and process the response answer, and obtain the sampling word segmentation, and from the sampling inspection Sampling keywords are extracted from the word segmentation, and the sampling keywords are stored in the preset answer thesaurus. The obtained sampling keywords are used for subsequent evaluation of the reliability value of the response object, which makes the evaluation of the reliability value of the response object easier. It is pertinent; at the same time, for the sampling word segmentation in each response answer of each response object, the number of times the sampling word segmentation hits the sampling inspection keyword is counted as the basic number of times corresponding to each response answer, and each response object is determined according to the basic number of times. The corresponding reliability value is determined, and then the sampling object is determined according to the reliability value, and the response answer corresponding to the sampling object is checked. By determining the reliability value corresponding to each response object, and determining the sampling object according to the reliability value, the sampling inspection can be made more targeted, and by arranging the response objects according to the corresponding reliability value, it is beneficial to improve the sampling efficiency.

Description of drawings

In order to illustrate the solutions in the present application more clearly, the following will briefly introduce the accompanying drawings used in the description of the embodiments of the present application. For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic diagram of the application environment of the sampling method for crowdsourcing tasks provided by an embodiment of the present application;

Fig. 2 is a realization flow chart of the sampling method for crowdsourcing tasks provided according to an embodiment of the present application;

3 is a flow chart of an implementation of step S2 in the sampling method for crowdsourcing tasks provided by the embodiment of the present application;

4 is a flow chart of an implementation of step S221 in the sampling method for crowdsourcing tasks provided by the embodiment of the present application;

5 is a flow chart of an implementation of step S3 in the sampling method for crowdsourcing tasks provided by the embodiment of the present application;

6 is a flow chart of an implementation of step S4 in the sampling method for crowdsourcing tasks provided by the embodiment of the present application;

7 is a schematic diagram of a sampling device for crowdsourcing tasks provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of a computer device provided by an embodiment of the present application.

Detailed ways

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field of this application; the terms used herein in the specification of the application are for the purpose of describing specific embodiments only It is not intended to limit the application; the terms "comprising" and "having" and any variations thereof in the description and claims of this application and the above description of the drawings are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the description and claims of the present application or the above drawings are used to distinguish different objects, rather than to describe a specific order.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings.

The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

Referring to FIG. 1 , the system architecture 100 may include terminal devices 101 , 102 , and 103 , a network 104 and a server 105 . The network 104 is a medium used to provide a communication link between the terminal devices 101 , 102 , 103 and the server 105 . The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages and the like. Various communication client applications may be installed on the terminal devices 101 , 102 and 103 , such as web browser applications, search applications, instant communication tools, and the like.

The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

The server 105 may be a server that provides various services, such as a background server that provides support for the pages displayed on the terminal devices 101 , 102 , and 103 .

It should be noted that a method for sampling a crowdsourcing task provided by the embodiment of the present application is generally performed by a server, and correspondingly, a device for sampling a crowdsourcing task is generally set in the server.

It should be understood that the numbers of terminal devices, networks and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Please refer to FIG. 2, which shows a specific implementation of the crowdsourcing task sampling method.

It should be noted that, if there is substantially the same result, the method of the present invention is not limited to the flow sequence shown in FIG. 2, and the method includes the following steps:

S1: For each historical crowdsourcing task, obtain each response object participating in the historical crowdsourcing task, and the corresponding response answer of each response object.

Specifically, the server stores each historical crowdsourcing task, the response object, the response answer, and the mapping relationship between the response object, the response answer and the historical crowdsourcing task. For any historical crowdsourcing task, there is at least one Answer answer, each answer object has at most one answer answer for the same historical crowdsourcing task, and for each historical crowdsourcing task, obtain the answer answer of each answer object for the historical crowdsourcing task.

The crowdsourcing task in this embodiment refers to a network-based task in which objects are allowed to participate in the task and give corresponding answers through the network.

Among them, the response object refers to the object that gives the corresponding response answer to the historical crowdsourcing task, and may specifically be a plurality of preset network models. For example, for the crowdsourcing task A, a network model K uses a preset method to The crowdsourcing task A is identified and analyzed, and the answer is given, and the network model K can be called a response object of the crowdsourcing task A.

In this embodiment, an electronic device (for example, the server shown in FIG. 1 ) on which a method for spot checking a crowdsourcing task runs may be connected by a wired connection or a wireless connection. It should be noted that the above wireless connection methods may include, but are not limited to, 3G/4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or developed in the future.

S2: Perform analysis processing on the response answer to obtain sampling word segmentations, extract sampling keywords from the sampling segmentations, and store the sampling keywords in a preset answer thesaurus.

Specifically, the response answer is parsed to obtain word segmentation information that can express the semantics of the response answer, which is used as the sampling word segmentation, and keywords are extracted from the sampling segmentation word as the sampling inspection keyword, and the sampling keyword is stored in the preset answer. Thesaurus, the specific process can also refer to the description of step S21 to step S24, in order to avoid repetition, it will not be repeated here.

The parsing processing specifically includes but is not limited to: word segmentation processing, data cleaning, deduplication processing, and synonymous substitution.

Among them, word segmentation processing refers to the process of recombining consecutive word sequences into word sequences according to certain specifications. In this embodiment, it specifically refers to dividing the response answer into individual random word segmentations, so that these random word segmentations can be used for subsequent use. Sampling keyword extraction.

Among them, the word segmentation processing can be specifically performed by a third-party word segmentation tool or a word segmentation algorithm.

Among them, common third-party word segmentation tools include but are not limited to: Stanford NLP word segmentation tool, ICTClAS word segmentation system, ansj word segmentation tool and HanLP Chinese word segmentation tool.

Wherein, word segmentation algorithms include but are not limited to: maximum forward matching (Maximum Matching, MM) algorithm, reverse maximum matching (ReverseDirectionMaximum Matching Method, RMM) algorithm, two-way maximum matching (Bi-directction Matching method, BM) algorithm, dynamic programming algorithm , Hidden Markov Model (Hidden Markov Model, HMM) and N-gram model, etc.

Preferably, a dynamic programming algorithm is used to perform word segmentation processing in this embodiment, and reference may be made to the description of step S21 for the specific process. To avoid repetition, details are not repeated here.

Among them, data cleaning refers to a process of finding and correcting identifiable errors in data files, including checking data consistency, dealing with invalid and missing values, etc. In this embodiment, the text normative check is performed on the response answer and word segmentation, and the interference of invalid items is eliminated, so as to improve the accuracy and efficiency of subsequent extraction of random keywords.

It should be noted that, as a preferred method, in this embodiment, the sampling keywords are stored in the preset answer thesaurus, which specifically includes: obtaining historical crowdsourcing tasks corresponding to preset task types; storing the same preset tasks The sampling keywords corresponding to the type of historical crowdsourcing tasks are used as the sampling keywords in the same group; the mapping relationship between the sampling keywords in the same group and the preset task types is established, and the mapping relationship is stored in the preset answer thesaurus middle.

The preset task types can be set according to actual needs, which is not specifically limited here. For example, in a specific implementation manner, the preset task types include: quiz questions, multiple-choice questions, fill-in-the-blank questions, and judgment questions, etc. For another example, in another specific implementation manner, the preset task types include: material collection, art production, planning, publicity design, and the like.

By analyzing and processing the response answer, the sampling word segmentation is obtained, and the sampling keyword is extracted from the sampling segmentation word, and the sampling keyword can be extracted from the response answer of each response object from many historical tasks, thereby effectively improving the sampling inspection. targeted and able to provide sampling efficiency.

S3: For the sampling word segmentation in each response answer of each answering object, count the number of times the sampling wording word hits the sampling inspection keyword in the preset answer thesaurus, as the basic number corresponding to each answering object.

Specifically, in the preset answers, the number of times that the sampling word segmentation hits the sampling keyword is counted, and the basic number of times that can express the reliability of the sampling word segmentation is obtained. The specific process can also refer to the description of steps S31 and S32. To avoid repetition, here No longer.

The reliability value refers to evaluating the reliability of the object's response to the crowdsourcing task according to the object's response to the historical crowdsourcing task. Generally speaking, the higher the reliability value, the higher the risk value of the response answer returned by the object. smaller.

S4: Determine the reliability value corresponding to each response object according to the basic times corresponding to each response answer.

Specifically, each response answer corresponds to a response object, and the reliability value corresponding to each response object can be determined through the basic times corresponding to each response answer. The reliability value corresponding to each response object is obtained, which is used to determine the sequence of subsequent sampling inspections, which avoids random sampling inspections and enhances the pertinence of sampling inspections. For the specific process of S4, reference may also be made to the descriptions of steps S41 and S42, which are not repeated here to avoid repetition.

S5: Select a preset number of response objects in an ascending order of reliability values as sampling objects, and check the response answers corresponding to the sampling objects.

Specifically, the response objects are sorted in ascending order according to the reliability value to obtain a sequence of response objects; in the order from front to back, a preset number of response objects are selected from the sequence of response objects to be used for random inspection target object.

Among them, the selection of the preset number of response objects is set according to the actual sampling inspection needs, which is not specifically limited here. For example, the number of sampling inspections can be arranged according to the manpower situation of the sampling inspection. If the sampling inspection manpower can only check 100 selected objects, the reliability Values are in the top 100 answering objects.

Send the sampling results to the terminal devices 101, 102, and 103, so that the respondent can know the sampling results.

In this embodiment, for each historical crowdsourcing task, the response answer corresponding to the historical crowdsourcing task of each response object is obtained, and the response answer is parsed to obtain the spot check word, and the spot check keyword is extracted from the spot check word. The sampling keywords are stored in the preset answer thesaurus, and the obtained sampling keywords are used for subsequent evaluation of the reliability value of the response object, which makes the evaluation of the reliability value of the response object more targeted; Sampling word segmentation in each response answer of the respondent, count the number of times that the sampling word segment hit the sampling keyword, as the basic number of times corresponding to each response answer, and determine the reliability value corresponding to each response object according to the basic number of times, and then follow The reliability value determines the sampling object, and checks the response answer corresponding to the sampling object. By determining the reliability value corresponding to each response object, and determining the sampling object according to the reliability value, the sampling inspection can be made more targeted, and by arranging the response objects according to the corresponding reliability value, the situation of random sampling is avoided. It is beneficial to improve the sampling efficiency.

Please refer to FIG. 3. FIG. 3 shows a specific implementation of step S2. In step S2, the response answer is parsed to obtain the sampling word segmentation, and the sampling keyword is extracted from the sampling segmentation, and the sampling keyword is stored in the The specific implementation process to the preset answer thesaurus is described in detail as follows:

S21: Using a dynamic programming algorithm, perform word segmentation on the answer to obtain an initial word segmentation.

Specifically, in the corresponding historical response answers of the respondent, the response answers are often complicated and need to be simplified. By using the dynamic programming algorithm, the response answers are processed by word segmentation, and the word segmentation related to the sampling inspection is extracted to obtain the initial word segmentation. .

Among them, the dynamic programming algorithm is usually used to solve a problem with a certain optimal property; in this application, the dynamic programming algorithm is used to obtain the optimal initial word segmentation. The basic idea of the dynamic programming algorithm is to decompose the problem to be solved into several sub-problems, first solve the sub-problems, and then obtain the solution of the original problem from the solutions of these sub-problems.

Preferably, a Vibiter algorithm is used to perform word segmentation processing on the answer to obtain an initial word segmentation. The Vibit algorithm is a dynamic programming algorithm for finding observations that are most likely to explain the correlation.

S22: Perform filtering processing on the initial word segmentation, and perform synonymous replacement on the initial word segmentation after the filtering processing to obtain the word segmentation by sampling inspection.

Specifically, for the initial word segmentation obtained after the parsing process, the initial word segmentation is filtered, and unnecessary or redundant words are processed to obtain a word segmentation that is more suitable for the needs of sampling inspection, and then the filtered initial word segmentation is synonymous. Substitute to get random word segmentation. The purpose of synonymous substitution is to convert the synonyms and synonyms in the participle into a consistent standard vocabulary, and further simplify the participle, so that the spot-checked participle can be obtained.

Among them, the filtering process refers to filtering out the initial participles such as adjectives and adverbs that have little influence on the semantic description, and retaining the initial participles that play a key role in semantic expression, such as names, verbs and quantifiers.

Among them, synonymous substitution refers to converting synonyms and synonyms into unified standard vocabulary for representation.

For example, in a specific implementation manner, the obtained initial participles are "Zhang San", "Zheng", "Crazy", "CALL", and the two words "Zheng" and "Crazy" are both adverbs, It can be filtered, "Zhang San" is a specific person's name, and "CALL" indicates a specific action. Therefore, after the filtering process, the obtained "Zhang San" and "CALL" are two participles, and then the "Zhang San" and "CALL" are obtained. "CALL" to perform synonymous replacement, and get the sampling participles "Zhang San" and "Cheers".

S23: For the same historical crowdsourcing task, obtain all the sampling word segments corresponding to the historical crowdsourcing task, and count the occurrences of each sampling word segment, and obtain the sampling word frequency corresponding to the sampling word segment.

Specifically, for the same historical crowdsourcing task, all sampling word segments corresponding to the historical crowdsourcing task are obtained, and the occurrences of each sampling word segment are counted to obtain the sampling word frequency corresponding to the sampling word segment in the historical crowdsourcing task.

Among them, the frequency of sampling words is the frequency of the occurrence of the sampling word in the same historical crowdsourcing task, and the way of expressing the frequency of sampling words can be the number of occurrences of the sampling word, or the proportion of the occurrence of the statistical sampling, which can be based on the actual situation. Make settings.

For example, in a specific embodiment, a historical crowdsourcing task corresponds to 8 random word segmentations, which are: word segmentation 1, word segmentation 2, word segmentation 3, word segmentation 4, word segmentation 5, word segmentation 6, word segmentation 7 and word segmentation 8, folded 8 The number of occurrences of each sampling word is: 22, 5, 19, 25, 8, 1, 20, 2, and the corresponding sampling word frequencies are 22, 5, 19, 25, 8, 1, 20, 2.

S24: Use the sampling word frequency with the sampling word frequency greater than the preset word frequency as the sampling keyword, and store the sampling keyword in the preset answer thesaurus.

Specifically, the server stores the preset word frequency, and compares each sampling word frequency with the preset word frequency. When the sampling word frequency is greater than the preset word frequency, the sampling word corresponding to the sampling word frequency is used as the sampling keyword, and is stored in the preset word frequency. set in the answer thesaurus.

Among them, the preset word frequency can be set according to actual sampling needs.

In this embodiment, by performing word segmentation processing on the response answer, the initial word segmentation related to the random inspection can be obtained, and the initial word segmentation can be filtered and replaced by synonyms, which can further simplify the word segmentation, so that the sampling inspection word segmentation can be obtained, and then extracted from the sampling inspection word segmentation keyword, you can obtain more accurate sampling keywords.

In one embodiment, in step S22, filter processing is performed on the initial word segmentation, and synonymous replacement is performed on the initial word segmentation after the filtering processing to obtain the word segmentation by sampling inspection, which is described in detail, and the specific process is as follows:

By means of named entity recognition, the initial participles are replaced by synonyms, and the sampling participles are obtained.

Specifically, by means of named entity recognition, synonymous replacement is performed on the initial word segmentation after filtering to obtain the word segmentation by sampling.

Among them, Named Entity Recognition (NER) is a basic method for determining entity boundaries mainly related to word segmentation and discovering named entities. It is used to identify entities with specific meanings in texts. It is an important part of the practical application of natural language processing. It plays an important fundamental role in application fields such as information extraction, syntactic analysis, and machine translation. Named entity recognition needs to identify entity boundaries on the one hand, and entity categories, such as person names, place names, and institution names, on the other hand.

In this embodiment, by means of named entity recognition, synonymous replacement is performed on the initial word segmentation that has been filtered, and the initial word segmentation is further simplified to obtain random word segmentation, which improves the accuracy of random word segmentation.

Please refer to Fig. 4, Fig. 4 shows that in step S22, by means of named entity recognition, synonymous replacement is performed on the initial word segmentation to obtain the specific implementation process of the sampling word segmentation, which is described in detail as follows:

S221: Obtain a preset standard vocabulary dictionary.

Specifically, a standard vocabulary dictionary is set in the server in advance, and the standard vocabulary dictionary can effectively filter out relatively redundant vocabulary, such as some unnecessary prepositions, adjectives, or Synonyms and Synonyms.

S222: For each initial word segmentation, perform named entity recognition on the initial word segmentation and each word in the standard vocabulary dictionary by means of traversal, to obtain an entity recognition result.

Specifically, by performing named entity identification on each word in the standard vocabulary dictionary with the initial segmentation, different identification results are obtained, and the identification results may be the same or different named entities.

For example, the two identified named entities "NYC" and "New York" appear to be different strings, but they actually refer to the city of New York, which needs to be merged. For example, the two identified named entities " "CALL" and "Cheers" have different literal descriptions, but the corresponding semantics are the meaning of "CALL", which requires entity naming and merging.

S223: If the entity recognition result is that the same named entity exists, acquire the initial word segmentation and standard vocabulary corresponding to the recognition result, and use the standard word segmentation to replace the initial word segmentation.

Specifically, the entity recognition result is the existence of the same named entity, that is, the semantics corresponding to the two participles are the same meaning. By obtaining the initial participle and standard vocabulary corresponding to the recognition result, and using the standard participle to replace the initial participle, the sampling participle is obtained.

For example, an initial participle named entity "calls", another initial participle named entity "cheers", through the named entity recognition in step S222, it can be concluded that the entity recognition result is the same named entity, by obtaining the initial participle "call CALL" ", "Cheers" and the standard word "Cheers", replace "CALL" and "Cheers" with "Cheers", and get the sampling participle "Cheers".

In this embodiment, by further simplifying the initial participle, a sampling participle that is closer to the purpose of sampling inspection is obtained, words such as prepositions and adjectives that have no meaning in the initial participle are filtered out, and words with the same or similar meaning are replaced, which improves the selection of the sampling participle. precision.

Please refer to FIG. 5. FIG. 5 shows a specific implementation of step S3. In step S3, for the sampling word segmentation in each response answer of each response object, in the preset answer thesaurus, the sampling word segmentation hits the The number of sampling keywords, as the specific implementation process of the basic number corresponding to each answer, is described in detail as follows:

S31: Obtain each historical crowdsourcing task that the respondent participated in as a reference task.

Specifically, each historical crowdsourcing task participated by each respondent is stored in the server, and each respondent's response answer can be included by acquiring each historical crowdsourcing task participated by the respondent as a reference task.

S32: For each reference task, obtain the sampling keyword from the preset answer thesaurus, and count the number of times the sampling keyword corresponding to the response answer of the respondent hits the sampling keyword, as the basic number of times.

Specifically, due to the targeted requirements of sampling inspection, it is necessary to spot-check whether the response answer of each response object is more correct to the response answer of the crowdsourcing task; Statistics can be used to know the ability of different respondents to complete the crowdsourcing task.

In this embodiment, by acquiring each historical crowdsourcing task that the respondent participated in as a reference task, and determining the basic number of times, a basis can be provided for the subsequent determination of the reliability value.

Please refer to FIG. 6. FIG. 6 shows a specific implementation of step S4. In step S4, according to the basic times corresponding to each response answer, the specific implementation process of determining the reliability value corresponding to each response object is described in detail. as follows:

S41: According to the basic number of times, count the sum M of the sampling keywords corresponding to each reference task, and count the sum N of the hit times of the response object for each reference task, where M and N are both positive integers, and N is less than or equal to M .

S42: Calculate by using the formula δ=N/M to obtain the reliability value δ.

For example, the sum M of the sampling keywords corresponding to the reference task is 10, and the sum N of the number of hits of a response object for each reference task is 2, through the formula δ=N/M, that is, the reliability value δ is 0.2, and in addition The sum N of hits of a responding object for each reference task is 4, that is, the reliability value δ is 0.4; since the higher the reliability value, the smaller the risk value of the response answer fed back by the object, so the reliability value δ can be obtained. A risk value of 0.4 is smaller than a reliability value of δ of 0.2.

In this example, by counting the basic times and using the formula to obtain the specific reliability value, the random sampling of the response answers in the prior art can be solved, and through the accurate reliability value, it is possible to accurately know which response objects have responded. The answers are trustworthy, and which respondent's responses are too random.

It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

Those of ordinary skill in the art can understand that the realization of all or part of the processes in the methods of the above embodiments can be accomplished by instructing relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the program is During execution, it may include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM).

Please refer to FIG. 7 , as an implementation of the method shown in FIG. 2 above, the present application provides an embodiment of a sampling inspection device for crowdsourcing tasks. The device embodiment corresponds to the method embodiment shown in FIG. 2 . The device can be specifically applied to various electronic devices.

As shown in FIG. 7 , a sampling inspection device for crowdsourcing tasks in this embodiment includes: an acquisition module 51 , an analysis module 52 , a statistics module 53 , a determination module 54 , and a selection module 55 . in:

The obtaining module 51 is configured to obtain, for each historical crowdsourcing task, a response answer corresponding to the historical crowdsourcing task for each response object.

The parsing module 52 is used for parsing and processing the response answer, obtaining the sampling word segmentation, extracting the sampling keyword of the sampling segmentation word, and storing the sampling keyword in the preset answer thesaurus.

The statistics module 53 is used to count the number of times that the sampling word hits the sampling keyword for the sampling word in each response answer of each response object, as the basic number of times corresponding to each answering answer.

The determination module 54 is configured to determine the reliability value corresponding to each response object according to the basic times corresponding to each response answer.

The selection module 55 is used to select a preset number of response objects in the order of increasing reliability values as sampling objects, and to check the response answers corresponding to the sampling objects.

Further, the parsing module 52 includes:

The word segmentation unit is used to perform word segmentation processing on the response answer by using the dynamic programming algorithm to obtain the initial word segmentation;

The spot check word segmentation determination unit is used to filter the initial word segmentation, and perform synonymous replacement for the filtered initial word segmentation to obtain the spot check word segmentation;

The sampling word frequency determination unit is used to obtain all sampling word segments corresponding to the historical crowdsourcing task for the same historical crowdsourcing task, and count the occurrences of each sampling word segment to obtain the sampling word frequency corresponding to the sampling word segment;

The sampling keyword determination unit is used for sampling the word frequency of the sampling inspection is greater than the preset word frequency as the sampling inspection keyword, and storing the sampling inspection keyword in the preset answer thesaurus.

Further, the sampling and segmentation determination unit includes:

The named entity recognition sub-unit is used to synonymously replace the initial participle by means of named entity recognition to obtain the sample participle.

Further, the sampling and segmentation determination unit also includes:

The standard vocabulary dictionary obtaining subunit is used to obtain the preset standard vocabulary dictionary;

The entity recognition result determination subunit is used to perform named entity recognition on the initial word segmentation and each word in the standard vocabulary dictionary by traversing for each initial word segmentation, and obtain the entity recognition result;

The standard participle replacement subunit is used to obtain the initial participle and standard vocabulary corresponding to the recognition result if the entity recognition result is that the same named entity exists, and use the standard participle to replace the initial participle.

Further, the statistics module 53 includes:

The reference task determination unit is used to obtain each historical crowdsourcing task that the respondent participated in as a reference task;

The basic number determination unit is used for obtaining the sampling keyword from the preset answer thesaurus for each reference task, and counting the number of times that the sampling word corresponding to the response answer of the respondent hits the sampling keyword as the basic number.

Further, the determining module 54 includes:

The basic count statistics unit is used to count the sum M of the sampling keywords corresponding to each reference task according to the basic count, and count the sum N of the hit times of the response object for each reference task, where M and N are both positive integers, and N is less than or equal to M;

The reliability value determination unit is configured to perform calculation by adopting the formula δ=N/M to obtain the reliability value δ.

A sampling inspection device for crowdsourcing tasks in the above scheme, through the obtaining module 51, for each historical crowdsourcing task, obtains the response answer corresponding to the historical crowdsourcing task for each response object; the parsing module 52 parses the response answer to obtain Sampling and segmenting words, extracting sampling keywords from the sampling segmentation, and saving the sampling keywords into the preset answer thesaurus; extracting the more targeted sampling keywords from the complicated sampling segmentation words, which can effectively improve the sampling efficiency. The statistics module 53 counts the number of times that the sampling word hits the sampling keyword for the sampling word segmentation in each response answer of each response object, as the basic number of each response answer; the response object can be converted into a corresponding reliability value, enhancing the The pertinence of the random inspection; the determination module 54 determines the reliability value corresponding to each response object according to the corresponding basic times of each response answer, and then the selection module 55 selects the preset number of responses according to the order of the reliability values from small to large The object, as a sampling object, can make the sampling more targeted and help improve the efficiency of sampling.

To solve the above technical problems, the embodiments of the present application also provide computer equipment. For details, please refer to FIG. 8 , which is a block diagram of a basic structure of a computer device according to this embodiment.

The computer device 6 includes a memory 61 , a processor 62 , and a network interface 63 connected to each other through a system bus. It should be pointed out that the figure only shows the computer device 6 with three components, the memory 61, the processor 62, and the network interface 63, but it should be understood that it is not required to implement all the shown components, and alternative implementations are possible. More or fewer components. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, special-purpose Integrated circuit (Application Specific Integrated Circuit, ASIC), Programmable Gate Array (Field-Programmable Gate Array, FPGA), Digital Signal Processor (Digital Signal Processor, DSP), embedded devices, etc.

The computer equipment may be a desktop computer, a notebook computer, a palmtop computer, and a cloud server and other computing equipment. Computer devices can interact with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory ( SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6 , such as a hard disk or a memory of the computer device 6 . In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD card) equipped on the computer device 6 ) card, flash card (Flash Card) and so on. Of course, the memory 61 may also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6 , such as program codes of the sampling method for crowdsourcing tasks, and the like. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.

The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is typically used to control the overall operation of the computer device 6 . In this embodiment, the processor 62 is configured to run the program code or process data stored in the memory 61, for example, the program code for running a sampling method for crowdsourcing tasks.

The network interface 63 may comprise a wireless network interface or a wired network interface, and the network interface 63 is typically used to establish a communication connection between the computer device 6 and other electronic devices.

The present application also provides another embodiment, which is to provide a computer-readable storage medium, where the computer-readable storage medium stores a sampling inspection program, and the sampling inspection program can be executed by at least one processor, so that the at least one processor executes the above-mentioned The steps of a sampling method for crowdsourcing tasks.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods of the various embodiments of the present application.

Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The accompanying drawings show the preferred embodiments of the present application, but do not limit the scope of the patent of the present application. This application may be embodied in many different forms, rather these embodiments are provided so that a thorough and complete understanding of the disclosure of this application is provided. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features. . Any equivalent structure made by using the contents of the description and drawings of the present application, which is directly or indirectly used in other related technical fields, is also within the scope of protection of the patent of the present application.

Claims (10)

1. a sampling method of crowdsourcing task, is characterized in that, comprises: For each historical crowdsourcing task, obtain each response object participating in the historical crowdsourcing task, and the response answer corresponding to each response object; Analyzing and processing the response answer, obtaining the word-sampling check, and extracting the keyword for spot-checking from the word-sampling check, and storing the keyword for spot-checking into a preset answer thesaurus; For the sampling word segmentation in each response answer of each responding object, count the number of times that the sampling word segmentation hits the sampling keyword in the preset answer thesaurus, as the basis for each answer answer frequency; According to the basic times corresponding to each response answer, determine the reliability value corresponding to each response object; According to the order of the reliability values from small to large, a preset number of response objects are selected as sampling objects, and the response answers corresponding to the sampling objects are checked. 2. the sampling method of crowdsourcing task according to claim 1, is characterized in that, described answering answer is analyzed and processed, obtains the word of sampling inspection and comprises: Using a dynamic programming algorithm, word segmentation is performed on the response answer to obtain an initial word segmentation; Filter processing is performed on the initial word segmentation, and synonymous replacement is performed on the initial word segmentation after the filtering processing to obtain the sampling word segmentation. 3. the sampling method of crowdsourcing task according to claim 1, is characterized in that, described extracting sampling key word from described sampling inspection participle, described sampling inspection key word is stored in the preset answer thesaurus comprising: For the same historical crowdsourcing task, obtain all the sampling word segments corresponding to the historical crowdsourcing task, and count the number of occurrences of each of the sampling word segmentations, to obtain the sampling word frequency corresponding to the sampling word segmentation; The sampling word frequency of the sampling inspection word frequency is greater than the preset word frequency, as the sampling inspection keyword, and the sampling inspection keyword is stored in the preset answer thesaurus. 4. the sampling method of crowdsourcing task according to claim 2, is characterized in that, described initial participle after filtering processing is carried out synonymous replacement, obtain described sampling participle comprises: By means of named entity recognition, synonymous replacement is performed on the initial participle to obtain the sampling participle. 5. the sampling method of crowdsourcing task according to claim 4, is characterized in that, described by the mode of named entity recognition, described initial participle is carried out synonymous replacement and comprises: Get the preset standard vocabulary dictionary; For each of the initial participles, by means of traversal, perform named entity recognition on the initial participles and each vocabulary in the standard vocabulary dictionary, respectively, to obtain an entity recognition result; If the entity recognition result is that the same named entity exists, the initial participle and standard vocabulary corresponding to the recognition result are acquired, and the standard participle is used to replace the initial participle. 6. according to the random inspection method of the crowdsourcing task described in any one of claim 1 to 5, it is characterized in that, described for the described spot inspection participle in each answer answer of described each response object, statistical preset answer In the thesaurus, the number of times that the sampling word segment hits the sampling keyword, as the basic number of times corresponding to each response answer includes: Obtain each historical crowdsourcing task that the respondent participates in, as a reference task; For each of the reference tasks, the sampling keyword is obtained from the preset answer thesaurus, and the number of times that the sampling word corresponding to the response answer of the answering object hits the sampling keyword is counted as the basic number of times. 7. according to the sampling method of the crowdsourcing task described in any one of claim 1 to 5, it is characterized in that, described according to the basic times corresponding to each response answer, determine that the reliability value corresponding to each response object comprises: According to the basic number of times, count the sum M of the sampling keywords corresponding to each of the reference tasks, and count the sum N of the hit times of the response object for each of the reference tasks, where M and N are both positive integers , and N is less than or equal to M; The formula δ=N/M is used for calculation to obtain the reliability value δ. 8. A sampling device for crowdsourcing tasks, comprising: an obtaining module, used for obtaining the response answer of each response object corresponding to the historical crowdsourcing task for each historical crowdsourcing task; The parsing module is used to parse and process the response answer, obtain the word-sampling check, and extract the keyword of the spot-checking of the word-sampling check, and store the keyword of the spot-checking into a preset answer thesaurus; The statistical module is used to count the number of times that the sampling word hits the sampling keyword in the preset answer thesaurus for the sampling word in each response answer of the each response object, as each The basic times corresponding to each answer; A determination module, used for determining the reliability value corresponding to each response object according to the basic times corresponding to each response answer; The selection module is configured to select a preset number of response objects in the order of the reliability values from small to large as sampling objects, and to check the response answers corresponding to the sampling objects. 9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the crowdsourcing task according to any one of claims 1 to 7 is realized. The steps of the sampling method. 10. A computer-readable storage medium, characterized in that, a computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, any one of claims 1 to 7 is implemented. The steps of the sampling method for the package task.

Images