CN105447141A - Data processing method and node - Google Patents

Abstract

Embodiments of the invention provide a data processing method and node, which relate to the technical field of computers and are used for quickly searching for data to improve the efficiency of mass data search. The method comprises the steps that a master control node determines to-be-searched data information; the master control node generates a search instruction according to the to-be-searched data information, and sends the search instruction to at least one slave node, wherein the at least one slave node stores elastic distribution type data sent by the master control node, and the search instruction carries identifier information of the to-be-searched data information; the master control node obtains the elastic distribution type data corresponding to the to-be-searched data information and returned by the at least one slave node, and determines a response data set corresponding to the to-be-searched data information according to the elastic distribution data corresponding to the to-be-searched data information returned by the at least one slave node; and the main control node outputs the response data set.

Description

A data processing method and node

technical field

The present invention relates to the field of computer technology, in particular to a data processing method and node.

Background technique

With the advent of the electronic information age, people's lives have undergone tremendous changes. Everyone is surrounded by massive amounts of data, whether it is work, study or life, data is everywhere. Massive data in various fields of people's lives, such as medical care, meteorology, life, etc., have been developed into business applications, providing people with a high quality of life. However, with the prevalence of informatization today, people have realized the importance of data. It has powerful energy and contains huge wealth. People can use massive data to make decisions and bring more beneficial effects to enterprises and individuals. However, in the face of massive data, how to quickly search for the data required by users from the stored massive data is an urgent problem to be solved.

Contents of the invention

Embodiments of the present invention provide a data processing method and node, which are used to realize fast data search and improve the efficiency of massive data search.

In order to achieve the above object, embodiments of the present invention adopt the following technical solutions:

An embodiment of the present invention provides a data processing method, which is applied to a data storage cluster, the cluster includes a master control node and at least one slave node, and the method includes: the master control node determines the data information to be searched ; The master control node generates a search instruction according to the data information to be searched, and sends the search instruction to the at least one slave node, so that the at least one slave node is stored according to the search instruction Find the elastic data corresponding to the data information to be searched in the elastic distributed data, and return the elastic data corresponding to the data information to be searched to the master control node; the at least one slave node stores the The elastic distributed data sent by the master control node; the search command carries the identification information of the data information to be searched; the master control node acquires the data to be searched returned by the at least one slave node Elastic distributed data corresponding to the information, and according to the elastic distributed data corresponding to the data information to be searched returned by the at least one slave node, determine the response data set corresponding to the data information to be searched; The node outputs the response data set.

Further, before the master control node determines the data information to be searched, it also includes: the master control node acquires the data to be stored; the master control node divides the data to be stored into at least One elastic distributed data; the master control node sends the at least one elastic distributed data to the at least one slave node.

Further, dividing the data to be stored by the master control node into at least one elastic distributed data according to a preset division rule includes: using the spark.textFile function by the master control node to convert the data to be stored into Stored data is divided into at least one elastic distributed data.

Further, an embodiment of the present invention provides a data processing method, which is applied to a data storage cluster, the cluster includes a master node and at least one slave node, and the method includes: the slave node receives the master According to the search instruction sent by the control node, and according to the identification information of the data information to be searched in the search instruction, search in the memory storing elastic distributed data, and obtain the search information corresponding to the identification information of the data information to be searched elastic distributed data; the search instruction carries the identification information of the data information to be searched; the slave node returns the acquired elastic distributed data corresponding to the identification information of the data information to be searched to the Master control node.

Further, the slave node receives the search command sent by the master control node, and performs a search in the memory storing elastic distributed data according to the identification information of the data information to be searched in the search command, and obtains the search command. Before the elastic distributed data corresponding to the identification information of the data information to be searched, it also includes: the slave node receives the elastic distributed data sent by the master node; the slave node stores the elastic distributed data to memory.

Further, an embodiment of the present invention provides a master control node, including: a determining unit, configured to determine data information to be searched; a processing unit, configured to generate a search instruction according to the data information to be searched, and send the The search instruction is sent to the at least one slave node, so that the at least one slave node finds the elastic data corresponding to the data information to be searched in the stored elastic distributed data according to the search instruction, and sends the The elastic sub-data corresponding to the data information to be searched is returned to the master control node; the at least one slave node stores the elastic distributed data sent by the master control node; the search command carries the The identification information of the searched data information; the processing unit is further configured to acquire the elastic distributed data corresponding to the data information to be searched returned by the at least one slave node, and according to the at least one returned slave node Elastic distributed data corresponding to the data information to be searched, determining a response data set corresponding to the data information to be searched; an output unit configured to output the response data set.

Further, it also includes: an acquisition unit, configured to acquire data to be stored; a division unit, configured to divide the data to be stored acquired by the acquisition unit into at least one elastic distributed data according to a preset division rule; a sending unit , configured to send the at least one elastic distributed data acquired by the division unit to the at least one slave node.

Further, the division unit is specifically configured to divide the data to be stored into at least one piece of elastic distributed data by using the spark.textFile function according to preset division rules.

Further, an embodiment of the present invention provides a slave node, including: a receiving unit, configured to receive the search instruction sent by the master control node; a processing unit, configured to receive the search instruction according to the receiving unit The identification information of the data information to be searched is searched in the memory storing the elastic distributed data, and the elastic distributed data corresponding to the identification information of the data information to be searched is obtained; the search command carries the Identification information of the data information to be searched; a sending unit configured to return elastic distributed data corresponding to the identification information of the data information to be searched acquired by the processing unit to the master control node.

Further, the receiving unit is further configured to receive the elastic distributed data sent by the master control node; the processing unit is further configured to store the elastic distributed data received by the receiving unit in memory.

The embodiment of the present invention provides a data processing method and node, which are applied in a data storage cluster, the cluster includes a master control node and at least one slave node, and the method includes: the master control node determines the data information to be searched; the master control node determines the data information to be searched; The control node generates a search instruction according to the data information to be searched, and sends the search instruction to the at least one slave node, so that at least one slave node finds the data to be searched in the stored elastic distributed data according to the search instruction information corresponding to the elastic sub-data, and return the elastic sub-data corresponding to the data information to be searched to the master control node; the master control node obtains at least one elastic sub-data corresponding to the data information to be searched returned from the From the elastic sub-data corresponding to the data information to be searched returned by the node, determine the response data set corresponding to the data to be searched. The master control node outputs the response data set. In this way, when data search is required, the master control node can generate search instructions based on the data to be searched, and send the search instructions to the slave nodes that store elastic distributed data, so that all slave nodes can obtain the data to be searched in parallel. The elastic distributed data corresponding to the data. That is to say, in the cluster of the present invention, the elastic distributed data stored by multiple slave nodes are searched at the same time, and the elastic distributed data corresponding to the data to be searched is obtained. In turn, fast data search is realized, and the efficiency of massive data search is improved.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a schematic flowchart of a data processing method provided by an embodiment of the present invention;

FIG. 2 is a schematic flowchart of another data processing method provided by an embodiment of the present invention;

FIG. 3 is a schematic flowchart of another data processing method provided by an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a master control node provided by an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of another master control node provided by an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a slave node provided by an embodiment of the present invention.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

An embodiment of the present invention provides a data processing method, which is applied to a data storage cluster, and the cluster includes a master control node and at least one slave node.

It should be noted that, in all embodiments of the present invention, the master control node and at least one slave node in the cluster are composed of servers. And the memory of the server is not less than 96G. In the cluster, the master control node and at least one slave node are installed and deployed according to the environment that Spark depends on, and Shark is installed and deployed on the master control node and at least one slave node. And, install Hadoop components on the master control node and at least one slave node, for example, install HDFS (Hadoop Distributed File System, distributed file system) components.

As shown in Figure 1, the method includes:

Step 101, the master control node determines the data information to be searched.

Specifically, when the user needs to obtain data in the cluster where the data storage is stored, the data information to be searched can be sent to the master control node. That is, tell the master control node what data the user needs to obtain.

It should be noted that the master control node of the cluster is preset.

Exemplarily, if the user needs to search for text data as data data, the user can send the data information to the master control node of the cluster. The master control node can obtain the data information to be searched, which is data information.

Step 102, the master control node generates a search instruction according to the data information to be searched, and sends the search instruction to at least one slave node, so that at least one slave node can find the data to be searched in the stored elastic distributed data according to the search instruction The elastic sub-data corresponding to the information, and return the elastic sub-data corresponding to the data information to be searched to the main control node.

Among them, at least one slave node stores elastic distributed data sent by the master control node. The search instruction carries identification information of the data information to be searched.

Specifically, after acquiring the data information to be searched, the master control node generates a search instruction according to the data information to be searched. The search instruction may carry identification information of the data information to be searched, so that the slave node may know which data needs to be searched according to the identification information of the data information to be searched. After the master control node generates the search command, it sends the search command to the slave nodes in the cluster. Since the elastic distributed data is stored in the slave node, the slave node can find the identity of the data information to be searched in the respectively stored elastic distributed data according to the identification information of the data information to be searched in the search instruction Elastic distributed data matching information. That is, find the elastic sub-data corresponding to the data information to be searched. After at least one slave node in the cluster finds the elastic data corresponding to the data information to be searched, it returns the respectively found elastic data corresponding to the data information to be searched to the master control node.

It should be noted that when the slave nodes in the cluster search for the elastic data corresponding to the data information to be searched, if the elastic distributed data is also stored in the master control node, it also needs to be stored in its own memory according to the data information to be searched. Stored elastically distributed data for lookup.

It should be noted that, in the embodiment of the present invention, the data to be stored needs to be stored in the cluster first, that is, the data to be stored is first divided into multiple elastic distributed data, and stored in each cluster in the form of elastic distributed data. From the node and the master node. In this way, the elastic distributed data stored by the master control node and all slave nodes in the cluster can be regarded as an elastic distributed data set. Furthermore, when the master control node and all the slave nodes store the elastic distributed data, they all store the elastic distributed data in the memory.

As described in the above example, after the master control node obtains the information of the data, it can generate a search command for searching the data according to the information of the data, and the search command carries the identification information of the data. And send the search command for searching data to all slave nodes in the cluster. In this way, after receiving the search command for searching data, the slave node can parse the search command, and then obtain the identification information of the data carried in the large search command. After obtaining the identification information of the data, the slave node can know It is necessary to search for data. At this time, the slave node can search for the data of data in the elastic distributed data stored in its own memory according to the identification information of data, and then obtain the elastic distributed data of data. The slave nodes in the cluster can return the elastic distributed data of the acquired data to the master node.

Step 103, the master control node obtains at least one elastic distributed data corresponding to the data information to be searched returned by the slave node, and determines the data to be searched according to the elastic distributed data corresponding to the data information to be searched returned by at least one slave node The response data set corresponding to the information.

Specifically, after the master control node receives the elastic distributed data corresponding to the data information to be searched returned by at least one slave node in the cluster, the elastic distributed data corresponding to the data information to be searched returned by the at least one slave node can be The formed data set is determined as the response data set corresponding to the data information to be searched.

As mentioned in the above example, after the master control node receives the elastic distributed data of data returned by at least one slave node in the cluster, it can combine all the elastic distributed data of data returned by the slave nodes to form the response data set of data . Further, the master control node can obtain the response data set of data according to the instruction valdata=file.filter(line=>line.contains("data")).

Step 104, the master control node outputs the response data set.

Specifically, after obtaining the response data set corresponding to the data information to be searched, the master control node may output the response data set, so that the user can obtain the data required by the user for subsequent processing.

As described in the above example, after the master control node determines the response data set of data, it can output the response data set of data to the user, so that the user can obtain the data they need for subsequent processing.

An embodiment of the present invention provides a data processing method, which is applied to a data storage cluster. The cluster includes a master control node and at least one slave node. The method includes: the master control node determines the data information to be searched; the master control node According to the data information to be searched, a search instruction is generated, and the search instruction is sent to the at least one slave node, so that at least one slave node finds the corresponding data information to be searched in the stored elastic distributed data according to the search instruction Elastic score data, and return the elastic score data corresponding to the data information to be searched to the master control node; the master control node obtains the elastic score data corresponding to the data information to be searched returned by at least one slave node, and according to at least one slave node The elastic sub-data corresponding to the returned data information to be searched is used to determine the response data set corresponding to the data to be searched. The master control node outputs the response data set. In this way, when data search is required, the master control node can generate search instructions based on the data to be searched, and send the search instructions to the slave nodes that store elastic distributed data, so that all slave nodes can obtain the data to be searched in parallel. The elastic distributed data corresponding to the data. That is to say, in the cluster of the present invention, the elastic distributed data stored by multiple slave nodes are searched at the same time, and the elastic distributed data corresponding to the data to be searched is obtained. In turn, fast data search is realized, and the efficiency of massive data search is improved.

An embodiment of the present invention provides a data processing method, which is applied to a data storage cluster, and the cluster includes a master control node and at least one slave node. As shown in Figure 2, the method includes:

Step 201, the slave node receives the search instruction sent by the master control node, and searches in the memory storing elastic distributed data according to the identification information of the data information to be searched in the search instruction, and obtains the identification of the data information to be searched Elastic distributed data corresponding to information.

Wherein, the search instruction carries identification information of the data information to be searched.

Specifically, after receiving the search instruction sent by the master control node, the slave node can parse the search instruction, and then can obtain the identification information of the data information to be searched carried in the search instruction. The slave node may search in its memory storing elastic distributed data according to the identification information of the data information to be searched, and obtain the elastic distributed data corresponding to the identification information of the data information to be searched. That is, the elastic distributed data corresponding to the data information to be searched has been obtained

Step 202, the slave node returns the acquired elastic distributed data corresponding to the identification information of the data information to be searched to the master control node.

Specifically, after obtaining the elastic distributed data corresponding to the identification information of the data information to be searched, the slave node may return the obtained elastic distributed data corresponding to the identification information of the data information to be searched to the master control node. That is, the elastic distributed data corresponding to the data information to be searched is returned to the master control node.

An embodiment of the present invention provides a data processing method, which is applied to a data storage cluster. The cluster includes a master control node and at least one slave node. The method includes: the slave node receives a search instruction sent by the master control node, and according to The identification information of the data information to be searched in the search command is searched in the memory storing the elastic distributed data, and the elastic distributed data corresponding to the identification information of the searched data information is obtained; The elastic distributed data corresponding to the identification information of the data information is returned to the master control node. In this way, when data search is required, all slave nodes in the cluster can search according to the search instruction after receiving the search instruction, and obtain elastic distributed data corresponding to the data information to be searched. That is to say, in the cluster of the present invention, elastic distributed data corresponding to the data information to be searched is obtained by searching the elastic distributed data stored separately by multiple slave nodes at the same time. In turn, fast data search is realized, and the efficiency of massive data search is improved.

An embodiment of the present invention provides a data processing method, which is applied to a data storage cluster, and the cluster includes a master control node and at least one slave node. As shown in Figure 3, the method includes:

Step 301, the master control node acquires data to be stored.

Specifically, since the data storage cluster is used to store data, when the user needs to store data, the data can be stored in the cluster. At this point, users can send data to the master node of the cluster. At this point, the master control node can obtain the data to be stored.

Of course, when other devices need to store data, they can also be stored in the cluster. At this point, other devices can send the data to be stored to the master control node of the cluster. At this point, the master control node can obtain the data to be stored.

Step 302, the master control node divides the data to be stored into at least one piece of elastic distributed data according to preset division rules.

Specifically, after acquiring the data to be stored, the master control node may divide the acquired data to be stored into at least one piece of elastic distributed data according to preset division rules.

It should be noted that the preset division rule is a preset rule for dividing data into multiple elastic distributed data. For example, the preset division rule may be to divide the data according to a size. At this time, the master control node divides the data to be stored according to the size of a, and divides each data of the size a into elastic distributed data, thereby dividing the data to be stored into at least one elastic distributed data.

Further, the master control node divides the data to be stored into at least one elastic distributed data according to the preset division rules, including: the master control node uses the spark.textFile function to divide the data to be stored into at least one elastic distribution according to the preset division rules format data.

That is to say, the master control node can divide the data to be stored through the spark.textFile function according to the preset division rules, and divide the data to be stored into at least one elastic distributed data.

Step 303, the master control node sends at least one elastic distributed data to at least one slave node. The slave node receives the elastic distributed data sent by the master node.

Specifically, after the master control node divides the data to be stored into at least one elastic distributed data, it needs to distribute the at least one elastic distributed data among the nodes in the cluster. At this point, the master control node can sequentially send elastic distributed data to each node in the cluster. In this way, each slave node in the cluster can respectively receive elastic distributed data sent by the master control node.

It should be noted that the master control node can also store elastic distributed data. At this time, when the master control node sends elastic distributed data to at least one slave node, it can evenly send elastic distributed data to in each node.

It should be noted that the elastic distributed data sent by the master control node to different nodes is different. That is, the master control node sends different elastic distributed data to different nodes.

For example, the master control node divides the data to be stored into elastic distributed data 1, elastic distributed data 2, elastic distributed data 3, and elastic distributed data 4. The cluster includes the master control node and slave node 1, slave node 2 and slave node 3. The master node can send elastic distributed data 1 to slave node 1, elastic distributed data 2 to slave node 2, elastic distributed data 3 to slave node 3, and elastic distributed data 4 to the master node internal storage. In this way, slave node 1 can receive resilient distributed data 1 . Resilient distributed data 2 can be received from node 2. Resilient distributed data 3 can be received from node 3 . The master control node stores elastic distributed data 4 in its own memory.

Step 304, the slave node stores the elastic distributed data in memory.

Specifically, after the slave node receives the elastic distributed data, it needs to store the elastic distributed data. In the cluster for storing data, all the data can be stored in the memory, thereby improving the speed of data processing. Therefore, the slave nodes can store the acquired elastic distributed data in their own memory.

Furthermore, slave nodes can use the cache function to load elastic distributed data into memory for storage.

Further, when each node in the cluster, including the slave node and the master node, stores the elastic distributed data in the memory, the elastic distributed data may be stored in the memory as configuration data.

Step 305, the master control node determines the data information to be searched.

For details, reference may be made to step 101, which will not be repeated here.

Step 306, the master control node generates a search command according to the data information to be searched, and sends the search command to at least one slave node, so that at least one slave node can find the search command in the stored elastic distributed data according to the search command. The elastic data corresponding to the data information to be searched is returned to the main control node. The slave node receives the search command sent by the master control node, and searches in the memory storing the elastic distributed data according to the identification information of the data information to be searched in the search command, and obtains the search command corresponding to the identification information of the data information to be searched. Elastic Distributed Data.

Among them, at least one slave node stores elastic distributed data sent by the master control node. The search instruction carries identification information of the data information to be searched.

Specifically, reference may be made to step 102 and step 201, which will not be repeated here.

Step 307, the slave node returns the obtained elastic distributed data corresponding to the identification information of the data information to be searched to the master control node. The master control node obtains at least one elastic distributed data corresponding to the data information to be searched returned by the slave node, and determines the corresponding Response dataset.

Specifically, reference may be made to step 103 and step 202, which will not be repeated here.

Step 308, the master control node outputs the response data set.

For details, reference may be made to step 104, which will not be repeated here.

An embodiment of the present invention provides a data processing method, which is applied to a data storage cluster. The cluster includes a master control node and at least one slave node. The method includes: the master control node determines the data information to be searched; the master control node According to the data information to be searched, a search instruction is generated, and the search instruction is sent to the at least one slave node, so that at least one slave node finds the corresponding data information to be searched in the stored elastic distributed data according to the search instruction Elastic score data, and return the elastic score data corresponding to the data information to be searched to the master control node; the master control node obtains the elastic score data corresponding to the data information to be searched returned by at least one slave node, and according to at least one slave node The elastic sub-data corresponding to the returned data information to be searched is used to determine the response data set corresponding to the data to be searched. The master control node outputs the response data set. In this way, when data search is required, the master control node can generate search instructions based on the data to be searched, and send the search instructions to the slave nodes that store elastic distributed data, so that all slave nodes can obtain the data to be searched in parallel. The elastic distributed data corresponding to the data. That is to say, in the cluster of the present invention, the elastic distributed data stored by multiple slave nodes are searched at the same time, and the elastic distributed data corresponding to the data to be searched is obtained. In turn, fast data search is realized, and the efficiency of massive data search is improved.

An embodiment of the present invention provides a master control node, as shown in FIG. 4 , including:

A determining unit 401, configured to determine data information to be searched.

The processing unit 402 is configured to generate a search instruction according to the data information to be searched, and send the search instruction to the at least one slave node, so that at least one slave node can find the search instruction in the stored elastic distributed data according to the search instruction. The elastic data corresponding to the data information to be searched is returned to the main control node.

Among them, at least one slave node stores elastic distributed data sent by the master control node. The search instruction carries identification information of the data information to be searched.

The processing unit 402 is further configured to acquire at least one elastic distributed data corresponding to the data information to be searched returned from the node, and determine the elastic distributed data to be searched according to at least one elastic distributed data corresponding to the data information to be searched returned from the node The response data set corresponding to the data information.

The output unit 403 is configured to output the response data set.

Further, the above-mentioned master control node, as shown in Figure 5, also includes:

The acquiring unit 404 is configured to acquire data to be stored.

The division unit 405 is configured to divide the data to be stored acquired by the acquisition unit 404 into at least one elastic distributed data according to a preset division rule.

Specifically, the division unit 405 is specifically configured to divide the data to be stored into at least one piece of elastic distributed data by using the spark.textFile function according to preset division rules.

The sending unit 406 is configured to send at least one elastic distributed data acquired by the dividing unit 405 to at least one slave node.

An embodiment of the present invention provides a master control node, including: the master control node determines the data information to be searched; the master control node generates a search instruction according to the data information to be searched, and sends the search instruction to the at least one The slave node is so that at least one slave node finds the elastic data corresponding to the data information to be searched in the stored elastic distributed data according to the search instruction, and returns the elastic data corresponding to the data information to be searched to the master control node; The master control node obtains at least one elastic data corresponding to the data information to be searched returned by the slave node, and determines the response corresponding to the data to be searched according to the elastic data corresponding to the data information to be searched returned by at least one slave node data set. The master control node outputs the response data set. In this way, when data search is required, the master control node can generate search instructions based on the data to be searched, and send the search instructions to the slave nodes that store elastic distributed data, so that all slave nodes can obtain the data to be searched in parallel. The elastic distributed data corresponding to the data. That is to say, in the cluster of the present invention, the elastic distributed data stored by multiple slave nodes are searched at the same time, and the elastic distributed data corresponding to the data to be searched is obtained. In turn, fast data search is realized, and the efficiency of massive data search is improved.

An embodiment of the present invention provides a slave node, as shown in FIG. 6, including:

The receiving unit 601 is configured to receive a search instruction sent by the master control node.

The processing unit 602 is configured to search in the memory storing the elastic distributed data according to the identification information of the data information to be searched in the search instruction received by the receiving unit 601, and obtain the elasticity corresponding to the identification information of the data information to be searched. distributed data.

Wherein, the search instruction carries identification information of the data information to be searched.

The sending unit 603 is configured to return the elastic distributed data corresponding to the identification information of the data information to be searched acquired by the processing unit 602 to the master control node.

Further, the above-mentioned receiving unit 601 is also configured to receive elastic distributed data sent by the master control node.

The processing unit 602 is further configured to store the elastic distributed data received by the receiving unit 601 in memory.

An embodiment of the present invention provides a slave node, including: the slave node receives a search command sent by the master control node, and performs a search in the memory storing elastic distributed data according to the identification information of the data information to be searched in the search command The elastic distributed data corresponding to the identification information of the data information to be searched is acquired; the slave node returns the acquired elastic distributed data corresponding to the identification information of the data information to be searched to the master control node. In this way, when data search is required, all slave nodes in the cluster can search according to the search instruction after receiving the search instruction, and obtain elastic distributed data corresponding to the data information to be searched. That is to say, in the cluster of the present invention, elastic distributed data corresponding to the data information to be searched is obtained by searching the elastic distributed data stored separately by multiple slave nodes at the same time. In turn, fast data search is realized, and the efficiency of massive data search is improved.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (10)

1. A method for data processing, characterized in that it is applied in a cluster of data storage, the cluster includes a master control node and at least one slave node, and the method comprises: The master control node determines the data information to be searched; The master control node generates a search instruction according to the data information to be searched, and sends the search instruction to the at least one slave node, so that the at least one slave node is stored according to the elasticity of the search instruction. find the elastic sub-data corresponding to the data information to be searched in the distributed data, and return the elastic sub-data corresponding to the data information to be searched to the master control node; the at least one slave node stores the The elastic distributed data sent by the master control node; the search command carries the identification information of the data information to be searched; The master control node acquires elastic distributed data corresponding to the data information to be searched returned by the at least one slave node, and according to the elastic distributed data corresponding to the data information to be searched returned by the at least one slave node data, determining a response data set corresponding to the data information to be searched; The master control node outputs the response data set. 2. The method according to claim 1, wherein, before the master control node determines the data information to be searched, further comprising: The master control node obtains the data to be stored; The master control node divides the data to be stored into at least one elastic distributed data according to preset division rules; The master control node sends the at least one elastic distributed data to the at least one slave node. 3. The method according to claim 2, wherein the master control node divides the data to be stored into at least one elastic distributed data according to preset division rules comprising: The master control node divides the data to be stored into at least one piece of elastic distributed data by using a spark.textFile function according to a preset division rule. 4. A method for data processing, characterized in that it is applied in a cluster of data storage, the cluster includes a master node and at least one slave node, and the method comprises: The slave node receives the search instruction sent by the master control node, and performs a search in the memory storing the elastic distributed data according to the identification information of the data information to be searched in the search instruction, and obtains and searches the search instruction. The elastic distributed data corresponding to the identification information of the data information; the identification information of the data information to be searched is carried in the search instruction; The slave node returns the acquired elastic distributed data corresponding to the identification information of the data information to be searched to the master control node. 5. The method according to claim 4, wherein the slave node receives the search instruction sent by the master control node, and stores the search instruction according to the identification information of the data information to be searched in the search instruction. Searching in the memory of the elastic distributed data, before obtaining the elastic distributed data corresponding to the identification information of the data information to be searched, further includes: The slave node receives the elastic distributed data sent by the master control node; The slave node stores the elastic distributed data in memory. 6. A master control node, characterized in that, comprising: a determining unit, configured to determine the data information to be searched; a processing unit, configured to generate a search instruction according to the data information to be searched, and send the search instruction to the at least one slave node, so that the at least one slave node can store elastic find the elastic sub-data corresponding to the data information to be searched in the distributed data, and return the elastic sub-data corresponding to the data information to be searched to the master control node; the at least one slave node stores the The elastic distributed data sent by the master control node; the search command carries the identification information of the data information to be searched; The processing unit is further configured to acquire the elastic distributed data corresponding to the data information to be searched returned by the at least one slave node, and according to the data information corresponding to the data information to be searched returned by the at least one slave node Elastic distributed data, determining a response data set corresponding to the data information to be searched; an output unit, configured to output the response data set. 7. The master control node according to claim 6, further comprising: an acquisition unit, configured to acquire data to be stored; A division unit, configured to divide the data to be stored acquired by the acquisition unit into at least one piece of elastic distributed data according to a preset division rule; A sending unit, configured to send the at least one elastic distributed data acquired by the dividing unit to the at least one slave node. 8. The master control node according to claim 7, characterized in that, The division unit is specifically configured to divide the data to be stored into at least one piece of elastic distributed data by using a spark.textFile function according to a preset division rule. 9. A slave node, characterized in that, comprising: a receiving unit, configured to receive a search instruction sent by the master control node; A processing unit, configured to search in the memory storing elastic distributed data according to the identification information of the data information to be searched in the search instruction received by the receiving unit, and obtain an identification for searching the data information to be searched The elastic distributed data corresponding to the information; the search instruction carries the identification information of the data information to be searched; A sending unit, configured to return the elastic distributed data corresponding to the identification information of the data information to be searched acquired by the processing unit to the master control node. 10. The slave node according to claim 9, characterized in that, The receiving unit is further configured to receive elastic distributed data sent by the master control node; The processing unit is further configured to store the elastic distributed data received by the receiving unit in a memory.