CN115729922A - High-speed data cleaning method based on plug-in hot update technology - Google Patents

Abstract

The invention relates to a high-speed data cleaning method based on a plug-in hot update technology, and belongs to the technical field of computers. Judging whether the access queue equipment has data of a new source needing to be input and integrated; checking whether the existing data cleaning plug-in has a data format capable of processing the data of the new source; converting the data format of the new source data into a standard target data format, compiling and packaging the conversion process into a plug-in, and issuing the plug-in to a server; configuring the transmitted plug-in on the plug-in deployment server as a data cleaning plug-in corresponding to the new source data format; downloading a newly added plug-in from a plug-in deployment server by a data cleaning processing equipment node, and carrying out hot updating on the plug-in through dynamic loading; carrying out data cleaning processing on the data of the new source by adopting a corresponding plug-in; and outputting the standard data after data cleaning. The invention can realize quick access and cleaning treatment, output standard data content and is easy to popularize and apply.

Description

A high-speed data cleaning method based on plug-in hot update technology

technical field

The invention belongs to the field of computer technology, and specifically relates to a high-speed data cleaning method based on plug-in hot update technology, which is specifically applied to the standardization processing of data access from different sources and different formats in the electric power market.

Background technique

In May 2022, due to a document clearly proposing to improve the ability of the power grid to accept distributed new energy, it is necessary to steadily promote the participation of new energy in electricity market transactions. Therefore, all regions have formulated market-oriented trading rules for distributed power generation to accelerate the development of distributed energy.

As a large number of different types of green electricity join the electricity market to participate in the transaction, the access of a large number of data from different sources and in different formats poses a huge challenge to the data integration of the power trading system. The ideal solution is to formulate standards and standardize the data content and format that are required to be accessed in a unified manner. However, different equipment and systems have different manufacturers, especially a large number of existing equipment, and it is difficult to achieve unified requirements for source-end data. Data cleaning and conversion can only be performed through the development of programs. Traditional processing programs need to avoid downtime through shutdown or dynamic expansion of scripting languages. It is difficult to simultaneously meet the processing requirements of flexible configuration, seamless update, and high-speed processing. Therefore, how to overcome the deficiencies in the prior art is an urgent problem in the field of computer technology.

Contents of the invention

The purpose of the present invention is to provide a high-speed data cleaning method based on plug-in hot update technology in order to solve the deficiencies of the prior art.

To achieve the above object, the technical scheme adopted in the present invention is as follows:

A high-speed data cleaning method based on plug-in hot update technology, comprising the following steps:

Step (1), determine whether the access queue device has data from a new source that needs to be input and integrated; if so, go to step (2);

Step (2), check whether the existing data cleaning plug-in has a data format that can process the data from the new source; if it cannot be processed, then step (3); if it can be processed, then go to step (6);

Step (3), convert the data format of the new source data into a standard target data format, compile and package the conversion process as a plug-in, and publish it to the server;

Step (4), on the plug-in deployment server, configure the plug-in from step (3) as a data cleaning plug-in corresponding to the new source data format;

Step (5), the data cleaning processing device node downloads the new plug-in from the plug-in deployment server, and performs hot update of the plug-in through dynamic loading;

Step (6), using corresponding plug-ins to perform data cleaning processing on the data from the new source;

Step (7), output the standardized data after data cleaning.

Further, preferably, in step (3), according to the characteristics of the data to be cleaned, the data format of the new source data is converted into a standard target data format, and the conversion process is compiled and packaged into a binary plug-in, and released to the server .

Further, preferably, in step (5), the data cleaning processing device node downloads the new plug-in from the plug-in deployment server according to the data format, loads the class in the plug-in to the Java virtual machine JVM through Java ClassLoader class loading, and loads the plug-in The program is registered to the registration system to realize the hot update of the data cleaning plug-in program logic.

Compared with the prior art, the present invention has the beneficial effects of:

One is flexible configuration. Plug-in realizes flexible configuration. For data in different sources and formats from various sources of power equipment data and power systems, it can realize fast access, cleaning processing, and output standard data content;

The second is seamless update, which can update the cleaning processing logic of new types of data at any time without interrupting the operation of other data cleaning programs;

The third is fast running speed, because the plug-in is a binary program that can be directly run. Compared with the interpreted script code scheme, the running and processing speed of the binary plug-in program is faster. This method has certain practical social value.

Description of drawings

Fig. 1 is the system structural diagram of the high-speed data cleaning method based on the plug-in thermal update technology of the present invention;

FIG. 2 is a flowchart of a high-speed data cleaning method based on plug-in hot update technology in an application example of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the examples.

Those skilled in the art will understand that the following examples are only for illustrating the present invention and should not be considered as limiting the scope of the present invention. If no specific technique or condition is indicated in the examples, it shall be carried out according to the technique or condition described in the literature in this field or according to the product specification. The materials or equipment used are not indicated by the manufacturer, and they are all conventional products that can be obtained through purchase.

Example 1

A high-speed data cleaning method based on plug-in hot update technology, comprising the following steps:

Step (1), determine whether the access queue device has data from a new source that needs to be input and integrated; if so, go to step (2);

Step (2), check whether the existing data cleaning plug-in has a data format that can process the data from the new source; if it cannot be processed, then step (3); if it can be processed, then go to step (6);

Step (3), convert the data format of the new source data into a standard target data format, compile and package the conversion process as a plug-in, and publish it to the server;

Step (4), on the plug-in deployment server, configure the plug-in from step (3) as a data cleaning plug-in corresponding to the new source data format;

Step (5), the data cleaning processing device node downloads the new plug-in from the plug-in deployment server, and performs hot update of the plug-in through dynamic loading;

Step (6), using corresponding plug-ins to perform data cleaning processing on the data from the new source;

Step (7), output the standardized data after data cleaning.

Example 2

A high-speed data cleaning method based on plug-in hot update technology, comprising the following steps:

Step (1), determine whether the access queue device has data from a new source that needs to be input and integrated; if so, go to step (2);

Step (2), check whether the existing data cleaning plug-in has a data format that can process the data from the new source; if it cannot be processed, then step (3); if it can be processed, then go to step (6);

Step (3), convert the data format of the new source data into a standard target data format, compile and package the conversion process as a plug-in, and publish it to the server;

Step (4), on the plug-in deployment server, configure the plug-in from step (3) as a data cleaning plug-in corresponding to the new source data format;

Step (5), the data cleaning processing device node downloads the new plug-in from the plug-in deployment server, and performs hot update of the plug-in through dynamic loading;

Step (6), using corresponding plug-ins to perform data cleaning processing on the data from the new source;

Step (7), output the standardized data after data cleaning.

In step (3), according to the characteristics of the data to be cleaned, the data format of the new source data is converted into a standard target data format, and the conversion process is compiled and packaged into a binary plug-in and released to the server.

In step (5), the data cleaning processing device node downloads the new plug-in from the plug-in deployment server according to the data format, loads the class in the plug-in to the Java virtual machine JVM through Java ClassLoader class loading, and registers the plug-in program to the registration system. Realize hot update of data cleaning plug-in program logic.

Applications

1. System structure diagram of the present invention

As shown in Figure 1, the system structure includes a plug-in deployment server, a data cleaning processing device node, an access queue device, and an output queue device.

2. The following will be further described in detail in conjunction with Figure 2, specifically including the following steps:

(1) The access queue device has data from a new source that needs to be input and integrated, and configure the code for the data source, such as C000N;

(2) Check whether the existing data cleaning plug-in can handle the data source format coded as C000N; if it cannot be processed, go to step (3); if it can, go to step (6);

(3) Convert the data format of the new source data into a standard target data format, compile and package the conversion process into a plug-in, and publish it to the server;

Specifically, write a cleaning and conversion program for new source data, and convert the newly added data format into a standard target data format according to the characteristics of the data to be cleaned by implementing the predefined data cleaning processing interface DataPurifyHandler, such as converting the time format to a unified The format and business type codes are converted into standard codes, etc.; after the created plug-in program is tested and verified to be correct, it is compiled and packaged into a binary plug-in and released to the server;

(4) On the plug-in deployment server, configure the plug-in from step (3) as the data cleaning plug-in corresponding to the data source C000N;

(5) The data cleaning processing device node downloads the new plug-in from the plug-in deployment server, and performs hot update of the plug-in through dynamic loading;

Specifically, the data cleaning processing device node downloads the newly deployed plug-in from the plug-in deployment server according to the data source code C000N, loads the class in the plug-in to the Java virtual machine JVM through Java ClassLoader class loading, and registers the plug-in program to the registration system, Realize hot update of data cleaning plug-in program logic.

(6) The hot-updated plug-in starts to run, and runs data cleaning processing on the newly added input data source C000N;

(7) Output the standardized data after data cleaning.

The basic principles, main features and advantages of the present invention have been shown and described above. Those skilled in the industry should understand that the present invention is not limited by the above-mentioned embodiments. What are described in the above-mentioned embodiments and the description only illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have Variations and improvements are possible, which fall within the scope of the claimed invention. The protection scope of the present invention is defined by the appended claims and their equivalents.

Claims (3)

1. A high-speed data cleaning method based on plug-in hot update technology, characterized in that, comprising the steps: Step (1), determine whether the access queue device has data from a new source that needs to be input and integrated; if so, go to step (2); Step (2), check whether the existing data cleaning plug-in has a data format that can process the data from the new source; if it cannot be processed, then step (3); if it can be processed, then go to step (6); Step (3), convert the data format of the new source data into a standard target data format, compile and package the conversion process as a plug-in, and publish it to the server; Step (4), on the plug-in deployment server, configure the plug-in from step (3) as a data cleaning plug-in corresponding to the new source data format; Step (5), the data cleaning processing device node downloads the new plug-in from the plug-in deployment server, and performs hot update of the plug-in through dynamic loading; Step (6), using corresponding plug-ins to perform data cleaning processing on the data from the new source; Step (7), output the standardized data after data cleaning. 2. The high-speed data cleaning method based on plug-in hot update technology according to claim 1, characterized in that in step (3), according to the characteristics of the data to be cleaned, the data format of the data from the new source is converted into a standard Target data format, compile and package the conversion process into a binary plug-in, and publish it to the server. 3. The high-speed data cleaning method based on plug-in hot update technology according to claim 1, characterized in that in step (5), the data cleaning processing device node downloads the new plug-in from the plug-in deployment server according to the data format, and uses Java The ClassLoader class loads, loads the class in the plug-in to the Java virtual machine JVM, registers the plug-in program to the registration system, and realizes the hot update of the logic of the data cleaning plug-in program.

Images