CN115729998A - Large-scale processing and analyzing system for arbitrary data hybrid optimization - Google Patents

Abstract

The invention discloses a large-scale processing and analyzing system for arbitrary data mixing optimization, and relates to the field of data large-scale processing and analyzing systems. The method comprises a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module and a data interpretation analysis module, wherein the parameter field of a source file is determined to be in a first data format and the parameter field of a target file is in a second data format by setting the corresponding relation between the parameter field of the source file and the parameter field of the target file, a format conversion protocol corresponding to each first data format is generated according to the similarity between the parameter field of the source file and the parameter field of the target file, each format conversion protocol is issued to the corresponding data conversion module, the data of the parameter field of the source file is matched to the parameter field corresponding to the target file according to the format conversion protocol, the data integration difficulty of the system facing multi-source heterogeneous mass data is reduced, and the analysis efficiency is improved.

Description

A Large-Scale Processing and Analysis System for Arbitrary Data Mixing Optimization

technical field

The invention relates to the field of large-scale data processing and analysis systems, in particular to a large-scale processing and analysis system for arbitrary data mixing optimization.

Background technique

Data is the result of facts or observations, a logical induction of objective things, and unprocessed raw materials used to represent objective things. Data can be continuous values, such as sounds and images, which are called analog data; they can also be discrete, such as symbols and text, which are called digital data; in computer systems, data are represented in the form of binary information units 0 and 1.

In the "multi-source mass data processing system and method" disclosed in the publication number "CN108427709B", the system includes a calculation module and a task management module, wherein: the calculation module is used to receive multi-source mass data and retrieve data The receiving service parses the multi-source massive data; the multi-source massive data is open-source data generated according to tasks pre-configured in the task management module; if the calculation module receives the confirmation of the preset model selected by the outside An action of inputting the analyzed multi-source mass data into the preset model for analyzing the output result of the preset model. The method uses the system. The multi-source mass data processing system and method provided by the embodiments of the present invention analyze the multi-source mass data by calling the data receiving service, input the parsed multi-source mass data into the preset model, and analyze the multi-source mass data according to the output result of the preset model. The source of massive data not only makes the enterprise system efficiently compatible with multi-source massive data, but also can effectively use the multi-source massive data for data analysis.

With the development of big data technology, data sources are becoming more and more extensive. Existing processing and analysis systems often have difficulties in data integration and low analysis efficiency when faced with multi-source heterogeneous massive data; and different business scenarios require Coding based on the business requires a series of operations such as requirements review, design, development, launch, and deployment when there is a slight change in the business, which is inefficient and cumbersome.

Contents of the invention

The purpose of the present invention is to provide a large-scale processing and analysis system for arbitrary data mixing and optimization, which solves the problem of data integration difficulties and low analysis efficiency in existing processing and analysis systems when faced with multi-source heterogeneous massive data; and different The business scenario requires coding based on the business. When the business changes slightly, a series of operations such as requirements review, design, development, launch, and deployment need to be carried out, which is inefficient and cumbersome:

The present invention is a large-scale processing and analysis system for arbitrary data mixing and optimization, including a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module and a data interpretation and analysis module;

The data acquisition module is used to execute acquisition tasks in parallel in a distributed manner by utilizing the computing power of the big data platform;

The data conversion module is used to convert massive heterogeneous data into isomorphic data and transmit it to the data processing module;

The data processing module is used to analyze the data packet, process the result according to the parameter information configured by the user, and send it to the data management module;

The data management module is used to receive and store the original data sent by the data processing module, establish a storage structure according to the test items, and complete the real-time data storage; receive the data processing results released by the data processing module through the network in real time, and display them in real time;

The learning module is used to integrate existing algorithm models to form an algorithm model database, and continuously optimize the accuracy of the model based on massive sample data training;

Among them, the bottom layer of the algorithm model database is integrated with open source technology components, such as Impala, YARN, Spark, Hbase, HDFS, Hive, Kafka, Flink, ElasticSearch, ZooKeeper, etc. For different application fields, the functional components are extended in the form of plug-ins to quickly respond to specific computing needs;

And it supports PB-level complex query and analysis, and the single-cluster deployment scale exceeds 1,000 nodes. It also provides efficient conversion of data files in various formats and analysis service loading of custom formats, supports separate management of data and applications, and seamless data translation of applications. Provide an algorithm model warehouse, and continuously optimize the model accuracy rate based on massive sample data training.

The data interpretation and analysis module is used to automatically complete the interpretation of telemetry parameters according to different test states and test procedures;

A transmission sub-module capable of transmitting data to each other is set between the data acquisition module and the data interpretation and analysis module, and the data acquisition module includes a field sub-module, the data acquisition module collects source file data, and analyzes the parameters of the source file field, to extract the data of each parameter field; the field submodule is used to set the corresponding relationship between the parameter field of the source file and the parameter field of the target file, and determine that the parameter field of the source file is the first data format, and the parameter field of the target file For the second data format, the field submodule generates a format conversion protocol corresponding to each first data format according to the similarity between the parameter field of the source file and the parameter field of the target file, and sends each format conversion protocol To the corresponding data conversion module, according to the format conversion protocol, match the data of the parameter field of the source file to the parameter field corresponding to the target file, and match the data of the parameter field of the source file to the corresponding parameter field of the target file The step of the parameter field includes calculating and judging the data of one or more parameter fields of the source file, generating data matched to the corresponding parameter field of the target file, and integrating the data into a data package;

By setting the data acquisition module, data conversion module, data processing module, learning module, data management module and data interpretation and analysis module, and setting the field sub-module in the data acquisition module, by setting the parameter field of the source file and the parameter field of the target file The corresponding relationship, determine the parameter field of the source file is the first data format, the parameter field of the target file is the second data format, generate each first data according to the similarity between the parameter field of the source file and the parameter field of the target file The format conversion protocol corresponding to the format, and each format conversion protocol is sent to the corresponding data conversion module, according to the format conversion protocol, the data in the parameter field of the source file is matched to the parameter field corresponding to the target file, and the The step of matching the data of the parameter field of the source file to the corresponding parameter field of the target file includes calculating and judging the data of one or more parameter fields of the source file, and generating a corresponding Data in the parameter field, and integrate the data into data packets, reducing the difficulty of data integration when the system faces multi-source heterogeneous massive data, and improving analysis efficiency;

And, when the business changes slightly, since the conversion of data is based on the similarity between the parameter fields of the source file and the parameter fields of the target file to generate a format conversion protocol corresponding to each first data format, the similarity between the data The changes are small, so a series of operations such as requirements review, design, development, launch, and deployment are also small, with high efficiency and convenient processes, greatly reducing time-consuming, and making users more flexible and convenient to use.

Among them, the working steps of the system are as follows:

S1: the data acquisition module collects the source file data, and analyzes the parameter fields of the source file, and extracts the data of each parameter field; the field sub-module is used to set the corresponding relationship between the parameter fields of the source file and the parameter fields of the target file , determining that the parameter field of the source file is the first data format, and the parameter field of the target file is the second data format;

S2: Generate a format conversion protocol corresponding to each first data format according to the similarity between the parameter field of the source file and the parameter field of the target file, and deliver each format conversion protocol to the corresponding data conversion module;

S3: According to the format conversion protocol, the step of matching the data of the parameter field of the source file to the parameter field corresponding to the target file, and matching the data of the parameter field of the source file to the parameter field corresponding to the target file After calculating and judging the data of one or more parameter fields of the source file, generating data matched to the parameter field corresponding to the target file, and integrating the data into a data package;

S4: After receiving the data, the data processing module completes the data processing according to the configured parameter processing information, and forwards the processing results to the data management software;

S5: The data management module establishes a storage structure according to the test items, and stores the data after receiving the processing results sent by the data processing and publishing software in real time; the user can monitor the data in the test process in real time through the real-time monitoring module of the data management software ;

S6: After the test is completed, the data interpretation and analysis module reads the data files stored in the hard disk by the data management module, and invokes the criteria to automatically interpret the data and generate a report; the software can also call the previously stored test data to analyze the data A horizontal comparison of different experiments was carried out.

Among them, the data interaction of the system involves the whole process from data generation to data calculation, distribution, data storage and call. The entire process of data transmission involves the uploading of data with various business units and superiors. According to the different requirements for data at all levels, the connection before the system and data transmission should be carried out. Various data interfaces are reserved for the entire data. necessary part of the life cycle.

Preferably, the data acquisition module further includes a microprocessor unit, a sending interface control submodule and a receiving interface control submodule; the microprocessor unit is used to implement interface control according to usage requirements; the sending interface control submodule is used to realize data packet sending based on bus protocol; the receiving interface control submodule is used to realize data packet reception based on bus protocol.

Preferably, the data processing module includes a bus data processing sub-module, an auxiliary interpretation sub-module and a real-time data processing sub-module; the bus data processing sub-module is used to implement data analysis based on the bus protocol; the auxiliary interpretation sub-module uses Interpret and analyze the data in the auxiliary data interpretation and analysis module; the real-time data processing sub-module is used to analyze the real-time data.

Preferably, the data management module includes a database management submodule, a real-time detection parameter configuration submodule, a data storage and publishing submodule, and a database configuration import submodule; a real-time database is established in the data management module, and the real-time database is used for The real-time data delivered by the storage data processing module; the real-time detection parameter configuration sub-module is used for binding and importing parameter information, and correctness verification of the parameter information; the data storage and publishing sub-module is used for publishing data processing through the network The results are displayed in real time; the database configuration import sub-module is used for migrating and backing up previous test data, and supports the import of unit test data in the same format.

Preferably, the data interpretation and analysis module includes an automatic data interpretation sub-module and a data comparison analysis sub-module; the automatic data interpretation sub-module is used to automatically complete the interpretation of telemetry parameters according to different test states and test procedures; The data comparison and analysis sub-module is used for horizontal comparison of different test data, storage of criteria under different tasks and different states, and creation, editing, deletion and copying of criteria.

Preferably, the data collection module includes an automatic collection sub-module for automatically receiving the data of the source file transmitted from outside.

Preferably, the data interpretation and analysis module further includes a user sub-module and a report sub-module; the user sub-module is used for hierarchical data and criterion hierarchical management of users, roles and permissions; the report sub-module is used for interpretation result reporting Automatically generated, signed and confirmed based on the network.

Preferably, the data processing module can perform dynamic compilation, packaging and dynamic scheduling to achieve hot deployment capabilities;

Among them, a large amount of R&D resources and tedious development process are saved through flexible configuration of data processing logic and data processing flow and hot deployment.

The present invention has the following beneficial effects:

1. The present invention sets the data acquisition module, data conversion module, data processing module, learning module, data management module and data interpretation and analysis module, and sets the field sub-module in the data acquisition module, by setting the parameter field and target of the source file The corresponding relationship of the parameter field of the file, determine the parameter field of the source file is the first data format, the parameter field of the target file is the second data format, generate each parameter field according to the similarity between the parameter field of the source file and the target file A format conversion protocol corresponding to the first data format, and each format conversion protocol is sent to the corresponding data conversion module, and according to the format conversion protocol, the data in the parameter field of the source file is matched to the parameter corresponding to the target file field, the step of matching the data of the parameter field of the source file to the corresponding parameter field of the target file includes calculating and judging the data of one or more parameter fields of the source file, and generating the data matched to the The data in the parameter field corresponding to the target file is integrated into a data package, which reduces the difficulty of data integration when the system faces multi-source heterogeneous massive data and improves analysis efficiency.

2. The present invention generates a format conversion protocol corresponding to each first data format according to the similarity between the parameter field of the source file and the parameter field of the target file when the business changes slightly, and the data conversion The similarity changes are small, so a series of operations such as requirements review, design, development, launch, and deployment also change little, with high efficiency, more convenient process, greatly reduced time-consuming, and more flexible use of users ,convenient.

Of course, any product implementing the present invention does not necessarily need to achieve all the above-mentioned advantages at the same time.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that are required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. Those of ordinary skill in the art can also obtain other drawings based on these drawings without any creative effort.

Fig. 1 is a system block diagram of a large-scale processing analysis system optimized for arbitrary data mixing in the present invention;

Fig. 2 is a work flow diagram of a large-scale processing and analysis system optimized for arbitrary data mixing in the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In the description of the present invention, it should be understood that the terms "upper", "middle", "outer", "inner" and the like indicate orientation or positional relationship, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or It should not be construed as limiting the invention by implying that a referenced component or element must have a particular orientation, be constructed and operate in a particular orientation.

As shown in Figure 1-2, this embodiment implements a large-scale processing and analysis system for arbitrary data mixing and optimization, including a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module, and a data interpretation and analysis module;

The data acquisition module is used to execute acquisition tasks in parallel in a distributed manner by utilizing the computing power of the big data platform;

The data conversion module is used to convert massive heterogeneous data into isomorphic data and transmit it to the data processing module;

The data processing module is used to analyze the data packet, process the result according to the parameter information configured by the user, and send it to the data management module;

The data management module is used to receive and store the original data sent by the data processing module, establish a storage structure according to the test items, and complete the real-time data storage; receive the data processing results released by the data processing module through the network in real time, and display them in real time;

The learning module is used to integrate the existing algorithm models to form an algorithm model database, and continuously optimize the accuracy of the model based on massive sample data training;

Among them, the bottom layer of the algorithm model database is integrated with open source technology components, such as Impala, YARN, Spark, Hbase, HDFS, Hive, Kafka, Flink, ElasticSearch, ZooKeeper, etc. For different application fields, the functional components are extended in the form of plug-ins to quickly respond to specific computing needs;

And it supports PB-level complex query and analysis, and the single-cluster deployment scale exceeds 1,000 nodes. It also provides efficient conversion of data files in various formats and analysis service loading of custom formats, supports separate management of data and applications, and seamless data translation of applications. Provide an algorithm model warehouse, and continuously optimize the model accuracy rate based on massive sample data training.

The data interpretation and analysis module is used to automatically complete the interpretation of telemetry parameters according to different test states and test procedures;

A transmission sub-module capable of transmitting data to each other is set between the data acquisition module and the data interpretation and analysis module, and the data acquisition module includes a field sub-module, the data acquisition module collects the source file data, analyzes the parameter field of the source file, and extracts each parameter The data of the field; the field sub-module is used to set the corresponding relationship between the parameter field of the source file and the parameter field of the target file, determine that the parameter field of the source file is the first data format, and the parameter field of the target file is the second data format, according to the source file The similarity between the parameter field of the file and the parameter field of the target file generates a format conversion protocol corresponding to each first data format, and sends each format conversion protocol to the corresponding data conversion module, and the field sub-module is based on the format The conversion protocol is to match the data of the parameter field of the source file to the corresponding parameter field of the target file, and the step of matching the data of the parameter field of the source file to the corresponding parameter field of the target file includes converting the parameter field of one or more source files to After the data is calculated and judged, the data matching the parameter field corresponding to the target file is generated, and the data is integrated into a data package;

By setting the data acquisition module, data conversion module, data processing module, learning module, data management module and data interpretation and analysis module, and setting the field sub-module in the data acquisition module, by setting the parameter field of the source file and the parameter field of the target file The corresponding relationship, determine the parameter field of the source file is the first data format, the parameter field of the target file is the second data format, generate each first data according to the similarity between the parameter field of the source file and the parameter field of the target file The format conversion protocol corresponding to the format, and each format conversion protocol is sent to the corresponding data conversion module. According to the format conversion protocol, the data in the parameter field of the source file is matched to the parameter field corresponding to the target file, and the parameter field of the source file is The step of matching the data of the target file to the parameter field corresponding to the target file includes calculating and judging the data of the parameter field of one or more source files, generating the data matching the parameter field corresponding to the target file, and integrating the data into data package, which reduces the difficulty of data integration when the system is faced with multi-source heterogeneous massive data, and improves analysis efficiency;

And, when the business changes slightly, since the conversion of data is based on the similarity between the parameter fields of the source file and the parameter fields of the target file to generate a format conversion protocol corresponding to each first data format, the similarity between the data The changes are small, so a series of operations such as requirements review, design, development, launch, and deployment are also small, with high efficiency and convenient processes, greatly reducing time-consuming, and making users more flexible and convenient to use.

Among them, the working steps of the system are as follows:

S1: The data acquisition module collects the source file data, analyzes the parameter fields of the source file, and extracts the data of each parameter field; the field sub-module is used to set the corresponding relationship between the parameter fields of the source file and the parameter fields of the target file, and determine the source file The parameter field of the target file is the first data format, and the parameter field of the target file is the second data format;

S2: Generate a format conversion protocol corresponding to each first data format according to the similarity between the parameter field of the source file and the parameter field of the target file, and deliver each format conversion protocol to the corresponding data conversion module;

S3: According to the format conversion protocol, match the data of the parameter field of the source file to the corresponding parameter field of the target file, and the step of matching the data of the parameter field of the source file to the corresponding parameter field of the target file includes converting one or more source files to After calculating and judging the data in the parameter field of the target file, generate data that matches the parameter field corresponding to the target file, and integrate the data into a data package;

S4: After receiving the data, the data processing module completes the data processing according to the configured parameter processing information, and forwards the processing results to the data management software;

S5: The data management module establishes a storage structure according to the test items, and stores the data after receiving the processing results sent by the data processing and publishing software in real time; the user can monitor the data in the test process in real time through the real-time monitoring module of the data management software ;

S6: After the test is completed, the data interpretation and analysis module reads the data files stored in the hard disk by the data management module, and invokes the criteria to automatically interpret the data and generate a report; the software can also call the previously stored test data to analyze the data A horizontal comparison of different experiments was carried out.

Among them, the data interaction of the system involves the whole process from data generation to data calculation, distribution, data storage and call. The entire process of data transmission involves the uploading of data with various business units and superiors. According to the different requirements for data at all levels, the connection before the system and data transmission should be carried out. Various data interfaces are reserved for the entire data. necessary part of the life cycle.

The data acquisition module includes a microprocessor unit, a sending interface control submodule and a receiving interface control submodule; the microprocessor unit is used to realize interface control according to usage requirements; the sending interface control submodule is used to realize the data packet based on the bus protocol The sending and receiving interface control submodule is used to realize the data packet reception based on the bus protocol.

The data processing module includes a bus data processing sub-module, an auxiliary interpretation sub-module and a real-time data processing sub-module; the bus data processing sub-module is used to implement data analysis based on the bus protocol; the auxiliary interpretation sub-module is used to assist the data interpretation and analysis module to perform data analysis. Interpretation and analysis; the real-time data processing sub-module is used for real-time data analysis.

The data management module includes a database management sub-module, a real-time detection parameter configuration sub-module, a data storage and release sub-module, and a database configuration import sub-module; a real-time database is established in the data management module, and the real-time database is used to store real-time data delivered by the data processing module ; The real-time detection parameter configuration sub-module is used for the binding and import of parameter information, and the correctness of the parameter information is verified; the data storage and release sub-module is used for publishing the data processing results through the network and displaying them in real time; the database configuration import sub-module is used for Migrate and back up previous test data, and support the import of unit test data in the same format.

The data interpretation and analysis module includes the data automatic interpretation sub-module and the data comparison and analysis sub-module; the data automatic interpretation sub-module is used to automatically complete the interpretation of telemetry parameters according to different test states and test procedures; the data comparison and analysis sub-module is used for Horizontal comparison of different test data, storage of criteria under different tasks and different states, creation, editing, deletion and copying of criteria.

The data collection module includes an automatic collection sub-module, which is used to automatically receive data from externally transmitted source files.

The data interpretation and analysis module also includes a user sub-module and a report sub-module; the user sub-module is used for hierarchical data and criterion hierarchical management of users, roles and permissions; the report sub-module is used for automatic generation of interpretation result reports, and completes signing, confirm.

The data processing module can be dynamically compiled, packaged and dynamically scheduled to achieve hot deployment capabilities;

Among them, a large amount of R&D resources and tedious development process are saved through flexible configuration of data processing logic and data processing flow and hot deployment.

The preferred embodiments of the invention disclosed above are only to help illustrate the invention. The preferred embodiments are not exhaustive in all detail, nor are the inventions limited to specific embodiments described. Obviously, many modifications and variations can be made based on the contents of this specification. This description selects and specifically describes these embodiments in order to better explain the principle and practical application of the present invention, so that those skilled in the art can well understand and utilize the present invention. The invention is to be limited only by the claims, along with their full scope and equivalents.

Claims (8)

1. A large-scale processing and analysis system for arbitrary data mixing optimization, characterized in that, comprising a data acquisition module, a data conversion module, a data processing module, a learning module, a data management module and a data interpretation and analysis module; The data acquisition module is used to utilize the computing power of the big data platform to execute acquisition tasks in parallel in a distributed manner; The data conversion module is used to convert massive heterogeneous data into isomorphic data and transmit it to the data processing module; The data processing module is used to analyze the data packet, process the result according to the parameter information configured by the user, and send it to the data management module; The data management module is used to receive and store the original data sent by the data processing module, establish a storage structure, and complete real-time data storage; The learning module is used to integrate existing algorithm models to form an algorithm model database, and continuously optimize the model based on massive sample data training; The data interpretation and analysis module is used to automatically complete the interpretation of telemetry parameters according to different test states and test procedures; A transmission sub-module capable of transmitting data to each other is set between the data acquisition module and the data interpretation and analysis module, and the data acquisition module includes a field sub-module, the data acquisition module collects source file data, and analyzes the parameters of the source file field, to extract the data of each parameter field; the field submodule is used to set the corresponding relationship between the source file parameter field and the target file parameter field, and determine that the parameter field of the source file is the first data format, and the parameter field of the target file is the first data format. Two data formats, the field submodule generates a format conversion protocol corresponding to each first data format according to the similarity between the source file parameter field and the target file parameter field, and sends each format conversion protocol to the corresponding data conversion A module, according to the format conversion protocol, matching the data of the parameter field of the source file to the parameter field corresponding to the target file, and matching the data of the parameter field of the source file to the parameter field corresponding to the target file After calculating and judging the data of one or more parameter fields of the source file, generating data matched to the corresponding parameter field of the target file, and integrating the data into a data package. 2. the large-scale processing analysis system of a kind of arbitrary data mixing optimization according to claim 1, is characterized in that: described data acquisition module also comprises microprocessor unit, sending interface control submodule and receiving interface control submodule; The microprocessor unit is used to realize interface control according to usage requirements; the sending interface control submodule is used to realize data packet transmission based on bus protocol; the receiving interface control submodule is used to realize data packet transmission based on bus protocol Packet received. 3. the large-scale processing analysis system of a kind of arbitrary data mixing optimization according to claim 1, it is characterized in that: described data processing module comprises bus data processing submodule, auxiliary interpretation submodule and real-time data processing submodule; The bus data processing sub-module is used to implement data analysis based on the bus protocol; the auxiliary interpretation sub-module is used to assist the data interpretation and analysis module to interpret and analyze data; the real-time data processing sub-module is used to analyze real-time data. 4. A large-scale processing and analysis system for arbitrary data mixing optimization according to claim 1, wherein the data management module includes a database management submodule, a real-time detection parameter configuration submodule, and a data storage and release submodule And the database configuration import submodule; set up a real-time database in the data management module, the real-time database is used to store the real-time data delivered by the data processing module; the real-time detection parameter configuration submodule is used for the binding import of parameter information, for Verify the correctness of parameter information; the data storage and publishing sub-module is used to publish data processing results through the network and display them in real time; the database configuration import sub-module is used to migrate and backup previous test data, and supports the same format Import of unit test data. 5. The large-scale processing and analysis system of a kind of arbitrary data mixing optimization according to claim 1, it is characterized in that: described data interpretation and analysis module comprises data automatic interpretation sub-module and data comparison analysis sub-module; The interpretation sub-module is used to automatically complete the interpretation of telemetry parameters according to different test states and test procedures; the data comparison and analysis sub-module is used for horizontal comparison of different test data and stores criteria under different tasks and states , to create, edit, delete and copy the criteria. 6. A kind of large-scale processing analysis system of arbitrary data mixing optimization according to claim 1, it is characterized in that: described data collection module comprises automatic collection sub-module, is used for automatically receiving the described source file of external transmission data. 7. A large-scale processing and analysis system for arbitrary data mixing optimization according to claim 1, characterized in that: the data interpretation and analysis module also includes a user sub-module and a report sub-module; the user sub-module is used for user , hierarchical data of roles and permissions, and hierarchical management of criteria; the report sub-module is used for automatic generation of interpretation result reports, and completes signing and confirmation based on the network. 8. A large-scale processing and analysis system for arbitrary data mixing and optimization according to claim 1, characterized in that: said data processing module can perform dynamic compilation, packaging and dynamic scheduling to realize hot deployment capability.

Images