CN112364084B - A visual data processing method and system with in-depth customized algorithm integration - Google Patents

Abstract

The present invention relates to the field of data processing, and in particular to a visual data processing method and system for deeply customized algorithm integration. The method comprises the following steps: S1. By analyzing the algorithm library management, the required algorithms are packaged and uploaded, and the basic information of the algorithms is configured; S2. By constructing a data service model, the data is processed by the algorithm and arranged as a data service; S3. After the service is arranged, the service is approved by the data service approval module; S4. After the approval, the service is managed by the data service management module. The present invention integrates the management of algorithms and visualizes the data processing arrangement. And it can generate services after the arrangement, which is convenient for users to use to the greatest extent, and the present invention solves the problems of high learning cost and cumbersome processing for data processing personnel in a very simple and low-cost way.

Description

A visual data processing method and system with in-depth customized algorithm integration

Technical Field

The present invention relates to the field of data processing, and in particular to a visualization data processing method and system with in-depth customized algorithm integration.

Background Art

At present, with the advancement of science and technology, we have entered a data-driven era. Every organization and enterprise hopes to find value in data, and data has become an indispensable part of this new era. However, most data processing analysts still use traditional code writing, that is, using custom algorithms in languages such as python and java, or algorithms provided by big data processing tools such as Spark and flink to process data; and SQL statement analysis, that is, writing SQL statements in databases or hive data warehouse tools, and using their own functions to process data. However, these data processing methods either have high learning costs, requiring learning to write code, SQL writing, and the use of data processing tools, or are cumbersome to use, and when processing similar data, it is necessary to copy the previously written processing logic and modify it, or have poor reusability, and it is quite troublesome to hand over your own processing methods to others, and the other party may not necessarily use them. In general, the current technologies used for data processing have high costs, time-consuming and costly due to high learning costs, cumbersome use, and poor reusability.

At present, in terms of data processing, most data processors use traditional code or SQL statements to process data. Generally, users need to learn a variety of software such as hive data warehouse, Spark offline computing, and ozzie scheduling, or process data through code writing, which leads to high learning and usage costs. In order to solve these problems of data processors, it is necessary to design a system that can integrate various processing algorithms and can be easily used by data processors.

The current solution is the visual ETL tool Kettle. Kettle is an open source ETL tool written in pure Java. It can run on Windows, Linux, and Unix. It is green and does not require installation. It extracts data efficiently and stably. It allows you to manage data from different databases by providing a graphical user environment to describe what you want to do. There are two script files in Kettle, transformation and job. Transformation completes the basic transformation of data, and job completes the control of the entire workflow. However, Kettle cannot effectively manage the algorithms it has, and it is difficult to add new algorithms. In addition, the data conversion process is too cumbersome to arrange, and it is necessary to arrange the transformation and work. It cannot be published as a service after arrangement to facilitate data docking. Therefore, this technical solution has not yet been widely used.

At present, there is no effective system or technical solution to solve the problems of high learning cost, poor algorithm reusability, and cumbersome algorithm use in data processing. Processors need to learn a lot of software and master a lot of algorithms. The existing data processing software on the market is often not practical enough, too cumbersome to use, and unable to effectively integrate various algorithms, resulting in a small number of users.

Summary of the invention

The present invention provides a visual data processing method and system with in-depth customized algorithm integration, aiming to solve the current problems in data processing such as high learning cost, poor algorithm reusability, and cumbersome algorithm use.

The present invention provides a visual data processing method with in-depth customized algorithm integration, comprising the following steps:

S1. By analyzing the algorithm library management, package and upload the required algorithms, and configure the basic information of the algorithms;

S2. Through the construction of data service model, the data is processed by algorithms and arranged into data services;

S3. After arranging the service, the service is approved by the data service approval module;

S4. After approval, use the data service management module to manage the service.

As a further improvement of the present invention, the step S1 specifically includes:

The analysis algorithm library management provides an algorithm library including basic business calculation formulas, algorithm models, and general calculation methods. It also provides an editing entry for new algorithm tools. Users can package the algorithms they need, register the algorithms on the algorithm management page, and configure basic information including algorithm properties, algorithm details, and algorithm name.

As a further improvement of the present invention, the analysis algorithm library management in step S1 includes the following sub-execution modules:

S11. Algorithm list management: Display the currently added algorithms in a list and manage them by category;

S12. Algorithm query: search and query based on algorithm name and creation time;

S13. Add algorithm: The user adds an algorithm, fills in the detailed information, and imports the local algorithm package;

S14. Modify algorithm: User changes to algorithm details and re-uploads the algorithm package;

S15. Remove algorithm: delete useless algorithms and give reminders before deletion; meanwhile, give reminders not to allow deletion of occupied algorithms;

S16. Enable/disable algorithm: enable or disable the algorithm.

As a further improvement of the present invention, the step S2 specifically includes: after the user uploads the algorithm, the data is processed by building a data service model, and the data service model provides a visualization page, and the registered algorithm is introduced by dragging and dropping to process the data; including the following sub-steps:

S21. Pull in the data extraction operator and select the data source;

S22. Add the processing algorithm uploaded by the user, edit the configuration properties of the algorithm, and build the service model;

S23. Run the constructed service model for testing, and save the data source and processing algorithm after connecting them;

S24. Publish the connection result.

As a further improvement of the present invention, the uploaded algorithm in step S22 processes data by constructing a data service model, which specifically includes the following sub-steps:

S22a. Registration of basic model information: Registration information includes service model name, service model description, model permission settings, applicable area scope, and applicable time period;

S22b. Model label setting: Use label setting to give the data service a specified label;

S22c. Model parameter setting: Use visualization technology to intuitively display the model parameter configuration results. Parameter setting operations include adding, deleting, and modifying parameters;

S22d. Add business process: provide data service theme library API or form API sharing interface according to user needs, modularize the business process by adding sub-nodes; and configure the properties of the nodes of the data processing process.

As a further improvement of the present invention, step S23 specifically includes the following sub-steps:

S23a. Service model integrity assessment: Test the integrity of the service model, including task allocation feedback, process sub-nodes, data sources, integrity of logical rules, and continuity.

S23b. Service model trial run: If the integrity assessment of the service model is reasonable, a trial run operation is performed to connect all business processes, process sub-nodes, algorithms, and logical rules, and generate a new data form for storage, while forming computer-readable information.

As a further improvement of the present invention, the step S24 specifically includes:

Submit the reviewed service model to the service release management and save it as a shared service. Edit the service information including basic information, service operation strategy, and service permission configuration for release.

As a further improvement of the present invention, the process of the data service approval module executing approval in step S3 specifically includes:

Display service release approval items according to user permissions, and perform approval or failure operations based on user application content. When the approval result is failure, fill in the reason for approval failure.

As a further improvement of the present invention, the process of managing the service by the data service management module in step S4 includes:

S41. Control the generated services, specifically the following sub-execution sections:

S41a. Service information preview: preview all relevant information and authorization information of the service, including service metadata and model metadata, such as input and output parameters, service address, and call permissions;

S41b service start: start the suspended service and the sharing of authorization information;

S41c. Service suspension: temporarily stop the selected service and the sharing of authorization information, and retain the service;

S41d. Service removal: Remove the selected service and related authorization information, and retain the service version;

S41e. Service deletion: Delete the selected service and related authorization information;

S42. Manage the operation strategy, specifically the following sub-execution sections:

S42a regular operation: set the service automatic operation time, data source update method, update frequency, scheduled service call and save the operation results, according to the user-provided parameters to filter the result set service call;

S42b. Call and run: According to the data source and data logic saved by the service, during the user call process, the service automatically extracts data and calculates and returns service results in real time based on the parameters provided by the user.

The present invention also provides a visual data processing system with in-depth customized algorithm integration, including an analysis algorithm library management module, a data service model construction module, a data service approval module, and a data service management module; the analysis algorithm library management module, the data service model construction module, the data service approval module, and the data service management module are connected in sequence;

The analysis algorithm library management module: provides basic general information and provides an editing entry for newly added algorithm tools;

The data service model construction module: provides an entry for model construction and maintains the constructed model, mainly including data service model construction, service model operation test, and data service model release;

The data service approval module is used to approve the published model and determine whether the service can be used externally;

The data service management module controls the generated services and manages the operation strategies.

The beneficial effects of the present invention are: by integrating and managing algorithms, visualizing data processing and arranging, and being able to generate services after arranging, maximizing the convenience for users to use, and the present invention solves the problems of high learning cost and cumbersome processing for data processing personnel in a simple and low-cost manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of the visual data processing method of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

Embodiment 1:

This technical solution mainly configures the algorithm through the algorithm management module, and performs visual data processing for the submitted data according to the business application scenario. In terms of technology, B/S architecture is adopted, and the page design supports responsive adaptation to mobile browsing. It adopts the front-end and back-end separation technology architecture. The front-end is based on the VUE.JS technology system, and the server is developed based on the SpringBoot framework. It can be used as an independent service and can be seamlessly connected to the Spring Cloud microservice governance framework. The data storage adopts the distributed framework hadoop3.x. Based on the storage, Hive query and Spark calculation of hadoop3.x, it brings more convenient expansion capabilities and excellent performance. As the amount of data increases, it will not only not cause service unavailability, but also ensure that data is returned within a tolerable time.

As shown in FIG1 , a visualization data processing method with in-depth customized algorithm integration of the present invention comprises the following steps:

S1. By analyzing the algorithm library management, package and upload the required algorithms, and configure the basic information of the algorithms;

S2. Through the construction of data service model, the data is processed by algorithms and arranged into data services;

S3. After arranging the service, the service is approved by the data service approval module;

S4. After approval, use the data service management module to manage the service.

Wherein, step S1 specifically includes:

The analysis algorithm library management provides an algorithm library including basic business calculation formulas, algorithm models, general calculation methods, etc., and also provides an editing entry for new algorithm tools. Users can package the algorithms they need, register the algorithms on the algorithm management page, and configure basic information including algorithm properties, algorithm details, algorithm name, etc., for use in visual data modeling.

The analysis algorithm library management in step S1 includes the following sub-execution modules:

S11. Algorithm list management: Display the currently added algorithms in a list and manage them by category;

S12. Algorithm query: search and query based on algorithm name and creation time;

S13. Add algorithm: The user adds an algorithm, fills in the detailed information, and imports the local algorithm package;

S14. Modify algorithm: User changes to algorithm details and re-uploads the algorithm package;

S15. Remove algorithm: delete useless algorithms and give reminders before deletion; meanwhile, give reminders not to allow deletion of occupied algorithms;

S16. Enable/disable algorithm: enable or disable the algorithm.

Among them, step S2 specifically includes: after the user uploads the algorithm, the data is processed by building a data service model, and the data service model provides a visualization page, and the registered algorithm is introduced by dragging and dropping to process the data; it includes the following sub-steps:

S21. Pull in the data extraction operator and select the data source;

S22. Add the processing algorithm uploaded by the user, edit the configuration properties of the algorithm, and build the service model;

S23. Run the constructed service model for testing, and save the data source and processing algorithm after connecting them;

S24. Publish the connection result.

In step S22, the uploaded algorithm is processed by constructing a data service model, which specifically includes the following sub-steps:

S22a. Registration of basic model information: Registration information includes service model name, service model description, model permission settings, applicable area scope, and applicable time period;

S22b. Model label setting: Use label setting to give the data service a specified label;

S22c. Model parameter setting: Use visualization technology to intuitively display the model parameter configuration results. Parameter setting operations include adding, deleting, and modifying parameters;

S22d. Add business process: provide data service theme library API or form API sharing interface according to user needs, modularize the business process by adding sub-nodes; and configure the properties of the nodes of the data processing process.

Step S23 specifically includes the following sub-steps:

S23a. Service model integrity assessment: Test the integrity of the service model, including task allocation feedback, process sub-nodes, data sources, integrity and continuity of logical rules;

S23b. Service model trial run: If the integrity assessment of the service model is reasonable, a trial run operation is performed to connect all business processes, process sub-nodes, algorithms, and logical rules, and generate a new data form for storage, while forming computer-readable information.

Step S24 specifically includes:

Submit the reviewed service model to the service release management and save it as a shared service. Edit the service information including basic information, service operation strategy, and service permission configuration for release.

The process of the data service approval module executing approval in step S3 specifically includes:

Display service release approval items according to user permissions, and perform approval and approval failure operations according to the user's application content. When the approval result is failure, fill in the reason for approval failure. The compiled data model needs to be approved before it can be released as a service. It needs to be approved by authorized users to ensure the reasonable use of the service.

The data service management module manages the service in step S4, including:

S41. Control the generated services, specifically the following sub-execution sections:

S41a. Service information preview: preview all relevant information and authorization information of the service, including service metadata and model metadata, such as input and output parameters, service address, and call permissions;

S41b service start: start the suspended service and the sharing of authorization information;

S41c. Service suspension: temporarily stop the selected service and the sharing of authorization information, and retain the service;

S41d. Service removal: Remove the selected service and related authorization information, and retain the service version;

S41e. Service deletion: Delete the selected service and related authorization information.

S42. Manage the operation strategy, specifically the following sub-execution sections:

S42a regular operation: set the service automatic operation time, data source update method, update frequency, scheduled service call and save the operation results, according to the user-provided parameters to filter the result set service call;

S42b. Call and run: According to the data source and data logic saved by the service, during the user call process, the service automatically extracts data and calculates and returns service results in real time based on the parameters provided by the user.

Approved services can be managed on the data service management page. Data service management mainly controls the generated services and manages the operation strategies. Services can be started and stopped freely and the service availability time period can be specified.

The method of the present invention can effectively manage and integrate algorithms by analyzing algorithm library management as an entry point for algorithm integration; through visual process construction, it can be convenient for users to use and reduce learning costs; data service publishing can directly publish processed data as services, which is convenient for users to use.

The present invention provides a method of inheriting various algorithms and visually arranging data processors to eventually form data services, which is simple to use and highly practical. The algorithm is integrated and managed in a simple way, data processing and arrangement are visualized, and the data is published as a service after processing, which greatly reduces the cost of use.

Embodiment 2:

The present invention provides a visual data processing system with in-depth customized algorithm integration, including an analysis algorithm library management module, a data service model construction module, a data service approval module, and a data service management module; the analysis algorithm library management module, the data service model construction module, the data service approval module, and the data service management module are connected in sequence.

Analysis algorithm library management module: provides basic and general business calculation formulas, algorithm models, general calculation methods, etc., and also provides editing entry for newly added algorithm tools. The algorithms include Hive extraction algorithm, count algorithm, hive query algorithm, data aggregation algorithm, statistical algorithm, calculation algorithm, Spark algorithm, and spatial fusion calculation algorithm provided by the basic support platform algorithm set. The specific functions are as follows:

1) Algorithm list management: Display the currently added algorithms in a list format and manage them by category. The list display information includes algorithm name, algorithm description, creation time, current status (enabled/disabled), etc. The list is arranged in reverse order according to the creation time and supports page turning.

2) Algorithm query: supports search queries based on algorithm name and creation time.

3) New algorithm: supports users to add new algorithms and fill in detailed information, and supports importing local algorithm packages.

4) Modify algorithm: Support users to modify algorithm details and re-upload algorithm packages.

5) Remove algorithm: Supports deleting useless algorithms and gives reminders before deletion. At the same time, for occupied algorithms, a reminder is given that deletion is not allowed.

6) Enable/disable algorithm: supports enabling and disabling the algorithm. The start and stop of the algorithm directly affects the operation of the data service model.

Data service model construction module: provides an entry for model construction and maintenance of the completed model, mainly including data service model construction, service model operation testing, and data service model publishing.

The data service model construction includes:

1) Registration of basic model information: Registration information includes service model name, service model description, model permission settings, applicable area scope, applicable time period, etc.

2) Model tag setting: Use tag setting to give the data service a specific tag, which is similar to a description information, allowing new tags to be added and multiple tags to be selected.

3) Model parameter setting: Use visualization technology to intuitively display the model parameter configuration results. The main functions include adding, deleting and modifying parameters.

4) Add business process: Provide data service theme library API or form API sharing interface according to user needs, and modularize business process by adding sub-nodes (algorithms). At the same time, you can configure the properties of the nodes of the data processing process.

Service model run test runs the built service model, including the following functions:

1) Service model integrity assessment: Test the integrity of the service model, including task allocation feedback, process sub-nodes, data sources, integrity and continuity of logical rules, and other information.

2) Service model trial run: If the integrity assessment of the service model is reasonable, a trial run can be performed, which connects all business processes, process sub-nodes, algorithms, and logical rules, generates a new data form for storage, and forms computer-readable information.

Data service model release: Submit the reviewed service model to the service release management, edit the basic information of the service, service operation strategy, service permission configuration and other information for sharing services, and publish them.

Data service approval module: A key step to approve the published model and determine whether the service can be used externally. Confirmation by authorized users is required. This module displays all pending and completed tasks of the user. Service release approval items are displayed according to user permissions. Approval is performed based on the user's application content. If the approval result is failure, fill in the reason for the approval failure.

Data service management module: controls the generated services and manages the operation strategies. The specific contents are as follows:

1) Service operation management:

Service Information Preview: Preview all relevant information and authorization information of the service, including service metadata, model metadata, such as input and output parameters, service address, calling permissions, etc.;

Service startup: Start the suspended service and share the related authorization information;

Service suspension: temporarily stop the selected service and the sharing of related authorization information, and retain the service;

Service delisting: delist the selected service and related authorization information, and retain the service version;

Service Delete: Delete the selected service and related authorization information.

2) Service operation strategy management: Service operation strategy management is to manage the service calling mechanism. The specific functions are as follows:

Regular operation: set the service automatic operation time, data source update method, update frequency, perform scheduled service calls and save the operation results, and filter and provide result set service calls based on the parameters provided by the user;

Call and run: According to the data source and data logic saved by the service, the service automatically extracts data and calculates and returns service results in real time based on the parameters provided by the user during the user call process.

The system of the present invention is a visual data processing tool customized for various data processing needs and services. Through the customization of the underlying data processing algorithm, it is possible to easily build ETL and ETL processes without code, easily introduce, move, prepare, convert and process data, and complete data modeling in an intuitive visual environment. It is a strong support for realizing multi-source data fusion and multi-service fusion, and can improve quality and efficiency in the process of providing data service capabilities to the outside world. Through customized algorithms and visual data processing methods, it better solves the problems of high learning cost, cumbersome use, poor reusability, etc. in the data processing process, which can provide convenience for data processing personnel and reduce the cost of use.

The present invention can directly generate data services, which is convenient for data docking; the present invention has low learning cost, and the visual construction service can quickly and simply build the processing flow; the analysis algorithm library management can effectively manage and integrate the algorithms.

The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims (8)

1. A visual data processing method with in-depth customized algorithm integration, characterized by comprising the following steps: S1. By analyzing the algorithm library management, package and upload the required algorithms, and configure the basic information of the algorithms; S2. Through the construction of data service model, the data is processed by algorithms and arranged into data services; S3. After arranging the service, the service is approved by the data service approval module; S4. After approval, use the data service management module to manage the service; The step S2 specifically includes: after the user uploads the algorithm, the data is processed by building a data service model, and the data service model provides a visualization page, and the registered algorithm is introduced by dragging and dropping to process the data; it includes the following sub-steps: S21. Pull in the data extraction operator and select the data source; S22. Add the processing algorithm uploaded by the user, edit the configuration properties of the algorithm, and build the service model; S23. Run the constructed service model for testing, and save the data source and processing algorithm after connecting them; S24. Publish the connection result; In step S22, the uploaded algorithm is processed by constructing a data service model, which specifically includes the following sub-steps: S22a. Registration of basic model information: Registration information includes service model name, service model description, model permission settings, applicable area scope, and applicable time period; S22b. Model label setting: Use label setting to give the data service a specified label; S22c. Model parameter setting: Use visualization technology to intuitively display the model parameter configuration results. Parameter setting operations include adding, deleting, and modifying parameters; S22d. Add business process: provide data service theme library API or form API sharing interface according to user needs, modularize the business process by adding sub-nodes; and configure the properties of the nodes of the data processing process. 2. The visual data processing method with in-depth customized algorithm integration according to claim 1 is characterized in that the step S1 specifically comprises: The analysis algorithm library management provides an algorithm library including basic business calculation formulas, algorithm models, and general calculation methods. It also provides an editing entry for new algorithm tools. Users can package the algorithms they need, register the algorithms on the algorithm management page, and configure basic information including algorithm properties, algorithm details, and algorithm name. 3. According to the visual data processing method with in-depth customized algorithm integration as claimed in claim 2, it is characterized in that the analysis algorithm library management in step S1 includes the following sub-execution modules: S11. Algorithm list management: Display the currently added algorithms in a list and manage them by category; S12. Algorithm query: search and query based on algorithm name and creation time; S13. Add algorithm: The user adds an algorithm, fills in the detailed information, and imports the local algorithm package; S14. Modify algorithm: User changes to algorithm details and re-uploads the algorithm package; S15. Remove algorithm: delete useless algorithms and give reminders before deletion; meanwhile, give reminders not to allow deletion of occupied algorithms; S16. Enable/disable algorithm: enable or disable the algorithm. 4. According to the visual data processing method with in-depth customized algorithm integration as claimed in claim 1, it is characterized in that the step S23 specifically includes the following sub-steps: S23a. Service model integrity assessment: Test the integrity of the service model, including task allocation feedback, process sub-nodes, data sources, integrity and continuity of logic rules; S23b. Service model trial run: If the integrity assessment of the service model is reasonable, a trial run operation is performed to connect all business processes, process sub-nodes, algorithms, and logical rules, and generate a new data form for storage, while forming computer-readable information. 5. According to the visual data processing method with in-depth customized algorithm integration as claimed in claim 1, it is characterized in that the step S24 specifically comprises: Submit the reviewed service model to the service release management and save it as a shared service. Edit the service information including basic information, service operation strategy, and service permission configuration for release. 6. According to the visual data processing method with in-depth customized algorithm integration as claimed in claim 1, it is characterized in that the process of the data service approval module executing approval in step S3 specifically includes: Display service release approval items according to user permissions, and perform approval or failure operations based on user application content. When the approval result is failure, fill in the reason for approval failure. 7. The visual data processing method with in-depth customized algorithm integration according to claim 1 is characterized in that the data service management module in step S4 manages the service including: S41. Control the generated services, specifically the following sub-execution sections: S41a. Service information preview: preview all relevant information and authorization information of the service, including service metadata and model metadata, such as input and output parameters, service address, and call permissions; S41b service start: start the suspended service and the sharing of authorization information; S41c. Service suspension: temporarily stop the selected service and the sharing of authorization information, and retain the service; S41d. Service removal: Remove the selected service and related authorization information, and retain the service version; S41e. Service deletion: Delete the selected service and related authorization information; S42. Manage the operation strategy, specifically the following sub-execution sections: S42a regular operation: set the service automatic operation time, data source update method, update frequency, scheduled service call and save the operation results, according to the user-provided parameters to filter the result set service call; S42b. Call and run: According to the data source and data logic saved by the service, during the user call process, the service automatically extracts data and calculates and returns service results in real time based on the parameters provided by the user. 8. A visual data processing system with in-depth customized algorithm integration, characterized by comprising an analysis algorithm library management module, a data service model construction module, a data service approval module, and a data service management module; the analysis algorithm library management module, the data service model construction module, the data service approval module, and the data service management module are connected in sequence; The analysis algorithm library management module: provides basic general information and provides an editing entry for newly added algorithm tools; The data service model construction module: provides an entry for model construction and maintains the constructed model, mainly including data service model construction, service model operation test, and data service model release; The data service approval module is used to approve the published model and determine whether the service can be used externally; The data service management module controls the generated services and manages the operation strategies; The data service model construction includes: 1) Registration of basic model information: Registration information includes service model name, service model description, model permission settings, applicable area scope, and applicable time period; 2) Model tag setting: Use tag setting to give a data service a specific tag, which is similar to a description information. It allows adding tags and selecting multiple tags. 3) Model parameter setting: Use visualization technology to intuitively display the model parameter configuration results. The main functions include adding, deleting and modifying parameters; 4) Add business processes: Provide a data service theme library API or form an API sharing interface based on user needs, modularize the business process by adding sub-nodes (algorithms); and configure the properties of the nodes of the data processing process.