WO2025001683A1 - Data processing method and device, task scheduling method, and storage medium - Google Patents

Abstract

The present invention relates to the field of data processing, and provides a data processing method and device, a task scheduling method, and a storage medium. According to embodiments of the present invention, when an SQL association query statement is obtained, a data table to be processed is obtained on the basis of data source information in the SQL association query statement; on the basis of data table information of said data table, the SQL association query statement is converted into an executable SQL query statement; and the executable SQL query statement is executed to obtain a query result. An SQL association query statement which is input by a user and conforms to a preset specification is converted to obtain an executable SQL query statement, and thus, data in different data sources can be fused together in real time simply by utilizing an SQL language.

Description

A data processing method, task scheduling method, device and storage medium Technical Field

The present invention relates to the field of data processing, and in particular to a data processing method, a task scheduling method, a device and a storage medium.

Background Art

With the advent of the big data era, enterprises have begun to use big data platforms to uniformly store, manage, and analyze data generated during production and operations. When storing data, the same enterprise can store data in different data sources according to demand, but when analyzing data, the data stored in different data sources must be merged together for query analysis. Generally, ETL tools can be used to synchronize data from different data sources into a certain data warehouse and analyze it in the data warehouse, but this method requires pre-synchronization of data, which is not only complex and time-consuming, but the synchronized data has become historical data and cannot perform real-time data calculation and analysis.

Summary of the invention

The present invention provides a data processing method, a task scheduling method, a device and a storage medium to solve the deficiencies in the related art.

According to a first aspect of an embodiment of the present invention, there is provided a data processing method, which is applied to a big data platform, wherein the big data platform includes at least two different data sources; the method comprises:

Displaying a data processing task page, and acquiring an SQL associated query statement in response to an input operation of a user on the data processing task page, wherein the SQL associated query statement includes data source information of a plurality of first data tables to be processed, wherein at least two of the plurality of first data tables to be processed are from different data sources;

Acquire a second table of data to be processed according to the data source information;

Replace the data source information of the first to-be-processed data table in the SQL associated query statement with the corresponding The data table information of the second data table to be processed is used to obtain an executable SQL query statement;

Execute the executable SQL query statement to obtain the query result.

In some implementations, the data processing method is implemented based on a PySpark framework.

In some implementations, after obtaining the query result, the method further includes:

Determining a processing condition for processing the query result based on the user's input operation;

Processing the query result based on the processing condition to obtain a processing result;

A column is added to the query result, and the processing result is displayed in the column.

In some implementations, after obtaining the query result, the method further includes:

Based on the configuration operation of the user, determine the target database for storing the query results and the name of the data table for storing the query results;

If the target database does not include a data table corresponding to the data table name, create a data table corresponding to the data table name;

If the target database includes a data table corresponding to the data table name, an output mode is determined according to a selection operation of the user, and the query result is written into the data table of the target database in the output mode.

According to a second aspect of an embodiment of the present invention, a task scheduling method is provided, which is applied to a big data platform, and the method includes:

Displaying a task scheduling page, and creating a scheduling task based on the user's input operation on the task scheduling page;

Determining multiple target data tasks of the scheduling task from multiple pre-created candidate data tasks based on a user's selection operation, wherein the candidate data tasks include data processing tasks, and the data processing tasks are implemented using any of the above-mentioned data processing methods;

Displaying icons of each target data task on the task scheduling page;

Determining the execution order of each target data task in the scheduling task based on the user's movement operation on each icon;

Execute each target data task in the scheduled task according to the execution order.

In some implementations, the candidate data task also includes a data integration task; the method also includes include:

Based on the user's operation information on the data integration page, determine the data integration task name, data source information, and data destination information;

A data integration task corresponding to the data integration task name is created according to the data source information and the data destination information.

In some implementations, the method further comprises:

Based on the user's selection operation of the icon, displaying the development page of the target data task corresponding to the icon;

In response to detecting a modification operation of the user on the development page, the modified target data task is saved.

In some implementations, the input operation includes a first run time of a scheduled task;

The executing each target data task in the scheduled task according to the execution order includes:

In response to the current time reaching the first running time of the scheduled task, executing each target data task in the scheduled task according to the execution order, and obtaining the running status of each target data task;

If the running state is an error state, stopping the execution of the scheduled task and returning the state of the scheduled task to an error state;

If all target data tasks are executed in the execution order of the target data tasks, the status of the scheduled task is returned as completed.

In some implementations, the method further comprises:

In the scheduling task template page, the corresponding scheduling task template is displayed based on the user's selection operation;

Obtaining user configuration information for the scheduling task template;

A scheduling task matching the configuration information is generated based on the scheduling task template.

According to a third aspect of an embodiment of the present invention, there is provided a data processing device, which is applied to a big data platform, wherein the big data platform includes at least two different data sources; the device comprises:

The acquisition unit is used to display the data processing task page, and obtain the SQL associated query statement in response to the user's input operation on the data processing task page, wherein the SQL associated query statement includes multiple data source information of first data tables to be processed, wherein at least two of the plurality of first data tables to be processed are from different data sources;

a conversion unit, configured to obtain a second data table to be processed according to the data source information, replace the data source information of the first data table to be processed in the SQL association query statement with data table information corresponding to the second data table to be processed, and obtain an executable SQL query statement;

The execution unit is used to execute the executable SQL query statement to obtain the query result.

According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, and when an executable computer program in the storage medium is executed by a processor, any of the above-mentioned methods can be implemented.

According to the above embodiments, after obtaining the SQL associated query statement, the present invention obtains the data table to be processed according to the data source information in the SQL associated query statement; according to the data table information of the data table to be processed, the SQL associated query statement is converted into an executable SQL query statement, and the executable SQL query statement is executed to obtain the query result. By converting the SQL associated query statement input by the user that meets the pre-specified specifications, an executable SQL query statement is obtained, and the data located in different data sources can be merged together in real time using the SQL language.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

FIG. 1 is a flow chart showing a data processing method according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of a data processing task page according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of an operation flow of a data processing task according to an embodiment of the present invention.

FIG. 4 is a schematic diagram showing a page for adding a calculated column according to an embodiment of the present invention.

FIG. 5 is a schematic diagram showing a newly added column in a query result according to an embodiment of the present invention.

FIG. 6 is a schematic diagram showing output to a library page according to an embodiment of the present invention.

FIG. 7 is a flow chart showing a task scheduling method according to an embodiment of the present invention.

FIG. 8 is a schematic diagram showing a task scheduling sequence according to an embodiment of the present invention.

FIG. 9A is a schematic diagram of a first data integration page according to an embodiment of the present invention.

FIG9B is a schematic diagram of a second data integration page according to an embodiment of the present invention.

FIG9C is a schematic diagram of a third data integration page according to an embodiment of the present invention.

FIG. 10A is a schematic diagram of a task scheduling page according to an embodiment of the present invention.

FIG. 10B is a schematic diagram showing a data integration task list according to an embodiment of the present invention.

FIG. 10C is a schematic diagram showing a data processing task list according to an embodiment of the present invention.

FIG. 11 is a schematic diagram showing an operation sequence according to an embodiment of the present invention.

FIG. 12 is a schematic diagram showing a template reference process according to an embodiment of the present invention.

FIG. 13 is a schematic diagram of a data processing device according to an embodiment of the present invention.

DETAILED DESCRIPTION

Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

Data from different data sources can be stored in different forms and rely on different database management systems, but in some scenarios, cross-data source queries need to be implemented, that is, data from different data sources are brought together for analysis. There are some technical difficulties in cross-data source queries, mainly including the following aspects:

Heterogeneity of data sources: Different data sources have different storage structures and query syntaxes, requiring data conversion and syntax conversion.

Data security: During cross-data source query processes, data security and privacy need to be guaranteed, and operations such as data permission verification and data encryption are required.

Performance issues: Cross-data source queries require data transmission and communication between different data sources. May affect query performance.

Therefore, in order to bring together data from different data sources for analysis, ETL tools can usually be used to synchronize data from different data sources into a certain data warehouse for analysis. However, this method requires pre-synchronization of data, and the synchronization process is complex and time-consuming. In addition, the synchronized data has become historical data, and real-time data calculation and analysis cannot be performed.

In view of this, the present invention provides a data processing method, which can bring together data from different data sources based on the PySpark framework. That is, when performing data processing tasks, if the current data analysis involves data from different data sources, this method can be used to fuse the required data from different data sources, which is referred to as cross-source fusion.

The present invention can be applied to a big data platform. By adding the data source to the big data platform through information such as the IP, port, database name and table name of the database in the data source, the big data platform can include at least two different data sources.

The following embodiments will illustrate the present invention with reference to the accompanying drawings.

FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps 101 to 104 .

In step 101, a data processing task page is displayed, and an SQL associated query statement is obtained in response to an input operation of a user on the data processing task page.

When the data tables to be processed are located in different data sources, the user needs to enter the SQL association query statement in the agreed format, that is, identify the table name in the format of "data source.data table". In this embodiment, the data source and the data table are referred to as data source information, and the data source information is used to indicate the data table in the specified data source.

In order to distinguish it from the data tables to be processed mentioned later, the data table to be processed in the SQL associated query statement input by the user is called the first data table to be processed, and the SQL associated query statement includes data source information of multiple first data tables to be processed, wherein at least two of the multiple first data tables to be processed are from different data sources.

In an optional implementation, FIG. 2 is a schematic diagram of a data processing task page according to an embodiment of the present invention. As shown in FIG. 2 , a user can enter an SQL associated query in area 201. sentence.

For example, the SQL query statement entered by the user complies with the following specifications:

select*from(data source.data table)(left/right)join(data source.data table)on connection condition

Among them, the data source, data table, and connection conditions can be determined according to actual needs.

When multiple data tables queried by the user belong to different data sources, the user can write the table name of the first data table to be processed in the format of "data source.database" when entering the SQL association query statement. For the user, although this method is implemented through the PySpark framework, the SQL language can be entered instead of Python code, which reduces the requirements for operators.

Since the table name format in the SQL association query statement input by the user is "data source.database", the SQL association query statement cannot be executed and needs to be converted through subsequent steps 102 and 103.

In step 102, a second to-be-processed data table is obtained according to the data source information.

After obtaining the SQL associated query statement input by the user, the SQL associated query statement can be verified, and after the verification is passed, SQL parsing is performed to obtain data source information. In the PySpark framework, after obtaining the data source information, the data table in the data source is read according to the indication of the data source information to generate a distributed data set (Dataframe) grouped by column names. In order to distinguish it from the previously mentioned data table to be processed, in this embodiment, the data table to be processed generated by reading the data table in the data source is called the second data table to be processed.

In step 103, the data source information of the first data table to be processed in the SQL association query statement is replaced with the data table information corresponding to the second data table to be processed, so as to obtain an executable SQL query statement.

The executable SQL query statement can be obtained by replacing the data source information representing the first data table to be processed in the SQL association query statement with the data table information corresponding to the second data table to be processed.

In step 104, the executable SQL query statement is executed to obtain a query result.

The present invention selects the PySpark framework to perform cross-source fusion analysis and data processing, submits the task to the Yarn cluster manager for execution, and uses Yarn to uniformly manage and schedule the resources of all nodes in the cluster. In addition to the Yarn cluster manager, the Standalone cluster manager and the Mesos cluster manager can also be used for management, which is not limited in this embodiment. The data processing method provided by the present invention can integrate data scattered in different data sources for query and analysis, thereby improving the efficiency of query and the accuracy of data analysis, and can also realize functions such as data sharing and collaborative analysis, which helps to optimize the efficiency of data management and application development.

The following embodiments will illustrate the data processing method of the present invention by means of specific examples.

Assume that MySql is used as a business database to store basic user information and order information tables. ClickHouse is used as a column storage database. It has the advantages of storing large amounts of data and fast query speed, and is suitable for storing detailed data browsed by users. For example, the user's current demand is to calculate the user conversion rate of a store every day in the current month. The calculation logic is: conversion rate = number of users who place orders / number of users who browse.

According to the above requirements, it is necessary to perform association analysis between the order data in MySql and the browsing data in ClickHouse using user information as the association field. In order to achieve association analysis, users need to write SQL in the format of "data source.data table" as the table name.

FIG3 is a schematic diagram of the operation flow of a data processing task according to an embodiment of the present invention. As shown in FIG3 , assuming that the data source mysql2 in MySql and the data source ch3 in ClickHouse have been added on the big data platform, taking the user information table user_info and the browse details table browse_details as examples, the writing format of the associated query syntax can be:

select*from mysql2.user_info left join ch3.browse_details on user_info.id=browse_details.user_id;

On the data processing task page, after the user writes the SQL associated query statement according to the rules, in response to the user clicking the run control, the SQL associated query statement is obtained, and SQL verification is performed. After the verification passes, SQL parsing is performed, and the data source, table name and field name can be parsed according to the rules; based on the parsed information, in the PySpark framework, the data table in the corresponding data source is read to generate the corresponding Dataframe, such as user_info in mysql2, to obtain the second data table to be processed, represented by df1, and the browse_details in ch3 is read to obtain the second data table to be processed, represented by df2.

The data source information of the first to-be-processed data table in the SQL associated query statement input by the user is replaced with the data table information corresponding to the second to-be-processed data table, that is, the data source information of the first to-be-processed data table in the SQL associated query statement input by the user is replaced with the data table information corresponding to the second to-be-processed data table. Replace mysql2.user_info with df1 and ch3.browse_details with df2 to get an executable SQL query statement, namely select * from df1 left join df2 on df1.id=df2.user_id.

In the PySpark framework, the executable SQL query statement is executed to obtain a result data set, that is, the query result.

It should be noted that although the cross-data source query and cross-database query in the present invention both involve accessing multiple data sources or multiple databases in one query, they are different from each other in the following ways:

Cross-data source query usually refers to accessing multiple different database types or different data storage systems in one query, such as relational databases, NoSQL databases, text files, Hadoop clusters, etc. The underlying storage structure, query syntax, etc. of these data sources may be different. Therefore, cross-data source query requires some technical means to connect these different data sources, and then perform unified data query and processing.

Cross-database query refers to accessing multiple databases of the same type in one query, such as querying in the same MySQL instance. In this case, the data storage structure and query syntax are the same, so cross-database query is relatively simple.

If two databases are deployed on different machines with different IP addresses and ports, the two databases belong to different data sources. For example, MySQL-10.10.111.111 and MySQL-47.22.22.22 belong to different data sources of the same type. There are multiple databases under the MySQL-10.10.111.111 data source, such as bdmp, datax, information_schema, and mysql111, and each database includes multiple data tables. There are also multiple databases under the MySQL-47.22.22.22 data source, such as mysql222.

In an example, the bdmp database includes a user_info data table, and the datax database includes a df data table. In this case, if a query involves the user_info data table under the bdmp database and the df data table under the datax database, the query involves different databases under the same data source and is a cross-database query.

In another example, the bdmp database includes a user_info data table, and the mysql222 database includes a test data table. When the test data table and the user_info data table are involved in a query, it indicates that the user_info data table in the MySQL-10.10.111.111 data source and the test data table in the MySQL-47.22.22.22 data source need to be cross-databaseed using the method provided by the present invention. Source fusion.

The present invention can be applied to the following scenarios:

1. When demonstrating to users or performing preliminary analysis, it takes a lot of time to synchronize the user's data to the big data platform. Therefore, the user's data can be accessed through the access method of the present invention.

2. Considering data security, when the user does not open the data synchronization authority, the access method of the present invention can be used to access the user's data.

3. In the case where the user's data is not updated regularly, if the updated data of the user is not updated synchronously, the analysis result will be inaccurate. The present invention can be used to analyze the latest data stored by the user, which can improve the accuracy of the analysis result.

In some embodiments, after obtaining the query results, it may also include: determining the processing conditions for processing the query results based on the user's input operation; processing the query results based on the processing conditions to obtain processing results; adding a column to the query results and displaying the processing results in the column.

As shown in FIG. 2 , the data processing task page includes adding a calculation column control. The processing condition in this embodiment may be a user-defined calculation rule or a user-defined screening condition.

In one embodiment, in response to detecting that the add calculated column control is triggered, the add calculated column page shown in FIG4 may be displayed, the add calculated column page includes a calculation rule input box and a determination control, and in response to detecting that the determination control is triggered, the python code entered by the user in the calculation rule input box is run. Functions that are inconvenient or impossible to implement with SQL statements can be implemented through custom calculated columns, by obtaining the column name of the newly added column entered by the user and the written python code, and executing the code segment as a function after the code format is verified to pass, to obtain the newly added column.

Figure 5 is a schematic diagram of adding a new column in the query results according to an embodiment of the present invention. As shown in Figure 5, the query results are displayed in area 501 of the data processing task page, and a custom column is added to the list corresponding to the query results, namely the col1 column on the far right.

In some embodiments, after obtaining the query result, the method may further include: determining a target database for storing the query result and a data table name for storing the query result based on the configuration operation of the user; if the target database does not include a data table corresponding to the data table name, creating the a data table corresponding to the data table name; if the target database includes a data table corresponding to the data table name, determining an output mode according to a user's selection operation, and writing the query result into the data table of the target database in the output mode.

That is, the data processing task page includes an output to library control, which is used to determine the location where the query results are stored.

FIG6 is a schematic diagram of an output to library page according to an embodiment of the present invention. In response to detecting that the output to library control is triggered, the output to library page shown in FIG6 is displayed, and the configuration information entered by the user in the output to library page, namely the target source and table name, and the output mode, is obtained. Check whether the table name configured by the user exists in the database. If the table does not exist, create the table and store the query results; if the table exists, store it according to the selected output mode. If the append mode is selected, the query results are appended to the table; if the overwrite mode is selected, the table is deleted and the table is re-created to write the data.

In some embodiments, the data processing task page as shown in FIG2 also includes a download control, and in response to the download control being triggered, the query results are output to a csv file.

The current big data platform needs to determine the execution order of each data task in the scheduling task according to the running script input by the developer, which is costly and error-prone for the developer. In view of this, the present invention provides a task scheduling method, based on which the user can drag and drop the icons corresponding to multiple data tasks in the target area to realize the execution order of each data task in the scheduling task, which reduces the workload of the developer and improves the development efficiency.

The following embodiments will illustrate the task scheduling method in conjunction with the accompanying drawings.

FIG. 7 is a flow chart of a task scheduling method according to an embodiment of the present invention. As shown in FIG. 7 , the method includes the following steps 701 - 705 .

In step 701, a task scheduling page is displayed, and a scheduling task is created based on the user's input operation on the task scheduling page.

In step 702, a plurality of target data tasks of the scheduling task are determined from a plurality of pre-created candidate data tasks based on a user's selection operation.

The candidate data tasks include data processing tasks, and the data processing tasks are implemented using any of the data processing methods described above.

In step 703, icons of each target data task are displayed on the task scheduling page.

In step 704, based on the user's movement operation on each icon, the execution order of each target data task in the scheduled task is determined.

In step 705, each target data task in the scheduled task is executed according to the execution order.

The scheduling task in this embodiment can be an offline scheduling task. In one implementation, the icon of the target data task can be in the form of a card, that is, each offline scheduling task can contain multiple target data tasks, each target data task has a dependency relationship, and each target data task can be executed serially or in parallel. After each target data task card is displayed on the task scheduling page, the execution order of the target data tasks can be defined by dragging the task card.

FIG8 is a schematic diagram of the task scheduling sequence according to an embodiment of the present invention. As shown in FIG8 , when a task card is behind another task card, it is sequential execution, and the next task is executed after one task is completed; when multiple task cards are dragged to the same vertical position, it is parallel execution, and multiple tasks can be executed at the same time, and the number of parallel execution tasks is not fixed, for example, n tasks can be executed at the same time. It should be noted that there are also two situations for the parallel scheduling of tasks. Task 4 needs to wait until tasks 1-n are all executed before it can be executed; and task 9 can start execution as soon as task 8 is completed.

Parallel application scenarios may include: a large number of tasks need to be processed in the same scheduling task, there is no dependency between the tasks executed in parallel, and a quick response to task requests is required. By executing multiple independent target data tasks in parallel, the task processing efficiency of the entire system can be improved, the task processing time can be shortened, and the system throughput and response speed can be improved.

In addition to data processing tasks, the candidate data tasks in this embodiment may also include data integration tasks, shell script tasks, python script tasks, etc., which can be added according to actual scheduling needs, and the present invention is not limited to this.

The data processing task in the present invention refers to the ability to query and perform correlation calculations on data between various data sources by editing SQL statements, and supports the direct output of data analysis results to the library and downloading, and supports custom calculation columns to implement complex logical functions. This has been explained in the processing method and will not be repeated here.

The data integration task in the present invention is data synchronization between various data sources. The data sources may include databases, file systems, service interfaces, and message queues. Data can be integrated between various data sources through integration tasks. The main application scenarios of data integration tasks are data synchronization, data integration (aggregation), data migration, and data exchange.

The shell script task in the present invention is a code that implements a certain function by a set of shell syntax or instructions. When periodic/non-periodic scheduling is required, the .sh file can be uploaded and added to the scheduling task for execution.

The python script task in the present invention is a code written in python language to implement a certain function. When periodic/non-periodic scheduling is required, it can be added to the scheduling task for execution by uploading the .py file.

The following embodiments will introduce the data integration task.

In some embodiments, the method may also include: determining the data integration task name, data source information and data destination information based on the user's operation information on the data integration page; and creating a data integration task corresponding to the data integration task name based on the data source information and the data destination information.

Figures 9A, 9B and 9C are schematic diagrams of three data integration pages according to an embodiment of the present invention, respectively, to obtain the data integration task name, label information and description information entered by the user in the data integration page shown in Figure 9A, to obtain the data source information selected by the user on the data source page shown in Figure 9B, and to obtain the data destination information selected by the user on the data destination page shown in Figure 9C. A data integration task corresponding to the data integration task name is created based on the data source information and the data destination information.

Create a data integration task on the big data platform to synchronize data from one data source to another. Take synchronizing a data table in PG to the ClickHouse database as an example to configure the data integration task: First, fill in the basic information, that is, fill in the task name and description in the data integration page shown in Figure 9A; secondly, select the data source, that is, select the table to be synchronized in PG on the data source page shown in Figure 9B, and check the data fields to be synchronized; then select the data destination, that is, in the On the data destination page shown in Figure 9C, select the destination table and corresponding fields in ClickHouse.

The task scheduling page of this embodiment includes a reference data integration task control and a reference data processing task control. The target data integration task is selected from the candidate data integration tasks through the reference data integration task control, and the target data processing task is selected from the candidate data processing tasks through the reference data processing task control.

In response to detecting that the reference data integration task control is triggered, a pre-created data integration task list corresponding to the reference data integration task control is displayed; in response to detecting that the data integration task in the data integration task list is triggered, a corresponding data integration task card is displayed in the target area of the task scheduling page.

In response to detecting that the reference data processing task control is triggered, a pre-created data processing task list corresponding to the reference data processing task control is displayed; in response to detecting that the data processing task in the data processing task list is triggered, a corresponding data processing task card is displayed in the target area of the task scheduling page.

In one embodiment, Figure 10A is a schematic diagram of a task scheduling page according to an embodiment of the present invention, showing the task scheduling page as shown in Figure 10A, and creating a scheduled task based on the basic information entered by the user on the task scheduling page and the selected first run time and run cycle (for example, every hour).

The task scheduling page shown in FIG. 10A includes a log package download control, a reference data integration task control, and a reference data processing task control.

When the task scheduling details page does not reference any data task, the log package download control cannot be triggered; after referencing the target data task, triggering this control can package and download the running log content of all target data tasks to the local computer. This function is generally used to check the task running status or to troubleshoot problems when a task is abnormal.

In response to the reference data integration task control being clicked, a list of data integration tasks that have been created as shown in FIG. 10B can be displayed. Select the required data integration tasks (multiple selections are allowed) to generate a data task card. By default, data integration tasks are displayed in reverse chronological order. When there are many data integration tasks, the search box can be used to find the target integration task.

In response to the reference data processing task control being clicked, a list of data processing tasks that have been created as shown in FIG. 10C can be displayed, and the required data processing tasks (multiple selections are allowed) can be selected to generate data task cards. By default, data processing tasks are displayed in reverse chronological order. When there are many data processing tasks, the search box can be used to find the target processing task.

Drag the task cards according to the dependencies to adjust the order of data tasks. When it is detected that the user clicks the save control, the creation of the offline scheduling task is completed.

In some embodiments, based on the user's selection operation on the icon, a development page of the target data task corresponding to the icon is displayed; in response to detecting the user's modification operation on the development page, the modified target data task is saved.

In the case where the icon is a data task card, the data task card includes a viewing control; in response to detecting that the viewing control is triggered, the development page corresponding to the data task card is displayed; in response to detecting that the user modifies the target data task on the development page, the modified target data task is saved.

The data tasks in this embodiment are all stored on the big data platform. Compared with the related art that requires inputting commands to view the running status of data tasks, this embodiment allows data tasks to be modified by clicking on the viewing control, which is convenient for user operation.

In some embodiments, the data task card may further include a manual run control; the method further includes: in response to detecting that the manual run control is triggered, running the data task corresponding to the data task card and displaying the run result. Without uploading the run script, the data task can be run independently.

In some embodiments, the input operation includes a first run time of a scheduled task;

The executing each target data task in the scheduled task according to the execution order includes:

In response to the current time reaching the first running time of the scheduled task, executing each target data task in the scheduled task according to the execution order, and obtaining the running status of each target data task;

If the running state is an error state, stopping the execution of the scheduled task and returning the state of the scheduled task to an error state;

If all target data tasks are executed in the execution order of the target data tasks, the status of the scheduled task is returned as completed.

Figure 11 is a schematic diagram of the operating sequence according to an embodiment of the present invention. As shown in Figure 11, the basic information of the offline scheduling task configured by the user, the operating cycle, the first operating time, all the target data tasks contained in each scheduling task, and the execution order of the target data tasks are stored in the specified data table; when the current time reaches the first operating time of the task, the round of scheduling tasks is executed: the target data tasks are traversed and executed, and the operating status of each target data task is obtained. When the operating status of the target data task is an error, the scheduling task is terminated, the execution is no longer continued, and an error status is returned; when the target data task is completed normally, the next target data task is executed until all target data tasks are completed and the completed status is returned.

In some embodiments, the big data platform may further include a scheduling task template page, wherein the scheduling task template page includes a plurality of preset scheduling task template controls, and the method further includes:

In the scheduling task template page, in response to detecting that any scheduling task template control is triggered, displaying the scheduling task template corresponding to the scheduling task template control;

Obtaining user configuration information for the scheduling task template;

A scheduling task matching the configuration information is generated based on the scheduling task template.

Scheduling task templates can be used to easily generate offline scheduling tasks by referencing templates during data analysis, thus reducing duplication of work. There are two ways to generate templates: user-generated templates and system-provided templates. After creating an offline scheduling task, users can click the Generate Template button to save the template to the template list for direct reference the next time they perform a similar task. The system-provided templates are based on common indicators of big data analysis, such as daily active users, monthly active users, online rate, conversion rate, etc. Users can select them as needed.

For example, taking the calculation conversion rate as an example, FIG12 is a schematic diagram of the template reference process according to an embodiment of the present invention. As shown in FIG12, after referencing the scheduling task template, the user can modify the task name, description, set the first run time and run cycle, etc.; the scheduling task template is preset with task cards, and task cards can be added or deleted according to the user's selection operation. You can also enter the task card details page to modify the task content. For example, if it is a data integration task, you can modify the content of the data source and data destination. content, specify the target data source; if it is a data processing task, you can modify the SQL code and run debugging, edit the newly added calculated columns, output to the library content, etc. as needed, which is similar to the content in the previous embodiment and will not be repeated here.

The present invention enables users to implement a full set of development processes for data analysis on a big data platform. Users can create and edit data tasks, then enter the scheduling task page to reference previously created tasks, configure the operating cycle and first operating time, and implement periodic/non-periodic operation of tasks; the operating status of all scheduled tasks can be seen on the scheduling task list page; click on a scheduled task to enter the scheduling task details page, where all target data task cards can be seen, and the execution order between target data tasks can be adjusted by dragging the task cards, and the target data task card can be clicked to directly jump to the task details page for viewing, editing, manual operation, and viewing logs, etc. This greatly facilitates the process of data analysis for users, lowers the threshold for users to develop big data, and the visual method is easier to maintain and adjust, giving users a better sense of use.

FIG13 is a schematic diagram of a data processing device according to an embodiment of the present invention. As shown in FIG13 , the data processing device provided by the present invention is applied to a big data platform, and the big data platform includes at least two different data sources; the device includes:

The acquisition unit 1301 is used to display a data processing task page, and acquire an SQL associated query statement in response to an input operation of a user on the data processing task page, wherein the SQL associated query statement includes data source information of multiple first data tables to be processed, wherein at least two of the multiple first data tables to be processed are from different data sources;

The conversion unit 1302 is used to obtain a second data table to be processed according to the data source information, replace the data source information of the first data table to be processed in the SQL association query statement with data table information corresponding to the second data table to be processed, and obtain an executable SQL query statement;

The execution unit 1303 is used to execute the executable SQL query statement to obtain the query result.

The specific contents executed by the above-mentioned units can be found in the above-mentioned embodiments, which will not be described again here.

The present invention also provides a task scheduling device, which is applied to a big data platform, and the device comprises:

A creation unit, used to display a task scheduling page and create a scheduling task based on the user's input operation on the task scheduling page;

a determining unit, configured to determine a plurality of target data tasks of the scheduling task from a plurality of pre-created candidate data tasks based on a selection operation of a user, wherein the candidate data tasks include a data processing task, and the data processing task is implemented by using the data processing method described in the above embodiment;

A display unit, used to display icons of each target data task on the task scheduling page;

A moving unit, used to determine the execution order of each target data task in the scheduling task based on the user's moving operation on each icon;

The scheduling unit is used to execute each target data task in the scheduled task according to the execution order.

The device implementing this embodiment has a display device, and the display device can be: electronic paper, mobile phone, tablet computer, television, notebook computer, digital photo frame, navigator, or any other product or component with a display function.

It should be noted that in the accompanying drawings, the sizes of layers and regions may be exaggerated for clarity of illustration. It is also understood that when an element or layer is referred to as being "on" another element or layer, it may be directly on the other element, or there may be an intermediate layer. In addition, it is understood that when an element or layer is referred to as being "under" another element or layer, it may be directly under the other element, or there may be more than one intermediate layer or element. In addition, it is also understood that when a layer or element is referred to as being "between" two layers or two elements, it may be the only layer between the two layers or two elements, or there may also be more than one intermediate layer or element. Similar reference numerals throughout the text indicate similar elements.

In the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.

Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the disclosure disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

It should be understood that the present invention is not limited to the precise details described above and shown in the accompanying drawings. The present invention is structured and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims (11)

A data processing method, characterized in that it is applied to a big data platform, wherein the big data platform includes at least two different data sources; the method comprises: Displaying a data processing task page, and acquiring an SQL associated query statement in response to an input operation of a user on the data processing task page, wherein the SQL associated query statement includes data source information of a plurality of first data tables to be processed, wherein at least two of the plurality of first data tables to be processed are from different data sources; Acquire a second table of data to be processed according to the data source information; The data source information of the first data table to be processed in the SQL association query statement is replaced with the data table information corresponding to the second data table to be processed, so as to obtain an executable SQL query statement; Execute the executable SQL query statement to obtain the query result. The method according to claim 1 is characterized in that the data processing method is implemented based on the PySpark framework. The method according to claim 1, characterized in that after obtaining the query result, the method further comprises: Determining a processing condition for processing the query result based on the user's input operation; Processing the query result based on the processing condition to obtain a processing result; A column is added to the query result, and the processing result is displayed in the column. The method according to claim 1, characterized in that after obtaining the query result, the method further comprises: Based on the configuration operation of the user, determine the target database for storing the query results and the name of the data table for storing the query results; If the target database does not include a data table corresponding to the data table name, create a data table corresponding to the data table name; If the target database includes a data table corresponding to the data table name, the output mode is determined according to the user's selection operation, and the query result is written to the target database in the output mode. In the data table of the target database. A task scheduling method, characterized in that it is applied to a big data platform, and the method comprises: Displaying a task scheduling page, and creating a scheduling task based on the user's input operation on the task scheduling page; Determining multiple target data tasks of the scheduling task from multiple pre-created candidate data tasks based on a user's selection operation, wherein the candidate data tasks include data processing tasks, and the data processing tasks are implemented using the data processing method described in any one of claims 1 to 4; Displaying icons of each target data task on the task scheduling page; Determining the execution order of each target data task in the scheduling task based on the user's movement operation on each icon; Execute each target data task in the scheduled task according to the execution order. The method according to claim 5, characterized in that the candidate data task also includes a data integration task; the method further includes: Based on the user's operation information on the data integration page, determine the data integration task name, data source information, and data destination information; A data integration task corresponding to the data integration task name is created according to the data source information and the data destination information. The method according to claim 5, characterized in that the method further comprises: Based on the user's selection operation of the icon, displaying the development page of the target data task corresponding to the icon; In response to detecting a modification operation of the user on the development page, the modified target data task is saved. The method according to claim 5, characterized in that the input operation includes the first running time of the scheduling task; The executing each target data task in the scheduled task according to the execution order includes: In response to the current time reaching the first running time of the scheduled task, the execution Execute each target data task in the scheduling task in sequence, and obtain the running status of each target data task; If the running state is an error state, stopping the execution of the scheduled task and returning the state of the scheduled task to an error state; If all target data tasks are executed in the execution order of the target data tasks, the status of the scheduled task is returned as completed. The method according to claim 5, characterized in that the method further comprises: In the scheduling task template page, the corresponding scheduling task template is displayed based on the user's selection operation; Obtaining user configuration information for the scheduling task template; A scheduling task matching the configuration information is generated based on the scheduling task template. A data processing device, characterized in that it is applied to a big data platform, wherein the big data platform includes at least two different data sources; the device comprises: an acquisition unit, configured to display a data processing task page, and acquire an SQL associated query statement in response to an input operation of a user on the data processing task page, wherein the SQL associated query statement includes data source information of a plurality of first data tables to be processed, wherein at least two of the plurality of first data tables to be processed are from different data sources; a conversion unit, configured to obtain a second data table to be processed according to the data source information, replace the data source information of the first data table to be processed in the SQL association query statement with data table information corresponding to the second data table to be processed, and obtain an executable SQL query statement; The execution unit is used to execute the executable SQL query statement to obtain the query result. A computer-readable storage medium, characterized in that when an executable computer program in the storage medium is executed by a processor, it can implement the method as claimed in any one of claims 1 to 4 or claims 5 to 9.