CN116166634A - Data blood relationship graph construction method and device, storage medium and electronic equipment - Google Patents

Abstract

The present disclosure relates to a method, device, storage medium, and electronic device for constructing a data kinship graph, wherein the method includes: acquiring task component information, the task component information including the execution sequence of multiple task components and the execution order of the multiple task components Configuration information; based on the execution sequence, determine the field association relationship corresponding to the task component according to the configuration information of each task component in turn, and the field association relationship is used to represent the relationship between the field information of the source table and the field information of the target table Corresponding relationship, the target table field information is obtained after the task component processes the source table field information; according to each of the field correlations, a data blood relationship diagram is determined. By constructing the data kinship diagram through the above method, we can intuitively see the corresponding relationship between databases and databases, data tables and data tables, and/or data fields and data fields, so as to provide support for the later construction and maintenance of the system.

Description

Method, device, storage medium and electronic equipment for constructing data kinship graph

technical field

The present disclosure relates to the technical field of data processing, and in particular, relates to a method, device, storage medium, and electronic device for constructing a data kinship relationship graph.

Background technique

In the actual data development and governance process, it is necessary to extract data from different data sources, and process and/or format the extracted data according to certain data processing rules to obtain target data tables or target files containing target data, etc. , and finally load the table data required by the user.

With the expansion of enterprise business, the number of data tables in the system continues to increase, and it is difficult for development and management personnel to be familiar with the relationship between databases and databases, data tables and data tables, and data fields and data fields, which is not conducive to the later construction and maintenance of the system.

Contents of the invention

The purpose of the present disclosure is to provide a method, device, storage medium and electronic equipment for constructing a data kinship graph, so as to solve the above-mentioned technical problems.

In order to achieve the above purpose, the first aspect of the present disclosure provides a method for constructing a data kinship relationship graph, the method comprising:

Obtaining task component information, the task component information including the execution sequence of multiple task components and configuration information of the multiple task components;

Based on the execution sequence, determine the field association relationship corresponding to the task component according to the configuration information of each task component in turn, and the field association relationship is used to represent the correspondence between the field information of the source table and the field information of the target table , the target table field information is obtained after the task component processes the source table field information;

According to the association relationship of each of the fields, a data blood relationship diagram is determined.

Optionally, the determining the field association relationship corresponding to the task component according to the configuration information of each task component includes:

For the configuration information of each of the described task components, perform the following steps:

Determine a source table field information identifier, a target table field information identifier, and a correspondence between the source table field information identifier and the target table field information identifier according to the configuration information;

The field information identifier of the source table and the field information identifier of the target table are bound according to the corresponding relationship to obtain the field association relationship.

Optionally, the determining the source table field information identifier according to the configuration information includes:

When the configuration information includes source database information, determine source table field information and a source table identifier according to the source database information, and determine the source table field information identifier according to the source table field information and the source table identifier;

When the configuration information includes a structured query statement and does not include source library information, parse the structured query statement to obtain source table field information and source table name, determine the source table identifier according to the source table name, and Determine the source table field information identifier according to the source table field information and the source table identifier.

Optionally, the parsing of the structured query statement to obtain source table field information and source table name includes:

Performing statement analysis on the structured query statement to obtain a statement analysis result;

determining the statement type of the structured query statement according to the statement parsing result;

A statement parsing strategy is determined according to the statement type, and the structured query statement is parsed according to the statement parsing strategy to obtain field information of the source table and a table name of the source table.

Optionally, when the statement type is a selected statement type, parsing the structured query statement according to the statement parsing strategy to obtain the source table field information and the source table name includes:

determining an operation instruction according to the structured query statement;

When the operation instruction is a combined operation instruction, split the structured query statement to obtain a target structured query statement whose operation instruction is a selected operation instruction, and determine the source table according to the target structured query statement Field information and the name of the source table;

When the operation instruction is a selection operation instruction, the source table field information and the source table name are determined according to the structured query statement.

Optionally, when the statement type is an embedded statement type or a created statement type, the structured query statement is parsed according to the statement parsing strategy to obtain the source table field information and the source table name, include:

Determine whether there is a selection statement in the structured query statement;

When the selection statement does not exist in the structured query statement, determine the source table field information and the source table name according to the structured query statement;

When the selection statement exists in the structured query statement, determine an operation instruction according to the structured query statement;

When the operation instruction is a combined operation instruction, split the structured query statement to obtain a target structured query statement whose operation instruction is a selected operation instruction, and determine the source table according to the target structured query statement Field information and the name of the source table;

When the operation instruction is a selection operation instruction, the source table field information and the source table name are determined according to the structured query statement.

Optionally, when the configuration information includes information indicating that the source table field information does not support query through a database connection protocol, the determining the source table field information identifier according to the configuration information includes:

determining the task type of the task component according to the configuration information;

When the task type is a dictionary table loading task, determine the source table field information and source table identifier from the target database according to the configuration information, and determine the source table according to the source table field information and the source table identifier Field information identification;

When the task type is a conversion task, determine a structured query statement according to the configuration information, and parse the structured query statement to obtain a source table field and a source table name;

Determine the source table identifier according to the source table name, and determine the field type of the source table field in the parsed data according to the source table identifier and the source table field;

The source table field information is determined according to the source table field and the field type, and the source table field information identifier is determined according to the source table field information and the source table identifier.

Optionally, the determining the data kinship diagram according to each of the field associations includes:

For each field association relationship, determine the corresponding source table field information identifier and the target table field information identifier, and determine the source table identifier according to the source table field information identifier, and determine the source table identifier according to the target table field information identifier Determine the target table ID;

Using the source table field information identifier with the same source table identifier as the first data, and the corresponding source table identifier as the first table name, construct a first data table, and use the same target table identifier as the first data table The field information identifier of the target table is used as the second data, and the corresponding target table identifier is used as the second table name to construct the second data table;

Binding the first data in the first data table and the second data in the second data table according to the field association relationship to obtain the data blood relationship graph.

The second aspect of the present disclosure provides a device for constructing a data kinship relationship graph, the device comprising:

An acquisition module, configured to acquire task component information, the task component information including the execution order of multiple task components and configuration information of the multiple task components;

The first determination module is configured to determine the field association relationship corresponding to the task component according to the configuration information of each task component in turn based on the execution order, and the field association relationship is used to represent the field information of the source table and the target table Correspondence between field information, the target table field information is obtained after the task component processes the source table field information;

The second determining module is configured to determine a data blood relationship diagram according to each of the field association relationships.

A third aspect of the present disclosure provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

A fourth aspect of the present disclosure provides an electronic device, including:

a memory on which a computer program is stored;

A processor, configured to execute the computer program in the memory, so as to implement the steps of any one of the methods in the first aspect.

Through the above technical solution, the field association relationship between the field information of the source table and the field information of the target table is extracted, and the data blood relationship diagram is constructed according to the field association relationship. /or the corresponding relationship between data fields and data fields provides support for the later construction and maintenance of the system; on the other hand, the database/data table involved in the data development process can be visualized through the data blood relationship diagram, which can be seen at a glance See the source of information for each data field in the data kinship diagram, no longer need to consult relevant data developers, and reduce the project's dependence on personnel.

Other features and advantages of the present disclosure will be described in detail in the detailed description that follows.

Description of drawings

The accompanying drawings are used to provide a further understanding of the present disclosure, and constitute a part of the description, together with the following specific embodiments, are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the attached picture:

Fig. 1 is a flow chart showing a method for constructing a data kinship relationship graph according to an exemplary embodiment of the present disclosure;

Fig. 2 is a schematic diagram of a data kinship relationship diagram according to an exemplary embodiment of the present disclosure;

Fig. 3 is a structural block diagram of a device for constructing a data kinship relationship graph according to an exemplary embodiment of the present disclosure;

Fig. 4 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method implementations of the present disclosure may be executed in different orders, and/or executed in parallel. Additionally, method embodiments may include additional steps and/or omit performing illustrated steps. The scope of the present disclosure is not limited in this respect.

As used herein, the term "comprise" and its variations are open-ended, ie "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one further embodiment"; the term "some embodiments" means "at least some embodiments." Relevant definitions of other terms will be given in the description below.

It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more" Multiple".

First, the application scenarios of the present disclosure are described. In the process of data development and management, it is necessary to extract data from different data sources, and process and/or format the extracted data according to certain data processing rules to obtain target data. Then write or transmit the target data to the target data table or target file to obtain the target data table or target file containing the target data. Finally, the table data required by the user is formed by loading the target data table or target file. However, with the expansion of enterprise business, the number of data tables in the system continues to increase, and it is difficult for developers and managers to be familiar with the relationship between databases, data tables and data tables, and/or data fields and data fields, which is not conducive to system development. Later construction and maintenance.

In view of this, an embodiment of the present disclosure provides a method, device, storage medium, and electronic device for constructing a data lineage relationship graph, by extracting the field association relationship between the field information of the source table and the field information of the target table, and constructing the data according to the field association relationship The data blood relationship diagram can intuitively see the corresponding relationship between databases and databases, data tables and data tables, and/or data fields and data fields, so as to provide support for the later construction and maintenance of the system.

The embodiments of the present disclosure will be further explained below in conjunction with the accompanying drawings.

Fig. 1 is a flow chart of a method for constructing a data kinship relationship graph according to an exemplary embodiment of the present disclosure. Referring to Fig. 1 , the method for constructing a data kinship relationship graph may include the following steps:

S101. Obtain task component information, where the task component information includes an execution sequence of multiple task components and configuration information of the multiple task components.

It should be understood that a task component is the smallest unit of a task, and each task component is used to complete a step or a function of the task. When a task includes multiple task components, the multiple task components can be sorted according to the order in which the task processes data, so as to obtain the execution order of the multiple task components. When multiple tasks are included, the execution sequence of the multiple tasks may be determined first, and then the execution sequence of the task components of each task may be sequentially determined according to the execution sequence of the multiple tasks.

In addition, it should be understood that the configuration information of each task component may be configured according to the function of the task component, which is not limited in this embodiment of the present disclosure. In a possible implementation, the configuration information of each task component may include the source table field information that the task component needs to process, the reading location of the source table field information, the storage location of the target table field information, and the like. For example, the configuration information may include source library information (source library address, source library name and/or source table name), structured query statement and/or target library information (target library address, target library name and/or target table name) name).

S102. Based on the execution sequence, determine the field association relationship corresponding to the task component according to the configuration information of each task component in sequence, and the field association relationship is used to represent the relationship between the field information of the source table and the field information of the target table. Corresponding relationship, the target table field information is obtained after the task component processes the source table field information.

In a possible implementation manner, the determining the field association relationship corresponding to the task component according to the configuration information of each task component may include:

For the configuration information of each of the described task components, perform the following steps:

Determine the source table field information identifier, the target table field information identifier, and the correspondence between the source table field information identifier and the target table field information identifier according to the configuration information; The information identifier is bound with the field information identifier of the target table to obtain the field association relationship.

It should be understood that the source table field information processed by each task component is generally different, and each source table field information may include more information, for example, source table fields, source table field types, and so on. Correspondingly, the target table field information will also include corresponding information, that is, the target table field, the target table field type, and the like. If the field association relationship is generated directly according to the field information of the source table and the field information of the target table, more data information needs to be stored. On the one hand, it greatly occupies the storage space of the system; on the other hand, due to the complex data structure of a field association relationship, the data structure is easily damaged in the subsequent storage or reading process. In view of this, this embodiment generates a corresponding identifier for each source table field information and each target table field information, that is, the source table field identifier and the target table field identifier, and the corresponding source table field identifier and target table field The identifier is bound to form a field association relationship. Compared with directly generating the field association relationship based on the field information of the source table and the field information of the target table, the method in the embodiment of the present application can greatly simplify the data structure of the field association relationship, thereby overcoming the above-mentioned problems.

Schematically, if the source table field information includes source table fields and source table field types, the target table field information includes target table fields and target table field types. If the field association relationship is generated directly according to the field information of the source table and the field information of the target table, the field association relationship can be expressed as: (source table field + source table field type) & (target table field + target table field type). If the field association relationship is generated according to the field information identifier of the source table and the field information identifier of the target table, the field association relationship can be expressed as: the field information identifier of the source table & the field information identifier of the target table.

Wherein, it is worth noting that the representation of the above-mentioned field association relationship is only for illustration, and does not constitute a limitation to this solution. For example, in a possible implementation manner, the field association relationship may also be expressed as: a source table field information identifier and a target table field information identifier.

In a possible implementation manner, the determining the field information identifier of the source table according to the configuration information may include:

When the configuration information includes source database information, determine source table field information and a source table identifier according to the source database information, and determine the source table field information identifier according to the source table field information and the source table identifier; When the configuration information includes a structured query statement and does not include source library information, parse the structured query statement to obtain source table field information and source table name, determine the source table identifier according to the source table name, and Determine the source table field information identifier according to the source table field information and the source table identifier.

It should be understood that the corresponding field information of the source database can be obtained according to the source database information and the structured query statement (Structured Query Language, SQL). The difference is that the complete source table data (including required source database field information and unnecessary source database field information) can be obtained according to the source database information, but according to the structured query statement, only the structured query statement can be obtained The declared source library field information and source table name. Whether the configuration information specifically configures source database information or structured query statements is set according to actual conditions, and the embodiments of the present disclosure do not impose any restrictions on this.

In addition, it should be understood that field information of different source tables may come from the same source table, or may come from different source tables. In order to confirm the corresponding relationship between data tables and data tables according to the data blood relationship diagram, after determining the field information of each source table, you can first bind the field information of the source table with the name of the source table, and then in After binding, the corresponding source table field information identifier is generated. At the same time, in order to simplify the binding relationship between the source table field information and the source table name, this embodiment first generates the source table identifier, then binds the source table identifier and the source table field information, and finally generates the corresponding Source table field information identifier. That is, the field association relationship can be expressed as: (source table identifier+source table field information identifier)&(target table identifier+target table field information identifier).

Wherein, it is worth noting that the manner of generating the source table identifier may be set according to actual conditions, and this embodiment of the present disclosure does not impose any limitation on this. In a possible implementation, it can be obtained by encrypting the source library address, source library name, and source table name; it can also be obtained by properly encrypting the source table name; it is also possible to pre-set the corresponding source table for the source table logo.

Schematically, when the configuration information includes source database information, first access the source table according to the source database information, and obtain the corresponding source table field information from the source table according to the data that needs to be processed according to the task component; then source database information Encrypt to obtain the source table identifier; finally, bind the source table identifier and source table field information and perform encryption processing to obtain the source table field information identifier.

When the configuration information does not include source database information, first obtain the corresponding source table data field information and source table name by parsing the data query statement; then encrypt the source table name to obtain the source table ID; finally, source table ID Bind with the field information of the source table and perform encryption processing to obtain the field information identifier of the source table.

In a possible implementation manner, the parsing the structured query statement to obtain the field information of the source table and the name of the source table may include:

Perform statement analysis on the structured query statement to obtain a statement analysis result; determine the statement type of the structured query statement according to the statement analysis result; determine a statement analysis strategy according to the statement type, and determine the statement analysis strategy according to the statement analysis strategy The structured query statement is parsed to obtain the field information of the source table and the name of the source table.

Wherein, performing sentence parsing on a structured query statement to obtain a statement parsing result may be implemented based on a structured query sentence parsing plug-in in related technologies, which will not be described in detail in the embodiments of the present disclosure.

It should be understood that, the structured query statement is parsed by the structured query statement parsing plug-in, and the obtained statement analysis result is the data structure of the structured query statement. Due to the complex data structure of the structured query statement, it is impossible to directly obtain the data field information of the source table and the name of the source table. Therefore, further judgment can be made on the analysis result of the structured query statement to determine the statement type of the structured query statement, so as to adopt different analysis strategies for different statement types to obtain the field information of the source table and the source table. table name.

In a possible implementation manner, when the statement type is a selected statement type, the structured query statement is parsed according to the statement parsing strategy to obtain the field information of the source table and the name of the source table, which may be include:

Determine the operation instruction according to the structured query statement; when the operation instruction is a joint operation instruction, split the structured query statement to obtain the target structured query statement that the operation instruction is the selected operation instruction, and according to the specified The target structured query statement determines the source table field information and the source table name; when the operation instruction is a selection operation instruction, determine the source table field information and the source table field information according to the structured query statement table name.

In a possible implementation manner, when the statement type is an embedded statement type or a created statement type, the structured query statement is parsed according to the statement parsing strategy to obtain field information of the source table and the source table Table name, which can include:

Determine whether there is a selection statement in the structured query statement; when the selection statement does not exist in the structured query statement, determine the source table field information and the source table name according to the structured query statement ; When the selection statement exists in the structured query statement, determine the operation instruction according to the structured query statement; when the operation instruction is a joint operation instruction, split the structured query statement to obtain The operation instruction is to select the target structured query statement of the operation instruction, and determine the field information of the source table and the table name of the source table according to the target structured query statement; when the operation instruction is the selection operation instruction, according to the The structured query statement determines the field information of the source table and the name of the source table.

In this embodiment, by layering and recursively parsing the structured query statement layer by layer, the attribution relationship between the underlying entity table and its fields can be obtained, and the field information of the source table can be obtained by recording the corresponding relationship between layers and fields between layers The corresponding relationship with the underlying entity table.

In addition, it is worth noting that there are various types of databases. Some databases can directly obtain the data in the database according to the database connection protocol, while some databases do not support obtaining the data in the database through the database connection protocol. For example, the Apache open source Two large data computing frameworks FlinkSql and SparkSql. Therefore, in order to obtain the source table field information identifier from FlinkSql and/or SparkSql, the source table field can be determined by parsing the structured query statement or from the target database (data source in FlinkSql and/or SparkSql). That is, according to an embodiment of the present disclosure, when the configuration information includes information indicating that the source table field information does not support query through a database connection protocol, the determination of the source table field information identifier according to the configuration information, Can include:

Determine the task type of the task component according to the configuration information; when the task type is a dictionary table loading task, determine the source table field information and the source table identifier from the target database according to the configuration information, and according to the source The table field information and the source table identifier determine the source table field information identifier; when the task type is a conversion task, determine the structured query statement according to the configuration information, and parse the structured query statement to obtain the source table field and source table name; determine the source table identifier according to the source table name, and determine the field type of the source table field in the parsed data according to the source table identifier and the source table field; according to The source table field and the field type determine the source table field information, and determine the source table field information identifier according to the source table field information and the source table identifier.

S103. Determine a data kinship relationship diagram according to each of the field association relationships.

In a possible implementation manner, the determining the data kinship relationship diagram according to each of the field association relationships may include:

For each field association relationship, determine the corresponding source table field information identifier and the target table field information identifier, and determine the source table identifier according to the source table field information identifier, and determine the source table identifier according to the target table field information identifier Determine the target table identifier; use the source table field information identifier with the same source table identifier as the first data, and the corresponding source table identifier as the first table name, construct the first data table, and use the same source table identifier as the first table name The target table field information identification of the target table identification is used as the second data, and the corresponding target table identification is used as the second table name to construct the second data table; according to the field association relationship in the first data table Bind the first data in the second data table with the second data in the second data table to obtain the blood relationship graph of the data.

It should be understood that after all task components are executed, multiple field associations will be generated, and some field associations represent the correspondence between different data fields in the same source table and the same target table. For example, the first field association is the corresponding relationship between the first source table field information in the source table A and the first target table field information in the target table A; the second field association is the second source table field in the source table A The corresponding relationship between the information and the field information of the second target table in the target table A. Therefore, by classifying the source table identifier and the target table identifier, the source table field information identifiers from the same source table are classified to form the first data table, and the target table field information identifiers from the same target table are classified, Form the second data table, and then bind the field information identifier of the source table in the first data table and the field information identifier of the target table in the second data table according to the field association relationship, so as to obtain the The data kinship diagram of the corresponding relationship between tables, data fields and data fields is shown in Figure 2.

In a possible implementation, in order to see the correspondence between data tables and data tables and/or data fields and data fields more intuitively, when generating the data kinship diagram, the The information ID, target table ID, and target table field information ID respectively determine the source table name, source table field information, target table ID, and target table field information, and then classify by source table name and target table name, that is , classify the field information of the source table from the same source table into the first data table, classify the field information of the target table from the same target table into the second data table, and finally classify the field information in the first data table according to the field association relationship Binding the field information of the source table and the field information of the target table in the second data table, so as to obtain a data blood relationship diagram that can intuitively see the corresponding relationship between data tables and data tables, and data fields.

To sum up, through the above technical solutions, the field association relationship between the field information of the source table and the field information of the target table is extracted, and the data blood relationship diagram is constructed according to the field association relationship. The corresponding relationship between data tables and/or data fields and data fields provides support for the later construction and maintenance of the system; on the other hand, the database/data tables involved in the data development process can be visualized through the data blood relationship diagram, It is possible to see at a glance the source of information for each data field in the data kinship diagram, and it is no longer necessary to consult relevant data developers, reducing the project's dependence on personnel.

Based on the same idea, an embodiment of the present disclosure also provides a device for constructing a data blood relationship graph. As shown in FIG. 3 , the data blood relationship graph construction device 300 may include:

An acquisition module 310, configured to acquire task component information, the task component information including the execution sequence of multiple task components and configuration information of the multiple task components;

The first determination module 320 is configured to determine the field association relationship corresponding to the task component according to the configuration information of each task component in sequence based on the execution order, and the field association relationship is used to represent the source table field information and the target Correspondence between table field information, the target table field information is obtained after the task component processes the source table field information;

The second determination module 330 is configured to determine a data blood relationship diagram according to each field association relationship.

Optionally, the first determining module 320 may include:

A first determining submodule, configured to determine a source table field information identifier, a target table field information identifier, and a correspondence between the source table field information identifier and the target table field information identifier according to the configuration information;

The first binding submodule is configured to bind the source table field information identifier and the target table field information identifier according to the corresponding relationship to obtain the field association relationship.

Optionally, the first determining submodule may include:

A first determining unit, configured to determine source table field information and a source table identifier according to the source database information when the configuration information includes source library information, and determine the source table field information and the source table identifier according to the source table field information and the source table identifier The field information identifier of the source table;

The second determining unit is configured to parse the structured query statement to obtain source table field information and source table name when the configuration information includes a structured query statement and does not include source database information, according to the source table The source table identifier is determined according to the source table name, and the source table field information identifier is determined according to the source table field information and the source table identifier.

Optionally, the second determining unit may include:

The first parsing subunit is configured to perform sentence parsing on the structured query sentence to obtain a sentence parsing result;

A first determining subunit, configured to determine the statement type of the structured query statement according to the statement analysis result;

The second parsing subunit is configured to determine a sentence parsing strategy according to the sentence type, and parse the structured query sentence according to the sentence parsing strategy to obtain the field information of the source table and the name of the source table.

Optionally, when the sentence type is a selected sentence type, the second parsing subunit may include:

A first determining component, configured to determine an operation instruction according to the structured query statement;

The second determining component is configured to split the structured query statement when the operation instruction is a combined operation instruction, obtain a target structured query statement whose operation instruction is a selected operation instruction, and structure the query according to the target The query statement determines the field information of the source table and the name of the source table;

The third determining component is configured to determine the source table field information and the source table name according to the structured query statement when the operation instruction is a selection operation instruction.

Optionally, when the statement type is an embedded statement type or a created statement type, the second parsing subunit may include:

The fourth determination component is used to determine whether there is a selection statement in the structured query statement;

A fifth determining component, configured to determine the source table field information and the source table name according to the structured query statement when the selection statement does not exist in the structured query statement;

A sixth determining component, configured to determine an operation instruction according to the structured query statement when the selection statement exists in the structured query statement;

The seventh determination component is used to split the structured query statement when the operation instruction is a joint operation instruction, obtain the target structured query statement whose operation instruction is a selected operation instruction, and structure the query statement according to the target The query statement determines the field information of the source table and the name of the source table;

The eighth determining component is configured to determine the field information of the source table and the name of the source table according to the structured query statement when the operation instruction is a selection operation instruction.

Optionally, when the configuration information includes information indicating that the source table field information does not support query through a database connection protocol, the first determining submodule may further include:

a third determining unit, configured to determine the task type of the task component according to the configuration information;

The fourth determining unit is configured to determine the source table field information and the source table identifier from the target database according to the configuration information when the task type is a dictionary table loading task, and determine the source table field information and the source table identifier according to the source table field information and the source table The table identifier determines the field information identifier of the source table;

The fifth determining unit is configured to determine a structured query statement according to the configuration information when the task type is a conversion task, and parse the structured query statement to obtain source table fields and source table names;

The sixth determining unit is configured to determine the source table identifier according to the source table name, and determine the field type of the source table field in the parsed data according to the source table identifier and the source table field;

A seventh determining unit, configured to determine the source table field information according to the source table field and the field type, and determine the source table field information identifier according to the source table field information and the source table identifier.

Optionally, the second determining module 330 may include:

The second determining submodule is configured to determine the corresponding source table field information identifier and the target table field information identifier for each of the field association relationships, and determine the source table identifier according to the source table field information identifier, Determine the target table identifier according to the target table field information identifier;

Constructing a submodule for constructing a first data table with the field information identifier of the source table having the same source table identifier as the first data, and the corresponding source table identifier as the first table name, and constructing a first data table with the same The target table field information identifier of the target table identifier is used as second data, and the corresponding target table identifier is used as a second table name to construct a second data table;

The second binding submodule is configured to bind the first data in the first data table and the second data in the second data table according to the field association relationship, to obtain the blood relationship of the data relation chart.

Through the above device, the field association relationship between the field information of the source table and the field information of the target table is extracted, and the data blood relationship diagram is constructed according to the field association relationship. Or the corresponding relationship between data fields and data fields provides support for the later construction and maintenance of the system; on the other hand, the database/data table involved in the data development process can be visualized through the data blood relationship diagram, which can be seen at a glance The source of information for each data field in the data blood relationship diagram is displayed, and it is no longer necessary to consult relevant data developers, reducing the project's dependence on personnel.

Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

Fig. 4 is a block diagram of an electronic device according to an exemplary embodiment. As shown in FIG. 4 , the electronic device 400 may include: a processor 401 and a memory 402 . The electronic device 400 may also include one or more of a multimedia component 403 , an input/output (I/O) interface 404 , and a communication component 405 .

Wherein, the processor 401 is used to control the overall operation of the electronic device 400, so as to complete all or part of the steps in the above-mentioned method for constructing the data blood relationship graph. The memory 402 is used to store various types of data to support the operation of the electronic device 400 , such data may include instructions for any application or method operating on the electronic device 400 , as well as application-related data. The memory 402 can be implemented by any type of volatile or non-volatile storage device or their combination, such as Static Random Access Memory (Static Random Access Memory, referred to as SRAM), Electrically Erasable Programmable Read-Only Memory (EPROM) Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), Erasable Programmable Read-Only Memory (Erasable Programmable Read-Only Memory, referred to as EPROM), Programmable Read-Only Memory (Programmable Read-Only Memory, referred to as PROM), read-only Memory (Read-Only Memory, ROM for short), magnetic memory, flash memory, magnetic disk or optical disk. Multimedia components 403 may include screen and audio components. The screen can be, for example, a touch screen, and the audio component is used for outputting and/or inputting audio signals. For example, an audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in memory 402 or sent via communication component 405 . The audio component also includes at least one speaker for outputting audio signals. The I/O interface 404 provides an interface between the processor 401 and other interface modules, which may be a keyboard, a mouse, buttons, and the like. These buttons can be virtual buttons or physical buttons. The communication component 405 is used for wired or wireless communication between the electronic device 400 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC for short), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or one or more of them Combinations are not limited here. Therefore, correspondingly, the communication component 405 may include: a Wi-Fi module, a Bluetooth module, an NFC module and the like.

In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), digital signal processors (Digital Signal Processor, DSP for short), digital signal processing equipment (Digital Signal Processing Device, referred to as DSPD), programmable logic device (Programmable Logic Device, referred to as PLD), field programmable gate array (Field Programmable Gate Array, referred to as FPGA), controller, microcontroller, microprocessor or other electronic components to achieve , which is used to implement the above-mentioned method for constructing a data kinship relationship graph.

In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned method for constructing a data blood relationship graph are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 402 including program instructions, and the above-mentioned program instructions can be executed by the processor 401 of the electronic device 400 to complete the above-mentioned method for constructing a data kinship graph.

In another exemplary embodiment, there is also provided a computer program product comprising a computer program executable by a programmable device, the computer program having a function for performing the above-mentioned The code part of the data lineage graph construction method.

The preferred embodiments of the present disclosure have been described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure. These simple modifications all belong to the protection scope of the present disclosure.

In addition, it should be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner if there is no contradiction. The combination method will not be described separately.

In addition, various implementations of the present disclosure can be combined arbitrarily, as long as they do not violate the idea of the present disclosure, they should also be regarded as the content disclosed in the present disclosure.

Claims (11)

1. A method for constructing a data relationship diagram, characterized in that the method comprises: Obtaining task component information, the task component information including the execution sequence of multiple task components and configuration information of the multiple task components; Based on the execution sequence, determine the field association relationship corresponding to the task component according to the configuration information of each task component in turn, and the field association relationship is used to represent the correspondence between the field information of the source table and the field information of the target table , the target table field information is obtained after the task component processes the source table field information; According to the association relationship of each of the fields, a data blood relationship diagram is determined. 2. The method according to claim 1, wherein the determining the field association relationship corresponding to the task component according to the configuration information of each task component comprises: For the configuration information of each of the described task components, perform the following steps: Determine a source table field information identifier, a target table field information identifier, and a correspondence between the source table field information identifier and the target table field information identifier according to the configuration information; The field information identifier of the source table and the field information identifier of the target table are bound according to the corresponding relationship to obtain the field association relationship. 3. The method according to claim 2, wherein the determining the source table field information identifier according to the configuration information comprises: When the configuration information includes source database information, determine source table field information and a source table identifier according to the source database information, and determine the source table field information identifier according to the source table field information and the source table identifier; When the configuration information includes a structured query statement and does not include source library information, parse the structured query statement to obtain source table field information and source table name, determine the source table identifier according to the source table name, and Determine the source table field information identifier according to the source table field information and the source table identifier. 4. The method according to claim 3, wherein said parsing said structured query statement to obtain source table field information and source table name includes: Performing statement analysis on the structured query statement to obtain a statement analysis result; determining the statement type of the structured query statement according to the statement parsing result; A statement parsing strategy is determined according to the statement type, and the structured query statement is parsed according to the statement parsing strategy to obtain field information of the source table and a table name of the source table. 5. The method according to claim 4, wherein, when the statement type is a selection statement type, the structured query statement is analyzed according to the statement analysis strategy to obtain the source table field information and The name of the source table includes: determining an operation instruction according to the structured query statement; When the operation instruction is a combined operation instruction, split the structured query statement to obtain a target structured query statement whose operation instruction is a selected operation instruction, and determine the source table according to the target structured query statement Field information and the name of the source table; When the operation instruction is a selection operation instruction, the source table field information and the source table name are determined according to the structured query statement. 6. The method according to claim 4, wherein when the statement type is an embedded statement type or a created statement type, the structured query statement is parsed according to the statement analysis strategy to obtain the source Table field information and the name of the source table, including: Determine whether there is a selection statement in the structured query statement; When the selection statement does not exist in the structured query statement, determine the source table field information and the source table name according to the structured query statement; When the selection statement exists in the structured query statement, determine an operation instruction according to the structured query statement; When the operation instruction is a combined operation instruction, split the structured query statement to obtain a target structured query statement whose operation instruction is a selected operation instruction, and determine the source table according to the target structured query statement Field information and the name of the source table; When the operation instruction is a selection operation instruction, the source table field information and the source table name are determined according to the structured query statement. 7. The method according to claim 2, wherein when the configuration information includes information indicating that the field information of the source table does not support query through a database connection protocol, determining the source according to the configuration information Table field information identification, including: determining the task type of the task component according to the configuration information; When the task type is a dictionary table loading task, determine the source table field information and source table identifier from the target database according to the configuration information, and determine the source table according to the source table field information and the source table identifier Field information identification; When the task type is a conversion task, determine a structured query statement according to the configuration information, and parse the structured query statement to obtain a source table field and a source table name; Determine the source table identifier according to the source table name, and determine the field type of the source table field in the parsed data according to the source table identifier and the source table field; The source table field information is determined according to the source table field and the field type, and the source table field information identifier is determined according to the source table field information and the source table identifier. 8. The method according to any one of claims 1-7, wherein the determination of the data blood relationship diagram according to each of the field associations includes: For each field association relationship, determine the corresponding source table field information identifier and the target table field information identifier, and determine the source table identifier according to the source table field information identifier, and determine the source table identifier according to the target table field information identifier Determine the target table ID; Using the source table field information identifier with the same source table identifier as the first data, and the corresponding source table identifier as the first table name, construct a first data table, and use the same target table identifier as the first data table The field information identifier of the target table is used as the second data, and the corresponding target table identifier is used as the second table name to construct the second data table; Binding the first data in the first data table and the second data in the second data table according to the field association relationship to obtain the data blood relationship graph. 9. A data consanguinity graph construction device, characterized in that the device comprises: An acquisition module, configured to acquire task component information, the task component information including the execution order of multiple task components and configuration information of the multiple task components; The first determination module is configured to determine the field association relationship corresponding to the task component according to the configuration information of each task component in turn based on the execution order, and the field association relationship is used to represent the field information of the source table and the target table Correspondence between field information, the target table field information is obtained after the task component processes the source table field information; The second determining module is configured to determine a data blood relationship diagram according to each of the field association relationships. 10. A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that, when the program is executed by a processor, the steps of the method according to any one of claims 1-8 are implemented. 11. An electronic device, characterized in that it comprises: a memory on which a computer program is stored; A processor, configured to execute the computer program in the memory, so as to implement the steps of the method according to any one of claims 1-8.

Images