CN114925688A - Data processing method and device based on artificial intelligence, electronic equipment and medium - Google Patents

Abstract

The present application relates to the technical field of artificial intelligence, and provides a data processing method, device, electronic device and medium based on artificial intelligence. The method includes: acquiring a database script corresponding to the data to be processed; performing text recognition on the database script, and determining A plurality of databases corresponding to the database script; obtaining a data table corresponding to each database; determining a data link corresponding to each database based on a pre-built data map and the data table; determining a data link corresponding to each two databases A public link in the road; based on the public link, determine the target data link corresponding to the database script, and perform data processing based on the target data link. In the present application, the text recognition algorithm is used to find out some links with common dependencies in the data to be processed, so as to avoid repeated processing of the same data table, reduce the consumption of resources and improve the efficiency of data processing.

Description

Data processing method, device, electronic device and medium based on artificial intelligence

technical field

The present application relates to the technical field of artificial intelligence, in particular to a data processing method, device, electronic device and medium based on artificial intelligence.

Background technique

A data mart is a data cube that is used to meet the needs of a specific department or user, store it in a multi-dimensional manner, and then generate decision analysis. There are many types of dimensions and indicators in the data mart, all of which are processed through various data reports to provide services to upstream applications. The traditional data mart has many deficiencies in hierarchical division and data link sharing, and there is a serious problem of repeated processing. The same data table in the mart is repeatedly processed. A data table is repeatedly processed, which will lead to serious waste of memory, and serious irregularities in data flow and data dependencies.

At present, a data processing method is urgently needed to solve the problem of wasting resources in repeated data processing and reading.

SUMMARY OF THE INVENTION

In view of the above content, it is necessary to propose a data processing method, device, electronic device and medium based on artificial intelligence, by combining text recognition algorithm to find out some links that have common dependencies in the data to be processed, so as to avoid the same data table from being processed. Repeated processing reduces resource consumption and improves data processing efficiency.

In a first aspect, the present application provides a data processing method based on artificial intelligence, the method comprising:

Obtain the database script corresponding to the data to be processed;

Perform text recognition on the database script, and determine a plurality of databases corresponding to the database script;

Get the data table corresponding to each database;

Based on the pre-built data map and the data table, determine the data link corresponding to each database;

Determine the common link in the data link corresponding to each two databases;

Based on the public link, a target data link corresponding to the database script is determined, and data processing is performed based on the target data link.

According to an optional implementation manner of the present application, performing text recognition on the database script to determine multiple databases corresponding to the database script includes:

Perform word segmentation processing on the database script to obtain word segmentation results;

Perform keyword detection on the word segmentation result, and identify multiple databases corresponding to the database script from the word segmentation result.

According to an optional embodiment of the present application, the keyword detection is performed on the word segmentation result, and multiple databases corresponding to the database script are identified from the word segmentation result, including:

The location of the target feature word is determined from the word segmentation result by using a preset feature word database, the feature word database includes a plurality of feature words, and the feature word refers to the feature word of the database operation statement used by the database script;

Using keyword matching technology, the database library name is identified from the text content after the position of the target feature word in the word segmentation result;

According to the database library name, multiple databases corresponding to the database script are determined.

According to an optional implementation manner of the present application, the acquiring a data table corresponding to each database includes:

Obtain the database library name of the database, and determine the database library name as the first keyword;

determining a second keyword included in the database script according to the first keyword, where the second keyword is located after the first keyword;

A data table name is generated according to the second keyword, and a data table corresponding to the database is determined according to the data table name.

According to an optional implementation manner of the present application, the determining, based on the public link, the target data link corresponding to the database script includes:

determining multiple coincidence databases corresponding to the public link;

obtaining the data link corresponding to each of the overlapping databases;

Based on the common link, the data links corresponding to every two overlapping databases are combined to obtain a first data link;

Obtain a second data link according to a database not corresponding to the public link in the database script;

A target data link corresponding to the database script is determined according to the first data link and the second data link.

According to an optional embodiment of the present application, before determining the data link corresponding to each database based on the pre-built data map and the data table, the method further includes:

Get the history database script;

Text recognition is performed on the historical database script, and the data flow direction text is determined in the historical database script;

Determine the flow direction of the data table according to the data flow direction text;

Based on the flow direction of the data table, a data map is generated.

According to an optional implementation manner of the present application, the performing text recognition on the historical database script, and determining the data flow direction text in the historical database script includes:

Call the preset word segmentation toolkit to perform word segmentation processing on the historical database script, and obtain a word segmentation result;

From the word segmentation result, determine the position of the preset flow direction word in the historical database script;

According to the position of the preset flow direction word in the historical database script, the data flow direction text is extracted from the historical database script.

In a second aspect, the present application provides a data processing device based on artificial intelligence, the device comprising:

The script obtaining module is used to obtain the database script corresponding to the data to be processed;

a text recognition module, configured to perform text recognition on the database script, and determine a plurality of databases corresponding to the database script;

The data acquisition module is used to acquire the data table corresponding to each database;

a link determination module, configured to determine the data link corresponding to each database based on the pre-built data map and the data table;

The link identification module is used to determine the common link in the data link corresponding to each two databases;

A data processing module, configured to determine a target data link corresponding to the database script based on the public link, and perform data processing based on the target data link.

In a third aspect, the present application provides an electronic device, the electronic device includes a processor and a memory, and the processor is configured to implement the artificial intelligence-based data processing method when executing a computer program stored in the memory.

In a fourth aspect, the present application provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program implements the artificial intelligence-based data processing method when executed by a processor.

To sum up, the artificial intelligence-based data processing method, device, electronic device and medium described in this application identify the database script corresponding to the data to be processed based on the text recognition algorithm, and determine multiple databases corresponding to the database script. ; Then, based on the corresponding data table in the database and the pre-built data map, determine the data link corresponding to each database; then determine the public link in the data link corresponding to each two databases, and the public link is the existence of repeated processing Finally, based on the public link, determine the target data link corresponding to the database script, and perform data processing based on the target data link. Processing based on the target data link can avoid the same data table It is processed repeatedly, which reduces the consumption of resources and improves the efficiency of data processing.

Description of drawings

FIG. 1 is a flowchart of an artificial intelligence-based data processing method provided in Embodiment 1 of the present application.

FIG. 2 is a schematic diagram of a flow direction of a data table provided by an embodiment of the present application.

FIG. 3 is a schematic diagram of a data link provided by an embodiment of the present application.

FIG. 4 is a schematic diagram of a data link provided by an embodiment of the present application.

FIG. 5 is a schematic diagram of a data link provided by an embodiment of the present application.

FIG. 6 is a schematic diagram of a target link provided by an embodiment of the present application.

FIG. 7 is a structural diagram of an artificial intelligence-based data processing apparatus provided in Embodiment 2 of the present application.

FIG. 8 is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present application.

Detailed ways

In order to more clearly understand the above objects, features and advantages of the present application, the present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments may be combined with each other in the case of no conflict.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terminology used herein in the specification of the application is for the purpose of describing an example in an alternative embodiment only and is not intended to limit the application.

The artificial intelligence-based data processing method provided by the embodiments of the present application is executed by an electronic device, and correspondingly, the artificial intelligence-based data processing apparatus runs in the electronic device. The electronic devices may include mobile phones, tablet computers, notebook computers, desktop computers, personal digital assistants, wearable devices, and the like.

The embodiments of the present application can process data based on artificial intelligence technology, so as to improve the efficiency of data processing. Among them, artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. .

The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning/deep learning.

Example 1

FIG. 1 is a flowchart of an artificial intelligence-based data processing method provided in Embodiment 1 of the present application. The artificial intelligence-based data processing method specifically includes the following steps. According to different requirements, the sequence of the steps in the flowchart can be changed, and some can be omitted.

S11, acquiring a database script corresponding to the data to be processed.

The data to be processed corresponding to the data processing task is acquired, wherein the data to be processed may include a plurality of corresponding data tables in a plurality of databases. Obtain the database script corresponding to the data to be processed. The database script may be a Hibernate Query Language (Hibernate Query Language, HQL) script or the like.

S12: Perform text recognition on the database script to determine multiple databases corresponding to the database script.

The data in the database script comes from multiple databases, each database has a corresponding name and/or code, the database script can be identified based on a text recognition algorithm, the library name corresponding to the database script can be identified, and the database script can be identified. Multiple databases corresponding to the database script.

In an optional implementation manner, performing text recognition on the database script to determine multiple databases corresponding to the database script includes:

Perform word segmentation processing on the database script to obtain word segmentation results;

Perform keyword detection on the word segmentation result, and identify multiple databases corresponding to the database script from the word segmentation result.

Exemplarily, a preset word segmentation toolkit (such as a stammer toolkit) may be invoked to perform word segmentation processing on the database script, and a word segmentation result may be obtained. Using this method can improve the efficiency and accuracy of keyword recognition.

In an optional embodiment, the keyword detection is performed on the word segmentation result, and multiple databases corresponding to the database script are identified from the word segmentation result, including:

The location of the target feature word is determined from the word segmentation result by using a preset feature word database, the feature word database includes a plurality of feature words, and the feature word refers to the feature word of the database operation statement used by the database script;

Using keyword matching technology, the database library name is identified from the text content after the position of the target feature word in the word segmentation result;

According to the database library name, multiple databases corresponding to the database script are determined.

Wherein, the feature thesaurus may include a plurality of feature words. The characteristic word refers to the characteristic word of the database operation statement used by the database script, and the content of the characteristic word is not limited here. For example, the database operation statement is a structured query language (Structured Query Language, SQL) statement, and correspondingly, the characteristic words can be where, from, join and some symbols (eg $), and so on.

In order to identify the database library name included in the database operation text, a word segmentation process can be performed on the database script to obtain a word segmentation result. specifically. You can use a word segmentation toolkit, such as the stammer toolkit, to perform word segmentation processing on each SQL statement in the database script to obtain the word segmentation result. Suppose, the word segmentation result after word segmentation processing is performed on each SQL statement in the database script corresponding to the data processing task using the stutter toolkit, including the following content:

SelectA.name

, B.type

, C.group

From cx_XX_safe: .text_table b1, .text_table c1, .text_table e1.

SelectB.name

, B.type

, C.group

From cx_XXX_safe: text_table b1, .text_table c2, .text_table e2.

After the word segmentation result is obtained, keyword detection can be performed on the word segmentation result, and the database library name included in the database script can be identified from the word segmentation result. Specifically, a preset feature thesaurus can be used to determine the location of the target feature word from the word segmentation result, perform keyword matching on the target word segmentation, and identify the target feature word from the text content after the location of the target feature word in the word segmentation result. out the database name. For example, a preset feature thesaurus can be used to determine the location of From from the word segmentation result, and perform keyword matching on the target word segmentation to identify the library name cx_XX_safe from cx_XX_safe: text_table b1, .text_table c1, .text_table e1 , identify the library name cx_XXX_safe from cx_XXX_safetext_table b1, .text_table c2, .text_table e2.

After the database name is identified, multiple databases corresponding to the database script can be determined according to the database library name.

S13, obtain a data table corresponding to each database.

Each database corresponds to the corresponding data table of the database. After the database database name is identified, the table name following the database name can be extracted, and the data table corresponding to the database can be determined according to the table name.

In an optional implementation manner, the obtaining the data table corresponding to the database includes:

Obtain the database library name of the database, and determine the database library name as the first keyword;

determining a second keyword included in the database script according to the first keyword, where the second keyword is located after the first keyword;

A data table name is generated according to the second keyword, and a data table corresponding to the database is determined according to the data table name.

A keyword matching technique is performed on the word segmentation result, and after identifying the database name of the database from the text content after the target feature word in the word segmentation result, the database name is determined as the first keyword. According to a preset extraction rule, several word segmentations after the first keyword are extracted and determined as the second keyword. The data table name is determined according to the content of the second keyword, thereby obtaining the data table corresponding to the database. For example, the processor may determine the library name cx_XX_safe as the first keyword, and determine the extracted text table statement as the second keyword according to preset extraction rules (such as extracting text table statements), such as text_table b1, .text_table c1, .text_tablee1 are determined as the second keyword, and the data table name (text_table b1, .text_table c1, .text_table e1) is determined according to the content of the second keyword, so as to obtain the corresponding data table (b1, c1, .text_table e1) of the database. e1); the library name cx_XXX_safe can be determined as the first keyword, and the extracted text table statement can be determined as the second keyword according to the preset extraction rules (such as extracting text table statements), such as text_table b1, . text_table c2, .text_tablee2 are determined as the second keyword, and the data table name (text_table b1, .text_table c2, .text_table e2) is determined according to the content of the second keyword, so as to obtain the data table (b1, c2, e2) corresponding to the database ).

S14, based on the pre-built data map and the data table, determine the data link corresponding to each database.

There may be a data flow direction in every two data tables, and the data flow direction is used to indicate the source of the data in the data table. For example, data table A flows to data table B, indicating that data table A is the source of data table B, and data table B is processed on the basis of data table A. The data flow direction corresponding to the data table is recorded in the pre-built data map, and the data link corresponding to each database can be determined according to the data map and the data table. The data link is the processing flow of the data table when processing the to-be-processed data. According to the data link, the number of data tables to be processed when processing the data to be processed can be determined.

In an optional implementation manner, before determining the data link corresponding to each database based on the pre-built data map and the data table, the method further includes:

Get the history database script;

Text recognition is performed on the historical database script, and the data flow direction text is determined in the historical database script;

Determine the flow direction of the data table according to the data flow direction text;

Based on the flow direction of the data table, a data map is generated.

The instruction has a fixed instruction word, text recognition is performed on the historical database script, the instruction word corresponding to the data flow direction is identified to obtain the data flow direction text, and according to the data flow direction text, the flow direction of the data table is determined, and the data map is obtained.

In an optional implementation manner, the performing text recognition on the historical database script, and determining the data flow direction text in the historical database script includes:

Call the preset word segmentation toolkit to perform word segmentation processing on the historical database script, and obtain a word segmentation result;

From the word segmentation result, determine the position of the preset flow direction word in the historical database script;

According to the position of the preset flow direction word in the historical database script, the data flow direction text is extracted from the historical database script.

The preset flow direction word can be determined according to the language corresponding to the database operation statement used by the historical database script, and there is no restriction on the preset flow direction word here. For example, if the language corresponding to the database operation statement is an object-oriented query language (Hibernate Query Language, HQL), the preset flow words can be insert and select...from, and the historical database script can be written based on the preset flow direction words in the historical database. In the position, extract the sentence corresponding to the position as the data flow text.

For example, the data flow direction text is "insert overwrite table T2; select a, b from tableT1", and the data flow direction can be determined in the data flow direction text based on text recognition: table T1->table T2 (table T1 flows to table T2). If the data of table T1 is processed by table T0 through the above process, it will form table T0->tableT1 (table T0 flows to table T1), and finally form a data link: table T0->table T1->table T2 (table T0->table T1->table T2 (table T0 flows to table T1) T0 flows to table T1, which flows to table T2).

If there is also a table T3, as shown in Figure 2, the table T3 is obtained from the simultaneous association of T0 and T4.

At this time, from table T1, table T2 and table T3, a data map can be obtained. As shown in Figure 3, the data map includes three links: table T0->table T1->table T2 (table T0 flows to table T1, table T1 flows to table T2); table T0->table T3 (table T0 flows to table T3); table T4->table T3 (table T4 flows to table T3).

S15: Determine a common link in the data links corresponding to each of the two databases.

The common link is the overlapping link in the data links corresponding to the two databases.

Exemplarily, based on a pre-built data map, the data link corresponding to the database cx_XX_safe is determined, as shown in FIG. 4 ; the data link corresponding to the database cx_XXX_safe is determined, as shown in FIG. 5 .

According to the data link corresponding to the database cx_XX_safe and the data link corresponding to the database cx_XXX_safe, it is determined that there are two public links in these two databases: table a1>table b1 (table a1 flows to table b1); table d1->table b1 ( Table d1 flows to table b1).

S16, based on the public link, determine a target data link corresponding to the database script, and perform data processing based on the target data link.

Merging the data links of the two databases with the common link to obtain the target data link corresponding to the database script, and then processing the target data link, so as to avoid waste caused by processing the duplicate links, Reduce the consumption of CPU resources.

In an optional implementation manner, the determining, based on the public link, the target data link corresponding to the database script includes:

determining multiple coincidence databases corresponding to the public link;

obtaining the data link corresponding to each of the overlapping databases;

Based on the common link, the data links corresponding to every two overlapping databases are combined to obtain a first data link;

Obtain a second data link according to a database not corresponding to the public link in the database script;

A target data link corresponding to the database script is determined according to the first data link and the second data link.

Multiple overlapping databases refer to two or more overlapping databases. A common link can correspond to two databases or more than two databases. For the convenience of distinguishing the databases, the database corresponding to the public link is now called the coincidence database. A plurality of coincidence databases corresponding to each public link are determined, and data links corresponding to every two coincidence databases in the plurality of coincidence databases corresponding to the public link are combined to obtain a first data link.

For example, it is determined that the public link corresponds to two coincidence databases (cx_XX_safe, cx_XXX_safe). The public links corresponding to the two databases are shown in FIG. 4 and FIG. 5 . The corresponding data links are combined to obtain a first data link, as shown in FIG. 6 .

The second data link is obtained according to the database that does not contain the public link in the database script. The data link of the database that does not include the common link may be determined as the second data link.

The artificial intelligence-based data processing method described in the present application identifies a database script corresponding to the data to be processed based on a text recognition algorithm, and determines multiple databases corresponding to the database script; determine the data link corresponding to each database; then determine the public link in the data link corresponding to each two databases, and the public link is the link with repeated processing; finally, based on the public link, Determine the target data link corresponding to the database script, and perform data processing based on the target data link. Processing based on the target data link can avoid repeated processing of the same data table, reduce resource consumption, and improve Efficiency of data processing.

Embodiment 2

FIG. 7 is a structural diagram of an artificial intelligence-based data processing apparatus provided in Embodiment 2 of the present application.

In some embodiments, the artificial intelligence-based data processing apparatus 20 may include a plurality of functional modules composed of computer program segments. The computer program of each program segment in the data processing device 20 based on artificial intelligence can be stored in the memory of the electronic device and executed by at least one processor to execute (see description in FIG. 1 for details) the data based on artificial intelligence. The function of the processing method.

In this embodiment, the artificial intelligence-based data processing apparatus 20 can be divided into a plurality of functional modules according to the functions it performs. The functional modules may include: a script acquisition module 201 , a text recognition module 202 , a data acquisition module 203 , a link determination module 204 , a link recognition module 205 and a data processing module 206 . A module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, and are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

The script obtaining module 201 is used for obtaining the database script corresponding to the data to be processed.

The data to be processed corresponding to the data processing task is acquired, wherein the data to be processed may include a plurality of corresponding data tables in a plurality of databases. Obtain the database script corresponding to the data to be processed. The database script may be a Hibernate Query Language (Hibernate Query Language, HQL) script or the like.

The text recognition module 202 is configured to perform text recognition on the database script, and determine multiple databases corresponding to the database script.

The data in the database script comes from multiple databases, each database has a corresponding name and/or code, the database script can be identified based on a text recognition algorithm, the library name corresponding to the database script can be identified, and the database script can be identified. Multiple databases corresponding to the database script.

In an optional implementation manner, the text recognition module 202 performs text recognition on the database script, and determining multiple databases corresponding to the database script includes:

Perform word segmentation processing on the database script to obtain word segmentation results;

Perform keyword detection on the word segmentation result, and identify multiple databases corresponding to the database script from the word segmentation result.

Exemplarily, a preset word segmentation toolkit (such as a stammer toolkit) may be invoked to perform word segmentation processing on the database script, and a word segmentation result may be obtained. Using this method can improve the efficiency and accuracy of keyword recognition.

In an optional embodiment, the text recognition module 202 performs keyword detection on the word segmentation result, and identifies a plurality of databases corresponding to the database script from the word segmentation result, including:

The location of the target feature word is determined from the word segmentation result by using a preset feature word database, the feature word database includes a plurality of feature words, and the feature word refers to the feature word of the database operation statement used by the database script;

Using keyword matching technology, the database library name is identified from the text content after the position of the target feature word in the word segmentation result;

According to the database library name, multiple databases corresponding to the database script are determined.

Wherein, the feature thesaurus may include a plurality of feature words. The characteristic word refers to the characteristic word of the database operation statement used by the database script, and the content of the characteristic word is not limited here. For example, the database operation statement is a structured query language (Structured Query Language, SQL) statement, and correspondingly, the characteristic words can be where, from, join and some symbols (eg $), and so on.

In order to identify the database library name included in the database operation text, a word segmentation process can be performed on the database script to obtain a word segmentation result. specifically. You can use a word segmentation toolkit, such as the stammer toolkit, to perform word segmentation processing on each SQL statement in the database script to obtain the word segmentation result. Suppose, the word segmentation result after word segmentation processing is performed on each SQL statement in the database script corresponding to the data processing task using the stutter toolkit, including the following content:

SelectA.name

, B.type

, C.group

From cx_XX_safe: .text_table b1, .text_table c1, .text_table e1.

SelectB.name

, B.type

, C.group

From cx_XXX_safe: text_table b1, .text_table c2, .text_table e2.

After the word segmentation result is obtained, keyword detection can be performed on the word segmentation result, and the database library name included in the database script can be identified from the word segmentation result. Specifically, a preset feature thesaurus can be used to determine the location of the target feature word from the word segmentation result, perform keyword matching on the target word segmentation, and identify the target feature word from the text content after the location of the target feature word in the word segmentation result. out the database name. For example, a preset feature thesaurus can be used to determine the location of From from the word segmentation result, and keyword matching can be performed on the target word segmentation to identify the library name from cx_XX_safe..text_table b1, .text_table c1, .text_table e1 cx_XX_safe, identifies the library name cx_XXX_safe from cx_XXX_safetext_table b1, .text_table c2, .text_table e2.

After the database name is identified, multiple databases corresponding to the database script can be determined according to the database library name.

The data acquisition module 203 is used for acquiring the data table corresponding to each database.

Each database corresponds to the corresponding data table of the database. After the database database name is identified, the table name following the database name can be extracted, and the data table corresponding to the database can be determined according to the table name.

In an optional implementation manner, the data acquisition module 203 acquiring the data table corresponding to the database includes:

Obtain the database library name of the database, and determine the database library name as the first keyword;

determining a second keyword included in the database script according to the first keyword, where the second keyword is located after the first keyword;

A data table name is generated according to the second keyword, and a data table corresponding to the database is determined according to the data table name.

The keyword matching technology is performed on the target word segmentation, and after identifying the database database name of the database from the text content after the location of the target feature word in the word segmentation result, the database database name is determined as the first keyword. According to a preset extraction rule, several word segmentations after the first keyword are extracted and determined as the second keyword. The data table name is determined according to the content of the second keyword, thereby obtaining the data table corresponding to the database. For example, the processor may determine the library name cx_XX_safe as the first keyword, and determine the extracted text table statement as the second keyword according to preset extraction rules (such as extracting text table statements), such as text_table b1, .text_table c1, .text_tablee1 are determined as the second keyword, and the data table name (text_table b1, .text_table c1, .text_table e1) is determined according to the content of the second keyword, so as to obtain the corresponding data table (b1, c1, .text_table e1) of the database. e1); the library name cx_XXX_safe can be determined as the first keyword, and the extracted text table statement can be determined as the second keyword according to the preset extraction rules (such as extracting text table statements), such as text_table b1, . text_table c2, .text_tablee2 are determined as the second keyword, and the data table name (text_table b1, .text_table c2, .text_table e2) is determined according to the content of the second keyword, so as to obtain the data table (b1, c2, e2) corresponding to the database ).

The link determination module 204 is configured to determine the data link corresponding to each database based on the pre-built data map and the data table.

There may be a data flow direction in every two data tables, and the data flow direction is used to indicate the source of the data in the data table. For example, data table A flows to data table B, indicating that data table A is the source of data table B, and data table B is processed on the basis of data table A. The data flow direction corresponding to the data table is recorded in the pre-built data map, and the data link corresponding to each database can be determined according to the data map and the data table. The data link is the processing flow of the data table when processing the to-be-processed data. According to the data link, the number of data tables to be processed when processing the data to be processed can be determined.

In an optional implementation manner, based on the pre-built data map and the data table, before determining the data link corresponding to each database, the link determination module 204 is further configured to:

Get the history database script;

Text recognition is performed on the historical database script, and the data flow direction text is determined in the historical database script;

Determine the flow direction of the data table according to the data flow direction text;

Based on the flow direction of the data table, a data map is generated.

The instruction has a fixed instruction word, text recognition is performed on the historical database script, the instruction word corresponding to the data flow direction is identified to obtain the data flow direction text, and according to the data flow direction text, the flow direction of the data table is determined, and the data map is obtained.

In an optional implementation manner, the link determination module 204 performs text recognition on the historical database script, and determining the data flow direction text in the historical database script includes:

Call the preset word segmentation toolkit to perform word segmentation processing on the historical database script, and obtain a word segmentation result;

From the word segmentation result, determine the position of the preset flow direction word in the historical database script;

According to the position of the preset flow direction word in the historical database script, the data flow direction text is extracted from the historical database script.

The preset flow direction word can be determined according to the language corresponding to the database operation statement used in the historical database script, and there is no restriction on the preset flow direction word here. For example, if the language corresponding to the database operation statement is an object-oriented query language (Hibernate Query Language, HQL), the preset flow words can be insert and select...from, and the historical database script can be written based on the preset flow direction words in the historical database. In the position, extract the sentence corresponding to the position as the data flow text.

For example, the data flow direction text is "insert overwrite table T2; select a, b from tableT1", and the data flow direction can be determined in the data flow direction text based on text recognition: table T1->table T2 (table T1 flows to table T2). If the data of table T1 is processed by table T0 through the above process, it will form table T0->tableT1 (table T0 flows to table T1), and finally form a data link: table T0->table T1->table T2 (table T0->table T1->table T2 (table T0 flows to table T1) T0 flows to table T1, which flows to table T2).

If there is also a table T3, as shown in Figure 2, the table T3 is obtained from the simultaneous association of T0 and T4.

At this time, from table T1, table T2 and table T3, a data map can be obtained. As shown in Figure 3, the data map includes three links: table T0->table T1->table T2 (table T0 flows to table T1, table T1 flows to table T2); table T0->table T3 (table T0 flows to table T3); table T4->table T3 (table T4 flows to table T3).

The link identification module 205 is configured to determine a common link in the data links corresponding to each of the two databases.

The common link is the overlapping link in the data links corresponding to the two databases.

Exemplarily, based on a pre-built data map, the data link corresponding to the database cx_XX_safe is determined, as shown in FIG. 4 ; the data link corresponding to the database cx_XXX_safe is determined, as shown in FIG. 5 .

According to the data link corresponding to database cx_XX_safe and the data link corresponding to database cx_XXX_safe, it is determined that there are two public links in these two databases: table a0->table b3 (table a0 flows to table b1); table d1->table b1 (table d1 flows to table b1).

The data processing module 206 is configured to determine a target data link corresponding to the database script based on the public link, and perform data processing based on the target data link.

Merging the data links of the two databases with the common link to obtain the target data link corresponding to the database script, and then processing the target data link, so as to avoid waste caused by processing the duplicate links, Reduce the consumption of CPU resources.

In an optional implementation manner, the data processing module 206 determines, based on the public link, the target data link corresponding to the database script includes:

determining multiple coincidence databases corresponding to the public link;

obtaining the data link corresponding to each of the overlapping databases;

Based on the common link, the data links corresponding to every two overlapping databases are combined to obtain a first data link;

Obtain a second data link according to a database not corresponding to the public link in the database script;

A target data link corresponding to the database script is determined according to the first data link and the second data link.

Multiple overlapping databases refer to two or more overlapping databases. A common link can correspond to two databases or more than two databases. For the convenience of distinguishing the databases, the database corresponding to the public link is now called the coincidence database. A plurality of coincidence databases corresponding to each public link are determined, and data links corresponding to every two coincidence databases in the plurality of coincidence databases corresponding to the public link are combined to obtain a first data link.

For example, it is determined that the public link corresponds to two coincidence databases (cx_XX_safe, cx_XXX_safe). The public links corresponding to the two coincidence databases are shown in FIG. 4 and FIG. 5 . The corresponding data links are combined to obtain a first data link, as shown in FIG. 6 .

The second data link is obtained according to the database that does not contain the public link in the database script. The data link of the database that does not include the common link may be determined as the second data link.

The artificial intelligence-based data processing device described in the present application identifies multiple databases corresponding to the database scripts by identifying the database scripts corresponding to the data to be processed based on a text recognition algorithm; determine the data link corresponding to each database; then determine the public link in the data link corresponding to each two databases, and the public link is the link with repeated processing; finally, based on the public link, Determine the target data link corresponding to the database script, and perform data processing based on the target data link. Processing based on the target data link can avoid repeated processing of the same data table, reduce resource consumption, and improve Efficiency of data processing.

Embodiment 3

This embodiment provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, implements the steps in the above-mentioned embodiment of the data processing method based on artificial intelligence, for example, FIG. 1 S11-S16 shown:

S11, obtaining a database script corresponding to the data to be processed;

S12, performing text recognition on the database script to determine multiple databases corresponding to the database script;

S13, obtain the data table corresponding to each database;

S14, based on the pre-built data map and the data table, determine the data link corresponding to each database;

S15, determine the public link in the data link corresponding to every two databases;

S16, based on the public link, determine a target data link corresponding to the database script, and perform data processing based on the target data link.

Alternatively, when the computer program is executed by the processor, the functions of the modules/units in the foregoing apparatus embodiments are implemented, for example, the modules 201-206 in FIG. 7 :

The script acquisition module 201 is used for acquiring the database script corresponding to the data to be processed;

A text recognition module 202, configured to perform text recognition on the database script, and determine a plurality of databases corresponding to the database script;

The data acquisition module 203 is used to acquire the data table corresponding to each database;

a link determination module 204, configured to determine the data link corresponding to each database based on the pre-built data map and the data table;

a link identification module 205, configured to determine a common link in the data links corresponding to every two databases;

The data processing module 206 is configured to determine a target data link corresponding to the database script based on the public link, and perform data processing based on the target data link.

Embodiment 4

Referring to FIG. 8 , it is a schematic structural diagram of an electronic device according to Embodiment 3 of the present application. In a preferred embodiment of the present application, the electronic device 3 includes a memory 31 , at least one processor 32 , a transceiver 33 and at least one communication bus 34 .

Those skilled in the art should understand that the structure of the electronic device shown in FIG. 8 does not constitute a limitation of the embodiments of the present application, and may be a bus-type structure or a star-shaped structure, and the electronic device 3 may also include a ratio diagram more or less other hardware or software, or a different arrangement of components is shown.

In some embodiments, the electronic device 3 is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, Programmable gate arrays, digital processors and embedded devices, etc. The electronic device 3 may also include a client device, which includes but is not limited to any electronic product that can perform human-computer interaction with a client through a keyboard, a mouse, a remote control, a touchpad, or a voice-activated device, for example, Personal computers, tablets, smartphones, digital cameras, etc.

It should be noted that the electronic device 3 is only an example, and other existing or future electronic products, if applicable to the present application, should also be included within the protection scope of the present application, and incorporated herein by reference .

In some embodiments, a computer program is stored in the memory 31, and when the computer program is executed by the at least one processor 32, all or part of the steps in the data processing method based on artificial intelligence as described above are implemented. Described memory 31 comprises read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), One-time Programmable Read-Only Memory (OTPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), CompactDisc Read-Only Memory, CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function, and the like; The data created by the use of the node, etc.

The blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

In some embodiments, the at least one processor 32 is a control core (ControlUnit) of the electronic device 3, uses various interfaces and lines to connect various components of the entire electronic device 3, and stores in the memory by running or executing 31, and call the data stored in the memory 31 to execute various functions of the electronic device 3 and process data. For example, when the at least one processor 32 executes the computer program stored in the memory, all or part of the steps of the artificial intelligence-based data processing method described in the embodiments of the present application are implemented; or an artificial intelligence-based data processing apparatus is implemented. all or part of the functions. The at least one processor 32 may be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits packaged with the same function or different functions, including one or more central processing units. (Central Processing Unit, CPU), microprocessor, digital processing chip, graphics processor and combination of various control chips, etc.

In some embodiments, the at least one communication bus 34 is arranged to enable connection communication between the memory 31 and the at least one processor 32 and the like.

Although not shown, the electronic device 3 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 32 through a power management device, so as to be implemented by the power management device Manage charging, discharging, and power management functions. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device 3 may further include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

The above-mentioned integrated units implemented in the form of software functional modules may be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium, and include several instructions to enable a computer device (which may be a personal computer, an electronic device, or a network device, etc.) or a processor (processor) to execute the methods described in the various embodiments of the present application. part.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, and may be located in one place or distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, but that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Accordingly, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the application is to be defined by the appended claims rather than the foregoing description, which is therefore intended to fall within the scope of the claims. All changes within the meaning and scope of the equivalents of , are included in this application. Any reference signs in the claims shall not be construed as limiting the involved claim. Furthermore, it is clear that the word "comprising" does not exclude other units or, and the singular does not exclude the plural. A plurality of units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The terms first, second, etc. are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than limitations. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present application.

Claims (10)

1. a data processing method based on artificial intelligence, is characterized in that, described method comprises: Obtain the database script corresponding to the data to be processed; Perform text recognition on the database script, and determine a plurality of databases corresponding to the database script; Get the data table corresponding to each database; Based on the pre-built data map and the data table, determine the data link corresponding to each database; Determine the common link in the data link corresponding to each two databases; Based on the public link, a target data link corresponding to the database script is determined, and data processing is performed based on the target data link. 2. the data processing method based on artificial intelligence as claimed in claim 1 is characterized in that, described carrying out text recognition to described database script, confirming that the multiple databases corresponding to described database script comprises: Perform word segmentation processing on the database script to obtain word segmentation results; Perform keyword detection on the word segmentation result, and identify multiple databases corresponding to the database script from the word segmentation result. 3. The data processing method based on artificial intelligence as claimed in claim 2, characterized in that, said word segmentation result is carried out keyword detection, and multiple databases corresponding to said database script are identified from said word segmentation result ,include: The location of the target feature word is determined from the word segmentation result by using a preset feature word database, the feature word database includes a plurality of feature words, and the feature word refers to the feature word of the database operation statement used by the database script; The word segmentation result is subjected to keyword matching, and the database library name is identified from the text content after the location of the target feature word in the word segmentation result; According to the database library name, multiple databases corresponding to the database script are determined. 4. The data processing method based on artificial intelligence as claimed in claim 1, is characterized in that, described obtaining the corresponding data table of each database comprises: Obtain the database library name of the database, and determine the database library name as the first keyword; determining a second keyword included in the database script according to the first keyword, where the second keyword is located after the first keyword; A data table name is generated according to the second keyword, and a data table corresponding to the database is determined according to the data table name. 5. The data processing method based on artificial intelligence as claimed in claim 1, wherein, determining the target data link corresponding to the database script based on the public link comprises: determining multiple coincidence databases corresponding to the public link; obtaining the data link corresponding to each of the overlapping databases; Based on the common link, the data links corresponding to every two overlapping databases are combined to obtain a first data link; Obtain a second data link according to a database not corresponding to the public link in the database script; A target data link corresponding to the database script is determined according to the first data link and the second data link. 6. The data processing method based on artificial intelligence as claimed in claim 1, characterized in that, before determining the data link corresponding to each database based on a pre-built data map and the data table, the method further comprises: Get the history database script; Text recognition is performed on the historical database script, and the data flow direction text is determined in the historical database script; Determine the flow direction of the data table according to the data flow direction text; Based on the flow direction of the data table, a data map is generated. 7. The data processing method based on artificial intelligence as claimed in claim 6, characterized in that, the described historical database script is carried out text recognition, and in the historical database script, it is determined that the data flow direction text comprises: Call the preset word segmentation toolkit to perform word segmentation processing on the historical database script, and obtain a word segmentation result; From the word segmentation result, determine the position of the preset flow direction word in the historical database script; According to the position of the preset flow direction word in the historical database script, the data flow direction text is extracted from the historical database script. 8. A data processing device based on artificial intelligence, wherein the device comprises: The script obtaining module is used to obtain the database script corresponding to the data to be processed; a text recognition module, configured to perform text recognition on the database script, and determine a plurality of databases corresponding to the database script; The data acquisition module is used to acquire the data table corresponding to each database; a link determination module, configured to determine the data link corresponding to each database based on the pre-built data map and the data table; The link identification module is used to determine the common link in the data link corresponding to each two databases; A data processing module, configured to determine a target data link corresponding to the database script based on the public link, and perform data processing based on the target data link. 9. An electronic device, characterized in that the electronic device comprises a processor and a memory, and when the processor is used to execute the computer program stored in the memory, the artificial intelligence-based artificial intelligence-based artificial intelligence as described in any one of claims 1 to 7 is realized when the processor is used to execute the computer program stored in the memory. data processing method. 10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the computer program according to any one of claims 1 to 7 is implemented. Data processing methods for artificial intelligence.

Images