CN112486981B

CN112486981B - Data processing method, device, electronic device and computer storage medium

Info

Publication number: CN112486981B
Application number: CN201910866658.3A
Authority: CN
Inventors: 王烨; 周祥
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2019-09-12
Filing date: 2019-09-12
Publication date: 2025-04-22
Anticipated expiration: 2039-09-12
Also published as: CN112486981A

Abstract

The embodiment of the invention provides a data processing method, a data processing device, electronic equipment and a computer storage medium. The method comprises the steps of obtaining a creation instruction of a target data table, obtaining setting dynamic field information from the creation instruction, wherein the setting dynamic field information is used for indicating definition rules of dynamic fields in the target data table, creating the target data table according to the definition rules indicated in the setting dynamic field information, wherein the target data table comprises the dynamic fields defined by the definition rules, and storing the setting dynamic field information into metadata of the target data table. By the embodiment of the invention, the usability and convenience can be improved.

Description

Data processing method, device, electronic equipment and computer storage medium

Technical Field

The embodiment of the invention relates to the technical field of computers, in particular to a data processing method, a data processing device, electronic equipment and a computer storage medium.

Background

With the advent of the large Data age, the amount of Data that needs to be stored has suddenly increased, and in order to meet new Data storage, management, and usage requirements, data lakes (Data lakes) have been used to store and manage Data. In a data lake usage scenario, a user also needs to define databases and data tables in a data lake and map the databases or data tables to various data sources through information such as paths, protocols, and the like. When inquiring the data table, the data source is found through the mapping relation, and the data is read from the data source and then analyzed.

The foregoing, whether it be a conventional database or a data lake, requires that all the fields in the data table be completely and explicitly defined when defining the data table, so that various processes can be performed on the fields in the data table during subsequent query analysis.

However, in a data lake usage scenario, it is often the case that field information in the real data source to which the data table corresponds is constantly changing, even unknown. Such as a key-value type, broad-table storage database of HBase, bigTable, tableStore or the like, which have only the primary key field fixed and not the primary key field allowed to change.

For this situation, in order to ensure that the data can be processed in the database using SQL normally, it is necessary to repeatedly change the data table structure and update the metadata of the data table by means such as DDL (Data Definition Language ) every time a change occurs in a field in the data table. The problem with this approach is that after the data content changes, fields in the data table are processed using DDL, such as adding fields, deleting fields, etc. This way of frequently changing the data table structure using DDL is required, because if the data table is not normally processed without changing the table structure and metadata, the processing of storing metadata corresponding to the data table in the database is complicated, the usability is poor, and the operation is complicated.

Disclosure of Invention

In view of the above, an embodiment of the present invention provides a data processing scheme to solve some or all of the above problems.

According to a first aspect of an embodiment of the present invention, there is provided a data processing method, including obtaining a creation instruction of a target data table, obtaining setting dynamic field information from the creation instruction, where the setting dynamic field information is used to indicate a definition rule of a dynamic field in the target data table, creating the target data table according to the definition rule indicated in the setting dynamic field information, where the target data table includes the dynamic field defined by the definition rule, and storing the setting dynamic field information in metadata of the target data table.

According to a second aspect of the embodiment of the invention, a data processing method is provided, which comprises the steps of obtaining information of data to be queried, which is requested by a data query request, from the data query request, wherein the information of the data to be queried comprises information of a target data table and information of a target field, obtaining information of a field matched with set dynamic field information in the metadata in the target field or obtaining information of a field matched with the set dynamic field information and information of a fixed field in the target data table according to the information of the target field and metadata of the target data table, and obtaining an analysis result of analyzing the data query request according to the obtained information of the field.

According to a third aspect of the embodiment of the invention, a data processing device is provided, which comprises an instruction receiving module, a table creating module and a table creating module, wherein the instruction receiving module is used for obtaining a creating instruction of a target data table, obtaining setting dynamic field information from the creating instruction, wherein the setting dynamic field information is used for indicating definition rules of dynamic fields in the target data table, the table creating module is used for creating the target data table according to the definition rules indicated in the setting dynamic field information, the target data table comprises dynamic fields defined by the definition rules, and the setting dynamic field information is stored in metadata of the target data table.

According to a fourth aspect of the embodiment of the invention, a data processing device is provided, which comprises an acquisition module, a matching module and a first generation module, wherein the acquisition module is used for acquiring information of data to be queried requested by a data query request from the data query request, the information of the data to be queried comprises information of a target data table and information of a target field, the matching module is used for acquiring information of a field matched with set dynamic field information in the metadata in the target field or acquiring information of a field matched with the set dynamic field information and information of a fixed field in the metadata according to the information of the target field and the metadata of the target data table, the set dynamic field information is used for indicating definition rules of the dynamic field in the target data table, and the first generation module is used for acquiring an analysis result for analyzing the data query request according to the acquired information of the field.

According to a fifth aspect of the embodiment of the invention, a database system is provided, which comprises a front node and a computing node, wherein the front node is used for acquiring a creation instruction of a target data table and acquiring set dynamic field information from the creation instruction, the front node interacts with the computing node so that the computing node creates the target data table according to the definition rule indicated in the set dynamic field information, the target data table comprises dynamic fields defined by the definition rule and stores the set dynamic field information into metadata of the target data table, or the front node is used for acquiring a data query request and generating an analysis result by acquiring information of data to be queried requested by the data query request and metadata of the target data table, and the front node interacts with the computing node so that the computing node acquires data of the target field requested by the data query request from a data source corresponding to the target data table according to the analysis result.

According to a sixth aspect of the embodiment of the present invention, there is provided an electronic device, including a processor, a memory, a communication interface, and a communication bus, where the processor, the memory, and the communication interface complete communication with each other through the communication bus, and the memory is configured to store at least one executable instruction, where the executable instruction causes the processor to perform an operation corresponding to the data processing method according to the first aspect or the second aspect.

According to a seventh aspect of embodiments of the present invention, there is provided a computer storage medium having stored thereon at least one target data table, the metadata of the target data table including at least one dynamic field, the at least one dynamic field being a field created according to a definition rule.

According to an eighth aspect of an embodiment of the present invention, there is provided a computer storage medium having stored thereon a computer program which, when executed by a processor, implements the data processing method according to the first or second aspect.

According to the data processing scheme provided by the embodiment of the invention, the target data table containing the dynamic fields can be created according to the set dynamic field information in the creation instruction, so that the target data table can seamlessly support the change of the dynamic fields of the data source, seamless compatibility between the change of the data and the user data query request is realized, the operation of a user is greatly simplified, the performance is saved, and the problem that the fields in the data table need to be updated by using DDL (data storage and distribution) every time the data source is changed in the prior art is solved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present invention, and other drawings may be obtained according to these drawings for a person having ordinary skill in the art.

FIG. 1 is a flow chart showing steps of a data processing method according to a first embodiment of the present invention;

FIG. 2 is a flow chart showing steps of a data processing method according to a second embodiment of the present invention;

FIG. 3 is a flow chart showing the steps of a data processing method according to a third embodiment of the present invention;

FIG. 4 is a flow chart showing the steps of a data processing method according to a fourth embodiment of the present invention;

FIG. 5 is a block diagram of a data processing apparatus according to a fifth embodiment of the present invention;

FIG. 6 is a block diagram showing a data processing apparatus according to a sixth embodiment of the present invention;

FIG. 7 is a block diagram showing a data processing apparatus according to a seventh embodiment of the present invention;

FIG. 8 is a block diagram showing a data processing apparatus according to an eighth embodiment of the present invention;

FIG. 9 is a schematic diagram of a database system in a scenario of a data lake analysis system according to a ninth embodiment of the present invention;

FIG. 10 is a schematic diagram illustrating a first computing node according to a ninth embodiment of the present invention;

FIG. 11 is a schematic diagram illustrating a second computing node according to a ninth embodiment of the present invention;

Fig. 12 is a schematic structural diagram of an electronic device according to a ninth embodiment of the present invention.

Detailed Description

In order to better understand the technical solutions in the embodiments of the present invention, the following description will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which are derived by a person skilled in the art based on the embodiments of the present invention, shall fall within the scope of protection of the embodiments of the present invention.

The implementation of the embodiments of the present invention will be further described below with reference to the accompanying drawings.

Example 1

Referring to fig. 1, a flowchart of steps of a data processing method according to a first embodiment of the present invention is shown.

The data processing method of the present embodiment includes the steps of:

Step S102, obtaining a creation instruction of a target data table, and obtaining setting dynamic field information from the creation instruction.

The creation instructions are for directing the creation of a target data table in a database. Those skilled in the art may instruct the creation of the target data table in an appropriate manner according to the business needs. For example, using SQL, the target data table is created by the create table key indication.

The content contained in the create instruction may be different for different ways of indication or different target data table structures. For example, if it is necessary to create a target data table including a dynamic field, a table name of the target data table, setting dynamic field information, fixed field information, and the like may be included in the creation instruction. For another example, if it is necessary to create a target data table including only fixed fields, the creation instruction may include a table name of the target data table, information of the fixed fields, and the like.

The setting dynamic field information is used for indicating definition rules of dynamic fields in a target data table. Those skilled in the art can configure appropriate definition rules as needed, which are not limited in this embodiment. For example, the definition rules include information defining regular expressions of dynamic fields in the target data table.

Wherein, the dynamic field may be a field whose field name meets a definition rule, and the addition or deletion of the field may not change the metadata of the target data table.

For example, the set dynamic field information may contain information defining rule identifications and dynamic fields. The definition rule identifier is used for indicating the definition rule corresponding to the definition rule identifier so that the database can create dynamic fields in the target data table according to the definition rule. The information of the dynamic field includes, but is not limited to, part or all of the field name of the dynamic field.

The setting dynamic field information may be obtained in any suitable manner for different creation instructions, for example, by using a lexical analysis method.

Step S104, creating the target data table according to the definition rules indicated in the setting dynamic field information, wherein the target data table comprises dynamic fields defined by the definition rules, and storing the setting dynamic field information into metadata of the target data table.

The definition rules may be preset or acquired from other systems. Taking information of a regular expression defining a dynamic field in the target data table as an example, a database defines the dynamic field (such as a field name defining the dynamic field) in the target data table by using the regular expression rule and the information of the dynamic field according to the definition rule, thereby creating the target data table, and storing the definition rule identification and the information of the dynamic field into metadata of the target data table.

Of course, in other embodiments, any appropriate definition rule may be set by those skilled in the art according to the service requirement, and this embodiment is not limited thereto.

Alternatively, the created target data table may establish a connection with the data source in a protocol, path, or the like. Therefore, loose coupling of the target data table and the data source can be realized, when the data source changes the fields in the target data table according to the requirement, the metadata of the target data table can not be changed as long as the changed fields meet the set dynamic field information, so that the condition that the metadata are changed when the fields in the target data table need to be changed due to data change in the data source in the prior art is avoided, and the load of the database is reduced.

According to the embodiment, the target data table containing the dynamic fields can be created according to the set dynamic field information in the creation instruction, so that the target data table can seamlessly support the change of the dynamic fields of the data source, seamless compatibility between data change and a user data query request is realized, the operation of a user is greatly simplified, the performance is saved, and the problem that fields in the data table need to be updated by using DDL (data storage layer) every time the data source is changed in the prior art is solved.

The data processing method of the present embodiment may be performed by any suitable electronic device having data processing capabilities, including, but not limited to, servers, mobile terminals (e.g., tablet computers, cell phones, etc.), PCs, and the like.

Example two

Referring to fig. 2, a flowchart of steps of a data processing method according to a second embodiment of the present invention is shown.

The data processing method of the present embodiment includes the steps S102 to S104. In this embodiment, the method further includes:

Step S106, obtaining the information of the data to be queried, which is requested by the data query request, from the data query request.

The data query request may be a request generated using a structured query language (e.g., SQL) for requesting queries and processing of data in the database. Among these, processing data may be selecting data, deleting data, updating data, and so forth.

The information of the data to be queried may be obtained by lexical analysis (e.g., lexical analysis using flex) of the data query request.

The information of the data to be queried comprises information of a target data table and information of a target field. The target data table may be one or more data tables stored in a database. The information of the target data table may include a table name of the target data table, and may further include other information of the target data table. The information of the target field may include field names of one or more target fields, and may also include other information of the target field, such as screening conditions of the target field, and so on.

Step S108, according to the information of the target field and the metadata of the target data table, obtaining the information of the field matched with the set dynamic field information in the metadata in the target field, or obtaining the information of the field matched with the set dynamic field information and the information of the fixed field.

Metadata of the target data table may be understood as related attribute data of the target data table, and the metadata includes, but is not limited to, field names of fields included in the target data table, data types of the fields, whether null (null), default values, autonomy, whether primary keys are remarks, and the like.

The setting dynamic field information in the metadata is used for indicating definition rules of dynamic fields in the target data table. Dynamic fields are understood to be non-fixed fields defined by corresponding definition rules that can be dynamically increased or decreased in the target data table.

Setting the dynamic field information may indicate the definition rules in any suitable manner. For example, in the present embodiment, the setting dynamic field information includes definition rule identification for indicating definition rules and information for indicating dynamic fields of the dynamic fields. Of course, the setting dynamic field information may also include the data type of the dynamic field, whether it is empty, and the like, which is not limited in this embodiment.

The definition rule identifier may be any suitable identifier, as long as the definition rule identifier indicates a corresponding definition rule. For example, the definition rule is identified as "regex_column" which is used to indicate that the definition rule is a regular rule. For another example, the definition rule is identified as "full" for indicating a full word match definition rule among preset definition rules, or for indicating a full match definition rule, or the like. The definition rule identifier may be any suitable identifier, which is not limited in this embodiment, as required.

The information of the dynamic field may include a partial field name of the dynamic field, etc. For example, the information of the dynamic field includes ' abc_, and the field name of the dynamic field indicated by the information of the dynamic field starts with ' abc_ ' according to its corresponding definition rule, and may be, for example, ' abc_1', ' abc_2', and so on.

By setting the dynamic field information in the metadata, at least one dynamic field in the target data table is stored, so that even if the dynamic field in the target data table is changed (such as new fields are added, new fields are reduced or field names are changed), the metadata can not be modified as long as the dynamic field information is set, and accordingly when the target data table needs to be updated, added or deleted, the dynamic field in the target data table can be directly processed correspondingly without frequently changing the metadata, load pressure and resource consumption of a metadata system are reduced, and response speed of a database can be effectively improved.

Taking regex_column' abc_ as an example, the field indicated by the information of the field matching the dynamic field information in the metadata in the target field should be a field with a field name beginning with abc, for example, abc_1, abc_2, etc.

The information of the fixed field indicates a target field that matches the fixed field of the target data table. For example, the fixed field of the target data table includes name, age, height, etc., and the target field indicated by the information of the fixed field may be a name, an age, etc.

And step S110, according to the obtained information of the fields, obtaining an analysis result for analyzing the data query request.

If the target field indicated by the obtained field information is all the target fields in the target field information, directly obtaining an analysis result for analyzing the data query request according to the obtained field information. For example, a parsed structure tree is generated from the information of the obtained fields.

If the target field indicated by the obtained field information is a part of the target fields in the target field information, that is, another part of the target fields are not fixed fields or dynamic fields in the target data table, the target field is not in the target data table, the data query request is incorrect, and the analysis and the error reporting can be terminated.

In addition, for the data query request, the information of the data to be queried requested by the data query request is obtained, and because the setting dynamic field information is used for indicating the definition rule of the dynamic field in the target data table, the information of the field matched with the setting dynamic field information is obtained by matching the target field with the setting dynamic field information, or the information of the field matched with the setting dynamic field information and the information of the fixed field are obtained, and the analysis result for analyzing the data query request is obtained according to the obtained information of the field, so that the analysis of the data query request can be completed under the condition of not frequently changing metadata, and the usability and the simplicity are improved.

Example III

Referring to fig. 3, a flowchart of the steps of a data processing method according to a third embodiment of the present invention is shown.

The data processing method of the present embodiment includes the steps of:

step S202, obtaining information of data to be queried, which is requested by a data query request, from the data query request.

The information of the data to be queried may be obtained by lexical analysis (e.g., lexical analysis using flex) of the data query request. The information of the data to be queried comprises information of a target data table and information of a target field. The target data table may be one or more data tables stored in a database. The information of the target data table may include a table name of the target data table, and may further include other information of the target data table. The information of the target field may include field names of one or more target fields, and may also include other information of the target field, such as screening conditions of the target field, and so on.

Step S204, according to the information of the target field and the metadata of the target data table, obtaining the information of the field matched with the set dynamic field information in the metadata in the target field, or obtaining the information of the field matched with the set dynamic field information and the information of the fixed field.

And S206, according to the obtained information of the fields, obtaining an analysis result for analyzing the data query request.

According to the embodiment, aiming at the data query request, the information of the requested data to be queried is obtained, and because the set dynamic field information is used for indicating the definition rule of the dynamic field in the target data table, the information of the field matched with the set dynamic field information is obtained by matching the target field with the set dynamic field information, or the information of the field matched with the set dynamic field information and the information of the fixed field are obtained, and the analysis result for analyzing the data query request is obtained according to the obtained information of the field, so that the analysis of the data query request can be completed under the condition of not frequently changing metadata, and the usability and the convenience are improved.

Example IV

Referring to fig. 4, a flowchart of the steps of a data processing method according to a fourth embodiment of the present invention is shown.

The data processing method of the present embodiment includes the steps of:

Step S300a, obtaining a creation instruction of the target data table, and obtaining the setting dynamic field information from the creation instruction.

It should be noted that this step is an optional step, and may be performed at any appropriate timing before step S302 described below.

Wherein the set dynamic field information includes information defining a rule identification and a dynamic field. The definition rule identifier is used for indicating the definition rule corresponding to the definition rule identifier so that the database can create dynamic fields in the target data table according to the definition rule. The information of the dynamic field includes, but is not limited to, part or all of the field name of the dynamic field.

And step S300b, creating the target data table according to the definition rule identification and the dynamic field information in the dynamic field setting information, and storing the definition rule identification and the dynamic field information into metadata of the target data table.

It should be noted that this step is an optional step, and may be performed at any suitable timing after step S300a and before step S302 described below.

The database can determine the corresponding definition rule according to the definition rule identification, and the definition rule can be a preset definition rule or can be acquired from other systems according to the definition rule identification.

Taking information including a regular expression defining a dynamic field in the target data table as an example in a definition rule, a database defines the dynamic field (e.g., a field name defining the dynamic field) in the target data table according to the definition rule using the regular expression rule and the information of the dynamic field, thereby creating the target data table and saving the definition rule identification and the information of the dynamic field to metadata of the target data table.

Step S302, obtaining information of data to be queried, which is requested by a data query request, from the data query request.

The step may be implemented by the implementation of step S202 in the third embodiment, so that the description is omitted here.

Step S304, according to the information of the target field and the metadata of the target data table, obtaining the information of the field matched with the set dynamic field information in the metadata in the target field, or obtaining the information of the field matched with the set dynamic field information and the information of the fixed field.

In a first specific implementation, the step S304 includes the following sub-steps:

And a substep S3041, namely determining a field which is not matched with a fixed field in the target data table in the target field according to the information of the target field.

In a specific embodiment, according to the information of the target field, the indicated target field is matched with the fixed field in the target data table, for example, a field name of the target field is matched with a field name of the fixed field in the target data table in an accurate matching manner, and the target field, of which the field name is not matched with the field name of the fixed field, is determined as the field, of the target field, which is not matched with the fixed field in the target data table.

And a substep S3042, wherein the information of the unmatched fields is matched with the set dynamic field information, and the information of the fields matched with the set dynamic field information is obtained according to the matching result.

And matching the information of the unmatched fields with the information of the set dynamic fields. For example, according to the definition rule indicated by the setting dynamic field information, a matching manner between the information of the unmatched field and the setting dynamic field information is determined, and the field name of the target field indicated by the information of the unmatched field and the information of the dynamic field indicated by the setting dynamic field information are matched. And obtaining the information of the field matched with the set dynamic field information according to the matching result.

Taking information defining regular expressions of dynamic fields in the target data table as an example, the corresponding matching mode is a regular matching mode. If the information of the dynamic field includes abc, the target field with the field name beginning with abc can be determined as the target field matched with the information of the set dynamic field, so that a matching result can be determined according to the target field, and the information of the field matched with the information of the set dynamic field can be obtained according to the matching result.

It should be noted that, the metadata may include a plurality of pieces of setting dynamic field information, and when matching is performed, information of a field that is not matched may be matched with the setting dynamic field information one by one.

By the method, the information of the field matched with the set dynamic field information in the metadata in the target field can be obtained quickly and accurately, the matching efficiency is improved, the information of different set dynamic fields can be matched in a matching mode corresponding to the information, and the matching accuracy is ensured.

Or in a second specific implementation manner, the step S304 includes, in addition to the sub-steps S3041 to S3042 in the first specific implementation manner, further:

substep S3043 determines information of a fixed field in the target field.

For example, according to the information of the target field, the indicated target field is matched with a fixed field in the target data table, for example, the field name of the target field is matched with the field name of the fixed field in the target data table in an accurate matching manner, the target field with the field name of the fixed field matched with the field name of the fixed field is determined as the fixed field in the target field, and the information of the fixed field is acquired.

It should be noted that, in order to improve efficiency, the matching result of the substep S3043 may be directly used in the substep S3043, a target field with a field name matching with a field name of a fixed field is determined as a fixed field in the target field, and information of the fixed field is acquired.

And step S306, according to the obtained information of the fields, obtaining an analysis result for analyzing the data query request.

The step may be implemented by the implementation of step S206 in the third embodiment, so that the description is omitted here.

And step 308, generating a query execution instruction corresponding to the data query request according to the analysis result.

It should be noted that this step is an optional step, and may be performed at any appropriate timing after step S306.

The query execution instructions are used to instruct the corresponding query actions to be performed, such as scanning the data source of the target data table, etc. Any suitable means may be employed by those skilled in the art to generate query instructions from the parsing results.

Optionally, in order to increase the response speed to the data query request, the database generates an execution plan with minimum resource consumption according to the analysis result, where the execution plan includes one or more query execution instructions, and this method can optimize the order of the query execution instructions compared with a method of directly generating the query execution instructions, thereby further reducing the resource consumption of executing the query execution instructions.

And step S310, according to the query execution instruction, acquiring the data of the target field requested by the data query request from the data source corresponding to the target data table.

In a specific embodiment, the step S310 includes the following substeps:

Substep S3101 is to determine, according to the query execution instruction, whether the information of the target field contains a wild card identifier for indicating all fields.

Taking SQL as an example, the wild card identifier for indicating all fields may be '×', and of course, those skilled in the art may define other characters as the wild card identifier for indicating all fields.

If it is determined that the information of the target field contains the wild card identifier, the sub-step S3102 is executed, and if it is determined that the information of the target field does not contain the wild card identifier, the data of the target field related to the query execution instruction is directly obtained from the data source corresponding to the target data table.

And step S3102, if the information is included, acquiring the information of all fields in the data source corresponding to the target data table.

Because the metadata of the target data table stores the dynamic fields in the target data table in a manner of setting the dynamic field information, the dynamic fields indicated by the wild card identifier cannot be directly determined according to the metadata, so that when the wild card identifier is contained, the information of all the fields in the data source corresponding to the target data table is acquired, and the field indicated by the wild card identifier is determined according to the information of all the fields actually existing in the data source of the target data table.

Substep S3103, matching the information of all fields with the set dynamic field information, and determining the dynamic field in all fields according to the matching result.

In a specific embodiment, when the information of all the fields is matched with the information of the set dynamic field, a matching mode corresponding to a definition rule indicated by the information of the set dynamic field may be used to match the field names in the information of all the fields with the information of the dynamic field, and all the matched fields are determined as the dynamic fields in all the fields.

For example, if the matching method is a regular matching method and the information of the dynamic field is ' abc_, the field with the field name beginning with ' abc_ ' in all the fields can be determined as the dynamic field. The wild card identifier can then be expanded to determine the specific field it indicates.

And step S3104, acquiring data of all fields including the dynamic field from a data source corresponding to the target data table.

And acquiring data of all fields including the dynamic field from a data source corresponding to the target data table according to the field indicated by the expanded wild card identifier.

If the query execution instruction relates to the wild card identifier, the method can determine the dynamic field by scanning the fields in the data source, matching the fields with the set dynamic field information, and expanding the set dynamic field information in the metadata, so as to determine the specific field corresponding to the wild card identifier, and further obtain all fields corresponding to the wild card identifier from the data source.

According to the embodiment, aiming at the data query request, the information of the requested data to be queried is obtained, and because the set dynamic field information is used for indicating the definition rule of the dynamic field in the target data table, the information of the field matched with the set dynamic field information is obtained by matching the target field with the set dynamic field information, or the information of the field matched with the set dynamic field information and the information of the fixed field are obtained, and the analysis result for analyzing the data query request is obtained according to the obtained information of the field, so that the analysis of the data query request can be completed under the condition of not frequently changing metadata, and the usability and convenience are improved.

Example five

Referring to fig. 5, there is shown a block diagram of a data processing apparatus according to a fifth embodiment of the present invention.

The data processing device of the embodiment comprises an instruction receiving module 502, a table creating module 504 and a table storing module, wherein the instruction receiving module is used for obtaining a creating instruction of a target data table, obtaining setting dynamic field information from the creating instruction, wherein the setting dynamic field information is used for indicating definition rules of dynamic fields in the target data table, and the table creating module 504 is used for creating the target data table according to the definition rules indicated in the setting dynamic field information, wherein the target data table comprises dynamic fields defined by the definition rules, and storing the setting dynamic field information into metadata of the target data table.

Example six

Referring to fig. 6, there is shown a block diagram of a data processing apparatus according to a sixth embodiment of the present invention.

The data processing device of the embodiment comprises an instruction receiving module 602, a table creating module 604 and a table creating module, wherein the instruction receiving module is used for obtaining a creation instruction of a target data table, obtaining setting dynamic field information from the creation instruction, wherein the setting dynamic field information is used for indicating definition rules of dynamic fields in the target data table, the table creating module 604 is used for creating the target data table according to the definition rules indicated in the setting dynamic field information, the target data table comprises dynamic fields defined by the definition rules, and the setting dynamic field information is stored in metadata of the target data table.

Optionally, the definition rule includes information of a regular expression defining a dynamic field in the target data table.

Optionally, the device further comprises an acquisition module 606, a matching module 608 and a first generation module 610, wherein the acquisition module is used for acquiring information of data to be queried requested by the data query request from the data query request, the information of the data to be queried comprises information of a target data table and information of a target field, the matching module 608 is used for acquiring information of a field matched with set dynamic field information in the metadata in the target field or acquiring information of a field matched with the set dynamic field information and information of a fixed field in the target field according to the information of the target field and metadata of the target data table, and the first generation module 610 is used for acquiring an analysis result for analyzing the data query request according to the acquired information of the field.

The data processing device in this embodiment is configured to implement the corresponding data processing method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described herein again.

Example seven

Referring to fig. 7, there is shown a block diagram of a data processing apparatus according to a seventh embodiment of the present invention.

The data processing device of the embodiment comprises an acquisition module 702, a matching module 704 and a first generation module 706, wherein the acquisition module is used for acquiring information of data to be queried requested by a data query request from the data query request, the information of the data to be queried comprises information of a target data table and information of a target field, the matching module is used for acquiring information of a field matched with set dynamic field information in the metadata in the target field or information of a field matched with the set dynamic field information and information of a fixed field in the target field according to the information of the target field and the metadata of the target data table, the set dynamic field information is used for indicating definition rules of the dynamic field in the target data table, and the first generation module 706 is used for acquiring an analysis result for analyzing the data query request according to the acquired information of the field.

Example eight

Referring to fig. 8, there is shown a block diagram of a data processing apparatus according to an eighth embodiment of the present invention.

The data processing device of the embodiment comprises an acquisition module 802, a matching module 804 and a first generation module 806, wherein the acquisition module is used for acquiring information of data to be queried requested by a data query request from the data query request, the information of the data to be queried comprises information of a target data table and information of a target field, the matching module is used for acquiring information of a field matched with set dynamic field information in the metadata in the target field or information of a field matched with the set dynamic field information and information of a fixed field in the target data table according to the information of the target field and metadata of the target data table, the set dynamic field information is used for indicating definition rules of the dynamic field in the target data table, and the first generation module 806 is used for acquiring an analysis result for analyzing the data query request according to the acquired information of the field.

Optionally, the matching module 804 is configured to determine, according to the information of the target field, a field in the target field that is not matched with a fixed field in the target data table, match the information of the unmatched field with the set dynamic field information, obtain, according to a matching result, information of a field that is matched with the set dynamic field information, or obtain, according to the information of the target field and metadata of the target data table, information of a field that is matched with the set dynamic field information and information of a fixed field, where the set dynamic field information is used to indicate a definition rule of a dynamic field in the target data table.

Optionally, the device further includes a second generating module 808, configured to generate a query execution instruction corresponding to the data query request according to the analysis result, and a data obtaining module 810, configured to obtain, according to the query execution instruction, data of a target field requested by the data query request from a data source corresponding to the target data table.

Optionally, the data acquisition module 810 includes a wild card determining module 8101 for determining whether the information of the target field includes wild card identifiers for indicating all fields according to the query execution instruction, a field information acquiring module 8102 for acquiring information of all fields in a data source corresponding to the target data table when the information is included, a dynamic field determining module 8103 for matching the information of all fields with the set dynamic field information and determining dynamic fields in all fields according to a matching result, and a field data extracting module 8104 for acquiring data of all fields including the dynamic fields from the data source corresponding to the target data table.

Example nine

Referring to FIG. 9, a schematic diagram of a database system for use in the context of a data lake analysis system (DATA LAKE ANALYTICS) is shown. Wherein the database system comprises a pre-node (FRONTNODE) and a compute node (COMPUTENODE).

The pre-node is used for acquiring a creation instruction of a target data table and acquiring setting dynamic field information from the creation instruction, and the pre-node interacts with the computing node so that the computing node creates the target data table according to the definition rule indicated in the setting dynamic field information, the target data table comprises dynamic fields defined by the definition rule, and the setting dynamic field information is stored in metadata of the target data table.

Therefore, a target data table containing dynamic fields can be created according to the set dynamic field information in the creation instruction, further, the target data table can seamlessly support the change of the dynamic fields of the data source, seamless compatibility between data change and user data query requests is achieved, operation of a user is greatly simplified, performance is saved, and the problem that fields in the data table need to be updated by DDL (data storage) in each data source change in the prior art is solved.

Or the front node is used for acquiring a data query request, acquiring information of data to be queried requested by the data query request and metadata of a target data table to generate an analysis result, and interacting with the computing node so that the computing node acquires data of a target field requested by the data query request from a data source corresponding to the target data table according to the analysis result.

Thus, for the data query request, the information of the data to be queried requested by the data query request is obtained, and because the setting dynamic field information is used for indicating the definition rule of the dynamic field in the target data table, the information of the field matched with the setting dynamic field information is obtained by matching the target field with the setting dynamic field information, or the information of the field matched with the setting dynamic field information and the information of the fixed field are obtained, and the analysis result for analyzing the data query request is obtained according to the obtained information of the field, so that the analysis of the data query request can be completed under the condition of not frequently changing metadata, and the usability and the simplicity are improved.

The front node may be used to execute the steps S102, S106 to S110 in the foregoing method embodiment, and achieve the corresponding effect. The computing node may be configured to perform step S104 in the foregoing method embodiment, and achieve the corresponding effect.

Or the front node may be further configured to execute steps S202 to S206, S300a, and S302 to S308 in the foregoing method embodiment, and achieve corresponding effects. The computing node may be configured to perform step S300b and step S310 in the foregoing method embodiments, and achieve corresponding effects.

In a particular implementation, a database system includes a plurality of front-end nodes and a plurality of compute nodes. And after the plurality of front-end nodes are mounted on the load balancing equipment (SLB), load balancing of request processing is realized.

A plurality of pre-node peer-to-peer services form a pool of pre-nodes (FRONTNODE Pool). The front node receives a data Query request (Query), analyzes the data Query request, generates and optimizes a distributed execution plan, and the like. After the distributed execution plan is generated, the distributed execution plan is distributed and scheduled to a plurality of computing nodes for execution. A plurality of compute nodes form a pool of compute nodes (COMPUTENODE Pool). The whole database system architecture adopts a storage and calculation separation architecture, namely a Data Source is separated from a calculation service cluster (the calculation service cluster at least comprises a calculation node pool) of the database system, a Data table in the database system adopts one or more fields in a dynamic field association Data Source created by defining rules, and thus the fields in the Data Source can be independently changed without frequently changing the fields of the Data table in the database system. When running (runtime), the computing node reads data from the data source, and binds the fields in the data source corresponding to the dynamic fields according to the matching result of the dynamic fields and the fields in the data source.

In this embodiment, the database system performs the foregoing method, and the process of creating the target data table including the dynamic field is, for example:

and step 1, acquiring setting dynamic field information according to the creation instruction.

Taking SQL as an example, the creation instruction is:

create table users(

name varchar(120),

age varchar(20),

regex_column‘abc_.*’,

regex_column‘xy[1-3]?’,

)

wherein the create table key is used to indicate the creation of the table. Users are the table names of the target data tables. name, age are field names of fixed fields. The varchar indicates that the data type of the corresponding field is a variable length string type. regex_column is a definition rule identifier in setting dynamic field information. ' abc_, x '; ' xy [1-3]? the information of the dynamic field in the information, regex_column ' abc_ ' refers to all fields beginning with abc_. regex_column 'xy [1-3]? the' finger field is named 'xy' xy1, xy2, xy3 ".

And 2, creating the target data table according to the definition rule identification and the dynamic field information in the dynamic field setting information, and storing the definition rule identification and the dynamic field information into metadata of the target data table.

And creating fixed fields in the target data table according to the creation instruction, creating dynamic fields in the target data table according to the definition rule identification in the information for setting the dynamic fields and the information of the dynamic fields so as to create the target data table, and associating each field of the target data table with a data source.

It should be noted that, depending on the creation instruction, the target data table may include only a dynamic field or only a fixed field.

And storing the information defining the rule identification and the dynamic field into metadata of the target data table. And storing the information of the fixed field into the metadata of the target data table. I.e. saved to the metadata module in fig. 9. For example, metadata includes:

name varchar(120),

age varchar(20),

regex_column‘abc_.*’,

regex_column' xy [1-3 ].

If the data types of the dynamic fields are not determined or consistent, the data types of the dynamic fields may be configured into binary types or character string types in the metadata.

In this embodiment, the database system executes the foregoing method, and the process of performing the data query with respect to the created target data table is:

And A, the load balancing equipment receives the data query request, distributes the data query request to the front-end nodes in the front-end node pool, analyzes the front-end nodes and obtains an analysis result.

And the load balancing equipment performs load balancing according to the resources of each front-end node, and distributes the data query request to the proper front-end node.

The process of analyzing the data query request by the front node is as follows:

and A1, obtaining information of data to be queried, which is requested by the data query request, from the data query request.

For example, lexical analysis is performed on the data query request, and information of the data to be queried requested by the data query request is obtained. The information of the data to be queried includes information of a target data table and information of a target field.

For example, the data query request is SELECT NAME, abc_3from users where age>10and xy1 in ('abc', 'zzz').

The information of the acquired target data table includes a table name, i.e., users. The information of the target field includes field names of the target fields, that is, name, abc_3, age, and xy1.

And A2, according to the information of the target field and the metadata of the target data table, obtaining the information of the field matched with the set dynamic field information in the metadata in the target field, or obtaining the information of the field matched with the set dynamic field information and the information of the fixed field.

When the information of the field matched with the set dynamic field information in the metadata is obtained according to the information of the target field and the metadata of the target data table, firstly matching the field name of the target field with the field name of the fixed field in the metadata, if matching the name with the field name of the fixed field, the matching can be successful, and the information of the fixed field can be obtained according to the information of the successfully matched target field.

Matching abc_3 and xy1 with the field names of the fixed fields fails to match, and these target fields that fail to match are determined to be unmatched fields.

And matching the information of the unmatched fields with the set dynamic field information, and obtaining the information of the fields matched with the set dynamic field information according to a matching result. I.e., abc_3 and xy1 match regex_column 'abc_, and regex_column' xy 1-3, respectively? information of a field matching the set dynamic field information is obtained from the information of the two target fields.

Because the target field which cannot be matched with the fixed field is subjected to rule matching with the set dynamic field information in the analysis process, and the set dynamic field information can indicate a plurality of dynamic fields defined by adopting the same definition rule, even if a certain field in a data source is temporarily deleted or added, metadata can not be changed, correct matching can still be ensured, metadata can not be changed when the field in the data source is changed, load is reduced, and correct processing of a data query request is ensured.

For example, the field names abc_1, abc_2, abc_3 are included in the data source, and metadata need not be changed when the abc_3 field in the data source is deleted.

And step A3, according to the obtained information of the field, obtaining an analysis result for analyzing the data query request.

And generating and obtaining an analysis result according to the obtained field information and the keywords in the data query request. In this usage scenario, the parsing result may be a structure tree.

And B, analyzing and processing the analysis result.

And according to the analysis result, analyzing and processing the analysis result. For example, an index of a target field related to the analysis result is established, an index of a target data table related to the analysis result is established, and the like, so that the analysis result is convenient for subsequent processing and use. Of course, the person skilled in the art may omit this step as required.

And C, carrying out rewriting operation on the data query request according to the analysis result.

The rewrite operation is used to rewrite the data query request into a more efficient query instruction. For example, an algebraizer (rewriter) is used that can perform an equivalent transformation on the data query request to make the execution of the transformed query instruction more efficient and less resource consuming.

And D, generating an execution plan.

And judging the non-nullability, the uniqueness, the sequency, the type and the like of the fields related to the query instruction according to the rewritten query instruction and the metadata. For example, if the query instruction includes a clause for querying a null value in a certain field, but when the non-null judgment is performed, it is determined that the attribute of the field is specified as a non-null column, the execution result of the clause may be directly determined as the null value, and the clause is not actually executed, which is helpful for improving efficiency.

And then, generating an execution plan according to the judging result and the physical resource, wherein the execution plan comprises one or more inquiry execution instructions. Each query execution instruction may be used to instruct execution of a query clause.

And E, distributing and scheduling the execution plan to the corresponding computing node.

Scheduling is carried out according to the generated execution plan, and query execution instructions in the scheduling are distributed to corresponding computing nodes.

And F1, executing the execution plan by the computing node.

A schematic of the architecture of a compute node for executing an execution plan without wild card identification is shown in fig. 10. For example, the data query request is SELECT NAME, abc_3 from users where age>10and xy1 in ('abc', 'zzz').

The scanner (table scan operator) of the computing node acquires data of fields of field names age, name, xy1 and abc_3 (denoted as data set A1) from the target data table, a filter (filter operator) acquires data of a row of age >10 (denoted as data set B1) from the data set A1, and acquires data of a row containing 'abc' and 'zz' in the xy1 field (denoted as data set C1) on the basis of the data set B1. The engineer acquires data of fields with field names name and abc_3 (denoted as data set D1) from the data set C1, and the exporter exports the data set D1 as an execution result.

And G, the result manager displays the result according to the execution result.

The results manager may process the execution results as needed and generate final display results, e.g., adjust the layout of the display, etc.

The query processing procedure for a data query request containing a wild card identifier is similar to the processing procedure without a wild card identifier, except that step H2 is used instead of step H1.

Step F2. A schematic diagram of a computing node for executing an execution plan with wild card identifiers is shown in FIG. 11. For example, the data query request is select x from users where age >10.

The scanner (table scan operator) of the compute node obtains information of all fields from the target data table, such as obtaining field names of the fields name, age, abc _1, abc_2, abc_3, xy1, etc. Matching the information of all the fields with the set dynamic field information, and determining the dynamic field in all the fields according to the matching result, wherein the dynamic field comprises fields with field names abc_1, abc_2, abc_3 and xy1 as abc_1, abc_2 and abc_3 can be matched with regex_column 'abc_, xy1 can be matched with regex_column' xy [1-3 ]. The extension is performed according to the dynamic field, and all the fields indicated by the determination include field names name, age, abc _1, abc_2, abc_3, xy1, and the like. Data of all fields including the dynamic field is acquired. Namely, data (denoted as a dataset A2) of fields having field names age, name, xy1, abc_1, abc_2, abc_3 are acquired.

A filter (filter operator) acquires data of a row of age >10 from the data set A2 (denoted as data set B2).

The engineer may omit or not act because field screening is not required. The outputter outputs the data set B2 as an execution result.

In this embodiment, the execution plan for a single data query request will scan and pass up data for data in the data source, if the fields for which the execution plan was designed are determined, it may be pushed down to the underlying operator at the beginning of the computation, e.g., the scanner may filter the fields as the underlying pulls the data. If the field involved is uncertain (i.e. when the field contains a wild card identifier), the information is always transmitted downwards, the scanner pulls all line data in the target data table from the bottom layer, and then the specific field involved is determined by back-pushing, so that the target data table containing the dynamic field can be ensured to perform normal and reliable data query.

In this embodiment, by creating a target data table with dynamic fields, the target data table does not need to explicitly and clearly define all the fields as in conventional database generic data tables. In a data lake scenario, a target data table with dynamic fields allows a user to define a non-deterministic table structure and define a set of dynamic field information (i.e., field rules). The field rules can participate in calculation when analyzing, optimizing and executing the data query request, and specific comparison judgment is made.

If the field related to the data query request needs to be judged whether the field exists in the target data table or not when the data query request is analyzed, the field can be matched with a fixed field of the target data table in the metadata, if the field is matched, the field exists in the target data table, otherwise, the field is matched on the set dynamic field rule of the metadata, and if the field is matched, the field is considered to exist in the target data table. If the field is not matched, the field is determined not to exist in the target data table, and the error can be directly reported.

In this embodiment, if a field related to a certain data query request matches the set dynamic field information, but the data source does not include a field matching the set dynamic field information, the value of the field is determined to be null.

For data query requests with ambiguity, errors are reported at the time of parsing. For example, the data query request is select abc_1 from users1 join users2, where users1 and 2 are both target data tables with dynamic field definitions, and users1 and 2 both have abc_1 fields. This case may overwrite the data query request with select user1.Abc_1 from users1 join users2.

For the case that the type of field data involved in the data query request is uncertain, whether the field is in a presence function, an expression, a selection field, etc., the type inference and the type conversion can be performed according to the current context. The content of any field can be predetermined as binary or string type, and then type conversion is performed according to the requirement. The value of the current expression is a null value (null) if the conversion fails.

By supporting dynamic field definition, repeated changes of a table structure, control of the width of the table structure and the like can be reduced to the greatest extent, and the requirements of large numbers of data source butt joint of wide tables and super-multiple fields in a data lake scene are supported seamlessly.

Examples ten

Referring to fig. 12, a schematic structural diagram of an electronic device according to a tenth embodiment of the present invention is shown, and the specific embodiment of the present invention is not limited to the specific implementation of the electronic device.

As shown in FIG. 12, the electronic device may include a processor 1202, a communication interface (Communications Interface) 1204, a memory 1206, and a communication bus 1208.

Wherein:

the processor 1202, the communication interface 1204, and the memory 1206 communicate with each other via a communication bus 1208.

A communication interface 1204 for communicating with other electronic devices, such as terminal devices or servers.

The processor 1202 is configured to execute the program 1210 and may specifically perform relevant steps in the above-described data processing method embodiment.

In particular, program 1210 may include program code including computer operating instructions.

The processor 1202 may be a central processing unit CPU, or an Application-specific integrated Circuit ASIC (Application SPECIFIC INTEGRATED Circuit), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the electronic device may be the same type of processor, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs.

Memory 1206 for storing program 1210. The memory 1206 may comprise high-speed RAM memory or may further comprise non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 1210 is specifically configured to cause the processor 1202 to obtain a creation instruction of a target data table, obtain setting dynamic field information from the creation instruction, where the setting dynamic field information is used to indicate a definition rule of a dynamic field in the target data table, create the target data table according to the definition rule indicated in the setting dynamic field information, where the target data table includes a dynamic field defined by the definition rule, and store the setting dynamic field information in metadata of the target data table.

In one possible manner, the definition rules include information defining regular expressions of dynamic fields in the target data table.

In one possible manner, the program 1210 may be further configured to cause the processor 1202 to obtain, from a data query request, information of data to be queried requested by the data query request, where the information of the data to be queried includes information of a target data table and information of a target field, obtain, according to the information of the target field and metadata of the target data table, information of a field in the target field that matches with set dynamic field information in the metadata, or obtain information of a field that matches with the set dynamic field information and information of a fixed field, and obtain, according to the obtained information of the field, an analysis result that analyzes the data query request.

Or alternatively

The program 1210 may be specifically configured to cause the processor 1202 to obtain, from a data query request, information of data to be queried requested by the data query request, where the information of the data to be queried includes information of a target data table and information of a target field, obtain, according to the information of the target field and metadata of the target data table, information of a field in the target field that matches with set dynamic field information in the metadata, or obtain information of a field that matches with the set dynamic field information and information of a fixed field, where the set dynamic field information is used to indicate a definition rule of a dynamic field in the target data table, and obtain, according to the obtained information of the field, an analysis result for analyzing the data query request.

In one possible manner, the program 1210 may be further configured to cause the processor 1202 to determine, when obtaining information of a field of the target fields that matches the set dynamic field information in the metadata based on the information of the target fields and the metadata of the target data table, a field of the target fields that does not match the fixed field in the target data table based on the information of the target fields, match the information of the unmatched field with the set dynamic field information, and obtain information of a field that matches the set dynamic field information based on a result of the matching.

In a possible manner, the program 1210 may be further configured to cause the processor 1202 to generate a query execution instruction corresponding to the data query request according to the analysis result, and obtain, according to the query execution instruction, data of a target field requested by the data query request from a data source corresponding to the target data table.

In a possible manner, the program 1210 may be further configured to, when acquiring, according to the query execution instruction, data of a target field requested by the data query request from a data source corresponding to the target data table, determine, according to the query execution instruction, whether information of the target field includes wild card identifiers for indicating all fields, if so, acquire information of all fields in the data source corresponding to the target data table, match the information of all fields with the set dynamic field information, determine, according to a matching result, a dynamic field in all fields, and acquire, from the data source corresponding to the target data table, data of all fields including the dynamic field.

According to the electronic device of the embodiment, the target data table containing the dynamic fields can be created according to the set dynamic field information in the creation instruction, so that the target data table can seamlessly support the change of the dynamic fields of the data sources, seamless compatibility between the change of the data and the user data query request is realized, the operation of a user is greatly simplified, the performance is saved, and the problem that fields in the data table need to be updated by using DDL (data storage) every time the data sources change in the prior art is solved.

Or aiming at the data query request, acquiring the information of the data to be queried, which is requested by the data query request, wherein the set dynamic field information is used for indicating the definition rule of the dynamic field in the target data table, so that the information of the field matched with the set dynamic field information is acquired by matching the target field with the set dynamic field information, or the information of the field matched with the set dynamic field information and the information of the fixed field are acquired, and the analysis result for analyzing the data query request is acquired according to the acquired information of the field, so that the analysis of the data query request can be completed under the condition of not frequently changing metadata, and the usability and convenience are improved.

In this embodiment, a computer storage medium is provided, on which at least one target data table is stored, where metadata of the target data table includes at least one dynamic field, and the at least one dynamic field is a field created according to a definition rule. The computer storage medium may be any suitable storage medium, such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk, etc.

It should be noted that, according to implementation requirements, each component/step described in the embodiments of the present invention may be split into more components/steps, or two or more components/steps or part of operations of the components/steps may be combined into new components/steps, so as to achieve the objects of the embodiments of the present invention.

The above-described methods according to embodiments of the present invention may be implemented in hardware, firmware, or as software or computer code storable in a recording medium such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and to be stored in a local recording medium downloaded through a network, so that the methods described herein may be stored on such software processes on a recording medium using a general purpose computer, special purpose processor, or programmable or special purpose hardware such as an ASIC or FPGA. It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a memory component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the data processing methods described herein. Further, when a general-purpose computer accesses code for implementing the data processing methods illustrated herein, execution of the code converts the general-purpose computer into a special-purpose computer for executing the data processing methods illustrated herein.

Those of ordinary skill in the art will appreciate that the elements and method steps of the examples described in connection with the embodiments disclosed herein can be implemented as electronic hardware, or as a combination of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments of the present invention.

The above embodiments are only for illustrating the embodiments of the present invention, but not for limiting the embodiments of the present invention, and various changes and modifications may be made by one skilled in the relevant art without departing from the spirit and scope of the embodiments of the present invention, so that all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A data processing method, characterized in that the method comprises:

Obtaining a creation instruction of a target data table, and obtaining setting dynamic field information from the creation instruction, wherein the setting dynamic field information is used to indicate a definition rule of a dynamic field in the target data table;

According to the definition rule indicated in the set dynamic field information, create the target data table, the target data table includes the dynamic fields defined by the definition rule, and save the set dynamic field information into the metadata of the target data table;

Obtaining information of the data to be queried requested by the data query request from the data query request, wherein the information of the data to be queried includes: information of a target data table and information of a target field;

According to the information of the target field and the metadata of the target data table, obtaining the information of the field in the target field that matches the set dynamic field information in the metadata, or obtaining the information of the field that matches the set dynamic field information and the information of the fixed field;

Obtaining a parsing result of parsing the data query request according to the obtained field information;

According to the information of the target field and the metadata of the target data table, obtaining information of a field in the target field that matches the set dynamic field information in the metadata includes:

Determine, according to the information of the target field, the fields in the target field that do not match the fixed fields in the target data table;

The information of the unmatched field is matched with the set dynamic field information, and the information of the field matching the set dynamic field information is obtained according to the matching result.

2 . The method according to claim 1 , wherein the definition rule includes information of a regular expression defining the dynamic field in the target data table.

3. A data processing method, comprising:

According to the information of the target field and the metadata of the target data table, obtaining the information of the field in the target field that matches the set dynamic field information in the metadata, or obtaining the information of the field that matches the set dynamic field information and the information of the fixed field; wherein the set dynamic field information is used to indicate the definition rule of the dynamic field in the target data table;

4 . The method according to claim 3 , wherein the definition rule includes information of a regular expression defining the dynamic field in the target data table.

5. The method according to claim 3, characterized in that the method further comprises:

Generating a query execution instruction corresponding to the data query request according to the analysis result;

According to the query execution instruction, the data of the target field requested by the data query request is obtained from the data source corresponding to the target data table.

6. The method according to claim 5, characterized in that the step of obtaining the data of the target field requested by the data query request from the data source corresponding to the target data table according to the query execution instruction comprises:

Determining, according to the query execution instruction, whether the information of the target field contains a wildcard identifier for indicating all fields;

If included, obtain information of all fields in the data source corresponding to the target data table;

Matching the information of all the fields with the set dynamic field information, and determining the dynamic fields in all the fields according to the matching result;

The data of all fields including the dynamic field are obtained from the data source corresponding to the target data table.

7. A data processing device, comprising:

An instruction receiving module, used for obtaining a creation instruction of a target data table, and obtaining setting dynamic field information from the creation instruction, wherein the setting dynamic field information is used for indicating a definition rule of a dynamic field in the target data table;

A table creation module, configured to create the target data table according to the definition rule indicated in the set dynamic field information, the target data table including the dynamic fields defined by the definition rule, and save the set dynamic field information into the metadata of the target data table;

An acquisition module, used for obtaining information of the to-be-queried data requested by the data query request from the data query request, wherein the information of the to-be-queried data includes: information of a target data table and information of a target field;

A matching module, configured to obtain, according to the information of the target field and the metadata of the target data table, the information of the field in the target field that matches the set dynamic field information in the metadata, or to obtain the information of the field that matches the set dynamic field information and the information of the fixed field;

A first generating module, configured to obtain a parsing result of parsing the data query request according to the obtained field information;

The matching module is specifically used to determine, based on the information of the target field, the fields in the target field that do not match the fixed fields in the target data table; match the information of the unmatched fields with the set dynamic field information, and obtain the information of the fields that match the set dynamic field information based on the matching result.

8 . The device according to claim 7 , wherein the definition rule includes information of a regular expression defining the dynamic field in the target data table.

9. A data processing device, comprising:

a matching module, configured to obtain, according to the information of the target field and the metadata of the target data table, the information of the field in the target field that matches the set dynamic field information in the metadata, or to obtain the information of the field that matches the set dynamic field information and the information of the fixed field; wherein the set dynamic field information is used to indicate the definition rule of the dynamic field in the target data table;

A first generating module, for obtaining a parsing result of parsing the data query request according to the obtained field information;

10 . The device according to claim 9 , wherein the definition rule includes information of a regular expression defining the dynamic field in the target data table.

11. The device according to claim 9, characterized in that the device further comprises:

A second generating module, configured to generate a query execution instruction corresponding to the data query request according to the parsing result;

The data acquisition module is used to acquire the data of the target field requested by the data query request from the data source corresponding to the target data table according to the query execution instruction.

12. The device according to claim 11, characterized in that the data acquisition module comprises:

A wildcard determination module, used to determine whether the information of the target field contains a wildcard identifier for indicating all fields according to the query execution instruction;

A field information acquisition module, used to acquire information of all fields in the data source corresponding to the target data table when included;

A dynamic field determination module, used for matching the information of all the fields with the set dynamic field information, and determining the dynamic field in all the fields according to the matching result;

The field data extraction module is used to obtain data of all fields including the dynamic field from the data source corresponding to the target data table.

13. A database system, characterized by comprising a front-end node and a computing node;

The front node is used to obtain a creation instruction of a target data table and obtain the set dynamic field information from the creation instruction, the front node interacts with the computing node, so that the computing node creates the target data table according to the definition rule indicated in the set dynamic field information, the target data table includes the dynamic fields defined by the definition rule, and saves the set dynamic field information into the metadata of the target data table;

The front node is further used to obtain a data query request, and obtain information of the to-be-queried data requested by the data query request from the data query request, wherein the information of the to-be-queried data includes: information of a target data table and information of a target field; according to the information of the target field and metadata of the target data table, obtain information of a field in the target field that matches the set dynamic field information in the metadata, or obtain information of a field that matches the set dynamic field information and information of a fixed field; according to the obtained field information, obtain a parsing result of parsing the data query request, and the front node interacts with the computing node so that the computing node obtains data of the target field requested by the data query request from a data source corresponding to the target data table according to the parsing result;

The front node is specifically used to determine, based on the information of the target field, the fields in the target field that do not match the fixed fields in the target data table; match the information of the unmatched fields with the set dynamic field information, and obtain the information of the fields that match the set dynamic field information based on the matching result.

14. An electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data processing method as described in any one of claims 1-2, or to perform an operation corresponding to the data processing method as described in any one of claims 3-6.

15. A computer storage medium storing at least one target data table as claimed in any one of claims 1 to 2, wherein metadata of the target data table includes at least one dynamic field, and the at least one dynamic field is a field created according to a definition rule.

16. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the data processing method as claimed in any one of claims 1 to 2, or, when executed, implements the data processing method as claimed in any one of claims 3 to 6.