WO2021134594A1 - Data processing method and apparatus - Google Patents

Abstract

Embodiments of the present invention provide a data processing method and apparatus. The method comprises: obtaining the metadata of data in a first data system; and converting the metadata of the data in the first data system from a first format into a second format, wherein the first format is the format of the metadata of the data in the first data system, and the second format is a metadata format that can be identified by the first data system and a second data system. The method provided by the embodiments of the present invention is used for defining, for a plurality of data systems, a unified metadata format that can be identified by the plurality of data systems, thereby simplifying data transmission between different data systems.

Description

Data processing method and device Technical field

This application relates to the storage field, and in particular to a data processing method and device.

Background technique

A data lake is a repository or system that stores data in its original format, without prior structured processing of the data. A data lake can store structured data (such as tables in a relational database), semi-structured data (such as CSV, logs, XML, JSON), unstructured data (such as emails, documents, PDF), and binary data ( Such as graphics, audio, video). In order to manage the data in the data lake, it is required that all data entering the data lake must provide related metadata, and organize the metadata of different types of data into a directory service. When using the data in the future, the data can be analyzed based on the directory service. Provide users with analyzed data.

Currently, the directory services in each data lake are independent of each other, and the directory services used by each data lake are from different vendors. For example, in the security field, each province or city purchases data lake products from different vendors, and data lake products from different vendors have different catalog services. Therefore, in higher-level units such as the province or the central government, they want to compare data lake products from different vendors. For example, when the data of cities or provinces are integrated, they cannot be directly integrated because they cannot identify the directory services of each city or province. One current implementation method is: If the directory services of data lake A (for example, province) and data lake B (for example, city) are different, but data lake A needs to obtain data from data lake B, then data lake A needs to notify data lake B , Tell Data Lake B the data it wants to access. Data Lake B puts the metadata of the data to be accessed by Data Lake A on the exchange platform, and the exchange platform converts the metadata of the data to be accessed by Data Lake A into a directory service that can be identified by Data Lake A, and informs Data Lake A from the said The exchange platform obtains the metadata, and then obtains the data in the data lake B based on the metadata.

In the above implementation, the metadata format defined in the catalog service of Data Lake A needs to be directly converted to the metadata format defined in the catalog service of Data Lake B. In this way, data lake B needs to obtain multiple data with different When the data of the data lake of the catalog service is used, the exchange platform needs to know the catalog service of all the data lakes, which leads to a relatively high complexity of metadata format conversion.

Summary of the invention

The invention provides a data processing method and device. The invention simplifies the communication between the two data systems by defining the metadata format that can be recognized by the two data systems.

A first aspect of the present invention provides a data processing method. The method includes: acquiring metadata of data in a first data system; and converting the metadata of data in the first data system from a first format to a second format. Format, the first format is a metadata format of data in the first data system, and the second format is a metadata format that can be recognized by both the first data system and the second data system.

In the data processing method provided by the embodiment of the present invention, a unified metadata format is defined for different data systems, so that when the first data system needs to obtain data from the second data system, the second data system first The metadata format is converted to the metadata format defined by the catalog template. After the first data system obtains the metadata that has been converted into a unified format, it then converts it into the metadata format of the first data system. In this way, each data system As long as the unified metadata format can be recognized, data can be transferred with other data lakes, which simplifies the communication between different data systems.

In a possible implementation of the first aspect, the method further includes predefining a metadata template, and the metadata format adopted by the metadata template is the second format.

In a possible implementation of the first aspect, the method further includes: sending the metadata converted into the second format to the messaging platform, so that the second data system obtains the converted into the message platform from the messaging platform. Metadata in the second format.

The second data system sends the metadata to the messaging platform. In this way, the first data system can obtain the metadata of the second system from the messaging platform, so there is no need to actively obtain the metadata from the second data system.

In a possible implementation of the first aspect, the sending the metadata converted into the second format to the messaging platform includes: judging whether the metadata converted into the second format meets a preset publishing rule;

When it is determined that the metadata converted into the second format meets a preset publishing rule, the metadata converted into the second format is sent to the messaging platform.

By setting metadata publishing rules in the second data system, only metadata that meets the publishing rules will be published to the messaging platform. In this way, authorization authentication is not required, and data security can also be ensured.

In a possible implementation manner, the method further includes:

Send a release metadata queue creation request to the message platform, so that the message platform establishes a release metadata

Data queue

Sending the metadata converted into the second format to the messaging platform includes:

The metadata converted into the second format is written into the publishing metadata queue of the messaging platform.

A data federation is formed in the first data system and the second data system, and a publishing metadata queue is established. The data transmission channel between the first data system and the second data system is established through the publishing metadata queue, which optimizes the data transmission process.

In a possible implementation manner, the method further includes:

Obtain the address of the publishing metadata queue sent by the messaging platform, and send the address of the publishing metadata queue to the second data system.

In a possible implementation manner, the method further includes:

Sending an original metadata queue creation request to the message platform, so that the message platform establishes an original metadata queue;

Converting the format of the metadata in the first data system from the first format to the second format includes:

Converting the metadata of the first data system in the original metadata queue from the first format to the second format.

By establishing the original metadata queue, a buffer can be provided for the converted data, which ensures the reliability of the data.

A second aspect of the present invention provides a data processing method, the method comprising: acquiring metadata of data of a first data system; and converting the format of the data metadata of the first data system from a second format to a third format The second format is a metadata format that can be recognized by both the first data system and the second data system, and the third format is a metadata format of data in the second data system.

In the data processing method provided by the embodiment of the present invention, a unified metadata format is defined for different data systems, so that when the first data system needs to obtain data from the second data system, the second data system first The metadata format is converted to the metadata format defined by the catalog template. After the first data system obtains the metadata that has been converted into a unified format, it then converts it into the metadata format of the first data system. In this way, each data system As long as the unified metadata format can be recognized, data can be transferred with other data lakes, which simplifies the communication between different data systems.

In a possible implementation manner, the method includes predefining a metadata template, and the metadata format adopted by the metadata template is the second format.

In a possible implementation manner, the acquiring metadata of the first data system includes:

The metadata of the data of the first data system is obtained from the message platform.

Obtaining the metadata of the first data system through the messaging platform allows the second data system to obtain the metadata of the first system without accessing the first system.

In a possible implementation manner, the method further includes:

Judging whether the metadata meets preset receiving rules;

When it is determined that the converted metadata satisfies the preset receiving rule, the format of the metadata of the data of the first data system is converted from the first format to the second format.

By setting the receiving rule, data not needed by the second data system can be filtered out, thereby ensuring data security.

In a possible implementation manner, the method further includes:

Receiving the address of the publishing metadata queue in the message platform sent by the first data system;

The obtaining the metadata of the data of the first data system from the message platform includes:

Acquiring the metadata from the publishing metadata address according to the address of the publishing metadata queue.

A third aspect of the present invention provides a data processing device, the data processing device includes a plurality of modules, and the plurality of modules are used to perform the functions of each step in the data processing method of the first aspect. The beneficial effects achieved are also the same as the beneficial effects of the various methods provided in the first aspect, and will not be repeated here.

A fourth aspect of the present invention provides a data processing device, the data processing device includes a plurality of modules, and the plurality of modules are used to perform the functions of each step in the data processing method of the second aspect. The beneficial effects achieved are also the same as the beneficial effects of the various methods provided in the first aspect, and will not be repeated here.

A fifth aspect of the present invention provides a server. The server includes a processor and a memory, where program instructions are stored in the memory, and the processor executes the program instructions to implement the functions of the method provided in the first aspect.

A sixth aspect of the present invention provides a server. The server includes a processor and a memory, where program instructions are stored in the memory, and the processor executes the program instructions to implement the functions of the method provided in the second aspect.

A seventh aspect of the present invention provides a storage medium in which program instructions are stored, and the program instructions are executed by a processor to implement the functions of the method provided in the first aspect.

An eighth aspect of the present invention provides a storage medium in which program instructions are stored, and the program instructions are executed by a processor to implement the functions of the method provided in the second aspect.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art.

FIG. 1 is a schematic diagram of the structure of a data lake provided by an embodiment of the present invention.

Fig. 2 is a format of metadata in a directory template defined in an embodiment of the present invention.

Fig. 3 is a structural diagram of a federated system in an embodiment of the present invention.

Fig. 4 is a flowchart of a method for registering as a federation between data lakes in an embodiment of the present invention.

Fig. 5 is a flowchart of a method for data transmission between data lakes in an embodiment of the present invention.

Fig. 6 is the metadata format of the Hive database in the embodiment of the present invention.

FIG. 7 is a schematic diagram of the metadata format of a Hive database in a data lake being converted into a metadata format defined by a catalog template in an embodiment of the present invention.

FIG. 8 is a schematic diagram of the metadata format defined by the catalog template in the embodiment of the present invention being converted to the metadata format in another data lake.

FIG. 9 is a schematic diagram of an application scenario of forming a data federation for multiple data lakes in an embodiment of the present invention.

Fig. 10 is a functional block diagram of a first data processing device and a second data processing device in an embodiment of the present invention.

Fig. 11 is a structural diagram of a server operated by a federated service in an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

Metadata: also known as intermediary data and relay data. It is data describing data, mainly information describing data attributes, used to support functions such as indicating storage location, historical data, resource search, and file recording. Metadata can be regarded as an electronic catalog. In order to achieve the purpose of compiling the catalog, the attributes of the data must be described, so as to achieve the purpose of assisting data retrieval.

Metadata format: defines the attributes included in the metadata, the expression of each attribute, and the order in which the attributes are arranged.

The data lake provided by the embodiment of the present invention can be deployed in a server cluster or in a cloud environment, as shown in FIG. 1, which is a schematic diagram of the structure of the data lake deployed in a cloud environment.

The data lake mainly includes two parts, one part is computing resources 10, and the other part is storage resources 20. The storage resource 20 generally includes multiple storage devices 201, and the data stored in the data lake is generally distributed and stored in the multiple storage devices 201. The computing resource 10 generally uses a virtual machine 101 in a cloud environment, and the virtual machine 101 is deployed in a computing server (not shown). When data is stored in the data lake, the virtual machine 101 determines the storage device where the data is stored, obtains metadata of the data, and records the metadata in the directory service 102.

If the data lake is deployed in a server cluster, the computing resources of the data lake are at least one computing server, and the storage resources are storage servers.

At present, there are many vendors on the market that provide data lake products, and the directory services set by the data lake products of different vendors are not the same. However, in some scenarios, data between data lakes need to be able to flow between each other, for example, in the first grade of high school. When a level unit such as a province or the central government wants to integrate the data of a lower level unit such as a city or province, it cannot be directly integrated because it cannot identify the directory service of each city or province. One current implementation method is: If the directory services of data lake A (for example, province) and data lake B (for example, city) are different, but data lake A needs to obtain data from data lake B, then data lake A needs to notify data lake B , Tell Data Lake B the data it wants to access. Data Lake B needs to confirm whether Data Lake A has the right to access the data it wants to access. If so, put the data and metadata to be accessed by Data Lake A on the exchange platform, and the exchange platform will store the metadata of the data that Data Lake A wants to access. Convert it into metadata that can be identified by Data Lake A, and notify Data Lake A to obtain the metadata from the exchange platform, and obtain data from Data Lake A based on the obtained metadata.

In the above implementation, the metadata format defined in the catalog service of Data Lake A needs to be directly converted to the metadata format defined in the catalog service of Data Lake B. In this way, data lake B needs to obtain multiple data with different When the data of the data lake of the catalog service is used, the exchange platform needs to know the catalog service of all the data lakes, which leads to a relatively high complexity of metadata format conversion.

In addition, data lake A needs to actively obtain data from data lake B, so data lake A needs to know in advance what data is in data lake B. When the scale of data lake is relatively large, it is difficult to predict what data is in data lake B.

In addition, every time data lake A obtains data from data lake B, data lake B also needs to authenticate data lake A, which affects the efficiency of data access.

The data processing method provided by the embodiment of the present invention is provided with a catalog template, which defines a unified metadata format for data lakes that adopt different catalog services, and all data lakes can recognize the metadata format defined in the catalog template . In this way, when data lake A needs to obtain data from data lake B, data lake B first converts the metadata format in the catalog service of data lake B into the metadata format defined by the catalog template, and the acquisition of data lake A is converted into catalog After the metadata in the format defined by the template, it is converted to the metadata format in the catalog service of Data Lake A. In this way, as long as each data lake can recognize the metadata format defined in the catalog template, it can be combined with other data. Data transmission between data lakes simplifies the communication between different data lakes.

In addition, Data Lake A and Data Lake B will form a data federation. After forming a data federation, when data lake B generates new metadata, data lake B will publish the newly generated metadata to the messaging platform, and the messaging platform will receive After the metadata released by Data Lake B, the received metadata will be pushed to Data Lake A. In this way, Data Lake A does not need to actively obtain metadata from Data Lake B.

In addition, metadata publishing rules will be set in Data Lake B. Only metadata that meets the publishing rules will be published to the messaging platform. Data Lake A also has metadata subscription rules. Only metadata that meets the subscription rules will be published. It is stored in Data Lake A. In this way, there is no need for permission authentication, and data security can also be ensured.

Since the data lake includes multiple types of data, such as Hive database and Hbase database that belong to structured data, and video and audio that belong to unstructured data, different types of data are set in the directory service. Metadata format. For different vendors, when establishing the catalog service of the data lake, the description format of the metadata of each type of data will be different. Therefore, the embodiment of the present invention provides a catalog template for providing catalog services of different vendors. The format of metadata in is converted to a unified metadata format.

As shown in FIG. 2, it is a schematic diagram of the metadata format of the Hive database in the directory template defined in the embodiment of the present invention.

In the template shown in Figure 2, the expression and sequence are defined for each attribute of the metadata. For example, if the table name for Hive metadata in the directory service in Data Lake A is represented by TalX and the column is represented by ColX, the table name for Hive metadata in the directory service in Data Lake B is represented by TalY. The column is represented by ColX. After being transformed into the format defined by the catalog template, the table name is represented by Name, and the column is represented by Column.

As shown in Figure 3, it is a schematic diagram of two data lakes forming a federation. In the embodiment of the present invention, the data lake 30 and the data lake 31 provide a directory service service 301 and a federation service 302 at the same time. The directory service service 301 and the federation service 302 can be provided by different virtual machines in a cloud environment. It can also be different processes in the same virtual machine; in a server cluster, it can be provided by different servers, or it can be different processes in the same server. The message platform 33 is used to provide queue services for the data lake 30 and the data lake 31, and provide the original metadata queue 331 and the publishing metadata queue 332 for the data lakes 30 and 31 to facilitate metadata between the data lake 30 and the data lake 31 exchange. The message platform may be a virtual machine cluster composed of multiple virtual machines, or a server cluster composed of multiple servers.

Please also refer to FIG. 4, which is a flowchart of a method for registering with the data lake 31 when the data lake 30 needs to form a data federation with the data lake 31 to obtain data in the data lake 31.

In step S401, the federated service 302 of the data lake 30 sends a registration request to the virtual machine (which may also be a server or process, hereinafter referred to as federated service) that provides federated services in the data lake 31.

In the embodiment of the present invention, the application of the federation service will be installed in the data lake. When the data lake 30 needs to obtain the data in the data lake 31, the user will start the application of the federation service 302 and select the required The federal data lake 31 is established, and then a registration request is sent to the data lake 31 through the registration function provided by the federal service, and the registration request carries the address of the federal service of the data lake 30.

Step S402: After the federation service 312 of the data lake 31 receives the registration request, it sends an original metadata queue creation request and a release metadata queue creation request to the message platform 33, and simultaneously sends the federation service 312 and the federation service 312 to the message platform 33. The address of the service 302 is sent to the message platform 33.

In step S403, the message platform 33 receives the original metadata queue creation request and the publishing metadata queue creation request, and establishes the original metadata queue and the publishing metadata queue respectively.

The original metadata queue and the publishing metadata queue are processes running in the messaging platform 33. When creating the original metadata queue and the publishing metadata queue, the messaging platform allocates a certain amount of storage space for each queue. The original metadata queue is used to store metadata newly generated in the data lake 30, and the publishing metadata queue is used to store metadata that the data lake 30 needs to publish. The process of storing metadata in the original metadata queue and publishing the metadata queue will be described below.

Step S404: After the original metadata queue and the publishing metadata queue are established, the message platform 33 returns the addresses of the original metadata queue and the publishing metadata queue to the federated service 312, the The federation service 312 sends the address of the publishing metadata queue to the federation service 302, and returns the address of the original metadata queue to the directory service 311. In this way, the federation service 302 or the directory service 311 respectively obtains metadata from the publishing metadata queue or the original metadata queue according to the address. In this way, a data federation is formed between the data lake 30 and the data lake 31.

The following will combine Figure 5 to form the process of data transmission between the data lakes of the federation.

As shown in FIG. 5, after the data lake 30 and the data lake 31 form a data federation, metadata can be transmitted between the data lake 30 and the data lake 31, and the data transmission process is shown in FIG. 5.

In step S501, a virtual machine (which may also be a process or a server, which is referred to as a directory service hereinafter) that provides the directory service 311 detects new metadata generated in the directory service in the data lake 31.

When a user generates a new table or file in the data lake 31, the directory service 311 obtains and stores the metadata of the newly generated table or file. In the embodiment of the present invention, a plug-in is added to the directory service to detect whether the directory service has acquired new metadata.

In step S502, the directory service 311 sends a metadata write request to the message platform 33 based on the newly generated metadata according to the address of the original metadata queue.

When establishing a data federation, the federation service 312 sends the address of the original metadata queue to the directory service. Therefore, when the directory service 311 generates new metadata, a metadata write request can be generated to transfer the The metadata write request is sent to the message platform 33.

In step S503, the message platform 33 writes the newly generated metadata into the original metadata queue.

In step S504, the message platform 33 notifies the federation service 312 of the data lake 31 that there is new source data generated in the original metadata queue.

In step S505, the federation service 301 obtains the metadata from the original metadata queue, and then converts the format of the metadata into the metadata format defined in the catalog template.

For example, if the metadata is the metadata in the Hive database, in the data lake 31, the expression of the metadata is shown in FIG. 6, and it is passed through the Hive metadata defined in the catalog template shown in FIG. After the data format is converted, as shown in Figure 7. The corresponding relationship between each attribute in the metadata template and each attribute in the Hive metadata is predefined in the federation service 312. During conversion, first read the metadata template shown in Figure 2, then obtain the value corresponding to the attribute in the Hive metadata that is the same as the attribute in the metadata template, and fill the obtained value into the metadata template corresponding For example, the attribute "Name" in the Hive metadata and the attribute "table name" in the metadata template represent the same attribute, and the value of the attribute "Name" is obtained from the Hive metadata and written in "Hive-table1" In the attribute "table name" defined by the metadata template. Doing similar operations for other attributes can convert the format of the metadata of the Hive database in the data lake 31 shown in FIG. 6 into the format defined by the metadata template shown in FIG. 7. In addition, the value of some attributes can also be simplified by only extracting key information. For example, for the DB attribute in Figure 6, the expression in the data lake 31 is more complicated, and it also includes metadata identifying the database to which this table belongs. When converting to the format of the metadata template, only the database identifier "hive_db" can be extracted.

Step S506: Determine whether the metadata can be published to the federal data lake 30 of the data lake 31 according to the publishing rule.

In the embodiment of the present invention, the metadata published to the federal data lake 30 is filtered, and only data that meets the publishing rules will be published to the federal data lake. The publishing rule is set by the user according to the attributes of the metadata in the catalog template. The publishing rules include rules and actions.

For example, the rule setting of the publishing rule is: creator=Zhang San&generation time>2019/9/18&table type=financial, and the action is set to publish, which means that data that meets the rule can be published.

In the rules, the creator, generation time, and table type are all attributes defined in the metadata template. According to the rules, after September 18, 2019, after Zhang San creates a table about finances, the metadata information related to this table will be synchronized to other data lakes that form the federation.

For another example, the rule of the publishing rule is set as: metadata source=XX company, and the action is set to not publish. If the metadata comes from XX company, it will not be synchronized to the federal data lake 30 of the data lake 31.

Step S507: When it is determined according to the publishing rule that the metadata can be published to the data lake 31, the federated service 312 sends the metadata to the messaging platform 33.

When the data lake 30 is registered as the federal data lake of the data lake 31, the data lake 31 applies for the establishment of a publishing metadata queue in the message platform 33, and returns the address of the publishing metadata queue to the federal service 312 of the data lake 31. In this way, when the federation service 312 sends the metadata to the messaging platform 33, it can carry the address of the metadata publishing queue.

In step S508, the message platform 33 stores the received metadata in the publishing metadata queue.

In step S509, the message platform 33 notifies the federation service 302 of the data lake 30 to obtain data from the publishing metadata queue.

During the registration process of the Federal Data Lake, the federation service 312 passed the address of the federation service 302 to the messaging platform, so the messaging platform can notify the federation according to the address of the federation service 302 The service 302 obtains the metadata.

In step S510, the federation service 302 of the data lake 30 obtains the metadata from the publishing metadata queue according to the address of the publishing metadata queue.

During the registration process of the Federal Data Lake, the federation service 312 sends the address of the publishing metadata queue to the federation service 302, so the federation service 302 can send the address of the publishing metadata queue from all sources. Obtain the metadata from the release metadata queue.

In step S511, the federation service 302 determines whether the metadata is the data required by the data lake B according to the subscription rule.

The subscription rules are also set according to the attributes of the metadata in the directory template. The subscription rules also include two parts: rules and actions. If the acquired metadata meets the set rules, corresponding actions are executed.

For example, the rule of the subscription rule can be set as: metadata source=XX, and the action is set to not receive, which means that if the metadata obtained from Data Lake A is XX, the metadata will not be received.

By setting subscription rules, data that is not needed by the data lake 30 can be filtered out.

In step S512, the federation service 302 converts the metadata into the format of the metadata in the data lake 30.

The federation service 302 defines the correspondence between each attribute of the metadata in the data lake 30 and each attribute of the metadata in the metadata template. Through the defined corresponding relationship, the metadata format in the metadata template can be converted into the metadata format in the data lake 30, and the converted format is shown in FIG. 8.

In step S513, the federation service 302 sends the converted metadata to the directory service 301 of the data lake 30.

In step S514, the catalog service 301 of the data lake 30 stores the converted metadata into the catalog service of the data lake 30.

The data processing method provided by the embodiment of the present invention is provided with a catalog template, which defines a unified metadata format for data lakes that adopt different catalog services, and all data lakes can recognize the metadata format defined in the catalog template . In this way, when data lake A needs to obtain data from data lake B, data lake B first converts the metadata format in the catalog service of data lake B into the metadata format defined by the catalog template, and the acquisition of data lake A is converted into catalog After the metadata in the format defined by the template, it is converted to the metadata format in the catalog service of Data Lake A. In this way, as long as each data lake can recognize the metadata format defined in the catalog template, it can be combined with other data. Data transmission between data lakes simplifies the communication between different data lakes.

In addition, Data Lake A and Data Lake B will form a data federation. After forming a data federation, when data lake B generates new metadata, data lake B will publish the newly generated metadata to the messaging platform, and the messaging platform will receive After the metadata released by Data Lake B, the received metadata will be pushed to Data Lake A. In this way, Data Lake A does not need to actively obtain metadata from Data Lake B.

In addition, metadata publishing rules will be set in Data Lake B. Only metadata that meets the publishing rules will be published to the messaging platform. Data Lake A also has metadata subscription rules. Only metadata that meets the subscription rules will be published. It is stored in Data Lake A. In this way, there is no need for permission authentication, and data security can also be ensured.

Since the data lake includes multiple types of data, such as Hive database and Hbase database that belong to structured data, and video and audio that belong to unstructured data, different types of data are set in the directory service. Metadata format. For different vendors, when establishing the catalog service of the data lake, the description format of the metadata of each type of data will be different. Therefore, the embodiment of the present invention provides a catalog template for providing catalog services of different vendors. The format of metadata in is converted to a unified metadata format.

In practical applications, the data lake 31 may be a lower-level unit such as a city or provincial data lake, and the data lake 30 may be a higher-level unit such as a provincial or central data lake. In this way, since the data lake 30 can directly obtain data from the data lake 31, it is convenient to integrate the data of lower-level units.

As shown in FIG. 9, it is an application scenario of an embodiment of the present invention. Data Lake B forms a catalog service federation with Data Lake D and Data Lake E, then Data Lake B can obtain metadata in Data Lake D and Data Lake E respectively, and Data Lake A forms a catalog with Data Lake B and Data Lake C respectively Service federation, data lake A can obtain metadata in data lake D and data lake E respectively. If Data Lake D and Data Lake E are the data lakes of municipal units, Data Lake B and Data Lake C are the data lakes of provincial units, and Data Lake A is the data lake of the central unit. The metadata of the unit is integrated, and the directory services between units of the same level are isolated from each other.

The above embodiments only take the data lake as an example for description, but the present invention is also applicable to other data systems, such as databases, data warehouses, and so on. Both the database and the data warehouse have metadata, so the implementation method is basically the same as that of the data lake, so I won't repeat it here.

As shown in FIG. 10, it is a functional block diagram of the first data processing device 1001 and the second data processing device 2001. The first data processing device 1001 and the second data processing device 2001 are mainly used to implement the functions of the federated service 302 and the federated service 312.

The first data processing device 1001 includes a definition module 1002, a creation module 1003, an acquisition module 1004, a conversion module 1005, and a publishing module 1006. The second data processing device includes a definition module 2001, an acquisition module 2002, and a conversion module 2003.

The definition module 1002 and the definition module 2002 are used to define the format of the metadata in the catalog template shown in FIG. 2 in the first data processing device 1001 and the second data processing device 2001, respectively. For the definition method, please refer to the description of Figure 2.

The creation module 1003 is configured to create an original metadata queue and publish a metadata queue in the messaging platform after receiving the registration request of the second data processing device 2001. For the specific creation process, please refer to the relevant description in FIG. 4. I won't repeat them here.

The acquisition module 1004 is used to acquire the metadata of the data in the data lake 30. For a specific acquisition method, please refer to the description of step S501 to step S505 in FIG. 5, which will not be repeated here.

The conversion module 1005 is used to convert the metadata format of the data in the data lake 30 into the metadata format defined in the catalog template. For the specific conversion method, please refer to the description of step S505 in FIG. 5, which will not be repeated here.

The publishing template 1006 is used to publish the metadata converted into the metadata format defined in the catalog template to the publishing metadata queue of the messaging platform. For details, please refer to the description of steps 507 and 508 in FIG. 5, which will not be repeated here.

After receiving the notification from the message platform, the acquiring module 2003 of the second data processing device 2001 acquires the metadata converted into the format defined by the catalog template from the publishing metadata queue of the message platform. For the process, please refer to the related description of step S510, which will not be repeated here.

The conversion module 2004 is configured to filter the acquired metadata according to preset filtering rules. When it is determined that the metadata is data that can be received by the data lake 31, the format of the acquired metadata is changed from the catalog The format defined by the template is converted into the metadata format of the data in the data lake 31, and the converted metadata is stored in the data directory of the data lake 31. For details, please refer to the description of step S511 to step S514, which will not be repeated here. Go into details.

When any of the above modules or units are implemented by software, the software exists in the form of computer program instructions and is stored in the memory, and the processor can be used to execute the program instructions to implement the above method flow. The processor may include, but is not limited to, at least one of the following: central processing unit (CPU), microprocessor, digital signal processor (DSP), microcontroller (microcontroller unit, MCU), or artificial intelligence Various computing devices such as processors that run software. Each computing device may include one or more cores for executing software instructions to perform operations or processing. The processor can be a single semiconductor chip, or it can be integrated with other circuits to form a semiconductor chip. For example, it can form an SoC (on-chip) with other circuits (such as codec circuits, hardware acceleration circuits, or various bus and interface circuits). System), or it can be integrated into the ASIC as a built-in processor of an ASIC, and the ASIC integrated with the processor can be packaged separately or together with other circuits. In addition to the core used to execute software instructions for calculation or processing, the processor may further include necessary hardware accelerators, such as field programmable gate array (FPGA) and PLD (programmable logic device) , Or a logic circuit that implements dedicated logic operations.

When the above modules are implemented by hardware circuits, the hardware circuits may be general-purpose CPU (Central Processing Unit, Central Processing Unit), MCU (Micro Controller Unit, Microcontroller), MPU (Micro Processing Unit, Microprocessor), DSP (Digital signal processing, digital signal processor), SoC (System on Chip, system-on-chip) to achieve, of course, it can also be implemented with application-specific integrated circuit (ASIC), or programmable logic device (programmable logic) device, PLD). The above-mentioned PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (generic array logic, GAL) or its In any combination, it can run necessary software or does not rely on software to execute the above method flow.

As shown in Figure 11, it is a hardware structure diagram of a server. The server is a server operated by a federated service in the data lake 30 or the data lake 31. The server 1101 includes a processing unit 1102, a storage unit 1103, and a communication unit 1104.

The processing unit 1101 has multiple implementation forms, for example, it can be a central processing unit (CPU) or graphics processing unit (GPU), it can be a single-core processor or a multi-core processor, if it is implemented as multi-core processing The number of processors is not limited.

The storage unit 1103 may be a dynamic random access memory (Dynamic Random Access Memory, DRAM), or may be a storage class memory (storage class memory, SCM). The storage unit 1103 is used to store program codes and data so that the processing unit can call and execute to implement related functions. In the embodiment of the present invention, the program code corresponding to the federated service may be stored. The processor calls the program code and has executed the function performed by the federated service of the data lake 30 or the federated service of the data lake 31 in the method shown in FIG. The function performed.

The communication unit 1104 is used to communicate with the messaging platform and other servers for data transmission.

The above are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention. It should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (24)

A data processing method, characterized in that the method includes: Acquiring metadata of data in the first data system; Convert the metadata of the data in the first data system from a first format to a second format, where the first format is the format of the metadata of the data in the first data system, and the second format is A metadata format that can be recognized by both the first data system and the second data system. The data processing method according to claim 1, further comprising: A metadata template is predefined, and the metadata format adopted by the metadata template is the second format. The data processing method according to claim 1, wherein the method further comprises: The metadata converted into the second format is sent to the messaging platform, so that the second data system obtains the metadata converted into the second format from the messaging platform. The data processing method according to claim 3, wherein the sending the metadata converted into the second format to the messaging platform comprises: Judging whether the metadata converted into the second format meets a preset publishing rule; When it is determined that the metadata converted into the second format meets a preset publishing rule, the metadata converted into the second format is sent to the messaging platform. The data processing method according to claim 3, further comprising: Sending a publishing metadata queue creation request to the messaging platform, so that the messaging platform can establish a publishing metadata queue; Sending the metadata converted into the second format to the messaging platform includes: The metadata converted into the second format is written into the publishing metadata queue of the messaging platform. The data processing method according to claim 5, further comprising: Obtain the address of the publishing metadata queue sent by the messaging platform, and send the address of the publishing metadata queue to the second data system. The data processing method according to claim 5, further comprising: Sending an original metadata queue creation request to the message platform, so that the message platform establishes an original metadata queue; Converting the format of the metadata in the first data system from the first format to the second format includes: Converting the metadata of the first data system in the original metadata queue from the first format to the second format. A data processing method, characterized in that the method includes: Acquiring metadata of the data of the first data system; Converting the format of the data metadata of the first data system from a second format to a third format, where the second format is a metadata format that can be recognized by both the first data system and the second data system, The third format is a format of metadata of data in the second data system. 8. The data processing method according to claim 8, further comprising: predefining a metadata template, and the metadata format adopted by the metadata template is the second format. . The data processing method according to claim 8, wherein: The acquiring metadata of the first data system includes: The metadata of the data of the first data system is obtained from the message platform. The data processing method according to claim 8, further comprising: Judging whether the metadata meets preset receiving rules; When it is determined that the converted metadata satisfies the preset receiving rule, the format of the metadata of the data of the first data system is converted from the first format to the second format. The data processing method according to claim 10, further comprising: Receiving the address of the publishing metadata queue in the message platform sent by the first data system; The obtaining the metadata of the data of the first data system from the message platform includes: Acquiring the metadata from the publishing metadata address according to the address of the publishing metadata queue. A data processing device, characterized in that the device comprises: An acquisition module, used to acquire metadata of data in the first data system; The conversion module is used to convert the metadata of the data in the first data system from a first format to a second format, where the first format is the format of the metadata of the data in the first data system, so The second format is a metadata format that can be recognized by both the first data system and the second data system. The data processing device according to claim 13, further comprising: The definition module is used to predefine a metadata template, and the metadata format adopted by the metadata template is the second format. The data processing device according to claim 13, wherein the device further comprises: The publishing module is configured to send the metadata converted into the second format to the message platform, so that the second data system obtains the metadata converted into the second format from the message platform. The data processing device according to claim 15, wherein the publishing module is further configured to: Judging whether the metadata converted into the second format meets a preset publishing rule; When it is determined that the metadata converted into the second format meets a preset publishing rule, the metadata converted into the second format is sent to the messaging platform. The data processing device of claim 15, further comprising: A creation module, configured to send a release metadata queue creation request to the message platform, so that the message platform establishes a release metadata queue; The publishing module is specifically configured to write the metadata converted into the second format into the publishing metadata queue of the messaging platform. The data processing device according to claim 17, wherein the creation module is specifically configured to: Obtain the address of the publishing metadata queue sent by the messaging platform, and send the address of the publishing metadata queue to the second data system. The data processing device according to claim 17, wherein the creation module is further configured to: Sending an original metadata queue creation request to the message platform, so that the message platform establishes an original metadata queue; The conversion module is specifically configured to: convert the metadata of the first data system in the original metadata queue from the first format to the second format. A data processing device, characterized in that the device comprises: An acquisition module for acquiring metadata of the data of the first data system; A conversion module for converting the format of the data metadata of the first data system from a second format to a third format, where the second format is identifiable by both the first data system and the second data system The third format is the metadata format of the data in the second data system. The data processing device according to claim 20, further comprising: The definition module is used to predefine a metadata template, and the metadata format adopted by the metadata template is the second format. The data processing device according to claim 20, wherein the acquisition module is specifically configured to: The metadata of the data of the first data system is obtained from the message platform. The data processing device according to claim 20, wherein the conversion module is further configured to: Judging whether the metadata meets preset receiving rules; When it is determined that the converted metadata satisfies the preset receiving rule, the format of the metadata of the data of the first data system is converted from the first format to the second format. The data processing device according to claim 20, wherein the acquisition module is specifically configured to: Receiving the address of the publishing metadata queue in the message platform sent by the first data system; Acquiring the metadata from the publishing metadata address according to the address of the publishing metadata queue.

Images