CN110222110A - A kind of resource description framework data conversion storage integral method based on ETL tool - Google Patents

Abstract

The invention discloses a resource description framework data conversion and storage integration method based on an ETL tool. The steps are as follows: 1. Preprocessing relational data, adding a unique key value through a database as a resource name in the data conversion process; 2. Obtaining the relationship 3. Add RDF output format to the data conversion script to form a complete program; 4. Determine the namespace and namespace prefix according to the data type to be converted, and limit variables Define the scope; 5. Set the subject and instance mapping rules, convert the obtained source data fields into HttpURI; 6. Set the attribute mapping rules; 7. Set the storage format and storage target location, and perform conversion and storage operations. The invention can improve data processing efficiency and coherence; can improve resource scheduling efficiency; can support zero coding and low labor cost data conversion, and has good scalability.

Description

An integrated method for data conversion and storage of resource description framework based on ETL tools

technical field

The invention relates to the field of computer technology, in particular to an integrated method for data conversion and storage of a resource description framework based on an ETL tool.

Background technique

With the development of the Semantic Web, the data published in the form of Resource Description Framework (RDF) surges rapidly and gradually becomes the basis of the Internet. The basic structure of RDF is a triple, including subject, predicate and object, which correspond to resources, attributes and attribute values respectively. RDF data can be associated with other databases to build a shared data space on the Semantic Internet, making the data more available and operable.

At present, due to the transformation from document network to data network, a large amount of data is stored in a relational database (Related Database, referred to as RDB). RDB has certain advantages in small data processing, complex SQL operations and visibility, but lacks semantics. This limits its value in practical application scenarios. Therefore, the conversion of relational data to RDF has attracted the attention of relevant institutions and scholars, and various general or specific domain mapping tools or application software have been developed. For example, D2RQ publishes relational data as RDF graphs to mapping Platform, Virtuoso RDF view converts SQL SELECT query result set into RDF set, and R2RML formulated by W3C realizes the mapping conversion from RDB to RDF by defining vocabulary, mapping rules and syntax, mapping framework and working mechanism. However, at present, various tools or languages have great limitations in the preparation of mapping rule files, scalability and converted data storage mode, and the cost of labor time investment needs to be optimized.

In the way of mapping RDB to RDF, Amazon Cloud (Amazon Cloud) tries to convert the source database into a static method represented by a graph through the Extract-Transform-Load (ETL) process, but in terms of the storage of large data sets Encountered difficulties, and because professional ETL tools do not support RDF output and other reasons, this method does not use related tools, but implements it by coding, which lacks flexibility and scalability. However, ETL tools are widely used in data warehouse construction due to their user-friendliness, rich components, flexible customization rules, and powerful management functions, forming a complete data management ecosystem with a good application environment. The limitations in graph data format support and other aspects limit its application in RDF conversion or generation practice.

Contents of the invention

The present invention aims at the limitations of high time cost, poor scalability, single data storage format and other limitations caused by the dependence on manual code matching in the existing relational database to RDF triple data conversion and storage process, and proposes an ETL-based The resource description framework data conversion and storage integration method of the tool finally realizes highly adaptable RDF triple data conversion and storage. Above-mentioned invention realizes by following technical scheme:

On the one hand, a method for transforming existing ETL tool mapping rules and graph data storage modes is provided, the method comprising:

Step 1. Define conversion rules for mapping tables, columns, and rows in RDB to classes, attributes, and resources in RDF in the ETL tool code layer, and perform functional encapsulation on them;

Step 2. Design the RDF data hybrid storage mode of the ETL tool, which can flexibly choose local or graph database storage, or both, and encapsulate its functions.

Wherein, the conversion rule in which RDB is mapped to RDF is defined as: RDB.table->RDF.class, RDB.column->RDF.attribute, RDB.row->RDF.resource, wherein any instance data Multiple classes can be specified.

On the other hand, an ETL tool-based resource description framework data conversion and storage integration method is provided, the method comprising:

Step 1. Relational data preprocessing, adding a unique key value (UniqueKey) through the database as the resource name in the data conversion process.

Step 2. Obtain the field information of the structured data in the relational database, and determine the necessary fields of the data to be converted. Add a data conversion script in the ETL tool, input and connect to the source database, that is, the relational database that stores the data table to be converted, and query all the fields in the source data table through SQL query, select and obtain the data table according to the pre-built semantic model Mandatory fields to be converted in .

Step 3. Add the RDF output format for the data conversion script to form a complete program.

Step 4. Determine the namespace and namespace prefix according to the type of data to be converted, and limit the scope of variable definition.

Step 5. Set the subject and instance mapping rules, and convert the acquired source data fields into HttpURIs.

Step 6. Set attribute mapping rules, including: obtained source field, object attribute, data attribute, object and predicate instance URI, multi-value delimiter, data type; wherein, the obtained source field is the field selected in step 2; Object attribute, the value range is all classes in the pre-built semantic model, which can be multi-valued; data attribute, the value range is literal value, which can be multi-valued; object and predicate instance URI: unique identifier, corresponding to a namespace in the namespace The entity object of the class; multi-value delimiter: optional for aggregation operations; data type: generate data traceability information, and support users to search for statistical analysis of data based on xml data type.

Step 7. Set the storage format and storage target location, and perform conversion and storage operations.

The advantage of the present invention compared with prior art is:

The present invention applies mainstream ETL tools to the RDF generation process for the first time, integrates a series of processes of data warehouse construction, RDF generation, and RDF storage settings, and improves data processing efficiency and consistency;

The invention transforms the ETL tool based on RDF conversion and storage requirements, so that it supports the RDF triple data output mode, and designs a hybrid storage mode of "local + multiple graph databases", which can improve resource scheduling efficiency;

The modified ETL tool encapsulates the mapping rules and tools in the form of plug-ins, which can support data conversion with zero coding and low labor costs, and can pre-install word segmentation and indexing tools, which has good scalability.

Description of drawings

FIG. 1 is a schematic diagram of the mapping rules from RDB to RDF provided in the present invention.

FIG. 2 is a flow chart of data conversion and storage integration of a resource description framework based on an ETL tool in the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the following will further describe in detail the embodiments of the present invention in conjunction with the accompanying drawings.

Embodiment one

Examples of the present invention provide a method for transforming existing ETL tool mapping rules and graph data storage modes, as shown in Figure 2, the processing flow of the method includes the following steps:

Step 1. Define the conversion rules for mapping tables, columns, and rows in RDB to classes, attributes, and resources in RDF at the ETL tool code layer, as shown in Figure 1, and perform functional encapsulation on them; the details are as follows:

S1.1 Define the namespace and prefix of the class to which the resource belongs, and support common namespaces and customization. Among them, the common namespaces added include:

The xmlns:rdf namespace specifies elements prefixed with rdf from "http://www.w3.org/1999/02/22-rdf-syntax-ns#"

The xmlns:rdfs namespace specifies elements with the prefix rdfs from "http://www.w3.org/2000/01/rdf-schema/"

The xmlns:xsd namespace specifies elements with the prefix xsd from "http://www.w3.org/2001/XM LSchema#"

The xmlns:owl namespace specifies that elements with the prefix owl come from "http://www.w3.org/2002/07/owl#"

The xmlns:dc namespace specifies that elements prefixed with dc come from "http://purl.org/dc/elements/1.1/"

The xmlns:dcterms namespace specifies elements prefixed with dcterms from "http://purl.org/dc/terms/"

S1.2 follows the "Direct Mapping" specification proposed by W3C, and defines simple mapping rules from RDB to RDF at the ETL tool code layer.

In the subject-predicate-object representation of RDF triples, the subject is the resource to be described, which is anything that can have a URI; the predicate is an attribute of a resource, which is a resource with a name; the object is an attribute value, which can be another resource or a string. The classes and properties of RDF are generated directly from the name of the RDB schema, so the simple mapping rules from RDB to RDF are defined as: RDB.Table -> RDF. Class, RDB. Column -> RDF. Property, RDB. Row -> RDF.Resource, where any piece of instance data can specify multiple classes.

S1.3 Design a data record limit function. If the data is not the last piece of data, it needs to be saved in batches and initialize the next file.

Taking the KETTLE tool applicable to the embodiment of the present invention as an example, this example should not bring any limitation to the function and scope of use of the embodiment of the present invention. Define the data record restriction function processRow() in the Java operating environment, and judge whether it is the last piece of data through getSplitEvery.

Step 2. Design the RDF data hybrid storage mode of the ETL tool, which can flexibly choose local or graph database storage, or both, and encapsulate each functional interface based on the ETL tool components.

Embodiment two

Embodiment 1 is the tool preparation for the example of the present invention. The example of the present invention provides a resource description framework data conversion and storage integration method based on ETL tools, including the following processing steps:

Step 1. Relational data preprocessing, adding a unique key value (UniqueKey) through the database as the resource name in the data conversion process.

Add the self-incrementing value as the unique key value in the relational model of the relational database, and it will be automatically generated as the part of the unique resource identifier in the RDF triple during the data conversion process.

Step 2. Add a data conversion script in the ETL tool modified by Embodiment 1, input and connect to the relational database, and obtain the necessary fields of the data to be converted.

Add data conversion script in ETL tool, add table input (table input), be connected to source database (namely store the relational database of data table to be converted), in the implementation case of the present invention, take relational database MySQL as example, through SQL The query gives all the fields in the source data table, selects and obtains the necessary fields to be converted in the data table according to the pre-built semantic model, and extracts the results such as the "journal name" and "issue number" of the journal data.

Step 3. Add RDF triple output (RDF output) to the data conversion script to form a complete program.

Step 4. Add a namespace and a namespace prefix according to the type of data to be converted to limit the scope of element definition.

Namespace (namespace) is a method to resolve name conflicts of elements and attributes aggregated under a common title, which can give different meanings to elements with the same name under different namespaces. In XML, a namespace ensures that all element and attribute names are unique throughout the system, so you don't have to worry about confusion caused by the same element name. In this implementation case, xmlns:rdf, xmlns:rdfs, xmlns:xsd, xmlns:owl, xmlns:dc, xmlns:dcterms, etc. are commonly used. Users can also customize or specify a defined self-owned space according to the pre-built semantic model requirements.

Step 5. Set up the data conversion script program based on the mapping rules.

The conversion of structured data to RDF triples needs to clarify the mapping rules from source data fields to target triples, and configure corresponding parameters to ensure the corresponding conversion of each element, attribute, and instance.

The specific implementation steps are as follows:

(5.1) Add the source field in the data to be converted, set the subject and instance mapping rules, specify the resource type URI (Unified Resource Identifier, Uniform Resource Identifier) and the class of the data to be converted, where the class can have multiple values.

(5.2) Map the field to be converted to the namespace added in step 4, corresponding to an entity object of a certain class in the namespace.

(5.3) Set the predicate and attribute name, object instance URI.

(5.4) Multi-value delimiters can be selected when performing aggregation operations on similar instances.

(5.5) Set the data type, that is, the traceability information of the generated data, which is used for searching based on xml datatype in statistical analysis.

(5.6) Set metadata mapping rules for datasets, including predicates and attribute types, predicates and attribute names, attribute values, and data types.

Step 6. Edit the storage format and storage target location. According to the needs, you can flexibly choose local or graph database storage, or both.

Step 7. Perform data conversion operations and store the converted RDF triples at the same time.

The contents not described in detail in the description of the present invention belong to the prior art known to those skilled in the art.

The above is only a preferred embodiment of the method for integrating resource description framework data conversion and storage based on ETL tools in the present invention. On the premise of the resource description framework data conversion and storage integration method principle of the tool, some improvements and modifications can be made, and these improvements and modifications should belong to the protection scope of the appended claims of the present invention.

Claims (4)

1. a kind of remodeling method to existing ETL tool mapping ruler and diagram data memory module, it is characterised in that: the method Include:

Step 1. ETL instrumentation code layer defines table in RDB, column and row is respectively mapped to class in RDF, the conversion of attribute, resource rule Then, and to it function package is carried out；

The RDF data that step 2. designs ETL tool mixes memory module, and local or chart database storage, or both may be selected It stores simultaneously, and function package is carried out to it.

2. remodeling method according to claim 1, it is characterised in that: the RDB in the step 1, which is mapped in RDF, to be turned Rule is changed, is defined as: RDB. table -> RDF. class, RDB. column -> RDF. attribute, RDB. row -> RDF. resource, wherein any one Instance data all may specify multiple classes.

3. a kind of resource description framework data conversion storage integral method based on ETL tool, it is characterised in that: the storage Integral method includes:

The pretreatment of step 1. relational data, adds unique key assignments by database, as the resource name in data conversion process；

Step 2. obtains the field information of structural data in relevant database, determines the mandatory field of data to be converted；

Step 3. forms complete routine for data conversion script addition RDF output format；

Step 4. determines NameSpace and namespace prefix according to the type of data to be converted, limits variable-definition range；

Subject and example mapping ruler is arranged in step 5., and the source data field corresponding conversion that will acquire is HttpURI；

Step 6. sets a property mapping ruler, comprising: source field, object properties, data attribute, object and the predicate example of acquisition URI, multivalue separator, data type；

Storage format and storage target position is arranged in step 7., executes conversion and storage operation.

4. storage integral method according to claim 3, it is characterised in that: the step 2 specifically: in ETL tool Middle addition data conversion script, input are connected to source database, that is, store the relevant database of tables of data to be converted, pass through SQL query provides all fields in source data table, is selected and is obtained to be converted in tables of data according to the semantic model of prebuild Mandatory field.