KR20230091749A - Mehtod and device for generating datamap - Google Patents

Abstract

Disclosed are a method and device for generating a data map to describe various metadata information about a dataset. According to the present invention, the method for generating a data map comprises: a step of generating, by a relational DB data map generation module, a relational data map describing asset-related metadata of a catalog classified according to a classification system and a category in a preset standard and using a standard vocabulary to convert the relational data map to an RDF triple structure; and a step of transmitting, by a graphic DB data map generation module, change data related to a change event in the relational data map to a message queue and using a standard vocabulary for the change data input to the message queue to generate a graph-type data map mapping nodes, node attributes, relationships between the nodes.

Description

Method and device for generating data map {MEHTOD AND DEVICE FOR GENERATING DATAMAP}

The present disclosure relates to a method and apparatus for generating a data map, and more specifically, data describing various metadata information about a dataset to support mutual sharing and utilization of various datasets generated in various areas such as the public and private sectors. It relates to a method and apparatus for generating a map.

Data-based convergence business models and the frequency of new data are rapidly increasing in various fields at home and abroad, and the demand for data sharing between the public and private sectors is growing. Accordingly, sharing of data maps describing information on datasets held by public and private sectors is on the rise, but there is a problem in that accurate interpretation and utilization of data in different formats are not smooth.

In the past, data sets have been managed and shared in relation to work, but the need for mutual data sharing is further increasing with the recent spread of data openness and expectations for new industry development. Accordingly, there is a need for a structure and generation method of a data map capable of responding to changes in the data industry environment.

The technical problem of the present disclosure is to provide a method and apparatus for generating a data map that describes various metadata information about a dataset in order to support mutual sharing and utilization of various datasets generated in various areas such as the public and private sectors. there is

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

According to one aspect of the present disclosure, a method for generating a datamap is provided. The data map creation method generates a relational data map in which asset-related metadata of a catalog classified according to a classification system and category is described in a preset standard by a data map creation module for a relational DB, and uses a standard vocabulary. converting the relational data map into an RDF triple structure by using; and a data map generation module for a graph-type DB, which transmits change data related to a change event in the relational data map to a message queue, and uses a standard vocabulary for the change data input to the message queue to obtain nodes and nodes. and generating a graph-type data map that maps attributes and relationships between nodes.

According to another embodiment of the present disclosure, the graph-type data map allocates the asset to the node, and according to the range of terms of the asset-related metadata, the asset-related metadata is assigned to the node attribute and the relationship between the nodes. It can be created to assign to any one of them.

In addition to the above embodiment, the node attribute is generated when the asset-related metadata falls within the scope of a term in the standard vocabulary, and the inter-node relationship is such that the asset-related metadata falls within the scope of a term in the standard vocabulary. It can be created if it does not belong.

In addition to the above embodiment, the graph-type data map may be created so as to assign attribute values related to the relationship between the nodes to new nodes.

According to another embodiment of the present disclosure, the asset-related metadata may include at least one of a usage method for each asset, a standard vocabulary, and a quality index, together with at least one of a dataset, a data service, and an analysis model. .

According to another aspect of the present disclosure, an apparatus for generating a data map is provided. The data map generation device generates a relational data map in which asset-related metadata of a catalog classified according to a classification system and category is described in a preset standard, and uses a standard vocabulary to convert the relational data map into an RDF triple structure. Data map creation module for relational DB that converts to; and graph-type data that transfers change data related to a change event in the relational data map to a message queue, and maps nodes, node attributes, and relationships between nodes using a standard vocabulary for the change data input to the message queue. Includes a data map creation module for graph-type DB that creates maps.

The features briefly summarized above with respect to the disclosure are merely exemplary aspects of the detailed description of the disclosure that follows, and do not limit the scope of the disclosure.

According to the present disclosure, it is possible to provide a method and apparatus for generating a data map describing various metadata information on a dataset in order to support mutual sharing and utilization of various datasets generated in various areas, such as the public and private sectors.

According to the present disclosure, metadata information, which is various information about data, such as whereabouts, formats, and contents of data, is distributed in a standard format, so that mutual data search and sharing can be smoothly performed.

According to the present disclosure, data scattered in various fields can be efficiently shared and search accuracy can be improved.

Effects obtainable in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below. will be.

1 is a schematic block diagram of an apparatus for generating a data map according to an embodiment of the present disclosure.
2 is a diagram showing the structure of a data map.
3 is a diagram illustrating the management of catalogs in which assets are classified according to classification systems and categories.
4 is a diagram illustrating an RDF triple structure for describing metadata of an asset.
5 is a block diagram of a data map generation module for relational DB.
6 is a block diagram of a data map generation module for a graph type DB.
7 is a diagram illustrating the addition of nodes in a graph-like database.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

In describing the embodiments of the present disclosure, if it is determined that a detailed description of a known configuration or function may obscure the gist of the present disclosure, a detailed description thereof will be omitted. And, in the drawings, parts irrelevant to the description of the present disclosure are omitted, and similar reference numerals are attached to similar parts.

In the present disclosure, when a component is said to be "connected", "coupled" or "connected" to another component, this is not only a direct connection relationship, but also an indirect connection relationship where another component exists in the middle. may also be included. In addition, when a component "includes" or "has" another component, this means that it may further include another component without excluding other components unless otherwise stated. .

In the present disclosure, terms such as first and second are used only for the purpose of distinguishing one element from another, and do not limit the order or importance of elements unless otherwise specified. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment. can also be called

In the present disclosure, components that are distinguished from each other are to clearly explain each characteristic, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to form a single hardware or software unit, or a single component may be distributed to form a plurality of hardware or software units. Accordingly, even such integrated or distributed embodiments are included in the scope of the present disclosure, even if not mentioned separately.

In the present disclosure, components described in various embodiments do not necessarily mean essential components, and some may be optional components. Therefore, an embodiment composed of a subset of components described in one embodiment is also included in the scope of the present disclosure. In addition, embodiments including other components in addition to the components described in various embodiments are also included in the scope of the present disclosure.

Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings of this specification.

1 is a schematic block diagram of an apparatus for generating a data map according to an embodiment of the present disclosure.

The data map generating system includes a data map generating device 100 , a data governance system 200 and a data portal system 300 .

The data map generating device 100 may include a data map generating module 110 for relational DB and a data map generating module 120 for graph type DB.

The data map generation module 110 for a relational DB creates a relational data map in which metadata related to assets of catalogs classified according to classification schemes and categories is described in a preset standard, and uses a standard vocabulary to create a relational data map. can be converted to RDF (Resource Description Framework) triple structure. An RDF triple structure can be created by integrating search results into an RDF file.

The data map generation module 120 for graph-type DB transfers change data related to change events in the relational data map to the message queue, and uses a standard vocabulary for the change data input to the message queue to identify nodes, node attributes, and nodes. You can create a graph-type data map that maps the relationship between

The apparatus 100 for generating a data map may use a relational database and a graph database in parallel, and convert a relational data map stored in the relational database into a graph data map suitable for the graph database.

By describing the metadata, including the data properties of assets and the relationship information between assets, in a data map based on the preset specifications (i.e., standards), semantic search is possible based on various metadata of data for user queries. do. In addition, by converting a relational database to a graph database, it is possible to support more improved search performance in a search process.

The data governance system 200 can manage the classification system and categories of the data map, as well as standard vocabulary and quality indicators for each category.

The data portal system 300 communicates with the data map generator 100, collects various data sets, transfers them to the data map generator 100, and provides the RDF triple structure generated by the data map generator 100. Based on the RDF file, search results according to user search requests can be provided.

2 is a diagram showing the structure of a data map. 2 relates to a relational data map generated by the data map generation module 110 for relational DB.

A data map may be a data structure expressed in various metadata information for convergence, utilization, search, etc. between datasets. When data maps described as standard metadata are exchanged between heterogeneous platforms operated by each institution, operability between platforms is increased and necessary data can be accurately retrieved.

The data map may be configured so that a catalog having at least one asset is assigned to a classification system classified for each field, and each asset constituting the catalog may be configured to belong to a category, which is a lower level of the classification system. Datamaps can describe metadata information related to catalogs and assets. In each category, how to use the asset, vocabulary, quality index, etc. can be added.

A catalog is a data catalog and can be configured to catalog assets. Assets can collectively refer to all data existing on the web. As illustrated in FIG. 2 , the asset may be at least one of a data set, which is a set of data, a data service, and an analysis model, which is a set of services or endpoints that can access data. An analytic model may consist of, for example, a set of algorithms and solutions that analyze data. It is defined as a data map that lists asset-related metadata, including information such as metadata about assets, relationships between assets, categories to which assets belong, and managed classification systems, while listing datasets, data services, and analysis models. It can be. Datamap is a kind of data catalog defined by W3C.

The data map information can be stored in a relational database or graph database, and then created and distributed in a standard data structure such as RDF or JONS LD.

3 is a diagram illustrating the management of catalogs in which assets are classified according to classification systems and categories.

For example, if a healthcare catalog (e.g., catalog 001) includes a disease dataset, a prescription dataset (e.g., dataset 001, dataset 002), and an API service accessible to the dataset, the data map is a data map for the healthcare catalog. Metadata can be described. Metadata of the healthcare catalog may include, for example, a creation date of an asset, a creator of the asset, assets included in the catalog, and a classification system used in asset management. In addition, the data map may include, for example, information such as metadata for each dataset and metadata for API (Application Program Interface), which is a data service. Accordingly, when a user searches for data based on the data map, a more accurate search according to the user's intention is possible due to the correlation between various metadata and assets included in the data map.

As illustrated in FIG. 3 , the classification system may have a hierarchical structure of categories. A catalog can select a classification system and manage the assets included in the catalog according to the categories of the classification system.

4 is a diagram illustrating an RDF triple structure for describing metadata of an asset.

First of all, RDF may be a standard specification for expressing information of resources on the web. In order to express the metadata of resources more clearly, it is a type of fixed format, and by describing resources and their metadata in a common format called RDF, it is easy to search and manage resources (or data), and heterogeneous platforms or applications Data exchange between programs can be implemented smoothly. According to the RDF format, a resource can be described in a triples structure. The triple structure can be expressed with three elements that describe resources: <subject>, <predicate>, and <object>. <predicate> can be a structure that describes the relationship between <subject> and <object>. <subject> corresponds to a resource (data) to be explained, and <object> may be a resource related to <subject>. <predicate> can express the relationship and characteristics between <subject> and <object>. <predicate> may have the same meaning as 'property property' used in DCAT of W3C and data map according to the present disclosure. In the data map according to the present disclosure, an RDF triple structure is used to express metadata of assets to be described, assets are described as subjects, and metadata of assets, that is, properties and relationships, etc. are expressed and described as objects and properties. can

4 shows an example of an RDF triple structure, and when the title of dataset 001 is 'blood pressure collection data', the dataset can be described as the RDF illustrated in FIG. 4 . Here, one of the heterogeneous platforms describes the metadata as 'title' and the other platform describes the metadata as 'name', so that the metadata can be expressed differently. In this case, the title of the dataset is indicated. Due to different metadata, difficulties may occur in data sharing, distribution, or search. To solve this problem, metadata can be described using standard vocabulary. The vocabulary used in Figure 4 is the Dublin Core vocabulary, and ' To indicate that the term title' is a term defined in the Dublin Core vocabulary, it can be described as 'dcterms:title', with the prefix 'dcterms' added. As a set of vocabularies, it can be distinguished by specifying and indicating the vocabulary to which the term belongs In addition, various vocabularies such as SKOS (Simple Knowledge Organization System) vocabulary, FOAF (Friend Of A Friend) vocabulary, and DCAT (Data CATalog vocabulary) vocabulary can be distinguished. In this disclosure, when expressing a data map, metadata can be described using these standard vocabularies.

In expressing metadata, validation related to whether standard vocabulary is correctly used, if constraints are defined for each metadata, whether the corresponding metadata and triple structure are created in accordance with the constraints, etc. may be accompanied. . In this case, the quality determination index or criterion of the metadata applied to the validity check may be a quality index. A data governance system (SODAS+ Data Governance System; see 200 in FIG. 5) that manages classification schemes and categories of data maps can manage quality indicators along with standard vocabularies.

5 is a block diagram of a data map generation module for relational DB.

The data map generation module 110 for relational DB may include a relational data map generation unit 111 and a distribution unit 112 that generate a relational data map stored in a relational DB.

The relational data map generating unit 111 may generate a relational data map related to metadata information about an asset, and the relational data map may be stored in a relational database of the relational data map generating unit 111 . The distribution unit 112 may apply a standard vocabulary, convert the relational data map into an RDF triple structure, and generate an RDF file in which search results are integrated.

As described in FIG. 2 , the relational data map may be configured such that a catalog having at least one asset is assigned to a classification system classified for each field. Each asset that makes up the relational catalog can be organized to belong to a category, which is a lower level of the taxonomy. Datamaps can describe metadata information related to catalogs and assets. In each category, how to use the asset, vocabulary, quality index, etc. can be added. A detailed description related to the foregoing point is described through FIG. 2 and will be omitted.

6 is a block diagram of a data map generation module for a graph type DB.

The data map generation module 120 for graph type DB may include a relational DB 121, a graph DB processing unit 123, a graph type data map generation unit 124, and a distribution unit 125.

The relational DB 121 associated with the data map creation module 110 for relational DB can recognize change events such as storage, modification, and deletion of data in the relational database. The relational DB 121 may identify and track changed data in the relational database (or the relational data map generator 111) by a CDC (Change Data Capture) 122 and input the changed data to the message queue. The message queue can synchronize both database systems so that there is no difference in content and structure between the data map created in the relational database and the data map created in the graph database. The message queue is an element included in the CDC 122 technology, and the CDC 122 records insert, update, and delete operations occurring or applied to tables in the source database, and processing according to change events so that they can be applied to the target database. it could be technology. A typical source system using CDC technology would be Debezium, for example. Debezium records the changes that occur in the source database to the message queue stream so that the corresponding change events can be sequentially applied in the target database. In this way, synchronization between the two databases is possible by applying change events that occur in the relational database to the graph database through the message queue.

The graph DB processor 123 maps the data waiting in the message queue to elements constituting a graph-type database (or graph-type data map), that is, nodes, node attributes, and relationships between nodes using a standard vocabulary. elements can be created. Each data asset is created as a single node. Attributes of each node may be mapped with asset metadata. In this case, the range of each metadata in the standard vocabulary may be mapped to an attribute of a graph node or may be mapped to a relationship between nodes by creating a new node.

The graph data map generation unit 124 may store assets matched with each element of the graph database and metadata of the assets as a graph database having a graph form according to the structure of the data map.

Like the data map generation module 110 for relational DB, the distribution unit 125 may generate a data map constructed in a graph form as an RDF file. RDF files may be configured in the form of, for example, Turtle, N-Triples, JSON-LD (my: JavaScript object notation for connected data), TriG, and RDF/XML.

7 is a diagram illustrating the addition of nodes in a graph-like database.

As shown in FIG. 7, the graph database may be composed of nodes, node properties, and relationship lines corresponding to relationships between nodes. Each node has a node property, and a relationship can be formed between nodes with a relationship line.

One node can be created for one asset (eg, dataset 001). Depending on the type of metadata, some of the asset's metadata may be generated as graph database node properties (e.g., descriptions within the nodes of dataset 001), and other portions may be assigned as relationship lines. The criterion determined by the node properties and relationship lines may follow terms of a standard vocabulary describing metadata. For terms, a range to which the term can be applied is set.

Referring to FIG. 7 as an example, the range of the 'dcterms:title' property is 'rdfs:Literal'. Literal values such as text strings or numbers belong to 'rdfs:Literal'. Therefore, when describing metadata called the title of 'dataset 001', if the term 'dcterms:title' is used, only literal values such as text strings or numbers can be written as the value. As another example, 'dcat:mediaType' is a term belonging to the DCAT vocabulary, and is a metadata property indicating the media type to which the asset of the corresponding distribution file belongs. The range of 'dcat:mediaType' is 'dcterms:MediaType', and only the media types defined by IANA can be the value of 'dcat:mediaType'.

According to the example of FIG. 7, an asset is created as one node, and the range of terms representing metadata (attributes) of the asset is 'rdfs:Literal', 'xsd:duration', 'xsd:decimal' In this case, the property of the asset may be set as the node property of the corresponding node.

Attributes with a range of terms other than three can be created as relation lines, as in 'dcterms:license'. In addition, 'https://creativecommons.org/licenses/by/4.0', the value of the corresponding attribute, may be assigned to a new node and configured in a form having a relationship with the corresponding node.

According to the present disclosure, relational data maps describe associations between datasets, and storing information including associations in a graph database can improve the creation of triple-structured files such as RDF and the performance of association searches. .

Specifically, when a relational database managing catalogs and asset-related metadata is converted to a graph database, the graph database is superior to the relational database in terms of search performance. Accordingly, data search and search accuracy can be improved by creating and managing a data map based on a graph database.

Exemplary methods of this disclosure are presented as a series of operations for clarity of explanation, but this is not intended to limit the order in which steps are performed, and each step may be performed concurrently or in a different order, if desired. In order to implement the method according to the present disclosure, other steps may be included in addition to the exemplified steps, other steps may be included except for some steps, or additional other steps may be included except for some steps.

Various embodiments of the present disclosure are intended to explain representative aspects of the present disclosure, rather than listing all possible combinations, and matters described in various embodiments may be applied independently or in combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), It may be implemented by a processor (general processor), controller, microcontroller, microprocessor, or the like.

The scope of the present disclosure is software or machine-executable instructions (eg, operating systems, applications, firmware, programs, etc.) that cause operations according to methods of various embodiments to be executed on a device or computer, and such software or It includes a non-transitory computer-readable medium in which instructions and the like are stored and executable on a device or computer.

Claims (1)

The relational data map creation module for relational DB creates a relational data map that describes asset-related metadata in a catalog classified according to classification system and category in a preset standard, and uses a standard vocabulary to generate the relational data map. Converting to RDF triple structure; and
The data map creation module for graph-type DB transfers change data related to the change event in the relational data map to the message queue, and uses standard vocabulary for the change data input to the message queue to determine node and node properties. and generating a graph-type data map mapping relationships between nodes.

Images