CN115391333A - Data acquisition method and device - Google Patents

Abstract

The invention discloses a data acquisition method and a data acquisition device. The method comprises the following steps: configuring an acquisition task, and issuing the configured acquisition task; based on the issued acquisition task, acquiring data from various heterogeneous data sources in an increment manner by utilizing corresponding data acquisition tools; and storing the acquired data to a service database.

Description

A data acquisition method and device

technical field

The present invention relates to the technical field of data processing, and more specifically, to a data collection method and device.

Background technique

Data gateway refers to a type of equipment that realizes the interconnection of multiple business data systems and realizes the integration, sharing and management of data between various systems. Intelligent Clinical&Health DataGateWay (hereinafter referred to as IDCHGW) refers to a device that provides data governance, reproduction and data services for medical data, provides visual, intelligent, interactive and programmable operation methods, and provides security data exchange capabilities.

Data collection is an indispensable link in the construction process of the entire intelligent medical and health data gateway. The medical and health data gateway can be connected to multiple medical information system data sources. For the hospital's multi-source heterogeneous data, how to achieve full collection and aggregation has become an urgent problem in the existing technology.

Contents of the invention

Aiming at the deficiencies of the prior art, the present invention provides a data collection method and device.

According to one aspect of the present invention, a data collection method is provided, comprising:

Configure collection tasks and publish the configured collection tasks;

Based on the released collection tasks, use the corresponding data collection tools to incrementally collect data from various heterogeneous data sources;

Store the collected data in the business database.

Optionally, using corresponding data collection tools to incrementally collect data from various heterogeneous data sources, including: using DataX middleware to incrementally collect data from various heterogeneous data sources.

Optionally, use corresponding data collection tools to incrementally collect data from various heterogeneous data sources, including:

Use DataX middleware to collect incremental data from various heterogeneous data sources to temporary tables;

Create upate table, new table and delete table;

By comparing the main key of the temporary table with the md5 string, insert the filtered data in the temporary table into the upate table, new table and delete table respectively;

When no data is inserted or the insertion ends, the upate table, new table, and delete table are deleted.

Optionally, use DataX middleware to collect incremental data from various heterogeneous data sources to temporary tables, including:

Based on the OGG (Oracle GoldenGate) acquisition technology, the incremental data of various heterogeneous data sources is identified through logs, and the identified incremental data is collected to a temporary table;

Based on the CDC (Change Data Capture) acquisition technology, the incremental data of various heterogeneous data sources is identified through logs, and the identified incremental data is collected to a temporary table;

Based on the ETL (Extraction-Transformation-Loading) acquisition technology, the incremental data of various heterogeneous data sources is identified by means of time stamps and primary keys, and the identified incremental data is collected to a temporary table.

According to another aspect of the present invention, a data collection device is provided, comprising:

The configuration and release module is used to configure the collection tasks and publish the configured collection tasks;

The data collection module is used to collect data incrementally from various heterogeneous data sources by using corresponding data collection tools based on published collection tasks;

The data storage module is used to store the collected data in the business database.

Optionally, the data collection module is specifically configured to: use DataX middleware to incrementally collect data from various heterogeneous data sources.

Optionally, the data acquisition module is specifically used for:

Use DataX middleware to collect incremental data from various heterogeneous data sources to temporary tables;

Create upate table, new table and delete table;

By comparing the main key of the temporary table with the md5 string, insert the filtered data in the temporary table into the upate table, new table and delete table respectively;

When no data is inserted or the insertion ends, the upate table, new table, and delete table are deleted.

Optionally, the data acquisition module is also specifically used for:

Based on the Ogg (Oracle GoldenGate) acquisition technology, the incremental data of various heterogeneous data sources is identified through logs, and the identified incremental data is collected to a temporary table;

Based on the Cdc (Change Data Capture) collection technology, the incremental data of various heterogeneous data sources is identified through logs, and the identified incremental data is collected to a temporary table;

Based on the ETL (Extraction-Transformation-Loading) acquisition technology, the incremental data of various heterogeneous data sources is identified by means of time stamps and primary keys, and the identified incremental data is collected to a temporary table.

The present invention firstly configures the collection tasks, and releases the configured collection tasks. Then, based on the published collection tasks, use the corresponding data collection tools to incrementally collect data from various heterogeneous data sources. Finally, the collected data is stored in the business database. The present invention adopts database synchronization technology and corresponding data collection tools to extract, synchronize and collect data from various heterogeneous data sources, thereby realizing the collection and collection of multi-source heterogeneous data for hospitals.

Description of drawings

A more complete understanding of the exemplary embodiments of the present invention can be had by referring to the following drawings:

Fig. 1 is a schematic flow chart of a data acquisition method provided by an exemplary embodiment of the present invention;

Fig. 2 is a frame diagram of a data collection service module applied with a data collection method provided by an exemplary embodiment of the present invention;

Fig. 3 is a schematic structural diagram of a data collection device provided by an exemplary embodiment of the present invention.

Detailed ways

Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Apparently, the described embodiments are only some embodiments of the present invention, rather than all embodiments of the present invention, and it should be understood that the present invention is not limited by the exemplary embodiments described here.

It should be noted that the relative arrangements of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

Those skilled in the art can understand that terms such as "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc. necessary logical sequence.

It should also be understood that in the embodiments of the present invention, "plurality" may refer to two or more than two, and "at least one" may refer to one, two or more than two.

It should also be understood that for any component, data or structure mentioned in the embodiments of the present invention, it can generally be understood as one or more unless there is a clear limitation or a contrary suggestion is given in the context.

In addition, the term "and/or" in the present invention is only an association relationship describing associated objects, indicating that there may be three relationships, for example, A and/or B may indicate: A exists alone, and A and B exist at the same time , there are three cases of B alone. In addition, the character "/" in the present invention generally indicates that the contextual objects are an "or" relationship.

It should also be understood that the description of the various embodiments of the present invention emphasizes the differences between the various embodiments, and the same or similar points can be referred to each other, and for the sake of brevity, details are not repeated one by one.

At the same time, it should be understood that, for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

The following description of at least one exemplary embodiment is merely illustrative in nature and in no way taken as limiting the invention, its application or uses.

Techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, techniques, methods and devices should be considered part of the description.

It should be noted that like numerals and letters denote like items in the following figures, therefore, once an item is defined in one figure, it does not require further discussion in subsequent figures.

Embodiments of the present invention can be applied to electronic equipment such as terminal equipment, computer systems, servers, etc., which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well known terminal devices, computing systems, environments and/or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick client Computers, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the foregoing, etc.

Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system-executable instructions, such as program modules, being executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types. The computer system/server can be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computing system storage media including storage devices.

exemplary method

Fig. 1 is a schematic flowchart of a data collection method provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic equipment, for example but not limited to a data collection service system.

As shown in Figure 1, the data collection method 100 includes the following steps:

Step 101, configure collection tasks, and publish the configured collection tasks.

In the embodiment of the present invention, the data to be collected is, for example but not limited to, hospital data. The hospital data can be obtained from multiple business data systems, and each business data system corresponds to a data source. Therefore, as shown in Figure 2, before collecting data, business units can configure data sources according to actual needs through the data source configuration management module, that is, determine which data sources to connect to. After the configuration is complete, according to the configuration information, collect data from the corresponding data source for subsequent synchronization and aggregation.

Further, as shown in Figure 2, the business unit can also configure corresponding collection tasks according to actual needs through the collection task configuration management module, and then publish the configured collection tasks through the collection task release module.

Step 102, based on the released collection tasks, use corresponding data collection tools to incrementally collect data from various heterogeneous data sources.

Optionally, using corresponding data collection tools to incrementally collect data from various heterogeneous data sources, including: using DataX middleware to incrementally collect data from various heterogeneous data sources.

In the embodiment of the present invention, the data acquisition tool adopts DataX, which is an off-line synchronization tool for heterogeneous data sources, and is dedicated to realizing various databases including relational databases (MySQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, FTP, etc. Stable and efficient data synchronization function between heterogeneous data sources.

In the embodiment of the present invention, the CDH (Cloudera's Distribution Including Apache Hadoop) big data processing scheme is used for data storage and calculation, providing an expandable, flexible, and integrated enterprise-level big data management platform, which can be used to conveniently manage the rapid growth of hospitals. A variety of data, while providing security protection and integration with hardware and software solutions. Support structured data and unstructured data storage in the medical industry.

In the embodiment of the present invention, the data access method adopts the table or view method, directly reads the existing table or view in the original business system, and uses SQL to filter and import data. This method is also the most convenient and mature method at present. The system uses DataX middleware as the data import component, which can support relational databases such as mysql, oracle, pg, and sqlserver, as well as non-relational data such as hive, HBase, etc. There are dozens of databases interconnected with each other.

Optionally, use corresponding data collection tools to incrementally collect data from various heterogeneous data sources, including: using DataX middleware to collect incremental data from various heterogeneous data sources to temporary tables; create upate tables, new table and delete table; by comparing the main key of the temporary table with the md5 string, the data filtered out of the temporary table are inserted into the upate table, new table and delete table respectively; when no data is inserted or the insertion is completed, the upate Table, new table and delete table are deleted.

Optionally, use DataX middleware to collect incremental data from various heterogeneous data sources to temporary tables, including: based on OGG (Oracle GoldenGate) collection technology, identify incremental data from various heterogeneous data sources through logs data, and collect the identified incremental data to a temporary table; based on CDC (Change Data Capture) collection technology, identify the incremental data of various heterogeneous data sources through logs, and collect the identified incremental data To a temporary table; based on ETL (Extraction-Transformation-Loading) collection technology, identify incremental data from various heterogeneous data sources by means of timestamps and primary keys, and collect the identified incremental data to a temporary table.

In the embodiment of the present invention, incremental data collection based on three methods of OGG, CDC and ETL is constructed, which can not only realize the separation of reading and writing of the business system, but also realize real-time data backup. The incremental data model is built based on the full data model of the hospital, that is, the incremental data model is created to be consistent with the full data model, so that new data can be quickly collected into the original data platform. The main steps of incremental data collection are as follows:

1) Create a temporary table;

2) Collect the incremental data to the temporary table through DataX (data collection tool). OGG, CDC, and ETL identify incremental data based on different approaches. CDC and OGG through logs; ETL through timestamps and primary keys.

3) Create upate, new, delete tables respectively.

4) By comparing the primary key with the md5 string, insert the filtered data into the update, new and delete tables respectively.

5) Insert the data of the three tables; delete the table if there is no data or the insertion is complete.

Step 103, storing the collected data in the business database.

In the embodiment of the present invention, it supports one-stop configuration of full and incremental (time stamp, self-incrementing primary key, expression, etc.) tasks, which greatly improves configuration efficiency and reduces configuration error rate. Use adaptation and extraction tools to import various data tables related to clinical research in the hospital HIS, LIS, RIS/PACS, surgical anesthesia, pathology, ECG, intensive care (ICU/CCU) and other systems to the original master of the medical data platform in the database. Supports flexible configuration of multiple data sources, supports database connectivity testing, and supports flexible addition of timestamp fields to the collection target model, forming a complete closed loop with incremental tasks. It supports the release of collection tasks by different versions, which can appropriately improve the reusability when creating new tasks in the future, and supports direct configuration of scheduled tasks to implement multi-task scenarios such as manual execution and scheduled tasks.

In the embodiment of the present invention, database synchronization technology, ETL and other technologies are used to extract, synchronize, and collect data to realize the collection and aggregation of multi-source heterogeneous data in hospitals, and support cooperation with more than 150 mainstream medical information system manufacturers. Seamless integration, supporting intelligent mapping of more than 300 medical information system data sources. Integration and access to hospital production systems: full collection of historical data from clinical systems such as HIS system, EMR system, PACS, RIS system, inspection information system, ultrasound information management system, and pathology system, and new data sources can be integrated in real time.

Therefore, the present invention firstly configures the collection task, and releases the configured collection task. Then, based on the published collection tasks, use the corresponding data collection tools to incrementally collect data from various heterogeneous data sources. Finally, the collected data is stored in the business database. The present invention adopts database synchronization technology and corresponding data collection tools to extract, synchronize and collect data from various heterogeneous data sources, thereby realizing the collection and collection of multi-source heterogeneous data for hospitals.

Exemplary device

Fig. 3 is a schematic structural diagram of a data collection device provided by an exemplary embodiment of the present invention. As shown in Figure 3, the data acquisition device 300 includes:

The configuration and release module 310 is configured to configure the collection tasks and release the configured collection tasks;

The data collection module 320 is used to incrementally collect data from various heterogeneous data sources using corresponding data collection tools based on published collection tasks;

The data storage module 330 is configured to store the collected data in a business database.

Optionally, the data collection module 320 is specifically configured to: use DataX middleware to incrementally collect data from various heterogeneous data sources.

Optionally, the data acquisition module 320 is specifically used for:

Use DataX middleware to collect incremental data from various heterogeneous data sources to temporary tables;

Create upate table, new table and delete table;

By comparing the main key of the temporary table with the md5 string, insert the filtered data in the temporary table into the upate table, new table and delete table respectively;

When no data is inserted or the insertion ends, the upate table, new table, and delete table are deleted.

Optionally, the data collection module 320 is also specifically used for:

Based on the OGG (Oracle GoldenGate) acquisition technology, the incremental data of various heterogeneous data sources is identified through logs, and the identified incremental data is collected to a temporary table;

Based on the CDC (Change Data Capture) acquisition technology, the incremental data of various heterogeneous data sources is identified through logs, and the identified incremental data is collected to a temporary table;

Based on the ETL (Extraction-Transformation-Loading) acquisition technology, the incremental data of various heterogeneous data sources is identified by means of time stamps and primary keys, and the identified incremental data is collected to a temporary table.

The data collection device 300 in the embodiment of the present invention corresponds to the data collection method 100 in another embodiment of the present invention, and will not be repeated here.

Claims (8)

1. A method of data acquisition, comprising:

configuring the acquisition task and issuing the configured acquisition task;

based on the issued acquisition task, acquiring data from various heterogeneous data sources in an incremental manner by using corresponding data acquisition tools;

and storing the collected data in a service database.

2. The method of claim 1, wherein incrementally collecting data from a variety of disparate data sources using a corresponding data collection tool comprises: data is incrementally acquired from various heterogeneous data sources using DataX middleware.

3. The method of claim 1, wherein incrementally collecting data from a variety of disparate data sources using corresponding data collection tools comprises:

acquiring incremental data of various heterogeneous data sources to a temporary table by using DataX middleware;

creating an update table, a new table and a delete table;

respectively inserting the screened data in the temporary table into an update table, a new table and a delete table by comparing the main key of the temporary table with the md5 character string;

when there is no data insertion or the insertion is finished, the update table, the new table, and the delete table are deleted.

4. The method of claim 3, wherein collecting incremental data for various heterogeneous data sources into a temporary table using DataX middleware comprises:

identifying incremental data of various heterogeneous data sources in a log mode based on an OGG (Oracle golden Gate) acquisition technology, and acquiring the identified incremental data to a temporary table;

based on a CDC (Change Data Capture) acquisition technology, identifying incremental Data of various heterogeneous Data sources in a log mode, and acquiring the identified incremental Data to a temporary table; or

Based on an ETL (Extraction-Transformation-Loading) acquisition technology, incremental data of various heterogeneous data sources are identified through a time stamp and a main key mode, and the identified incremental data are acquired to a temporary table.

5. A data acquisition device, comprising:

the configuration and release module is used for configuring the acquisition tasks and releasing the configured acquisition tasks;

the data acquisition module is used for incrementally acquiring data from various heterogeneous data sources by using corresponding data acquisition tools based on the issued acquisition tasks;

and the data storage module is used for storing the acquired data to a service database.

6. The device of claim 5, wherein the data acquisition module is specifically configured to: data is incrementally acquired from various heterogeneous data sources using DataX middleware.

7. The device of claim 5, wherein the data acquisition module is specifically configured to:

acquiring incremental data of various heterogeneous data sources to a temporary table by using DataX middleware;

creating an update table, a new table and a delete table;

respectively inserting the screened data in the temporary table into an update table, a new table and a delete table by comparing the main key of the temporary table with the md5 character string;

when there is no data inserted or the insertion is finished, the update table, the new table, and the delete table are deleted.

8. The apparatus of claim 7, wherein the data acquisition module is further specifically configured to:

identifying incremental data of various heterogeneous data sources in a log mode based on an OGG (Oracle golden Gate) acquisition technology, and acquiring the identified incremental data to a temporary table;

based on a CDC (Change Data Capture) acquisition technology, identifying incremental Data of various heterogeneous Data sources in a log mode, and acquiring the identified incremental Data to a temporary table;

based on an ETL (Extraction-Transformation-Loading) acquisition technology, incremental data of various heterogeneous data sources are identified through a time stamp and a primary key mode, and the identified incremental data are acquired to a temporary table.

Images