CN111400061A - A data processing method and system - Google Patents

Abstract

The invention discloses a data processing method and system, and relates to the technical field of computers. A specific implementation of the method includes: the configuration management center obtains the configuration information generated for different types of data sources, and pushes the configuration information to the front-end computer; after the front-end computer encapsulates the configuration information into the configuration information entity class, data calibration is performed. After verification, the configuration information in the configuration information entity class that has passed the verification is written into the local configuration file; after the current host computer receives the data extraction request, it is obtained from the configuration information of the local configuration file according to the target data source identifier of the data extraction request. Corresponding data source type and attribute information; the front-end loads the corresponding data engine according to the data source type to establish a connection with the target data source based on the data engine and attribute information, and then obtains business data from the target data source and returns the business data. The method realizes hot deployment of configuration information, and can acquire business data of target data sources flexibly and quickly.

Description

A data processing method and system

technical field

The present invention relates to the field of computers, in particular to a data processing method and system.

Background technique

Insurance companies need to obtain a large amount of basic data from the hospital system and third-party institutions in the process of underwriting and underwriting for customers. In the prior art, insurance companies generally take message middleware as the core of the architecture, and use message queues to process data extraction requests to achieve connection with hospital systems and third-party institutions.

In the process of realizing the present invention, the inventor found that there are at least the following problems in the prior art:

Using the message queue method, the back-row requests will be processed only after the front-row messages are processed, and the processing speed is slow; the dependence on middleware is strong, and the flexibility is poor; the technical level of various hospital systems and third-party institutions Adjusting the docking method according to the specific conditions of each hospital system and third-party institutions, the development workload is large, and the labor and time costs are high.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present invention provide a data processing method and system, after receiving a data extraction request, obtain the data source type and attribute information corresponding to the target data source from the configuration information, so as to load the corresponding data engine, and A connection is established with the target data source, so that the business data of the target data source can be obtained flexibly and quickly.

To achieve the above object, according to an aspect of the embodiments of the present invention, a data processing method is provided.

A data processing method according to an embodiment of the present invention includes: a configuration management center obtains configuration information generated for different types of data sources, and pushes the configuration information to a front-end computer; wherein the configuration information includes: a data source identifier , data source type and attribute information; the front-end machine encapsulates the configuration information into a configuration information entity class, performs data verification on the configuration information entity class, and checks the configuration information in the configuration information entity class that has passed the verification. Write the local configuration file; when the front-end computer receives the data extraction request, according to the target data source identifier of the data extraction request, obtain the corresponding data source type and attribute information from the configuration information of the local configuration file ; The front-end machine loads the corresponding data engine according to the data source type, to establish a connection with the target data source based on the data engine and the attribute information, then obtain business data from the target data source, return the business data.

Optionally, acquiring business data from the target data source includes: acquiring original data from the target data source, and preprocessing the original data to obtain business data; wherein the preprocessing includes any one of the following Or more: data validation, data cleaning, data conversion, adding default values.

Optionally, performing data verification on the original data includes: according to a set value rule, filtering out field information that satisfies the value rule from the field information of the original data, and will satisfy the value rule. The field information of the value rule is summarized.

Optionally, performing data conversion on the raw data includes: parsing the raw data to obtain formatting information, and encapsulating the formatting information into an entity class.

Optionally, after the step of parsing the original data to obtain the formatted information, it further includes: according to pre-established dictionary information, performing data assignment on the set field information in the formatted information; encapsulating the formatted information To the entity class, including: encapsulating the formatted information after the assignment into the entity class.

Optionally, the data source type includes any one or more of the following: database, webservice interface and server file.

When the data source type is a database, the attribute information includes the connection address of the data source, the connection login user, the connection login password, and the execution SQL; when the data source type is a webservice interface, the attribute information includes the IP address and port|URL of the data source; when the data source type is a server file, the attribute information includes the FTP protocol address, account number and password.

To achieve the above object, according to another aspect of the embodiments of the present invention, a data processing system is provided.

A data processing system according to an embodiment of the present invention includes: a configuration management center and a front-end computer, the front-end computer includes a package writing module, a data acquisition module and a data return module, wherein the configuration management center is used for Acquire configuration information generated for different types of data sources, and push the configuration information to the front-end computer; wherein, the configuration information includes: data source identification, data source type and attribute information; the package writing module is used to encapsulate the configuration information into a configuration information entity class, perform data verification on the configuration information entity class, and write the configuration information in the configuration information entity class that has passed the verification into a local configuration file; the data acquisition a module, configured to obtain the corresponding data source type and attribute information from the configuration information of the local configuration file according to the target data source identifier of the data extraction request after receiving the data extraction request; the data return module, Used to load the corresponding data engine according to the data source type, to establish a connection with the target data source based on the data engine and the attribute information, then obtain business data from the target data source, and return the business data .

Optionally, the data return module is further configured to: obtain raw data from the target data source, and preprocess the raw data to obtain business data; wherein, the preprocessing includes any one or more of the following Items: data validation, data cleaning, data conversion, adding default values.

Optionally, the data return module is further configured to: according to the set value rule, filter out the field information that satisfies the value rule from the field information of the original data, and will satisfy the value rule The field information is summarized.

Optionally, the data return module is further configured to: parse the raw data to obtain formatted information, and encapsulate the formatted information into an entity class.

Optionally, the system further includes an assignment module for performing data assignment on the set field information in the formatted information according to the pre-established dictionary information; the data return module is also used for: assigning the value after the assignment. The formatting information is encapsulated into the entity class.

To achieve the above object, according to yet another aspect of the embodiments of the present invention, an electronic device is provided.

An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, The one or more processors are caused to implement a data processing method according to an embodiment of the present invention.

To achieve the above object, according to yet another aspect of the embodiments of the present invention, a computer-readable medium is provided.

A computer-readable medium of an embodiment of the present invention stores a computer program thereon, and when the program is executed by a processor, a data processing method of the embodiment of the present invention is implemented.

An embodiment of the above invention has the following advantages or beneficial effects: after receiving the data extraction request, obtain the data source type and attribute information corresponding to the target data source from the configuration information, so as to load the corresponding data engine, and match the target data source. The source establishes a connection, so that the business data of the target data source can be obtained flexibly and quickly; by obtaining the configuration information and writing the configuration information to the local configuration file, the configuration file can be updated without stopping, and the hot deployment of the configuration information is realized; The original data obtained by the target database is processed by data verification, data cleaning, data conversion, adding default values, etc. to ensure the accuracy and integrity of business data and meet actual business needs; the database and webservice interfaces are set in the configuration information and server files and other data source types, as well as the corresponding attribute information, realizes flexible connection with various types of data sources, fast data extraction, and reduces development costs.

Further effects of the above non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

The accompanying drawings are used for better understanding of the present invention and do not constitute an improper limitation of the present invention. in:

1 is a schematic diagram of the main steps of a data processing method according to an embodiment of the present invention;

2 is a schematic diagram of a system structure of a data processing method according to an embodiment of the present invention;

3 is a schematic diagram of a technical architecture of a data processing method according to an embodiment of the present invention;

4 is a schematic diagram of a data architecture of a data processing method according to an embodiment of the present invention;

5 is a schematic flow chart of a data processing method according to an embodiment of the present invention;

6 is a schematic diagram of a configuration information management interface according to an embodiment of the present invention;

7 is a schematic diagram of an interface for generating configuration information of a data source of a database type according to an embodiment of the present invention;

8 is a schematic diagram of main modules of a data processing system according to an embodiment of the present invention;

FIG. 9 is an exemplary system architecture diagram to which an embodiment of the present invention may be applied;

FIG. 10 is a schematic structural diagram of a computer apparatus suitable for implementing the electronic device according to the embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

FIG. 1 is a schematic diagram of main steps of a data processing method according to an embodiment of the present invention. As shown in FIG. 1 , the data processing method according to the embodiment of the present invention mainly includes the following steps:

Step S101: The configuration management center obtains configuration information generated for different types of data sources, and pushes the configuration information to the front-end computer. The configuration management center provides the function of setting configuration information through the configuration information management interface. The administrator fills in the above fields on the configuration information management interface to generate configuration information. The configuration information includes: the data source identifier and data source type of each data source. and corresponding attribute information, the attribute information includes connection information of the data source.

Step S102: The front-end computer encapsulates the configuration information into a configuration information entity class, performs data verification on the configuration information entity class, and writes the configuration information in the configuration information entity class that has passed the verification into a local configuration file. After receiving the configuration information, the front-end computer encapsulates the configuration information into the configuration information entity class, and then performs data verification on the configuration information of the configuration information entity class. The data verification includes the non-empty checksum of each field and the rule check. The configuration information that has passed the verification is written into the local configuration file to achieve the goal of updating the configuration information without stopping.

Step S103: After receiving the data extraction request, the current host computer obtains the corresponding data source type and attribute information from the configuration information of the local configuration file according to the target data source identifier of the data extraction request. The data calling direction sends a data extraction request to the front-end computer, and the data extraction request includes the target data source identifier, such as the name and serial number of the target data source. After receiving the data extraction request, the front-end computer obtains the data source type and attribute information corresponding to the target data source from the locally stored configuration information according to the target data source identifier.

Step S104: The front-end computer loads the corresponding data engine according to the data source type, so as to establish a connection with the target data source based on the data engine and the attribute information, then obtain business data from the target data source, and return The business data is returned. The front-end computer loads the corresponding data engine according to the acquired data source type. For example, if the data source type is a database, the database engine is loaded; for another example, if the data source type is a webservice interface, the webservice interface engine is loaded.

Use the data engine and the connection information of the target data source indicated in the attribute information to establish a connection with the target data source. For example, if the data source type is database, the connection information is the database account name and password. After the front-end computer is connected to the target data source, the original data can be obtained from the target data source. Data is returned to the data caller.

FIG. 2 is a schematic diagram of a system structure of a data processing method according to an embodiment of the present invention. As shown in FIG. 2 , the front-end computer in the embodiment of the present invention is respectively connected with the hospital system and the data platform system. The front-end machine is divided into interface layer, business processing layer and data layer. The interface layer is used to expose the configuration information interface to the data platform system to receive the configuration information pushed by it, and send the configuration information to the business processing layer; it is also used to receive the data extraction request sent by the data platform system (that is, the data caller), The data extraction request is sent to the business processing layer; it is also used to provide external business data API (Application Programming Interface) and basic API. The business data layer includes inpatient data API and outpatient data API. The basic API includes: spring actuator monitoring component, Log uploading components, etc., to realize remote adjustment of configuration information, remote monitoring of front-end network and server running status.

The business processing layer is used to write the configuration information into the configuration information entity class, perform data verification on the configuration information of the configuration information entity class, and write the verified configuration information into the local configuration file; it is also used to receive a data extraction request when After that, read the configuration information entity class and establish a connection with the corresponding target data source; it is also used to perform data verification, data cleaning, data conversion and adding default values for the medical data fed back by the data layer. Process, get business data, and return the business data to the interface layer.

The data layer is used to dynamically adapt to different data sources, load the corresponding data engine according to the data source type provided in the configuration information, then obtain medical data, and feed the medical data to the business processing layer.

FIG. 3 is a schematic diagram of a technical architecture of a data processing method according to an embodiment of the present invention, and FIG. 4 is a schematic diagram of a data architecture of a data processing method according to an embodiment of the present invention. As shown in FIG. 3 , the operating system adopted by the front-end of the embodiment of the present invention is Ubuntu Linux 16.04, the version of JDK (Java Development Kit, a software development kit) is 1.8, and the version of J2EE (Java 2 Platform Enterprise Edition, Java 2 Platform Enterprise Edition) is The container is Tomcat7, and the framework is SpringBoot 1.5.6. As shown in FIG. 4 , the data processing system of the embodiment of the present invention is implemented by a hospital layer, a front-end computer layer, a platform layer and a background layer. The hospital layer includes various types of data sources, such as databases, webservice interfaces, and server files. The front-end layer implements the functions described in Figure 2. The platform layer and the background layer are the data callers.

FIG. 5 is a schematic flowchart of a main flow of a data processing method according to an embodiment of the present invention. As shown in FIG. 5 , the data processing method according to the embodiment of the present invention mainly includes the following steps:

Step S501: The configuration management center of the data platform system generates configuration information for different types of data sources, and pushes the configuration information to the front-end computer. The configuration management center will push the configuration information to the interface layer of the front end. The configuration information includes: data source identification, data source type, connection information and other parameter information. In the embodiment, the data source types include: a database view, a webservice interface, and an FTP (File Transfer Protocol, file transfer protocol) file.

When the data source type is database view, the connection information includes: connection address, connection login user, connection login password; other parameter information includes data configuration name, data access method, execution sql (structured query language, structured query language), usage status , operation, the data configuration name is used to indicate which part of the data source to obtain, and the status is used to indicate enabled or disabled status.

When the data source type is a webservice interface, the connection information includes: IP address, port|URL (Uniform Resource Locator, Uniform Resource Locator) of the data source, and other parameter information includes encrypted fields and the like.

When the data source type is an FTP file, the connection information includes: FTP protocol address, account number and password; other parameter information includes document path information, etc. The document path information is used to indicate the document path of the data source to obtain data. In an optional embodiment, before the configuration information is pushed to the front-end computer, a non-null check may be performed on all the configuration information.

The configuration management center manages the configuration information of each data source through the configuration information management interface. FIG. 6 is a schematic diagram of a configuration information management interface according to an embodiment of the present invention. As shown in FIG. 6 , the configuration information management interface of the embodiment of the present invention may include fields such as creation time, hospital code, hospital name, hospital level, province and city where the hospital is located, whether to point to health insurance, data source, and operation. The operation field can realize functions such as modifying, defining, and pushing configuration information. The administrator clicks the his configuration button to jump to the configuration information generation interface corresponding to the data source type of the hospital system, and set specific configuration information. Among them, his is hospital information system, hospital information system. The administrator can push the configuration information to the front-end computer by clicking the Push Configuration button.

FIG. 7 is a schematic diagram of an interface for generating configuration information of a data source of a database type according to an embodiment of the present invention. As shown in FIG. 7 , the configuration information generation interface of the data source of the database type in the embodiment of the present invention may include data configuration name, connection login password, connection address, connection login user, data source type, data access method, execution sql, Use fields like status, action, etc. The operation field can realize the function of modifying and deleting the configuration information of the hospital system. The interface also includes the function of adding his configuration. The administrator can click the corresponding position on the interface to add a new his configuration.

Step S502: the front-end computer encapsulates the configuration information into the configuration information entity class, performs data verification on the configuration information of the configuration information entity class, writes the verified configuration information into the local configuration file, and loads the configuration bean into the spring container . After receiving the configuration information, the interface layer of the front-end computer sends the configuration information to the business processing layer; the business processing layer encapsulates the configuration information into the configuration information entity class, and then performs data verification on the configuration information of the configuration information entity class, and then checks the configuration information. The configuration information that has passed the verification is written into the local configuration file (config_his.properties), which realizes the goal of updating the configuration information without stopping, and loads the configuration bean into the spring container for invocation during data business processing.

The business processing layer performs data verification on the configuration information of the configuration information entity class, including non-null verification and rule verification. The data source type is a required field, and the non-null check is to verify whether the data source type is empty. If it is empty, it can be set as the default type, such as a database view. The rule verification is to verify whether the fields of the configuration information meet the set rules, for example, the connection address field cannot be empty. If the verification fails, feedback prompt information to the configuration management center.

When the program is started for the first time, the interface layer of the front-end computer loads the local configuration file into the cached configuration information entity class. At this time, the attribute in the local configuration information entity class is null, and the program starts normally.

Step S503: After receiving the data extraction request from the data platform system, the front-end computer obtains the corresponding data source type and attribute information from the configuration information of the configuration information entity class according to the target data source identifier of the data extraction request. After receiving the data extraction request, the interface layer of the front-end computer sends the data extraction request to the business processing layer, and the data extraction request includes the target data source identifier; the business processing layer obtains the corresponding data from the local configuration file according to the target data source identifier. data source type and attribute information.

Step S504: The front-end computer loads the corresponding data engine according to the data source type, so as to establish a connection with the target data source based on the data engine and attribute information. The business processing layer of the front-end computer instantiates a data implementation entity class to load the corresponding data engine, and then establishes a connection with the target data source corresponding to the connection information indicated in the attribute information through the data engine. For example, if the data source type is a database view, the business processing layer will instantiate a data implementation entity class named SqlDataDaoImpl.java to load the corresponding data engine, and then establish a connection with the target database corresponding to the connection information through the data engine.

For another example, if the data source type is a webservice interface, the business processing layer instantiates a data implementation entity class to load the webservice interface engine, and then establishes a connection with the webservice interface corresponding to the connection information through the webservice interface engine.

For another example, if the data source type is an FTP file, a data implementation entity class is instantiated to load the FTP engine, and then a connection is established with the FTP server corresponding to the connection information through the FTP engine.

Step S505: Obtain original data from the target data source, preprocess the original data to obtain business data, and return the business data to the data platform system. After the business processing layer of the front-end computer successfully establishes a connection with the target data source, after obtaining the original data from the target data source through the data layer, the original data is processed such as data verification, data cleaning, data conversion, and adding default values. The obtained business data is returned to the data platform system. This embodiment can flexibly meet the different data supply modes of most hospital systems in the country, and has good versatility. Among them, data cleaning is to check data consistency and delete invalid values.

Data verification includes non-null verification and data exception verification. Data verification is realized by setting the value rules of the data (such as the default value space), and the obtained data value conforming to the value rules is considered to pass the verification, including but not limited to: the basic medical expense information is a non-empty positive value and is a numeric value , The unit price, quantity, amount, medical insurance price limit and other data in the cost details information are four decimals, the unit price and quantity cannot be zero, the discharge diagnosis cannot be empty, the discharge time cannot be empty, the ID card is 18 digits and only the last One can be a letter, and the admission diagnosis cannot be empty.

Data transformation includes large text field processing, dictionary value transformation. The content to be processed in data conversion is mainly the processing of electronic medical record data. Electronic medical record data is generally a large text field, and key information is mixed in a large piece of text. After parsing this text, format information is obtained and packaged into the electronic medical record. In the entity class, it is then returned to the data platform system through the interface layer. The field value conversion is realized by establishing dictionary information (dictionary table in the embodiment): when data is received, call the dictionary table to assign data to the set field information, for example, the detailed category obtained from the hospital system is 1, which is automatically converted to " Class A".

Add default value: When some specified data cannot get a value, a default value will be added. For example, the length of stay in hospital, if it is not available, the time of discharge from the hospital can be subtracted from the time of admission. For another example, birthday and gender, if they cannot be obtained, the birthday and gender can be inferred from the ID card information. Another example is the type of consultation. If it cannot be obtained, the default type of consultation can be added according to the calling method of the data.

In addition, when the data source type is database view, the implementation process of the data layer to obtain the original data from the target data source is as follows: instantiate a data implementation entity class named SqlDataDaoImpl.java, the function implemented by this entity class is to load the view information, Send a database call request, encapsulate the data returned by the request, and return it to the business processing layer. When the data source type is a webservice interface, the implementation process of the data layer obtaining the original data from the target data source is as follows: instantiate the corresponding implementation class to automatically load the webservice interface engine, obtain medical data by parsing the data format provided by the webservice interface, and return to the business processing layer.

In a preferred embodiment, if an error message occurs during the execution of steps S502-S505, an exception that the configuration information is wrong can be thrown, and the configuration management center can perform corresponding inspection and reconfiguration after receiving such abnormal information. work. The configuration management center can change the configuration information at any time, which can cope with the changeable scenes in the complex environment of the hospital system, and can speed up the connection with the hospital system at the same time.

It can be seen from the data processing method of the embodiment of the present invention that, after receiving the data extraction request, the data source type and attribute information corresponding to the target data source are obtained from the configuration information, so as to load the corresponding data engine and connect with the target data source. Establish a connection, and then obtain business data of the target data source flexibly and quickly; by obtaining configuration information and writing the configuration information to the local configuration file, the configuration file can be updated without stopping; the original data obtained from the target database can be used for data calibration. A series of processes such as verification, data cleaning, data conversion, and adding default values are performed to ensure the accuracy and integrity of business data and meet actual business needs; data source types such as database, webservice interface, and server file are set in the configuration information, and The corresponding attribute information realizes flexible connection with various types of data sources, fast data extraction, and reduces development costs.

FIG. 8 is a schematic diagram of main modules of a data processing system according to an embodiment of the present invention. As shown in FIG. 8 , the data processing system 800 according to the embodiment of the present invention mainly includes: a configuration management center 801 and a front-end 802 , and the front-end 802 includes: a package writing module 8021 , a data acquisition module 8022 and a data return module 8023 ,in,

The configuration management center 801 is configured to acquire configuration information generated for different types of data sources, and push the configuration information to the front-end computer. The configuration management center provides the function of setting configuration information through the configuration information management interface. The administrator fills in the above fields on the configuration information management interface to generate configuration information. The configuration information includes: the data source identifier and data source type of each data source. and corresponding attribute information, the attribute information includes connection information of the data source.

The encapsulation and writing module 8021 is used to encapsulate the configuration information into a configuration information entity class, perform data verification on the configuration information entity class, and write the configuration information in the configuration information entity class that has passed the verification into a local configuration file . After receiving the configuration information, the front-end computer encapsulates the configuration information into the configuration information entity class, and then performs data verification on the configuration information of the configuration information entity class. The data verification includes the non-empty checksum of each field and the rule check. The configuration information that has passed the verification is written into the local configuration file to achieve the goal of updating the configuration information without stopping.

The data acquisition module 8022 is configured to acquire the corresponding data source type and attribute information from the configuration information of the local configuration file according to the target data source identifier of the data extraction request after receiving the data extraction request. The data calling direction sends a data extraction request to the front-end computer, and the data extraction request includes the target data source identifier, such as the name and serial number of the target data source. After receiving the data extraction request, the front-end computer obtains the data source type and attribute information corresponding to the target data source from the locally stored configuration information according to the target data source identifier.

The data return module 8023 is used to load the corresponding data engine according to the data source type, to establish a connection with the target data source based on the data engine and the attribute information, and then obtain business data from the target data source, Return the business data. The front-end computer loads the corresponding data engine according to the acquired data source type. For example, if the data source type is a database, the database engine is loaded; for another example, if the data source type is a webservice interface, the webservice interface engine is loaded.

Use the data engine and the connection information of the target data source indicated in the attribute information to establish a connection with the target data source. For example, if the data source type is database, the connection information is the database account name and password. After the front-end computer is connected to the target data source, the original data can be obtained from the target data source. Data is returned to the data caller.

In addition, the front-end computer 802 in the embodiment of the present invention may further include: an assignment module, configured to perform data assignment on the setting field information in the formatted information according to the pre-established dictionary information.

It can be seen from the above description that, after receiving the data extraction request, the data source type and attribute information corresponding to the target data source are obtained from the configuration information to load the corresponding data engine and establish a connection with the target data source, which enables flexible , Quickly obtain the business data of the target data source; at the same time, by writing the configuration information into the local configuration file, the configuration file can be updated without stopping, and the hot deployment of the configuration information is realized.

FIG. 9 shows an exemplary system architecture 900 of a data processing method or data processing system to which embodiments of the present invention may be applied.

As shown in FIG. 9 , the system architecture 900 may include terminal devices 901 , 902 , and 903 , a network 904 and a server 905 . The network 904 is a medium used to provide a communication link between the terminal devices 901 , 902 , 903 and the server 905 . Network 904 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 901, 902, 903 to interact with the server 905 through the network 904 to receive or send messages and the like. Various communication client applications may be installed on the terminal devices 901 , 902 and 903 , such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, and the like.

The terminal devices 901, 902, 903 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

The server 905 may be a server that provides various services, for example, a background management server that an administrator uses to process data extraction requests sent by the terminal devices 901 , 902 , and 903 . The background management server can obtain the corresponding data source type and attribute information according to the target data source identifier of the data extraction request, establish a connection with the target data source, obtain business data from the target data source, and feed back the processing results (such as business data) to Terminal Equipment.

It should be noted that the data processing method provided by the embodiments of the present application is generally executed by the server 905 , and accordingly, the data processing system is generally set in the server 905 .

It should be understood that the numbers of terminal devices, networks and servers in FIG. 9 are only illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

According to an embodiment of the present invention, the present invention further provides an electronic device and a computer-readable medium.

The electronic device of the present invention comprises: one or more processors; storage means for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more programs A plurality of processors implement a data processing method according to an embodiment of the present invention.

The computer-readable medium of the present invention stores a computer program thereon, and when the program is executed by a processor, a data processing method according to an embodiment of the present invention is implemented.

Referring to FIG. 10 below, it shows a schematic structural diagram of a computer system 1000 suitable for implementing an electronic device according to an embodiment of the present invention. The electronic device shown in FIG. 10 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present invention.

As shown in FIG. 10, a computer system 1000 includes a central processing unit (CPU) 1001, which can be loaded into a random access memory (RAM) 1003 according to a program stored in a read only memory (ROM) 1002 or a program from a storage section 1008 Instead, various appropriate actions and processes are performed. In the RAM 1003, various programs and data necessary for the operation of the computer system 1000 are also stored. The CPU 1001 , the ROM 1002 , and the RAM 1003 are connected to each other through a bus 1004 . An input/output (I/O) interface 1005 is also connected to the bus 1004 .

The following components are connected to the I/O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc. ; and a communication section 1009 including a network interface card such as a LAN card, a modem, and the like. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I/O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.

In particular, according to the disclosed embodiments of the present invention, the processes described in the main step diagrams above may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods illustrated in the main step diagrams. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 1009, and/or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-described functions defined in the system of the present invention are executed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The modules involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described modules can also be set in the processor, for example, it can be described as: a processor includes a package writing module, a data acquiring module and a data returning module. Among them, the names of these modules do not constitute a limitation of the module itself under certain circumstances. For example, the encapsulation writing module can also be described as "encapsulate the configuration information into the configuration information entity class, and The entity class performs data verification, and writes the configuration information in the configuration information entity class that has passed the verification to the module of the local configuration file".

As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by a device, the device includes: the configuration management center obtains configuration information generated for different types of data sources, The configuration information is pushed to the front-end computer; wherein, the configuration information includes: data source identification, data source type and attribute information; the front-end computer encapsulates the configuration information into a configuration information entity class, and the configuration information entity The class performs data verification, and writes the configuration information in the configuration information entity class that has passed the verification into the local configuration file; when the front-end computer receives the data extraction request, according to the target data source identifier of the data extraction request, Obtain the corresponding data source type and attribute information from the configuration information of the local configuration file; the front-end computer loads the corresponding data engine according to the data source type, so as to be based on the data engine and the attribute information and the corresponding data engine. The target data source establishes a connection, then obtains business data from the target data source, and returns the business data.

It can be seen from the above description that, after receiving the data extraction request, the data source type and attribute information corresponding to the target data source are obtained from the configuration information to load the corresponding data engine and establish a connection with the target data source, which enables flexible , Quickly obtain the business data of the target data source; at the same time, by writing the configuration information into the local configuration file, the configuration file can be updated without stopping, and the hot deployment of the configuration information is realized.

The above product can execute the method provided by the embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference may be made to the method provided by the embodiment of the present invention.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A data processing method, comprising:

the method comprises the steps that a configuration management center obtains configuration information generated for different types of data sources and pushes the configuration information to a front-end processor; wherein the configuration information comprises: data source identification, data source type and attribute information;

the front-end processor packages the configuration information into a configuration information entity class, performs data verification on the configuration information entity class, and writes the configuration information in the configuration information entity class passing the verification into a local configuration file;

after receiving a data extraction request, the front-end processor acquires corresponding data source type and attribute information from the configuration information of the local configuration file according to a target data source identifier of the data extraction request;

and the front-end processor loads a corresponding data engine according to the type of the data source so as to establish connection with the target data source based on the data engine and the attribute information, then acquires service data from the target data source and returns the service data.

2. The method of claim 1, wherein obtaining traffic data from the target data source comprises:

acquiring original data from the target data source, and preprocessing the original data to obtain service data; wherein the pre-treatment comprises any one or more of: data verification, data cleaning, data conversion and default value addition.

3. The method of claim 2, wherein performing a data check on the raw data comprises:

and according to a set value-taking rule, screening field information meeting the value-taking rule from the field information of the original data, and summarizing the field information meeting the value-taking rule.

4. The method of claim 2, wherein data transforming the raw data comprises:

and analyzing the original data to obtain formatting information, and packaging the formatting information into an entity class.

5. The method of claim 4, wherein the step of parsing the raw data to obtain the formatted information further comprises:

performing data assignment on set field information in the formatting information according to pre-established dictionary information;

encapsulating the formatted information into an entity class, including: and packaging the formatted information after value assignment into the entity class.

6. The method of any of claims 1 to 5, wherein the data source types include any one or more of: the system comprises a database, a webservice interface and a server file.

7. The method as claimed in claim 6, wherein when the data source type is database, the attribute information includes connection address, connection login user, connection login password and execution SQ L of the data source;

when the data source type is a webservice interface, the attribute information comprises an IP address and a port | UR L of the data source, and when the data source type is a server file, the attribute information comprises an FTP protocol address, an account and a password.

8. A data processing system, comprising: configuring a management center and a front-end processor, wherein,

the configuration management center is used for acquiring configuration information generated for different types of data sources and pushing the configuration information to the front-end processor; wherein the configuration information comprises: data source identification, data source type and attribute information;

the front-end processor is used for packaging the configuration information into a configuration information entity class, carrying out data verification on the configuration information entity class and writing the configuration information in the configuration information entity class passing the verification into a local configuration file;

the front-end processor is further configured to, after receiving a data extraction request, obtain corresponding data source type and attribute information from the configuration information of the local configuration file according to a target data source identifier of the data extraction request;

the front-end processor is further configured to load a corresponding data engine according to the data source type, establish a connection with the target data source based on the data engine and the attribute information, then acquire service data from the target data source, and return the service data.

9. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.

Images