CN114817391A - A method and device for analyzing risk control business data - Google Patents

Abstract

The invention provides a method and a device for analyzing wind control service data, wherein the method comprises the following steps: acquiring an analysis task, analyzing an analysis data source, and acquiring metadata of data to be analyzed; loading a corresponding analysis template, and calling metadata of the data to be analyzed; loading the data to be analyzed corresponding to the corresponding metadata into a built-in memory database and executing SQL to obtain an analysis result index; and generating a data analysis report according to the analysis template and the analysis result index. The method is used for solving the problems of long required processing period and low processing speed of the existing wind control service data analysis method.

Description

A method and device for analyzing risk control business data

technical field

The invention relates to the field of data analysis, in particular to a data analysis method and device for a risk control business.

Background technique

In the risk control business, there are often user groups with suspected risks. It is necessary to analyze and evaluate the quality of the user groups and individual users, such as the unpaid rate of x days, the unpaid amount of x days and other indicators. Crowd quality assessment in risk control business scenarios often relies on professional BI pass, through offline data warehouse data, and data analysis through complex SQL. There are problems such as high professional requirements for demand processing personnel, long demand processing cycle, and slow processing speed.

SUMMARY OF THE INVENTION

In view of the above shortcomings of the prior art, the present invention provides a method and device for analyzing data of risk control business, so as to improve the problems of long processing period and slow processing speed in the existing method for analyzing data of risk control business.

To achieve the above purpose and other related purposes, the present invention provides a data analysis method for risk control business, comprising the steps of:

Obtain analysis tasks, parse and analyze data sources, and obtain metadata of the data to be analyzed;

Load the corresponding analysis template, and retrieve the metadata of the data to be analyzed;

Loading data corresponding to the corresponding metadata into the built-in memory database and executing SQL to obtain an analysis result indicator;

A data analysis report is generated according to the analysis template and the analysis result index.

In an embodiment of the present invention, before the step of acquiring the analysis task includes: storing the metadata of the data of the risk control service in HBASE.

In an embodiment of the present invention, the step of storing the metadata of the data of the risk control service in the HBASE includes: synchronizing the metadata of the data of the risk control service in HIVE to the HBASE.

In an embodiment of the present invention, the step of loading the corresponding analysis template includes: based on the analysis task, loading the corresponding template from the template center.

The present invention also provides a risk control business data analysis device, comprising: an operation layer, a business layer, a storage layer and a processing layer, wherein the operation layer is used for generating operation instructions; the business layer is used for business management, receiving operation instructions and performing operations. Corresponding business analysis, and send the analysis result; the storage layer is used to store the data of the risk control business; the processing layer is used to receive the analysis result, call the data from the storage layer and execute the operation instruction.

In an embodiment of the present invention, the operation instructions include data synchronization and analysis tasks.

In an embodiment of the present invention, the business layer includes: an analysis data source management module, an analysis template management module, and an analysis task management module; the analysis data source management module is used to analyze and edit metadata and manage data that conforms to permissions source; the analysis template management module is used to manage templates of SQL and analysis reports; the analysis task management module is used to manage analysis tasks.

In an embodiment of the present invention, a template center is set in the analysis template management module.

In an embodiment of the present invention, the processing layer includes: a task execution module, a SQL operation module, a task indicator storage module, and an analysis report module; wherein the task execution module is used to execute analysis tasks, load analysis data, and load analysis template; the SQL operation module is used to perform SQL operation; the task index storage module is used to save the analysis result index of the analysis task; the analysis report template is used to generate an analysis report according to the analysis result index and the analysis template.

In an embodiment of the present invention, the storage layer includes HIVE, HBASE and MONGO; the HIVE is used to store the data of the risk control service; the HBASE is used to store the metadata of the data of the risk control service; the MONGO uses Used to store relevant data for analysis tasks.

The risk control business data analysis method of the present invention loads the data to be analyzed corresponding to the metadata into the built-in memory database, executes SQL for data analysis, obtains the analysis result, and outputs the report file according to the data analysis template, thereby ensuring the fast and efficient data requirements. , which improves the processing speed of data analysis and shortens the processing cycle.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

Fig. 1 is the flow chart of the risk control business data analysis method of the present invention;

FIG. 2 is a structural block diagram of a risk control business data analysis device according to the present invention.

Component label description

100, operation layer; 200, business layer; 210, analysis data source management module; 220, analysis template management module; 230, analysis task management module; 300, processing layer; 310, task execution module; 320, SQL operation module; 330 , task indicator storage module; 340, analysis report module; 400, storage layer; 410, HIVE; 420, HBASE; 430, MONGO; 500, task monitoring layer.

Detailed ways

The embodiments of the present invention are described below through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other under the condition of no conflict. It should also be understood that the terms used in the embodiments of the present invention are for describing specific specific embodiments, rather than for limiting the protection scope of the present invention. In the following examples, the test methods without specific conditions are usually in accordance with conventional conditions, or in accordance with the conditions suggested by various manufacturers.

It should be noted that the terms such as "up", "down", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of description and clarity, and are not used to limit this The implementable scope of the invention, and the change or adjustment of the relative relationship thereof, shall also be regarded as the implementable scope of the present invention without substantially changing the technical content.

Please refer to FIG. 1 to FIG. 2, the present invention provides a data analysis method for risk control business, including the steps:

The metadata of the data of the synchronization risk control business, such as order data or user data, is synchronized by HIVE 410 to HBASE 420 for storage through data synchronization tasks or one-time data synchronization. HIVE is an offline warehouse. It is a data warehouse tool based on Hadoop for data extraction, transformation and loading. It is a mechanism that can store, query and analyze large-scale data stored in Hadoop. The hive data warehouse tool can map the structured data file into a database table and provide SQL (Structured Query Language, Structured Query Language) query function. HBASE is a distributed, column-oriented open source database, belonging to NOSQL (generally refers to non-relational database) database.

Obtain analysis tasks through user submission, analyze and analyze data sources, and obtain metadata of the data to be analyzed. Wherein, the step of acquiring the analysis task by the user submission method includes: receiving the user's new task instruction and the user information uploaded by the staff, providing an analysis template and selecting and confirming the analysis template by the user, and using the user information and the analysis template selected by the user. The analysis task is generated and fed back to the user for the user to submit the analysis task to the risk control business data analysis device. User information includes, but is not limited to, the user's mobile phone number and/or the user's serial number.

Based on the analysis task, load the corresponding template from the template center, query and retrieve the metadata of the data to be analyzed from the HBASE 420;

Load the data corresponding to the corresponding metadata into the built-in memory database and execute the SQL corresponding to the analysis template to obtain the analysis result indicators; the analysis result indicators include the detailed items of the analysis results, such as the unpaid rate of X days and the unpaid amount of X days Wait.

A data analysis report is generated according to the analysis template and the analysis result indicators, saved and fed back to the user. The user can select subsequent related operations according to their own needs, and subsequent related operations include editing, deleting, running, and downloading.

The risk control business data analysis method of the present invention stores the metadata to be analyzed in a NOSQL (generally referred to as a non-relational database) database, loads the data within the condition range into the built-in memory, and executes the template SQL through the built-in memory database to perform data analysis, The analysis results are obtained, and the report file is output according to the data analysis template; thereby ensuring the fast and efficient data requirements, improving the processing speed of data analysis, and shortening the processing cycle.

Referring to FIG. 2, the present invention also provides a risk control business data analysis device, including: an operation layer 100, a business layer 200, a storage layer 400 and a processing layer 300, wherein the operation layer 100 is used to generate operation instructions; the business layer 200 It is used for business management, receiving operation instructions, performing corresponding business analysis, and sending the analysis results; the storage layer 400 is used to store the data of the risk control business; the processing layer 300 is used to receive the analysis results and call from the storage layer 400 data and execute the operating instructions.

Referring to FIG. 2 , in an embodiment of the present invention, the operation instructions include data synchronization and analysis tasks. Among them, the metadata of data synchronization risk control business analysis, such as order data or user data, is synchronized by HIVE 410 to HBASE 420 for storage through data timing synchronization tasks or one-time data synchronization. Analysis tasks are tasks that submit data analysis requests. For example, after analyzing the quality of the input user's orders in the past 7 days, create a new analysis task in the system, upload the analyst, and submit the task form, which is a data analysis request.

Referring to FIG. 2, in an embodiment of the present invention, the business layer 200 includes: an analysis data source management module 210, an analysis template management module 220, and an analysis task management module 230; the analysis data source management module 210 is used for analysis And edit metadata and manage data sources that comply with permissions, for example, analyze the addition, deletion, modification and query of metadata, and manage data sources allowed by the system, such as order data, user data, etc. The analysis template management module 220 is used to manage the template of SQL and analysis report executed by the analysis engine for index processing, and is the core module of the data analysis device. The analysis task management module 230 is used to manage the analysis tasks, for example, the management of the life cycle of the analysis tasks, such as adding, starting, canceling, and so on. The analysis template management module 220 is provided with a template center, and the template center is used for storing analysis templates.

Referring to FIG. 2, in an embodiment of the present invention, the processing layer 300 includes: a task execution module 310, an SQL operation module 320, a task indicator storage module 330, and an analysis report module 340; wherein, the task execution module 310 uses For performing analysis tasks, loading analysis data and loading analysis templates; the SQL operation module 320 is used to perform SQL operations, that is, the memory data analysis engine performs SQL index operations based on recorded memory data; the task index storage module 330 is used to save the data of the analysis task. Analysis result index; the analysis report module 340 is configured to generate an analysis report according to the analysis result index and the analysis template.

Referring to FIG. 2, in an embodiment of the present invention, the storage layer 400 includes HIVE 410, HBASE 420, and MONGO 430; the HIVE 410 is an offline data warehouse for storing all data of risk control services; the HBASE 420 Used to store metadata of the data of the risk control business; the MONGO 430 is used to store the relevant data of the analysis task, and the relevant tasks of the analysis task include the analysis task data and the analysis input data.

Referring to FIG. 2 , in an embodiment of the present invention, the risk control business data analysis device further includes a task monitoring layer 500 for monitoring and analyzing tasks to facilitate management.

Compared with the previous BI (Business Intelligence, business intelligence) data analysis method, the present invention is simpler and has no threshold. It only needs to submit the data to be analyzed, the analysis report can be downloaded in a few seconds, and the work of the previous professional BI can be completed in an hour. quantity. According to the data analysis requirements, the invention can analyze the input data at any time by configuring a template once, and only need to create a new analysis template, select a data source, and configure the analysis report format according to the analysis requirements, and then the analysis data results can be obtained. The invention can realize the standardization, engineering and systemization of the demand process, and establish an agile and efficient data analysis method. Users only need the following operations: select the specified time range, determine the analysis template, submit the analysis task, and download the analysis report, and then the data analysis results can be obtained in seconds. Therefore, the present invention greatly lowers the threshold for data analysis and improves the efficiency of data analysis. Therefore, the present invention effectively overcomes some practical problems in the prior art and thus has high utilization value and use significance.

The above-mentioned embodiments merely illustrate the principles and effects of the present invention, but are not intended to limit the present invention. Anyone skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed in the present invention should still be covered by the claims of the present invention.

Claims (10)

1. A method for analyzing wind control service data is characterized by comprising the following steps:

acquiring an analysis task, analyzing an analysis data source, and acquiring metadata of data to be analyzed;

loading a corresponding analysis template, and calling metadata of the data to be analyzed;

loading the data to be analyzed corresponding to the corresponding metadata into a built-in memory database and executing SQL to obtain an analysis result index;

and generating a data analysis report according to the analysis template and the analysis result index.

2. The method for analyzing wind control business data according to claim 1, wherein the step of obtaining the analysis task is preceded by: and storing the metadata of the data of the wind control service into HBASE.

3. The method of claim 2, wherein the step of storing metadata of the data of the wind services in the HBASE comprises: and synchronizing the metadata of the data of the wind control service in the HIVE into HBASE.

4. The method for analyzing wind-controlled business data according to claim 1, wherein the step of loading the corresponding analysis template comprises: and loading a corresponding template from a template center according to the analysis task.

5. A wind-controlled business data analysis device is characterized by comprising:

the operation layer is used for generating an operation instruction;

the service layer is used for carrying out service management, receiving the operation instruction, carrying out corresponding service analysis and sending an analysis result;

the storage layer is used for storing data of the wind control service;

and the processing layer is used for receiving the analysis result, calling data from the storage layer and executing the operation instruction.

6. The wind control business data analysis device of claim 5, wherein the operational instructions comprise data synchronization and analysis tasks.

7. The wind-controlled business data analysis device of claim 5, wherein the business layer comprises: the analysis data source management module, the analysis template management module and the analysis task management module; the analysis data source management module is used for analyzing and editing metadata and managing data sources conforming to the authority; the analysis template management module is used for managing SQL and the template of the analysis report; the analysis task management module is used for managing analysis tasks.

8. The wind control business data analysis device according to claim 7, wherein a template center is disposed in the analysis template management module.

9. The wind control business data analysis device of claim 5, wherein the processing layer comprises: the system comprises a task execution module, an SQL operation module, a task index storage module and an analysis report module; the task execution module is used for executing analysis tasks, loading analysis data and loading analysis templates; the SQL operation module is used for carrying out SQL operation; the task index storage module is used for storing the analysis result index of the analysis task; and the analysis report template is used for generating an analysis report according to the analysis result index and the analysis template.

10. The wind controlled business data analysis device of claim 5, wherein the storage tier comprises HIVE, HBASE, and MONGO; the HIVE is used for storing data of the wind control service; the HBASE is used for storing metadata of data of the wind control service; the MONGO is used for storing relevant data of the analysis task.

Images