CN118484474A - Data index result calculation processing method, calculation processing system and application - Google Patents

Abstract

The invention discloses a data index result calculation processing method, which comprises the following steps: pre-constructing an index information database, and reading index information loading index information of the index information database; caching the index name through redis; forming a custom rule base from the index information according to the custom rule; multithreading uses the credit information obtained, and invokes the index calculation method from the user-defined rule base to calculate the index value; and storing the index result obtained by calculation into an Hbase database. The invention also discloses a computing system for realizing the computing method, which comprises the following steps: the system comprises a verification module, an index processing module and a result storage module; the verification module is used for verifying the calculation request according to the verification rule; the index processing module is used for calculating an index value according to the calculation request; the result storage module is used for storing information such as index results obtained through calculation into the Hbase database. The invention also discloses application of the method or the system in data index calculation processing.

Description

Data index result calculation processing method, calculation processing system and application

Technical Field

The invention belongs to the technical field of data computing and processing, and relates to a data index result computing and processing method, a computing and processing system and application.

Background

In the prior art, the sql statement is used for calculating the credit report index result, the credit report is required to be stored in a database for calculation after being analyzed, and the efficiency of calculating the index result is low.

The problem of operating database and efficiency of sql statement is solved by writing java logic codes and calculating indexes in memory, but each index item needs to write corresponding method logic, so that if the index item is too many, a plurality of methods need to be written, and when indexes are calculated, the corresponding method needs to be called according to judgment conditions for calculation, if a newly added index item needs to change codes, and each update needs to restart the item.

In practice, it is desirable to be able to complete the calculation of the index directly in memory without frequent database operations and to simplify the if/else structure, allowing the separation of business logic and data; the separated business logic must be easy to write, at least independently, faster than writing code; the separated business logic must be easier to read and understand and maintain than the original code; the application does not need to be restarted if new index calculation logic needs to be added.

Disclosure of Invention

In order to solve the defects in the prior art, the invention aims to provide a data index result calculation processing method, a calculation processing system and application.

The method solves the problems of updating maintenance and efficiency of java and sql calculation credit reporting indexes, and uses a rule engine to separate business logic from data. By loading the rule base when the item is started, the index calculation logic can be increased or modified in an incremental mode, so that the application program is prevented from being restarted every time the index is changed.

The rule engine is a component capable of reducing the complexity of a complex service logic component and the maintenance and expandability costs of an application program, and can separate a service decision from an application program code, receive data input, interpret a service rule and make the service decision according to the service rule.

When a user attempts to add a new index, the system checks whether the index has been added to the database based on the index name. If so, the system may prevent the new increase in the index. If not, the system adds the new index into the database, acquires the rule base stored in the Map according to the index name prefix, and adds the index calculation rule increment of the new index into the rule base corresponding to the index name prefix. If the rule base corresponding to the new index English name prefix does not exist in the Map, a new KieBase base is created for the new index name prefix, and the new KieBase base is updated into the Map. In addition, the system updates the new index name into the redis cache, which allows the system to dynamically add and calculate new indexes without restarting the entire application. Meanwhile, since repeated indexes cannot be added to the database, the uniqueness of the indexes is ensured.

Map is a data structure for storing key-value pairs, where each key is unique for quick lookup and retrieval of the value associated with it.

The KieBase library refers to a knowledge base in the rules engine that stores and manages rules. The rules engine may calculate and infer data from defined rules. The KieBase libraries in the present invention are used to store rules corresponding to different index name prefixes, and each KieBase library contains a set of rules for calculating the index associated with the index name prefix. When a new index is added to the system, the index calculation rule increment of the index is added to the corresponding KieBase library, so that the rule engine can dynamically process the new index calculation.

Redis is a high-performance key-value store database that is commonly used to cache commonly used data to increase the read speed of the system. The system updates the newly added index name to the redis cache, so that the system can be ensured to quickly access and retrieve the newly added index information without frequently accessing the database. By storing the newly added index name in the redis cache, the system can quickly verify the uniqueness of the index, thereby avoiding duplicate indexes from being added to the database.

In the invention, the data index result calculation processing method specifically comprises the following steps:

Firstly, constructing an index information database in advance, and reading index information and loading index information from the index information database;

Caching index names in the index information through redis;

thirdly, forming a custom rule base from the index information according to the custom rule;

step four, multithreading uses the credit investigation data obtained, call the index calculation method from the custom rule base to calculate the index value;

and step five, storing the index result obtained by calculation into an Hbase database.

In the first step, a data index information table is created and used as a database for storing all the existing indexes, the table stores index information including initialized index numbers, index names, index calculation logic, whether indexes are invalid and other fields, when the system is loaded, whether the indexes to be loaded are required to be loaded into a memory is judged according to the information stored in the database and corresponding to whether the indexes to be loaded are invalid, if the indexes to be loaded are not invalid, the indexes to be loaded are loaded into the memory, and if the indexes to be loaded are invalid, the indexes to be loaded are not loaded into the memory; when loading, whether the index to be loaded needs to be loaded or not is judged in advance, so that the memory utilization rate of the system can be improved, and only the needed index information is loaded.

And step two, caching the index name set inquired in the step one to Redis, retrieving the index name set from the Redis when the full index is required to be calculated, traversing the elements in the index name set, and calculating corresponding index values for each index name according to index calculation logic.

By using Redis cache, the system can quickly store and retrieve index name sets, avoiding frequent access to the database. In Redis, key-value pairs may be stored, retrieved, deleted, and checked using different commands such as SET, GET, DEL, EXISTS, etc. Specifically, in Redis, a SET Key Value command may be used to store Key Value pairs in Redis, a GET Key command may be used to retrieve Key values from Redis, a DEL Key command may be used to delete Key Value pairs from Redis, an EXISTS Key command may be used to check whether a Key EXISTS in Redis, a EXPIRE Key seconds command may be used to SET the expiration time of the Key, a fixed Key may be used in Redis to store index data, value may be stored as a Map SET, where Key is an index name, value may be used to store whether an index is enabled, and the latest data of the index may be obtained in real time using the Redis storage index.

Step three, one or more rule bases KieBase are created according to the size of the rule base or index name group configured by user definition and stored in maps, the Key value of each KieBase rule base corresponds to one type of index name prefix, the value in the maps corresponds to the KieBase rule base of the current class, kieBase rule bases are obtained according to keys, and then related rules are called according to index English names;

each class of index has a fixed prefix to facilitate distinguishing between each class of index, for example:

Number of non-credit transaction records: noncreditdetailJ2AA1; the non-credit transaction records a normal amount of payment status: noncreditdetailJ2CD1; total amount of tax owed to public information: publicinfoGAA1; public information owe taxes record number: publicinfoGAA2, etc., other index prefixes can be set according to actual needs;

KieBase is a knowledge repository, and KieBase only contains a plurality of rules, procedures, methods and the like, so that separation of data and business logic is realized. While Map is a key-value pair mapping container that maps keys to values.

In the fourth step, it is necessary to determine whether to calculate the existing full-quantity index or the index requested by the user according to the request parameters of the user.

If the user request parameter contains the index calculation name set, calculating the index requested by the user, otherwise, returning the total index of the calculation system to the user.

During calculation, the corresponding index in the custom rule base can be searched according to the index name prefix, then the related index calculation rule is called to calculate the index value,

In order to realize the index calculation without affecting the normal request of the user, an asynchronous thread mode is generally adopted for the index calculation. Each thread runs independently and does not interfere with the other thread, so that the stability and the reliability of the index calculation program can be ensured, and the concurrency capacity and the throughput of the system can be improved.

The thread may be set according to the number of CPU cores deploying the machine, with an upper limit of twice the number of CPU cores. Through continuous 62-hour pressure test, 830 ten thousand credit report stability and reliability are calculated to be 100%, the credit report about 1M is tested, the time for calculating 2034 indexes by using single threads is about 450 milliseconds, the time for calculating 2034 indexes by using multiple threads is about 180 milliseconds, and the multithread calculation efficiency is 2.5 times of that of the single threads.

And fifthly, storing the calculated result in an Hbase database by taking a report number on the personal credit report as a main key. Because Hbase is a column-oriented distributed database, a large amount of data can be processed quickly and efficiently.

The report number is a unique code for identifying a personal credit report, and the length of the report number is 22 bits, which is composed of a query date and a query sequence number.

By the method, the complex credit reporting index calculation process can be decomposed into application and processing processes of a plurality of independent components (comprising a relational database module, a redis cache module, a rule engine module, an Hbase database module and the like), and maintainability and flexibility of the whole system are improved. Meanwhile, by using technologies such as a rule engine, a distributed database and the like, rapid and accurate index calculation and data storage can be realized, and high-quality service is provided for users.

The invention also provides a processing system for realizing the data index result calculation processing method, which comprises the following steps: the system comprises a verification module, an index processing module and a result storage module;

the verification module is used for verifying the calculation request according to the verification rule, releasing the calculation request passing the verification, and intercepting the request which cannot pass the verification;

the index processing module is used for calculating an index value according to the calculation request;

The result storage module is used for storing the index result obtained by calculation, the calculated personal credit report file path, the personal credit report number and the calculation time into the Hbase database.

The index processing module may further comprise a plurality of components, each component being:

The component 1 is used for storing a full index initialization calculation rule;

the component 2 and the redis cache module are used for storing the full index rule names without requesting the database each time;

the component 3 and the rule engine module are used for calculating index results;

And the component 4 and the Hbase database module are used for storing calculation results.

The rule engine calculates the index result by storing the analyzed credit report data in a database like sql sentences and writing complicated call judgment like java codes, and searching corresponding calculation logic in a rule base according to the index name by putting the analyzed data in a memory.

The invention has the innovation point that the rule engine is combined with credit report analysis to realize rapid calculation of each index of the credit report. Various complex algorithms and calculation rules can be converted into a rule base, so that the high efficiency and accuracy of index calculation are realized. Meanwhile, because the rule engine is based on rules, a developer can flexibly formulate calculation rules according to actual needs, and maintainability and expandability of the system are greatly improved. Compared with the traditional index calculation method, the method not only improves the calculation speed, but also enables the codes to be more concise and understandable, and is applicable to various credit reporting scenes. These are all the innovative points of the present invention.

The credit report is stored in an XML file, and after the API analysis file is provided through the DOM4J, each data element and each attribute in the XML are obtained. Then, according to the structure of the credit report, the large memory object (including a plurality of small objects such as basic information, information summary, credit transaction details and the like) of the credit report is assembled, so that the credit report is represented in the memory by a unified data structure, and the subsequent data processing and calculation are convenient; and then, the XML is analyzed by the Dom4j to establish mapping, and the data in the credit report is mapped to a plurality of small object attributes to obtain data items.

Dom4j is a Java open source library for processing XML documents, and by using the API provided by the Dom4j library, XML files can be flexibly parsed and required element and attribute information can be obtained

The invention also provides an application of the processing method or the processing system in credit report data index calculation processing.

Compared with the prior art, the invention has the following beneficial effects:

1. firstly, the system stores the analyzed credit report data in a memory so as to be accessed quickly;

2. The system then uses a rule engine to generate rule bases, each of which contains a set of index calculation logic. Each index calculation logic is defined based on user-configured algorithms and parameters;

3. when calculating the index, the system searches corresponding calculation logic from the rule base according to the index name to be calculated, and takes the analyzed data as input parameters to be transmitted into the calculation logic;

4. Supporting dynamic updating of the rule base, if the index calculation logic needs to be added or modified, the rule base can be directly updated without restarting the project;

5. Compared with the prior art that the index result is calculated by using SQL sentences, the method has the advantages that the calculation speed is tens of times faster, and the calculation speed is more obvious along with more credit reporting and index items.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of the rule engine based calculation of metrics in accordance with the present invention.

Fig. 2 is a diagram of the architecture of the system of the present invention.

FIG. 3 is a diagram showing the new index.

FIG. 4 is a diagram showing the results of the indicators in detail.

Detailed Description

The present invention will be described in further detail with reference to the following specific examples and drawings. The procedures, conditions, experimental methods, etc. for carrying out the present invention are common knowledge and common knowledge in the art, except for the following specific references, and the present invention is not particularly limited.

The invention provides a credit sign index result calculation processing method which can effectively solve the problems of low efficiency, restarting after updating and the like faced by SQL and Java code calculation indexes based on a rule engine. The method not only can improve the index calculation efficiency, but also can flexibly update the rule base, and improves the maintainability and the flexibility of the system. Meanwhile, due to the fact that the rule engine is used, writing of calculation logic is simpler and more visual, and development difficulty and cost are reduced.

According to the invention, a plurality of credit indexes of each credit report are calculated based on a business rule engine, hbase is used as an index result storage database, redis is adopted to carry out caching processing on all index name data to be calculated, mongab is used to store item logs, so that a business scene of inquiring a large data amount and index result change is met, and online transaction efficiency is improved.

Specifically, the following steps are adopted:

1. First, the system will calculate multiple credit indices in each credit report based on the business rules engine. For each index, the system executes corresponding calculation logic according to the rules configured by the user, and stores the calculation result in the Hbase database. Because Hbase is a column-oriented distributed database, a large amount of data can be processed quickly and efficiently.

2. In order to avoid the need to acquire the index name from the database every time the index is calculated, the system uses redis to perform cache processing on all index name data to be calculated. Therefore, when calculating the index, the system only needs to acquire the index name from the cache, and performance loss caused by repeated access to the database is avoided.

3. At the same time, the system will also use mongab to store project logs for subsequent monitoring and analysis. By recording various events and abnormal conditions in the running process of the system, the problems can be found out in time and processed, and the stability and reliability of the system are ensured.

The method can realize rapid and accurate calculation of the credit reporting index and store the result in the Hbase database. The use of redis for caching can improve the performance and efficiency of the system. Meanwhile, the mongolib is used for storing the project log, so that developers can be helped to find and solve problems in time, and stability and reliability of the system are guaranteed.

The architecture diagram of the system of the invention is shown in fig. 2, and the client terminal covers credit systems (retail credit, public credit and the like), internal evaluation systems, credit cards and the like in a financial institution, and the client terminal systems interact with the computing processing system of the invention through an ajax request mode (GET, POST) and land processing result data.

The system functions related in the invention comprise main business modules such as business system management, individual/enterprise index library management, individual/enterprise index application, index application test, index result query, log management and the like.

After receiving the request from the financial institution business system, the computing processing system firstly checks whether the requester is registered in the business system management module, if not, the requester is intercepted, otherwise, the requester is released.

Then, according to the index item set or index application number stored in the request body, the computing processing system expands the calculation, calculates the credit report content according to the index business logic, lands the result in the Hbase database, and stores the calculated credit report file path, report number and calculation time in the database. Meanwhile, the computing processing system returns the result to the business system of the financial institution, and the financial institution can check the result through the index result query module.

To ensure efficient operation of the system, the entire request link needs to be logged, including the request log and the process log, which can be viewed in the log management module.

FIG. 3 is a diagram of new index addition, comprising:

1. The calculation rules are configured, the relevant calculation logic is configured in a mode of selecting from a front-end page, and the page has all credit report fields and sql function buttons.

2. The analysis and calculation rule generates an algorithm formula, and the configuration formula is analyzed into a corresponding rule code by logically judging through java codes according to the configuration formula (only supporting a fixed configuration format) transmitted from the front end.

3. And checking the correctness of the algorithm formula, and calling a built-in credit reporting file to test whether the result can be calculated or not by the generated rule code.

4. And adding rule code increment to the rule base, firstly finding out a corresponding rule base group according to the index name prefix, and then adding increment.

Fig. 4 is a diagram showing the detail of the index result.

And checking the details of the index calculation results, wherein all index results under the change credit report number are contained, and the index calculation results comprise an index Chinese name, an index English name (index number) and an index calculation result.

The related art related to the present invention is shown in the following table:

The specific application scene is as follows:

Scene one

And uploading the original file of the credit report to a file platform by the credit prepositive inquiry system, and storing the Hbase database through processing calculation after the index processing system acquires the file.

(1) The system automatically executes the task of acquiring the credit report file:

1.1, the system will perform timing tasks every month and month.

1.2, The timing task reads the index report numbers and the file storage paths calculated in all previous months of the database according to the date.

(2) System loading full index numbering

And 2.1, reading the cache to obtain all the existing index numbers.

And 2.2, inquiring the Hbase database according to the acquired credit report number to acquire the result of the existing index.

2.3, Filtering out the index number which is not calculated according to the existing index and the index calculated by Hbase.

(3) Performing credit index calculation

And transmitting the credit report file path and the index number set to be calculated to a corresponding method call rule engine to perform incremental calculation.

(4) And storing the index calculation result to be landed on the Hbase database.

Scene two

And calling an external interface of the index processing platform by other service systems, and returning an index result to the corresponding service system:

(1) External interface of calling system of other business system

And the other systems call the index interfaces to transmit the corresponding credit reporting paths and index numbers to be calculated to the index processing platform.

(2) System resolving request information

And 2.1, the system judges whether the request parameters meet the requirements.

2.2, Judging whether to calculate the full index or calculate the index transmitted by the user according to the transmission parameters.

2.3, If the user calculates the full index, all index numbers in the cache are obtained.

(3) Performing credit index calculation

And transmitting the credit report file path and the index number set to be calculated to a corresponding method call rule engine to perform incremental calculation.

(4) And storing the index calculation result, landing the index calculation result on the Hbase database, and returning the result to the corresponding service system.

The protection of the present invention is not limited to the above embodiments. Variations and advantages that would occur to one skilled in the art are included within the invention without departing from the spirit and scope of the inventive concept, and the scope of the invention is defined by the appended claims.

Claims (10)

1. A data index result calculation processing method, the method comprising:

Firstly, constructing an index information database in advance, and reading index information and loading index information from the index information database;

Caching index names in the index information through redis;

thirdly, forming a custom rule base from the index information according to the custom rule;

step four, multithreading uses the credit investigation data obtained, call the index calculation method from the custom rule base to calculate the index value;

and step five, storing the index result obtained by calculation into an Hbase database.

2. The computing method of claim 1, wherein in step one, the index information database stores index information including an initialized index number, an index name, index calculation logic, and whether an index is invalid;

when index information is loaded, judging whether index loading is invalid according to the index to be loaded; and if the index to be loaded is not invalid, loading the index to be loaded into the memory, and if the index to be loaded is invalid, not loading the index to be loaded into the memory.

3. The computing method of claim 1, wherein in the second step, index names in the index information of the first step are cached as a set in Redis; in Redis, different commands, including SET, GET, DEL, EXISTS, are used to store, retrieve, delete, and check key-value pairs.

4. The computing method of claim 1, wherein in step three, one or more rule bases KieBase are created according to the size of the rule base or the index name group configured by user, and stored in maps, and Key values of each KieBase rule base correspond to one type of index name prefix; the value in the Map corresponds to KieBase rule base of the current class, kieBase rule base is obtained according to key, and then related rules are called according to index English names; kieBase is a knowledge repository containing a number of rules, flows, methods; the Map is a key value pair mapping container, and maps keys to values; the index name prefix refers to a preset prefix for distinguishing each type of index.

5. The computing method of claim 1, wherein in the fourth step, when the index value is computed, it is determined that the existing full-quantity index or the index requested by the user is computed according to the user request parameter; the judgment standard is whether the request parameters of the user contain the index calculation name set, and if the request parameters of the user contain the index calculation name set, the index requested by the user is calculated.

6. The computing method of claim 5, wherein before the index value is computed, the index computing rule in the index information is invoked by searching for a corresponding index in the custom rule base according to the index name prefix.

7. The method according to claim 1, wherein in the fifth step, the result obtained by calculation is stored in the Hbase database with the report number on the personal credit report as a primary key.

8. A computing processing system implementing the method of any of claims 1-7, the processing system comprising: the system comprises a verification module, an index processing module and a result storage module;

the verification module is used for verifying the calculation request according to the verification rule, releasing the calculation request passing the verification, and intercepting the request which cannot pass the verification;

the index processing module is used for calculating an index value according to the calculation request;

The result storage module is used for storing the index result obtained by calculation, the calculated credit reporting file path, the reporting number and the calculation time into the Hbase database.

9. The computing system of claim 8, wherein the index processing module further comprises a relational database module, a redis cache module, a rule engine module, and an Hbase database module;

The relational database module is used for storing a full index initialization calculation rule; the redis cache module is used for storing the full index rule names; the rule engine module is used for calculating index results; the Hbase database module is used for storing calculation results.

10. Use of the computing process method according to any one of claims 1-7 or the computing process system according to claim 8 or 9 in credit report data index computing processes.