CN111090708B - Method and system for generating user characteristics based on data warehouse - Google Patents

Abstract

The present disclosure provides a data warehouse-based user feature yield method, comprising: acquiring transaction detail data of a plurality of users and dividing the transaction detail data into a plurality of subsets according to different dimensions; aggregating data in the first dimension subset per user; acquiring a second dimension for data screening; correlating the data in the second subset of dimensions to the data in the aggregated first subset of dimensions to generate data to be screened; screening the data to be screened to obtain user characteristics; and outputting at least one of the user characteristics, the data in the first subset of dimensions, and the data in the second subset of dimensions.

Description

Method and system for generating user characteristics based on data warehouse

technical field

The present disclosure generally relates to data processing, and more particularly to data processing based on data aggregation.

Background technique

In the era of big data, big data correlation and traceability technology is usually used to deeply mine massive transaction data, and find abnormal transaction networks based on transaction network structure discovery and abnormal transaction discovery.

In the field of big data, due to the huge and complicated volume of transaction data, grouping operations and data aggregation are often used to obtain results in the data mining process. Such a processing process is relatively simple and fast, but it also brings its limitations, that is, when outputting the final result of a certain feature, there is often no intermediate evidence information, so it is necessary to manually collect evidence or evidence later, which is a heavy workload and extremely expensive. May be missed.

In this case, operators in the big data field have a strong demand for more powerful tools to provide intermediate evidence information when quickly outputting the final result of a certain feature, thereby improving the efficiency of data processing in the big data field and targeting at sex.

Contents of the invention

In order to solve the above technical problems, the present disclosure provides a data warehouse-based user feature output solution. The scheme can provide the necessary intermediate evidence information when outputting the final result of user characteristics quickly, thereby improving the efficiency and pertinence of data processing.

In an embodiment of the present disclosure, a method for generating user characteristics based on a data warehouse is provided, including: obtaining transaction detail data of multiple users and splitting the transaction detail data into multiple subsets according to different dimensions; Aggregating data in the subset of the first dimension; obtaining a second dimension for data filtering; associating the data in the subset of the second dimension with data in the aggregated subset of the first dimension to generate data to be filtered; filtering the data to be filtered obtaining user features; and outputting at least one of the user features, data in the first dimension subset, and data in the second dimension subset.

In another embodiment of the present disclosure, the plurality of subsets each include key-value data pairs.

In yet another embodiment of the present disclosure, the data warehouse is based on Spark, Hadoop, MapReduce, Hive or SQL.

In another embodiment of the present disclosure, the operation of slicing or dicing may be used to split the transaction detail data into multiple subsets according to different dimensions.

In another embodiment of the present disclosure, outputting at least one of user characteristics, data in the first dimension subset, and data in the second dimension subset includes: outputting user characteristics, data in the first dimension subset Data or data in a subset of the second dimension.

In yet another embodiment of the present disclosure, outputting at least one of user characteristics, data in the first dimension subset, and data in the second dimension subset includes: outputting user characteristics, data in the first dimension subset when evidence is required, Key data and key data in the second dimension subset.

In an embodiment of the present disclosure, a data warehouse-based user feature output system is provided, including: an acquisition module that acquires transaction detail data of multiple users and splits the transaction detail data into multiple sub-components according to different dimensions. set, and obtain the second dimension used for data screening; the aggregation and association module aggregates the data in the first dimension subset by user, and associates the data in the second dimension subset with the aggregated first dimension subset to generate data to be screened; a screening module to screen the data to be screened to obtain user features; and an output module to output the user features, the data in the first dimension subset and the data in the second dimension subset at least one of the data.

In another embodiment of the present disclosure, the plurality of subsets each include key-value data pairs.

In yet another embodiment of the present disclosure, the data warehouse is based on Spark, Hadoop, MapReduce, Hive or SQL.

In another embodiment of the present disclosure, the acquisition module may divide the transaction detail data into multiple subsets according to different dimensions by slicing or dicing operations.

In another embodiment of the present disclosure, the output module outputting at least one of the user characteristics, the data in the first dimension subset and the data in the second dimension subset includes: when evidence is required, the output module outputs the user characteristics, the first dimension subset Data in a dimension subset or data in a second dimension subset.

In yet another embodiment of the present disclosure, the output module outputting at least one of the user characteristics, the data in the first dimension subset, and the data in the second dimension subset includes: when evidence is required, the output module outputs the user characteristics, the first dimension subset The key data in the dimension subset and the key data in the second dimension subset.

In an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, and when executed, the instructions cause a machine to perform the aforementioned method.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Description of drawings

The above summary of the invention and the following detailed description of the present disclosure will be better understood when read in conjunction with the accompanying drawings. It should be noted that the drawings are merely examples of the claimed invention. In the drawings, the same reference numerals represent the same or similar elements.

Fig. 1 shows a flow chart of a spark-based user feature production method according to an embodiment of the present disclosure;

Fig. 2 shows a schematic diagram of a spark-based user feature production method according to an embodiment of the present disclosure;

FIG. 3 shows a flow chart of a method for generating user characteristics based on a data warehouse according to an embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of an application scenario of a data warehouse-based user characteristic production method according to an embodiment of the present disclosure;

Fig. 5 shows a block diagram of a data warehouse-based user feature production system according to an embodiment of the present disclosure.

Detailed ways

In order to make the above objects, features and advantages of the present disclosure more comprehensible, specific implementations of the present disclosure will be described in detail below in conjunction with the accompanying drawings.

Many specific details are set forth in the following description to facilitate a full understanding of the present disclosure, but the present disclosure can also be implemented in other ways than those described here, so the present disclosure is not limited by the specific embodiments disclosed below.

In the era of big data, massive data needs to be processed and analyzed based on effective storage. The currently used Hadoop (HDFS) distributed file system can effectively store very large data sets. On the basis of effective data storage, the statistics and analysis of data is essentially the calculation of data. Common computing tools in the field of big data include MapReduce, Spark, etc. Effective management of big data can use data query in Hadoop mode. The data warehouse provides the basis for data mining, and establishes models through classification, prediction, and correlation analysis for pattern recognition and machine learning, thereby building an expert system.

As a distributed file system (HDFS), Hadoop can store massive data, and MapReduce is responsible for data calculation, thus forming a distributed framework for processing massive data. HDFS enables data to span different machines and devices, and uses paths to manage data on different platforms.

As a computing engine, MapReduce is responsible for allocating different data resources when processing different types of data, and dealing with data exchange and communication issues between different devices. And Spark is the second generation. Classification is required when counting data. The finer the classification and the more people involved in the statistics, the shorter the calculation time. In big data computing, hundreds of machines read various parts of the target file at the same time, and then calculate statistics for each part. Therefore, the calculation process of data is classified and summarized on the basis of HDFS.

Spark is a big data processing cluster based on data computing, in which data exchange and disk reading and writing are very convenient, so the data processing capacity is further improved.

As a data warehouse tool, Hive is based on HDFS. It maps the structured data in HDFS to tables in the database, so that the calculation results of MapReduce can be queried only through SQL statements, and files can also be processed through SQL. The data in the system is modified to improve work efficiency.

In the following description of the present disclosure, Spark will be taken as an example to describe the method and system for generating user characteristics based on a data warehouse. Those skilled in the art can understand that the technical solution of the present disclosure is not limited to be applied to Spark, in fact, the technical solution of the present disclosure is applicable to any data warehouse. More advanced data warehouse technologies can also be included in the technical solutions of the present disclosure, and will not be repeated here.

The method and system for generating user characteristics based on data warehouses according to various embodiments of the present disclosure will be described in detail below based on the accompanying drawings.

In a data warehouse, as mentioned above, splitting or grouping data sets and applying different functions to each subset or grouping is an important part of data analysis. The data in a dataset is split into subsets or groupings according to one or more selected keys. Split operations are performed on a specific dimension or axis of a data object. For example, a DataFrame can be split on its rows (axis=0) or columns (axis=1). Then, a function is applied to each subset or grouping and produces new values. Finally, the execution results of all these functions are aggregated or combined into the final feature result.

In such aggregation calculation process, usually the output result is only the final feature and does not contain intermediate data. In a big data scenario, when it is determined or judged that the characteristics of the end user are "suspicious transaction users", it is often necessary to provide evidence information related to the characteristics of the user. It is often inefficient to look for evidence information after the calculation is completed. Therefore, it is required in this field to output evidence information as required while outputting the characteristics of the end user.

FIG. 1 is a flowchart of a method 100 for generating user characteristics based on Spark according to an embodiment of the present disclosure. Fig. 2 shows a schematic diagram of a Spark-based user feature generation method according to an embodiment of the present disclosure. A Spark-based user feature generation method according to an embodiment of the present disclosure will be described with reference to FIG. 1 and FIG. 2 .

In an embodiment of the present disclosure, the Spark-based user feature production method 100 intends to produce the target user feature is the number of times the customer's transaction location is in Chengdu.

At 102, transaction detail data of multiple users is acquired.

Get the transaction detail data of multiple users from the data store. The data storage may be a data warehouse based on Spark, Hadoop, MapReduce, Hive, or SQL, etc.

The transaction detail data is used as a data set, which contains data of multiple dimensions. As shown in Figure 2, the subsets split by different dimensions will include different data. For example, subset 1 has multiple fields centered on customer ID and transaction UUID, including customer ID, transaction UUID, transaction amount, payment IP, and transaction time. Subset 2 has two fields with payment IP as the core, including payment IP and transaction location.

For example, in subset 1, it can be seen that the user whose customer ID is A111 has two transactions (transaction UUIDs are T100 and T101, and payment IPs are 192.1.2.3 and 192.1.2.4 respectively. In subset 1 It can also be seen from the figure that the user whose customer ID is A112 also has two transactions (the transaction UUIDs are T102 and T103, and the payment IPs are 192.1.2.5 and 192.1.2.6 respectively.

It can be seen in subset 2 that the transaction locations of the two transactions whose payment IPs are 192.1.2.3 and 192.1.2.4 respectively are in Chengdu. The transaction location of the payment IP 192.1.2.5 is also in Chengdu, while the transaction location of the payment IP 192.1.2.6 is Hangzhou.

At 104, data is aggregated by user dimension in Spark, and the data includes transaction UUID.

When aggregating data, the data of a certain dimension is often used as the key-value (key) to construct a key-value pair (key:value).

When aggregating the data in subset 1, the user ID can be used as the key value to construct a set of value values (transaction UUID, payment IP, ...). In the present embodiment shown in FIG. 2, for user A111, transactions (transaction UUID: T100, payment IP: 192.1.2.3) and (transaction UUID: T101, payment IP 192.1.2.4) are aggregated under his name. For user A112, aggregate transactions under his name (transaction UUID: T102, payment IP: 192.1.2.5) and (transaction UUID: T103, payment IP 192.1.2.6).

At 106, the data of another dimension is correlated.

Since the target user feature to be produced is the payment times of the user's transaction place in Chengdu, only aggregating subset 1 according to the user dimension will not be able to produce the target user feature. In this case, another dimension of data will need to be associated, namely the address mapping table (ie the subset 2 consisting of payment IP and transaction location).

This other dimension that needs to be correlated depends on the target user characteristics to be produced. When the filtering condition is "the user's transaction location is in Chengdu", since the transaction detail data (subset 1) of the user dimension does not contain the transaction location data, in order to generate the characteristics of the target user, it is necessary to establish the association of the relevant subsets.

Based on the aggregated data of subset 1, associate subset 2 to form the data to be screened. For user A111, the transactions to be screened are (transaction UUID: T100, payment IP: 192.1.2.3, location: Chengdu) and (transaction UUID: T101, payment IP 192.1.2.4, location: Chengdu). For user A112, the transactions to be screened are (transaction UUID: T102, payment IP: 192.1.2.5, location: Chengdu) and (transaction UUID: T103, payment IP 192.1.2.6, location: Hangzhou).

At 108, the associated data is screened.

Based on the filtering condition "the user's transaction location is in Chengdu" to filter the above associated data, the filtered data is: For user A111, the filtered transaction is (transaction UUID: T100, payment IP: 192.1.2.3, location: Chengdu) and (transaction UUID: T101, payment IP 192.1.2.4, location: Chengdu). For user A112, the filtered transaction is (transaction UUID: T102, payment IP: 192.1.2.5, location: Chengdu).

At 110, the user characteristics and transaction UUID are output.

Based on the transaction data screened out at 108, calculate the number of times the user's transaction location is in Chengdu, that is: for user A111, there are 2 transactions in Chengdu; for user A112, there is 1 transaction in Chengdu.

In the big data scenario, the transaction UUID can be output as evidence, namely:

For user A111, there are 2 transactions in Chengdu, and the transaction UUIDs are T100 and T101; for user A112, there is 1 transaction in Chengdu, and the transaction UUID is T102.

Of course, those skilled in the art can understand that for different big data scenarios, different relevant transaction details can be output as evidence, and multiple relevant detailed data can be output as evidence as required.

In the field of big data, abnormal transfer structure discovery can be carried out by discovering different transaction structures. Connectivity is the most ubiquitous feature of today's networks and systems, and connections are not evenly distributed or static, from social networks to transactional networks. Studying connected relations and their dynamic properties will fully describe and even predict behaviors in connected systems.

The different transaction structures found in the discovery of abnormal transfer structures include chain transaction structures, nested ring transaction structures, centralized transfer-in decentralized transfer-out transaction structures, decentralized transfer-in centralized transfer-out transaction structures, etc. For such an abnormal transaction structure, abnormal user characteristics can be excavated by using the data warehouse-based user characteristic output scheme of the present disclosure, and corresponding evidence can be provided.

FIG. 3 is a flowchart of a method 300 for generating user characteristics based on a data warehouse according to an embodiment of the present disclosure. Fig. 4 is a schematic diagram of an application scenario of a method for generating user characteristics based on a data warehouse according to an embodiment of the present disclosure. A method for generating user characteristics based on a data warehouse according to an embodiment of the present disclosure will be described with reference to FIGS. 3 and 4 .

In the data warehouse, multi-dimensional analysis can perform various analysis operations such as roll-up, drill-down, slicing, dicing, and rotation on data organized in multi-dimensional forms, so as to analyze data and output characteristics, so that analysts, managers and Decision makers can observe the data in the database from multiple angles and sides, so as to gain an in-depth understanding of the information and content contained in the data. This multidimensional analysis fits the human mindset, reducing confusion and reducing the possibility of misinterpretation.

At 302, the transaction detailed data of multiple users is acquired and the transaction detailed data is split into multiple subsets according to different dimensions.

For the multi-dimensional data set stored in the data warehouse, it is necessary to split the data set before performing multi-dimensional analysis on it. It can be split according to one dimension, that is, slice (slice), and the result of slice is usually two-dimensional plane data. ; It can also be split according to several dimensions, that is, dice, and the result of dice is usually a sub-cube of the data.

In an embodiment of the present disclosure, the transaction detail data of multiple users is split into subset 1 according to the user dimension, that is, multiple fields including customer ID, transaction UUID, transaction amount, payment IP and transaction time; according to the payment IP dimension Split into subset 2, including payment IP and transaction location.

At 304, data in the first dimension subset is aggregated by user.

When aggregating the data in subset 1, the user ID can be used as the key value to construct a set of value values (transaction UUID, payment IP, ...). For example, for user A111, aggregate transactions under his name (transaction UUID: T100, payment IP: 192.1.2.3) and (transaction UUID: T101, payment IP 192.1.2.4).

In a data warehouse, the dimensions of data are hierarchical. For example, the time dimension may be composed of year, month, and day, and the level of the dimension actually reflects the comprehensiveness of the data. The higher the level of the dimension, the higher the comprehensiveness of the data represented, the fewer the details, and the smaller the amount of data; the lower the level of the dimension, the lower the comprehensiveness of the data represented, the more sufficient the details, and the greater the amount of data. Data aggregation may be referred to as roll-up, to view more generalized data by going up in the dimension hierarchy or by eliminating a dimension or dimensions.

As an inverse operation of data aggregation, drill-down is also called data drill-down, which observes the data in more detail by descending the dimension level or by introducing one or some dimensions. In addition, data from different perspectives can be obtained through data rotation (pivot). The data rotation operation is equivalent to rotating the coordinate axis based on the plane data. For example, a rotation might involve swapping rows and columns, or rotating one dimension into another.

Those skilled in the art can understand that the method for generating user characteristics based on the data warehouse in the present disclosure should not be limited to data aggregation operations, but can also be extended to other analysis operations, which will not be repeated here.

At 306, a second dimension for data filtering is acquired.

Since the data set is split into multiple subsets, the data will inevitably be scattered into multiple dimensions. Therefore, when performing data filtering, it is necessary to obtain one or more other relevant dimensions for data filtering or filter conditions.

In an embodiment of the present disclosure, if the filter condition is "the user's transaction location is in Chengdu", then the dimensions related to the filter condition need to include the transaction location data. Thus, the obtained second dimension is the address mapping table (that is, the subset 2 formed by the payment IP and the transaction location).

At 308, the data in the second dimension subset is associated with the aggregated data in the first dimension subset to generate data to be filtered.

For the obtained second dimension, the data in the second dimension subset is associated with the aggregated data in the first dimension subset, that is, a data set that can be filtered is generated. In this embodiment, the data to be screened is: for user A111, the transaction to be screened is (transaction UUID: T100, payment IP: 192.1.2.3, location: Chengdu) and (transaction UUID: T101, payment IP 192.1.2.4, Location: Chengdu). For user A112, the transactions to be screened are (transaction UUID: T102, payment IP: 192.1.2.5, location: Chengdu) and (transaction UUID: T103, payment IP 192.1.2.6, location: Hangzhou).

At 310, the data to be screened is screened to obtain user characteristics.

In this embodiment, the filtered data is: for user A111, the filtered transaction is (transaction UUID: T100, payment IP: 192.1.2.3, location: Chengdu) and (transaction UUID: T101, payment IP 192.1.2.4 , Location: Chengdu). For user A112, the filtered transaction is (transaction UUID: T102, payment IP: 192.1.2.5, location: Chengdu).

Based on the transaction data screened out at 108, calculate the number of times the user's transaction location is in Chengdu, that is: for user A111, there are 2 transactions in Chengdu; for user A112, there is 1 transaction in Chengdu.

At 312, at least one of the user characteristics, the data in the first dimension subset, and the data in the second dimension subset is output.

In big data scenarios, in addition to outputting user characteristics, relevant evidence also needs to be output.

Therefore, in this embodiment, the data that can be output is: for user A111, there are 2 transactions with the transaction location in Chengdu, and the transaction UUIDs are T100 and T101; for user A112, there is 1 transaction with the transaction location in Chengdu, The transaction UUID is T102.

In scenarios where data is stored in the form of key-value pairs, relevant evidence can be output in the form of key-value in the corresponding dimension.

Those skilled in the art can understand that in different scenarios, user characteristics and evidence data can be output as required. That is, only user characteristics can be output; user characteristics and necessary evidence can be output; when important affairs are involved, user characteristics, necessary evidence and auxiliary evidence can be output.

The user characteristic output method based on the data warehouse of the present disclosure can provide necessary intermediate evidence information together when quickly outputting the final result of the user characteristic, thereby improving the efficiency and pertinence of data processing. In this method, feature-related evidence is produced while user features are produced, and there is no need to manually retrieve data when reporting, which greatly improves the efficiency and integrity of evidence.

FIG. 5 shows a block diagram of a data warehouse-based user feature production system 500 according to an embodiment of the present disclosure.

The user feature production system 500 based on the data warehouse includes an acquisition module 502 , a screening module 504 , an aggregation and correlation module 506 and an output module 508 .

The obtaining module 502 obtains transaction detailed data of multiple users and splits the transaction detailed data into multiple subsets according to different dimensions.

For the multi-dimensional data set stored in the data warehouse, it is necessary to split the data set before performing multi-dimensional analysis on it. It can be split according to one dimension, that is, slice (slice), and the result of slice is usually two-dimensional plane data. ; It can also be split according to several dimensions, that is, dice, and the result of dice is usually a sub-cube of the data.

In an embodiment of the present disclosure, the transaction detail data of multiple users is split into subset 1 according to the user dimension, that is, multiple fields including customer ID, transaction UUID, transaction amount, payment IP and transaction time; according to the payment IP dimension Split into subset 2, including payment IP and transaction location

Further, the obtaining module 502 obtains the second dimension used for data filtering. Since the data set is split into multiple subsets, the data will inevitably be scattered into multiple dimensions. Therefore, when performing data filtering, it is necessary to obtain one or more other relevant dimensions for data filtering or filter conditions.

In an embodiment of the present disclosure, if the filter condition is "the user's transaction location is in Chengdu", then the dimensions related to the filter condition need to include the transaction location data. Thus, the obtained second dimension is the address mapping table (that is, the subset 2 formed by the payment IP and the transaction location).

The aggregation and association module 506 aggregates the data in the first dimension subset by user, and associates the data in the second dimension subset with the aggregated data in the first dimension subset to generate data to be filtered.

In a data warehouse, the dimension of data is hierarchical, and the level of dimension actually reflects the comprehensive degree of data. The higher the level of the dimension, the higher the comprehensiveness of the data represented, the fewer the details, and the smaller the amount of data; the lower the level of the dimension, the lower the comprehensiveness of the data represented, the more sufficient the details, and the greater the amount of data. Data aggregation looks at more generalized data by going up in the dimensional hierarchy or by eliminating one or more dimensions.

When aggregating the data in subset 1, the aggregation and association module 506 uses the user ID as the key value to construct a set of value values (transaction UUID, payment IP, . . . ). For example, for user A111, the transactions aggregated under his name are (transaction UUID: T100, payment IP: 192.1.2.3) and (transaction UUID: T101, payment IP 192.1.2.4).

For the obtained second dimension, the aggregation and association module 506 associates the data in the second dimension subset with the aggregated data in the first dimension subset, that is, generates a data set that can be filtered. In this embodiment, the data to be screened is: for user A111, the transaction to be screened is (transaction UUID: T100, payment IP: 192.1.2.3, location: Chengdu) and (transaction UUID: T101, payment IP 192.1.2.4, Location: Chengdu). For user A112, the transactions to be screened are (transaction UUID: T102, payment IP: 192.1.2.5, location: Chengdu) and (transaction UUID: T103, payment IP 192.1.2.6, location: Hangzhou).

The screening module 504 screens the data to be screened to obtain user characteristics.

In this embodiment, the data screened by the screening module 504 is: for user A111, the screened transactions are (transaction UUID: T100, payment IP: 192.1.2.3, location: Chengdu) and (transaction UUID: T101, payment IP 192.1.2.4, location: Chengdu). For user A112, the filtered transaction is (transaction UUID: T102, payment IP: 192.1.2.5, location: Chengdu).

For the transaction data screened out at 108, the screening module 504 calculates the payment times of the user's transaction location in Chengdu, that is: for user A111, there are 2 transactions with the transaction location in Chengdu; for user A112, there are 1 transactions with the transaction location in Chengdu. Second-rate.

The output module 508 outputs at least one of user characteristics, data in the first dimension subset, and data in the second dimension subset.

Therefore, in this embodiment, the data that can be output is: for user A111, there are 2 transactions with the transaction location in Chengdu, and the transaction UUIDs are T100 and T101; for user A112, there is 1 transaction with the transaction location in Chengdu, The transaction UUID is T102.

In scenarios where data is stored in the form of key-value pairs, relevant evidence can be output in the form of key-value in the corresponding dimension.

The user feature output system based on the data warehouse of the present disclosure can provide necessary intermediate evidence information together with the final result of the user feature output quickly, thereby improving the efficiency and pertinence of data processing. In this system, the feature-related evidence is produced while the user features are produced, and there is no need to manually retrieve the data when reporting, which improves the efficiency and integrity of the evidence.

Each step and module of the data warehouse-based user characteristic production method and system described above can be realized by hardware, software, or a combination thereof. If implemented in hardware, the various illustrative steps, modules, and circuits described in connection with the present invention can be implemented with a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic components, hardware components, or any combination thereof. A general-purpose processor may be a processor, microprocessor, controller, microcontroller, or state machine, among others. If implemented in software, the various illustrative steps, modules described in connection with the invention may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. The software modules implementing the various operations of the present invention may reside in storage media such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, cloud storage, and the like. The storage medium can be coupled to the processor so that the processor can read and write information from/to the storage medium, and execute corresponding program modules to realize various steps of the present invention. Furthermore, software-based embodiments may be uploaded, downloaded or accessed remotely through appropriate communication means. Such suitable means of communication include, for example, the Internet, the World Wide Web, an intranet, software applications, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such means of communication.

It should also be noted that the embodiments may be described as processes depicted as flowcharts, flow diagrams, structural diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many of these operations can be performed in parallel or concurrently. Additionally, the order of these operations may be rearranged.

The disclosed methods, apparatus and systems should not be limited in any way. On the contrary, the invention covers all novel and nonobvious features and aspects of the various disclosed embodiments, both alone and in various combinations and subcombinations with each other. The disclosed methods, devices and systems are not limited to any specific aspect or feature or combination thereof, nor do any disclosed embodiments require that any one or more specific advantages be present or that specific or all technical problems be solved.

Embodiments of the present invention have been described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementations, and the above-mentioned specific implementations are only illustrative, rather than restrictive, and those of ordinary skill in the art will Under the enlightenment of the present invention, without departing from the gist of the present invention and the protection scope of the claims, many changes can also be made, and these all fall within the protection scope of the present invention.

Claims (13)

1. A data warehouse-based user feature yield method, comprising:

acquiring transaction detail data of a plurality of users and dividing the transaction detail data into a plurality of subsets according to different dimensions;

aggregating data in the first dimension subset per user;

acquiring a second dimension for data screening, wherein the second dimension subset corresponds to the same dimension and different dimension as the first dimension subset, the field value of the dimension which corresponds to the second dimension subset and is different from the first dimension is used for determining a final result, and the field value of the dimension which corresponds to the first dimension subset and is different from the second dimension is used as intermediate evidence information of the final result;

associating data in a second dimension subset to the aggregated data in the first dimension subset based on field values of the same dimensions corresponding to the second dimension subset as the first dimension subset to generate data to be screened;

screening the data to be screened based on field values of different dimensions corresponding to the second dimension subset and the first dimension subset to obtain user characteristics; and

outputting at least one of the user characteristics, the data in the first subset of dimensions, and the data in the second subset of dimensions.

2. The method of claim 1, wherein the plurality of subsets each comprise key-value data pairs.

3. The method of claim 1, wherein the data warehouse is based on Spark, hadoop, mapReduce, hive or SQL.

4. The method of claim 1, wherein splitting the transaction detail data into subsets in different dimensions may employ a slicing or dicing operation.

5. The method of claim 1, wherein outputting at least one of the user characteristic, the data in the first subset of dimensions, and the data in the second subset of dimensions comprises: outputting the user features, the data in the first subset of dimensions, or the data in the second subset of dimensions when evidence is required.

6. The method of claim 2, wherein outputting at least one of the user characteristic, the data in the first subset of dimensions, and the data in the second subset of dimensions comprises: and outputting the user characteristics, the key data in the first dimension subset and the key data in the second dimension subset when evidence is needed.

7. A data warehouse-based user feature yield system, comprising:

the system comprises an acquisition module, a data filtering module and a data filtering module, wherein the acquisition module acquires transaction detail data of a plurality of users, divides the transaction detail data into a plurality of subsets according to different dimensions, and acquires a second dimension for data filtering;

the aggregation and association module aggregates data in a first dimension subset according to users, associates the data in a second dimension subset with the aggregated data in the first dimension subset based on field values of the same dimension as the first dimension subset, which correspond to the same and different dimensions as the first dimension subset, wherein the field values of the different dimension as the first dimension are used for determining a final result, and the field values of the different dimension as the second dimension are used as intermediate evidence information of the final result;

the screening module is used for screening the data to be screened based on the field values of the dimensions, corresponding to the second dimension subset, which are different from the first dimension subset, so as to obtain user characteristics; and

and an output module outputting at least one of the user characteristics, the data in the first subset of dimensions, and the data in the second subset of dimensions.

8. The system of claim 7, wherein the plurality of subsets each comprise key-value data pairs.

9. The system of claim 7, the data warehouse is based on Spark, hadoop, mapReduce, hive or SQL.

10. The system of claim 7, wherein the obtaining module splits the transaction detail data into subsets in different dimensions using a slicing or dicing operation.

11. The system of claim 7, wherein the output module outputting at least one of the user characteristic, the data in the first subset of dimensions, and the data in the second subset of dimensions comprises: the output module outputs the user characteristic, the data in the first subset of dimensions, or the data in the second subset of dimensions when evidence is required.

12. The system of claim 7, wherein the output module outputting at least one of the user characteristic, the data in the first subset of dimensions, and the data in the second subset of dimensions comprises: when evidence is needed, the output module outputs the user feature, the key data in the first dimension subset, and the key data in the second dimension subset.

13. A computer readable storage medium storing instructions that, when executed, cause a machine to perform the method of any of claims 1-6.

Images