CN111444162A - Big data initialization method and device, electronic equipment and storage medium - Google Patents

Abstract

The present invention provides a big data initialization method, device, electronic device and storage medium. The method can map the historical data set imported into the distributed file system into a data table set, which improves the fault tolerance of the data, further performs missing value detection on the data in the data table set according to the historical data set, and obtains The standard data table set ensures the accuracy and integrity of the data, and then performs data analysis on the data in the standard data table set through the preset association degree dependency to obtain the key field set of the data, and the keyword The segment set is randomly distributed into the standard data table set, and a random landing data table set is generated, which improves the speed of data processing, and further combines the random landing data tables in the random landing data table set according to preset conditions to obtain initialization data. The table set realizes the initialization of big data.

Description

Big data initialization method, device, electronic device and storage medium

technical field

The present invention relates to the technical field of data processing, and in particular, to a big data initialization method, device, electronic device and storage medium.

Background technique

At present, big data has been applied to various application systems, and various application systems are also being upgraded and transformed according to the big data system. For example, for system architecture upgrade, it is only necessary to replace the data processing module with big data processing technology in terms of development technology, but for historical data, corresponding conversion processing is required.

When upgrading big data, it is usually necessary to perform system reconstruction, while historical data needs to be initialized according to the new rules to generate data that conforms to the new system rules. Simply stacking statements will cause serious performance problems and cannot be completed. initialization.

SUMMARY OF THE INVENTION

In view of the above, it is necessary to provide a big data initialization method, device, electronic device and storage medium, which can realize rapid initialization of big data, ensure the accuracy and integrity of the data, and improve the fault tolerance of the data.

A big data initialization method, the method includes:

Obtain historical datasets from pre-built databases, and import the historical datasets into a distributed file system;

mapping the imported historical data set into a data table set;

According to the historical data set, missing value detection is performed on the data in the data table set to obtain a standard data table set;

Perform data analysis on the data in the standard data table set by using a preset association degree dependency to obtain a key field set of the data;

Randomly distributing the key field set to the standard data table set to generate a random landing data table set;

The random landing data tables in the random landing data table set are combined according to preset conditions to obtain an initialization data table set.

According to a preferred embodiment of the present invention, the importing the historical data set into the distributed file system includes:

Obtain the IP address and SID number of the server where the database is located;

Log in to the database according to the IP address and the SID number;

Obtain the absolute path of the historical data set on the distributed file system from the database;

The historical dataset is imported into the distributed file system according to the absolute path.

According to a preferred embodiment of the present invention, the importing the historical data set into the distributed file system includes:

Obtain attribute information of the data in the historical data set;

Determine the priority of the data in the historical data set according to the attribute information;

The historical dataset is imported into a distributed file system according to the priority.

According to a preferred embodiment of the present invention, the mapping of the imported historical data set into a data table set includes:

Using a configuration tool to map the historical data set into a data table;

Check whether the historical data set is loaded into the data table according to the absolute path of the historical data set on the distributed file system;

When the historical data set is loaded into the data table, the data table set is constructed using the data in the data table.

According to a preferred embodiment of the present invention, performing missing value detection on the data in the data table set according to the historical data set to obtain a standard data table set includes:

Use missmap function to perform missing value detection on the data in the data table set;

When detecting that there is no missing value in the data table set, determining the data table set as the standard data table set; or

When it is detected that there are missing values in the data table set, a maximum likelihood estimation algorithm is used to fill in the missing values to obtain the standard data table set.

According to a preferred embodiment of the present invention, when using the maximum likelihood estimation algorithm to fill in the missing values to obtain the standard data table set, the following formula is used:

Wherein, L(θ) represents the filled missing value, θ represents the probability parameter corresponding to the missing value, n represents the number of the historical data set, and p( _xi |θ) represents the probability of the missing value.

According to a preferred embodiment of the present invention, randomly distributing the key field set to the standard data table set, and generating a random landing data table set includes:

determining the number of key fields in the key field set;

Generate a plurality of numerical values according to the number of the key fields, the number of the plurality of numerical values is the same as the number of the key fields;

establishing a mapping relationship between the plurality of numerical values and the standard data table set;

Randomly matching the plurality of numerical values and the key fields to obtain a matching result;

The key fields are distributed to the standard data table set according to the mapping relationship and the matching result to obtain the random landing data table set.

A big data initialization device, the device includes:

an importing unit, used to obtain a historical data set from a pre-built database, and import the historical data set into a distributed file system;

a mapping unit for mapping the imported historical data set into a data table set;

a detection unit, configured to perform missing value detection on the data in the data table set according to the historical data set to obtain a standard data table set;

an analysis unit, configured to perform data analysis on the data in the standard data table set through a preset association degree dependency relationship to obtain a key field set of the data;

a distribution unit, configured to randomly distribute the key field set to the standard data table set to generate a random landing data table set;

The merging unit is used for merging the random landing data tables in the random landing data table set according to preset conditions to obtain an initialization data table set.

According to a preferred embodiment of the present invention, the introduction unit is specifically used for:

Obtain the IP address and SID number of the server where the database is located;

Log in to the database according to the IP address and the SID number;

Obtain the absolute path of the historical data set on the distributed file system from the database;

The historical dataset is imported into the distributed file system according to the absolute path.

According to a preferred embodiment of the present invention, the device further comprises:

an acquiring unit, configured to acquire attribute information of data in the historical data set when the historical data set is imported into the distributed file system;

a determining unit, configured to determine the priority of the data in the historical data set according to the attribute information;

The importing unit is further configured to import the historical data set into the distributed file system according to the priority.

According to a preferred embodiment of the present invention, the mapping unit is specifically used for:

Using a configuration tool to map the historical data set into a data table;

Check whether the historical data set is loaded into the data table according to the absolute path of the historical data set on the distributed file system;

When the historical data set is loaded into the data table, the data table set is constructed using the data in the data table.

According to a preferred embodiment of the present invention, the detection unit is specifically used for:

Use missmap function to perform missing value detection on the data in the data table set;

When detecting that there is no missing value in the data table set, determining the data table set as the standard data table set; or

When it is detected that there are missing values in the data table set, a maximum likelihood estimation algorithm is used to fill in the missing values to obtain the standard data table set.

According to a preferred embodiment of the present invention, when the detection unit uses a maximum likelihood estimation algorithm to fill in the missing values and obtains the standard data table set, the following formula is used:

Wherein, L(θ) represents the filled missing value, θ represents the probability parameter corresponding to the missing value, n represents the number of the historical data set, and p( _xi |θ) represents the probability of the missing value.

According to a preferred embodiment of the present invention, the generating unit is specifically used for:

determining the number of key fields in the key field set;

Generate a plurality of numerical values according to the number of the key fields, the number of the plurality of numerical values is the same as the number of the key fields;

establishing a mapping relationship between the plurality of numerical values and the standard data table set;

Randomly matching the plurality of numerical values and the key fields to obtain a matching result;

The key fields are distributed to the standard data table set according to the mapping relationship and the matching result to obtain the random landing data table set.

An electronic device comprising:

a memory that stores at least one instruction; and

A processor executes the instructions stored in the memory to implement the big data initialization method.

A computer-readable storage medium, where at least one instruction is stored, the at least one instruction is executed by a processor in an electronic device to implement the big data initialization method.

It can be seen from the above technical solutions that the present invention can map the historical data set imported into the distributed file system into a data table set, thereby improving the fault tolerance of the data, and further according to the historical data set, the data table is centralized. The missing value detection is performed on the data of the standard data table, and the standard data table set is obtained, which ensures the accuracy and integrity of the data, and then the data in the standard data table set is analyzed through the preset correlation dependency relationship, and the key fields of the data are obtained. The key field set is randomly distributed into the standard data table set to generate a random landing data table set, which improves the speed of data processing. Further, the random landing data table in the random landing data table set is pre- Set the conditions to merge, get the initialization data table set, and realize the initialization of big data.

Description of drawings

FIG. 1 is a flow chart of a preferred embodiment of a method for initializing big data according to the present invention.

FIG. 2 is a functional block diagram of a preferred embodiment of the big data initialization device of the present invention.

FIG. 3 is a schematic structural diagram of an electronic device implementing a preferred embodiment of the method for initializing big data according to the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

As shown in FIG. 1 , it is a flow chart of a preferred embodiment of the big data initialization method of the present invention. According to different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

The big data initialization method is applied to one or more electronic devices, the electronic device is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions, and its hardware includes but not Limited to microprocessors, application specific integrated circuits (ASICs), programmable gate arrays (Field-Programmable Gate Arrays, FPGAs), digital processors (Digital SignalProcessors, DSPs), embedded devices, and the like.

The electronic device can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network TV ( Internet Protocol Television, IPTV), smart wearable devices, etc.

The electronic equipment may also include network equipment and/or user equipment. Wherein, the network device includes, but is not limited to, a single network server, a server group formed by multiple network servers, or a cloud formed by a large number of hosts or network servers based on cloud computing (Cloud Computing).

The network where the electronic device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (Virtual Private Network, VPN), and the like.

S10: Acquire a historical data set from a pre-built database, and import the historical data set into a distributed file system.

In at least one embodiment of the present invention, the pre-built database is a traditional database, also known as a relational database, for processing permanent and stable data.

For example, the database may be an Oracle database, a MySQL database, a graph database, or the like.

In at least one embodiment of the present invention, the historical data set is formed by combining data generated by historical user behavior.

In at least one embodiment of the present invention, the distributed file system may be a Hadoop Distributed File System (Hadoop Distributed File System, HDFS). Wherein, the distributed file system is a distributed file system deployed on a cluster, and data transmission needs to be performed through a network.

In at least one embodiment of the present invention, the electronic device importing the historical data set into the distributed file system includes:

The electronic device obtains the IP (Internet Protocol, Internet Protocol) address and SID (Security Identifiers, security identifier) number of the server where the database is located, and logs into the database according to the IP address and the SID number, The electronic device acquires the absolute path of the historical data set on the distributed file system from the database, and further imports the historical data set into the distributed file system according to the absolute path.

Specifically, the electronic device may import the historical data set from the database into the HDFS through the Sqoop transmission tool.

Among them, the Sqoop is an open source tool, which is mainly used for data transfer between Hadoop and traditional databases. The Sqoop can import data in a relational database into the HDFS cluster of Hadoop, or can Import HDFS data into a relational database.

In at least one embodiment of the present invention, the electronic device imports the historical data set into a distributed file system, including:

The electronic device acquires attribute information of the data in the historical data set, and determines the priority of the data in the historical data set according to the attribute information, and the electronic device imports the historical data set into the distribution system according to the priority. in the file system.

Wherein, the attribute information includes, but is not limited to, one or more of the following combinations:

The size of the table in the historical data set, whether there is a primary key, a field containing a time series or a number sequence, etc.

S11: Map the imported historical data set into a data table set.

In at least one embodiment of the present invention, mapping the imported historical data set into a data table set by the electronic device includes:

The electronic device uses a configuration tool to map the historical data set into a data table, and checks whether the historical data set is loaded into the data table according to the absolute path of the historical data set on the distributed file system , when the historical data set is loaded into the data table, the electronic device constructs the data table set by using the data in the data table.

Specifically, the configuration tool can be hive, the data table can include hive data table, the hive is a data warehouse tool based on Hadoop, can map structured data files into a database table, and provide simple SQL (Structured Query Language, Structured Query Language) query function.

Through the above embodiments, the data in the historical data set can be converted into a data table in a mapping manner, which improves the fault tolerance of the data.

S12, according to the historical data set, perform missing value detection on the data in the data table set to obtain a standard data table set.

It can be understood that data may be missing due to the developer's operation error and/or the failure of the import and transmission tool. Therefore, the electronic device performs missing value detection on the data in the data table set to obtain a standard data table set.

Specifically, the missing values include: completely random missing, random missing and non-random missing.

Among them, the completely random missing refers to the completely random missing value of a variable that does not depend on any other reasons; the said random missing refers to the missing of a variable that is related to other variables but not related to the value of the variable itself. Correlated missing; the non-random missing refers to the missing of a variable and the missing value of the variable itself.

In at least one embodiment of the present invention, the electronic device performs missing value detection on the data in the data table set according to the historical data set, and obtains a standard data table set including:

The electronic device uses the missmap function to perform missing value detection on the data in the data table set, specifically:

(1) When detecting that there are no missing values in the data table set, the electronic device determines the data table set as the standard data table set.

(2) When detecting that there are missing values in the data table set, the electronic device uses a maximum likelihood estimation algorithm to fill in the missing values to obtain the standard data table set.

Further, when the electronic device uses a maximum likelihood estimation algorithm to fill in the missing values and obtains the standard data table set, the following formula is used:

Wherein, L(θ) represents the filled missing value, θ represents the probability parameter corresponding to the missing value, n represents the number of the historical data set, and p( _xi |θ) represents the probability of the missing value.

Through the above-mentioned embodiments, the method for detecting missing values ensures the accuracy and integrity of the data.

S13: Perform data analysis on the data in the standard data table set by using a preset association degree dependency relationship to obtain a key field set of the data.

In at least one embodiment of the present invention, since a distributed system is different from a relational database, data analysis cannot be performed directly according to an index. In this case, the electronic device needs to analyze the standard data according to a preset correlation degree dependency Perform data analysis on the data in the table set to obtain the key field set that causes the data to be skewed.

Among them, the data skew refers to the large difference in the proportion of the number of occurrences of a field, for example, the number of boys in a school is 10,000, and the number of girls is 100.

Wherein, the preset association degree dependency includes: an unequal value key association rule and a joint detection rule of multiple data tables.

Further, the electronic device obtains the distribution of keys according to the correlation degree dependency, so as to obtain the key field set.

S14: Randomly distribute the key field set to the standard data table set to generate a random landing data table set.

In at least one embodiment of the present invention, the electronic device randomly distributes the key field set to the standard data table set, and generating a random landing data table set includes:

The electronic device determines the number of key fields in the key field set, and generates a plurality of numerical values according to the number of the key fields, and the number of the plurality of numerical values is the same as the number of the key fields, and the The electronic device establishes a mapping relationship between the plurality of numerical values and the standard data table set, and randomly matches the plurality of numerical values with the key fields to obtain a matching result, and the electronic device obtains a matching result according to the mapping relationship and the The matching result distributes the key fields to the standard data table set to obtain the random landing data table set.

Through the above-mentioned embodiments, the speed of data processing can be improved in combination with the random distribution of key fields.

S15: Combine the random landing data tables in the random landing data table set according to preset conditions to obtain an initialization data table set.

In at least one embodiment of the present invention, the preset conditions can be configured by developers according to different requirements of the random landing data table.

The preset conditions include, but are not limited to: merging of single tables, merging of multiple tables, merging of adjacent tables, and the like.

Through the above-mentioned embodiment, the random landing data table is merged according to the preset conditions, which can reduce the difficulty of data initialization, indirectly improve the data processing speed, and provide strong support for project switching.

It can be seen from the above technical solutions that the present invention can map the historical data set imported into the distributed file system into a data table set, thereby improving the fault tolerance of the data, and further according to the historical data set, the data table is centralized. The missing value detection is performed on the data of the standard data table, and the standard data table set is obtained, which ensures the accuracy and integrity of the data, and then the data in the standard data table set is analyzed through the preset correlation dependency relationship, and the key fields of the data are obtained. The key field set is randomly distributed into the standard data table set to generate a random landing data table set, which improves the speed of data processing. Further, the random landing data table in the random landing data table set is pre- Set the conditions to merge, get the initialization data table set, and realize the initialization of big data.

As shown in FIG. 2 , it is a functional block diagram of a preferred embodiment of the big data initialization apparatus of the present invention. The big data initialization device 11 includes an import unit 110 , a mapping unit 111 , a detection unit 112 , an analysis unit 113 , a distribution unit 114 , a merge unit 115 , an acquisition unit 116 and a determination unit 117 . The modules/units referred to in the present invention refer to a series of computer program segments that can be executed by the processor 13 and can perform fixed functions, and are stored in the memory 12 . In this embodiment, the functions of each module/unit will be described in detail in subsequent embodiments.

The importing unit 110 acquires historical data sets from a pre-built database, and imports the historical data sets into the distributed file system.

In at least one embodiment of the present invention, the pre-built database is a traditional database, also known as a relational database, for processing permanent and stable data.

For example, the database may be an Oracle database, a MySQL database, a graph database, or the like.

In at least one embodiment of the present invention, the historical data set is formed by combining data generated by historical user behavior.

In at least one embodiment of the present invention, the distributed file system may be a Hadoop Distributed File System (Hadoop Distributed File System, HDFS). Wherein, the distributed file system is a distributed file system deployed on a cluster, and data transmission needs to be performed through a network.

In at least one embodiment of the present invention, the importing unit 110 importing the historical data set into the distributed file system includes:

The importing unit 110 obtains the IP (Internet Protocol, Internet Protocol) address and SID (Security Identifiers, security identifier) number of the server where the database is located, and logs into the database according to the IP address and the SID number , the import unit 110 obtains the absolute path of the historical data set on the distributed file system from the database, and further imports the historical data set into the distributed file system according to the absolute path .

Specifically, the importing unit 110 may import the historical data set from the database into the HDFS through the Sqoop transmission tool.

Among them, the Sqoop is an open source tool, which is mainly used for data transfer between Hadoop and traditional databases. The Sqoop can import data in a relational database into the HDFS cluster of Hadoop, or can Import HDFS data into a relational database.

In at least one embodiment of the present invention, the importing unit 110 imports the historical data set into a distributed file system, including:

The acquiring unit 116 acquires attribute information of the data in the historical data set, the determining unit 117 determines the priority of the data in the historical data set according to the attribute information, and the importing unit 110 imports the historical data set according to the priority into a distributed file system.

Wherein, the attribute information includes, but is not limited to, one or more of the following combinations:

The size of the table in the historical data set, whether there is a primary key, a field containing a time series or a number sequence, etc.

The mapping unit 111 maps the imported historical data set into a data table set.

In at least one embodiment of the present invention, the mapping unit 111 maps the imported historical data set into a data table set including:

The mapping unit 111 uses a configuration tool to map the historical data set to a data table, and checks whether the historical data set is loaded into the data table according to the absolute path of the historical data set on the distributed file system , when the historical data set is loaded into the data table, the mapping unit 111 uses the data in the data table to construct the data table set.

Specifically, the configuration tool can be hive, the data table can include hive data table, the hive is a data warehouse tool based on Hadoop, can map structured data files into a database table, and provide simple SQL (Structured Query Language, Structured Query Language) query function.

Through the above embodiments, the data in the historical data set can be converted into a data table in a mapping manner, which improves the fault tolerance of the data.

The detection unit 112 performs missing value detection on the data in the data table set according to the historical data set to obtain a standard data table set.

It can be understood that data may be missing due to the developer's operation error and/or the failure of the import and transmission tool. Therefore, the detection unit 112 performs missing value detection on the data in the data table set to obtain a standard data table set. .

Specifically, the missing values include: completely random missing, random missing and non-random missing.

Among them, the completely random missing refers to the completely random missing value of a variable that does not depend on any other reasons; the said random missing refers to the missing of a variable that is related to other variables but not related to the value of the variable itself. Correlated missing; the non-random missing refers to the missing of a variable and the missing value of the variable itself.

In at least one embodiment of the present invention, the detection unit 112 performs missing value detection on the data in the data table set according to the historical data set, and obtains a standard data table set including:

The detection unit 112 uses the missmap function to perform missing value detection on the data in the data table set, specifically:

(1) When detecting that there is no missing value in the data table set, the detecting unit 112 determines the data table set as the standard data table set.

(2) When detecting that there are missing values in the data table set, the detection unit 112 uses a maximum likelihood estimation algorithm to fill in the missing values to obtain the standard data table set.

Further, when the detection unit 112 uses the maximum likelihood estimation algorithm to fill in the missing values to obtain the standard data table set, the following formula is used:

Wherein, L(θ) represents the filled missing value, θ represents the probability parameter corresponding to the missing value, n represents the number of the historical data set, and p( _xi |θ) represents the probability of the missing value.

Through the above-mentioned embodiments, the method for detecting missing values ensures the accuracy and integrity of the data.

The analyzing unit 113 performs data analysis on the data in the standard data table set according to the preset association degree dependency, and obtains a key field set of the data.

In at least one embodiment of the present invention, since the distributed system is different from the relational database, it is impossible to directly analyze the data according to the index. The data in the data table set is subjected to data analysis to obtain the key field set that causes the data to be skewed.

Among them, the data skew refers to the large difference in the proportion of the number of occurrences of a field, for example, the number of boys in a school is 10,000, and the number of girls is 100.

Wherein, the preset association degree dependency includes: an unequal value key association rule and a joint detection rule of multiple data tables.

Further, the analyzing unit 113 obtains the distribution of keys according to the correlation degree dependency, thereby obtaining the key field set.

The distribution unit 114 randomly distributes the key field set to the standard data table set to generate a random landing data table set.

In at least one embodiment of the present invention, the distribution unit 114 randomly distributes the key field set to the standard data table set, and generating a random landing data table set includes:

The distribution unit 114 determines the number of key fields in the key field set, and generates a plurality of numerical values according to the number of the key fields, and the number of the multiple numerical values is the same as the number of the key fields, so The distribution unit 114 establishes a mapping relationship between the multiple values and the standard data table set, and randomly matches the multiple values with the key fields to obtain a matching result. The distribution unit 114 determines the mapping relationship according to the mapping relationship. and the matching result distributes the key fields into the standard data table set to obtain the random landing data table set.

Through the above-mentioned embodiments, the speed of data processing can be improved in combination with the random distribution of key fields.

The merging unit 115 merges the random landing data tables in the random landing data table set according to preset conditions to obtain an initialization data table set.

In at least one embodiment of the present invention, the preset conditions can be configured by developers according to different requirements of the random landing data table.

The preset conditions include, but are not limited to: merging of single tables, merging of multiple tables, merging of adjacent tables, and the like.

Through the above-mentioned embodiment, the random landing data table is merged according to the preset conditions, which can reduce the difficulty of data initialization, indirectly improve the data processing speed, and provide strong support for project switching.

It can be seen from the above technical solutions that the present invention can map the historical data set imported into the distributed file system into a data table set, thereby improving the fault tolerance of the data, and further according to the historical data set, the data table is centralized. The missing value detection is performed on the data of the standard data table, and the standard data table set is obtained, which ensures the accuracy and integrity of the data, and then the data in the standard data table set is analyzed through the preset correlation dependency relationship, and the key fields of the data are obtained. The key field set is randomly distributed into the standard data table set to generate a random landing data table set, which improves the speed of data processing. Further, the random landing data table in the random landing data table set is pre- Set the conditions to merge, get the initialization data table set, and realize the initialization of big data.

As shown in FIG. 3 , it is a schematic structural diagram of an electronic device according to a preferred embodiment of the method for initializing big data according to the present invention.

The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a big data initialization program.

Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 1, and does not constitute a limitation on the electronic device 1. The electronic device 1 can be either a bus-type structure or a star-shaped structure. The device 1 may also include more or less other hardware or software than shown, or different component arrangements, for example, the electronic device 1 may also include input and output devices, network access devices, and the like.

It should be noted that the electronic device 1 is only an example. If other existing or future electronic products can be adapted to the present invention, they should also be included within the protection scope of the present invention, and are incorporated herein by reference. .

Wherein, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. . The memory 12 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1 . In other embodiments, the memory 12 may also be an external storage device of the electronic device 1, such as a pluggable mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) equipped on the electronic device 1 ) card, flash memory card (Flash Card) and so on. Further, the memory 12 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software installed in the electronic device 1 and various types of data, such as the code of a big data initialization program, etc., but also can be used to temporarily store data that has been output or will be output.

The processor 13 may be composed of integrated circuits in some embodiments, for example, may be composed of a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same function or different functions, including one or more central processing units. CPU (Central Processing Unit, CPU), microprocessor, digital processing chip, graphics processor and combination of various control chips, etc. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect the various components of the entire electronic device 1, by running or executing the program or module (for example, executing the program) stored in the memory 12. Big data initialization program, etc.), and call the data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

The processor 13 executes the operating system of the electronic device 1 and various installed application programs. The processor 13 executes the application program to implement the steps in each of the foregoing big data initialization method embodiments, such as steps S10 , S11 , S12 , S13 , S14 , and S15 shown in FIG. 1 .

Alternatively, when the processor 13 executes the computer program, the functions of the modules/units in the above device embodiments are implemented, for example:

Obtain historical datasets from pre-built databases, and import the historical datasets into a distributed file system;

mapping the imported historical data set into a data table set;

According to the historical data set, missing value detection is performed on the data in the data table set to obtain a standard data table set;

Perform data analysis on the data in the standard data table set by using a preset association degree dependency to obtain a key field set of the data;

Randomly distributing the key field set to the standard data table set to generate a random landing data table set;

The random landing data tables in the random landing data table set are combined according to preset conditions to obtain an initialization data table set.

Exemplarily, the computer program may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 12 and executed by the processor 13 to complete the present invention. invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 1 . For example, the computer program may be divided into an import unit 110 , a mapping unit 111 , a detection unit 112 , an analysis unit 113 , a distribution unit 114 , a merge unit 115 , an acquisition unit 116 , and a determination unit 117 .

The above-mentioned integrated units implemented in the form of software functional modules may be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium, and include several instructions to cause a computer device (which may be a personal computer, a computer device, or a network device, etc.) or a processor (processor) to execute the methods described in the various embodiments of the present invention. part.

If the modules/units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments, and can also be completed by instructing relevant hardware devices through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above method embodiments can be implemented.

Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory) .

The bus may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (extended industry standard architecture, EISA for short) bus, or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one arrow is shown in FIG. 3, but it does not mean that there is only one bus or one type of bus. The bus is arranged to enable connection communication between the memory 12 and at least one processor 13 and the like.

Although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to various components, preferably, the power source may be logically connected to the at least one processor 13 through a power management device, so as to be implemented by the power management device Charge management, discharge management, and power management functions. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device 1 may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

Further, the electronic device 1 may also include a network interface, optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used in the electronic device 1 Establish a communication connection with other electronic devices.

Optionally, the electronic device 1 may further include a user interface, and the user interface may be a display (Display), an input unit (eg, a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. The display may also be appropriately called a display screen or a display unit, which is used for displaying information processed in the electronic device 1 and for displaying a visualized user interface.

It should be understood that the embodiments are only used for illustration, and are not limited by this structure in the scope of the patent application.

FIG. 3 only shows the electronic device 1 with components 12-13. Those skilled in the art can understand that the structure shown in FIG. 3 does not constitute a limitation on the electronic device 1, and may include less than shown in the figure. Or more components, or a combination of certain components, or a different arrangement of components.

With reference to FIG. 1 , the memory 12 in the electronic device 1 stores multiple instructions to implement a big data initialization method, and the processor 13 can execute the multiple instructions to implement:

Obtain historical datasets from pre-built databases, and import the historical datasets into a distributed file system;

mapping the imported historical data set into a data table set;

According to the historical data set, missing value detection is performed on the data in the data table set to obtain a standard data table set;

Perform data analysis on the data in the standard data table set by using a preset association degree dependency to obtain a key field set of the data;

Randomly distributing the key field set to the standard data table set to generate a random landing data table set;

The random landing data tables in the random landing data table set are combined according to preset conditions to obtain an initialization data table set.

Specifically, for the specific implementation method of the above-mentioned instruction by the processor 13, reference may be made to the description of the relevant steps in the embodiment corresponding to FIG. 1 , which is not repeated here.

In the several embodiments provided by the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, but that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the invention.

Therefore, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the invention is to be defined by the appended claims rather than the foregoing description, which are therefore intended to fall within the scope of the claims. All changes within the meaning and range of the equivalents of , are included in the present invention. Any reference signs in the claims shall not be construed as limiting the involved claim.

Furthermore, it is clear that the word "comprising" does not exclude other units or steps and the singular does not exclude the plural. Several units or means recited in the system claims can also be realized by one unit or means by means of software or hardware. Second-class terms are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present invention.

Claims (10)

1. a big data initialization method, is characterized in that, described method comprises: Obtain historical datasets from pre-built databases, and import the historical datasets into a distributed file system; mapping the imported historical data set into a data table set; According to the historical data set, missing value detection is performed on the data in the data table set to obtain a standard data table set; Perform data analysis on the data in the standard data table set by using a preset association degree dependency to obtain a key field set of the data; Randomly distributing the key field set to the standard data table set to generate a random landing data table set; The random landing data tables in the random landing data table set are combined according to preset conditions to obtain an initialization data table set. 2. The big data initialization method according to claim 1, wherein the importing the historical data set into a distributed file system comprises: Obtain the IP address and SID number of the server where the database is located; Log in to the database according to the IP address and the SID number; Obtain the absolute path of the historical data set on the distributed file system from the database; The historical dataset is imported into the distributed file system according to the absolute path. 3. The big data initialization method according to claim 1, wherein the importing the historical data set into a distributed file system comprises: Obtain attribute information of the data in the historical data set; Determine the priority of the data in the historical data set according to the attribute information; The historical dataset is imported into a distributed file system according to the priority. 4. The big data initialization method according to claim 2, wherein the mapping of the imported historical data set into a data table set comprises: Using a configuration tool to map the historical data set into a data table; Check whether the historical data set is loaded into the data table according to the absolute path of the historical data set on the distributed file system; When the historical data set is loaded into the data table, the data table set is constructed using the data in the data table. 5. The big data initialization method according to claim 1, wherein, according to the historical data set, performing missing value detection on the data in the data table set, and obtaining a standard data table set comprises: Use missmap function to perform missing value detection on the data in the data table set; When detecting that there is no missing value in the data table set, determining the data table set as the standard data table set; or When it is detected that there are missing values in the data table set, a maximum likelihood estimation algorithm is used to fill in the missing values to obtain the standard data table set. 6. The big data initialization method according to claim 5, characterized in that, when using a maximum likelihood estimation algorithm to fill in the missing values to obtain the standard data table set, the following formula is used:

Wherein, L(θ) represents the filled missing value, θ represents the probability parameter corresponding to the missing value, n represents the number of the historical data set, and p( _xi |θ) represents the probability of the missing value. 7. The big data initialization method according to claim 1, wherein the random distribution of the key field set to the standard data table set, and the generation of a random landing data table set comprises: determining the number of key fields in the key field set; Generate a plurality of numerical values according to the number of the key fields, the number of the plurality of numerical values is the same as the number of the key fields; establishing a mapping relationship between the plurality of numerical values and the standard data table set; Randomly matching the plurality of numerical values and the key fields to obtain a matching result; The key fields are distributed to the standard data table set according to the mapping relationship and the matching result to obtain the random landing data table set. 8. A big data initialization device, wherein the device comprises: an importing unit, used to obtain a historical data set from a pre-built database, and import the historical data set into a distributed file system; a mapping unit for mapping the imported historical data set into a data table set; a detection unit, configured to perform missing value detection on the data in the data table set according to the historical data set to obtain a standard data table set; an analysis unit, configured to perform data analysis on the data in the standard data table set through a preset association degree dependency relationship to obtain a key field set of the data; a distribution unit, configured to randomly distribute the key field set to the standard data table set to generate a random landing data table set; The merging unit is used for merging the random landing data tables in the random landing data table set according to preset conditions to obtain an initialization data table set. 9. An electronic device, characterized in that the electronic device comprises: a memory that stores at least one instruction; and The processor executes the instructions stored in the memory to implement the big data initialization method according to any one of claims 1 to 7. 10. A computer-readable storage medium, characterized in that: the computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the method as claimed in claims 1 to 7 The big data initialization method described in any one of the items.

Images