CN112115164B - Data processing method and device, data query method and device and network equipment - Google Patents

Abstract

The invention provides a data processing method and device, a data query method and device and network equipment, wherein the data processing method comprises the following steps: collecting original data to be processed; carrying out structuring treatment on the original data according to a specified format to obtain structured data; carrying out aggregation treatment on the structured data to obtain aggregated data, and taking the aggregated data as data processed by original data; wherein the data amount of the aggregate data is smaller than the data amount of the original data. The invention can effectively reduce the data volume and solve the problem of huge data volume in the prior art.

Description

Data processing method and device, data query method and device, network equipment

Technical Field

The present invention relates to the field of data processing technology, and in particular to a data processing method and device, a data query method and device, and network equipment.

Background Art

In areas such as cloud services, huge amounts of data are usually involved. For example, cloud service providers will provide users with a variety of billing products (also known as resources), such as computing services, network services, and storage services, and record the amount of resources consumed by users in a preset time period in the form of data to obtain resource consumption data used as the basis for charging. The amount of resource consumption data recorded by cloud service providers is usually huge. For example, for a resource, if a cloud service provider records a resource consumption data at a frequency of once per second, 86,400 pieces of data will be recorded in a day. If a cloud service provider can provide 10,000 resources, the resource consumption data recorded in a day will be 800 million, not to mention that cloud service providers can usually provide tens of thousands, hundreds of thousands, or even millions of resources. The huge amount of data may bring many problems such as inconvenient storage and long query time.

Summary of the invention

In view of this, an object of the present invention is to provide a data processing method and device, a data query method and device, and a network device, which can effectively reduce the amount of data and improve the problem of huge data volume in the prior art.

In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

In a first aspect, an embodiment of the present invention provides a data processing method, comprising: collecting original data to be processed; performing structured processing on the original data according to a specified format to obtain structured data; performing aggregation processing on the structured data to obtain aggregated data, and using the aggregated data as data after processing the original data; wherein the data volume of the aggregated data is smaller than the data volume of the original data.

In some embodiments, the step of structuring the original data according to a specified format to obtain structured data includes: searching the original data for data corresponding to each field in a preset structured table, and filling the found data in the corresponding fields in the preset structured table; and using the filled preset structured table as structured data.

In some embodiments, the method further includes: if no data corresponding to the key field specified in the preset structured table is found in the original data, discarding the original data.

In some embodiments, the step of aggregating the structured data to obtain aggregated data includes: grouping the structured data to obtain at least one data group; wherein, multiple pieces of structured data in the same data group have the same data filled in corresponding preset fields; for each of the data groups, aggregating the multiple pieces of structured data in the data group to obtain aggregated data corresponding to the data group; wherein, the aggregated data is a piece of structured data selected from the multiple pieces of structured data in the data group, or, the aggregated data is a new piece of structured data formed by the multiple pieces of structured data in the data group according to a preset algorithm.

In some embodiments, the method further includes: storing the obtained aggregated data in a non-relational database; and deploying the non-relational database on a server cluster.

In some embodiments, the step of collecting the raw data to be processed includes: obtaining the raw data to be processed from a server where a billing product is deployed at a preset frequency; wherein the raw data is resource consumption data of the billing product recorded by the server; and the number of the servers is one or more.

In a second aspect, an embodiment of the present invention further provides a data query method, comprising: if a data query request is received, searching for aggregated data corresponding to the data query request from a preset database; wherein the database stores aggregated data obtained by the data processing method provided in any one of the embodiments of the first aspect; and feeding back the found aggregated data to the requester of the data query request.

In a third aspect, an embodiment of the present invention provides a data processing device, comprising: a data acquisition module, used to acquire original data to be processed; a data structuring module, used to perform structured processing on the original data according to a specified format to obtain structured data; a data aggregation module, used to perform aggregation processing on the structured data to obtain aggregated data, and use the aggregated data as data after processing the original data; wherein the data volume of the aggregated data is smaller than the data volume of the original data.

In a fourth aspect, an embodiment of the present invention provides a data query device, comprising: a data search module, for searching for aggregated data corresponding to a data query request from a preset database if a data query request is received; wherein the database stores aggregated data obtained by the data processing method provided in any one of the embodiments of the first aspect; and a data feedback module, for feeding back the found aggregated data to the requester of the data query request.

In a fifth aspect, an embodiment of the present invention provides a network device, comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the method provided by any embodiment of the first aspect, or the processor executes the machine-executable instructions to implement the method provided by the second aspect.

In a sixth aspect, an embodiment of the present invention provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the method provided by any one of the embodiments of the first aspect, or the machine-executable instructions cause the processor to implement the method provided by the second aspect.

The embodiment of the present invention provides a data processing method and device, which can perform structured processing on the collected raw data according to a specified format to obtain structured data, and then aggregate the structured data to obtain aggregated data, so as to use the aggregated data as the data after the raw data is processed. This embodiment adopts the method of performing structured processing and aggregating the raw data, which can effectively reduce the amount of raw data and improve the problem of huge data volume in the prior art.

An embodiment of the present invention provides a data query method and device, which can search for aggregated data corresponding to a received data query request from a preset database and feed back the aggregated data to the requester of the data query request. Since the database stores aggregated data obtained by the above-mentioned data processing method, the amount of data is smaller than the amount of original data directly stored in the database in the prior art. Therefore, this method can effectively shorten the data query time and improve the query efficiency.

Other features and advantages of the embodiments of the present invention will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned techniques of the embodiments of the present invention.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 shows a flow chart of a data processing method provided by an embodiment of the present invention;

FIG2 shows a specific flow chart of a data processing method provided by an embodiment of the present invention;

FIG3 shows a flow chart of a data query method provided by an embodiment of the present invention;

FIG4 shows a structural block diagram of a data processing device provided by an embodiment of the present invention;

FIG5 shows a structural block diagram of a data query device provided by an embodiment of the present invention;

FIG6 shows a schematic diagram of the structure of a network device provided by an embodiment of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

At present, fields such as cloud services usually involve huge amounts of data. Taking the charging model of cloud service providers as an example, a typical way for cloud service providers to charge users for services is to collect the resources consumed by users in a certain time period at a predetermined frequency, and use the peak value of resource consumption in the time period as the unit price standard in the time period. For example, a cloud service provider will collect the user's resource consumption data at a frequency of once every 5 seconds, obtain the user's resource consumption data set within an hour, and determine the user's unit price within this hour based on the highest value of the resource consumption data in the obtained resource consumption data set, thereby charging the user. Based on the above requirements, for each charged resource, the cloud service provider needs to frequently collect and store the user's resource consumption data. The amount of resource consumption data collected every day is huge, which greatly affects the storage space and query efficiency.

To improve the above problems, the embodiments of the present invention provide a data processing method and device, a data query method and device, and a network device. This technology can be applied to fields such as cloud services that easily involve large amounts of data. The embodiments of the present invention are described in detail below.

First, the embodiment of the present invention discloses a data processing method, which can be executed by a network device such as a server. Referring to the flowchart of the data processing method shown in FIG1 , the method mainly includes the following steps S102 to S106:

Step S102, collecting raw data to be processed.

The above raw data may be directly collected and unprocessed data, and the data volume is usually large, such as a large amount of resource consumption data directly collected by a cloud service provider through a server deployed with a billing product. In a specific implementation, the raw data to be processed may be obtained from a server deployed with a billing product at a preset frequency; the raw data is the resource consumption data of the billing product recorded by the server.

In actual applications, there are one or more servers. A billing product can be deployed on one server or on a server cluster. The server where the billing product is deployed can be equipped with a data collector for collecting raw data. Billing products can implement user-oriented services such as computing services, network services, and storage services. Specifically, they can be implemented using cloud servers, elastic IPs, storage blocks, etc. Cloud service providers usually have multiple billing products, and each billing product has a unique ID.

Compared with the prior art, most of the original data is directly stored in the database, resulting in a huge amount of data in the database. After collecting the original data, the present application will further process the collected original data in the following steps S104 and S106 to better reduce the amount of data.

Step S104, structuring the original data according to a specified format to obtain structured data.

Generally speaking, most raw data is unstructured data in the form of strings. The form, quantity, order, etc. of the fields of different raw data are mostly different, and the form is disorganized. In order to be able to process and store the raw data more conveniently, this embodiment converts the raw data into neatly ordered structured data, and the structured processing process can be offline processing.

Step S106, performing aggregation processing on the structured data to obtain aggregated data, and using the aggregated data as data after the original data is processed; wherein the data volume of the aggregated data is smaller than the data volume of the original data.

Specifically, the structured data can be aggregated according to the preset aggregation rules to obtain a smaller amount of aggregated data. For example, after 1000 pieces of structured data are aggregated, 1 piece of aggregated data is obtained, which can greatly reduce the amount of data. The specific aggregation rules can be flexibly set according to actual needs. Regardless of the aggregation rules, the amount of aggregated data finally obtained must be less than the amount of original data.

The above data processing method provided by the embodiment of the present invention can perform structured processing on the collected raw data according to the specified format to obtain structured data, and then aggregate the structured data to obtain aggregated data, so as to use the aggregated data as the data after the raw data is processed. This embodiment adopts the method of performing structured processing and aggregation on the raw data, which can effectively reduce the amount of raw data, improve the problem of large amount of data in the prior art, and thus also help alleviate the problems of storage inconvenience and low query efficiency caused by large amount of data.

This embodiment provides a specific implementation method of structured processing. The above step S104 can be implemented in the following manner: search for data corresponding to each field in the preset structured table from the original data, and fill the found data in the corresponding field in the preset structured table; and use the filled preset structured table as structured data. For ease of understanding, an exemplary description is as follows:

For example, a piece of original data is:

2019-01-29 07:31:39body:

{"status":"available","share_type_id":"473acf19-f54a-408a-83b0-f835519125f8","share_id":"2b151c02-eb6b-4962-a822-5d69e888aebc","usage_realtime":0,"quota_upperbound" :20971520,"az":"ksc_shp bstest_zone_raidssd_1001","size":20480,"user_id":"11e440e7c797415ca65421273fde3b2c"."share_type":"Capacity","tenant_id":"8ec373464e3f4a53b8ee6d32afc62c41","timestamp ":"2019-01-2907:31:39"}

From the above, we can see that the original data is unstructured data in the form of strings.

The preset structured table is shown in Table 1:

Table 1

As shown in Table 1, storage_type (storage type), user_id (user ID), availability_zone (available area, also known as computer room), share_id (shared ID, also known as file system ID), used_size (used amount, also known as resource usage), create_time (creation time, that is, the creation time of this data) are fields in the preset structured table, and the above fields are all useful information required. In actual applications, the field type, field quantity, and field sorting contained in the structured table can be flexibly set according to actual business and needs.

The original data is structured according to the above preset structured table, and the obtained structured data is shown in Table 2:

Table 2

As shown in Table 2, the data corresponding to each field in the preset structured table is searched from the messy unstructured raw data and filled into the structured table accordingly to obtain structured data. The unnecessary data in the structured table (such as share-type-id, usage_realtime and other information in the raw data) can be directly discarded. In this way, the raw data can be better organized, which is not only convenient for subsequent aggregation processing, but also can clean up the information of low importance, which helps to reduce the amount of data.

Considering that there may be data that does not meet the requirements in the acquired raw data, for example, some raw data does not contain the key information required for subsequent queries. Taking the raw data as resource consumption data as an example, resource ID, resource consumption and time are all essential key information for resource consumption data. Once the resource consumption data lacks such key information, it cannot be used later. In order to avoid such non-compliant data occupying storage space and wasting query time, the above method also includes: if the data corresponding to the key field specified in the preset structured table is not found from the raw data, the raw data is discarded. The key field can be an indispensable field, which can be set according to actual needs.

This embodiment provides an implementation method for aggregating structured data to obtain aggregated data, including the following steps (1) and (2):

(1) Grouping the structured data to obtain at least one data group; wherein the data filled in the corresponding preset fields of multiple structured data in the same data group are the same. For example, resource consumption data (which has been structured) with the same file system identifier within a preset time period are divided into one data group.

(2) For each data group, multiple structured data in the data group are aggregated to obtain aggregated data corresponding to the data group; wherein the aggregated data is a structured data selected from the multiple structured data in the data group, or the aggregated data is a new structured data formed by the multiple structured data in the data group according to a preset algorithm. Assume that the structured data is structured resource consumption data, which contains resource consumption. From each data group, a structured data with the largest resource consumption is selected as the aggregated data corresponding to the data group; for another example, the resource consumption in all the structured data in the data group is calculated according to a preset algorithm (such as a weighted algorithm) to obtain the resource consumption for charging, thereby forming a new structured data based on the calculated resource consumption.

Considering that most of the prior art directly stores the original data in a relational database, and the relational database is usually deployed on only one server, there is a storage bottleneck, which makes it difficult to meet the situation of large data volume. Based on processing the original data to obtain aggregated data with a smaller amount of data, in order to better expand the storage space and improve the existing storage bottleneck problem, the data processing method provided by this embodiment also includes: storing the obtained aggregated data in a non-relational database; the non-relational database is deployed on a server cluster. Through this distributed storage method, the storage problem that may be caused by the amount of data is further alleviated. In a specific implementation method, the non-relational database can be implemented based on a distributed ElasticSearch (ES), which can store and retrieve data online in near real time.

This embodiment further provides a specific implementation method of data processing, in which the original product is resource consumption data, and the billing product is dispersedly deployed in the server cluster. A data collector flume is set on each server. Flume is a data warehouse tool that can collect the required data and collect a large amount of data and send it to data warehouse tools such as Hive based on Hadoop for further processing. Hadoop is a distributed system infrastructure that allows users to develop distributed programs without understanding the details of the distributed underlying layer, and fully utilize the advantages of the cluster for high-speed computing and storage. Hadoop's storage system is HDFS (Hadoop Distributed File System), and Hive can perform underlying storage based on HDFS. Referring to the specific flow diagram of a data processing method shown in Figure 2, it includes the following steps:

Step S202: the data collector flume obtains the original data of the current server at a predetermined frequency, and transmits the original data to Hive to be recorded in the original Hive table.

Hive can be regarded as a data warehouse tool based on Hadoop. It is similar to a relational database and has the visualization capability of a relational database. In this embodiment, it can be used as an intermediate repository used in the raw data processing process. If the data collector Flume sends the raw data to Hive, an original Hive table storing the raw data can be formed.

Step S204: Use Spark to perform structured processing on the original data recorded in the original Hive table, and migrate the obtained structured data to the structured Hive table in batches.

Spark is a fast, general-purpose large-scale data processing engine. In this embodiment, Spark can be used to perform batch task cleaning and parsing on the raw data in the original Hive table, convert the data in the original Hive table into structured data, and ETL (Extract-Transform-Load) the obtained structured data to the structured Hive table. The above batch migration process is also the ETL process. For example, Spark can find the required useful information (such as the data corresponding to each field in the aforementioned preset structured table) from the messy and unstructured raw data, and store the obtained useful information in the corresponding field of the preset preset structured table to obtain structured data, and then store the structured data in batches in the structured Hive table.

Step S206: Use Spark to batch migrate structured Hive tables to ElasticSearch, and perform aggregation processing on the structured data.

ElasticSearch (ES) is an open source, highly scalable distributed full-text search engine that can store and retrieve data online in near real time. It has good scalability and can be expanded to hundreds of servers and process PB-level data. Migrating structured Hive tables to ElasticSearch is conducive to subsequent rapid retrieval through ElasticSearch (online query is possible), further shortening query time and improving query efficiency. In addition, Spark can further aggregate structured data based on preset aggregation rules to reduce the amount of data ultimately stored in ElasticSearch. In actual applications, Spark can use offline processing when performing data structuring and aggregation processing.

Step S208: storing the aggregated data obtained by the aggregation process in a structured table of the non-relational database ElasticSearch.

Since the structured tables of the non-relational database ElasticSearch can be deployed on a server cluster, the storage space can be expanded well, which can effectively improve the storage bottleneck problem caused by the traditional relational database being deployed on only one server. Moreover, the amount of aggregated data obtained after processing is much smaller than the amount of original data, which effectively improves the problem of large data volume, not only saving storage resources, but also helping to improve query efficiency.

Furthermore, this embodiment also provides a data query method. Referring to the flow chart of the data query method shown in FIG. 3 , the method mainly includes the following steps S302 to S304:

Step S302, if a data query request is received, the aggregated data corresponding to the data query request is searched from a preset database; wherein the database stores aggregated data obtained by any of the data processing methods described above; since the database stores aggregated data obtained after processing the original data, compared to the existing database that directly stores the original data and uses mysql for querying, the amount of data stored in the database in this embodiment is less, so the search time is shorter, which can effectively improve the query efficiency. In practical applications, the above database can be a single-point relational database or a multi-point distributed non-relational database. Preferably, the aggregated data can be stored in a non-relational database, such as storing the aggregated data online in ElasticSearch, which is not only conducive to distributed storage to expand the storage space, but also uses ElasticSearch for retrieval in a shorter time and faster speed.

Step S304: Feedback the found aggregated data to the requester of the data query request.

The above-mentioned data query method provided by the embodiment of the present invention can search for aggregated data corresponding to the received data query request from a preset database, and feed it back to the requester of the data query request. Since the database stores the aggregated data obtained by the above-mentioned data processing method, the amount of data is smaller than the amount of data of the original data directly stored in the database in the prior art. Therefore, this method can effectively shorten the data query time and improve the query efficiency.

Corresponding to the data processing method provided in the above embodiment, the present embodiment further provides a data processing device, as shown in FIG4 , the device comprises a data acquisition module 40, a data structuring module 42 and a data aggregation module 44 connected in sequence, wherein:

A data acquisition module 40 is used to acquire raw data to be processed;

The data structuring module 42 is used to perform structured processing on the original data according to a specified format to obtain structured data;

The data aggregation module 44 is used to aggregate the structured data to obtain aggregated data, and use the aggregated data as data after the original data is processed; wherein the data volume of the aggregated data is smaller than the data volume of the original data.

The above data processing device provided by the embodiment of the present invention can perform structured processing on the collected raw data according to the specified format to obtain structured data, and then aggregate the structured data to obtain aggregated data, so as to use the aggregated data as the data after the raw data is processed. This embodiment adopts the method of performing structured processing and aggregating the raw data, which can effectively reduce the amount of raw data and improve the problem of huge data volume in the prior art.

In a specific implementation, the data acquisition module 40 is further used to: obtain the original data to be processed from the server where the billing product is deployed at a preset frequency; wherein the original data is the resource consumption data of the billing product recorded by the server; and the number of servers is one or more.

In one embodiment, the data structuring module 42 is used to: search for data corresponding to each field in the preset structured table from the original data, and fill the found data in the corresponding field in the preset structured table; and use the filled preset structured table as structured data. On this basis, the device further includes: a data discarding module, which is used to discard the original data if the data corresponding to the key field specified in the preset structured table is not found from the original data.

In a specific embodiment, the data structuring module 42 is further used to: group the structured data to obtain at least one data group; wherein, the data filled in the corresponding preset fields of multiple structured data in the same data group are the same; for each data group, aggregate the multiple structured data in the data group to obtain aggregated data corresponding to the data group; wherein the aggregated data is a structured data selected from the multiple structured data in the data group, or the aggregated data is a new structured data formed by the multiple structured data in the data group according to a preset algorithm.

In one embodiment, the above-mentioned device further includes: a storage module, used to store the obtained aggregated data in a non-relational database; the non-relational database is deployed on a server cluster.

Corresponding to the data query method provided in the above embodiment, the present embodiment further provides a data query device, as shown in FIG5 , the device comprises a data query module 50 and a data feedback module 52 connected in sequence, wherein:

A data search module 50, for searching for aggregated data corresponding to the data query request from a preset database if a data query request is received; wherein the database stores aggregated data obtained by any of the above-mentioned data processing methods;

The data feedback module 52 is used to feed back the found aggregated data to the requester of the data query request.

The above-mentioned data query device provided by the embodiment of the present invention can search for aggregated data corresponding to the received data query request from a preset database, and feed back the aggregated data to the requester of the data query request. Since the database stores the aggregated data obtained by the above-mentioned data processing method, the amount of data is smaller than the amount of data of the original data directly stored in the database in the prior art. Therefore, this method can effectively shorten the data query time and improve the query efficiency.

The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

This embodiment further provides a network device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the aforementioned data processing method or the aforementioned data query method.

Figure 6 is a structural diagram of a network device provided in an embodiment of the present invention. The device 100 includes: a processor 60, a memory 61, a bus 62 and a communication interface 63. The processor 60, the communication interface 63 and the memory 61 are connected via the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.

The memory 61 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

The bus 62 may be an ISA bus, a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bidirectional arrow is used in FIG6 , but this does not mean that there is only one bus or one type of bus.

Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 60 or implemented by the processor 60.

The processor 60 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 60. The above processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.

An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the aforementioned data processing method or the aforementioned data query method. The specific implementation can be found in the method embodiment, which will not be repeated here.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims (9)

1. A method of data processing, comprising:

Collecting original data to be processed;

carrying out structuring treatment on the original data according to a specified format to obtain structured data;

performing aggregation processing on the structured data to obtain aggregated data, wherein the aggregated data is used as data processed by the original data; wherein the data volume of the aggregate data is smaller than the data volume of the original data;

The step of carrying out structuring treatment on the original data according to a specified format to obtain structured data comprises the following steps:

Searching data corresponding to each field in a preset structured table from the original data, and filling the searched data in the corresponding field in the preset structured table;

taking the filled preset structured table as structured data;

The step of performing aggregation processing on the structured data to obtain aggregated data comprises the following steps:

Grouping the structured data to obtain at least one data group; wherein, the data correspondingly filled in the plurality of structured data in the same data group in the preset field are the same;

For each data group, aggregating a plurality of pieces of structured data in the data group to obtain aggregation data corresponding to the data group; the aggregation data is one piece of structured data selected from a plurality of pieces of structured data in the data group, or the aggregation data is one new piece of structured data formed by the plurality of pieces of structured data in the data group according to a preset algorithm.

2. The method according to claim 1, wherein the method further comprises:

and discarding the original data if the data corresponding to the key field specified in the preset structural table is not found in the original data.

3. The method according to claim 1, wherein the method further comprises:

Storing the obtained aggregation data in a non-relational database; the non-relational database is deployed on a server cluster.

4. A method according to any one of claims 1 to 3, wherein the step of collecting raw data to be processed comprises:

Acquiring raw data to be processed from a server deployed with a charging product according to a preset frequency; wherein, the original data is the resource consumption data of the charging product recorded by the server; the number of the servers is one or more.

5. A method of querying data, comprising:

If a data query request is received, searching aggregate data corresponding to the data query request from a preset database; wherein the database stores therein aggregated data obtained by the data processing method according to any one of claims 1 to 4;

and feeding the searched aggregated data back to a requester of the data query request.

6. A data processing apparatus, comprising:

The data acquisition module is used for acquiring the original data to be processed;

The data structuring module is used for structuring the original data according to a specified format to obtain structured data;

The data aggregation module is used for carrying out aggregation processing on the structured data to obtain aggregated data, and taking the aggregated data as the data processed by the original data; wherein the data volume of the aggregate data is smaller than the data volume of the original data;

The data structuring module is further configured to: searching data corresponding to each field in a preset structured table from the original data, and filling the searched data in the corresponding field in the preset structured table; taking the filled preset structured table as structured data;

the data aggregation module is further configured to: grouping the structured data to obtain at least one data group; wherein, the data correspondingly filled in the plurality of structured data in the same data group in the preset field are the same; for each data group, aggregating a plurality of pieces of structured data in the data group to obtain aggregation data corresponding to the data group; the aggregation data is one piece of structured data selected from a plurality of pieces of structured data in the data group, or the aggregation data is one new piece of structured data formed by the plurality of pieces of structured data in the data group according to a preset algorithm.

7. A data query device, comprising:

the data searching module is used for searching aggregate data corresponding to the data query request from a preset database if the data query request is received; wherein the database stores therein aggregated data obtained by the data processing method according to any one of claims 1 to 4;

and the data feedback module is used for feeding the searched aggregated data back to a requester of the data query request.

8. A network device comprising a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor to perform the method of any one of claims 1 to 4 or to perform the method of claim 5.

9. A machine-readable storage medium storing machine-executable instructions which, when invoked and executed by a processor, cause the processor to implement the method of any one of claims 1 to 4 or cause the processor to implement the method of claim 5.