CN107798009A - Data aggregation method, apparatus and system - Google Patents

Abstract

The invention discloses a data aggregation method, device and system, which relate to the field of information technology, can improve the efficiency of data aggregation, and can also solve the problem of delay in data aggregation operations, thereby ensuring the real-time performance of data aggregation. The main technical solution of the present invention is: when it is necessary to aggregate the massive business data stored in the cloud environment according to a certain period, first obtain the business data currently received by the data collector from the data collector, and then judge the time stamp Whether it is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and if it is greater than, aggregate the business data with the saved timestamp as the aggregation trigger time point. The invention is mainly used for aggregating massive business data in the cloud environment.

Description

Data aggregation method, device and system

technical field

The present invention relates to the field of information technology, in particular to a data aggregation method, device and system.

Background technique

With the continuous development of the Internet, the storage and statistics of business data in the cloud environment are becoming more and more common. For example, to store and count different types of business data such as QPS (Query Per Second), PV (Page View, page views), the specific process can be shown in Figure 11. The data is stored in files or other storage media, and then the files are read at regular intervals, and the contents of the files are calculated using MapReduce. Since the amount of data stored in the cloud environment is large, it is necessary to aggregate the massive business data when performing statistics on the massive business data stored in the cloud environment.

At present, when aggregating massive business data stored in the cloud environment, first write the business data received in different time periods into files, then read all the business data of a certain category that needs to be aggregated from the files, and finally The read business data is sorted by time and then aggregated. However, since data aggregation requires reading and writing operations on files, in order to solve the problem of file lock conflicts in the process of file reading and writing operations, a file copy process for saving the data to be aggregated will be performed during data aggregation. , resulting in a high delay in data calculation; and because it is necessary to sort the read business data according to time, the process of data aggregation is cumbersome, the efficiency of data aggregation operation is low, and the delay of data aggregation operation is high.

Contents of the invention

In view of this, embodiments of the present invention provide a data aggregation method, device, and system, the main purpose of which is to solve the problem of high delay in data aggregation operations when aggregating massive business data stored in a cloud environment.

To achieve the above object, the present invention provides the following technical solutions:

In one aspect, the present invention provides a data aggregation method, comprising:

When data aggregation is performed according to the first preset period, the business data currently received by the data collector is obtained from the data collector, the business data carries a time stamp, and the time stamp is used to identify the data collector The time when the business data is received;

Judging whether the timestamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, the first preset period corresponds to multiple aggregation trigger time points, and the multiple aggregation trigger time points The corresponding time increases one by one;

If yes, aggregate the saved time stamp as the service data at the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required.

In another aspect, the present invention provides a data collection method, comprising:

Receive business data to be aggregated;

configuring a time stamp for the business data, where the time stamp is used to identify the time when the data collector receives the business data,

so that the analysis and aggregation server acquires the service data and performs data aggregation according to the time stamp of the service data.

In yet another aspect, the present invention provides a method for querying aggregated data, including:

sending a data reporting instruction to the analysis and aggregation server, the data reporting instruction carrying business data that needs to be aggregated, so that the analysis and aggregation server receives the business data according to the time of the data collector and the preset aggregation trigger time point, Carry out data aggregation for the business data that needs to be aggregated according to the preset cycle;

Sending a data aggregation result query instruction to the analysis aggregation server, the data query instruction carrying time information of service data to be queried;

The data aggregation result corresponding to the time information is received and displayed.

In another aspect, the present invention provides a data aggregation device, comprising:

An acquisition unit, configured to acquire from the data collector the business data currently received by the data collector when the data aggregation is performed according to the first preset period, the business data carries a time stamp, and the time stamp is used to identify The time when the data collector receives the business data;

A judging unit, configured to judge whether the timestamp acquired by the acquiring unit is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and the first preset period corresponds to multiple aggregation triggers Time points, the times corresponding to the plurality of aggregation trigger time points are incremented one by one;

The aggregation unit is configured to aggregate the business data with the saved time stamp as the aggregation trigger time point if the judging unit determines that it is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required .

In yet another aspect, the present invention provides a data collection device, comprising:

a receiving unit, configured to receive service data to be aggregated;

a configuration unit, configured to configure a time stamp for the service data, where the time stamp is used to identify the time when the data collector receives the service data,

so that the analysis and aggregation server acquires the service data and performs data aggregation according to the time stamp of the service data.

In yet another aspect, the present invention provides an aggregated data query device, comprising:

A sending unit, configured to send a data reporting instruction to the analysis and aggregation server, the data reporting instruction carries business data that needs to be aggregated, so that the analysis and aggregation server receives the business data according to the time and preset time of the data collector Aggregation trigger time point, data aggregation is performed on the business data that needs to be aggregated according to the preset cycle;

The sending unit is further configured to send a data aggregation result query instruction to the analysis aggregation server, where the data query instruction carries time information of service data to be queried;

The receiving unit is configured to receive and display the data aggregation result corresponding to the time information.

In yet another aspect, the present invention provides a data aggregation system, including: a data collector and an analysis aggregation server;

The data collector is configured to configure a time stamp for the business data when receiving the business data, and save the business data and the time stamp of the business data;

The analysis aggregation server is configured to obtain from the data collector the business data currently received by the data collector when the data aggregation is performed according to the first preset period, and the business data carries a time stamp;

The analysis and aggregation server is further configured to determine whether the timestamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and the first preset period corresponds to multiple aggregation trigger time points , the times corresponding to the plurality of aggregation trigger time points are incremented one by one;

If yes, aggregate the saved timestamp as the service data at the aggregation trigger time point.

With the above technical solutions, the technical solutions provided by the embodiments of the present invention have at least the following advantages:

In the data aggregation method, device and system provided by the embodiments of the present invention, when it is necessary to aggregate the massive business data stored in the cloud environment according to a certain period, first obtain the business data currently received by the data collector from the data collector data, and then determine whether the time stamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and if it is greater than, then save the time stamp as the business data at the aggregation trigger time point polymerization. Compared with the current aggregation of massive business data stored in the cloud environment, it is necessary to read and write files used to store business data, and also need to sort the read business data according to time, the present invention Since it is only necessary to detect whether the timestamp of the current business data is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, the aggregation of business data in different time periods can be realized, and the efficiency of data aggregation can be improved. Efficiency, and since the detection operation is performed in real time, the delay of the data aggregation operation can be reduced, and the real-time performance of the data aggregation can be improved at the same time.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same components. In the attached picture:

FIG. 1 shows a flowchart of a data aggregation method provided by an embodiment of the present invention;

FIG. 2 shows a flowchart of another data aggregation method provided by an embodiment of the present invention;

FIG. 3 shows a flow chart of an aggregated data query method provided by an embodiment of the present invention;

FIG. 4 shows a flowchart of a data collection method provided by an embodiment of the present invention;

FIG. 5 shows a schematic structural diagram of a data aggregation device provided by an embodiment of the present invention;

FIG. 6 shows a schematic structural diagram of another data aggregation device provided by an embodiment of the present invention;

Fig. 7 shows a schematic structural diagram of an aggregated data query device provided by an embodiment of the present invention;

Fig. 8 shows a schematic structural diagram of an aggregation data statistics device provided by an embodiment of the present invention;

FIG. 9 shows a schematic structural diagram of a data aggregation system provided by an embodiment of the present invention;

Fig. 10 shows a schematic diagram of business data flow provided by an embodiment of the present invention;

Fig. 11 shows a schematic diagram of the architecture of the data statistics method in the prior art;

Fig. 12 shows a schematic diagram of the logical architecture of the aggregated data query method provided by the embodiment of the present invention;

Fig. 13 shows a schematic diagram of the logical structure of the entry server provided by the embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

A data aggregation method provided by an embodiment of the present invention, as shown in FIG. 1, the method includes:

101. When data aggregation is performed according to a first preset period, acquire service data currently received by the data collector from the data collector.

Wherein, the first preset period is used to reflect the time granularity for data aggregation, and different time granularities can be configured according to different data aggregation requirements. For example, when data aggregation with a small time granularity is required, the first preset period can be configured relatively small, such as once every minute, once every five minutes, etc., for example, when data aggregation with a large time granularity is required , the first preset period can be configured to be relatively larger, such as aggregation once every hour, aggregation once every half day, and so on. For the embodiment of the present invention, configuring different aggregation time granularities according to different requirements can further improve the accuracy of data aggregation. The service data carries a time stamp, and the time stamp is used to identify the time when the service data is triggered after the data collector receives the service data. For example, if the service data is triggered at 10:01 after the data collector receives the service data, the time stamp carried in the service data currently obtained from the data collector is 10:01. The business data can be business data of different categories such as QPS (Query Per Second, query rate per second), PV (Page View, page views), or business data of different applications of the same category, such as application A and application A. The PV data of B is not limited in this embodiment of the present invention.

102. Determine whether the time stamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required.

Wherein, the first preset period corresponds to a plurality of aggregation trigger time points, and the times corresponding to the plurality of aggregation trigger time points are incremented one by one, and each aggregation trigger time point is used to trigger the pair timestamp as the last aggregation trigger time All business data of the point is aggregated. For example, if the first preset period is aggregated once every minute, then starting from 10:00, the first preset period corresponds to multiple aggregation trigger time points, respectively 10:01, 10:02, 10:03, etc., and so on. When the time stamp 10:02 is greater than the aggregation trigger time point 10:01 corresponding to the first preset period when the data aggregation is currently performed, the aggregation processing of all business data with the time stamp 10:01 is triggered.

It should be noted that the multiple aggregation trigger time points can be displayed in a list, time axis, etc. in chronological order, which is not limited in this embodiment of the present invention. For example, when multiple aggregation trigger time points are displayed in a time axis in chronological order etc., the time on the time axis will continuously roll forward to ensure that the time on the time axis is the current time. For the embodiment of the present invention, by detecting whether the time stamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, that is, the time between the time of the currently received business data and the time of the last received business data is detected. Whether the time of the data changes, so as to determine whether to trigger the business data aggregation operation, thereby ensuring the real-time performance of the business data aggregation operation.

For example, the first preset period is aggregated once per minute. If the currently received service data obtained from the data collector carries a timestamp of 10:01, the time stamp corresponding to the first preset period when data aggregation is currently required If the aggregation trigger time point is also 10:01, it is determined not to trigger the business data aggregation operation; if the time stamp carried by the currently received business data obtained from the data collector is 10:02, which is greater than the time stamp specified when data aggregation is currently required The aggregation trigger time point corresponding to the first preset period is 10:01, and it is determined to trigger the service data aggregation operation.

103. If it is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, aggregate the service data with the saved time stamp as the aggregation trigger time point.

Wherein, the time stamp is stored in a preset storage location for the service data at the aggregation trigger time point. The preset storage location may be a preset mapping table, a preset queue, etc., which are not limited in this embodiment of the present invention. For this embodiment of the present invention, if the timestamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, for example, from 10:01 to 10:02, or from 10:01:01 It has changed to 10:01:02, indicating that the timestamp carried by the current business data has changed forward, that is, the aggregation operation is triggered in real time, thereby ensuring the real-time performance of the business data aggregation operation and improving the efficiency of the business data aggregation operation.

In the data aggregation method provided by the embodiment of the present invention, when it is necessary to aggregate the massive business data stored in the cloud environment according to a certain period, first obtain the business data currently received by the data collector from the data collector, and then judge Whether the time stamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and if so, aggregate the business data with the saved time stamp as the aggregation trigger time point. Compared with the current aggregation of massive business data stored in the cloud environment, it is necessary to read and write files used to store business data, and also need to sort the read business data according to time, the present invention In the embodiment, it is only necessary to detect whether the time stamp of the current business data is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, so that the aggregation of business data in different time periods can be realized, and the data can be improved. The efficiency of aggregation, and since the detection operation is performed in real time, the delay of data aggregation operation can be reduced, and the real-time performance of data aggregation can be improved at the same time.

Further, the embodiment of the present invention provides another data aggregation method, as shown in FIG. 2, the method includes:

201. When data aggregation is performed according to a first preset period, acquire service data currently received by the data collector from the data collector.

Wherein, the service data carries a time stamp, and the time stamp is used to identify the time when the service server receives the service data. For the corresponding description of the first preset period and the service data, reference may be made to the corresponding description of the embodiment shown in FIG. 1 , which will not be repeated here.

For the embodiment of the present invention, step 201 may specifically include: when performing data aggregation according to the first preset period, obtaining the business data from the preset queue corresponding to the current aggregation task, and the data collection server provides Configure a preset queue for business data. Specifically, as shown in FIG. 10 , transponders such as ONS (message service) and Metaq (message middleware) can be configured in the data collector, and the transponders are used for data transfer between the data collector and the aggregation analysis server. Specifically, the data collector can map different types of business data to different virtual nodes 0, 1, 2, etc. through preset routing rules, and then save each type of business data to different queues Queue1 of the ONS through the virtual nodes , Queue2, etc., and then configure a task for each queue, such as task ServerNode1, so as to ensure business data flow and subsequent data aggregation processing by category, and further improve the accuracy of data aggregation.

202. Determine whether the time stamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required.

Wherein, the first preset period corresponds to multiple aggregation trigger time points, and the times corresponding to the multiple aggregation trigger time points are incremented one by one. For a specific description of the configuration manner of the first preset period, reference may be made to the corresponding description of the embodiment shown in FIG. 1 , and details are not repeated here.

For the embodiment of the present invention, by detecting whether the time stamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, that is, the time between the time of the currently received business data and the time of the last received business data is detected. Whether the time of the data changes forward determines whether to trigger the business data aggregation operation, thereby ensuring the real-time performance of the business data aggregation operation.

Further, after step 202, the method may further include: if the time stamp is the same as the aggregation trigger time point, saving the currently received service data into the preset queue corresponding to the current aggregation task. Wherein, all service data whose time stamp is the triggering time point of the aggregation is cached in the preset queue. Different preset queues cache business data with different aggregation triggering time points. Since the delay of caching and reading operations on the queues is low, caching business data in the queues can further reduce the delay of data aggregation operations and improve Real-time performance of data aggregation operations. If the time stamp is less than the aggregation trigger time point, the currently received service data is deleted. If the aggregation trigger time point is 10:01 and the timestamp is 09:50, the currently received service data with the timestamp of 09:50 will be deleted, so as to ensure the accuracy of data aggregation.

203. If it is greater than the aggregation triggering time point corresponding to the first preset period when data aggregation is currently required, determine whether the time stamp is received within the first preset time period after the service data is currently received. Aggregate business data at the trigger time point.

It should be noted that when the business data of a certain time stamp is aggregated, due to the delay of transmission data, etc., the data collector may receive the business data of this time stamp again. At this time, if the business data received with delay Processing will result in the loss of business data, which in turn will make the accuracy of business data aggregation lower. Therefore, before data aggregation, by configuring the time tolerance period, that is, the first preset time period, it is possible to avoid delaying the received business Data loss, improve the accuracy of business data aggregation.

Wherein, the first preset time period may be configured according to a first preset period. For example, if the first preset period is relatively small, such as once per second, then the first preset time period can be 1 second, 2 seconds, etc., if the first preset period is relatively moderate, such as once per minute, then The first preset time period may be 10 seconds, 20 seconds, etc. If the first preset period is relatively large, such as once per hour, the first preset time period may be 10 minutes, 20 minutes, etc. By configuring the first preset time period according to the first preset period, the configuration accuracy of the time tolerance period can be further improved, thereby further improving the accuracy of service data aggregation.

204a. If not received, aggregate the saved time stamp as the service data at the aggregation trigger time point.

In step 204b parallel to step 204a, if received, aggregate the service data received within the first preset time period and the saved time stamp as the service data at the aggregation trigger time point.

For example, if the first preset period is aggregated every minute, then the preset buffer time period is configured as 5 minutes. If the time stamp carried by the business data currently obtained from the data collector is 10:02, and the aggregation trigger time point corresponding to the first preset period is 10:01 when data aggregation is currently required, then the time stamp is 10:01. Before the business data of :01 is aggregated, it is judged whether the business data with the time stamp of 10:01 is received within five minutes from the current time. If received, the time stamp received within five minutes is The business data at 10:01 is aggregated with the previously received business data with a time stamp of 10:01; if not received, it means that there is no delayed business data at this time, and the business data with a time stamp of 10:01 is aggregated. Data aggregation, thus ensuring the accuracy of data aggregation.

205. Save the aggregated service data and the time stamp corresponding to the service data in a first preset period aggregation table.

Wherein, the aggregation table of the first preset period is used to save the data aggregated according to the first preset period. The aggregation table for the first preset period may be stored in a storage medium such as OTS (table storage), HBase (distributed column-oriented database). For example, if the first preset period is aggregated once every minute, the data aggregated every minute is stored in the first preset period aggregation table. The aggregation table of the first preset period may be as follows:

For the embodiment of the present invention, in order to realize data aggregation at different time granularities at the same time, after step 205, it may also include: when data aggregation needs to be performed according to the second preset period at the same time, obtain the data collector from the data collector The time stamp of the currently received service data, the second preset period is greater than the first preset period, and the second preset period is an integer multiple of the first preset period; then judge the Whether the time stamp reaches the aggregation trigger time point corresponding to the second preset period, the second preset period corresponds to a plurality of aggregation trigger time points, and the times corresponding to the plurality of aggregation trigger time points are incremented one by one; if so, Then, from the aggregation table of the first preset period, obtain the service data whose time stamp is between the last aggregation trigger time point corresponding to the second preset period and the current aggregation trigger time point, and aggregate the aggregated business data The data is saved in the aggregation table of the second preset period.

For example, the first preset period is aggregation every minute, the second preset period is aggregation every five minutes, and the timestamp of the currently received service data is 10:06, which is greater than the aggregation corresponding to the second preset period When the trigger time point is 10:05, the business data whose time stamp is from 10:01 to 10:05 is obtained from the 1-minute aggregation table for aggregation, and the aggregated business data is saved in the 5-minute aggregation table.

For the embodiment of the present invention, before the step of saving the aggregated business data in the aggregation table of the second preset period, it may further include: according to the aggregation trigger time corresponding to the current second preset period, the aggregated business data Data configuration time stamp, based on the above example, at this time, configure the time stamp for the aggregated service data as 10:05. At this time, the aggregated service data and the time stamp corresponding to the service data are saved in the second preset period aggregation table.

Further, after step 205, it may also include: receiving a data query instruction sent by the data query server, the data query instruction carrying time information; obtaining the time stamp and the time information from the first preset period aggregation table The corresponding business data is sent to the query server, so that the query server can display the data. For example, when the time information is 10:01-10:05, the time stamps obtained from the aggregation table of the first preset period are 10:01, 10:02, 10:03, 10:04, and 10: Aggregate data for 05.

Wherein, the query server may be a Portal (entry server), and the Portal may provide various types of query interfaces, so that the client may query the aggregated service data by invoking the data interface provided by the Portal. At the same time, the Portal can also provide a registration interface for business data, so that the data collector can verify the registration information of the business data according to the registration information stored in the Portal.

For the embodiment of the present invention, the aggregation trigger time point corresponding to the first preset period and the current first preset period may also be saved in a preset storage location. By storing the first preset period and the aggregation trigger time point corresponding to the current first preset period in the preset storage location, it can be guaranteed that when the system is restarted or the system recovers from a failure, the time state before the restart or failure can be restored , so as to prevent data loss and double calculation of data, and further improve the accuracy of data aggregation.

Another data aggregation method provided by the embodiment of the present invention, when it is necessary to aggregate the massive business data stored in the cloud environment according to a certain period, first obtain the business data currently received by the data collector from the data collector, and then Judging whether the time stamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and if so, aggregate the business data with the saved time stamp as the aggregation trigger time point. Compared with the current aggregation of massive business data stored in the cloud environment, it is necessary to read and write files used to store business data, and also need to sort the read business data according to time, the present invention In the embodiment, it is only necessary to detect whether the time stamp of the current business data is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, so that the aggregation of business data in different time periods can be realized, and the data can be improved. The efficiency of aggregation, and since the detection operation is performed in real time, the delay of data aggregation operation can be reduced, and the real-time performance of data aggregation can be improved at the same time.

Further, an embodiment of the present invention provides an aggregated data query method, as shown in FIG. 3, the method includes:

301. Send a data reporting instruction to the analysis aggregation server.

Wherein, the data reporting instruction carries the business data that needs to be aggregated, so that the analysis and aggregation server performs the required aggregation according to the preset cycle according to the time when the data collector receives the business data and the preset aggregation trigger time point. Data aggregation is performed on the business data of the data aggregation. For the embodiment of the present invention, step 301 may specifically include: sending a data reporting instruction to the data aggregation server by calling the Portal interface.

For the embodiment of the present invention, the logical structure diagram of the Portal can be shown in Figure 13. In the embodiment of the present invention, the Portal has two functions. The first is to query aggregated data and provide users with various types of business data. Query interface, so as to query the aggregation result from the storage server OTS in the aggregation system through the query interface, and return it to the client for aggregated data display. The second is to register the reported data metadata, and then write the registration information into RDS for data verification when the collector receives the data.

302. Send a data aggregation result query instruction to the analysis aggregation server.

Wherein, the data query instruction carries time information of the service data to be queried. For the embodiment of the present invention, step 302 may specifically include: sending a data aggregation result query instruction to the analysis aggregation server through a data query interface configured on the portal server Portal.

303. Receive and display a data aggregation result corresponding to the time information.

For the embodiment of the present invention, the specific application scenario can be shown in Figure 12, including: the process is as follows: first call the Portal interface to register the monitoring items that need to report data to the aggregation system, and then call the collector reporting interface to report the specified monitoring items to the system At this time, the collector will send the received data to the middleware ONS, and then the analyzer will pull the data from ONS for data analysis and aggregation, and store the data in the OTS (storage server) after aggregation calculation. When aggregated data query is required, users can query the aggregated data results through the query interface provided by Portal, and display the statistical results of the data through the data display page.

For the embodiment of the present invention, the portal server Portal provides users with an interface for reporting and querying aggregated data, that is, users can conveniently and quickly report the data to be aggregated and query the aggregation results, thereby improving the reporting efficiency of the data to be aggregated and improving the efficiency of the aggregated data. Query efficiency of aggregated results.

Further, an embodiment of the present invention provides a data collection method, as shown in FIG. 4, the method includes:

401. Receive service data to be aggregated. 402. Configure a time stamp for the service data.

Wherein, the timestamp is used to identify the time when the data collector receives the service data, so that the analysis and aggregation server acquires the service data and performs data aggregation according to the timestamp of the service data.

For the embodiment of the present invention, by configuring the time stamp for the received service data, it can be ensured that the aggregation of the service data is triggered in time, so as to realize the real-time aggregation of the service data.

Further, as a specific implementation of the method shown in FIG. 1 , an embodiment of the present invention provides a data aggregation device. As shown in FIG. 5 , the device may include: an acquisition unit 51 , a judging unit 52 , and an aggregation unit 53 .

The obtaining unit 51 is configured to obtain from the data collector the service data currently received by the data collector when the data aggregation is performed according to the first preset period, the service data carries a time stamp, and the time stamp is used for Identify the time when the data collector receives the service data;

A judging unit 52, configured to judge whether the timestamp acquired by the acquiring unit 51 is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is required, and the first preset period corresponds to a plurality of Aggregation triggering time points, the times corresponding to the plurality of aggregation triggering time points are incremented one by one;

The aggregation unit 53 is configured to store the saved timestamp as the service data at the aggregation trigger time point if the judging unit 52 determines that it is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required to aggregate.

It should be noted that this device embodiment corresponds to the foregoing method embodiment. For the convenience of reading, this device embodiment will not repeat the details of the foregoing method embodiment one by one, but it should be clear that the device in this embodiment can Correspondingly implement all the contents in the foregoing method embodiments.

A data aggregation device provided by an embodiment of the present invention, when it is necessary to aggregate massive business data stored in a cloud environment according to a certain period, first obtains the business data currently received by the data collector from the data collector, and then judges Whether the time stamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and if so, aggregate the business data with the saved time stamp as the aggregation trigger time point. Compared with the current aggregation of massive business data stored in the cloud environment, it is necessary to read and write files used to store business data, and also need to sort the read business data according to time, the present invention In the embodiment, it is only necessary to detect whether the time stamp of the current business data is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, so that the aggregation of business data in different time periods can be realized, and the data can be improved. The efficiency of aggregation, and since the detection operation is performed in real time, the delay of data aggregation operation can be reduced, and the real-time performance of data aggregation can be improved at the same time.

Further, as a specific implementation of the method shown in FIG. 2 , an embodiment of the present invention provides a data aggregation device. As shown in FIG. 6 , the device may include: an acquisition unit 61 , a judging unit 62 , and an aggregation unit 63 .

The obtaining unit 61 is configured to obtain from the data collector the business data currently received by the data collector when the data aggregation is performed according to the first preset period, the business data carries a time stamp, and the time stamp is used for Identify the time when the data collector receives the service data;

A judging unit 62, configured to judge whether the timestamp acquired by the acquiring unit 61 is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and the first preset period corresponds to a plurality of Aggregation triggering time points, the times corresponding to the plurality of aggregation triggering time points are incremented one by one;

The aggregation unit 63 is configured to store the saved timestamp as the service data at the aggregation trigger time point if the judging unit 62 determines that it is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required to aggregate.

Further, the device further includes: a saving unit 64 .

The saving unit 64 is configured to save the business data aggregated by the aggregation unit 63 and the time stamp corresponding to the business data in a first preset period aggregation table.

Further, the obtaining unit 61 is also configured to obtain from the data collector the time stamp of the service data currently received by the data collector when data aggregation needs to be performed according to the second preset period at the same time, the The second preset period is greater than the first preset period;

The judging unit 62 is further configured to judge whether the time stamp reaches the aggregation trigger time point corresponding to the second preset period, the second preset period corresponds to a plurality of aggregation trigger time points, and the plurality of aggregation trigger time points The times corresponding to the trigger time points are incremented one by one;

The aggregation unit 63 is further configured to, if the aggregation trigger time point corresponding to the second preset period is reached, obtain the time stamp from the aggregation table of the first preset period at the time point corresponding to the second preset period. Aggregate business data from the previous aggregation trigger time point to the current aggregation trigger time point;

The saving unit 64 is further configured to save the aggregated service data in a second preset period aggregation table.

Further, the device further includes: a configuration unit 65 .

The configuration unit 65 is configured to configure a time stamp for the aggregated service data according to the aggregation trigger time corresponding to the current second preset period;

The saving unit 64 is specifically configured to save the aggregated business data and the time stamp corresponding to the business data in the second preset period aggregation table.

Further, the acquisition unit 61 is specifically configured to acquire the business data from the preset queue corresponding to the current aggregation task when data aggregation is performed according to the first preset period, and the data collection server provides a Configure a preset queue for business data.

Further, the device further includes: a receiving unit 66 .

The receiving unit 66 is configured to receive a data query instruction sent by a data query server, and the data query instruction carries time information;

The obtaining unit 61 is further configured to obtain service data with a time stamp corresponding to the time information from the first preset period aggregation table and send it to the query server, so that the query server performs data presentation.

Further, the saving unit 64 is further configured to save the first preset period and the aggregation trigger time point corresponding to the current first preset period in a preset storage location.

Further, the judging unit 62 is also used to judge whether the business data whose timestamp is the aggregation trigger time point is received within the first preset time period after the business data is currently received;

The aggregation unit 63 is specifically configured to aggregate the stored timestamp as the service data at the aggregation trigger time point if not received; or

If received, aggregate the service data received within the first preset time period and the saved timestamp as the service data at the aggregation trigger time point.

Further, the device further includes: a deletion unit 67 .

The saving unit 64 is further configured to save the currently received service data in the preset queue corresponding to the current aggregation task if the time stamp is the same as the aggregation trigger time point;

The deleting unit 67 is configured to delete the currently received service data if the time stamp is smaller than the aggregation trigger time point.

It should be noted that this device embodiment corresponds to the foregoing method embodiment. For the convenience of reading, this device embodiment will not repeat the details of the foregoing method embodiment one by one, but it should be clear that the device in this embodiment can Correspondingly implement all the contents in the foregoing method embodiments.

Another data aggregation device provided by the embodiment of the present invention, when it is necessary to aggregate the massive business data stored in the cloud environment according to a certain period, first obtain the business data currently received by the data collector from the data collector, and then Judging whether the time stamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and if so, aggregate the business data with the saved time stamp as the aggregation trigger time point. Compared with the current aggregation of massive business data stored in the cloud environment, it is necessary to read and write files used to store business data, and also need to sort the read business data according to time, the present invention In the embodiment, it is only necessary to detect whether the time stamp of the current business data is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, so that the aggregation of business data in different time periods can be realized, and the data can be improved. The efficiency of aggregation, and since the detection operation is performed in real time, the delay of data aggregation operation can be reduced, and the real-time performance of data aggregation can be improved at the same time.

Still further, as a specific implementation of the method shown in FIG. 3 , an embodiment of the present invention provides an aggregated data query device. As shown in FIG. 7 , the device may include: a sending unit 71 and a receiving unit 72 .

The sending unit 71 is configured to send a data reporting instruction to the analysis and aggregation server, where the data reporting instruction carries business data that needs to be aggregated, so that the analysis and aggregation server receives the business data according to the time when the data collector receives the data. and the preset aggregation trigger time point, perform data aggregation on the business data requiring data aggregation according to the first preset cycle;

The sending unit 71 is further configured to send a data aggregation result query instruction to the analysis aggregation server, and the data query instruction carries time information of service data to be queried;

The receiving unit 72 is configured to receive and display the data aggregation result corresponding to the time information.

The sending unit 71 is specifically configured to send a data aggregation result query instruction to the analysis aggregation server through a data query interface configured on the portal server Portal.

It should be noted that this device embodiment corresponds to the foregoing method embodiment. For the convenience of reading, this device embodiment will not repeat the details of the foregoing method embodiment one by one, but it should be clear that the device in this embodiment can Correspondingly implement all the contents in the foregoing method embodiments.

An aggregated data query device provided by an embodiment of the present invention provides users with an interface for reporting and querying aggregated data through the portal server Portal, that is, users can conveniently and quickly report the aggregated data and query the aggregated results, thereby improving the number of aggregated data to be aggregated. The reporting efficiency of aggregated data and the query efficiency of aggregated results.

Still further, as a specific implementation of the method shown in FIG. 4 , an embodiment of the present invention provides a data collection device. As shown in FIG. 8 , the device may include: a receiving unit 81 and a configuration unit 82 .

a receiving unit 81, configured to receive service data to be aggregated;

The configuration unit 82 is configured to configure a time stamp for the business data, the time stamp is used to identify the time when the data collector receives the business data, so that the analysis aggregation server acquires the business data and The timestamp of business data is used for data aggregation.

The data collection device provided by the embodiment of the present invention can ensure that the aggregation of the business data is triggered in time by configuring a time stamp for the received business data, thereby realizing the real-time aggregation of the business data.

Furthermore, as a specific implementation of the method shown in FIG. 1 or FIG. 2 , an embodiment of the present invention provides a data aggregation system. As shown in FIG. 9 , the system may include: a data collector 91 and an analysis aggregation server 92 .

The data collector 91 is configured to configure a time stamp for the business data when receiving the business data;

The analysis aggregation server 92 is configured to obtain from the data collector the service data currently received by the data collector 91 when the data aggregation is performed according to the first preset period, and the service data carries a time stamp;

The analysis and aggregation server 92 is also used to determine whether the timestamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and the first preset period corresponds to multiple aggregation trigger times point, the times corresponding to the plurality of aggregation trigger time points are incremented one by one;

If yes, the time stamp stored in the aggregation server 92 is analyzed and aggregated as the service data at the aggregation trigger time point.

Further, the system further includes: a query server 93 .

The query server 93 is configured to send a data query instruction to the analysis and aggregation server 92, and the data query instruction carries time information;

The analysis and aggregation server 92 is further configured to acquire the business data with a time stamp corresponding to the time information from the first preset period aggregation table and send it to the query server;

The query server 93 is also used to output the business data.

It should be noted that this system embodiment corresponds to the foregoing method embodiment. For the convenience of reading, this system embodiment does not repeat the details of the foregoing method embodiment one by one, but it should be clear that the system in this embodiment can Correspondingly implement all the contents in the foregoing method embodiments.

In the data aggregation system provided by the embodiment of the present invention, when it is necessary to aggregate the massive business data stored in the cloud environment according to a certain period, the analysis aggregation server first obtains the business data currently received by the data collector from the data collector , and then analyze and aggregate the server to determine whether the time stamp is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, and if it is greater, the saved time stamp is the service at the aggregation trigger time point The data is aggregated. Compared with the current aggregation of massive business data stored in the cloud environment, it is necessary to read and write files used to store business data, and also need to sort the read business data according to time, the present invention In the embodiment, it is only necessary to detect whether the time stamp of the current business data is greater than the aggregation trigger time point corresponding to the first preset period when data aggregation is currently required, so that the aggregation of business data in different time periods can be realized, and the data can be improved. The efficiency of aggregation, and since the detection operation is performed in real time, the delay of data aggregation operation can be reduced, and the real-time performance of data aggregation can be improved at the same time.

The data aggregation device in the graph includes a processor and a memory. The acquisition unit, judgment unit, and aggregation unit are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

The processor includes a kernel, and the kernel fetches corresponding program units from the memory. One or more kernels can be set, and by adjusting the kernel parameters to solve the problem of display lag when displaying the selected graphics in the chart.

Memory may include non-permanent memory in computer-readable media, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM), memory includes at least one memory chip.

The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program code initialized with the following method steps: when data aggregation is performed according to a first preset cycle, all The business data currently received by the data collector, the business data carries a time stamp, and the time stamp is used to identify the time when the data collector receives the business data; then judge whether the time stamp is greater than the current The aggregation trigger time point corresponding to the first preset period when data aggregation is required, the first preset period corresponds to a plurality of aggregation trigger time points, and the times corresponding to the plurality of aggregation trigger time points are incremented one by one; if , the time stamp saved in the data collector is aggregated as the service data at the aggregation trigger time point.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of data aggregation methods, apparatuses, and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM. The memory is an example of a computer readable medium.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

The above are only examples of the present application, and are not intended to limit the present application. For those skilled in the art, various modifications and changes may occur in this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims (17)

1. A kind of 1. data aggregate system, it is characterised in that including：Data collector and analysis aggregate server；

   The data collector, during for receiving business datum, stabbed for the business datum setup time；

   The analysis aggregate server, for when carrying out data aggregate according to the first during cycle, being obtained from data collector The business datum that the data collector is currently received, the business datum carry timestamp；

   The analysis aggregate server, it is additionally operable to judge whether the timestamp is more than described in when being currently needed for carrying out data aggregate Polymerization triggered time point corresponding to first during cycle, first during cycle corresponds to multiple polymerization triggered time points, described Time corresponding to multiple polymerization triggered time points difference is incremented by one by one；

   If so, then business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.
2. 2. data aggregate system according to claim 1, it is characterised in that the system also includes：Inquire about server；

   The inquiry server, for sending data query instruction, the data query instruction to the analysis aggregate server In carry temporal information；

   The analysis aggregate server, it is additionally operable to obtain timestamp and time letter from the first during cycle Aggregation Table Business datum corresponding to breath is simultaneously sent to the inquiry server；

   The inquiry server, it is additionally operable to export the business datum.
3. A kind of 3. data aggregation method, it is characterised in that including：

   When carrying out data aggregate according to the first during cycle, obtain the data collector from data collector and be currently received Business datum, the business datum carries timestamp, and the timestamp, which is used to identifying the data collector, receives institute State the time of business datum；

   Judge whether the timestamp is more than polymerization corresponding to the first during cycle described in when being currently needed for carrying out data aggregate and touches Time point is sent out, first during cycle corresponds to multiple polymerization triggered time points, and the multiple polymerization triggered time point is right respectively The time answered is incremented by one by one；

   If so, polymerization corresponding to the first during cycle described in during then by the timestamp of preservation to be currently needed for carrying out data aggregate is touched The business datum at hair time point is polymerize.
4. 4. data aggregation method according to claim 3, it is characterised in that described to preserve in the data collector After timestamp is polymerize for the business datum of the polymerization triggered time point, methods described also includes：

   Timestamp corresponding to business datum after polymerization and the business datum is stored in the first during cycle Aggregation Table.
5. 5. data aggregation method according to claim 4, it is characterised in that the business datum by after polymerization and described Timestamp corresponding to business datum is stored in after the step in the first during cycle Aggregation Table, and methods described also includes：

   When needing to carry out data aggregate according to the second during cycle simultaneously, the Data Collection is obtained from the data collector The timestamp for the business datum that device is currently received, second during cycle are more than first during cycle, and described Second during cycle is the integral multiple of first during cycle；

   Judge whether the timestamp reaches polymerization triggered time point, the second preset week corresponding to second during cycle Phase corresponds to multiple polymerization triggered time points, and the time corresponding to the multiple polymerization triggered time point difference is incremented by one by one；

   It is located at if so, then obtaining timestamp from the first during cycle Aggregation Table upper one corresponding to second during cycle Individual polymerization triggered time point is polymerize to the business datum worked as in pre-polymerization triggered time point, and by the business datum after polymerization It is stored in the second during cycle Aggregation Table.
6. 6. data aggregation method according to claim 5, it is characterised in that the business datum by after polymerization is stored in Before step in second during cycle Aggregation Table, methods described also includes：

   It is the business datum setup time stamp after the polymerization to polymerize the triggered time according to corresponding to current second during cycle；

   The business datum by after polymerization, which is stored in the second during cycle Aggregation Table, to be included：

   Timestamp corresponding to business datum after polymerization and the business datum is stored in the second during cycle Aggregation Table.
7. 7. data aggregation method according to claim 3, it is characterised in that described to enter line number according to the first during cycle During according to polymerization, the business datum being currently received from the data collector acquisition data collector includes：

   When carrying out data aggregate according to the first during cycle, from described in obtained in the corresponding preset queue of pre-polymerization task A preset queue is respectively configured for the business datum of each classification in business datum, the data collection server.
8. 8. data aggregation method according to claim 4, it is characterised in that methods described also includes：

   The data query instruction that data query server is sent is received, temporal information is carried in the data query instruction；

   Timestamp business datum corresponding with the temporal information is obtained from the first during cycle Aggregation Table and is sent to The inquiry server, to cause the inquiry server to carry out data display.
9. 9. data aggregation method according to claim 3, it is characterised in that methods described also includes：

   Preset storage position is stored in by triggered time point is polymerize corresponding to first during cycle and current first during cycle Put.
10. 10. data aggregation method according to claim 3, it is characterised in that the timestamp by preservation is described poly- Before the business datum of conjunction triggered time point is polymerize, methods described also includes：

    Judge to be currently received in the preset buffer time section after the business datum, if receive timestamp to be described poly- Close the business datum of triggered time point；

    The timestamp by preservation carries out polymerization for the business datum of the polymerization triggered time point to be included：

    If not receiving, business datum of the timestamp of preservation for the polymerization triggered time point is polymerize；

    It is institute by the business datum received in the preset buffer time section and the timestamp preserved if receiving The business datum for stating polymerization triggered time point is polymerize.
11. 11. data aggregation method according to claim 7, it is characterised in that described to judge whether the timestamp is more than It polymerize corresponding to first during cycle described in when being currently needed for carrying out data aggregate after triggered time point, methods described is also wrapped Include：

    If the timestamp is identical with the polymerization triggered time point, the business datum being currently received is saved in institute State in preset queue corresponding with working as pre-polymerization task；

    If the timestamp is less than the polymerization triggered time point, the business datum being currently received is deleted.
12. A kind of 12. method of data capture, it is characterised in that including：

    Receive business datum to be polymerized；

    Stabbed for the business datum setup time, the timestamp receives the business number for identifying the data collector According to time,

    Gather so as to which aggregate server must be analyzed and obtain the business datum and carry out data according to the timestamp of the business datum Close.
13. A kind of 13. aggregated data querying method, it is characterised in that including：

    Data are sent to analysis aggregate server and report instruction, and the data report in instruction and carry carry out data aggregate in need Business datum so that must analyze aggregate server receives time of the business datum and preset poly- according to data collector Triggered time point is closed, according to during cycle to needing the business datum for carrying out data aggregate to carry out data aggregate；

    The instruction of data aggregate result queries is sent to the analysis aggregate server, is carried in the data query instruction to be checked Ask the temporal information of business datum；

    Receive data aggregate result corresponding with the temporal information and shown.
14. 14. aggregated data querying method according to claim 13, it is characterised in that described to the analysis aggregated service Device, which sends the instruction of data aggregate result queries, to be included：

    By the data-query interfaces that portal server Portal is configured data aggregate knot is sent to the analysis aggregate server Fruit query statement.
15. A kind of 15. data aggregate device, it is characterised in that including：

    Acquiring unit, for when carrying out data aggregate according to the first during cycle, obtaining the data from data collector and receiving The business datum that storage is currently received, the business datum carry timestamp, and the timestamp is used to identify the data Collector receives the time of the business datum；

    Judging unit, whether the timestamp for judging the acquiring unit acquisition, which is more than, is currently needed for carrying out data aggregate Polymerization triggered time point, first during cycle correspond to multiple polymerization triggered times corresponding to the during cycles of Shi Suoshu first Point, time corresponding to the multiple polymerization triggered time point difference are incremented by one by one；

    Polymerized unit, if determining to be more than the first during cycle described in when being currently needed for carrying out data aggregate for the judging unit Corresponding polymerization triggered time point, then business datum of the timestamp of preservation for the polymerization triggered time point is polymerize.
16. A kind of 16. transacter, it is characterised in that including：

    Receiving unit, for receiving business datum to be polymerized；

    Dispensing unit, for being stabbed for the business datum setup time, the timestamp connects for identifying the data collector The time of the business datum is received,

    Gather so as to which aggregate server must be analyzed and obtain the business datum and carry out data according to the timestamp of the business datum Close.
17. A kind of 17. aggregated data inquiry unit, it is characterised in that including：

    Transmitting element, instruction is reported for sending data to analysis aggregate server, the data are reported to carry in instruction and needed The business datum of data aggregate is carried out, so that must analyze aggregate server receives the business datum according to data collector Time and it is preset polymerization triggered time point, according to during cycle to need carry out data aggregate business datum carry out data gather Close；

    The transmitting element, it is additionally operable to send the instruction of data aggregate result queries, the data to the analysis aggregate server The temporal information of business datum to be checked is carried in query statement；

    Receiving unit, for receiving data aggregate result corresponding with the temporal information and being shown.