CN113791955B - Data aggregation device and method for monitoring system and server - Google Patents
Data aggregation device and method for monitoring system and server Download PDFInfo
- Publication number
- CN113791955B CN113791955B CN202111093632.3A CN202111093632A CN113791955B CN 113791955 B CN113791955 B CN 113791955B CN 202111093632 A CN202111093632 A CN 202111093632A CN 113791955 B CN113791955 B CN 113791955B
- Authority
- CN
- China
- Prior art keywords
- data
- aggregation
- module
- convergence
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
- G06F11/3082—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by aggregating or compressing the monitored data
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
- G06F16/24556—Aggregation; Duplicate elimination
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Quality & Reliability (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a data aggregation device, a method and a server for a monitoring system, wherein the device comprises: the system comprises a rule engine module, a data convergence processing module and a storage module, wherein: the rule engine module is used for configuring data aggregation rules; the data aggregation processing module comprises a plurality of data aggregation channels, and is used for acquiring data, transmitting the data to the corresponding data aggregation channels, processing the data based on the data aggregation rules configured by the rule engine module and outputting an aggregation result; the data aggregation processing module is also used for judging whether the aggregation result output by the corresponding data aggregation channel has a dependency relationship with the data transmitted to the rest data aggregation channels, and storing the aggregation result to a cache layer or a persistence layer of the storage module based on the judgment result. The invention can adjust the data convergence rule and the data convergence channel conveniently and dynamically, thereby accelerating the convergence processing speed of the data and improving the data convergence processing efficiency.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data aggregation device, method and server for a monitoring system.
Background
In the big data age, as the scale of data center equipment increases and the monitored data volume of the equipment increases with the lapse of time, the diversity of data processing increases, and a convergence dimension is generally required to be increased to revise the data convergence logic, so that the efficiency of data processing is greatly reduced, for example, the time period inquiry of different dimensions such as seconds, minutes, hours, days and the like needs to be supported in a monitoring system, and for time span such as months and years, the data convergence statistical processing of the years and months needs to be carried out again, and the convergence processing of dynamically adding some data dimensions is not convenient.
Disclosure of Invention
In view of this, the invention provides a data aggregation device, a method and a server for a monitoring system, through which data aggregation can be adjusted conveniently and dynamically, data aggregation processing is quickened, and data aggregation efficiency is improved.
Based on the above object, an aspect of the embodiments of the present invention provides a data aggregation device for a monitoring system, where the data aggregation device specifically includes: the system comprises a rule engine module, a data convergence processing module and a storage module, wherein:
the rule engine module is used for configuring data aggregation rules;
the data aggregation processing module comprises a plurality of data aggregation channels, and is used for acquiring data, transmitting the data to the corresponding data aggregation channels, processing the data based on the data aggregation rules configured by the rule engine module and outputting an aggregation result;
the data aggregation processing module is further configured to determine whether a dependency relationship exists between an aggregation result output by the corresponding data aggregation channel and data transmitted to other data aggregation channels, and store the aggregation result to a cache layer or a persistence layer of the storage module based on the determination result.
In some embodiments, the data aggregation processing module includes a judging sub-module, where the judging sub-module is configured to store, in response to a dependency relationship between an aggregation result output by the corresponding data aggregation channel and data transmitted to the other data aggregation channels, the aggregation result to the cache layer.
In some embodiments, the judging submodule is further configured to store the aggregation result to the persistence layer in response to the aggregation result output by the corresponding data aggregation channel having no dependency on the data transmitted to the remaining data aggregation channels.
In some embodiments, the data aggregation apparatus further comprises: the data receiving module is used for receiving the monitoring data, filtering the monitoring data and adding the filtered monitoring data into the buffer queue.
In some embodiments, the data aggregation processing module is configured to obtain the filtered monitoring data from the data receiving module or obtain the aggregation result from the cache layer.
In some embodiments, the data receiving module is configured to configure a data filtering rule, and perform a filtering process on the received monitoring data based on the data filtering rule.
In another aspect of the embodiment of the present invention, there is also provided a data aggregation method for a monitoring system, including:
configuring data aggregation rules based on a rule engine module;
acquiring data based on a data aggregation processing module, transmitting the data to corresponding data aggregation channels in a plurality of data aggregation channels of the data aggregation processing module, processing the data based on data aggregation rules configured by the rule engine module, and outputting an aggregation result;
and judging whether the aggregation result output by the corresponding data aggregation channel and the data transmitted to the other data aggregation channels have a dependency relationship or not based on the data aggregation processing module, and storing the aggregation result to a cache layer or a persistence layer of the storage module based on the judgment result.
In some embodiments, storing the aggregate result to the cache layer or persistence layer based on the determination result comprises:
Responding to the dependency relationship between the convergence result output by the corresponding data convergence channel and the data transmitted to the rest data convergence channels, and storing the convergence result into the cache layer;
and if the convergence result output by the corresponding data convergence channel has no dependency relationship with the data transmitted to the rest data convergence channels, storing the convergence result to the persistence layer.
In some embodiments, the method further comprises:
and receiving monitoring data based on the data receiving module, filtering the monitoring data, and adding the filtered monitoring data into a buffer queue.
In another aspect of the embodiment of the present invention, a server is provided, which includes a data aggregation device for a monitoring system according to the present invention.
The invention has at least the following beneficial technical effects: according to the scheme, when data processing is converged, the data converging rules and the data converging channels can be adjusted conveniently and dynamically, so that the data converging processing is accelerated; and the storage module is divided into a cache layer and a persistence layer, so that the processed data and the aggregation result dependent data are cached, and the persistence storage is carried out on the data aggregation processing result, thereby accelerating the efficiency of the data aggregation processing and improving the data aggregation efficiency.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other embodiments may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data aggregation device for a monitoring system according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data receiving module according to the present invention;
FIG. 3 is a schematic structural diagram of a data aggregation device for a monitoring system according to another embodiment of the present invention;
FIG. 4 is a block diagram of one embodiment of a data aggregation method for a monitoring system provided by the present invention;
fig. 5 is a schematic diagram of an embodiment of a server provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
It should be noted that, in the embodiments of the present invention, all the expressions "first" and "second" are used to distinguish two entities with the same name but different entities or different parameters, and it is noted that the "first" and "second" are only used for convenience of expression, and should not be construed as limiting the embodiments of the present invention, and the following embodiments are not described one by one.
Based on the above object, a first aspect of the embodiments of the present invention proposes a data aggregation device for a monitoring system. As shown in fig. 1, the data aggregation device specifically includes: a rules engine module 110, a data aggregation processing module 120, and a storage module 130, wherein:
the rule engine module 110 is configured to configure data aggregation rules.
By setting the convergence channels, different dimensionalities of data can be processed, and the processing results of a plurality of different convergence channels can be combined to perform cooperative data processing;
The data aggregation processing module 120 includes a plurality of data aggregation channels, and the data aggregation processing module 120 is configured to obtain data, transmit the data to a corresponding data aggregation channel, process the data based on a data aggregation rule configured by the rule engine module 110, and output an aggregation result.
By setting the convergence channel, different dimensions of the data can be processed.
The data aggregation processing module 120 is further configured to determine whether the aggregation result output by the corresponding data aggregation channel has a dependency relationship with the data transmitted to the other data aggregation channels, and store the aggregation result to the cache layer or the persistence layer of the storage module 130 based on the determination result.
The storage module is divided into the cache layer and the persistence layer, so that the processed data and the aggregation result dependent data are cached, and the persistence storage is carried out on the data aggregation processing result, so that the efficiency of the data aggregation processing is accelerated.
For a better understanding of embodiments of the present invention, a data aggregation scenario is illustrated by CPU usage as follows.
Configuring a data aggregation Rule by the Rule engine module 110, wherein the Rule is to average 60 times of input data and is named Rule1;
two data convergence channels are created by the data convergence processing module 120.
The data convergence channel 1 is named as Pipeline1, data input is set as 1 parameter, the parameter value is CPU usage performance data of the receiving module, and the convergence Rule is associated with a convergence Rule named as Rule 1;
the data convergence channel 2 is named as Pipeline2, data input is set to be 1 parameter, the parameter value is the output value of Pipeline1, and the association convergence Rule is the convergence Rule of Rule 1.
After the data aggregation module 120 obtains the CPU usage data, the CPU usage data is transmitted to the data aggregation channel named Pipeline1, and the aggregation Rule1 is associated by the Rule engine module 110, so that after 60 times of transmission of the CPU usage data to the data aggregation channel, the data aggregation channel obtains and outputs an aggregation result.
In the second-level monitoring system, the reporting frequency of data is 1 second for 1 time, and the average CPU utilization rate of the minute level can be obtained through the above aggregation configuration.
The data aggregation processing module 120 determines that the aggregation output result of the data aggregation channel 1 is the dependent data of the data aggregation channel 2, so that the aggregation result is stored in the buffer layer of the storage module 130 for standby.
The data aggregation module 120 obtains the average usage rate of the CPU at the minute level output by the data aggregation channel 1, and transmits the average usage rate to the data aggregation channel named Pipeline2, and the Rule engine module 110 associates the aggregation Rule1, so that after transmitting the CPU usage rate data at the minute level 60 times to the data aggregation channel 2, the data aggregation channel 2 obtains and outputs an aggregation result, and the average usage rate aggregation result of the CPU at the hour level is obtained.
Two sources of data acquired by the data aggregation module 120 are available, one being CPU usage data acquired from the monitoring system, for example, the CPU usage data transmitted to the data aggregation channel Pipeline1 in the above example; one is the aggregate result output for the other data aggregate channel, e.g., the average of 60 CPU utilization data output by the data aggregate channel Pipeline1 in the above example.
The aggregation result with the dependency relationship can be cached to the cache layer of the storage module 130 first, and is obtained from the cache layer when waiting for the data aggregation processing module 120 to be used; and the data can be directly output to a corresponding data convergence channel, and the data convergence channel can be used after triggering the associated rule.
According to the embodiment of the invention, when data processing is converged, the data converging rules and the data converging channels can be adjusted conveniently and dynamically, so that the data converging processing is accelerated, and the data converging efficiency is improved.
The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions described herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP and/or any other such configuration.
In some embodiments, the data aggregation processing module includes a judging sub-module, where the judging sub-module is configured to store, in response to a dependency relationship between an aggregation result output by the corresponding data aggregation channel and data transmitted to the other data aggregation channels, the aggregation result to the cache layer.
In some embodiments, the judging submodule is further configured to store the aggregation result to the persistence layer in response to the aggregation result output by the corresponding data aggregation channel having no dependency on the data transmitted to the remaining data aggregation channels.
The storage module for storing data generally selects a database, if the data is stored in the database after finishing the data arrangement, the data is queried from the database when the data is needed to be used, and the time for querying the data is needed, especially the time for acquiring the data is increased when the data amount stored in the database is large or the data query operation is performed frequently, and the data is reduced. Therefore, the storage modules are layered, two databases are established, one database is used as a cache layer of the cache module, and the other database is used as a persistence layer of the storage module. The data to be used by the caching layer or the convergence result is utilized to change the layer, so that the inquiry of a database can be reduced, the acquisition of the data is quickened, and the data convergence efficiency is improved.
In some embodiments, the data aggregation apparatus further comprises: the data receiving module is used for receiving the monitoring data, filtering the monitoring data and adding the filtered monitoring data into the buffer queue.
In some embodiments, the data aggregation processing module is configured to obtain the filtered monitoring data from the data receiving module or obtain the aggregation result from the cache layer.
In some embodiments, the data receiving module is configured to configure a data filtering rule, and perform a filtering process on the received monitoring data based on the data filtering rule.
As shown in fig. 2, a schematic diagram of the structure of the data receiving module is provided.
The data receiving module receives monitoring data of hardware equipment (such as CPU, hard disk, fan and other server-mounted hardware) through a receiving interface, invokes a data filtering rule through a filter, filters invalid data which are abnormal and do not accord with a processing format, and inputs the filtered monitoring data into a queue to buffer the data.
The data filtering rules may include: a threshold range rule, filtering out the numerical data which is not in the range through the threshold range rule; format validity rules by which data of a band unit is filtered, specified symbols are filtered, and the like.
The filtering processing is carried out on different types of data through the configured data filtering rule, and the filtering function is to carry out the pre-data checking processing to prevent invalid data in abnormal and inconsistent processing formats from being carried out in a later data aggregation module; and the data is buffered through the queue, so that instantaneous high concurrent data reception can be prevented, and the availability of the data receiving module is improved.
Embodiments of the present invention will be described below by way of specific examples.
Fig. 3 is a schematic diagram of a further embodiment of a data aggregation device for a monitoring system.
The data aggregation device comprises: the system comprises a data receiving module, a data aggregation processing module, a rule engine module and a storage module, wherein the storage module comprises a caching layer and a persistence layer.
The data receiving module receives the monitoring data based on the data receiving module, filters the monitoring data, and adds the filtered monitoring data into the buffer queue.
The data aggregation processing module acquires monitoring data from the cache queue and transmits the monitoring data to the corresponding data aggregation channel, and the data aggregation channel processes the monitoring data based on the data aggregation rule configured by the rule engine module and outputs an aggregation result;
and judging whether the aggregation result output by the corresponding data aggregation channel and the data transmitted to the other data aggregation channels have a dependency relationship or not based on the data aggregation processing module, and storing the aggregation result to a cache layer or a persistence layer of the storage module based on the judgment result.
According to the embodiment of the invention, when data processing is converged, the data converging rules and the data converging channels can be adjusted conveniently and dynamically, so that the data converging processing is accelerated; and the storage module is divided into a cache layer and a persistence layer, so that the processed data and the aggregation result dependent data are cached, and the persistence storage is carried out on the data aggregation processing result, thereby accelerating the efficiency of the data aggregation processing and improving the data aggregation efficiency.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 4, an embodiment of the present invention further provides a data aggregation method for a monitoring system, including:
Step S101, a data aggregation rule is configured based on a rule engine module.
Step S103, acquiring data based on a data convergence processing module, transmitting the data to corresponding data convergence channels in a plurality of data convergence channels of the data convergence processing module, processing the data based on a data convergence rule configured by the rule engine module, and outputting a convergence result;
step 105, based on the data convergence processing module, judging whether the convergence result output by the corresponding data convergence channel has a dependency relationship with the data transmitted to the rest data convergence channels, and based on the judgment result, storing the convergence result into a cache layer or a persistence layer of the storage module.
Those skilled in the art will appreciate that implementing all or part of the above-described methods in accordance with the embodiments may be accomplished by a computer program for instructing relevant hardware, where the program may be stored on a computer readable storage medium, and where the program, when executed, may comprise the steps of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (RAM), or the like. Embodiments of the computer program may achieve the same or similar effects as any of the method embodiments previously described.
The steps of a method or algorithm described in connection with the disclosure herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
In some embodiments, storing the aggregate result to the cache layer or persistence layer based on the determination result comprises:
Responding to the dependency relationship between the convergence result output by the corresponding data convergence channel and the data transmitted to the rest data convergence channels, and storing the convergence result into the cache layer;
and if the convergence result output by the corresponding data convergence channel has no dependency relationship with the data transmitted to the rest data convergence channels, storing the convergence result to the persistence layer.
In some embodiments, the method further comprises:
and receiving monitoring data based on the data receiving module, filtering the monitoring data, and adding the filtered monitoring data into a buffer queue.
Embodiments of the invention may also include corresponding computer devices. The computer device comprises a memory, at least one processor, and a computer program stored on the memory and executable on the processor, the processor executing the method according to the invention.
In a second aspect of the embodiment of the present invention, a server 11 is provided. Fig. 5 is a schematic diagram of an embodiment of a server provided by the present invention. As shown in fig. 5, the server 11 includes a data aggregation device 12 for a monitoring system as follows.
The data aggregation device 12 specifically includes: the system comprises a rule engine module, a data convergence processing module and a storage module, wherein:
the rule engine module is used for configuring data aggregation rules;
the data aggregation processing module comprises a plurality of data aggregation channels, and is used for acquiring data, transmitting the data to the corresponding data aggregation channels, processing the data based on the data aggregation rules configured by the rule engine module and outputting an aggregation result;
the data aggregation processing module is further configured to determine whether a dependency relationship exists between an aggregation result output by the corresponding data aggregation channel and data transmitted to other data aggregation channels, and store the aggregation result to a cache layer or a persistence layer of the storage module based on the determination result.
In some embodiments, the data aggregation processing module includes a judging sub-module, where the judging sub-module is configured to store, in response to a dependency relationship between an aggregation result output by the corresponding data aggregation channel and data transmitted to the other data aggregation channels, the aggregation result to the cache layer.
In some embodiments, the judging submodule is further configured to store the aggregation result to the persistence layer in response to the aggregation result output by the corresponding data aggregation channel having no dependency on the data transmitted to the remaining data aggregation channels.
In some embodiments, the data aggregation apparatus further comprises: the data receiving module is used for receiving the monitoring data, filtering the monitoring data and adding the filtered monitoring data into the buffer queue.
In some embodiments, the data aggregation processing module is configured to obtain the filtered monitoring data from the data receiving module or obtain the aggregation result from the cache layer.
In some embodiments, the data receiving module is configured to configure a data filtering rule, and perform a filtering process on the received monitoring data based on the data filtering rule.
In the context of the present application, the memory is used as a non-volatile computer readable storage medium, and may be used to store a non-volatile software program, a non-volatile computer executable program, and modules, such as program instructions/modules corresponding to the network resource coordination method in the embodiments of the present application. The processor executes various functional applications of the device and data processing by running non-volatile software programs, instructions and modules stored in the memory, i.e. implements the data aggregation method of the above method embodiments.
The memory may include a memory program area and a memory data area, wherein the memory program area may store an operating system, at least one application program required for a function; the storage data area may store data created according to the use of the device, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory optionally includes memory remotely located relative to the processor, the remote memory being connectable to the local module through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Finally, it should be noted that, as will be appreciated by those skilled in the art, all or part of the procedures in implementing the methods of the embodiments described above may be implemented by a computer program for instructing relevant hardware, and the program may be stored in a computer readable storage medium, and the program may include the procedures of the embodiments of the methods described above when executed. The storage medium of the program may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (RAM), or the like. The computer program embodiments described above may achieve the same or similar effects as any of the method embodiments described above.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The foregoing embodiment of the present invention has been disclosed with reference to the number of embodiments for the purpose of description only, and does not represent the advantages or disadvantages of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, and the program may be stored in a computer readable storage medium, where the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will appreciate that: the above discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the disclosure of embodiments of the invention, including the claims, is limited to such examples; combinations of features of the above embodiments or in different embodiments are also possible within the idea of an embodiment of the invention, and many other variations of the different aspects of the embodiments of the invention as described above exist, which are not provided in detail for the sake of brevity. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the embodiments should be included in the protection scope of the embodiments of the present invention.
Claims (7)
1. A data aggregation device for a monitoring system, comprising: the system comprises a rule engine module, a data convergence processing module and a storage module, wherein:
the rule engine module is used for configuring data aggregation rules;
the data aggregation processing module comprises a plurality of data aggregation channels, and is used for acquiring data, transmitting the data to the corresponding data aggregation channels, processing the data based on the data aggregation rules configured by the rule engine module and outputting an aggregation result;
The data aggregation processing module is further used for judging whether the aggregation result output by the corresponding data aggregation channel and the data transmitted to the other data aggregation channels have a dependency relationship, and storing the aggregation result to a cache layer or a persistence layer of the storage module based on the judgment result;
the data aggregation processing module comprises a judging sub-module, wherein the judging sub-module is used for responding to the dependency relationship between the aggregation result output by the corresponding data aggregation channel and the data transmitted to the rest data aggregation channels, and storing the aggregation result into the cache layer;
and the judging submodule is further used for storing the convergence result to the persistence layer in response to the fact that the convergence result output by the corresponding data convergence channel has no dependency relationship with the data transmitted to the rest data convergence channels.
2. The apparatus as recited in claim 1, further comprising: the data receiving module is used for receiving the monitoring data, filtering the monitoring data and adding the filtered monitoring data into the buffer queue.
3. The apparatus of claim 2, wherein the data aggregation processing module is configured to obtain the filtered monitoring data from the data receiving module or the aggregate result from the cache layer.
4. The apparatus of claim 2, wherein the data receiving module is configured to configure data filtering rules and to filter the received monitoring data based on the data filtering rules.
5. A data aggregation method for a monitoring system, comprising:
configuring data aggregation rules based on a rule engine module;
acquiring data based on a data aggregation processing module, transmitting the data to corresponding data aggregation channels in a plurality of data aggregation channels of the data aggregation processing module, processing the data based on data aggregation rules configured by the rule engine module, and outputting an aggregation result;
judging whether the aggregation result output by the corresponding data aggregation channel and the data transmitted to the other data aggregation channels have a dependency relationship or not based on a data aggregation processing module, and storing the aggregation result to a cache layer or a persistence layer of a storage module based on the judgment result;
Storing the aggregate result to the cache layer or persistence layer based on the determination result includes:
Responding to the dependency relationship between the convergence result output by the corresponding data convergence channel and the data transmitted to the rest data convergence channels, and storing the convergence result into the cache layer;
and if the convergence result output by the corresponding data convergence channel has no dependency relationship with the data transmitted to the rest data convergence channels, storing the convergence result to the persistence layer.
6. The method as recited in claim 5, further comprising:
and receiving monitoring data based on the data receiving module, filtering the monitoring data, and adding the filtered monitoring data into a buffer queue.
7. A server comprising a data aggregation device for a monitoring system according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111093632.3A CN113791955B (en) | 2021-09-17 | 2021-09-17 | Data aggregation device and method for monitoring system and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111093632.3A CN113791955B (en) | 2021-09-17 | 2021-09-17 | Data aggregation device and method for monitoring system and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113791955A CN113791955A (en) | 2021-12-14 |
| CN113791955B true CN113791955B (en) | 2024-09-17 |
Family
ID=78878806
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111093632.3A Active CN113791955B (en) | 2021-09-17 | 2021-09-17 | Data aggregation device and method for monitoring system and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113791955B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115037729B (en) * | 2022-04-21 | 2024-05-28 | 中国建设银行股份有限公司 | Data aggregation method, device, electronic equipment and computer readable medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101491055A (en) * | 2006-07-17 | 2009-07-22 | 国际商业机器公司 | Dispatching request fragments from a response aggregating surrogate |
| CN102209118A (en) * | 2011-06-10 | 2011-10-05 | 成都勤智数码科技有限公司 | Distributed mass data gathering method |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103810258B (en) * | 2014-01-26 | 2017-06-30 | 杭州东方通信软件技术有限公司 | Convergence dispatching method based on data warehouse |
| CN108804347A (en) * | 2017-05-05 | 2018-11-13 | 华中科技大学 | A kind of cache layer, collecting system and method for industrial big data convergence |
| CN111064591B (en) * | 2018-10-16 | 2021-03-26 | 杭州海康威视数字技术股份有限公司 | Data aggregation method, apparatus, device, storage medium and system |
| JP7025365B2 (en) * | 2019-03-25 | 2022-02-24 | 株式会社日立製作所 | Data linkage management method, data linkage management system, and node |
| CN111460495B (en) * | 2020-03-27 | 2023-06-23 | 北京锐安科技有限公司 | A data classification management system and method |
| CN111858739A (en) * | 2020-08-04 | 2020-10-30 | 浪潮卓数大数据产业发展有限公司 | Mapreduce-based data aggregation method and system |
-
2021
- 2021-09-17 CN CN202111093632.3A patent/CN113791955B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101491055A (en) * | 2006-07-17 | 2009-07-22 | 国际商业机器公司 | Dispatching request fragments from a response aggregating surrogate |
| CN102209118A (en) * | 2011-06-10 | 2011-10-05 | 成都勤智数码科技有限公司 | Distributed mass data gathering method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113791955A (en) | 2021-12-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110858843B (en) | Service request processing method and device and computer readable storage medium | |
| CN109743295B (en) | Access threshold adjusting method and device, computer equipment and storage medium | |
| CN111538572B (en) | Task processing method, device, scheduling server and medium | |
| CN113791955B (en) | Data aggregation device and method for monitoring system and server | |
| CN111641563B (en) | Traffic adaptive method and system based on distributed scenario | |
| CN109561403A (en) | A kind of short message dispatching method, equipment and storage medium | |
| CN112383585A (en) | Message processing system and method and electronic equipment | |
| CN110661717A (en) | Current limiting method and device and electronic equipment | |
| CN112073329B (en) | Distributed current limiting method and device, electronic equipment and storage medium | |
| CN115062035A (en) | Data processing method, electronic device and computer readable storage medium | |
| CN113835905B (en) | Message queue load balancing method and device, electronic equipment and medium | |
| CN109885384B (en) | Task parallelism optimization method and device, computer equipment and storage medium | |
| CN109862052A (en) | Invoking method, device, computer equipment and storage medium for service request | |
| US11265250B2 (en) | Targeted rate limiting of tenant systems in online services | |
| CN119232662A (en) | Flow transmission control method, device, electronic equipment, medium and program product | |
| CN109508433B (en) | Load fluctuation coping method and system based on matching algorithm performance adjustment | |
| CN109308219B (en) | Task processing method and device and distributed computer system | |
| CN117675866A (en) | Data processing methods, devices, equipment and media based on Bayesian inference | |
| CN115934313A (en) | Method, device, equipment, medium and product for processing service request | |
| CN114254001A (en) | Query request response method and device, storage medium and electronic device | |
| CN113535643B (en) | Data processing method, device and server | |
| CN111586822B (en) | Adaptive rate adjustment method, medium, terminal and device for communication of Internet of things | |
| CN110838888A (en) | Data processing method, device, system and computer readable storage medium | |
| CN113542012B (en) | Fault detection method, fault detection device and electronic equipment | |
| CN111193760A (en) | Information sending method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |