CN114090366B - Method, device and system for monitoring data - Google Patents
Method, device and system for monitoring dataInfo
- Publication number
- CN114090366B CN114090366B CN202010904700.9A CN202010904700A CN114090366B CN 114090366 B CN114090366 B CN 114090366B CN 202010904700 A CN202010904700 A CN 202010904700A CN 114090366 B CN114090366 B CN 114090366B
- Authority
- CN
- China
- Prior art keywords
- data
- index data
- indicator data
- format
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3024—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a central processing unit [CPU]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3037—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a memory, e.g. virtual memory, cache
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a device and a system for monitoring data, and relates to the technical field of computers. A specific implementation mode of the method comprises the steps that a server receives index data collected by a client, a unified interface is utilized to execute read-write operation on a third-party time sequence database so as to monitor storage and reading of the index data, the defect that an existing system writes the third-party time sequence database in one way is overcome, the efficiency of monitoring the data is improved, the mobility and the utilization rate of the index data are improved, the problem that the relevance of the collected index data is poor due to the fact that a global service label is not standard is solved, and the problem that mass data are high in concurrency is solved by placing the received index data into a message queue.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, and a system for monitoring data.
Background
The Prometheus system provides a complete data monitoring solution as a set of monitoring systems, and is widely deployed and applied in the field of data monitoring due to the ecology openness and the flexibility of multiple components.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
Index data received by the promethaus system is stored on a local storage medium, which makes mass data storage challenging due to storage space limitations. When index data is stored in the third-party database, only the writing of the index data into the third-party database is supported, and the direct reading of the data of the third-party database is not supported, and when a plurality of third-party databases exist, the complexity of monitoring the reading and writing of the third-party databases is increased, and meanwhile, the problems of poor data mobility and low data utilization rate are caused.
The promethaus system acquires index data in a complex and various manner, for example, acquires data of a target data source, acquires index data stored in a data gateway, or acquires index data discovered through service discovery, thereby bringing about the problem of lack of correlation between data caused by nonstandard monitoring data labels and bringing about difficulties for later data correlation analysis. When mass data are collected, the Prometaus system has the defect of high concurrency data collection, the problem of data loss exists in the process of collecting mass data, and meanwhile, the centralized processing of the mass data causes the Prometaus system to have downtime risk.
Disclosure of Invention
In view of the above, the embodiments of the present invention provide a method, an apparatus, and a system for monitoring data, where a server receives index data collected by a client, performs read-write operations on a third party time series database by using a unified interface to monitor storage and reading of the index data, so as to overcome the defect that the existing system unidirectional writes the third party time series database, improve the efficiency of monitoring data, improve the mobility and the utilization rate of the index data, solve the problem of poor relevance of the collected index data due to the non-standardization of a global service tag by determining the global service tag, and solve the problem of high concurrency of mass data by placing the received index data in a message queue.
In order to achieve the above object, according to one aspect of the embodiment of the present invention, there is provided a method for monitoring data, applied to a promethaus system, which is characterized by comprising the steps of receiving index data, determining a global service tag corresponding to the index data according to the category of the index data, adding the global service tag and corresponding content to each index data, forming target index data and storing the target index data in a message queue, acquiring the target index data from the message queue by using a promethaus server, monitoring the target index data according to a set monitoring policy, and storing the target index data in a time sequence database to monitor the storage of the index data, receiving a query request, determining a time sequence database corresponding to the target index data according to the target index data in the query request, and acquiring the target index data from the time sequence database by using the promethaus server to monitor the reading of the index data.
Optionally, the method for monitoring data is characterized in that,
And converting the data format of the index data based on the second format into the first format according to the format rule of the first format.
Optionally, the method for monitoring data is characterized in that,
And when the index data contains non-numerical values, converting the non-numerical values into corresponding numbers according to a predefined correspondence between the non-numerical values and the numbers.
Optionally, the method for monitoring data is characterized in that,
And acquiring a query request of the target index data, and converting operators contained in the query request according to grammar rules of a time sequence database.
Optionally, the method for monitoring data is characterized in that,
Storing the index data into the time sequence database based on a remote process call model, and reading the index data of the time sequence database.
In order to achieve the above object, according to a second aspect of the embodiment of the present invention, there is provided a method for monitoring data, which is characterized by comprising the steps of collecting index data, and sending the index data and the global service tag to a network address according to a set period according to the configured global service tag and the network address.
Optionally, the method for monitoring data is characterized in that,
And acquiring index data by utilizing a data acquisition software package.
Optionally, the method for monitoring data is characterized in that,
And adding a user-defined index by using a registration method contained in the index data acquisition software package, and acquiring index data corresponding to the user-defined index by using the data acquisition software package.
Optionally, the method for monitoring data is characterized in that,
And acquiring index data by using an index data acquisition script.
In order to achieve the above object, according to a third aspect of the embodiments of the present invention, there is provided an apparatus for monitoring data, which is characterized by being applied to a promethaus system, including a data processing module and a data read-write module, wherein,
The data processing module is used for receiving index data, determining a global service tag corresponding to the index data according to the category of the index data, and adding the global service tag and corresponding content into each index data;
The data read-write module is used for acquiring the target index data from the message queue by using a Prometaus server, monitoring the target index data according to a set monitoring strategy, storing the target index data in a time sequence database to monitor the storage of the index data, receiving a query request, determining a time sequence database corresponding to the target index data according to the target index data in the query request, and acquiring the target index data from the time sequence database by using the Prometaus server to monitor the reading of the index data.
Optionally, the means for monitoring data is characterized in that,
And converting the data format of the index data based on the second format into the first format according to the format rule of the first format.
Optionally, the means for monitoring data is characterized in that,
And when the index data contains non-numerical values, converting the non-numerical values into corresponding numbers according to a predefined correspondence between the non-numerical values and the numbers.
Optionally, the means for monitoring data is characterized in that,
And acquiring a query request of the target index data, and converting operators contained in the query request according to grammar rules of a time sequence database.
Optionally, the means for monitoring data is characterized in that,
Storing the index data into the time sequence database based on a remote process call model, and reading the index data of the time sequence database.
In order to achieve the above object, according to a fourth aspect of the embodiments of the present invention, there is provided a device for monitoring data, which is characterized by comprising a data acquisition module, wherein the data acquisition module is configured to acquire index data, and send the index data and the global service tag to a network address according to a configured global service tag and the network address according to a set period.
Optionally, the means for monitoring data is characterized in that,
And acquiring index data by utilizing a data acquisition software package.
Optionally, the means for monitoring data is characterized in that,
And adding a user-defined index by using a registration method contained in the index data acquisition software package, and acquiring index data corresponding to the user-defined index by using the data acquisition software package.
Optionally, the means for monitoring data is characterized in that,
And acquiring index data by using an index data acquisition script.
To achieve the above object, according to a fifth aspect of the embodiment of the present invention, there is provided a system 800 for monitoring data, including the apparatus 600 for monitoring data provided in the above third aspect, and the apparatus 700 for monitoring data provided in the above fourth aspect.
In order to achieve the above object, according to a sixth aspect of the embodiments of the present invention, there is provided an electronic device for monitoring data, which is characterized by comprising one or more processors, and storage means for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method as set forth in any one of the methods for monitoring data described above.
To achieve the above object, according to a seventh aspect of the embodiments of the present invention, there is provided a computer-readable medium having stored thereon a computer program, characterized in that the program, when executed by a processor, implements a method as described in any one of the above methods of monitoring data.
The method has the advantages that the server receives index data acquired by the client, performs read-write operation on the third-party time sequence database by utilizing the unified interface to monitor storage and reading of the index data, overcomes the defect that an existing system unidirectional writes the third-party time sequence database, improves the efficiency of monitoring data, improves the fluidity and the utilization rate of the index data, solves the problem that the relevance of the collected index data is poor due to the fact that the service label is not standard by determining the global service label, and solves the problem of high concurrency of mass data by placing the received index data into the message queue.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a flow chart of a method for monitoring data according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for acquiring index data according to an embodiment of the present invention;
FIG. 3 is a flow chart of monitoring data according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a prior art Prometheus system;
FIG. 5 is a schematic diagram of an improved prior art Prometaus system provided by one embodiment of the present invention;
FIG. 6 is a schematic diagram of an apparatus for monitoring data according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of an apparatus for acquiring index data according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of a system for monitoring data according to an embodiment of the present invention;
FIG. 9 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
fig. 10 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
As shown in fig. 1, an embodiment of the present invention provides a method for monitoring data by a server, where the method may include the following steps:
step S101, receiving index data, determining a global service tag corresponding to the index data according to the category of the index data, and adding the global service tag to each index data;
Specifically, the index data is monitored by the promethaus monitoring system, for example, the index data comprises index data related to a physical machine (such as temperature, hardware fault information of a router, a switch and a server, and the like), system index data running on the physical machine (such as CPU utilization rate, memory utilization rate, hard disk utilization rate, network card flow rate, TCP state, process number, and the like), service index data (such as operation index data of Nginx, tomcat, PHP, mySQL, redis services used by the system), service index data (index data generated by specific service scenes (such as e.g. e-commerce websites, order generation per minute, and the like), and it is understood that the invention is an improvement based on a component framework of the promethaus system, the range of the monitored index data is similar to the data range processed by the promethaus system, and the format of the index data contained by the promethaus system is as follows:
metric{tagk=tagv,tagk1=tagv1,...}value;
the method comprises the steps of determining a metric, wherein the metric is an identification of a monitored index, tagk and tagk1 are names of parameters related to the monitored index, tagv and tagv1 are values corresponding to the parameters tagk and tagk1, and the value is a value corresponding to the metric.
Further, according to the category of the index data, determining a global service tag corresponding to the index data, wherein the category of the index data is to distinguish the source, service type and the like of the index data, for example, index data of different clusters are divided into different categories, index data from different network address ranges are divided into different categories, index data of different services are divided into different categories (for example, index data of logistics and electronic commerce are divided into different categories), further, according to the category of the index data, determining a global service tag corresponding to the index data, for example, determining that the global service tag corresponding to the index data from the cluster1 is SERVICEID = "cluster11111", and the specific content of the category of the index data and the specific content of the global service tag are not limited.
Still further, the global traffic labels are added to each of the index data, e.g. receiving an index data identification finishJobAvgTime, which may originate from a different service, e.g. Hbase, hadoop, spark, etc., by which the source of the collected index data may be distinguished by category and in case the index data identifications are the same. According to the method, the receiving index data are unique and correlated by the global service label, the problem of low data correlation caused by nonstandard index data labels is solved, and the method solves the problems of low data correlation caused by nonstandard index data labels, for example, setting a global service label serviceid= "123456abcdef" corresponding to the index data of HBase service, setting a global service label serviceid= "789010abcdef" corresponding to the index data of Hadoop service, adding the global service label to each index data in a corresponding batch of index data to form target index data, and preferably determining the global service label according to the received global service label according to the category of the index data, or converting the received global service label into the global service label matched with the configuration rule according to the customized global service label configuration rule. Meanwhile, if the client sends the global service tag and other service tags at the same time, the server can also add the other service tags to each index data in a batch of index data.
Further, the target index data is stored in the message queue. Specifically, the existing Prometaus system acquires the data of the target network unit or the data gateway according to a set period, high concurrency data acquisition cannot be completed, and the data volume of related data generated by 4 ten thousand data processing units can be 219 GB/day through pressure test, the generated index volume is 150 hundred million/day, 17.3 ten thousand indexes are generated per second, and the stability of the Prometaus server is possibly reduced due to unscheduled massive data, so that before massive index data are sent to the Prometaus server, the index data are processed by using a message queue (for example, kafka), and the concurrency processing problem of the massive data is solved by using a mechanism of buffering the message queue. For example, one embodiment of the invention pushes the received index data to the message queue by setting the topic partition (for example, setting the topic1 to indicate as the index data related to HBase and setting the topic2 to indicate as the index data related to Hadoop), it can be understood that the topic partition (topic) can isolate the data reading and writing of different services in the message queue (kafka), preferably, the Prometaus server can acquire the index data from the message queue according to the set period or push the index data Prometaus server according to the set period, thereby partially solving the problem of task blocking or poor server stability caused by that massive data is directly pushed to the Prometaus server.
Still further, a data format based on the index data in a second format is converted into the first format according to a format rule of the first format. Specifically, the first format is a data format supported by the existing Prometaus system, the second format is a data format inconsistent with the data format supported by the existing Prometaus system, for example, the second format is any data format which can be converted into the first format, and the specific format of the second format is not limited in the invention.
The data format of the index data may be a data format defined based on the promethaus system (i.e., a first format), examples of which are as follows:
metric { tagk = tagv, tagk 1= tagv1, } value; wherein metric is an identifier of the monitored index, tagk and tagk1 are names of parameters associated with the monitored index, tagv and tagv1 are values corresponding to the parameters tagk and tagk1, value is a value corresponding to the metric, further, the data format of the received index data may be a second format, and the second format is described below by taking the data format defined based on the OpenTSDB database as an example, where the second format is as follows:
[{"metric":"query_info_12345",
"value":"200",
"tags":{
"status":"failed",
"cluster":"AAAA"
}}]
Wherein, metric is the monitored index identifier, in the above example, metric is "query_info_12345", tags contain the name of the parameter associated with the index, for example, parameter 1 is "status" and value is "failed", format the index data based on the second format into the first format according to the format rule of the first format, for example, convert the index data based on the second format into the first format as follows:
query_info_12345{status=”failed”,cluster=”AAAA”}200
it can be understood that, according to the format rule of the first format, the received index data of the second format is correspondingly converted into the first format according to the corresponding label or content, so that the index data of the first format is further monitored and analyzed, the range of the monitored index data is expanded, and the complexity of data processing is reduced due to the unification of the data formats.
Step S102, acquiring the target index data from the message queue by using a Prometaus server, monitoring the target index data according to a set monitoring strategy, storing the target index data in a time sequence database to monitor the storage of the index data, receiving a query request, determining a time sequence database corresponding to the target index data according to the target index data in the query request, and acquiring the target index data from the time sequence database by using the Prometaus server to monitor the reading of the index data.
In particular, the target index data is obtained from the message queue using a Prometaus server (i.e., a monitoring server), preferably the index data is obtained from the message queue at a set period, for example, set to 30 seconds, it being understood that the set period is determined according to the specific service or service scenario, and the frequency and granularity of the user monitoring data. For example, index data having a high real-time requirement may be set for a short period of time, for example, 1 second. The Prometaus server monitors the received index data according to the set monitoring strategy, and the monitoring strategy is set according to the monitored service scene and the index, and can be understood to comprise the steps of setting a trigger alarm rule, setting a threshold value, setting a monitoring period, setting the monitored index data and the like.
Further, as can be seen from the description of step S101, the received data is processed to form target index data with consistent format, and the promethaus server stores the target index data in a time series database to monitor the storage of the index data, wherein, the HBase-based time series database OpenTSDB is used as the time series database, it can be understood that the index data can be stored in a plurality of time series databases according to the category of the index data, each time series database can be a third party database, the problem of the limitation of the capacity of local storage is solved by storing the data in a plurality of third party time series databases, and the storage of the index data in a plurality of time series databases can be monitored through the step.
Further, according to the rules of the time series database (OpenTSDB), index data with a numerical value being non-numerical is not supported to be stored, and then according to the predefined correspondence between the non-numerical value and the numerical value, the value of the index data is converted into the corresponding numerical value. For example, the value "active" indicating the service status is corresponding to the number 1 and "standby" is corresponding to the number 0, i.e. when the index data contains a non-numeric value, the non-numeric value is converted into the corresponding number according to a predefined correspondence between non-numeric and numeric values.
Further, receiving a query request, determining a time sequence database corresponding to target index data according to the target index data in the query request, and acquiring the target index data from the time sequence database to monitor reading of the index data. Specifically, a query request for target index data is received, a time sequence database corresponding to the target index data is determined, and the target index data is acquired, so that the reading of the index data is monitored; compared with the existing Prometaus system, the system supports the monitoring of the reading of the target time sequence database by the Prometaus server, and acquires the index data in the time sequence database according to the request, so that the fluidity and the utilization rate of mass data are increased, and the index data can be analyzed more conveniently to generate index data analysis information. Meanwhile, index data in the time sequence database is directly acquired by using the Prometheus server, so that the efficiency of monitoring and managing the third-party time sequence database is improved, and the defect that the Prometheus server cannot read data from the third-party database in the prior art is overcome.
Because the query grammar of the time series database (OpenTSDB) is inconsistent with the query grammar of the data format of the Prometaus system, for example, the data format of the Prometaus system contains "=", "|=", "=" four operation symbols, "| to" and the data format of OpenTSDB does not contain the operation symbols, the query request of the target index data is acquired and converted according to the grammar rule of the time series database, for example "=" and "|=" in the query request can be converted into a filter condition operator "neutral_or" in the grammar rule in the OpenTSDB format, and the fuzzy matching operation symbols "=" and "|" do not have a matched filter condition operator in the OpenTSDB format, preferably, the return result corresponding to the query request containing the fuzzy matching operation symbol can be acquired first, then the grammar rule of the time series database defined by using the OR operator is converted based on the return result, and further query operation is executed. It will be appreciated that after the Prometaus server obtains the target index data from OpenTSDB, further data calculations and processing are performed in the Prometaus server. That is, a query request of the target index data is acquired, and an operator included in the query request is converted according to a grammar rule of a time sequence database.
Further, the index data is stored in the target database based on a remote process call model, and the index data of the target database is read. Specifically, a monitoring server (Prometaus server) performs read-write operations on a time-series database (e.g., openTSDB) through a remote process call model (e.g., gRPC), and monitors read-write operations on the time-series database, wherein gRPC can define an interface through a structured data serialization method (e.g., protobuf) that can serialize data into binary code and compress the data, thereby reducing the amount of data transmitted and improving the performance of data transmission.
As shown in fig. 2, an embodiment of the present invention provides a method for collecting index data by a client, where the method may include the following steps:
Step S201, index data are collected, and the index data and the global service label are sent to the network address according to a set period according to the configured global service label and the network address.
The client acquires the index data by two methods, wherein the client is a server or a computer used for acquiring the index data, and the specific equipment to which the client belongs is not limited.
The first method is to collect index data by using a data collection software package.
Specifically, one embodiment of the invention collects index data for an application developed by a client using Java language, the client is completed by using a data acquisition software package (for example: java agent. Jar), the client can add the following parameters in Java virtual machine parameters by using the following commands, -javaagent: { Java agent. Jar } = { IP: port }, labels = { SERVICEID: abcd }, file = { a. Yml }, jobName = { abc }, specifically, the description of the parameters for this example is shown in table 1;
Table 1Java class application acquisition index data parameters
The data acquisition software package comprises the following functions of supporting the addition of global service labels through starting parameters and directly pushing index data to a data gateway through configuration of a data gateway network address. Further, the client side scans and collects data through probe technology of Java after the application is started by configuring a global service tag and dynamically configuring a data gateway network address. And transmitting the collected index data to the configured data gateway through the hypertext transfer protocol, further, the data gateway pushes the received index data to a message queue (kafka) through the subject information, and the promethaus server can acquire the index data from the message queue.
Further, when the client needs to send index data based on the self-defined index of the service logic, the data can be reported by using a registration method contained in a data collection software package (java) so as to complete data collection. For example, the custom index metric register (name, METRICTYPE) may be registered by using a method shown in the following, wherein METRICREGI try register is a method used for registration, METRICTYPE is an index type, and for example, five types are included, namely, gauge, counter, meter, timer, and history, wherein Ga uge is used for recording instantaneous values of indexes, such as service current Java virtual machine use conditions including memory utilization, CPU utilization, thread use state, and the like, counter is a Counter, and by increasing and decreasing operations, cumulative indexes, such as a total number of tasks submitted in a cluster, are formed, and Meter is used for counting the occurrence frequency of events, such as counting the latest 1 minute, 5 minutes and 15 minutes of network traffic, is used for counting the aggregation calculation of indexes, and Timer is used for counting the request frequency and time-consuming data of a certain interface, and Histogra m is used for counting the numerical distribution conditions of index data, such as counting the minimum value, maximum value, average value, median, 75 bits, and the like. The client can define the index and put the index in the position of the logic code exception rejection, when the exception occurs, the logic code of the exception rejection part is triggered to send index data corresponding to the index to the data gateway, so that the client can call a registration method contained in the data acquisition software package to acquire index data corresponding to the custom index in the logic code section of the exception rejection part, namely, the user-defined index is increased by using the registration method contained in the index data acquisition software package, and the index data corresponding to the custom index is acquired by using the data acquisition software package.
And the second method is to collect index data by using an index data collection script.
Specifically, for applications developed in non-java languages, the client may collect index data by using an index data collection script, send the index data and the global service tag to a configured network address (for example, the network address of the data gateway) by using a hypertext transfer protocol according to a data format set by the data gateway according to a configured global service tag and network address, and develop the index data collection script in languages such as Phython and Perl.
As shown in fig. 3, an embodiment of the present invention provides a flowchart for monitoring data, and the method may include the following steps:
Step S301, the client collects index data.
Specifically, the description of the client collecting the index data by using the data collecting software package or the data collecting script is consistent with step S201, and will not be described herein. Namely, index data are collected, and the index data and the global service label are sent to the network address according to a set period according to the configured global service label and the network address. Further, index data is collected using a data collection software package. And adding a user-defined index by using a registration method contained in the index data acquisition software package, and acquiring index data corresponding to the user-defined index by using the data acquisition software package. And the index data corresponding to the service identifier is acquired by the index data acquisition script.
Step S302, the data gateway receives index data from the client.
Specifically, a data gateway receives index data, receives the index data, determines a global service tag corresponding to the index data according to the category of the index data, and adds the global service tag to each index data;
The description of the receiving index data and the processing index data is identical to that of step S101, and will not be repeated here.
And step S303, the data gateway puts the target index data into a message queue.
Specifically, the description of the data gateway placing the target index data into the message queue is consistent with step S101, and will not be described herein.
Step S304-step S305, acquiring the target index data from the message queue by using a Prometaus server, monitoring the target index data according to a set monitoring strategy, storing the target index data in a time sequence database to monitor the storage of the index data, receiving a query request, determining a time sequence database corresponding to the target index data according to the target index data in the query request, and acquiring the target index data from the time sequence database by using the Prometaus server to monitor the reading of the index data.
The descriptions of the Prometaus server obtaining data from the message queue and monitoring the storage and reading of the time series database are consistent with step S102, and will not be repeated here.
FIG. 4 shows a schematic diagram of a prior Prometheus system;
FIG. 5 shows a schematic diagram of an improved Prometheus system provided by one embodiment of the present invention;
an embodiment of the present invention is described below by comparing fig. 4 and 5.
1) In the existing Prometaus system, index data are stored locally, and the problem of capacity limitation exists in mass data storage. Although the index data may be stored in the third party database (TSDB as shown in fig. 4), the interaction with the third party database is unidirectional, and only unidirectional writing is supported, that is, the index data stored in the third party database is static, has poor fluidity and low data utilization, so that the problems of data relevance loss and possibly incompatibility of data formats are caused.
In contrast to fig. 4, as shown in fig. 5, the present invention implements bidirectional operation of the promethaus server for storage and reading of time series databases and monitors storage and reading for storage and reading of index data based on monitoring of a plurality of third party time series databases. For example, the promethaus server for monitoring stores and reads index data of a third party database (for example, TSDB as shown in fig. 5, namely OpenTSDB described in the present invention) and monitors stores and reads the third party database, and the promethaus server is utilized to perform read-write operation on the third party database through a unified interface of the promethaus server, so that the efficiency of monitoring the index data is improved, the mobility of the index data and the utilization rate of the index data are improved, and meanwhile, the problem of promethaus performance caused by storing massive data in a local hard disk is solved.
The method comprises the steps of obtaining target index data from the message queue by using a Prometaus server, monitoring the target index data according to a set monitoring strategy, storing the target index data in a time sequence database to monitor storage of the index data, receiving a query request, determining a time sequence database corresponding to the target index data according to the target index data in the query request, and obtaining the target index data from the time sequence database by using the Prometaus server to monitor reading of the index data.
2) As shown in fig. 4, in the existing promethaus system, promethaus periodically acquires index data of a static configuration monitoring target (targets) or a data gateway (Pushgateway), and may also acquire index data of a data container kubernetes (k 8 s) through service discovery (Service discovery). Therefore, the data acquisition mode is complex and various, and the data island (namely, the lack of correlation of data) is caused by the fact that the index data label used for acquisition possibly has an irregular condition, so that the later data correlation analysis is problematic, and the k8s machine is coupled with the monitoring service to a higher degree as shown in fig. 4.
As shown in fig. 5, the promethaus server may periodically acquire data (i.e., target index data) in a message queue, and in contrast to the existing promethaus system, the manner of acquiring data is converted from multiple types to single type and acquired from a message queue (kafka), where the target index data of the message queue (kafka) is obtained by processing the received index data through a data gateway (Pushgateway), and the data gateway determines a global service tag according to the type of the received index data and adds a global service table to fill the global service table into the index data.
The method comprises the steps of receiving index data, determining a global service tag corresponding to the index data according to the category of the index data, and adding the global service tag to each index data;
3) As can be seen from the description of 2), in one embodiment of the present invention, the message queue (kafka) is used to solve the problem of collecting high concurrency mass data, and the technical scheme is that the present invention is based on the newly added components of the existing promethaus system. The message queue caching mechanism can partially solve the problems caused by high throughput and low delay of the monitoring server, and has good fault tolerance under the condition of high concurrency.
As shown in fig. 6, an embodiment of the present invention provides an apparatus 600 for monitoring data, including a data processing module 601 and a data read-write module 602, wherein,
The data processing module 601 is configured to receive index data, determine a global service tag corresponding to the index data according to a category of the index data, and add the global service tag to each index data;
The data read-write module 602 is configured to obtain the target index data from the message queue by using a promethaus server, monitor the target index data according to a set monitoring policy, store the target index data in a time-series database to monitor storage of the index data, receive a query request, determine a time-series database corresponding to the target index data according to the target index data in the query request, and obtain the target index data from the time-series database by using the promethaus server to monitor reading of the index data.
Optionally, the data processing module 601 is further configured to convert a data format of the index data based on the second format into the first format according to a format rule of the first format.
Optionally, the data processing module 601 is further configured to, when the index data includes a non-numeric value, convert the non-numeric value into a corresponding numeric value according to a predefined correspondence between non-numeric values and numeric values.
Optionally, the data read-write module 602 is further configured to obtain a query request of the target index data, and convert an operator included in the query request according to a grammar rule of a time sequence database.
Optionally, the data read-write module 602 is further configured to store the index data into the time-series database based on a remote process call model, and read the index data of the time-series database.
As shown in fig. 7, an embodiment of the present invention provides an apparatus 700 for monitoring data, including a data acquisition module 701, wherein,
The data collection module 701 is configured to collect index data, and send the index data and the global service tag to the network address according to a set period according to the configured global service tag and the network address.
Optionally, the data acquisition module 701 is further configured to acquire the index data by using a data acquisition software package.
Optionally, the data collection module 701 is further configured to increase a custom index by using a registration method included in the index data collection software package, and collect index data corresponding to the custom index by using the data collection software package.
Optionally, the data acquisition module 701 is further configured to acquire the index data by using an index data acquisition script.
As shown in fig. 8, an embodiment of the present invention provides a system for monitoring data, which includes a device for monitoring data as shown in fig. 6, and a device for monitoring data as shown in fig. 7.
The embodiment of the invention also provides electronic equipment for monitoring data, which comprises one or more processors and a storage device, wherein the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the method provided by any embodiment.
The embodiment of the invention also provides a computer readable medium, on which a computer program is stored, which when executed by a processor implements the method provided by any of the above embodiments.
Fig. 9 illustrates an exemplary system architecture 900 of a method of monitoring data or an apparatus of monitoring data to which embodiments of the present invention may be applied.
As shown in fig. 9, system architecture 900 may include terminal devices 901, 902, 903, a network 904, and a server 905. The network 904 is the medium used to provide communications links between the terminal devices 901, 902, 903 and the server 905. The network 904 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 905 over the network 904 using the terminal devices 901, 902, 903 to receive or send messages, etc. Various client applications such as a web browser application, a search class application, an instant messaging tool, a mailbox client, and the like may be installed on the terminal devices 901, 902, 903.
Terminal devices 901, 902, 903 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to servers, smartphones, tablets, laptop and desktop computers, and the like.
The server 905 may be a server providing various services, such as a background management server providing support for data monitoring requests made by users using the terminal devices 901, 902, 903. The background management server can process the received data such as the data monitoring request and store the received index data, and feed back the analysis result of the index data to the terminal equipment.
It should be noted that, the method for monitoring data provided by the embodiment of the present invention is generally executed by the server 905, and accordingly, the device for monitoring data is generally disposed in the server 905.
It should be understood that the number of terminal devices, networks and servers in fig. 9 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 10, there is illustrated a schematic diagram of a computer system 1000 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 10 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 10, the computer system 1000 includes a Central Processing Unit (CPU) 1001, which can execute various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1002 or a program loaded from a storage section 1008 into a Random Access Memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the system 1000 are also stored. The CPU 1001, ROM 1002, and RAM 1003 are connected to each other by a bus 1004. An input/output (I/O) interface 1005 is also connected to bus 1004.
Connected to the I/O interface 1005 are an input section 1006 including a keyboard, a mouse, and the like, an output section 1007 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like, a storage section 1008 including a hard disk, and the like, and a communication section 1009 including a network interface card such as a LAN card, a modem, and the like. The communication section 1009 performs communication processing via a network such as the internet. The drive 1010 is also connected to the I/O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is installed as needed in the drive 1010, so that a computer program read out therefrom is installed as needed in the storage section 1008.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 1009, and/or installed from the removable medium 1011. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 1001.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules and/or units involved in the embodiments of the present invention may be implemented in software, or may be implemented in hardware. The described modules and/or units may also be provided in a processor, for example, a processor may be described as comprising a data processing module, a data reading and writing module, and a data acquisition module. The names of these modules do not limit the module itself in some cases, for example, the data processing module may also be described as "a module that receives index data, converts the data format of the index data into the first format, and adds a global service tag to the index data".
As a further aspect, the invention also provides a computer readable medium which may be comprised in the device described in the above embodiments or may be present alone without being fitted into the device. The computer readable medium carries one or more programs, which when executed by the apparatus, cause the apparatus to include receiving index data, determining a global service tag corresponding to the index data according to a category of the index data, adding the global service tag to each of the index data, forming target index data and storing the target index data in a message queue, acquiring the target index data from the message queue by using a Prometheus server, monitoring the target index data according to a set monitoring policy, and storing the target index data in a time-series database to monitor storage of the index data, receiving a query request, determining a time-series database corresponding to the target index data according to the target index data in the query request, and acquiring the target index data from the time-series database by using the Prometheus server to monitor reading of the index data. And acquiring index data, and sending the index data and the global service label to the network address according to a set period according to the configured global service label and the network address.
According to the technical scheme of the embodiment of the invention, the server receives the index data collected by the client, performs read-write operation on the third-party time sequence database by utilizing the unified interface to monitor storage and reading of the index data, overcomes the defect that the existing system unidirectionally stores the third-party time sequence database, improves the efficiency of monitoring the data, improves the mobility and the utilization rate of the index data, solves the problem of poor relevance of the collected index data due to non-standardization of the service label by determining the global service label, and solves the problem of high concurrency of mass data by putting the received index data into the message queue.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (7)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010904700.9A CN114090366B (en) | 2020-09-01 | 2020-09-01 | Method, device and system for monitoring data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010904700.9A CN114090366B (en) | 2020-09-01 | 2020-09-01 | Method, device and system for monitoring data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114090366A CN114090366A (en) | 2022-02-25 |
| CN114090366B true CN114090366B (en) | 2025-09-12 |
Family
ID=80295782
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010904700.9A Active CN114090366B (en) | 2020-09-01 | 2020-09-01 | Method, device and system for monitoring data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114090366B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114595080A (en) * | 2022-03-02 | 2022-06-07 | 平凯星辰(北京)科技有限公司 | Data processing method, apparatus, electronic device, and computer-readable storage medium |
| CN114911843A (en) * | 2022-05-11 | 2022-08-16 | 中国平安人寿保险股份有限公司 | Method, device, and computer-readable storage medium for reporting business indicators |
| CN115314452B (en) * | 2022-07-11 | 2025-01-24 | 中电通商数字技术(上海)有限公司 | Monitoring indicator collection and storage method, system and storage medium based on message queue |
| CN115460264B (en) * | 2022-08-23 | 2025-01-14 | 曙光信息产业股份有限公司 | Access method and system for target server |
| CN115309612B (en) * | 2022-10-10 | 2023-02-28 | 凯美瑞德（苏州）信息科技股份有限公司 | Method and device for monitoring data |
| CN116303571A (en) * | 2023-03-01 | 2023-06-23 | 杭州网易云音乐科技有限公司 | Data query method, device, equipment and storage medium |
| CN116303804B (en) * | 2023-05-19 | 2023-08-15 | 北京拓普丰联信息科技股份有限公司 | Data comparison method, device, equipment and medium |
| CN117194562B (en) * | 2023-07-25 | 2025-03-07 | 中国人民银行数字货币研究所 | Data synchronization method and device, electronic equipment and computer readable medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107844399A (en) * | 2017-10-10 | 2018-03-27 | 武汉斗鱼网络科技有限公司 | Method, storage medium, electronic equipment and the system of automatic monitoring data storehouse service |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
| CN111553560A (en) * | 2020-04-01 | 2020-08-18 | 车智互联（北京）科技有限公司 | Service index monitoring method, monitoring server and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109726074A (en) * | 2018-08-31 | 2019-05-07 | 网联清算有限公司 | Log processing method, device, computer equipment and storage medium |
-
2020
- 2020-09-01 CN CN202010904700.9A patent/CN114090366B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107844399A (en) * | 2017-10-10 | 2018-03-27 | 武汉斗鱼网络科技有限公司 | Method, storage medium, electronic equipment and the system of automatic monitoring data storehouse service |
| CN109408347A (en) * | 2018-09-28 | 2019-03-01 | 北京九章云极科技有限公司 | A kind of index real-time analyzer and index real-time computing technique |
| CN111553560A (en) * | 2020-04-01 | 2020-08-18 | 车智互联（北京）科技有限公司 | Service index monitoring method, monitoring server and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114090366A (en) | 2022-02-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114090366B (en) | Method, device and system for monitoring data | |
| CN111124819B (en) | Method and device for full link monitoring | |
| WO2021151312A1 (en) | Method for determining inter-service dependency, and related apparatus | |
| CN109039817B (en) | Information processing method, device, equipment and medium for flow monitoring | |
| CN112948138B (en) | A method and device for processing messages | |
| CN110532322B (en) | Operation and maintenance interaction method, system, computer-readable storage medium and device | |
| CN110928934A (en) | Data processing method and device for business analysis | |
| CN111831503B (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN110262951A (en) | A kind of business second grade monitoring method and system, storage medium and client | |
| CN115396523A (en) | Internet of things data processing method, device, equipment, medium and internet of things soft gateway | |
| CN110727563A (en) | Cloud service alarm method and device for preset customer | |
| CN110309028B (en) | Monitoring information acquisition method, service monitoring method, device and system | |
| CN115529271B (en) | Service request distribution method, device, equipment and medium | |
| CN116303761A (en) | Distributed warehousing method and device, electronic equipment and storage medium | |
| CN115048279A (en) | Server processing method and device, electronic equipment and computer readable medium | |
| US10516767B2 (en) | Unifying realtime and static data for presenting over a web service | |
| CN112783924B (en) | Dirty data identification method, device and system | |
| CN118552123A (en) | Method, apparatus, device and computer readable medium for processing asynchronous tasks | |
| CN113722193B (en) | Method and device for detecting page anomaly | |
| CN112749204B (en) | Method and device for reading data | |
| CN113079055B (en) | AGV operation data dynamic acquisition method and device | |
| CN115202973A (en) | Method, apparatus, electronic device and medium for determining application operating state | |
| CN115134419A (en) | Data transmission method, device, equipment and medium | |
| CN108805741B (en) | A method, device and system for fusion of power quality data | |
| CN110852537B (en) | Quality of service detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |