CN119271483A

CN119271483A - Database monitoring method, device, computer equipment, and storage medium

Info

Publication number: CN119271483A
Application number: CN202411117540.8A
Authority: CN
Inventors: 丁卫民; 管正伟; 王歆; 陈亮
Original assignee: Pujiang Kunying Technology Co ltd
Current assignee: Pujiang Kunying Technology Co ltd
Priority date: 2024-08-15
Filing date: 2024-08-15
Publication date: 2025-01-07

Abstract

The present application relates to the field of big data technology, and in particular, to a database monitoring method, apparatus, computer device, storage medium, and computer program product. The method comprises the steps of responding to a data request aiming at a target service, creating tracking identification information associated with the data request, determining a target execution statement aiming at the data request in a target database based on the tracking identification information, obtaining log data generated in the execution of the target execution statement, wherein the log data are used for recording parameter information when the target database executes the target execution statement, monitoring the abnormal condition of the target database based on the log data, and judging the abnormal condition according to preset monitoring logic. The method can improve the universality and the controllability of database monitoring and improve the monitoring efficiency.

Description

Database monitoring method, device, computer equipment and storage medium

Technical Field

The present application relates to the field of big data technology, and in particular, to a database monitoring method, apparatus, computer device, storage medium, and computer program product.

Background

A Database (Database) is an organized, structured collection of data used in a system for storing and managing related data. The database system is a key component for managing and storing data in the computer system, and the data is stored by structuring, so that the data is more efficient and convenient to access, manage and use. Databases are typically composed of one or more data tables, each of which contains a plurality of data records, each record being composed of a plurality of fields. Databases employ different data models (e.g., relational, document, graphic, etc.) to organize data, common database systems include relational databases (e.g., mySQL, oracle, SQL SERVER), noSQL databases (e.g., mongoDB, redis), and the like. Database operation (Database Operations) refers to the management and maintenance work of a database system, which aims to ensure safe, stable and efficient operation of the database system.

In the related art, database operation and maintenance mainly comprises backup and recovery, performance optimization, security management, capacity planning, version upgrading and patch management, fault elimination, monitoring alarm and the like. The monitoring alarm is usually dependent on a database monitoring system, the running state and performance index of the database are monitored regularly, reasonable alarm rules are set, and abnormal conditions are responded timely.

However, the existing database monitoring method has the following technical problems:

the current database monitoring platform depends on the support and specific configuration of the database, so that the universality and controllability of the monitoring platform are poor.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a database monitoring method, apparatus, computer device, computer readable storage medium, and computer program product that can improve the versatility and controllability of database monitoring and improve the monitoring efficiency.

In a first aspect, the present application provides a database monitoring method. The method comprises the following steps:

Creating tracking identification information associated with a data request for a target service in response to the data request;

determining a target execution statement executed for the data request in the target database based on the tracking identification information;

acquiring log data generated in the execution of the target execution statement, wherein the log data is used for recording parameter information when the target database executes the target execution statement;

and monitoring the abnormal situation of the target database based on the log data, wherein the abnormal situation is judged according to a preset monitoring logic.

In one embodiment, the monitoring of the abnormal situation of the target database based on the log data, before the abnormal situation is determined according to a preset monitoring logic, further includes:

Carrying out graphical processing on the log data to obtain a log chart;

And monitoring the abnormal condition based on the log chart.

In one embodiment, the performing the graphical processing on the log data to obtain a log chart includes:

Confirming a target field to be monitored in the log data, and creating an index between the target field and the tracking identification information;

And filling the target field into an initial chart matched with the target field to obtain the log chart.

In one embodiment, the obtaining the log data generated in the execution of the target execution statement, where the log data is used to record parameter information when the target database executes the target execution statement includes:

And acquiring the log data output by the target database based on a preset log output frame layer, wherein the log output frame layer and a service processing layer of the target service are mutually independent.

In one embodiment, the method further comprises:

Determining the target execution statement associated with the data request in a plurality of databases based on the tracking identification information;

And acquiring the log data output by a plurality of databases based on the log output framework layer.

In one embodiment, the obtaining the log data generated in the execution of the target execution statement includes:

acquiring a business method and stack information in the execution of the target execution statement;

And acquiring the log data based on the business method and the stack information, wherein the log data comprises a slow query log, a transaction log and a deadlock log.

In a second aspect, the application further provides a database monitoring device. The device comprises:

A request response module for responding to a data request for a target service and creating tracking identification information associated with the data request;

The statement positioning module is used for determining a target execution statement which is executed for the data request in the target database based on the tracking identification information;

the log data module is used for acquiring log data generated in the execution of the target execution statement, and the log data is used for recording parameter information when the target database executes the target execution statement;

And the abnormal monitoring module is used for monitoring abnormal conditions of the target database based on the log data, and the abnormal conditions are judged according to preset monitoring logic.

In one embodiment, before the anomaly monitoring module, the anomaly monitoring module further includes:

the imaging module is used for carrying out imaging processing on the log data to obtain a log chart;

And the chart monitoring module is used for monitoring the abnormal condition based on the log chart.

In one embodiment, the patterning module includes:

The index module is used for confirming a target field to be monitored in the log data, and creating an index between the target field and the tracking identification information;

And the chart filling module is used for filling the target field into an initial chart matched with the target field to obtain the log chart.

In one embodiment, the log data module includes:

And the output frame layer module is used for acquiring the log data output by the target database based on a preset log output frame layer, and the log output frame layer and the service processing layer of the target service are mutually independent.

In one embodiment, the apparatus further comprises:

a multi-database module for determining the target execution statement associated with the data request in a plurality of databases based on the tracking identification information;

And the data acquisition module is used for acquiring the log data output by the databases based on the log output framework layer.

In one embodiment, the log data module includes:

The business information module is used for acquiring a business method and stack information in the execution of the target execution statement;

And the log generation module is used for acquiring the log data based on the business method and the stack information, wherein the log data comprises a slow query log, a transaction log and a deadlock log.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of a database monitoring method according to any one of the embodiments of the first aspect when the processor executes the computer program.

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of a database monitoring method according to any one of the embodiments of the first aspect.

In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of a database monitoring method according to any one of the embodiments of the first aspect.

The database monitoring method, the database monitoring device, the database monitoring computer equipment, the database monitoring storage medium and the database monitoring computer program product can achieve the beneficial effects corresponding to the technical problems in the background art through the technical characteristics in the claims:

The application provides a database monitoring method, which comprises the steps of responding to a data request aiming at a target service, creating tracking identification information related to the data request, determining a target execution statement aiming at the data request in a target database based on the tracking identification information, acquiring log data generated in the execution of the target execution statement, wherein the log data is used for recording parameter information when the target database executes the target execution statement, and monitoring abnormal conditions of the target database based on the log data, wherein the abnormal conditions are judged according to preset monitoring logic. In implementation, the trace identification information is associated with the data request of the heap target service, so that the execution statement associated with the data request in the database can be marked in the request processing, then the parameter information generated in the execution process of the execution statement can be recorded to form log data, and finally the running state of the database is monitored depending on the log data. Therefore, an information link from a user request to service execution to database execution can be established, the database change related to the request can be intuitively and accurately found, targeted monitoring can be performed, the accuracy of a monitoring result can be improved, different databases can be flexibly adapted, and the flexibility of a monitoring system is improved.

Drawings

FIG. 1 is a diagram of an application environment for a database monitoring method in one embodiment;

FIG. 2 is a first flow chart of a database monitoring method according to one embodiment;

FIG. 3 is a second flow chart of a database monitoring method according to another embodiment;

FIG. 4 is a third flow chart of a database monitoring method according to another embodiment;

FIG. 5 is a fourth flowchart of a database monitoring method according to another embodiment;

FIG. 6 is a fifth flowchart of a database monitoring method according to another embodiment;

FIG. 7 is a sixth flowchart of a database monitoring method according to another embodiment;

FIG. 8 is a block diagram of a database monitoring apparatus in one embodiment;

Fig. 9 is an internal structural diagram of a computer device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

The database monitoring method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers.

In one embodiment, as shown in fig. 2, a database monitoring method is provided, and the method is applied to the terminal in fig. 1 for illustration, and includes the following steps:

Step 202, in response to a data request for a target service, creating tracking identification information associated with the data request.

A service may refer to content in a system that is related to a specific service requirement, function, and flow. A data request may refer to a request initiated when specific data needs to be invoked in order to achieve the flow of the target service. Tracking identification information may refer to a unique identifier used in a distributed system to track the condition of requests passing between different services. Since a request in the system needs to be completed through different service nodes, tracking identification information can be used for request tracking and monitoring across services. The tracking identifier may be a Trace ID.

For example, the terminal may create tracking identification information associated with a data request in response to the data request for the target service. Specifically, when a request is initiated, the first service generates a unique TraceID and passes it to downstream services. Each service will carry TraceID in the request header or context during processing of the request to the next service, thus forming a trace of the entire request link. In this way, the terminal can help developers and operators track the path of the request in the system, analyze the processing procedure and performance bottleneck of the request.

Specifically, traceID typically contains a globally unique identifier, which identifies the uniqueness of a particular request, and optionally information such as parent SpanID (span) and Sampled (sample identification). When a large request link is broken down into multiple small spans, each Span carries the same TraceID to associate them together. Through analysis TraceID, developers can know information such as trends of requests in the system, processing time of each service, service with problems and the like, and help to quickly locate and solve the problems. TraceID are also key parameters commonly used in monitoring systems and performance analysis tools.

And 204, determining a target execution statement executed for the data request in the target database based on the tracking identification information.

The execution statement may be a program statement for operating data in the database, may be SQL (Structured Query Language), etc., and the standardized language of the relational database system may be managed by the execution statement, so as to implement operations such as querying, inserting, updating, deleting, etc. on the database.

For example, the terminal may determine a target execution statement in the target database to be executed for the data request based on the tracking identification information.

And 206, acquiring log data generated in the execution of the target execution statement.

The log data may be used to record parameter information when the target database executes the target execution statement. In particular, the log data may record events, activities, or information occurring at the time of execution of the statement by the target in the database. The log data may be stored in the form of text files, data records, log servers, and the like.

For example, the terminal may acquire log data generated in execution of the target execution statement.

And step 208, monitoring the abnormal condition of the target database based on the log data, wherein the abnormal condition is judged according to a preset monitoring logic.

The abnormal condition may be an abnormal condition existing in the operation and the processing of the database, which may cause problems such as damaged data integrity, operation failure, and system crash in the database. Specific exception conditions may include unique constraint violations, null constraint violations, out of constraint ranges, deadlocks, connection timeouts, space starvation, file corruption, and the like. The monitoring logic may refer to a discrimination rule for discriminating an abnormal condition in the database, and the monitoring logic may be preset according to the performance parameters of the database and the abnormal condition.

The terminal may monitor the target database for an abnormal condition based on the log data, where the abnormal condition is determined according to a preset monitoring logic.

In the database monitoring method, the technical characteristics in the embodiment are combined to carry out reasonable deduction, so that the following beneficial effects of solving the technical problems in the background technology can be realized:

In one embodiment, as shown in fig. 3, before the step 208, the method further includes:

and 302, carrying out graphical processing on the log data to obtain a log chart.

The graphic processing can refer to a process of displaying and visualizing the data in the form of a chart and a graph, and the data can be more intuitively understood and the association, trend and mode among the data can be found through the data graphic, so that the efficiency of data analysis and decision making is improved. In particular, the patterning may include processing the data into a line graph, a bar graph, a pie graph, a scatter graph, a thermodynamic diagram, or the like.

For example, the terminal may perform a graphical process on the log data to obtain a log chart.

And 304, monitoring the abnormal condition based on the log chart.

For example, the terminal may monitor for an abnormal situation based on the log chart after the patterning.

In the embodiment, after the log data is obtained, the log data is subjected to graphical processing to obtain a log chart, so that the state of the database can be visually monitored, and the intuitiveness of checking the log data is improved.

In one embodiment, as shown in fig. 4, the step 302 may include:

and step 402, confirming a target field to be monitored in the log data, and creating an index between the target field and the tracking identification information.

The target field may refer to a parameter field associated with the monitored database exception in the log data. An index may refer to a data structure used to locate data that meets a condition.

For example, the terminal may determine a target field to be monitored in log data and create an index between the target field and tracking identification information.

And step 404, filling the target field into an initial chart matched with the target field to obtain the log chart.

For example, the terminal may associate and assign the log data with the log schema according to the index, thereby obtaining the log schema.

In the embodiment, in the graphical processing of obtaining the log chart, the effective fields in the log data are identified and extracted, so that the effective data proportion in the log data is improved, on the other hand, the index between the target field and the tracking identification information is established, the query of specific fields is facilitated, and the flexibility of monitoring the database state is improved.

In one embodiment, as shown in fig. 5, the step 206 may include:

step 502, acquiring the log data output by the target database based on a preset log output framework layer.

The log output framework layer may refer to a framework or library for recording and managing log information in software development, among other things. These frameworks provide a mechanism that enables developers to conveniently record various events and state information while the system is running, for debugging, monitoring, and troubleshooting. The log output framework layer is mutually independent of the service processing layer of the target service.

For example, the terminal may acquire the log data output by the target database based on a preset log output framework layer.

In this embodiment, the log output framework layer obtains the log data output by the target database, so that the output of the log data and the monitoring of the database are independent of the service side, which is conducive to improving the universality of the monitoring of the database, and the method for monitoring the database is adapted to different languages and systems.

In one embodiment, as shown in fig. 6, the method further includes:

step 602, determining the target execution statement associated with the data request in a plurality of databases based on the tracking identification information.

For example, a terminal may determine the target execution statement associated with the data request in a plurality of databases based on the tracking identification information.

Step 604, obtaining the log data output by a plurality of databases based on the log output framework layer.

For example, the terminal may acquire the log data output by a plurality of the databases based on the log output framework layer.

In this embodiment, through the log output framework layer, multiple databases may be associated and monitored at the same time, so as to further improve flexibility and compatibility of log data monitoring.

In one embodiment, as shown in fig. 7, the step 206 includes:

step 702, obtaining the business method and stack information in the execution of the target execution statement.

A Stack (Stack) may refer to a data structure used to store and manage call information for a program runtime method. When the program is executed, each method call generates a stack frame (STACK FRAME) containing information such as parameters, local variables, and execution instructions of the method. The stack frames are arranged in sequence according to the order of method call, and form a call chain, namely a stack. When an exception occurs in the program execution process or a method call needs to be made, a stack is continuously pushed (push) and popped (pop) with stack frames so as to maintain the sequence and execution state of the method call. When an exception occurs in the program, the system generates a stack trace (STACK TRACE) that records information about the current method call chain, including the name of the method called, class name, line number, etc., to facilitate the developer in locating and troubleshooting the problem. Stack information, typically presented in text form, contains the hierarchy and order of method calls, and is one of the very important references for developers in debugging and analyzing program problems.

For example, the terminal may obtain the service method and stack information in the execution of the target execution statement.

Step 704, acquiring the log data based on the business method and the stack information, wherein the log data comprises a slow query log, a transaction log and a deadlock log.

For example, the terminal may obtain the log data based on the business method and the stack information, where the log data includes a slow query log, a transaction log, and a deadlock log.

In this embodiment, the log data includes a plurality of different types, which is conducive to implementing monitoring and analysis of different abnormal conditions, and improves flexibility of monitoring processing of the database.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a database monitoring device for realizing the database monitoring method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the database monitoring device provided below may be referred to the limitation of one database monitoring method hereinabove, and will not be repeated herein.

In one embodiment, as shown in FIG. 8, there is provided a database monitoring apparatus including a request response module, a statement positioning module, a log data module, and an anomaly monitoring module, wherein:

In one embodiment, the patterning module includes:

In one embodiment, the log data module includes:

In one embodiment, the apparatus further comprises:

In one embodiment, the log data module includes:

Each of the modules in the database monitoring apparatus may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a terminal, and the internal structure thereof may be as shown in fig. 9. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a database monitoring method. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by persons skilled in the art that the architecture shown in fig. 9 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting as to the computer device to which the present inventive arrangements are applicable, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.

In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magneto-resistive random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (PHASE CHANGE Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims

1. A database monitoring method, characterized in that the method comprises:

In response to a data request for a target business, creating tracking identification information associated with the data request;

Determine, based on the tracking identification information, a target execution statement to be executed in a target database for the data request;

Acquire log data generated during the execution of the target execution statement, wherein the log data is used to record parameter information when the target database executes the target execution statement;

The target database is monitored for abnormal conditions based on the log data, and the abnormal conditions are determined according to a preset monitoring logic.

2. The method according to claim 1, characterized in that the monitoring of the target database for abnormal conditions based on the log data, before the abnormal conditions are determined according to a preset monitoring logic, further comprises:

Performing graphical processing on the log data to obtain a log chart;

The abnormal situation is monitored based on the log chart.

3. The method according to claim 2, wherein the step of graphically processing the log data to obtain a log chart comprises:

The target field is filled into an initial graph matching the target field to obtain the log graph.

4. The method according to claim 1, characterized in that the step of obtaining log data generated during the execution of the target execution statement, wherein the log data is used to record parameter information when the target database executes the target execution statement, comprises:

The log data output by the target database is obtained based on a preset log output framework layer, and the log output framework layer is independent of the business processing layer of the target business.

5. The method according to claim 4, characterized in that the method further comprises:

The log data output by the plurality of databases is obtained based on the log output framework layer.

6. The method according to claim 1, characterized in that the step of obtaining log data generated during the execution of the target execution statement comprises:

Obtaining the business method and stack information during the execution of the target execution statement;

The log data is acquired based on the business method and the stack information, wherein the log data includes a slow query log, a transaction log, and a deadlock log.

7. A database monitoring device, characterized in that the device comprises:

A request response module, configured to create tracking identification information associated with a data request for a target business in response to the data request;

A statement locating module, configured to determine a target execution statement executed in a target database for the data request based on the tracking identification information;

A log data module, used to obtain log data generated during the execution of the target execution statement, wherein the log data is used to record parameter information when the target database executes the target execution statement;

The abnormality monitoring module is used to monitor the target database for abnormal conditions based on the log data, and the abnormal conditions are determined according to a preset monitoring logic.

8. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.