CN117215897A - Hot spot cache dynamic monitoring method, device, equipment and medium - Google Patents
Hot spot cache dynamic monitoring method, device, equipment and medium Download PDFInfo
- Publication number
- CN117215897A CN117215897A CN202311259984.0A CN202311259984A CN117215897A CN 117215897 A CN117215897 A CN 117215897A CN 202311259984 A CN202311259984 A CN 202311259984A CN 117215897 A CN117215897 A CN 117215897A
- Authority
- CN
- China
- Prior art keywords
- cache
- access
- hot spot
- preset
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 178
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000012550 audit Methods 0.000 claims abstract description 181
- 238000012545 processing Methods 0.000 claims description 41
- 238000004590 computer program Methods 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 13
- 238000012806 monitoring device Methods 0.000 claims description 8
- 238000005516 engineering process Methods 0.000 description 9
- 230000001960 triggered effect Effects 0.000 description 9
- 238000010168 coupling process Methods 0.000 description 6
- 238000005859 coupling reaction Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000015556 catabolic process Effects 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000001808 coupling effect Effects 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 229940060321 after-bug Drugs 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
Landscapes
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The application discloses a method, a device, equipment and a medium for dynamically monitoring hot spot cache, which relate to the technical field of cache monitoring and comprise the following steps: receiving an access storage data instruction, accessing a preset cache based on the access storage data instruction, and acquiring corresponding cache data; the method comprises the steps that an audit log real-time monitoring process records access key information and access time corresponding to cache data to obtain a target audit log carrying a cache log mark; when the record quantity of the target audit logs in the preset time window exceeds a preset record quantity threshold value, taking the corresponding target audit logs in the preset time window as hot spot cache monitoring logs, and triggering a hot spot cache monitoring process; and acquiring access times corresponding to all access key information in the hot spot cache monitoring log, and setting cache data corresponding to the hot spot cache monitoring log as hot spot cache when the access times exceed a preset access number threshold. Through the hot spot cache monitoring, the memory occupation is reduced, and the utilization rate of the system memory is improved.
Description
Technical Field
The present invention relates to the field of cache monitoring technologies, and in particular, to a method, an apparatus, a device, and a medium for dynamically monitoring a hot spot cache.
Background
Storage systems to increase the efficiency of data access, items often introduce NoSQL (i.e., not Only SQL, non-relational database; SQL, i.e., structured Query Language, structured query language) technology, which is a memory-based database and provides some persistence functionality. Redis (REmote DIctionary Server, remote dictionary service) technology is one of the NoSQL technologies. The use of Redis cache greatly improves the performance and efficiency of the application program, especially in the aspect of data query. Redis cache, while capable of greatly improving overall system performance, introduces more complexity. When most or even all service requests hit the same cache data, and the requests reach thousands of times, that is, hot spot caching occurs, the pressure of a cache server where the data is located is also great. Once the buffer space is excessively occupied, the buffer is not effectively released or the hot spot buffer is out of date, and the phenomena such as buffer breakdown and the like occur, the system is blocked or dead.
Under the prior art, engineers often embed monitoring processes when calling the system internal API (Application Program Interface ) interface to realize the monitoring and updating of the hot spot cache. However, the scheme and the system frame code have high coupling property, occupy memory very and are difficult to maintain and migrate; meanwhile, under a part of scenes, the embedded point monitoring process cannot monitor cache access update, if an API interface is not called, but a background CLI (Command Line Interface ) command is directly called to access or modify data, then such a scheme cannot monitor the cache update of the hot point.
In summary, how to reduce the coupling between different reductions and the system framework is suitable for hot spot cache monitoring in different scenes such as various GUI (Graphical User Interface ) interfaces, CLI scenes and the like; and the hot spot cache is monitored, the memory occupation is effectively reduced, and the utilization rate of the system memory is improved, so that the technical problem to be solved in the field is solved.
Disclosure of Invention
Therefore, the invention aims to provide a method, a device, equipment and a medium for dynamically monitoring hot spot cache, which can reduce the coupling between different reductions and a system frame, and are suitable for hot spot cache monitoring in different scenes such as various GUI interfaces, CLI scenes and the like; and the memory occupation is effectively reduced and the utilization rate of the system memory is improved while the hot spot cache is monitored. The specific scheme is as follows:
In a first aspect, the present application discloses a method for dynamically monitoring a hot spot cache, which is applied to a storage system, and includes:
receiving an access storage data instruction, and accessing a preset cache based on the access storage data instruction to acquire corresponding cache data;
recording access key information and access time corresponding to the cache data through an audit log real-time monitoring process so as to obtain a target audit log carrying a cache log mark;
when the record quantity of the target audit logs in the preset time window exceeds a preset record quantity threshold value, taking the target audit logs in the corresponding preset time window as hot spot cache monitoring logs, and triggering a hot spot cache monitoring process;
and acquiring access times corresponding to all access key information in the hot spot cache monitoring log, and setting the cache data corresponding to the hot spot cache monitoring log as hot spot cache when the access times exceed a preset access number threshold.
Optionally, the receiving the access storage data instruction includes:
receiving an access storage data instruction sent by a user side through a preset graphical user interface of the storage system;
or receiving an access storage data instruction written by the user side through the reservation command line interface.
Optionally, the receiving the access storage data instruction, accessing a preset cache based on the access storage data instruction to obtain corresponding cache data, includes:
receiving an access storage data instruction, and judging whether an object to be accessed in the access data instruction is a preset cache;
if the object to be accessed is the preset cache, starting an audit log real-time monitoring process, and recording the data access and storage instruction through a first audit log;
accessing the preset cache based on the access storage data instruction so as to acquire corresponding cache data from the preset cache;
correspondingly, the process of monitoring the access key information and the access time corresponding to the cache data in real time through the audit log to obtain a target audit log carrying the cache log mark comprises the following steps:
acquiring access key information and access time when accessing the cache data in the preset cache;
recording the access key information and the access time through the audit log real-time monitoring process, and combining the access key information and the access time with the access storage data instruction in the first audit log to obtain a second audit log;
And marking the second audit log by using a preset cache log identifier to obtain a target audit log carrying the cache log identifier.
Optionally, after the real-time monitoring process of the audit log records the access key information and the access time corresponding to the cached data to obtain the target audit log carrying the cache log identifier, the method further includes:
inputting the target audit log carrying the cache log mark into a preset stream processing system so that the preset stream processing system calculates the target audit log in real time according to the access time in the target audit log to obtain the record quantity corresponding to the target audit log in a plurality of preset time windows;
judging whether the record quantity of the target audit log in the preset time window exceeds a preset record quantity threshold value or not so as to obtain a corresponding judgment result.
Optionally, after the determining whether the record amount of the target audit log in the preset time window exceeds the preset record amount threshold to obtain a corresponding determination result, the method further includes:
and when the judgment result is that the record quantity of the target audit log in the preset time window does not exceed a preset record quantity threshold value, not triggering a hot spot cache monitoring process.
Optionally, after the target audit log carrying the cache log identifier is input to a preset stream processing system, the method further includes:
and setting a time window of a preset stream processing system so that the preset stream processing system groups all the received target audit logs carrying the cache log identification according to the time window and the access time in the target audit logs.
Optionally, after setting the cache data corresponding to the hot spot cache monitoring log to be a hot spot cache, the method further includes:
triggering a main node updating logic to update all the hot spot caches to a preset cluster server main node;
and triggering a cache data synchronization logic to synchronize the hot spot caches in the preset cluster server main nodes to all preset cluster server standby nodes.
In a second aspect, the present application discloses a hot spot cache dynamic monitoring device, which is applied to a storage system, and includes:
the data acquisition module is used for receiving an access storage data instruction, accessing a preset cache based on the access storage data instruction, and acquiring corresponding cache data;
the log generation module is used for recording access key information and access time corresponding to the cache data through an audit log real-time monitoring process so as to obtain a target audit log carrying a cache log mark;
The cache monitoring starting module is used for taking the target audit log in the corresponding preset time window as a hot spot cache monitoring log and triggering a hot spot cache monitoring process when the record quantity of the target audit log in the preset time window exceeds a preset record quantity threshold;
and the cache judging module is used for acquiring the access times corresponding to all the access key information in the hot spot cache monitoring log, and setting the cache data corresponding to the hot spot cache monitoring log as hot spot cache when the access times exceed a preset access quantity threshold.
In a third aspect, the present application discloses an electronic device, comprising:
a memory for storing a computer program;
and the processor is used for executing the computer program to realize the steps of the hot spot cache dynamic monitoring method.
In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; the method comprises the steps of a hot spot cache dynamic monitoring method, wherein the steps of the hot spot cache dynamic monitoring method are realized when the computer program is executed by a processor.
The application discloses a hot spot cache dynamic monitoring method, which is applied to a storage system and comprises the following steps: receiving an access storage data instruction, and accessing a preset cache based on the access storage data instruction to acquire corresponding cache data; recording access key information and access time corresponding to the cache data through an audit log real-time monitoring process so as to obtain a target audit log carrying a cache log mark; when the record quantity of the target audit logs in the preset time window exceeds a preset record quantity threshold value, taking the target audit logs in the corresponding preset time window as hot spot cache monitoring logs, and triggering a hot spot cache monitoring process; and acquiring access times corresponding to all access key information in the hot spot cache monitoring log, and setting the cache data corresponding to the hot spot cache monitoring log as hot spot cache when the access times exceed a preset access number threshold. Therefore, the hot spot cache monitoring process is not required to be started in real time, and is triggered when the recording quantity exceeds a preset recording quantity threshold value, so that the access times corresponding to access key information in the hot spot cache monitoring log are acquired, then the fact that the cache data corresponding to the hot spot monitoring log which corresponds to a specific one or a plurality of access key information in the hot spot cache monitoring log are hot spot caches is further judged, and the hot spot cache monitoring method can be suitable for monitoring hot spot caches in various GUI interfaces, CLI commands and other scenes through the audit log recording mode; and the memory occupation is effectively reduced and the utilization rate of the system memory is improved while the hot spot cache is monitored.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for dynamically monitoring a hot spot cache according to the present application;
FIG. 2 is a flow chart of audit log cache storage in accordance with the present disclosure;
FIG. 3 is a flow chart of hot spot cache monitoring in accordance with the present disclosure;
FIG. 4 is a flowchart of a method for dynamically monitoring a hot spot cache according to the present application;
FIG. 5 is a flow chart of audit log real-time monitoring in accordance with the present disclosure;
FIG. 6 is a diagram of an overall architecture for hot spot cache monitoring in accordance with the present disclosure;
FIG. 7 is a schematic diagram of a hot spot cache dynamic monitoring device according to the present application;
fig. 8 is a block diagram of an electronic device according to the present disclosure.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Storage systems to increase the efficiency of data access, items often introduce NoSQL technology, which is a memory-based database and provides some persistence functionality. Redis technology is one of NoSQL technology. The use of Redis cache greatly improves the performance and efficiency of the application program, especially in the aspect of data query. Redis cache, while capable of greatly improving overall system performance, introduces more complexity. When most or even all service requests hit the same cache data, and the requests reach thousands of times, that is, hot spot caching occurs, the pressure of a cache server where the data is located is also great. Once the buffer space is excessively occupied, the buffer is not effectively released or the hot spot buffer is out of date, and the phenomena such as buffer breakdown and the like occur, the system is blocked or dead.
Under the prior art, engineers often embed monitoring processes when calling an API interface in the system to realize the monitoring and updating of the hot spot cache. However, the scheme and the system frame code have high coupling property, occupy memory very and are difficult to maintain and migrate; meanwhile, under partial scenes, the embedded point monitoring process cannot monitor cache access update, if an API interface is not called, but a background CLI command is directly called to access or modify data, the scheme cannot monitor the hot spot cache update.
Therefore, the application provides a hot spot cache dynamic monitoring scheme which can reduce the coupling between different reductions and a system frame, and is suitable for hot spot cache monitoring in different scenes such as various GUI interfaces, CLI scenes and the like; and the memory occupation is effectively reduced and the utilization rate of the system memory is improved while the hot spot cache is monitored.
Referring to fig. 1, an embodiment of the present application discloses a method for dynamically monitoring a hot spot cache, which is applied to a storage system and includes:
step S11: and receiving an access storage data instruction, and accessing a preset cache based on the access storage data instruction to acquire corresponding cache data.
In this embodiment, when an access storage data instruction sent by a user terminal is received, a preset cache is accessed based on the access storage data instruction, so as to obtain cache data corresponding to the access storage data instruction through the preset cache. The preset cache may be a dis cache.
In this embodiment, a command for accessing stored data sent by a user terminal through a preset graphical user interface of the storage system is received; or receiving an access storage data instruction written by the user side through the reservation command line interface. It can be understood that the access storage data instruction sent by the user side through the GUI interface mode of the system is received, or the access storage data instruction sent by the user side through the CLI command mode of the system is received. It can be seen that, in this embodiment, the instruction for accessing and storing data may be obtained in different manners, and accordingly, the hot spot cache monitoring method may also be deployed in different scenes. And the application range of hot spot cache monitoring is enlarged.
Step S12: and recording access key information and access time corresponding to the cache data through an audit log real-time monitoring process so as to obtain a target audit log carrying a cache log identifier.
In this embodiment, the audit log real-time monitoring process records the access key information and the access time corresponding to the cache data, so as to obtain the target audit log carrying the cache log identifier. It should be noted that the access key information and the access time are recorded through the audit log real-time monitoring process without recording the cache data, so that the access amount of the cache data can be rapidly identified by determining the cache log identification on the cache data recorded by the monitoring process according to the recorded target audit log, the access key information and the access time are recorded, the cache storage capacity of the target audit log can be reduced by adopting a recording mode without recording the cache data, and the storage capacity utilization rate is improved.
Referring to fig. 2, this embodiment provides an audit log cache storage flow, where when a user side accesses stored data, the audit log cache storage flow may be accessed by a system GUI interface or CLI command to obtain the stored data; when the audit log real-time monitoring process monitors that the accessed storage data is acquired through the Redis cache, the process acquires the cache key and the access time, namely the access key information and the access time, writes the cache key information and the access time into the cache audit log, marks the cache audit log to acquire a target audit log carrying a flag mark, and can determine that the target audit log is the cache access record through the carried flag mark. Monitoring hot spot cache dynamic update with audit log: the method has the advantages that the dynamic monitoring and updating of the hot spot cache are realized in a low-coupling mode through the audit log, the operation is independent of the system frame, and the coupling between the hot spot cache monitoring process and the frame is reduced; the flexibility of the monitoring process is improved, the hot spot cache monitoring in various scenes is realized, the cache access record format is optimized, the cache storage capacity of the audit log is effectively reduced, and the storage capacity utilization rate is improved.
In this embodiment, after the audit log real-time monitoring process records the access key information and the access time corresponding to the cache data to obtain the target audit log carrying the cache log identifier, the method further includes: inputting the target audit log carrying the cache log mark into a preset stream processing system so that the preset stream processing system calculates the target audit log in real time according to the access time in the target audit log to obtain the record quantity corresponding to the target audit log in a plurality of preset time windows; judging whether the record quantity of the target audit log in the preset time window exceeds a preset record quantity threshold value or not so as to obtain a corresponding judgment result. It can be understood that the target audit log carrying the cache log identifier is input to a preset stream processing system, and the target audit log can be calculated in real time by the stream processing system, so that the corresponding log record amount in a plurality of time windows can be judged, and the log record amount is judged to exceed a preset record amount threshold, wherein the preset record amount threshold is a record amount threshold set by a system of the stream processing system or manually set, and the method is not particularly limited.
Step S13: and when the record quantity of the target audit logs in the preset time window exceeds a preset record quantity threshold value, taking the target audit logs in the corresponding preset time window as hot spot cache monitoring logs, and triggering a hot spot cache monitoring process.
In this embodiment, if the recording amount of the target audit log in the preset time window exceeds the preset recording amount threshold, it indicates that the recording amount of the target audit log in the preset time window suddenly increases, that is, the number of corresponding access storage data instructions increases suddenly, and correspondingly, the number of access storage data requests increases suddenly. And indicating that one or more hot spot caches possibly exist in the preset time window, correspondingly triggering a hot spot cache monitoring process, and performing hot spot cache monitoring on cache data corresponding to all hot spot cache monitoring logs in the preset time window.
Step S14: and acquiring access times corresponding to all access key information in the hot spot cache monitoring log, and setting the cache data corresponding to the hot spot cache monitoring log as hot spot cache when the access times exceed a preset access number threshold.
In this embodiment, the number of accesses corresponding to the access key information in the hot spot cache monitoring log in the preset time window is counted, so that the total number of accesses corresponding to different access key information in the preset time window can be obtained, and when the total number of accesses exceeds a set access number threshold, cache data corresponding to the corresponding access key information is set as hot spot cache. And when the total access times do not exceed the access quantity threshold, setting the cache data corresponding to the hot spot cache monitoring log as cold spot cache. And the cold point cache is cache data corresponding to the condition that the total access times of the access key information does not exceed the manually set access quantity threshold.
In this embodiment, after setting the cache data corresponding to the hot spot cache monitoring log as a hot spot cache, the method further includes: triggering a main node updating logic to update all the hot spot caches to a preset cluster server main node; and triggering a cache data synchronization logic to synchronize the hot spot caches in the preset cluster server main nodes to all preset cluster server standby nodes. It can be understood that when most or even all service access requests hit the same cache data, the service access requests can reach thousands of times, in order to timely feed back the cache data corresponding to the service access requests to each user end, namely, hot spot cache, the problems of excessive pressure of a server where the hot spot cache is located, blocking dead caused by expiration or breakdown of the hot spot cache and the like are avoided, after the hot spot cache is set, a master node update logic is triggered so as to update the hot spot cache to a preset cluster server master node, and then a cache data synchronization logic in a cluster is triggered so as to synchronize the hot spot cache in the preset cluster server master node to all preset cluster server standby nodes. In this way, by utilizing the active-standby characteristics of the cluster server, the hot-spot cache is replicated and stored in different servers, so as to cope with the situation that a large number of unified access requests hit the same cache data.
Referring to fig. 3, this embodiment discloses a hot spot cache monitoring flow. When the hot spot cache monitoring process is triggered, it is indicated that a large number of cache access records appear in the period L, at this time, each cache key and access times N2 in the period L are obtained through the process, and then whether the access times N2 exceed a hot spot threshold M2 or not is judged, namely, a preset access number threshold M2 is set. When N2 of the corresponding key is larger than or equal to M2, the cache key is indicated to be hot spot cache, and then the hot spot cache is updated to the main node of the cache cluster server and synchronously updated to the standby node of the cache cluster server. Otherwise, the cache key is indicated as cold point cache and is not processed.
The application discloses a hot spot cache dynamic monitoring method, which is applied to a storage system and comprises the following steps: receiving an access storage data instruction, and accessing a preset cache based on the access storage data instruction to acquire corresponding cache data; recording access key information and access time corresponding to the cache data through an audit log real-time monitoring process so as to obtain a target audit log carrying a cache log mark; when the record quantity of the target audit logs in the preset time window exceeds a preset record quantity threshold value, taking the target audit logs in the corresponding preset time window as hot spot cache monitoring logs, and triggering a hot spot cache monitoring process; and acquiring access times corresponding to all access key information in the hot spot cache monitoring log, and setting the cache data corresponding to the hot spot cache monitoring log as hot spot cache when the access times exceed a preset access number threshold. Therefore, the hot spot cache monitoring process is not required to be started in real time, and is triggered when the recording quantity exceeds a preset recording quantity threshold value, so that the access times corresponding to access key information in the hot spot cache monitoring log are acquired, then the fact that the cache data corresponding to the hot spot monitoring log which corresponds to a specific one or a plurality of access key information in the hot spot cache monitoring log are hot spot caches is further judged, and the hot spot cache monitoring method can be suitable for monitoring hot spot caches in various GUI interfaces, CLI commands and other scenes through the audit log recording mode; and the memory occupation is effectively reduced and the utilization rate of the system memory is improved while the hot spot cache is monitored.
Referring to fig. 4, an embodiment of the present invention discloses a specific method for dynamically monitoring a hot spot cache, and compared with the previous embodiment, the present embodiment further describes and optimizes a technical solution. Specific:
step S21: receiving an access storage data instruction, and judging whether an object to be accessed in the access data instruction is a preset cache; if the object to be accessed is the preset cache, starting an audit log real-time monitoring process, and recording the data access and storage instruction through a first audit log; and accessing the preset cache based on the access storage data instruction so as to acquire corresponding cache data from the preset cache.
In this embodiment, the received access storage data instruction is judged, whether the object to be accessed in the access storage data instruction is a dis cache is judged, if not, the audit log real-time monitoring process is not required to be started, and the audit log real-time monitoring process is monitored according to other cache monitoring modes. If yes, starting an audit log real-time monitoring process, accessing a stored data instruction through a first audit log record, and accessing a Redis cache based on the accessed stored data instruction so as to acquire corresponding cache data from the Redis cache.
Step S22: acquiring access key information and access time when accessing the cache data in the preset cache; recording access key information and access time through the audit log real-time monitoring process, and combining the access key information and the access time with the access storage data instruction in the first audit log to obtain a second audit log; and marking the second audit log by using a preset cache log identifier to obtain a target audit log carrying the cache log identifier.
In this embodiment, access key information and access time when accessing the cache data in the preset cache are obtained; recording access key information and access time through the audit log real-time monitoring process, and combining the access key information and the access time with the access storage data instruction in the first audit log to obtain a second audit log; and marking the second audit log by using a preset cache log identifier to obtain a target audit log carrying the cache log identifier. It can be understood that the audit log real-time monitoring process records access key information and access time, then merges the recorded access storage data instruction of the first audit log with the access record containing the access key information and the access time to obtain a second audit log, and then marks the cache log identification flag on the second audit log to obtain a target audit log carrying the cache log identification flag.
Step S23: inputting the target audit log carrying the cache log mark into a preset stream processing system so that the preset stream processing system calculates the target audit log in real time according to the access time in the target audit log to obtain the record quantity corresponding to the target audit log in a plurality of preset time windows; judging whether the record quantity of the target audit log in the preset time window exceeds a preset record quantity threshold value or not so as to obtain a corresponding judgment result.
In this embodiment, a stream computing technology is used to process a target audit log carrying a buffer log identification flag, specifically, stream Processing (a stream processing system) may be used to calculate the target audit log in real time according to access time in the target audit log, it may be understood that the stream processing system through real-time data processing may calculate the target audit log in real time according to access time in the target audit log, so as to obtain a record amount corresponding to the target audit log under a plurality of continuous real-time preset time windows, and then determine whether the record amount of the target audit log in each time window exceeds a preset record amount threshold, so as to obtain a corresponding determination result.
In this embodiment, after the target audit log carrying the cache log identifier is input to the preset stream processing system, the method further includes: and setting a time window of a preset stream processing system so that the preset stream processing system groups all the received target audit logs carrying the cache log identification according to the time window and the access time in the target audit logs. Before the stream processing system processes the target audit log, a time window of the stream processing system is also set, where the time window is a mechanism for grouping a number of events by time or other characteristics so as to obtain target audit log packets under different time windows.
In this embodiment, in the process of performing real-time calculation on the target audit log by using the Flink in the stream processing system, the target audit log obtained in real time is divided according to the time window to obtain different target audit log groups, and then the Flink-based system has a distributed characteristic, and since the Flink-based system can run on hundreds of machines, the Flink-based system divides a large calculation task into a plurality of small parts, and each machine executes one part. And further, a large number of target audit logs can be calculated in real time, so that the recorded amounts in different time windows can be obtained. Furthermore, the flank can automatically ensure that the calculation continues when a machine failure or other error occurs, or is performed again in a planned manner after bug fixes or version upgrades are made. This capability eliminates the need for developers to worry about failure. The flank essentially uses fault tolerant data streams, which allows developers to analyze data that is continuously generated and never ends (i.e., stream processing).
Referring to fig. 5, the present embodiment discloses an audit log real-time monitoring process. The process monitors the writing of the cache record flag in the audit log in real time through a stream computing technology. When the cache records are monitored to be in the appointed period L, N1 cache records are recorded, and whether N1 exceeds a set access record threshold M1 or not is judged, namely, a preset record quantity threshold M1 is judged. If N1 is more than or equal to M1, a large number of cache record logs appear in the period L, and a hot spot cache monitoring process is triggered to judge whether the hot spot cache is the hot spot cache or not; if N1< M1, the cache access records are less in the period L, and the hot spot cache access phenomenon is not generated.
Step S24: and when the judgment result is that the record quantity of the target audit log in the preset time window does not exceed a preset record quantity threshold value, not triggering a hot spot cache monitoring process.
In this embodiment, when the record amount does not exceed the preset record amount threshold, it indicates that the number of cache access records in the preset time window is small, and no hot spot cache access condition is generated, that is, the hot spot cache monitoring process is not required to be triggered, so that the memory occupation of the hot spot cache monitoring process is reduced, and the utilization rate of the system memory is improved.
Step S25: and when the judgment result is that the record quantity of the target audit logs in the preset time window exceeds the preset record quantity threshold, taking the target audit logs in the corresponding preset time window as hot spot cache monitoring logs, and triggering a hot spot cache monitoring process.
Step S26: and acquiring access times corresponding to all access key information in the hot spot cache monitoring log, and setting the cache data corresponding to the hot spot cache monitoring log as hot spot cache when the access times exceed a preset access number threshold.
The more detailed processing procedures in steps S25 and S26 refer to the content of the foregoing disclosed embodiments, and are not described herein.
The disclosed system deploys the overall architecture as shown in fig. 6. In different systems, such as a system 1 and a system 2, for realizing monitoring and updating of a hot spot cache, specifically, cache data aiming at an access storage data instruction is obtained through audit log cache storage, and then access time and access key information corresponding to the cache data are stored to generate a target audit log carrying a cache log mark, so that the cache data is not required to be stored; and then, carrying out real-time monitoring on the writing condition of the cache log mark through the real-time monitoring of the audit log, when the number of the recorded target audit logs exceeds the preset record number threshold value in a specified period, setting the target audit log in the specified period as a hot spot cache monitoring log, further monitoring and acquiring the total access times of access key information in the hot spot cache monitoring log through a hot spot cache monitoring process, and taking cache data corresponding to the access key information in the hot spot cache monitoring log with the total access times exceeding the preset access number threshold value as hot spot caches again.
Therefore, deployment monitoring is carried out in a low-coupling mode through the audit log, the system is independent of the system and is not constrained by a system frame, a hot spot cache monitoring process is selectively triggered according to whether the access amount of the hot spot cache in a period exceeds a threshold value, the memory occupation of the process is reduced, and the memory utilization rate of the system is improved.
Referring to fig. 7, the embodiment of the invention also correspondingly discloses a hot spot cache dynamic monitoring device, which is applied to a storage system and comprises:
the data acquisition module 11 is configured to receive an access storage data instruction, and access a preset cache based on the access storage data instruction to acquire corresponding cache data;
the log generating module 12 is configured to record, through an audit log real-time monitoring process, access key information and access time corresponding to the cache data, so as to obtain a target audit log carrying a cache log identifier;
the cache monitoring starting module 13 is configured to take the target audit log in the corresponding preset time window as a hot spot cache monitoring log and trigger a hot spot cache monitoring process when the record amount of the target audit log in the preset time window exceeds a preset record amount threshold;
And the cache judging module 14 is configured to obtain the access times corresponding to all the access key information in the hot spot cache monitoring log, and set the cache data corresponding to the hot spot cache monitoring log as a hot spot cache when the access times exceed a preset access number threshold.
Therefore, the method and the device disclosed by the application receive the access storage data instruction, and access the preset cache based on the access storage data instruction so as to acquire corresponding cache data; recording access key information and access time corresponding to the cache data through an audit log real-time monitoring process so as to obtain a target audit log carrying a cache log mark; when the record quantity of the target audit logs in the preset time window exceeds a preset record quantity threshold value, taking the target audit logs in the corresponding preset time window as hot spot cache monitoring logs, and triggering a hot spot cache monitoring process; and acquiring access times corresponding to all access key information in the hot spot cache monitoring log, and setting the cache data corresponding to the hot spot cache monitoring log as hot spot cache when the access times exceed a preset access number threshold. Therefore, the hot spot cache monitoring process is not required to be started in real time, and is triggered when the recording quantity exceeds a preset recording quantity threshold value, so that the access times corresponding to access key information in the hot spot cache monitoring log are acquired, then the fact that the cache data corresponding to the hot spot monitoring log which corresponds to a specific one or a plurality of access key information in the hot spot cache monitoring log are hot spot caches is further judged, and the hot spot cache monitoring method can be suitable for monitoring hot spot caches in various GUI interfaces, CLI commands and other scenes through the audit log recording mode; and the memory occupation is effectively reduced and the utilization rate of the system memory is improved while the hot spot cache is monitored.
In some embodiments, the data acquisition module 11 may specifically include:
the data acquisition unit is used for receiving an access storage data instruction sent by a user side through a preset graphical user interface of the storage system; or receiving an access storage data instruction written by the user side through the reservation command line interface.
In some embodiments, the data acquisition module 11 may specifically include:
the cache judging unit is used for receiving an access storage data instruction and judging whether an object to be accessed in the access data instruction is a preset cache or not;
the process starting unit is used for starting an audit log real-time monitoring process if the object to be accessed is the preset cache, and recording the access storage data instruction through a first audit log;
and the cache access unit is used for accessing the preset cache based on the access storage data instruction so as to acquire corresponding cache data from the preset cache.
In some embodiments, the data acquisition module 11 may specifically include:
the access information acquisition unit is used for acquiring access key information and access time when the cache data in the preset cache are accessed;
The data merging unit is used for recording the access key information and the access time through the audit log real-time monitoring process and merging the access key information and the access storage data instruction in the first audit log to obtain a second audit log;
the log obtaining unit is used for marking the second audit log by using a preset cache log identifier so as to obtain a target audit log carrying the cache log identifier.
In some embodiments, the hotspot cache dynamic monitoring device may specifically include:
the log calculation unit is used for inputting the target audit log carrying the cache log mark into a preset stream processing system so that the preset stream processing system calculates the target audit log in real time according to the access time in the target audit log to obtain the record quantity corresponding to the target audit log in a plurality of preset time windows;
and the recording quantity judging unit is used for judging whether the recording quantity of the target audit log in the preset time window exceeds a preset recording quantity threshold value so as to obtain a corresponding judging result.
In some embodiments, the hotspot cache dynamic monitoring device may specifically include:
And the process triggering unit is used for not triggering a hot spot cache monitoring process when the judgment result is that the record quantity of the target audit log in the preset time window does not exceed a preset record quantity threshold value.
In some embodiments, the hotspot cache dynamic monitoring device may specifically include:
the window setting unit is used for setting a time window of a preset stream processing system so that the preset stream processing system groups all received target audit logs carrying the cache log mark according to the time window and the access time in the target audit log.
In some embodiments, the hotspot cache dynamic monitoring device may specifically include:
the node updating unit is used for triggering a main node updating logic to update all the hot spot caches to a main node of a preset cluster server;
and the data synchronization unit is used for triggering a cache data synchronization logic to synchronize the hot spot cache in the main node of the preset cluster server to all standby nodes of the preset cluster server.
Further, the embodiment of the present application further discloses an electronic device, and fig. 8 is a block diagram of an electronic device 20 according to an exemplary embodiment, where the content of the diagram is not to be considered as any limitation on the scope of use of the present application.
Fig. 8 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, where the computer program is loaded and executed by the processor 21 to implement relevant steps in the hot spot cache dynamic monitoring method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the present embodiment may be specifically an electronic computer.
In this embodiment, the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.
Processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor 21 may also comprise a main processor, which is a processor for processing data in an awake state, also called CPU (Central Processing Unit ); a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit, image processor) for rendering and drawing of content required to be displayed by the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.
The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon may include an operating system 221, a computer program 222, and the like, and the storage may be temporary storage or permanent storage.
The operating system 221 is used for managing and controlling various hardware devices on the electronic device 20 and the computer program 222, so as to implement the operation and processing of the processor 21 on the mass data 223 in the memory 22, which may be Windows Server, netware, unix, linux, etc. The computer program 222 may further comprise a computer program capable of performing other specific tasks in addition to the computer program capable of performing the hotspot cache dynamic monitoring method performed by the electronic device 20 as disclosed in any of the embodiments previously described. The data 223 may include, in addition to data received by the electronic device and transmitted by the external device, data collected by the input/output interface 25 itself, and so on.
Further, the application also discloses a computer readable storage medium for storing a computer program; the method for dynamically monitoring the hot spot cache is characterized in that the computer program is executed by a processor to realize the method for dynamically monitoring the hot spot cache. For specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application. The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in random access Memory RAM (Random Access Memory), memory, read-Only Memory ROM (Read Only Memory), electrically programmable EPROM (Electrically Programmable Read Only Memory), electrically erasable programmable EEPROM (Electric Erasable Programmable Read Only Memory), registers, hard disk, a removable disk, a CD-ROM (Compact Disc-Read Only Memory), or any other form of storage medium known in the art.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing describes in detail a method, apparatus, device and medium for dynamically monitoring a hot spot cache, and specific examples are applied to illustrate the principles and embodiments of the present invention, where the foregoing examples are only used to help understand the method and core idea of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (10)
1. A hot spot cache dynamic monitoring method is characterized by being applied to a storage system and comprising the following steps:
receiving an access storage data instruction, and accessing a preset cache based on the access storage data instruction to acquire corresponding cache data;
recording access key information and access time corresponding to the cache data through an audit log real-time monitoring process so as to obtain a target audit log carrying a cache log mark;
when the record quantity of the target audit logs in the preset time window exceeds a preset record quantity threshold value, taking the target audit logs in the corresponding preset time window as hot spot cache monitoring logs, and triggering a hot spot cache monitoring process;
and acquiring access times corresponding to all access key information in the hot spot cache monitoring log, and setting the cache data corresponding to the hot spot cache monitoring log as hot spot cache when the access times exceed a preset access number threshold.
2. The method for dynamically monitoring a hot spot cache according to claim 1, wherein the receiving the access storage data instruction comprises:
receiving an access storage data instruction sent by a user side through a preset graphical user interface of the storage system;
Or receiving an access storage data instruction written by the user side through the reservation command line interface.
3. The method for dynamically monitoring a hot spot cache according to claim 1, wherein the receiving the access storage data instruction, accessing the preset cache based on the access storage data instruction to obtain the corresponding cache data, comprises:
receiving an access storage data instruction, and judging whether an object to be accessed in the access data instruction is a preset cache;
if the object to be accessed is the preset cache, starting an audit log real-time monitoring process, and recording the data access and storage instruction through a first audit log;
accessing the preset cache based on the access storage data instruction so as to acquire corresponding cache data from the preset cache;
correspondingly, the process of monitoring the access key information and the access time corresponding to the cache data in real time through the audit log to obtain a target audit log carrying the cache log mark comprises the following steps:
acquiring access key information and access time when accessing the cache data in the preset cache;
recording the access key information and the access time through the audit log real-time monitoring process, and combining the access key information and the access time with the access storage data instruction in the first audit log to obtain a second audit log;
And marking the second audit log by using a preset cache log identifier to obtain a target audit log carrying the cache log identifier.
4. The method for dynamically monitoring a hot spot cache according to claim 1, wherein after the real-time monitoring process of the audit log records the access key information and the access time corresponding to the cache data to obtain the target audit log carrying the cache log identifier, the method further comprises:
inputting the target audit log carrying the cache log mark into a preset stream processing system so that the preset stream processing system calculates the target audit log in real time according to the access time in the target audit log to obtain the record quantity corresponding to the target audit log in a plurality of preset time windows;
judging whether the record quantity of the target audit log in the preset time window exceeds a preset record quantity threshold value or not so as to obtain a corresponding judgment result.
5. The method for dynamically monitoring a hot spot cache according to claim 4, wherein after determining whether the record amount of the target audit log in the preset time window exceeds a preset record amount threshold to obtain a corresponding determination result, further comprises:
And when the judgment result is that the record quantity of the target audit log in the preset time window does not exceed a preset record quantity threshold value, not triggering a hot spot cache monitoring process.
6. The method for dynamically monitoring the hot spot cache according to claim 4, wherein after the target audit log carrying the cache log identifier is input to a preset stream processing system, the method further comprises:
and setting a time window of a preset stream processing system so that the preset stream processing system groups all the received target audit logs carrying the cache log identification according to the time window and the access time in the target audit logs.
7. The method for dynamically monitoring a hot spot cache according to any one of claims 1 to 6, wherein after the setting the cache data corresponding to the hot spot cache monitoring log as a hot spot cache, further comprises:
triggering a main node updating logic to update all the hot spot caches to a preset cluster server main node;
and triggering a cache data synchronization logic to synchronize the hot spot caches in the preset cluster server main nodes to all preset cluster server standby nodes.
8. The utility model provides a hot spot buffer memory dynamic monitoring device which is characterized in that is applied to memory system, includes:
the data acquisition module is used for receiving an access storage data instruction, accessing a preset cache based on the access storage data instruction, and acquiring corresponding cache data;
the log generation module is used for recording access key information and access time corresponding to the cache data through an audit log real-time monitoring process so as to obtain a target audit log carrying a cache log mark;
the cache monitoring starting module is used for taking the target audit log in the corresponding preset time window as a hot spot cache monitoring log and triggering a hot spot cache monitoring process when the record quantity of the target audit log in the preset time window exceeds a preset record quantity threshold;
and the cache judging module is used for acquiring the access times corresponding to all the access key information in the hot spot cache monitoring log, and setting the cache data corresponding to the hot spot cache monitoring log as hot spot cache when the access times exceed a preset access quantity threshold.
9. An electronic device, comprising:
a memory for storing a computer program;
A processor for executing the computer program to implement the steps of the hot spot cache dynamic monitoring method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program; wherein the computer program when executed by a processor implements the steps of the hot spot cache dynamic monitoring method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311259984.0A CN117215897A (en) | 2023-09-27 | 2023-09-27 | Hot spot cache dynamic monitoring method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311259984.0A CN117215897A (en) | 2023-09-27 | 2023-09-27 | Hot spot cache dynamic monitoring method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117215897A true CN117215897A (en) | 2023-12-12 |
Family
ID=89050952
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311259984.0A Pending CN117215897A (en) | 2023-09-27 | 2023-09-27 | Hot spot cache dynamic monitoring method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117215897A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118779257A (en) * | 2024-09-10 | 2024-10-15 | 浪潮云洲工业互联网有限公司 | A multi-level multi-node caching method, device, equipment and storage medium |
-
2023
- 2023-09-27 CN CN202311259984.0A patent/CN117215897A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118779257A (en) * | 2024-09-10 | 2024-10-15 | 浪潮云洲工业互联网有限公司 | A multi-level multi-node caching method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11882054B2 (en) | Terminating data server nodes | |
| CN110457281A (en) | Data processing method, device, equipment and medium | |
| CN109446225B (en) | Data caching method and device, computer equipment and storage medium | |
| CN111522703A (en) | Method, apparatus and computer program product for monitoring access requests | |
| CN113495889B (en) | Distributed object storage method and device, electronic equipment and storage medium | |
| CN101136825A (en) | Asynchronous configuration information management method and system for client terminal/server structure | |
| CN111858284A (en) | A resource monitoring method and device for an artificial intelligence server | |
| WO2023165188A1 (en) | Sequential read prefetching method and apparatus, device, and medium | |
| CN114036195B (en) | A data request processing method, device, server and storage medium | |
| CN117215897A (en) | Hot spot cache dynamic monitoring method, device, equipment and medium | |
| US11055223B2 (en) | Efficient cache warm up based on user requests | |
| CN108052385A (en) | A kind of Container Management method, system, equipment and computer storage media | |
| CN112948336A (en) | Data acceleration method, cache unit, electronic device and storage medium | |
| CN111988427A (en) | Distributed local cache processing method and system | |
| CN119065612A (en) | A monitoring data cache flushing method, device and medium | |
| CN118540371A (en) | Cluster resource management method and device, storage medium and electronic equipment | |
| CN113868184B (en) | Configuration method and system of configuration center based on cache of master-slave architecture | |
| CN106302726B (en) | A kind of monitoring method and module of the management node of distributed type assemblies data | |
| JP2023022843A (en) | Failure information positioning method, device, equipment and storage medium | |
| CN116126411A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN109981391B (en) | A sampling method, device and readable storage medium | |
| CN120336007B (en) | Distributed cache optimization method, device, electronic device and storage medium | |
| CN102402397B (en) | Method and device for storing data | |
| CN112559568A (en) | Virtual article determination method and device and computer readable storage medium | |
| CN102902825B (en) | A kind of database optimizing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |