[go: up one dir, main page]

CN120386801B - Data analysis system and method based on distributed caching technology - Google Patents

Data analysis system and method based on distributed caching technology

Info

Publication number
CN120386801B
CN120386801B CN202510779687.1A CN202510779687A CN120386801B CN 120386801 B CN120386801 B CN 120386801B CN 202510779687 A CN202510779687 A CN 202510779687A CN 120386801 B CN120386801 B CN 120386801B
Authority
CN
China
Prior art keywords
data
node
cache
capacity
business
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202510779687.1A
Other languages
Chinese (zh)
Other versions
CN120386801A (en
Inventor
王陶
温砚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu Zhimeng Intelligent Technology Co ltd
Original Assignee
Jiangsu Zhimeng Intelligent Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu Zhimeng Intelligent Technology Co ltd filed Critical Jiangsu Zhimeng Intelligent Technology Co ltd
Priority to CN202510779687.1A priority Critical patent/CN120386801B/en
Publication of CN120386801A publication Critical patent/CN120386801A/en
Application granted granted Critical
Publication of CN120386801B publication Critical patent/CN120386801B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements
    • H04L67/288—Distributed intermediate devices, i.e. intermediate devices for interaction with other intermediate devices on the same level

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Memory System Of A Hierarchy Structure (AREA)

Abstract

The invention discloses a data analysis system and a data analysis method based on a distributed caching technology, which relate to the technical field of data analysis and are used for positioning service nodes based on target service data to acquire service node state data; the method comprises the steps of analyzing the service node data request demand degree in real time based on the service node state data, judging the database access state, constructing data cache nodes for each service node according to the database access state judgment result, analyzing cache memory allocation based on the corresponding service node data request demand data, analyzing the real-time cache node memory utilization degree according to the service node period data reading state of the cache nodes, carrying out periodic memory dynamic regulation on the cache nodes, generating regulation and control instructions according to the cache node period memory dynamic regulation and control data and feeding back to a management port, and realizing flexible regulation and analysis on cache demands in different demand scenes in the service node data calling process.

Description

Data analysis system and method based on distributed caching technology
Technical Field
The invention relates to the field of data analysis, in particular to a data analysis system and method based on a distributed caching technology.
Background
The distributed caching technology is a technical scheme for coping with high concurrency and big data scenes, can improve system throughput and remarkably reduce data read-write delay and high concurrency requests by storing data in a plurality of server nodes in a scattered manner, but when the distribution and regulation of caching nodes are single, the caching nodes cannot be adaptively regulated and controlled according to dynamic node data request environments, the capacity regulation and node state regulation of the caching nodes cannot be carried out according to complex data change scenes, and the situation that memory resource waste is caused by unreasonable capacity resource allocation exists in most caching nodes at present is caused.
Disclosure of Invention
The invention aims to provide a data analysis system and a data analysis method based on a distributed caching technology, which are used for solving the problems in the prior art.
In order to achieve the above purpose, the present invention provides the following technical solutions:
A data analysis method based on a distributed caching technique, the method comprising the steps of:
determining target service data, positioning service nodes based on the target service data, and acquiring service node state data;
According to the judging result of the access state of the database, constructing data cache nodes for each service node, and analyzing cache memory allocation based on the data request requirement data of the corresponding service node;
Analyzing the memory utilization degree of the real-time cache node according to the data reading state of the cache node in each service node period, carrying out periodic memory dynamic regulation and control on the cache node according to the memory utilization degree of the real-time cache node, and generating a regulation and control instruction according to the periodic memory dynamic regulation and control data of the cache node and feeding back to the management port.
Further, the service data is searched on the basis of the management port to determine target service data, wherein the service data comprises service name data and service node data, and the service node data is a service node number;
and positioning the service node through the target service node data, and acquiring the state data of the corresponding service node through acquiring the request data of the corresponding service node, wherein the state data of the service node comprises the number of the request data packets and the request data packet capacity of the service node.
Further, based on the state data of each service node of the target service, a corresponding service node state set is constructed, the real-time data request demand of the target service is analyzed according to the service node state set, the real-time data request data capacity of the target service is obtained, and the calculation formula is as follows
;
P (t) is the data request demand capacity of the target service at the current time t, P is the single data packet capacity, mj is the data packet request quantity at the time t of the service node of the corresponding number j, n is the service node quantity of the target service, analysis data are analyzed based on the actual target service data request demand, the real-time access state of the database is judged and analyzed, the maximum response request data packet capacity at the single time of the database is calculated by acquiring the request data packet quantity of the maximum response at the single time of the database, the access state of the database at the current time is determined by comparing the data request demand of the target service at the single time with the maximum response request data packet capacity at the single time of the database, the access state comprises busy and idle, the specific comparison step is that the pre-warning coefficient k is determined for the maximum response request data packet capacity at the single time of the database, the response request data packet capacity at the single time is acquired, the real-time is calculated as P e,s=k*Pmax,s, the maximum response request data packet capacity at the single time of the database is calculated, the current time of the target service data request capacity P (t) is compared, and the current access state is judged as the busy state if the P (t) is more than or equal to the current state of the current access state is judged as busy;
when the access state of the database is busy, constructing cache nodes of each service node of the target service, carrying out capacity prediction analysis on each cache node based on the real-time service data request demand data of each service node to obtain the real-time capacity data of the cache node corresponding to each service node, and calculating as
;
The method comprises the steps of taking C (j, T) as a time T, numbering a capacity prediction value of a corresponding cache node of a service node, taking r as a single data packet cache capacity pre-storage coefficient, wherein the single data packet cache capacity pre-storage coefficient refers to the fact that extra storage space is needed to be carried out when a single data packet is cached and stored, preventing data storage loss, carrying out capacity prediction analysis on each cache node in an observation period through dividing the observation period T1, taking the maximum value as the storage capacity value of the corresponding service node period T1, carrying out type division and overall preparation according to request demand data in the corresponding service node period T1, respectively determining the quantity of various types of request data in the period T1, carrying out analysis on the request data quantity proportion of various types of request data in the period T1, introducing a proportion threshold according to analysis results, comparing the request data quantity proportion of various types of request data with the proportion threshold, determining high request type data and low request type data, otherwise judging the request data type corresponding to be high request type data, judging the request data type corresponding to the cache type data is the low request type data, and the low request type data is used for the first-level cache space and the second-level cache type data storage space, and the first-level cache type data is compared with the low request type data storage space.
Further, based on the capacity prediction analysis and data storage division results of the corresponding cache nodes of each service node of the target service, a cache node update period T2 is set, the calling condition analysis of the data stored by the cache nodes of each service node in the period is analyzed, the utilization degree of the primary cache space and the secondary cache space is analyzed in real time, the calling hit rate of the data stored in the primary cache space and the secondary cache space is analyzed respectively, and the calculation formula is that
;
The hit1 and hit2 correspond to the call hit rate in the data storage period T2 in the first-level storage space and the second-level storage space in the cache nodes of each service node respectively, m (1, T2) and m (1, T2) correspond to the call number in the data storage period T2 in the first-level storage space and the second-level storage space in the cache nodes of each service node respectively, and m (T2) is the call number in the data storage period T2 in the cache nodes of each service node;
Based on the data calling hit rate of the first-level storage space and the second-level storage space in the corresponding cache nodes, the memory occupation amount of the first-level storage space and the second-level storage space in the corresponding cache nodes in the period T2 is analyzed, and the calculation formula is as follows
;
Wherein C (1, j, T2) and C (2, j, T2) are respectively the period memory occupation amount of the first-level memory space and the second-level memory space in the cache nodes corresponding to the service nodes with the numbers j in the period T2, C (1, j) and C (2, j) are respectively the memory capacity of the first-level memory space and the second-level memory space in the cache nodes corresponding to the service nodes with the numbers j, S is a cache fragment rate parameter of the cache nodes, the memory update capacity Cg1 and the memory update capacity Cg2 of the second-level memory space in the cache nodes corresponding to the service nodes are analyzed, the memory occupation amount of the first-level memory space and the second-level memory space in the cache nodes corresponding to the service nodes with the numbers j is set, the memory prompt capacity Cv is set, the memory occupation amount of the first-level memory space and the second-level memory space in the cache nodes corresponding to the continuous time points in the period T2 is updated and judged, the memory capacity regulation strategy of the corresponding to the cache nodes is generated according to the judgment result, wherein when the memory occupation amount of the first-level memory space in the cache nodes corresponding to the first-level memory space is larger than the cache prompt capacity and smaller than or equal to the memory update capacity of the cache nodes in the certain period, the memory occupation amount of the first-level memory space is corresponding to the memory prompt capacity of the cache nodes corresponding to the first-level memory space, the corresponding to the first-level memory space is C (1, j, T2) is the memory capacity of the first-level memory storage space and the second-level memory space is larger than the free data is required to be the free state, and the first-level memory storage capacity is directly or the free data is judged to be the free from the first level storage and the storage capacity is calculated to be the storage and the first level storage and the storage capacity is larger than the storage and the storage is corresponding is larger and the free storage and the storage and is corresponding storage and is larger and is corresponding and a high. The memory regulation is not performed, wherein the calculation formulas of the memory update capacity Cg1 of the primary storage space and the memory update capacity Cg2 of the secondary storage space corresponding to the cache node are as follows
。
Further, the service node state data of the target service and the state data of the corresponding cache node are fed back through a visual window, wherein the cache node state data comprises capacity data of the cache node and data calling records of the response service node;
outputting the generated memory capacity regulating strategy corresponding to each cache node to a management port, and executing the memory capacity regulating strategy corresponding to each cache node.
The system comprises a node positioning module, a node analysis module, a dynamic regulation and control module and a data feedback module;
The node positioning module determines target service data, positions service nodes based on the target service data and acquires service node state data, the node analysis module analyzes service node data request demand degrees in real time based on the service node state data and analyzes data according to the service node data request demand, the node analysis module judges database access states, builds data cache nodes for the service nodes according to the database access state judgment results and analyzes cache memory allocation based on corresponding service node data request demand data, the dynamic regulation module analyzes real-time cache node memory utilization degrees according to data reading states of the cache nodes in each service node period and carries out periodic memory dynamic regulation on the cache nodes according to the real-time cache node memory utilization degrees, a regulation command is generated according to the cache node periodic memory dynamic regulation data and is fed back to the management port, and the data feedback module outputs the service node state data and the corresponding cache node state data by utilizing the visual window and executes the regulation command.
Further, the node positioning module comprises a target service determining unit and a service node positioning unit;
The target service determining unit determines target service data by searching the service data based on the management port, wherein the service data comprises service name data and service node data, and the service node data is a service node number;
The service node positioning unit positions the service node through target service node data, and acquires state data of the corresponding service node through acquiring request data of the corresponding service node, wherein the state data of the service node comprises the number of request data packets and the request data packet capacity of the service node.
Further, the node analysis module comprises a service node demand analysis unit and a cache node construction unit;
The service node demand analysis unit is used for constructing a corresponding service node status set based on each service node status data of a target service, analyzing real-time data request demands of the target service according to the service node status set to obtain real-time data request data capacity of the target service, judging and analyzing a real-time access state of a database based on actual target service data request demand analysis data, calculating the maximum response request data packet capacity of the database at a single moment by obtaining the maximum response request data packet number of the database at the single moment, and determining the access state of the database at the current moment by comparing the target service data request demand analysis data at the single moment with the maximum response request data packet capacity of the database at the single moment, wherein the access state comprises busy and idle;
The method comprises the steps of establishing a cache node for each service node of a target service when an access state of a database is busy, carrying out capacity prediction analysis on each cache node based on real-time service data request demand data of each service node to obtain cache node real-time capacity data corresponding to each service node, carrying out capacity prediction analysis on each cache node in an observation period through dividing the observation period T1 to obtain a maximum value which is a memory value of the cache node period T1 corresponding to the service node, carrying out type division according to the request demand data in the corresponding service node period T1, respectively determining the quantity of each type of request data in the period T1, carrying out analysis on the request data quantity proportion of each type of request data in the period T1, introducing a proportion threshold according to an analysis result, comparing the request data quantity proportion of each type of request data with the proportion threshold, determining high request type data and low request type data, carrying out cache space division on the cache nodes into a first-level cache space and a second-level cache space according to the comparison analysis data of each type of request data in the corresponding service nodes, wherein the first-level cache space is used for storing the high request type data and the second-level request data.
Further, the dynamic regulation and control module comprises a cache node memory analysis unit and a cache node memory regulation and control unit;
The cache node memory analysis unit sets a cache node update period T2 based on capacity prediction analysis and data storage division results of corresponding cache nodes of each service node of target service, analyzes the calling condition of each service node in the period for data stored in the cache nodes, and analyzes the utilization degree of a first-level cache space and a second-level cache space in real time;
the memory regulating and controlling unit of the cache node analyzes data based on the calling condition of each service node for the data stored in the cache node in a cache node updating period T2, dynamically regulates and controls and analyzes the memory of each service node for the cache node, analyzes the memory occupation amount of the corresponding first-level memory space and second-level memory space of the cache node in the period T2 based on the data calling hit rate of the first-level memory space and the second-level memory space of the corresponding cache node respectively, analyzes the memory updating capacity Cg1 and the memory updating capacity Cg2 of the corresponding first-level memory space and the memory occupation amount of the second-level memory space of the corresponding cache node in the cache node of each service node, sets a cache prompting capacity Cv, updates and judges the memory occupation amount of the first-level memory space and the second-level memory space in each cache node at continuous time points in the period, and generates a memory capacity regulating and controlling strategy of the corresponding cache node according to the judging result.
Further, the data feedback module comprises a visualization unit and a strategy execution unit;
The visualization unit feeds back the service node state data of the target service and the state data of the corresponding cache node through a visualization window, wherein the cache node state data comprises capacity data of the cache node and a data calling record of the response service node;
And the strategy executing unit outputs the memory capacity regulating strategy generated by corresponding each cache node to the management port and executes the memory capacity regulating strategy corresponding to each cache node.
Compared with the prior art, the invention has the beneficial effects that:
The application realizes flexible regulation and analysis of the cache demands under different demand scenes in the service node data calling process by carrying out node determination on the target service and a series of node request quantity analysis and database access state analysis and combining cache node construction and cache node memory prediction and regulation analysis, can realize adaptive cache node regulation and control on dynamic node data request environments, and can carry out cache capacity regulation and state regulation on complex data change scenes, thereby improving the condition of memory resource waste caused by unreasonable capacity resource allocation of most cache nodes at present.
Drawings
FIG. 1 is a schematic diagram of a data analysis system based on a distributed caching technique according to the present invention;
fig. 2 is a flow chart of a data analysis method based on a distributed caching technology according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Embodiment as shown in fig. 1, the invention provides a technical scheme that:
the data analysis system based on the distributed caching technology comprises a node positioning module, a node analysis module, a dynamic regulation and control module and a data feedback module;
The node positioning module determines target service data, positions service nodes based on the target service data and acquires service node state data, the node analysis module analyzes service node data request demand degrees in real time based on the service node state data and judges database access states according to the service node data request demand analysis data, a data cache node is built for each service node according to the database access state judgment result and cache memory allocation is analyzed based on corresponding service node data request demand data, the dynamic regulation module analyzes real-time cache node memory utilization degrees according to data reading states of the cache nodes in each service node period and carries out periodic memory dynamic regulation on the cache nodes according to the real-time cache node memory utilization degrees, a regulation command is generated according to the cache node periodic memory dynamic regulation data and fed back to the management port, and the data feedback module outputs the service node state data and the state data of the corresponding cache nodes by using the visual window and executes the regulation command.
Further, the node positioning module comprises a target service determining unit and a service node positioning unit;
The target service determining unit determines target service data by searching the service data based on the management port, wherein the service data comprises service name data and service node data;
The service node positioning unit positions the service node through the target service node data, and acquires the state data of the corresponding service node through acquiring the corresponding service node request data, wherein the state data of the service node comprises the number of the request data packets and the request data packet capacity of the service node.
Further, the node analysis module comprises a service node demand analysis unit and a cache node construction unit;
The service node demand analysis unit is used for constructing a corresponding service node status set based on each service node status data of the target service, analyzing the real-time data request demand of the target service according to the service node status set to obtain the real-time data request data capacity of the target service, judging and analyzing the real-time access status of the database based on the real target service data request demand analysis data, calculating the maximum response request data packet capacity of the database at a single moment by obtaining the maximum response request data packet quantity of the database at a single moment, and determining the access status of the database at the current moment by comparing the target service data request demand analysis data at a single moment with the maximum response request data packet capacity of the database at a single moment;
The method comprises the steps of constructing a cache node for each service node of a target service when an access state of a database is busy, carrying out capacity prediction analysis on each cache node based on real-time service data request demand data of each service node to obtain real-time capacity data of the cache node corresponding to each service node, carrying out capacity prediction analysis on each cache node in an observation period through dividing the observation period T1, taking the maximum value as a memory value of the cache node period T1 of the corresponding service node, carrying out type division and overall according to the request demand data in the corresponding service node period T1, respectively determining the quantity of each type of request data in the period T1, carrying out analysis on the request data quantity proportion of each type of request data in the period T1, introducing a proportion threshold according to an analysis result, comparing the request data quantity proportion of each type of request data with the proportion threshold, determining high request type data and low request type data, carrying out cache space division on the cache nodes into a first-level cache space and a second-level cache space according to comparison analysis data of each type of request data in the corresponding service nodes, wherein the first-level cache space is used for storing the high request type data, and the second-level cache space is used for storing the low request type data.
Further, the dynamic regulation and control module comprises a cache node memory analysis unit and a cache node memory regulation and control unit;
The cache node memory analysis unit sets a cache node update period T2 based on capacity prediction analysis and data storage division results of corresponding cache nodes of each service node of the target service, analyzes the calling condition of the data stored by the cache nodes by each service node in the period, and analyzes the utilization degree of a first-level cache space and a second-level cache space in real time;
The method comprises the steps of analyzing the memory occupation amount of a first-level storage space and a second-level storage space of corresponding cache nodes in a period T2 based on the data of the cache node updating period T2, analyzing the data of the calling condition of each service node for the data stored in the cache nodes, dynamically adjusting and analyzing the memory occupation amount of the cache node by each service node, updating and judging the memory occupation amount of the first-level storage space and the second-level storage space in each cache node at continuous time points in the period according to the judging result, and generating a memory capacity adjusting strategy of the corresponding cache nodes.
Further, the data feedback module comprises a visualization unit and a strategy execution unit;
The visualization unit feeds back the service node state data of the target service and the state data of the corresponding cache node through a visualization window, wherein the cache node state data comprises capacity data of the cache node and a data calling record of the response service node;
The strategy execution unit outputs the generated memory capacity regulation strategy corresponding to each cache node to the management port, and executes the memory capacity regulation strategy corresponding to each cache node;
As shown in fig. 2, the present invention provides another technical solution:
A data analysis method based on a distributed caching technique, the method comprising the steps of:
determining target service data, positioning service nodes based on the target service data, and acquiring service node state data;
According to the judging result of the access state of the database, constructing data cache nodes for each service node, and analyzing cache memory allocation based on the data request requirement data of the corresponding service node;
Analyzing the memory utilization degree of the real-time cache node according to the data reading state of the cache node in each service node period, carrying out periodic memory dynamic regulation and control on the cache node according to the memory utilization degree of the real-time cache node, and generating a regulation and control instruction according to the periodic memory dynamic regulation and control data of the cache node and feeding back to the management port.
Further, the service data is searched on the basis of the management port to determine target service data, wherein the service data comprises service name data and service node data;
And positioning the service node through the target service node data, and acquiring the state data of the corresponding service node through acquiring the request data of the corresponding service node, wherein the state data of the service node comprises the number of the request data packets and the request data packet capacity of the service node.
Further, based on the state data of each service node of the target service, a corresponding service node state set is constructed, the real-time data request demand of the target service is analyzed according to the service node state set, the real-time data request data capacity of the target service is obtained, and the calculation formula is as follows
;
P (t) is the data request demand capacity of the target service at the current time t, P is the single data packet capacity, mj is the data packet request quantity at the time t of the service node of the corresponding number j, n is the service node quantity of the target service, based on the actual target service data request demand analysis data, judgment and analysis are carried out on the real-time access state of the database, the maximum response request data packet capacity of the database at the single time is calculated by acquiring the request data packet quantity of the maximum response of the database at the single time, the access state of the database at the current time is determined by comparing the target service data request demand analysis data at the single time with the maximum response request data packet capacity of the database at the single time, the access state comprises busy and idle, the specific comparison steps are that the request data packet capacity P e,s of the database at the single time is acquired by carrying out early warning coefficient k determination on the maximum response request data packet capacity of the database at the single time, P e,s=k*Pmax,s is calculated, wherein P max,s is the maximum response request data packet capacity at the single time, the current time is compared with the request data capacity P (t) at the maximum response request data packet capacity at the single time, and if P (t) is more than or equal to the current P e,s, the current access state is judged to be busy, otherwise, the access state of the database is judged to be idle;
when the access state of the database is busy, constructing cache nodes of each service node of the target service, carrying out capacity prediction analysis on each cache node based on the real-time service data request demand data of each service node to obtain the real-time capacity data of the cache node corresponding to each service node, and calculating as
;
The method comprises the steps of taking a time C (j, T) as a time T, numbering a service node according to a capacity prediction value of the corresponding buffer node, taking r as a single data packet buffer capacity pre-storage coefficient, comparing a request data volume ratio of each type of request data with a ratio threshold according to analysis results, determining high request type data and low request type data, judging the request data type of each buffer node in an observation period as high request type data by dividing the observation period T1, taking the maximum value as the content value of the buffer node period T1 of the corresponding service node, carrying out type division and overall formulation according to request data in the corresponding service node period T1, respectively determining the quantity of each type of request data in the period T1, analyzing the request data volume ratio of each type of request data in the period T1, introducing the ratio threshold according to analysis results, comparing the request data volume ratio of each type of request data with the ratio threshold, and determining the high request type data and the low request type data, otherwise judging the request type data with the ratio of the ratio threshold, and judging the request data type of each type of request data in the corresponding type of request data as low request type data, and the buffer node is used for the first-level buffer space storage of the low request type data.
Further, based on the capacity prediction analysis and data storage division results of the corresponding cache nodes of each service node of the target service, a cache node update period T2 is set, the calling condition analysis of the data stored by the cache nodes of each service node in the period is analyzed, the utilization degree of the primary cache space and the secondary cache space is analyzed in real time, the calling hit rate of the data stored in the primary cache space and the secondary cache space is analyzed respectively, and the calculation formula is that
;
The hit1 and hit2 correspond to the call hit rate in the data storage period T2 in the first-level storage space and the second-level storage space in the cache nodes of each service node respectively, m (1, T2) and m (1, T2) correspond to the call number in the data storage period T2 in the first-level storage space and the second-level storage space in the cache nodes of each service node respectively, and m (T2) is the call number in the data storage period T2 in the cache nodes of each service node;
Based on the data calling hit rate of the first-level storage space and the second-level storage space in the corresponding cache nodes, the memory occupation amount of the first-level storage space and the second-level storage space in the corresponding cache nodes in the period T2 is analyzed, and the calculation formula is as follows
;
Wherein C (1, j, T2) and C (2, j, T2) are respectively the period memory occupation amount of the first-level memory space and the second-level memory space in the cache nodes corresponding to the service nodes with the numbers j in the period T2, C (1, j) and C (2, j) are respectively the memory capacity of the first-level memory space and the second-level memory space in the cache nodes corresponding to the service nodes with the numbers j, S is a cache fragment rate parameter of the cache nodes, the memory update capacity Cg1 and the memory update capacity Cg2 of the second-level memory space in the cache nodes corresponding to the service nodes are analyzed, the memory occupation amount of the first-level memory space and the second-level memory space in the cache nodes corresponding to the service nodes with the numbers j is set, the memory prompt capacity Cv is set, the memory occupation amount of the first-level memory space and the second-level memory space in the cache nodes corresponding to the continuous time points in the period T2 is updated and judged, the memory capacity regulation strategy of the corresponding to the cache nodes is generated according to the judgment result, wherein when the memory occupation amount of the first-level memory space in the cache nodes corresponding to the first-level memory space is larger than the cache prompt capacity and smaller than or equal to the memory update capacity of the cache nodes in the certain period, the memory occupation amount of the first-level memory space is corresponding to the memory prompt capacity of the cache nodes corresponding to the first-level memory space, the corresponding to the first-level memory space is C (1, j, T2) is the memory capacity of the first-level memory storage space and the second-level memory space is larger than the free data is required to be the free state, and the first-level memory storage capacity is directly or the free data is judged to be the free from the first level storage and the storage capacity is calculated to be the storage and the first level storage and the storage capacity is larger than the storage and the storage is corresponding is larger and the free storage and the storage and is corresponding storage and is larger and is corresponding and a high. The memory regulation is not performed, wherein the calculation formulas of the memory update capacity Cg1 of the primary storage space and the memory update capacity Cg2 of the secondary storage space corresponding to the cache node are as follows
。
Further, the service node state data of the target service and the state data of the corresponding cache node are fed back through a visual window, wherein the cache node state data comprises capacity data of the cache node and data calling records of the response service node;
outputting the generated memory capacity regulating strategy corresponding to each cache node to a management port, and executing the memory capacity regulating strategy corresponding to each cache node.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.

Claims (10)

1.一种基于分布式缓存技术的数据分析方法，其特征在于：该方法包括以下步骤：1. A data analysis method based on distributed caching technology, characterized in that the method comprises the following steps: 确定目标业务数据，基于目标业务数据对业务节点进行定位，并获取业务节点状态数据；Determine target business data, locate business nodes based on the target business data, and obtain business node status data; 基于各业务节点状态数据实时分析业务节点数据请求需求程度，并根据业务节点数据请求需求程度分析数据判断数据库访问状态；根据数据库访问状态判断结果，对各业务节点构建数据缓存节点，并基于对应业务节点数据请求需求程度分析缓存内存分配；Analyze the data request demand level of each business node in real time based on the status data of each business node, and judge the database access status based on the data analysis of the business node data request demand level; build a data cache node for each business node based on the database access status judgment result, and analyze the cache memory allocation based on the corresponding business node data request demand level; 根据各业务节点周期对缓存节点的数据读取状态，分析实时缓存节点内存利用程度，并根据实时缓存节点内存利用程度对缓存节点进行周期内存动态调控；根据缓存节点周期内存动态调控数据生成调控指令反馈至管理端口；Analyze the real-time cache node memory utilization based on the data reading status of each business node periodically, and dynamically adjust the cache node memory periodically based on the real-time cache node memory utilization; generate adjustment instructions based on the cache node periodic memory dynamic adjustment data and feed them back to the management port; 进一步的，基于目标业务的各业务节点状态数据，构建对应业务节点状态集合；根据业务节点状态集合对目标业务实时的数据请求需求进行分析，获取目标业务实时的数据请求数据容量；其计算公式为：Furthermore, based on the status data of each service node of the target service, a corresponding service node status set is constructed; the real-time data request requirements of the target service are analyzed according to the service node status set to obtain the real-time data request data capacity of the target service; the calculation formula is: P(t)为当前t时刻目标业务的数据请求需求容量；p为单数据包容量；Mj为对应编号j业务节点t时刻的数据包请求数量；n为目标业务的业务节点数量；基于实际目标业务数据请求需求分析数据，对数据库的实时访问状态进行判断分析；通过获取数据库单时刻最大响应的请求数据包数量，对数据库单时刻最大响应请求数据包容量进行计算；通过比较单时刻目标业务数据请求需求分析数据与数据库单时刻最大响应请求数据包容量，确定当前时刻数据库的访问状态；访问状态包括繁忙和空闲；其具体比较步骤为，基于数据库单时刻最大响应请求数据包容量以及预警系数k，计算单时刻数据库响应请求数据包预警容量Pe,s，具体为Pe,s＝k*Pmax,s；其中Pmax,s为数据库单时刻最大响应请求数据包容量；与当前时刻目标业务数据请求需求容量P(t)进行比对，若P(t)≥Pe,s，则判断当前数据库访问状态为繁忙；反之则判断当前数据库访问状态为空闲。P(t) is the data request demand capacity of the target service at the current moment t; p is the capacity of a single data packet; Mj is the number of data packet requests corresponding to service node number j at moment t; and n is the number of service nodes of the target service. Based on the actual target service data request demand analysis data, the real-time access status of the database is judged and analyzed. By obtaining the maximum number of request packets that the database responds to at a single moment, the maximum capacity of the database's single-moment response request packet is calculated. The current database access status is determined by comparing the single-moment target service data request demand analysis data with the database's single-moment maximum capacity of the database's single-moment response request packet. Access status includes busy and idle. The specific comparison step is to calculate the single-moment database response request packet warning capacity P e,s based on the database's single-moment maximum capacity of the response request packet and the warning coefficient k, specifically P e,s = k*P max,s , where P max,s is the database's single-moment maximum capacity of the response request packet. This is compared with the current target service data request demand capacity P(t). If P(t) ≥ P e,s , the current database access status is determined to be busy; otherwise, the current database access status is determined to be idle. 2.根据权利要求1所述的一种基于分布式缓存技术的数据分析方法，其特征在于：2. The data analysis method based on distributed caching technology according to claim 1, characterized in that: 基于管理端口通过对业务数据进行检索确定目标业务数据；所述业务数据包括业务名称数据和业务节点数据；所述业务节点数据为业务节点编号；Determine target business data by searching business data based on the management port; the business data includes business name data and business node data; the business node data is the business node number; 通过目标业务节点数据对业务节点进行定位，通过获取对应业务节点请求数据获取对应业务节点的状态数据；所述业务节点的状态数据包括业务节点的请求数据包数量和请求数据包容量。The service node is located by the target service node data, and the status data of the corresponding service node is obtained by obtaining the corresponding service node request data; the status data of the service node includes the number of request data packets and the capacity of the request data packets of the service node. 3.根据权利要求2所述的一种基于分布式缓存技术的数据分析方法，其特征在于：3. The data analysis method based on distributed caching technology according to claim 2, characterized in that: 基于目标业务的各业务节点状态数据，构建对应业务节点状态集合；根据业务节点状态集合对目标业务实时的数据请求需求进行分析，获取目标业务实时的数据请求数据容量；基于实际目标业务数据请求需求分析数据，对数据库的实时访问状态进行判断分析；基于数据库单时刻最大响应的请求数据包数量，对单时刻数据库响应请求数据包预警容量进行计算；通过比较单时刻目标业务数据请求需求分析数据与数据库单时刻最大响应请求数据包预警容量，确定当前时刻数据库的访问状态；所述访问状态包括繁忙和空闲；Based on the status data of each business node of the target business, a corresponding business node status set is constructed; the real-time data request demand of the target business is analyzed according to the business node status set to obtain the real-time data request data capacity of the target business; based on the actual target business data request demand analysis data, the real-time access status of the database is judged and analyzed; based on the maximum number of request data packets responded by the database at a single moment, the single-moment database response request data packet warning capacity is calculated; by comparing the single-moment target business data request demand analysis data with the single-moment database maximum response request data packet warning capacity, the current moment database access status is determined; the access status includes busy and idle; 当数据库的访问状态为繁忙，则对目标业务的各业务节点进行缓存节点构建；基于各业务节点实时业务数据请求需求数据对各缓存节点进行容量预测分析，获取对应各业务节点的缓存节点实时容量数据；其计算为：When the database access status is busy, cache nodes are constructed for each business node of the target business. Capacity forecast analysis is performed on each cache node based on the real-time business data request demand data of each business node to obtain the real-time capacity data of the cache node corresponding to each business node. The calculation is: C(j,t)＝Mj*p*(1+r)；C(j,t)＝ Mj *p*(1+r); 其中，C(j,t)为t时刻，编号j业务节点对应缓存节点的容量预测值；r为单数据包缓存容量预存系数；单数据包缓存容量预存系数指对单数据包进行缓存存储时需要对存储空间进行额外存储空间，防止数据存储丢失；Where C(j,t) is the predicted capacity of the cache node corresponding to service node number j at time t; r is the single data packet cache capacity pre-storage coefficient; the single data packet cache capacity pre-storage coefficient refers to the additional storage space required for caching a single data packet to prevent data loss; 通过划分观测周期T1，对观测周期内各缓存节点进行容量预测分析进行统筹，取最大值为对应业务节点的缓存节点周期T1内存容量值；根据对应业务节点周期T1内请求需求数据进行类型划分统筹，分别确定周期T1内各类型请求数据数量，并对周期T1内各类型请求数据的请求数据量占比进行分析；根据分析结果引入占比阈值，将各类型请求数据的请求数据量占比与占比阈值进行对比，确定高请求类型数据和低请求类型数据；根据对应业务节点中各类型请求数据的对比分析数据，对缓存节点进行缓存空间划分为一级缓存空间和二级缓存空间；其中所述一级缓存空间用于存储高请求类型数据；所述二级缓存空间用于存储低请求类型数据。By dividing the observation period T1, the capacity prediction analysis of each cache node in the observation period is coordinated, and the maximum value is taken as the cache node period T1 memory capacity value of the corresponding business node; according to the request demand data in the corresponding business node period T1, the type is divided and coordinated, and the number of each type of request data in the period T1 is determined respectively, and the proportion of the request data volume of each type of request data in the period T1 is analyzed; according to the analysis result, a proportion threshold is introduced, and the proportion of the request data volume of each type of request data is compared with the proportion threshold to determine high request type data and low request type data; according to the comparative analysis data of each type of request data in the corresponding business node, the cache space of the cache node is divided into a first-level cache space and a second-level cache space; wherein the first-level cache space is used to store high request type data; the second-level cache space is used to store low request type data. 4.根据权利要求3所述的一种基于分布式缓存技术的数据分析方法，其特征在于：4. The data analysis method based on distributed caching technology according to claim 3, characterized in that: 基于目标业务的各业务节点对应缓存节点的容量预测分析和数据存储划分结果，设置缓存节点更新周期T2，对周期内各业务节点对缓存节点存储数据的调用情况分析，实时分析一级缓存空间和二级缓存空间的利用程度；其通过分别对一级缓存空间和二级缓存空间中存储的数据调用命中率进行分析；Based on the capacity forecast analysis and data storage partitioning results of the cache nodes corresponding to each business node of the target business, a cache node update cycle T2 is set. The call status of each business node to the cache node stored data within the cycle is analyzed, and the utilization of the first-level cache space and the second-level cache space is analyzed in real time. This is done by analyzing the call hit rate of the data stored in the first-level cache space and the second-level cache space respectively. 其计算公式为The calculation formula is 其中，hit1和hit2分别对应各业务节点的缓存节点中一级存储空间和二级存储空间中存储数据周期T2内调用命中率；m(1,T2)和m(2,T2)分别对应各业务节点的缓存节点中一级存储空间和二级存储空间中存储数据周期T2内调用数量；m(T2)为对应各业务节点的缓存节点中存储数据周期T2内调用数量；Among them, hit1 and hit2 correspond to the call hit rates of the primary storage space and secondary storage space in the cache node of each business node within the storage data period T2; m(1, T2) and m(2, T2) correspond to the number of calls to the primary storage space and secondary storage space in the cache node of each business node within the storage data period T2; m(T2) is the number of calls to the cache node of each business node within the storage data period T2; 基于缓存节点更新周期T2内，各业务节点对缓存节点存储数据的调用情况分析数据，对各业务节点对缓存节点存储内存进行动态调控分析；其分别基于对应缓存节点中一级存储空间和二级存储空间的数据调用命中率，对周期T2内对应缓存节点一级存储空间和二级存储空间的内存占用量进行分析；其计算公式为：Based on the analysis data of the call status of each business node to the cache node storage data within the cache node update cycle T2, the dynamic regulation and analysis of each business node's use of the cache node storage memory is performed; based on the data call hit rate of the first-level storage space and second-level storage space in the corresponding cache node, the memory usage of the first-level storage space and second-level storage space of the corresponding cache node within the cycle T2 is analyzed; the calculation formula is: 其中，C(1,j,T2)和C(2,j,T2)分别为周期T2内对应编号j业务节点的缓存节点中一级存储空间和二级存储空间的周期内存占用量；C(1,j)和C(2,j)分别对应编号j业务节点的缓存节点中一级存储空间和二级存储空间的内存容量；S为缓存节点的缓存碎片率参数；通过分析各业务节点的缓存节点中对应一级存储空间内存更新容量Cg1和二级存储空间内存更新容量Cg2，并设置缓存提示容量Cv，对周期内连续时间点各缓存节点中一级存储空间和二级存储空间的内存占用量进行更新判断，根据判断结果生成对应缓存节点的内存容量调控策略；其中，当某周期内所有连续时间点，缓存节点中对应一级存储空间内存占用量大于缓存提示容量且小于等于缓存节点内存更新容量，其对应为Cv＜C(1,j,T2)≤Cg1，则将对应一级存储空间内存容量调控为缓存节点内存更新容量Cg；若C(1,j,T2)≤Cv，则表示当前一级存储空间为空闲状态；依据上述判断方法，对二级存储空间占用量进行判断并调控内存容量；其中若某缓存节点的一级存储空间和二级存储空间均为空闲状态，则生成缓存节点休眠指令，则业务节点请求数据直接与数据库进行数据调用；而当一级存储空间或二级存储空间存在大于缓存节点更新容量时，则不进行内存调控；其中缓存节点对应一级存储空间内存更新容量Cg1和二级存储空间内存更新容量Cg2的计算公式为Among them, C(1,j,T2) and C(2,j,T2) are the periodic memory occupancy of the primary storage space and the secondary storage space in the cache node corresponding to the business node numbered j in period T2; C(1,j) and C(2,j) are the memory capacities of the primary storage space and the secondary storage space in the cache node corresponding to the business node numbered j respectively; S is the cache fragmentation rate parameter of the cache node; by analyzing the memory update capacity Cg1 of the primary storage space and the memory update capacity Cg2 of the secondary storage space in the cache node of each business node, and setting the cache prompt capacity Cv, the memory occupancy of the primary storage space and the secondary storage space in each cache node at continuous time points in the period is updated and judged, and the memory capacity control strategy of the corresponding cache node is generated according to the judgment result; among them, when the memory capacity of the corresponding primary storage space in the cache node at all continuous time points in a certain period is The memory occupancy is greater than the cache prompt capacity and less than or equal to the cache node memory update capacity, which corresponds to Cv＜C(1,j,T2)≤Cg1, then the corresponding primary storage space memory capacity is adjusted to the cache node memory update capacity Cg; if C(1,j,T2)≤Cv, it means that the current primary storage space is idle; according to the above judgment method, the secondary storage space occupancy is judged and the memory capacity is adjusted; if the primary storage space and the secondary storage space of a cache node are both idle, a cache node sleep instruction is generated, and the business node requests data directly with the database for data call; and when the primary storage space or the secondary storage space is greater than the cache node update capacity, no memory adjustment is performed; the calculation formula for the primary storage space memory update capacity Cg1 and the secondary storage space memory update capacity Cg2 corresponding to the cache node is 5.根据权利要求4所述的一种基于分布式缓存技术的数据分析方法，其特征在于：5. The data analysis method based on distributed caching technology according to claim 4, characterized in that: 通过可视化窗口将目标业务的业务节点状态数据和对应缓存节点的状态数据进行反馈；所述缓存节点状态数据包括缓存节点的容量数据和响应业务节点的数据调取记录；Feedback of the target service's service node status data and the corresponding cache node status data through a visualization window; the cache node status data includes the cache node's capacity data and the data retrieval record of the responding service node; 将对应各缓存节点的生成的内存容量调控策略输出管理端口，执行对应各缓存节点内存容量调控策略。The memory capacity control policy generated for each cache node is output to the management port, and the memory capacity control policy for each cache node is executed. 6.一种基于分布式缓存技术的数据分析系统，用于实现如权利要求1-5中任一项所述的一种基于分布式缓存技术的数据分析方法，其特征在于：所述系统包括节点定位模块、节点分析模块、动态调控模块和数据反馈模块；6. A data analysis system based on distributed caching technology, used to implement the data analysis method based on distributed caching technology according to any one of claims 1 to 5, characterized in that: the system includes a node positioning module, a node analysis module, a dynamic control module, and a data feedback module; 所述节点定位模块确定目标业务数据，基于目标业务数据对业务节点进行定位，并获取业务节点状态数据；所述节点分析模块基于各业务节点状态数据实时分析业务节点数据请求需求程度，并根据业务节点数据请求需求程度分析数据判断数据库访问状态；根据数据库访问状态判断结果，对各业务节点构建数据缓存节点，并基于对应业务节点数据请求需求程度分析缓存内存分配；所述动态调控模块根据各业务节点周期对缓存节点的数据读取状态，分析实时缓存节点内存利用程度，并根据实时缓存节点内存利用程度对缓存节点进行周期内存动态调控；根据缓存节点周期内存动态调控数据生成调控指令反馈至管理端口；所述数据反馈模块利用可视化窗口输出业务节点状态数据和对应缓存节点的状态数据并执行调控指令。The node positioning module determines the target business data, locates the business node based on the target business data, and obtains the business node status data; the node analysis module analyzes the business node data request demand level in real time based on the business node status data, and judges the database access status based on the business node data request demand level analysis data; according to the database access status judgment result, a data cache node is constructed for each business node, and the cache memory allocation is analyzed based on the corresponding business node data request demand level; the dynamic control module analyzes the real-time cache node memory utilization level based on the data reading status of the cache node of each business node cycle, and performs periodic memory dynamic control on the cache node based on the real-time cache node memory utilization level; generates control instructions based on the cache node periodic memory dynamic control data and feeds them back to the management port; the data feedback module uses a visualization window to output the business node status data and the status data of the corresponding cache node and executes the control instructions. 7.根据权利要求6所述的一种基于分布式缓存技术的数据分析系统，其特征在于：所述节点定位模块包括目标业务确定单元和业务节点定位单元；7. A data analysis system based on distributed cache technology according to claim 6, characterized in that: the node positioning module includes a target service determination unit and a service node positioning unit; 所述目标业务确定单元基于管理端口通过对业务数据进行检索确定目标业务数据；所述业务数据包括业务名称数据和业务节点数据；所述业务节点数据为业务节点编号；The target service determination unit determines target service data by retrieving service data based on the management port; the service data includes service name data and service node data; the service node data is a service node number; 所述业务节点定位单元通过目标业务节点数据对业务节点进行定位，通过获取对应业务节点请求数据获取对应业务节点的状态数据；所述业务节点的状态数据包括业务节点的请求数据包数量和请求数据包容量。The service node positioning unit locates the service node through the target service node data, and obtains the status data of the corresponding service node by obtaining the corresponding service node request data; the status data of the service node includes the number of request data packets and the capacity of the request data packets of the service node. 8.根据权利要求7所述的一种基于分布式缓存技术的数据分析系统，其特征在于：所述节点分析模块包括业务节点需求分析单元和缓存节点构建单元；8. A data analysis system based on distributed cache technology according to claim 7, characterized in that: the node analysis module includes a service node demand analysis unit and a cache node construction unit; 所述业务节点需求分析单元基于目标业务的各业务节点状态数据，构建对应业务节点状态集合；根据业务节点状态集合对目标业务实时的数据请求需求进行分析，获取目标业务实时的数据请求数据容量；基于实际目标业务数据请求需求分析数据，对数据库的实时访问状态进行判断分析；基于数据库单时刻最大响应的请求数据包数量，对数据库单时刻最大响应请求数据包预警容量进行计算；通过比较单时刻目标业务数据请求需求分析数据与数据库单时刻最大响应请求数据包预警容量，确定当前时刻数据库的访问状态；所述访问状态包括繁忙和空闲；The service node demand analysis unit constructs a corresponding service node state set based on the status data of each service node of the target service; analyzes the real-time data request demand of the target service according to the service node state set to obtain the real-time data request data capacity of the target service; judges and analyzes the real-time access state of the database based on the actual target service data request demand analysis data; calculates the maximum single-time response request data packet warning capacity of the database based on the maximum number of request data packets responded by the database at a single moment; determines the access state of the database at the current moment by comparing the single-time target service data request demand analysis data with the maximum single-time response request data packet warning capacity of the database; the access state includes busy and idle; 所述缓存节点构建单元当数据库的访问状态为繁忙，则对目标业务的各业务节点进行缓存节点构建；基于各业务节点实时业务数据请求需求数据对各缓存节点进行容量预测分析，获取对应各业务节点的缓存节点实时容量数据；通过划分观测周期T1，对观测周期内各缓存节点进行容量预测分析进行统筹，取最大值为对应业务节点的缓存节点周期T1内存容量值；根据对应业务节点周期T1内请求需求数据进行类型划分统筹，分别确定周期T1内各类型请求数据数量，并对周期T1内各类型请求数据的请求数据量占比进行分析；根据分析结果引入占比阈值，将各类型请求数据的请求数据量占比与占比阈值进行对比，确定高请求类型数据和低请求类型数据；根据对应业务节点中各类型请求数据的对比分析数据，对缓存节点进行缓存空间划分为一级缓存空间和二级缓存空间；其中所述一级缓存空间用于存储高请求类型数据；所述二级缓存空间用于存储低请求类型数据。When the access status of the database is busy, the cache node construction unit constructs a cache node for each business node of the target business; performs capacity prediction analysis on each cache node based on the real-time business data request demand data of each business node, and obtains the real-time capacity data of the cache node corresponding to each business node; divides the observation period T1, coordinates the capacity prediction analysis of each cache node within the observation period, and takes the maximum value as the cache node period T1 memory capacity value of the corresponding business node; divides and coordinates the types according to the request demand data within the corresponding business node period T1, determines the number of each type of request data within the period T1, and analyzes the proportion of the request data volume of each type of request data within the period T1; introduces a proportion threshold based on the analysis result, compares the request data volume proportion of each type of request data with the proportion threshold, and determines high request type data and low request type data; divides the cache space of the cache node into a first-level cache space and a second-level cache space based on the comparative analysis data of each type of request data in the corresponding business node; wherein the first-level cache space is used to store high request type data; the second-level cache space is used to store low request type data. 9.根据权利要求8所述的一种基于分布式缓存技术的数据分析系统，其特征在于：所述动态调控模块包括缓存节点内存分析单元和缓存节点内存调控单元；9. A data analysis system based on distributed cache technology according to claim 8, characterized in that: the dynamic control module includes a cache node memory analysis unit and a cache node memory control unit; 所述缓存节点内存分析单元基于目标业务的各业务节点对应缓存节点的容量预测分析和数据存储划分结果，设置缓存节点更新周期T2，对周期内各业务节点对缓存节点存储数据的调用情况分析，实时分析一级缓存空间和二级缓存空间的利用程度；其通过分别对一级缓存空间和二级缓存空间中存储的数据调用命中率进行分析；The cache node memory analysis unit sets a cache node update period T2 based on the capacity prediction analysis and data storage partitioning results of the cache nodes corresponding to each business node of the target business, analyzes the call status of each business node to the cache node stored data within the period, and analyzes the utilization level of the first-level cache space and the second-level cache space in real time; it analyzes the call hit rate of the data stored in the first-level cache space and the second-level cache space respectively; 所述缓存节点内存调控单元基于缓存节点更新周期T2内，各业务节点对缓存节点存储数据的调用情况分析数据，对各业务节点对缓存节点存储内存进行动态调控分析；其分别基于对应缓存节点中一级存储空间和二级存储空间的数据调用命中率，对周期T2内对应缓存节点一级存储空间和二级存储空间的内存占用量进行分析；通过分析各业务节点的缓存节点中对应一级存储空间内存更新容量Cg1和二级存储空间内存更新容量Cg2，并设置缓存提示容量Cv，对周期内连续时间点各缓存节点中一级存储空间和二级存储空间的内存占用量进行更新判断，根据判断结果生成对应缓存节点的内存容量调控策略。The cache node memory control unit dynamically controls and analyzes the cache node storage memory of each business node based on the call analysis data of the cache node storage data of each business node within the cache node update cycle T2; it analyzes the memory occupancy of the first-level storage space and the second-level storage space of the corresponding cache node within the cycle T2 based on the data call hit rate of the first-level storage space and the second-level storage space in the corresponding cache node respectively; by analyzing the corresponding first-level storage space memory update capacity Cg1 and the second-level storage space memory update capacity Cg2 in the cache node of each business node, and setting the cache prompt capacity Cv, it updates and judges the memory occupancy of the first-level storage space and the second-level storage space in each cache node at continuous time points within the cycle, and generates a memory capacity control strategy for the corresponding cache node based on the judgment result. 10.根据权利要求9所述的一种基于分布式缓存技术的数据分析系统，其特征在于：所述数据反馈模块包括可视化单元和策略执行单元；10. A data analysis system based on distributed cache technology according to claim 9, characterized in that: the data feedback module includes a visualization unit and a strategy execution unit; 所述可视化单元通过可视化窗口将目标业务的业务节点状态数据和对应缓存节点的状态数据进行反馈；所述缓存节点状态数据包括缓存节点的容量数据和响应业务节点的数据调取记录；The visualization unit feeds back the service node status data of the target service and the status data of the corresponding cache node through the visualization window; the cache node status data includes the capacity data of the cache node and the data retrieval record of the response service node; 所述策略执行单元将对应各缓存节点生成的内存容量调控策略输出管理端口，执行对应各缓存节点内存容量调控策略。The policy execution unit outputs the memory capacity control policy generated for each cache node to the management port, and executes the memory capacity control policy for each cache node.
CN202510779687.1A 2025-06-12 2025-06-12 Data analysis system and method based on distributed caching technology Active CN120386801B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202510779687.1A CN120386801B (en) 2025-06-12 2025-06-12 Data analysis system and method based on distributed caching technology

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202510779687.1A CN120386801B (en) 2025-06-12 2025-06-12 Data analysis system and method based on distributed caching technology

Publications (2)

Publication Number Publication Date
CN120386801A CN120386801A (en) 2025-07-29
CN120386801B true CN120386801B (en) 2025-10-14

Family

ID=96489760

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202510779687.1A Active CN120386801B (en) 2025-06-12 2025-06-12 Data analysis system and method based on distributed caching technology

Country Status (1)

Country Link
CN (1) CN120386801B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119652907A (en) * 2025-02-18 2025-03-18 江苏智檬智能科技有限公司 A distributed cache data control security management system and method
CN119719232A (en) * 2025-03-04 2025-03-28 浙江爱客智能科技有限责任公司 Distributed database management method and system based on artificial intelligence

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107093082A (en) * 2017-04-21 2017-08-25 北京恒冠网络数据处理有限公司 The Data Collection and management method of a kind of technical transaction platform
US11586630B2 (en) * 2020-02-27 2023-02-21 Sap Se Near-memory acceleration for database operations
CN114428796A (en) * 2022-01-26 2022-05-03 麒麟合盛网络技术股份有限公司 A data acquisition method and device
US12417136B2 (en) * 2023-03-24 2025-09-16 AtomBeam Technologies Inc. System and method for adaptive protocol caching in event-driven data communication networks

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119652907A (en) * 2025-02-18 2025-03-18 江苏智檬智能科技有限公司 A distributed cache data control security management system and method
CN119719232A (en) * 2025-03-04 2025-03-28 浙江爱客智能科技有限责任公司 Distributed database management method and system based on artificial intelligence

Also Published As

Publication number Publication date
CN120386801A (en) 2025-07-29

Similar Documents

Publication Publication Date Title
CN110166282B (en) Resource allocation method, device, computer equipment and storage medium
CN107404409B (en) Prediction method and system of container cloud elastic supply container quantity for sudden load
CN118012906A (en) Multi-level cache adaptive system and strategy based on machine learning
CN119938942B (en) A method for generating distributed data storage network based on knowledge graph
CN119862907B (en) A hybrid expert model training optimization method based on matrix routing and token allocation
CN119271398B (en) Heterogeneous computing power resource allocation optimization method for deep reinforcement learning model training
CN119691305A (en) Intelligent caching method, device, equipment and medium for front-end resource loading as required
CN115981863A (en) Intelligent cloud resource elastic expansion method and system combining business characteristics
CN120386801B (en) Data analysis system and method based on distributed caching technology
CN111629216B (en) VOD service cache replacement method based on random forest algorithm under edge network environment
CN115190135B (en) Distributed storage system and copy selection method thereof
CN114785856A (en) Edge calculation-based collaborative caching method, device, equipment and storage medium
CN110535894A (en) A method and system for dynamic allocation of container resources based on load feedback
CN111143411A (en) Dynamic streaming pre-calculation method and device and storage medium
CN111190737A (en) Memory allocation method for embedded system
CN110944050B (en) Reverse proxy server cache dynamic configuration method and system
CN117407921A (en) Differential privacy histogram publishing method and system based on must-connect and do-not-connect constraints
CN116541147A (en) Heterogeneous multi-core task scheduling method and system based on improved whale optimization algorithm
Kim et al. T-cachenet: Transformer-based deep reinforcement learning for next-generation internet content caching
CN114882713A (en) Multi-scene-based signal control method, system, device and storage medium
CN116644047B (en) A Spark-based file size adaptive processing method and system
CN121070629B (en) Load balancing method and system for low-power-consumption AI processor
CN120448054B (en) Scheduling methods, inference methods, devices and electronic equipment for large model parameters
CN120653451B (en) A method, apparatus, device, medium, and program product for cache allocation.
CN113742383B (en) Data storage method, device, equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant