CN119106005A - A computer device based on big data processing - Google Patents
A computer device based on big data processing Download PDFInfo
- Publication number
- CN119106005A CN119106005A CN202411562076.3A CN202411562076A CN119106005A CN 119106005 A CN119106005 A CN 119106005A CN 202411562076 A CN202411562076 A CN 202411562076A CN 119106005 A CN119106005 A CN 119106005A
- Authority
- CN
- China
- Prior art keywords
- unit
- data
- user
- cache
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/168—Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/31—User authentication
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/604—Tools and structures for managing or administering access control systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Human Computer Interaction (AREA)
- Automation & Control Theory (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a computer device based on big data processing, which comprises a data processing and storage module, a cache management module, a data verification module, an access control and authority management module, a user interaction module and a security module, wherein the data processing and storage module is responsible for receiving, processing, analyzing and storing data, the cache management module is responsible for managing cache data and reducing direct access to a database, the data verification module is responsible for carrying out multiple verification before deleting the data, the access control and authority management module is responsible for managing the access authority of a user, the user interaction module is responsible for providing a user interface for interacting with the user, and the security module is responsible for remote monitoring and management and data backup and recovery.
Description
Technical Field
The invention relates to the technical field of intelligent monitoring, in particular to a computer device based on big data processing.
Background
With the rapid development and popularization of information technology, computers have become an indispensable tool in modern society, and are widely applied to various fields of education, entertainment, work and the like. With the rise in video resolution (e.g., 4K, 8K video) and the popularity of streaming media services, users tend to download more high quality video for offline viewing. These video files occupy a large amount of memory. Computer devices based on big data processing technology have been developed. The devices realize the effective management and the efficient utilization of massive video data by integrating advanced storage technology, optimizing a data processing algorithm and constructing an efficient data management system.
Through retrieval, the invention patent with the Chinese patent number of CN115510272A discloses a computer data processing system based on big data analysis, and belongs to the technical field of computers. The processing system comprises a storage space analysis module, a target data selection module and a state information analysis module, wherein the storage space analysis module acquires video download data stored by a current computer as analysis data, if the storage space occupied by the analysis data is larger than a storage space threshold value, the target data selection module is enabled to analyze the analysis data, and if the time interval between the time when a certain video download data is triggered last time and the current time in the analysis data is longer than a time length threshold value, the video download data is the target data, and the state information analysis module analyzes the state information of the target data and judges whether the target data is to be deleted.
Compared with the prior art, the invention patent with the Chinese patent number of CN115510272A judges the probability that a subsequent user looks at the video by analyzing the video which is not watched for a long time, and directly deletes the video under the condition of lower probability, thereby reducing the occupation of idle video data to the storage space of a computer, ensuring the normal operation of the computer and improving the operation efficiency of the computer.
However, in the use of the above device, although the system designs complex analysis logic to determine which data should be deleted, there is still a possibility of erroneous determination that some users may occasionally watch some videos, but these videos are still important to them and should not be deleted, and thus a computer device based on big data processing is proposed.
Disclosure of Invention
The present invention aims to solve the drawbacks of the prior art that although the system designs complex analysis logic to determine which data should be deleted, there is still the possibility of misjudgment, some users may occasionally watch some videos, but these videos are still important to them and should not be deleted, and a computer device based on big data processing is proposed.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a big data processing based computer device, comprising:
the data processing and storing module is used for receiving, processing, analyzing and storing data;
The cache management module is responsible for managing cache data and reducing direct access to a database;
the data verification module is responsible for carrying out multiple verification before deleting the data, so as to ensure the accuracy and importance of the data;
the access control and authority management module is responsible for managing the access authority of the user and ensuring the security of the data;
the user interaction module is responsible for providing a user interface and interacting with a user;
the safety module is responsible for remote monitoring and management and data backup and recovery;
The data processing and storing module is a core, the cache management module receives data to be cached from the data processing and storing module, the data checking module receives data to be deleted and corresponding information thereof from the data processing and storing module, returns a checking result to the data processing and storing module and decides whether to continue deleting operation, and the access control and authority management module receives an access request and user identity information from the user interaction module and returns an access result (permission or refusal) to the user interaction module.
The technical scheme further comprises the following steps:
Further, the data processing and storing module comprises a storage space analyzing sub-module, a target data selecting sub-module, a state information analyzing sub-module and an event association analyzing sub-module, wherein the storage space analyzing sub-module triggers a data optimizing process by monitoring and analyzing the use condition of a storage space and sends a request to the target data selecting sub-module, the target data selecting sub-module screens target data according to preset conditions and transmits the screening result to the state information analyzing sub-module, and the state information analyzing sub-module processes (sorting, demarcating, collecting and dividing, collecting and analyzing, deleting and controlling) the target data and determines the processing strategy (such as deleting, reserving and archiving) of the data.
Further, the storage space analysis submodule comprises a monitoring unit, an analysis unit and a triggering unit, wherein the monitoring unit continuously monitors the storage space occupation condition of the video downloading data, collects real-time data of the storage space, the analysis unit receives the data of the monitoring unit, analyzes the use trend and the residual space amount of the storage space, evaluates whether to trigger a data optimization flow, and when the storage space reaches a preset threshold or meets other optimization conditions according to the result of the analysis unit, the triggering unit triggers the data optimization flow and sends a request to the target data selection submodule.
Further, the target data selecting submodule comprises a screening unit and a condition configuration unit, the condition configuration unit allows a user or a system administrator to configure conditions for screening target data, such as time intervals, file sizes, data types and the like, the screening unit receives a trigger request from the storage space analysis submodule and the condition configuration of the condition configuration unit, the target data is screened from video downloading data according to the trigger request of the storage space analysis submodule and the condition configuration of the condition configuration unit, and a screening result is transmitted to the state information analysis submodule.
Further, the status information analysis submodule includes a sorting unit, a demarcating unit, a set dividing unit, a set analysis unit and a deletion control unit, wherein the sorting unit receives target data from the target data selection submodule, sorts the data according to a preset sorting rule (such as time and size), the sorting result is transmitted to the demarcating unit, the demarcating unit analyzes the sorted data, demarcates data according to service requirements or data characteristics, such as dividing the data into a plurality of batches or paragraphs, the set dividing unit further divides the demarcated data into different sets or subsets so as to perform more detailed analysis or processing, the set analysis unit analyzes each set or subset, evaluates importance, value or processing priority of the set analysis unit, the evaluation result is used for guiding a processing strategy of the data, and the deletion control unit decides which data need to be deleted or reserved based on the result of the set analysis unit, if the data need to be deleted, sends a request to the data verification module to perform multiple verification operation, and finally files or moves the data to other storage positions if the data need to be reserved.
Further, the event association analysis submodule comprises an event collecting unit, an event normalizing unit, an event storing unit, an association rule engine, an association analysis unit and an interpretation generating unit;
When any data-related events occur within the system (e.g., data expiration notifications, illegal access records, data migration logs, etc.), the event collection unit captures and passes these events to an event normalization unit;
the event standardization unit processes the collected event data to enable the event data to accord with a unified format and standard;
The processed event data is stored in an event storage unit for subsequent analysis and inquiry;
When a user initiates a deletion request, the system triggers a correlation analysis flow, and the correlation analysis unit retrieves event data related to the data to be deleted from the event storage unit;
The association analysis unit performs association analysis on the retrieved event data by using an association rule engine, and identifies an event sequence and an association mode related to the data to be deleted by analyzing the attributes such as the type, time, source, content and the like of the event;
The interpretation generation unit generates deletion reason interpretations according to the result of the association analysis, wherein the interpretations may comprise specific date of data expiration, detailed information of illegal access, reasons and processes of data migration, and the like, and the deletion reason interpretations are sent to the user interaction module.
Further, the cache management module includes a cache layer integration unit, a cache policy unit and a cache synchronization unit, where the cache layer integration unit is responsible for integrating Redis, memcached cache systems, configuring cache connection, parameters and the like, so as to ensure that the cache systems can work normally, the cache layer integration unit communicates with the cache systems (such as Redis, memcached), sends a cache read-write request, receives a response of the cache systems, processes cache data, the cache policy unit is responsible for implementing cache elimination policies such as LRU (least recently used), LFU (least frequently used) and the like, the cache policy unit monitors cache use conditions, including cache hit rate, cache size and the like, the cache policy unit determines which cache items need to be eliminated according to the cache policy, and sends an elimination request to the cache layer integration unit, the cache policy unit receives feedback of results of the cache layer integration unit about the elimination operation, and the cache synchronization unit is responsible for ensuring consistency of cache data and database data, and processing cache failure and update problems.
Furthermore, the data verification module provides a manual auditing channel, so that an additional verification step can be added before high-risk operation, and the safety of data is further improved;
The data verification module builds a data dependency graph, checks the dependency of data through a graph theory algorithm, and can identify and process the dependency of the data before deleting the data, and the data verification module comprises the following steps:
defining data entities and dependencies:
Identifying data entities-data entities explicitly incorporated into the dependency graph (e.g., database tables, files, objects, etc.);
defining dependency relationships, namely determining dependency relationship types (such as direct dependency, indirect dependency, circular dependency and the like) and rules among data entities;
Constructing a dependency graph:
Creating a Graph structure, namely representing data entities and the dependency relationship between the data entities by using a Graph structure in Graph theory, wherein the Graph consists of nodes and edges, wherein the nodes represent the data entities, and the edges represent the dependency relationship;
adding a node, namely adding a corresponding node in the graph for each data entity;
adding edges, namely adding corresponding edges in the graph according to the dependency relationship among the data entities, wherein the directions of the edges generally represent the direction of dependence (such as from the relying party to the relied party);
Graph theory algorithms are implemented to examine dependencies:
a traversal algorithm that uses a depth-first search to examine dependencies in the graph;
Dependency analysis:
checking whether a direct edge in the graph connects two nodes or not, and indicating that a direct dependency relationship exists between the two nodes;
indirect dependence, namely finding all possible dependent paths through depth-first search, and identifying indirect dependence;
Checking if a loop (Cycle) exists in the graph, the existence of which represents the existence of a cyclic dependency, which is generally an undesirable situation because it may lead to increased complexity in the case of data deletion or update;
dependency conflict detection, when a data entity is scheduled to be deleted, using a depth-first search to check whether the operation would violate any defined dependency rules or cause other data entities to become invalid or unavailable;
Processing dependency relationship:
a corresponding dependent solution strategy is formulated according to the checking result, for example, if deleting a certain data entity can cause other data entities to fail, the dependent data entities may need to be updated or deleted first, or the deleting operation is cancelled;
Automated processing-the checking and processing of the dependency is automated.
Further, the access control and authority management module comprises a role management unit, an authority allocation unit, a user management unit, an access control decision unit, a sensitive data protection unit and an audit log unit, wherein the role management unit is responsible for defining different roles in the system and allocating a group of authorities for each role, the role management unit interacts with the authority allocation unit and the user management unit to ensure the correct association between the roles and the authorities and users, the authority allocation unit is responsible for allocating access authorities for system resources (such as data, functions and the like) according to the roles defined by the role management unit, the authority allocation unit interacts with the role management unit, the user management unit and the access control decision unit to ensure the accurate allocation and execution of the authorities, the user management unit is responsible for managing the registration, authentication, authorization and other information of the users, the user management unit interacts with the roles, the user management unit is responsible for ensuring that the user can access corresponding data or executing operation by the role identity, the access control decision unit is responsible for attempting to access the corresponding data or executing operation by the user, the access control decision unit is responsible for directly recording the sensitive data protection and the sensitive data protection unit, the sensitive data is directly transmitted by the sensitive data protection unit is required to be limited by the access control unit according to the roles, and other time, the access control unit is required by the user is required to directly recording the sensitive data protection unit, the sensitive data protection unit is required to be directly sensitive data protection unit is required to be protected, and the sensitive data protection unit is protected by the sensitive data protection unit is sensitive data. The sensitive data protection unit interacts with the access control decision unit and the storage system to ensure the security of the sensitive data in the storage and transmission processes, the audit log unit is responsible for recording the data access and operation behaviors of all users, including logging in, accessing resources, modifying data and the like, so as to facilitate post-hoc tracking and audit, and the audit log unit interacts with the user management unit, the access control decision unit and the storage system to collect and store audit log information.
Further, the user interaction module comprises an interface presentation unit, a user operation unit, a user interaction logic unit, a deletion confirmation unit and an option customization unit, wherein the interface presentation unit is responsible for rendering and displaying a Graphical User Interface (GUI) or a Command Line Interface (CLI) to enable a user to interact with the system, the interface presentation unit receives user input from the user operation unit and displays system feedback to the user, the user operation unit is responsible for capturing operation instructions of the user, such as clicking, inputting and the like, and converting the operation instructions into instructions identifiable by the system, the user operation unit interacts with the user interaction logic unit, transmits the user operation instructions, simultaneously receives feedback from the interface presentation unit and displays the feedback to the user, the user interaction logic unit is responsible for processing the instructions transmitted by the user operation unit, executing corresponding logic judgment such as authority verification, data verification and the like, and calling functions of other units, wherein the user interaction logic unit interacts with a user operation unit, a deletion confirmation unit, an option customization unit and a data management module, coordinates execution of user operation, and before deleting data, the deletion confirmation unit inquires whether a user agrees in a popup window, mail notification and the like manner and provides data information and deletion reason explanation, the deletion reason explanation is provided by an event association analysis submodule, the deletion confirmation unit receives a deletion request from the user interaction logic unit, displays deletion confirmation information to the user and sends an instruction to the option customization unit according to user feedback (agreeing/refusing), the option customization unit allows the user to select whether to delete immediately, delay deletion or mark as to-be-audited, the flexibility of user operation is increased, and the option customization unit receives the user selection from the deletion confirmation unit and sends a corresponding deletion or marking instruction to the data management module according to the user selection.
The invention has the following beneficial effects:
In the invention, the cache management module uses the cache layer to cache the frequently queried results, reduces the number of direct access times to the database, improves the system performance, and avoids users waiting for a long time to see the results.
In the invention, a multiple verification mechanism is added before deleting the data, so that the accuracy and the importance of the data are ensured, meanwhile, the system inquires whether the user agrees before deleting the data through enough user interaction in the decision process, and provides data information and deletion reason explanation, wherein the deletion reason explanation comprises an event sequence and an association mode related to the data to be deleted, so that the situations of misjudgment, misdeletion and the like are avoided.
Drawings
FIG. 1 is a system block diagram of a computer device based on big data processing according to the present invention;
FIG. 2 is a system block diagram of a data processing and storage module according to the present invention;
FIG. 3 is a system block diagram of an event correlation analysis sub-module in accordance with the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1-2, the present invention is a big data processing-based computer device, comprising:
the data processing and storing module is used for receiving, processing, analyzing and storing data;
The cache management module is responsible for managing cache data and reducing direct access to a database;
the data verification module is responsible for carrying out multiple verification before deleting the data, so as to ensure the accuracy and importance of the data;
the access control and authority management module is responsible for managing the access authority of the user and ensuring the security of the data;
the user interaction module is responsible for providing a user interface and interacting with a user;
the safety module is responsible for remote monitoring and management and data backup and recovery;
The data processing and storing module is a core, the cache management module receives data to be cached from the data processing and storing module, the data checking module receives data to be deleted and corresponding information thereof from the data processing and storing module, returns a checking result to the data processing and storing module, decides whether to continue deleting operation, the access control and authority management module receives an access request and user identity information from the user interaction module, returns an access result (permission or refusal) to the user interaction module, when a user initiates a data query request in the user interaction module, firstly acquires the required data from the cache management module, if the required data exists in the cache management module, the data is directly returned to the user, and if the required data does not exist, the data is returned to the user from the data processing and storing module.
The working principle of the computer equipment based on big data processing is that an external data source (such as a network and other systems) transmits data to a data processing and storing module, the data processing and storing module receives, analyzes and primarily processes the data, and the processed data is stored or transmitted to other modules according to requirements;
The cache management module receives data to be cached from the data processing and storage module, stores the data in a cache system such as Redis and the like, sets proper cache strategies (such as LRU and LFU), firstly tries to acquire the data from a cache when a data access request exists, and acquires the data from a database or other storage systems if the cache is not hit;
The data processing and storage module identifies the data to be deleted and transmits the related information to the data checking module, the data checking module executes multiple checking logics including data comparison, integrity checking, dependency checking and the like, and returns an instruction whether to continue the deleting operation or not to the data processing and storage module according to the checking result, if the checking result is passed, the data processing and storage module executes the deleting operation, provides more detailed explanation and options and inquires whether a user agrees when executing the deleting operation;
The user initiates an access request through the user interaction module and provides user identity information, the access control and authority management module receives the access request and the user identity information and performs identity verification and authority check, the access control and authority management module returns an access result (permission or refusal) to the user interaction module according to the check result, if the access is permitted, the user interaction module displays corresponding data or executes corresponding operation, if the access is refused, the user is displayed with refusal information, and when the user initiates a data query request in the user interaction module, the system firstly tries to acquire the required data from the cache management module. If the needed data exists in the cache, the data is directly returned to the user, so that the access times of the database are reduced, and the response speed is improved;
the user interacts with the system through a graphical user interface or a command line interface, the user can initiate operation requests such as data query, data deletion, authority application and the like, the system executes corresponding operations according to the requests of the user, and an operation result or prompt message is displayed to the user through a user interaction module.
In one embodiment, for the above-mentioned data processing and storage module, the data processing and storage module includes a storage space analysis sub-module, a target data selection sub-module, a status information analysis sub-module, and an event association analysis sub-module, where the storage space analysis sub-module triggers a data optimization process by monitoring and analyzing a storage space usage condition, and sends a request to the target data selection sub-module, where the target data selection sub-module screens target data according to a preset condition, and transmits a screening result to the status information analysis sub-module, where the status information analysis sub-module processes (sorts, delimits, divides, analyzes, deletes and controls) the target data, and determines a processing policy (such as deletion, retention, archiving, etc.) of the data.
In one embodiment, for the storage space analysis submodule, the storage space analysis submodule includes a monitoring unit, an analysis unit and a triggering unit, the monitoring unit continuously monitors the storage space occupation condition of the video download data, collects real-time data of the storage space, the analysis unit receives the data of the monitoring unit, analyzes the use trend and the residual space amount of the storage space, evaluates whether to trigger a data optimization flow, and according to the result of the analysis unit, when the storage space reaches a preset threshold or meets other optimization conditions, the triggering unit triggers the data optimization flow and sends a request to the target data selection submodule.
In one embodiment, for the above target data selecting sub-module, the target data selecting sub-module includes a filtering unit and a condition configuration unit, where the condition configuration unit allows a user or a system administrator to configure conditions for filtering target data, such as a time interval, a file size, a data type, etc., the filtering unit receives a trigger request from the storage space analyzing sub-module and the condition configuration of the condition configuration unit, and filters the target data from the video download data according to the trigger request of the storage space analyzing sub-module and the condition configuration of the condition configuration unit, and the filtering result is transferred to the status information analyzing sub-module.
In one embodiment, for the state information analysis submodule, the state information analysis submodule includes a sorting unit, a demarcation unit, a set dividing unit, a set analysis unit and a deletion control unit, the sorting unit receives the target data from the target data selecting submodule, sorts the data according to a preset sorting rule (such as time and size), the sorting result is transmitted to the demarcation unit, the demarcation unit analyzes the sorted data, demarcates the data according to service requirements or data characteristics, such as dividing the data into a plurality of batches or paragraphs, the set dividing unit further divides the demarcated data into different sets or subsets so as to perform more detailed analysis or processing, the set analysis unit analyzes each set or subset, evaluates importance, value or processing priority thereof, the evaluation result is used for guiding a processing strategy of the data, the deletion control unit determines which data needs to be deleted or reserved based on the result of the set analysis unit, if the data needs to be deleted, sends a request to the data verification module to perform multiple verification, and finally performs deletion operation, if the data needs to be reserved, files or moves to other storage locations.
In one embodiment, for the event correlation analysis submodule, the event correlation analysis submodule includes an event collecting unit, an event normalizing unit, an event storing unit, a correlation rule engine, a correlation analysis unit and an interpretation generating unit;
when any data-related events occur within the system (e.g., data expiration notifications, illegal access records, data migration logs, etc.), the event collection unit captures and passes these events to the event normalization unit;
the event standardization unit processes the collected event data to enable the event data to accord with a unified format and standard;
The processed event data is stored in an event storage unit for subsequent analysis and inquiry;
when a user initiates a deletion request, the system triggers a correlation analysis flow, and the correlation analysis unit retrieves event data related to the data to be deleted from the event storage unit;
The association analysis unit performs association analysis on the retrieved event data by using an association rule engine, and identifies an event sequence and an association mode related to the data to be deleted by analyzing the attributes such as the type, time, source, content and the like of the event;
the interpretation generating unit generates deletion reason interpretations according to the result of the association analysis, wherein the interpretations may comprise specific date of data expiration, detailed information of illegal access, reasons and processes of data migration, and the like, and the deletion reason interpretations are sent to the user interaction module.
In one embodiment, for the above cache management module, the cache management module includes a cache layer integration unit, a cache policy unit and a cache synchronization unit, where the cache layer integration unit is responsible for integrating Redis, memcached cache systems such as cache connection, parameters and the like, ensuring that the cache system can work normally, the cache layer integration unit communicates with the cache system (such as Redis, memcached), sends a cache read-write request, receives a response of the cache system, processes cache data, the cache policy unit is responsible for implementing cache elimination policies such as LRU (least recently used), LFU (least frequently used) and the like, the cache policy unit monitors cache use conditions, including cache hit rate, cache size and the like, the cache policy unit determines which cache items need to be eliminated according to the cache policy, and sends an elimination request to the cache layer integration unit, the cache policy unit receives feedback of the result of the cache layer integration unit about the elimination operation, and the cache synchronization unit is responsible for ensuring consistency of the cache data and database data, and processing cache failure and update problems.
In one embodiment, for the data verification module, the data verification module provides a manual auditing channel, so that an additional verification step can be added before high-risk operation, and the safety of data is further improved;
the data verification module builds a data dependency graph, checks the dependency of the data through a graph theory algorithm, and can identify and process the dependency of the data before deleting the data, and the data verification module comprises the following steps:
defining data entities and dependency relationships, namely identifying the data entities, and definitely incorporating the data entities into a dependency graph (such as database tables, files, objects and the like), defining the dependency relationships, and determining dependency relationship types (such as direct dependency, indirect dependency, circular dependency and the like) and rules among the data entities;
creating a Graph structure, wherein the Graph structure in Graph theory is used for representing data entities and the dependency relationship between the data entities, and the Graph consists of nodes and edges, wherein the nodes represent the data entities, and the edges represent the dependency relationship; adding a Node, adding a corresponding Node in the Graph for each data entity, adding an Edge, adding a corresponding Edge in the Graph according to the dependency relationship among the data entities, wherein the direction of the Edge generally represents the direction of dependence (such as from a relying party to a relied party);
The dependency analysis comprises the steps of directly relying on whether two nodes are connected by a direct edge in the diagram or not to represent the direct dependency relationship between the two nodes, indirectly relying on that all possible dependency paths are found through the depth-first search to identify the indirect dependency relationship, and circularly relying on that whether a ring (Cycle) exists in the diagram or not is checked, wherein the existence of the ring represents the existence of the circularly dependent relationship, which is generally undesirable, because the complexity of data deletion or update is increased, and depending on conflict detection, and when a certain data entity is scheduled to be deleted, the depth-first search is used for checking whether the operation violates any defined dependency rule or other data entity is invalid or not;
The dependency relationship is processed, namely, a dependency solution strategy is processed, a corresponding dependency solution strategy is formulated according to the checking result, for example, if deleting a certain data entity can cause other data entities to fail, the dependent data entities can be required to be updated or deleted first, or deleting operation is cancelled, and the checking and processing process of the dependency relationship is automated by automated processing.
In one embodiment, for the above access control and rights management module, the access control and rights management module includes a role management unit, a rights allocation unit, a user management unit, an access control decision unit, a sensitive data protection unit, and an audit log unit, the role management unit is responsible for defining different roles in the system and allocating a set of rights for each role, the role management unit interacts with the rights allocation unit and the user management unit to ensure proper association between roles and rights, users, the rights allocation unit is responsible for allocating access rights for system resources (such as data, functions, etc.) according to roles defined by the role management unit, the rights allocation unit interacts with the role management unit, the user management unit and the access control decision unit to ensure accurate allocation and execution of rights, the user management unit is responsible for managing registration, authentication, authorization, etc. information of users, associating users with roles, the user management unit interacts with the role management unit, the rights allocation unit and the access control decision unit to ensure that users can access corresponding data or perform operations in their role identities, the access control decision unit is responsible for deciding, when users access to system resources, and whether sensitive data is required to be directly transferred to the sensitive data protection units by the user, and other sensitive data protection units are restricted by the access control units, the sensitive data protection units are allowed to be directly transmitted by the access control units, the access control units is restricted by the sensitive data protection units, the sensitive data protection unit interacts with the access control decision unit and the storage system to ensure the security of the sensitive data in the storage and transmission process, and the audit log unit is responsible for recording the data access and operation behaviors of all users, including logging in, accessing resources, modifying data and the like, so as to facilitate post-hoc tracking and audit, and interacts with the user management unit, the access control decision unit and the storage system to collect and store audit log information.
In one embodiment, for the above-mentioned user interaction module package, the user interaction module includes an interface presenting unit, a user operation unit, a user interaction logic unit, a deletion confirmation unit and an option customization unit, where the interface presenting unit is responsible for rendering and displaying a Graphical User Interface (GUI) or a Command Line Interface (CLI) to enable a user to interact with the system, the interface presenting unit receives user input from the user operation unit and displays system feedback to the user, the user operation unit is responsible for capturing operation instructions of the user, such as clicking, inputting, etc., and converting them into instructions identifiable by the system, the user operation unit interacts with the user interaction logic unit, transmits the user operation instructions while receiving feedback from the interface presenting unit and displaying to the user, and the user interaction logic unit is responsible for processing the instructions transmitted by the user operation unit, executing corresponding logic judgment such as authority verification, data verification and the like, and calling functions of other units, wherein a user interaction logic unit interacts with a user operation unit, a deletion confirmation unit, an option customization unit and a data management module, coordinates the execution of user operation, and before deleting data, the deletion confirmation unit inquires whether a user agrees in a popup window, mail notification and the like manner and provides data information and deletion reason explanation, the deletion reason explanation is provided by an event association analysis submodule, the deletion confirmation unit receives a deletion request from the user interaction logic unit, displays the deletion confirmation information to the user and sends an instruction to the option customization unit according to user feedback (agreeing/refusing), the option customization unit allows the user to select whether to delete immediately, delay deletion or mark to be audited, the flexibility of user operation is increased, the option customization unit receives the user selection from the deletion confirmation unit, and sends a corresponding deletion or marking instruction to the data management module according to the user selection.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411562076.3A CN119106005A (en) | 2024-11-05 | 2024-11-05 | A computer device based on big data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411562076.3A CN119106005A (en) | 2024-11-05 | 2024-11-05 | A computer device based on big data processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN119106005A true CN119106005A (en) | 2024-12-10 |
Family
ID=93720143
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411562076.3A Pending CN119106005A (en) | 2024-11-05 | 2024-11-05 | A computer device based on big data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN119106005A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119540430A (en) * | 2025-01-21 | 2025-02-28 | 武创芯研科技(武汉)有限公司 | Three-dimensional graphics rendering method and device based on graph theory |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060282485A1 (en) * | 2005-06-10 | 2006-12-14 | Himanshu Aggarwal | Method and system for automatic management of storage space |
| CN114996675A (en) * | 2022-06-23 | 2022-09-02 | 平安科技（深圳）有限公司 | Data query method and device, computer equipment and storage medium |
| CN115510272A (en) * | 2022-09-20 | 2022-12-23 | 哈尔滨萌动科技有限公司 | Computer data processing system based on big data analysis |
| CN116820346A (en) * | 2023-07-11 | 2023-09-29 | 哈尔滨市唯美科技有限公司 | A computer big data storage control system and method |
| CN117971129A (en) * | 2024-02-27 | 2024-05-03 | 桂林理工大学 | An efficient storage system for big data statistical collection |
| CN118364462A (en) * | 2024-04-08 | 2024-07-19 | 广西电网有限责任公司电力科学研究院 | Software supply chain security assessment method and system based on static analysis |
-
2024
- 2024-11-05 CN CN202411562076.3A patent/CN119106005A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060282485A1 (en) * | 2005-06-10 | 2006-12-14 | Himanshu Aggarwal | Method and system for automatic management of storage space |
| CN114996675A (en) * | 2022-06-23 | 2022-09-02 | 平安科技（深圳）有限公司 | Data query method and device, computer equipment and storage medium |
| CN115510272A (en) * | 2022-09-20 | 2022-12-23 | 哈尔滨萌动科技有限公司 | Computer data processing system based on big data analysis |
| CN116820346A (en) * | 2023-07-11 | 2023-09-29 | 哈尔滨市唯美科技有限公司 | A computer big data storage control system and method |
| CN117971129A (en) * | 2024-02-27 | 2024-05-03 | 桂林理工大学 | An efficient storage system for big data statistical collection |
| CN118364462A (en) * | 2024-04-08 | 2024-07-19 | 广西电网有限责任公司电力科学研究院 | Software supply chain security assessment method and system based on static analysis |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119540430A (en) * | 2025-01-21 | 2025-02-28 | 武创芯研科技(武汉)有限公司 | Three-dimensional graphics rendering method and device based on graph theory |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7673324B2 (en) | Method and system for tracking an operating performed on an information asset with metadata associated therewith | |
| KR100732789B1 (en) | Method and apparatus for monitoring database system | |
| US7356545B2 (en) | Enabling relational databases to incorporate customized intrusion prevention policies | |
| CN102598021B (en) | Method and system for managing security objects | |
| CN110795257A (en) | Method, device and equipment for processing multi-cluster operation records and storage medium | |
| US5062045A (en) | System for maintaining a document and activity selective alterable document history log in a data processing system | |
| US9509722B2 (en) | Provisioning access control using SDDL on the basis of an XACML policy | |
| US12393472B2 (en) | Unified event processing and log management over multiple domains | |
| US8135762B2 (en) | System and method for determining true computer file type identity | |
| KR102734574B1 (en) | Device for analyzing large amounts of log data based on ruleset, its control method and program | |
| US12443604B2 (en) | Nested discovery and deletion of resources | |
| CN119106005A (en) | A computer device based on big data processing | |
| CN117744129B (en) | Intelligent operation and maintenance method and system based on CIM | |
| US20080208866A1 (en) | Identification, notification, and control of data access quantity and patterns | |
| CN120068088B (en) | Unified resource identification and parsing calculation method based on trusted data space | |
| CN119669167B (en) | File observation method, electronic device, chip and program product | |
| CN117951682A (en) | Application process detection processing method, system, device and computer equipment | |
| US12430461B2 (en) | Query limiting and tracking in a data sharing environment | |
| US20240256700A1 (en) | Mechanisms to predict system resource consumption of transactions | |
| WO2024164630A1 (en) | Microkernel operating system based security monitoring method, apparatus, device, and chip | |
| CN118094497A (en) | Mobile storage equipment safety management method based on authority allocation and cancellation | |
| CN115525924A (en) | An Information Security System Based on Cloud Computing | |
| CN118916906B (en) | Data permission configuration method, device and storage medium | |
| CN119442198A (en) | A data permission identification method, device, equipment and storage medium | |
| CN120692042A (en) | Account management method, device, equipment, system, program product and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |