CN119066077A - Metadata change audit method, system, computer equipment and storage medium - Google Patents
Metadata change audit method, system, computer equipment and storage medium Download PDFInfo
- Publication number
- CN119066077A CN119066077A CN202411005615.3A CN202411005615A CN119066077A CN 119066077 A CN119066077 A CN 119066077A CN 202411005615 A CN202411005615 A CN 202411005615A CN 119066077 A CN119066077 A CN 119066077A
- Authority
- CN
- China
- Prior art keywords
- change
- information
- metadata
- record
- actual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/022—Knowledge engineering; Knowledge acquisition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Human Computer Interaction (AREA)
- Artificial Intelligence (AREA)
- Fuzzy Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of information and data management and discloses a metadata change auditing method, a system, computer equipment and a storage medium. The method comprises the steps of collecting target metadata information of a service system, carrying out standardized processing on the target metadata information to obtain standard metadata information, constructing a knowledge graph according to the target metadata information, the standard metadata information and the corresponding relation between the target metadata information and the standard metadata information, obtaining a change related file, inputting the change related file and the knowledge graph into a large language model to determine pre-change information, generating a pre-change record according to the pre-change information, obtaining actual change information, generating an actual change record according to the actual change information, and auditing and comparing the pre-change record and the actual change record. According to the invention, the actual change record is checked through the pre-change record generated by changing the related file, so that whether the change is illegal or not can be judged, and the communication cost of the manager is reduced.
Description
Technical Field
The invention relates to the field of information and data management, in particular to a metadata change auditing method, a system, computer equipment and a storage medium.
Background
At present, digital economy has become new kinetic energy for national economic growth, and is also an important break for economic transformation and promotion of human economic form transformation. The volume of the super data will cause the quality change of the super intelligence, lead the rapid development of productivity, and the data capacity acts on the digital economy to realize the rapid development of the digital economy. The data elements are used as core kinetic energy for driving the innovation development of the digital economy, not only act on the development of digital industrialization, but also comprise the development of industrial digitization, namely the radiation driving effect of the data elements on various departments of national economy.
In the digital transformation process, the IT department of the enterprise needs to manage a plurality of informationized systems, and integrates the data in the informationized systems into a data platform for unified processing, thereby playing the data value. However, the informatization system is not invariable, and needs to be updated and updated along with the service, so that new service functions are expanded. IT is an important issue for the IT department how to manage changes in the informatization system, especially changes in the data. To solve the above problems, metadata integration modules in existing data governance tools are typically employed. However, if the database structure of the service system is changed due to service change, it is impossible for the data administration team to confirm whether the metadata change is an expected change or an unexpected change, and comprehensive judgment needs to be performed by combining information such as update description of the service system technical team, which results in high communication cost and low auditing efficiency.
Therefore, a metadata change auditing method with high management efficiency and low communication cost is needed.
Disclosure of Invention
Therefore, the technical problem to be solved by the invention is to overcome the problems of high communication cost and low auditing efficiency caused by the fact that management personnel are required to comprehensively judge according to information such as updating description in order to determine whether the change is the expected change or the unexpected change when metadata in a service system is changed in the related art.
In order to solve the above technical problems, the present invention provides a metadata change auditing method, where the metadata change auditing method includes:
Collecting target metadata information of a service system;
performing standardization processing on the target metadata information to obtain standard metadata information;
constructing a knowledge graph according to the target metadata information, the standard metadata information and the corresponding relation between the target metadata information and the standard metadata information;
The method comprises the steps of obtaining a change related file, inputting the change related file and the knowledge graph to a large language model to determine pre-change information, wherein the large language model is configured to determine keyword information according to the change related file, and determine pre-change information according to the keyword information and the knowledge graph;
generating a pre-change record according to the pre-change information;
acquiring actual change information, and generating an actual change record according to the actual change information;
and auditing and comparing the pre-change record and the actual change record.
In an alternative embodiment, the inputting the change related file and the knowledge-graph into a large language model to determine pre-change information includes:
Inputting the change related file into a large language model to generate keyword information;
Inputting the keyword information and the knowledge graph into the large language model to determine standard change information; the large language model is configured to search and determine the standard change information in the knowledge graph according to the keyword information;
And inputting the standard change information and the knowledge graph into the large language model to determine the pre-change information.
In an alternative embodiment, the obtaining the actual change information includes:
When detecting that the target metadata information changes, generating actual change information, and sending the actual change information to a message middleware;
and acquiring the actual change information from the message middleware.
In an alternative embodiment, the generating a pre-change record according to the pre-change information includes:
Generating a pre-change record according to the pre-change information, a predicted change mode corresponding to the pre-change information and a predicted change time;
The generating an actual change record according to the actual change information includes:
And generating an actual change record according to the actual change information, the actual change mode corresponding to the actual change information and the actual change time.
In an alternative embodiment, the auditing the pre-change record and the actual change record includes:
Determining a matching result of the pre-change record and the actual change record, and determining a matching state of the pre-change record according to the matching result, wherein the matching result comprises a passing or non-passing state, and the matching result comprises a non-matching state or a matched state;
And when the matching result of the pre-change record and the actual change record is failed, determining the matching state of the pre-change record as an unmatched state, and generating unexpected change alarm information, wherein the unexpected change alarm information is used for prompting a manager that the matching result of the pre-change record and the actual change record is failed.
In an alternative embodiment, the method further comprises:
Storing the matching state of the pre-change record and the pre-change record into a pre-change database;
determining unmatched change records in the pre-change database according to a preset period, wherein the unmatched change records are the pre-change records with unmatched state in the pre-change database;
determining metadata information corresponding to the unmatched change records according to the unmatched change records in the pre-change database;
auditing and comparing the unmatched change records in the pre-change database with metadata information corresponding to the unmatched change records;
And when the matching result of the unmatched change record in the pre-change database and the metadata information corresponding to the unmatched change record is that the metadata information does not pass, generating pre-change delay alarm information according to the unmatched change record in the pre-change database, wherein the pre-change delay alarm information is used for prompting a manager that the metadata information corresponding to the unmatched change record in the pre-change database is unchanged.
In an alternative embodiment, the target metadata information of the service system is stored in a metadata database, and the collecting the target metadata information of the service system includes:
Configuring connection information of the metadata database;
testing whether the connecting channels are communicated or not, wherein the connecting channels are determined according to the connecting information of the metadata database;
And when the connecting channels are communicated, executing a metadata integration task, wherein the metadata integration task is used for acquiring target metadata information of a service system in the metadata database according to the connection information of the metadata database.
In a second aspect, the present invention provides a metadata change auditing system, the metadata change auditing system including:
The metadata integration module is used for collecting target metadata information of the service system;
The metadata management module is connected with the metadata integration module;
the metadata management module is used for carrying out standardization processing on the target metadata information to obtain standard metadata information, and constructing a knowledge graph according to the target metadata information, the standard metadata information and the corresponding relation between the target metadata information and the standard metadata information;
The pre-change service module is connected with the metadata management module;
The pre-change service module is used for acquiring a change related file, inputting the change related file and the knowledge graph into a large language model to determine pre-change information, and generating a pre-change record according to the pre-change information;
the metadata change auditing module is connected with the pre-change service module;
The metadata change auditing module is used for acquiring actual change information, generating an actual change record according to the actual change information, and auditing and comparing the pre-change record and the actual change record.
In a third aspect, the present invention provides a computer device, including a memory and a processor, where the memory and the processor are communicatively connected to each other, and the memory stores computer instructions, and the processor executes the computer instructions, thereby executing the metadata change auditing method according to the first aspect or any implementation manner corresponding to the first aspect.
In a fourth aspect, the present invention provides a computer readable storage medium having stored thereon computer instructions for causing a computer to perform the metadata change auditing method of the first aspect or any of its corresponding embodiments.
In a fifth aspect, the present invention provides a computer program product comprising computer instructions for causing a computer to perform the metadata change auditing method of the first aspect or any of its corresponding embodiments.
The technical scheme provided by the invention has the following technical effects:
According to the invention, the pre-change record can be determined from the change related file through the text analysis and processing capacity of the large language model and the knowledge graph constructed according to the metadata information. By auditing and comparing the pre-change records and the actual change records, whether the change corresponding to the actual change record is an expected change or an unexpected change is judged. Through the technical scheme, audit comparison is carried out, management staff is not required to deeply understand the change technology and change details of the service system, management staff can be assisted in managing metadata information, communication cost can be reduced, workload is reduced, and audit efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a metadata change auditing method according to an embodiment of the present invention;
FIG. 2 is a schematic overall flow diagram of a metadata change auditing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a metadata change auditing system according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Therefore, the embodiment of the invention provides a metadata change auditing method, a system, a computer device and a storage medium, which are used for solving the problems of high communication cost and low auditing efficiency caused by the fact that management staff is required to comprehensively judge according to information such as update description in order to determine whether the change is an expected change or an unexpected change when metadata in a service system is changed in the related art.
In accordance with an embodiment of the present invention, a metadata change auditing method embodiment is provided, it being noted that the steps shown in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and, although a logical order is shown in the flow diagrams, in some cases, the steps shown or described may be performed in an order other than that shown or described herein.
FIG. 1 is a flow chart of a metadata change auditing method according to an embodiment of the present invention.
As shown in FIG. 1, the metadata change auditing method provided by the embodiment of the invention can be applied to a metadata change auditing system, wherein the metadata change auditing system can be a metadata change auditing system, and the metadata change auditing system can comprise a metadata management module, a pre-change service module and a metadata change auditing module, and the metadata management module comprises a metadata integration module and a metadata management module.
The metadata change auditing method comprises the following steps:
s101, collecting target metadata information of a service system.
In this embodiment, the target metadata information of the service system is stored in a metadata database, and S101 collecting the target metadata information of the service system specifically includes:
And S1011, configuring connection information of the metadata database.
In this embodiment, the connection information of the metadata database may be used as the data source connection information, and may include an access address, a port, a user name, a password, and the like of the metadata database.
And S1012, testing whether the connecting channels are communicated.
In this embodiment, the connection channel is determined based on the connection information of the metadata database. The connection channel is a communication channel between the metadata integration module and the metadata database.
And S1013, when the connection channels are communicated, performing metadata integration tasks. The metadata integration task is used for collecting target metadata information of the business system in the metadata database according to the connection information of the metadata database.
In this embodiment, the target metadata information of the service system may include connection information of a metadata database, a table list, a field list, a system name, and the like. The list of tables may include table names, table chinese names, and table annotation information. The field list may include a field name, an affiliated table name, a field chinese name, a field type, and field annotation information.
The metadata change auditing method provided by the embodiment of the invention can be applied to various service systems, and as an example, if the service system is a human resource management system, the S101 collects target metadata information of the service system, specifically, a user configures connection information of a metadata database of the human resource management system through a metadata integration module, fills in an access address, a port, a user name, a password and the like of the metadata database, and the metadata change auditing system tests whether connection channels reaching the metadata database are communicated or not and stores the connection information of the database into the database of the metadata change auditing system. The metadata integration module schedules metadata integration tasks at fixed time, accesses the corresponding metadata database according to the connection information of the metadata database of the metadata change auditing system, reads target metadata information, and stores the target metadata information in the database of the metadata change auditing system.
And S102, carrying out standardization processing on the target metadata information to obtain standard metadata information, and constructing a knowledge graph according to the target metadata information, the standard metadata information and the corresponding relation between the target metadata information and the standard metadata information.
In this embodiment, the metadata governance module may perform the step of S102 described above. The method comprises the steps of carrying out word segmentation and synonym processing on target metadata information to obtain words to be standardized, obtaining a pre-established data standard term table, and searching standard terms corresponding to the words to be standardized in the data standard term table to obtain standard metadata information. As an example, the table name, the field annotation information, and the like are subjected to word segmentation processing and standardized conversion, and then standard metadata information and a correspondence relationship between the target metadata information and the standard metadata information are obtained. The knowledge graph constructed according to the target metadata information, the standard metadata information and the corresponding relation between the target metadata information and the standard metadata information can be stored in a knowledge graph engine so as to carry out semantic association, and a large language model can analyze specific tables or fields from semantics conveniently.
As an example, the data management personnel performs standardized processing on the collected target metadata information of the service system through a data management module of the metadata change auditing system. The data management module compares the table, the field and the corresponding annotation information by word segmentation and synonym, matches the closest standard term in the data standard term table, and forms the association relation between the target metadata information and the corresponding standard term. For example, a target metadata information is collected from the human resource internal management system, a field is included in an employee basic information table, the name is an employee name, and a standard term is called an employee name in a standard term table. At this time, the "employee name" can be divided into two words, namely "employee" and "name", the "employee" is consistent with the standard term, and the "name" are synonymous, so that the "employee name" is the "employee name" after standardized processing. The above-described target metadata information includes "employee name", the words to be standardized include "employee", "name" and "name", the standard term includes "employee name", and the standard metadata information includes "employee name". The metadata change auditing system stores the original field names and the standardized field names, and establishes an association relation between the employee names and the employee names. After the standardization process is completed, the data management module can store the target metadata information, the standardized standard metadata information and the corresponding relation between the target metadata information and the standard metadata information into the knowledge graph engine. For example, 2 entities are contained in the knowledge graph engine, a field information entity: "employee name", a standard term entity: "employee name", and a connection of two entities "standardized".
And S103, acquiring a change related file, inputting the change related file and the knowledge graph into the large language model to determine pre-change information, and generating a pre-change record according to the pre-change information.
In the art, a large language model is configured to determine keyword information from a change-related file, and pre-change information from the keyword information and a knowledge-graph.
In the art, the pre-change service module may perform the step of S103 described above. A technician is required to provide a change related file before a change is made to the business system. The change related file may be a system upgrade scheme or update description or DML SQL document information.
In this embodiment, the large language model includes, but is not limited to, llama, GPT-3, BERT, transformer-X, and the like. In this embodiment, the large language model can be trained in a targeted manner based on the actual application scenario of the present invention, so as to improve the accuracy of determining the pre-change information.
In the scene aimed at by the invention, a manager needs to manage and control the change of a business system besides the task related to data management, and particularly, the change of a database structure is caused by the change of the business system, business upgrading and the like, and the change of the database can cause the change of metadata.
In the process, a technician of the service system provides a system upgrade scheme or an update description or DML SQL document information for a manager before the system is upgraded, the manager uploads a change related file at the function provided by the metadata change auditing system, and after receiving and storing the change related file, the pre-change service module uploads the file to the large language model and analyzes the pre-change information through a question. After the large language model performs semantic analysis on the change related file, metadata information needing to be changed is analyzed, compared with the metadata information in the knowledge graph engine, a table and field information needing to be changed are analyzed, a pre-change record is generated, and the pre-change record is stored in a pre-change database.
S104, acquiring actual change information, generating an actual change record according to the actual change information, and auditing and comparing the pre-change record and the actual change record.
In this embodiment, the metadata change auditing module may execute the step S104 described above. The obtaining of the actual change information specifically includes generating the actual change information when the change of the target metadata information is detected, and sending the actual change information to the message middleware. The actual change information is obtained from the message middleware. In this embodiment, the metadata management module may generate a metadata change event when a difference exists between the front and rear of the target metadata information, determine that the target metadata information changes, generate actual change information according to the difference between the front and rear of the target metadata information, and send the actual change information to the message middleware. The metadata change auditing module may obtain actual change information from the message middleware.
In the present embodiment, the step S104 is performed for each pre-change record and the actual change information corresponding to the pre-change record.
In an alternative embodiment, S103 inputs the change-related file and the knowledge-graph into the large language model to determine the pre-change information, including:
s1031, inputting the change related file into the large language model to generate keyword information.
S1032, inputting the keyword information and the knowledge graph into the large language model to determine standard change information.
In the present embodiment, the large language model is configured to retrieve determination criterion change information in the knowledge-graph from the keyword information.
In this embodiment, the standard change information is a first analysis result of the large language model, and the upper entity may be first retrieved and determined in the knowledge graph according to the keyword information, where the upper entity may be a database table name, a database annotation, a standardized table name, and the field is the same, and then inferred to the lower entity, such as a table and a field, through the corresponding relationship. In this embodiment, the tables and fields to be changed cannot be directly determined through the knowledge graph and the keyword information, and only the first analysis result can be obtained. To obtain a more accurate table and field to be changed, standard change information and knowledge maps need to be input into a large language model for further analysis.
S1033, inputting the standard change information and the knowledge graph into the large language model to determine pre-change information.
In this embodiment, the pre-change information is the second analysis result of the large language model, and is metadata information that the service system should actually change. The pre-change information can be used as a metadata change application form of the service system. In the present embodiment, some of the standard change information may be used as the pre-change information by analysis of the large language model. The accuracy of determining the pre-change information can be improved by performing the second analysis on the basis of the first analysis through multiple analyses of the large language model.
As an example, if the large language model is llama, S1031 inputs the change-related file into the large language model to generate keyword information, specifically including, based on the semantic analysis and code analysis capability of the llama large language model, asking the llama large language model using a specific prompt, and analyzing, by the llama large language model, the content of the change of the metadata database from the change-related file (the change-related file includes the system upgrade scheme, the update description, and the unstructured document such as DML SQL) as the keyword information, such as the related system, table, field, and change time. For example, in the system upgrade scheme and the update description, the change description is generally performed in a literal form, for example, when a field of 'staff work age' is newly added in a table of 'staff basic information', three key word information can be deduced, namely (1) influencing a system, namely a human resource management system, (2) influencing the table, namely the staff basic information table, and (3) influencing the content, namely, a field of 'staff work age' is newly added. If the uploaded information is a DML SQL file, the 3 key information can be deduced through SQL statement analysis, and the table names and the field names are relatively accurate. Meanwhile, the expected change mode and the expected change time can be analyzed from the upgrade scheme and the update description. The expected change modes can comprise modes of addition, replacement, deletion and the like.
After obtaining the keyword information, standard change information can be determined by a technical means of combining llama large language models with Graph RAG technology. The llama large language model searches the related metadata information of the table and the field in the knowledge graph according to the keyword information to serve as standard metadata information. And the standard metadata information and the knowledge graph are input into a large language model as a context, the large language model is used for matching the knowledge graph based on GraphRAG technology, the metadata information of the data table which is actually required to be changed is finally analyzed through matching key data such as a system, a table and the like, and the metadata information of the data table which is actually required to be changed is used as pre-change information.
After the standard change information and the knowledge graph are input into the large language model to determine the pre-change information, the background service generates a pre-change record based on the pre-change information, a predicted change mode corresponding to the pre-change information and a predicted change time, wherein the pre-change record comprises metadata identification, table name change information, field change information, matching state, predicted change time and the like, and the pre-change record is stored in a pre-change database of the system.
In an alternative embodiment, generating the pre-change record according to the pre-change information in S103 includes generating the pre-change record according to the pre-change information, a predicted change manner corresponding to the pre-change information, and a predicted change time.
In this embodiment, the pre-change service module may generate the pre-change record according to the pre-change information, the expected change mode corresponding to the pre-change information, and the expected change time.
In an alternative embodiment, generating the actual change record according to the actual change information in S104 includes generating the actual change record according to the actual change information, an actual change mode corresponding to the actual change information, and an actual change time.
In this embodiment, the metadata change auditing module may generate the actual change record according to the actual change information, the actual change mode corresponding to the actual change information, and the actual change time.
In an alternative embodiment, the step S104 of auditing the pre-change record and the actual change record includes:
sa1, determining the matching result of the pre-change record and the actual change record, and determining the matching state of the pre-change record according to the matching result.
In this embodiment, the matching result includes pass or fail, and the matching result includes a non-matching state or a matched state.
And Sa2, when the matching result of the pre-change record and the actual change record is failed, determining the matching state of the pre-change record as a non-matching state, and generating unexpected change alarm information.
In this embodiment, the unexpected change alert information is used to prompt the manager that the matching result of the pre-change record and the actual change record is failed. As one example, the unexpected change alert information may be stored in a database that is prompted to a manager when queried by a background service.
In this embodiment, when the matching result of the pre-change record and the actual change record is passing, the matching state of the pre-change record is determined to be the matched state, and the next pre-change record and the corresponding actual change record are checked and compared.
In this embodiment, the metadata change auditing module may compare the actual change record with the pre-change record stored in the pre-change database one by one, where the auditing content is whether the change content such as the system name, the table name, the field name and the like of the actual trigger change is consistent with the change content such as the system name, the table name, the field name and the like in the registered change information (the pre-change record stored in the pre-change database), if so, it is determined that the matching result of the pre-change record and the actual change record is passed, the matching state of the pre-change record stored in the pre-change database is updated to be the matched state, and the next change information is checked, if not, it is determined that the matching result of the pre-change record and the actual change record is failed, the matching state of the pre-change record stored in the pre-change database is updated to be the unmatched state, and the unmatched change alarm information is generated, the unmatched alarm information is stored in the database, and the background service is queried to the unmatched alarm information and then is prompted to the manager.
In this embodiment, the auditing process may involve some details, for example, when a table is newly added, since both the table and the field are newly added as metadata changes, but in general, this will only clarify the newly added table and not the specific field list in the materials such as the upgrade scheme provided. There may be cases where the changes are filtered from one another during a particular audit. The above problems can be solved by conventional technical means in the art, and the present invention is not specifically described, and the above embodiments mainly describe a modification auditing manner.
In the present invention, in addition to the above-mentioned change auditing method, the metadata change auditing method further includes another auditing method:
and Sb1, storing the matching state of the pre-change record and the pre-change record into a pre-change database.
And Sb2, determining unmatched change records in the pre-change database according to a preset period.
In this embodiment, the unmatched change record is a prematched change record whose matching state is an unmatched state in the prematched database.
And Sb3, determining the latest metadata information corresponding to the unmatched change records according to the unmatched change records in the pre-change database.
In this embodiment, the metadata information corresponding to the unmatched change record is metadata information corresponding to the unmatched change record.
And Sb4, auditing and comparing the unmatched change records in the pre-change database with metadata information corresponding to the unmatched change records.
And Sb5, when the matching result of the unmatched change records in the pre-change database and the metadata information corresponding to the unmatched change records is that the metadata information fails, generating pre-change delay alarm information according to the unmatched change records in the pre-change database.
In this embodiment, the pre-change delay alarm information is used to prompt a manager to pre-change metadata information corresponding to an unmatched change record in the database, and prompt the manager to perform further processing.
The metadata change auditing module can scan the pre-change information in the pre-change database according to a preset period to determine an unmatched change record in the pre-change database, wherein the unmatched change record is specifically a pre-change record in which the matching state in the pre-change database is in an unmatched state within a preset time period, and the unmatched state of the pre-change information is equivalent to the fact that the pre-change information is in an unmatched state for a long time. At this time, it is necessary to determine the latest metadata information corresponding to the unmatched change record according to the unmatched change record in the pre-change database, audit and compare the unmatched change record with the latest metadata information corresponding to the unmatched change record, and update the matching state of the unmatched change record to the matched state when the matching result of the unmatched change record and the metadata information corresponding to the unmatched change record in the pre-change database is passing.
The overall flow of the metadata change auditing method provided by the embodiment of the invention is shown in fig. 2, unexpected changes can be found out by auditing based on the technical scheme of the invention, the manager is not required to deeply understand the change details of the service system, and the intervention workload of the manager on the metadata changes is reduced. The management personnel does not need to have deep knowledge on the service system like the technical personnel, the communication between the management personnel and the technical personnel is reduced, and in the communication process between the management personnel and the technical personnel, the management personnel can improve the communication efficiency between the management personnel and the technical personnel according to the information such as the pre-change information, the actual change information, the matching result and the like. According to the technical scheme, the upgrading scheme and the updating description of the system can be compared and checked with the actual change of the database through the semantic understanding capability and the knowledge graph of the large language model, so that management personnel can be assisted to manage and control the change of the database.
It should be noted that what is not described in detail in the present specification belongs to the known technology of those skilled in the art.
In this embodiment, a metadata change auditing system is further provided, and a single system is used to implement the foregoing embodiments and optional implementations, which have already been described and will not be described again. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the system described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
Fig. 3 is a schematic diagram of a metadata change auditing system according to an embodiment of the present invention.
The invention provides a metadata change auditing system, as shown in fig. 3, which comprises:
The metadata integration module 11 is configured to collect target metadata information of the service system.
The metadata governance module 12 is connected with the metadata integration module 11.
The metadata governance module 12 is configured to perform standardization processing on the target metadata information to obtain standard metadata information, and construct a knowledge graph according to the target metadata information, the standard metadata information, and a correspondence between the target metadata information and the standard metadata information.
The pre-change service module 13 is connected to the metadata governance module 12.
The pre-change service module 13 is configured to obtain a change-related file, input the change-related file and the knowledge graph into the large language model, determine pre-change information, and generate a pre-change record according to the pre-change information. The large language model is configured to determine keyword information from the change-related file and pre-change information from the keyword information and the knowledge-graph.
The metadata change auditing module 14 is connected with the pre-change service module 13.
The metadata change auditing module 14 is configured to obtain actual change information, and generate an actual change record according to the actual change information. And auditing and comparing the pre-change record and the actual change record.
In an alternative embodiment, the target metadata information of the service system is stored in a metadata database, and the metadata integration module 11 includes:
And the first processing unit is used for configuring the connection information of the metadata database.
And the second processing unit is used for testing whether the connecting channels are communicated. The connection channel is determined based on the connection information of the metadata database.
And the third processing unit is used for executing the metadata integration task when the connection channels are communicated. The metadata integration task is used for collecting target metadata information of the business system in the metadata database according to the connection information of the metadata database.
In an alternative embodiment, the pre-change service module 13 includes:
and the first processing unit is used for inputting the change related file into the large language model to generate keyword information.
And the second processing unit is used for inputting the keyword information and the knowledge graph into the large language model to determine standard change information. The large language model is configured to retrieve the determination criterion change information in the knowledge-graph based on the keyword information.
And the third processing unit is used for inputting the standard change information and the knowledge graph into the large language model to determine the pre-change information.
In an alternative embodiment, the pre-change service module 13 further comprises a fourth processing unit.
And the fourth processing unit is used for generating a pre-change record according to the pre-change information, the expected change mode corresponding to the pre-change information and the expected change time.
In an alternative embodiment, the metadata governance module 12 is further configured to generate actual change information when a change in the target metadata information is detected, and send the actual change information to the message middleware.
The metadata change auditing module 14 includes a first processing unit.
And the first processing unit is used for acquiring the actual change information from the message middleware.
In an alternative embodiment, the metadata change auditing module 14 also includes a second processing unit.
And the second processing unit is used for generating an actual change record according to the actual change information, the actual change mode corresponding to the actual change information and the actual change time.
In an alternative embodiment, the metadata change auditing module 14 also includes a third processing unit.
The third processing unit includes:
And the first processing subunit is used for determining the matching result of the pre-change record and the actual change record and determining the matching state of the pre-change record according to the matching result. The match result includes a pass or fail, and the match result includes a no-match state or a matched state.
And the second processing subunit is used for determining the matching state of the pre-change record as an unmatched state and generating unexpected change alarm information when the matching result of the pre-change record and the actual change record is not passed. The unexpected change warning information is used for prompting the manager that the matching result of the pre-change record and the actual change record is failed.
In an alternative embodiment, the metadata change auditing module 14 also includes a fourth processing unit.
The fourth processing unit includes:
And the first processing subunit is used for storing the pre-change record and the matching state of the pre-change record into the pre-change database.
And the second processing subunit is used for determining unmatched change records in the pre-change database according to the preset period. The unmatched change record is a prematched change record with a matching state of unmatched state in the prematched change database.
And the third processing subunit is used for determining the metadata information corresponding to the unmatched change record according to the unmatched change record in the pre-change database.
And the fourth processing subunit is used for auditing and comparing the unmatched change records in the pre-change database and the metadata information corresponding to the unmatched change records.
And the fifth processing subunit is used for generating the pre-change delay alarm information according to the unmatched change records in the pre-change database when the matching result of the unmatched change records in the pre-change database and the metadata information corresponding to the unmatched change records is not passed. The pre-change delay alarm information is used for prompting a manager to pre-change metadata information corresponding to the unmatched change records in the database.
Further functional descriptions of the above respective modules and units are the same as those of the above corresponding embodiments, and are not repeated here.
The metadata change auditing system in this embodiment is presented in the form of functional units, where the units refer to ASIC (Application SPECIFIC INTEGRATED Circuit) circuits, processors and memory that execute one or more software or firmware programs, and/or other devices that can provide the above functions.
The embodiment of the invention also provides a computer device which is provided with the metadata change auditing system shown in the figure 3.
Referring to fig. 4, fig. 4 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present invention, and as shown in fig. 4, the computer device includes one or more processors 10, a memory 20, and interfaces for connecting components, including a high-speed interface and a low-speed interface. The various components are communicatively coupled to each other using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the computer device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In an alternative embodiment, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple computer devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multi-processor apparatus). One processor 10 is illustrated in fig. 4.
The processor 10 may be a central processor, a network processor, or a combination thereof. The processor 10 may further include a hardware chip, among others. The hardware chip may be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general-purpose array logic, or any combination thereof.
Wherein the memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform a method for implementing the embodiments described above.
The memory 20 may include a storage program area that may store an operating system, at least one application program required for functions, and a storage data area. The storage data area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In an alternative embodiment, memory 20 may optionally include memory located remotely from processor 10, such remote memory being connectable to the computer device through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The memory 20 may include volatile memory, such as random access memory. The memory may also include non-volatile memory, such as flash memory, a hard disk, or a solid state disk. The memory 20 may also comprise a combination of the above types of memories.
The computer device also includes a communication interface 30 for the computer device to communicate with other devices or communication networks.
The embodiments of the present invention also provide a computer readable storage medium, and the method according to the embodiments of the present invention described above may be implemented in hardware, firmware, or as a computer code which may be recorded on a storage medium, or as original stored in a remote storage medium or a non-transitory machine readable storage medium downloaded through a network and to be stored in a local storage medium, so that the method described herein may be stored on such software process on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, a solid state disk, or the like. Further, the storage medium may also comprise a combination of memories of the kind described above. It will be appreciated that a computer, processor, microprocessor controller or programmable hardware includes a storage element that can store or receive software or computer code that, when accessed and executed by the computer, processor or hardware, implements the methods illustrated by the above embodiments.
Portions of the present invention may be implemented as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or aspects in accordance with the present invention by way of operation of the computer. Those skilled in the art will appreciate that the existence of computer program instructions in a computer-readable medium includes, but is not limited to, source files, executable files, installation package files, and the like, and accordingly, the manner in which computer program instructions are executed by a computer includes, but is not limited to, the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled programs, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed programs. Herein, a computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by a computer.
Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope of the invention as defined by the appended claims.
Claims (10)
1. A method for auditing metadata changes, comprising:
Collecting target metadata information of a service system;
performing standardization processing on the target metadata information to obtain standard metadata information;
constructing a knowledge graph according to the target metadata information, the standard metadata information and the corresponding relation between the target metadata information and the standard metadata information;
The method comprises the steps of obtaining a change related file, inputting the change related file and the knowledge graph to a large language model to determine pre-change information, wherein the large language model is configured to determine keyword information according to the change related file, and determine pre-change information according to the keyword information and the knowledge graph;
generating a pre-change record according to the pre-change information;
acquiring actual change information, and generating an actual change record according to the actual change information;
and auditing and comparing the pre-change record and the actual change record.
2. The method of claim 1, wherein the inputting the change-related file and the knowledge-graph into a large language model to determine pre-change information comprises:
Inputting the change related file into a large language model to generate keyword information;
Inputting the keyword information and the knowledge graph into the large language model to determine standard change information; the large language model is configured to search and determine the standard change information in the knowledge graph according to the keyword information;
And inputting the standard change information and the knowledge graph into the large language model to determine the pre-change information.
3. The method of claim 1, wherein the obtaining actual change information comprises:
When detecting that the target metadata information changes, generating actual change information, and sending the actual change information to a message middleware;
and acquiring the actual change information from the message middleware.
4. The method of claim 1, wherein the generating a pre-change record from the pre-change information comprises:
Generating a pre-change record according to the pre-change information, a predicted change mode corresponding to the pre-change information and a predicted change time;
The generating an actual change record according to the actual change information includes:
And generating an actual change record according to the actual change information, the actual change mode corresponding to the actual change information and the actual change time.
5. The method of claim 1, wherein said auditing the pre-change record and the actual change record includes:
Determining a matching result of the pre-change record and the actual change record, and determining a matching state of the pre-change record according to the matching result, wherein the matching result comprises a passing or non-passing state, and the matching result comprises a non-matching state or a matched state;
And when the matching result of the pre-change record and the actual change record is failed, determining the matching state of the pre-change record as an unmatched state, and generating unexpected change alarm information, wherein the unexpected change alarm information is used for prompting a manager that the matching result of the pre-change record and the actual change record is failed.
6. The method of claim 5, wherein the method further comprises:
Storing the matching state of the pre-change record and the pre-change record into a pre-change database;
determining unmatched change records in the pre-change database according to a preset period, wherein the unmatched change records are the pre-change records with unmatched state in the pre-change database;
determining metadata information corresponding to the unmatched change records according to the unmatched change records in the pre-change database;
auditing and comparing the unmatched change records in the pre-change database with metadata information corresponding to the unmatched change records;
And when the matching result of the unmatched change record in the pre-change database and the metadata information corresponding to the unmatched change record is that the metadata information does not pass, generating pre-change delay alarm information according to the unmatched change record in the pre-change database, wherein the pre-change delay alarm information is used for prompting a manager that the metadata information corresponding to the unmatched change record in the pre-change database is unchanged.
7. The method of claim 1, wherein the target metadata information of the business system is stored in a metadata database, and wherein the collecting the target metadata information of the business system comprises:
Configuring connection information of the metadata database;
testing whether the connecting channels are communicated or not, wherein the connecting channels are determined according to the connecting information of the metadata database;
And when the connecting channels are communicated, executing a metadata integration task, wherein the metadata integration task is used for acquiring target metadata information of a service system in the metadata database according to the connection information of the metadata database.
8. A metadata change auditing system, comprising:
The metadata integration module is used for collecting target metadata information of the service system;
The metadata management module is connected with the metadata integration module;
the metadata management module is used for carrying out standardization processing on the target metadata information to obtain standard metadata information, and constructing a knowledge graph according to the target metadata information, the standard metadata information and the corresponding relation between the target metadata information and the standard metadata information;
The pre-change service module is connected with the metadata management module;
The pre-change service module is used for acquiring a change related file, inputting the change related file and the knowledge graph into a large language model to determine pre-change information, and generating a pre-change record according to the pre-change information;
the metadata change auditing module is connected with the pre-change service module;
The metadata change auditing module is used for acquiring actual change information, generating an actual change record according to the actual change information, and auditing and comparing the pre-change record and the actual change record.
9. A computer device, comprising:
A memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the metadata change auditing method of any of claims 1 to 7.
10. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the metadata change auditing method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411005615.3A CN119066077B (en) | 2024-07-25 | 2024-07-25 | Metadata change audit method, system, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411005615.3A CN119066077B (en) | 2024-07-25 | 2024-07-25 | Metadata change audit method, system, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN119066077A true CN119066077A (en) | 2024-12-03 |
| CN119066077B CN119066077B (en) | 2025-07-08 |
Family
ID=93645868
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411005615.3A Active CN119066077B (en) | 2024-07-25 | 2024-07-25 | Metadata change audit method, system, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN119066077B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20100008885A (en) * | 2008-07-17 | 2010-01-27 | 엔에이치엔(주) | Method and system for updating web page index |
| CN112231181A (en) * | 2020-12-08 | 2021-01-15 | 平安科技（深圳）有限公司 | Data abnormal update detection method and device, computer equipment and storage medium |
| US20220100611A1 (en) * | 2017-03-15 | 2022-03-31 | Acronis International Gmbh | System and method for selecting a backup plan based on sampled metadata |
| CN116912016A (en) * | 2022-12-26 | 2023-10-20 | 中国移动通信集团江苏有限公司 | Bill auditing method and device |
| CN117290589A (en) * | 2023-09-01 | 2023-12-26 | 鹏城实验室 | Data change prediction method, system, electronic device and storage medium |
| CN117851348A (en) * | 2023-12-14 | 2024-04-09 | 天翼云科技有限公司 | Method for efficiently exporting import file by RBD snapshot |
| KR20240097198A (en) * | 2022-12-20 | 2024-06-27 | 한국환경연구원 | Method Forecasting The Change Of Treatment Capacity In A Waste Landfill And Method Analyzing Network Of Waste Landfills Based On Optimum Transport Path Using It |
-
2024
- 2024-07-25 CN CN202411005615.3A patent/CN119066077B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20100008885A (en) * | 2008-07-17 | 2010-01-27 | 엔에이치엔(주) | Method and system for updating web page index |
| US20220100611A1 (en) * | 2017-03-15 | 2022-03-31 | Acronis International Gmbh | System and method for selecting a backup plan based on sampled metadata |
| CN112231181A (en) * | 2020-12-08 | 2021-01-15 | 平安科技（深圳）有限公司 | Data abnormal update detection method and device, computer equipment and storage medium |
| KR20240097198A (en) * | 2022-12-20 | 2024-06-27 | 한국환경연구원 | Method Forecasting The Change Of Treatment Capacity In A Waste Landfill And Method Analyzing Network Of Waste Landfills Based On Optimum Transport Path Using It |
| CN116912016A (en) * | 2022-12-26 | 2023-10-20 | 中国移动通信集团江苏有限公司 | Bill auditing method and device |
| CN117290589A (en) * | 2023-09-01 | 2023-12-26 | 鹏城实验室 | Data change prediction method, system, electronic device and storage medium |
| CN117851348A (en) * | 2023-12-14 | 2024-04-09 | 天翼云科技有限公司 | Method for efficiently exporting import file by RBD snapshot |
Non-Patent Citations (1)
| Title |
|---|
| 张岩;刘晓芸;: "商业银行信息系统中"血缘分析"技术的应用研究", 信息技术与信息化, no. 06, 25 June 2017 (2017-06-25) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN119066077B (en) | 2025-07-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11409764B2 (en) | System for data management in a large scale data repository | |
| US8219518B2 (en) | Method and apparatus for modelling data exchange in a data flow of an extract, transform, and load (ETL) process | |
| CN110659790B (en) | Experiment management and data analysis system | |
| US11615076B2 (en) | Monolith database to distributed database transformation | |
| US20180046675A1 (en) | Automatic adjustment of an execution plan for a query | |
| CN112835924A (en) | Real-time computing task processing method, device, equipment and storage medium | |
| US9104724B2 (en) | Dynamic bridging of application and data servers | |
| WO2017041578A1 (en) | Method and device for acquiring database change information | |
| US11487742B2 (en) | Consistency checks between database systems | |
| CN107766353B (en) | Method and device for migrating statistical information of database | |
| CN111625472B (en) | A unit testing method and device | |
| US20160063107A1 (en) | Data retrieval via a telecommunication network | |
| CN112579705B (en) | Metadata acquisition method, device, computer equipment and storage medium | |
| US10846291B2 (en) | Transforming a user-defined table function to a derived table in a database management system | |
| CN108959454B (en) | Prompting clause specifying method, device, equipment and storage medium | |
| US8832653B2 (en) | Centralized, object-level change tracking | |
| CN119066077B (en) | Metadata change audit method, system, computer equipment and storage medium | |
| CN117648339B (en) | Data exploration method and device, server and storage medium | |
| CN111177239B (en) | Unified log processing method and system based on HDP big data cluster | |
| CN113221528A (en) | Automatic generation and execution method of clinical data quality evaluation rule based on openEHR model | |
| US20130218928A1 (en) | Information processing device | |
| KR102202792B1 (en) | Method and device for performing multi-caching on data sources of same or different types by using cluster-based processing system | |
| CN112783758A (en) | Test case library and feature library generation method, device and storage medium | |
| US20250097123A1 (en) | System and method for long-term compilation and retrieval of past data in network performance analysis | |
| CN117435666A (en) | Database version synchronization method, apparatus, device, storage medium, and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |