CN111858539A - Method, system and equipment for realizing traceability of index relationship - Google Patents
Method, system and equipment for realizing traceability of index relationship Download PDFInfo
- Publication number
- CN111858539A CN111858539A CN202010708428.7A CN202010708428A CN111858539A CN 111858539 A CN111858539 A CN 111858539A CN 202010708428 A CN202010708428 A CN 202010708428A CN 111858539 A CN111858539 A CN 111858539A
- Authority
- CN
- China
- Prior art keywords
- index
- data
- traceability
- storage
- calculation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A method for realizing traceability of index relationship comprises the following steps: index calculation; storing data; and traceability storage and reading of the index; the index calculation step adopts a distributed mode for deployment, and the data storage step adopts a shared disk mode to point disk paths of a distributed server to a shared disk; wherein, the index calculating step further comprises: performing traceable analysis on a formula of the index calculation, and acquiring the relation between the index and the relation between the index and the data attribute after performing the traceable analysis; the data storage step also comprises the steps of storing the relation between the indexes and the data attributes; in the data storage step, the LBA logical address is converted and communicated with the physical address of the storage medium, and the LBA logical address is adopted for storage and data search; the method can respond to external application and realize the retroactive analysis and storage of indexes.
Description
Technical Field
The invention relates to the field of computers, in particular to a method, a system and equipment for realizing traceability of an index relation, which can respond to external application and realize the analysis and storage of the traceability of indexes.
Background
As shown in fig. 1, the common index calculation at the present stage includes the following two ways:
1. and compiling an SQL script through an SQL language by a database server to perform index calculation, and directly storing a calculation result into a relational database table.
2. And compiling code segments through a scripting language to perform index calculation through a big data platform, and then storing data results in a big data database (relational or non-relational).
The two common ways can not achieve traceability and traceability storage of the index relationship.
The solution and method for index calculation at the present stage mainly aim at specific index algorithm to calculate and then store. Mainly depends on the performance of a database server and software, and the traceability of indexes is not reflected at all.
1. Calculating and storing in a database mode: calculations are mainly done by SQL, the SQL fragment of the stored procedure. The SQL and the method for storing the process code segments are difficult to reflect and store the relation between the indexes and the relation between the indexes and the data attributes. When the data volume is large and the table association is large, the performance is seriously reduced and the expandability is poor.
2. The mode of the big data platform is as follows: the specific index is calculated through a calculation script (usually an SQL script of a big data database) mainly supported by a big data platform, and the traceability of the index is not reflected. The continuous change of the optimized data often causes the situation of data deviation, so that the whole calculation and storage are slow. There is a continuing need to adjust the allocation of computing policies and resources.
The scheme and the technology in the current stage pay attention to the calculation of specific indexes, the scripting language in the current stage cannot realize the analysis and the storage of the traceability of the indexes in the calculation process, the traceability of the indexes is not considered, and the indexes are often checked layer by layer through the traceability of the indexes in normal services.
Disclosure of Invention
One of the objectives of the present invention is to provide a method, a system, and a device for implementing traceability of an index relationship, which can respond to an external application to implement analysis and storage of traceability of an index.
One of the objectives of the present invention is to provide a method, a system, and a device for implementing traceability of an index relationship, which can implement fast storage and reading, and the data is directly addressed by LBA logical addresses, thereby speeding up the data reading.
One of the objectives of the present invention is to provide a method, a system, and a device for implementing traceability of an index relationship, which accelerate the speed of index forwarding, store the index-traceable data through a multi-way tree, quickly read the traceable data through an index data file and an LBA logical address, and display the data, thereby avoiding data reading through complex business logic during index forwarding.
One of the objectives of the present invention is to provide a method, a system, and a device for implementing traceability of an index relationship, in which data is provided for checking subsequent index data, traceability data related to the current calculation index is stored in traceability, and if the index cannot be checked after a period of time, the problem can be checked and traced through the part of traceability data.
In order to achieve at least one of the objectives of the present invention, the present invention provides a method for implementing traceability of an index relationship, where the method for implementing traceability of an index relationship includes the following steps:
index calculation;
storing data;
and
traceability storage and reading of indexes;
the index calculation step adopts a distributed mode for deployment, and the data storage step adopts a shared disk mode to point disk paths of a distributed server to a shared disk;
Wherein the index calculating step further comprises: performing traceable analysis on a formula of the index calculation, and acquiring the relation between the index and the relation between the index and the data attribute after performing the traceable analysis;
the data storage step further comprises the steps of storing the relation between the indexes and the data attributes;
the traceability relation of the indexes is stored in a multi-branch tree mode; the index data result of the index calculation is stored in a data file mode; the logical address of the index calculation result file is stored in a data file indexing mode, so that subsequent reading is facilitated;
when calculation or query is executed, the logic address of the data file is obtained through the index file, and then the data file is positioned through the logic address, so that data reading can be realized;
in the data storage step, the LBA logical address is converted and communicated with the physical address of the storage medium, and the LBA logical address is adopted for storage and data search.
In some embodiments, in the index calculating step, a traceable parsing step of the index is performed, wherein the traceable parsing step of the index includes the following steps:
And responding to an application index calculation request of an external application, analyzing a calculation formula of the index, acquiring the tracing attribute content and the tracing attribute content value required by index calculation, and simultaneously sending the tracing attribute content and the tracing attribute content value to a calculation engine to calculate the index to acquire an index result value.
In some embodiments, the index calculating step further includes the steps of:
a metadata level index calculation step, wherein the metadata level index is an index obtained by the operation of a plurality of attributes in a piece of data, and the traceable content of the metadata level index is the attribute and the attribute value of the data;
a summary level index calculation step, wherein the summary level index is an index obtained by aggregating the attributes of a plurality of pieces of data, and the traceable content of the summary level index is the attribute of the data and the attribute value of each piece of data used in aggregation; and
and calculating a composite index, wherein the composite index is an index obtained by calculation through the relation between the index and the index or four arithmetic operations of the index and the index, and the traceable content of the composite index is related index and index value.
In some embodiments, in the performing of the index calculating step, when one server is subjected to retrospective analysis of the index, result data running on another server and stored is obtained, wherein in the data storing step, shared storage is performed, wherein the shared storage step is performed by a NAS shared storage technique or a distributed data storage technique according to a range and a degree of concurrency and a data amount.
In some embodiments, the step of traceability storing and reading the indicator further comprises the steps of:
the index traceability data storage adopts a data structure of a multi-branch tree for storage, wherein the multi-branch tree is divided into a data field part and an address field part, the data field part is used for storing basic data of an index, such as an index name, an index code, an index dimension, an index value and the like, the address field part is used for storing an address set of the traceability data and directly points to the position of the traceability data of the index, and the data can be rapidly read during traceability;
the index data is stored by taking the index dimension as a file name, and the file content is stored by index basic information and address information of traceable data; simultaneously, creating an index data file according to the index dimension, and storing the LBA logical addresses of the index data in the dimension under the same dimension; when a user inquires information under a certain dimension, LBA logical addresses of all indexes under the dimension can be quickly found through the dimension index file, related index data can be quickly found according to the LBA logical addresses, and related traceable data can be quickly positioned in the index data;
And
and performing a rapid data reading step, wherein a plurality of index files are created according to different dimensions according to different service needs and requirements, and one piece of index data is in the index files of the plurality of dimensions, so that the required data can be rapidly searched under the condition of service needs.
According to another aspect of the present invention, a system for implementing traceability of an index relationship is further provided, where the system for implementing traceability of an index relationship includes an index calculation subsystem, a data storage subsystem, and an index traceability storage and reading subsystem;
wherein the index calculation subsystem is deployed in a distributed mode, the data storage subsystem directs the disk paths of the distributed servers to the shared disk in a disk sharing mode, in the index calculation process of the index calculation subsystem, a calculation formula is firstly analyzed, the relationship between an index and the relationship between the index and data attributes are obtained after the analysis, then the relationship is stored, the traceability relationship between the index in the index traceability storage and reading subsystem is stored in a multi-way tree mode, the calculated index data result is stored in a data file mode, for the convenience of subsequent reading, the logic address of the index calculation result file is stored in a data file indexing mode, when in calculation or query, the logic address of the data file is obtained through the index file, and then the data file is positioned through the logic address, thereby achieving the reading of data.
In some embodiments, the index calculation subsystem includes a metadata level index calculation unit, a summary level index calculation unit, and a composite index calculation unit, wherein the traceable content of the metadata level index calculation unit is an attribute and an attribute value of data, the traceable content of the summary level index calculation unit is an attribute of data and an attribute value of each piece of data used in aggregation, and the traceable content of the composite index calculation unit is a related index and an index value; when the external application applies for index calculation, the traceability realization system of the index relation analyzes the calculation formula of the index, obtains the traceability attribute content and traceability attribute content value required by index calculation through analysis, and simultaneously sends the traceability attribute content and traceability attribute content value to the calculation engine to calculate the index so as to obtain the index result value.
In some embodiments, the data storage subsystem directs the disk paths of the distributed servers to the shared disk in a shared disk manner, where the shared storage is NAS shared storage or distributed data storage; the data storage subsystem performs conversion and communication with a physical address of a storage medium through an LBA logical address, and the data storage subsystem performs storage and data search by using the LBA logical address when storing data.
In some embodiments, wherein the traceability storage and reading subsystem of the index comprises a traceability storage unit of the index and a traceability reading unit of the index, wherein the index traceability storage unit stores the index data by adopting a data structure of a multi-branch tree, the index dimension is used as a file name for storing the index data in the index traceability storage unit, the file content is used as the index basic information and the address information of the traceable data, and an index data file is created by the index dimension, storing the LBA logical addresses of the index data in the same dimension, when a user inquires information under a certain dimension, the LBA logical addresses of all indexes under the dimension are obtained through the dimension index file, acquiring related index data according to the LBA logical address, and positioning traceable data in the index data; the index traceability reading unit creates a plurality of index files according to different service needs and requirements and according to different dimensions, wherein one index data is in the index files of the plurality of dimensions, so that the data corresponding to the service needs is searched and obtained under the condition of the service needs.
According to another aspect of the present invention, an apparatus for implementing traceability of an index relationship is further provided, including:
the software application(s) may be,
a memory for storing the software application,
a processor for executing the software application,
wherein, each program of the software application program correspondingly executes the steps in the method for realizing the traceability of the index relationship.
Drawings
Fig. 1 is a schematic diagram of a prior art solution and method for index calculation.
Fig. 2 is a flowchart illustrating a method for implementing traceability of index relationship according to a preferred embodiment of the present invention.
Detailed Description
The following description is presented to disclose the invention so as to enable any person skilled in the art to practice the invention. The preferred embodiments in the following description are given by way of example only, and other obvious variations will occur to those skilled in the art. The basic principles of the invention, as defined in the following description, may be applied to other embodiments, variations, modifications, equivalents, and other technical solutions without departing from the spirit and scope of the invention.
It is understood that the terms "a" and "an" should be interpreted as meaning that a number of one element or element is one in one embodiment, while a number of other elements is one in another embodiment, and the terms "a" and "an" should not be interpreted as limiting the number.
The present invention relates to a computer program. Fig. 2 is a schematic flow chart of a traceability implementation method based on an index relationship of the present invention, which illustrates a solution for controlling or processing an external object or an internal object of a computer by executing a computer program compiled according to the above flow on the basis of a computer program processing flow to solve the problems of the present invention. By the method for realizing the traceability of the index relationship, the traceability of the index can be realized by utilizing a computer system and responding to external application. It should be understood that the term "computer" as used herein refers not only to desktop computers, notebook computers, tablet computers, etc., but also includes other intelligent electronic devices capable of operating according to programs and processing data.
Specifically, the method for implementing traceability of the index relationship includes the following steps:
index calculation;
a data storage step;
and
and traceability storing and reading the index.
The index calculation step adopts a distributed mode for deployment, and the data storage step adopts a shared disk mode to enable disk paths of a distributed server to point to a shared disk.
Wherein the index calculating step further comprises: and performing traceable analysis on the formula of the index calculation, and acquiring the relation between the index and the relation between the index and the data attribute after performing the traceable analysis.
And the data storage step further comprises the step of storing the relation between the indexes and the data attributes.
In the data storage step, the traceability relation of the indexes is stored in a multi-branch tree mode; wherein, the calculated index data result is stored in a data file mode; preferably, the logical address of the index calculation result file is stored in a manner of indexing the data file, thereby facilitating subsequent quick reading.
When calculation or query is executed, the logic address of the data file is quickly found through the index file, and then the data file is quickly positioned through the logic address, so that the data can be quickly read.
More specifically, in the index calculation step, a traceable parsing step of the index is performed, wherein the traceable parsing step of the index includes the steps of:
and responding to an application index calculation request of an external application, analyzing a calculation formula of the index, acquiring the tracing attribute content and the tracing attribute content value required by index calculation, and simultaneously sending the tracing attribute content and the tracing attribute content value to a calculation engine to calculate the index to acquire an index result value.
More specifically, the index calculation step further includes the steps of:
a metadata level index calculation step, wherein the metadata level index is an index obtained by the operation of a plurality of attributes in a piece of data, and the traceable content of the metadata level index is the attribute and the attribute value of the data;
a summary level index calculation step, wherein the summary level index is an index obtained by aggregating the attributes of a plurality of pieces of data, and the traceable content of the summary level index is the attribute of the data and the attribute value of each piece of data used in aggregation; and
and calculating a composite index, wherein the composite index is an index obtained by calculation through the relation between the index and the index or four arithmetic operations of the index and the index, and the traceable content of the composite index is related index and index value.
It is worth mentioning that in the index calculation step, the whole calculation application adopts distributed deployment, so as to provide expandability for subsequent calculation performance improvement. In the whole distributed computation, the thread pools are integrated in each distributed computation application, and the multithreading parallel computation in the computation meets the requirement of large concurrency, so that the computation efficiency is improved.
More specifically, since the whole computing application adopts a distributed application architecture, in index computation, when one server performs retrospective analysis of an index, result data running and stored on another server often needs to be acquired, and therefore shared storage is performed in the data storage step.
In a preferred embodiment, the shared Storage step is performed by NAS (Network Attached Storage) shared Storage or distributed data Storage.
The NAS shared storage is a technology for integrating distributed and independent data into a large-scale and centralized-management data center so as to facilitate access to different hosts and application servers. The NAS technology can reduce the storage cost and has good cost performance. The advantage of using NAS shared storage is that the special storage design of NAS and the characteristics of serving multiple clients make data more secure and accessible. Unlike typical storage devices such as U-disks, 1394 interface storage, or CD/DVD, NAS has an embedded operating system. The operating system gives built-in intelligence to the NAS and supports the NAS to independently provide services for various clients on the network. The NAS does not need to depend on any other host equipment, and the problem that the NAS cannot be accessed when other equipment is stopped does not occur. In addition, the NAS comprises RAID technology, data is redundantly backed up among a plurality of hard disks, and easy data recovery can be provided once a certain hard disk encounters hardware failure. The reason why NAS technology can replace servers: as a professional storage and file server, NAS is cheaper and more convenient than traditional servers. NAS is more advantageous in that it works for the most important storage. Other functions are stripped, one NAS device is less prone to system conflict and security attack problems, and higher read-write performance and system stability are provided. Also, when a problem occurs, the NAS system can process the failure faster due to lower system complexity. NAS technology supports different operating system platforms: as the redundant functions of the operating system of the NAS device are eliminated, the NAS device can be connected with more various operating systems than a common server. The NAS runs an own operating system, and can be accessed by operating systems such as Windows, UNIX, Linux, Mac and the like without requiring specific client support.
Distributed data storage is the decentralized storage of data on a plurality of independent devices. The traditional network storage system adopts a centralized storage server to store all data, and the storage server cannot meet the requirement of large-scale storage application. The distributed network storage system adopts an expandable system structure, utilizes a plurality of storage servers to share the storage load, and utilizes the position server to position the storage information, thereby not only improving the reliability, the availability and the access efficiency of the system, but also being easy to expand.
It should be noted that in the data storage step of the method for implementing traceability of the index relationship, when the concurrency amount and the data amount are within a certain range, the NAS shared storage mode may be used to store the data, and when the concurrency amount and the data amount reach a certain degree, the NAS disk may have a bottleneck of network and data reading, and may use a distributed data storage mode.
Further, after the distributed application server has a shared storage function, the storage media are mainly divided into high-speed storage disk media and solid-state storage media.
The addressing mode of the high-speed disk storage medium is as follows: the operating system uses LBA (logical Block Address) to linearly Address by sector unit, an Address translator is installed in the hard disk controller and is responsible for translating the C/H/S parameter into LBA Address, and the logical Address and the physical Address of the hard disk can be mutually converted. The C/H/S (Cylinder/Head/Sector) addressing mode is used for addressing according to absolute addresses formed by the number of Heads (Heads), the number of Cylinders (Cylinders) and the number of Sectors (Sectors), and is also called as the physical address of the hard disk.
The solid-state storage medium addressing mode comprises the following steps: SSDs are primarily composed of an SSD controller, FLASH storage array, on-board DRAM (optional), and HOST interface (such as SATA, SAS, PCIe, etc.). HOST accesses SSD through LBA (Logical block address), each LBA represents a Sector (generally 512B), the operating system accesses SSD generally in units of 4K, and we call HOST Page (HOST Page) the basic unit of HOST accessing SSD. In the SSD, a FLASH Page is used as a basic unit between the SSD main control and the FLASH to access the FLASH, and we call the FLASH Page as a Physical Page (Physical Page). When HOST writes a HOST Page, SSD master finds a Physical Page to write HOST data, and the SSD records the mapping (Map) inside. With such a mapping relationship, the SSD knows from which position of FLASH the data is read the next time HOST needs to read a HOST Page.
It should be noted that in the data storage step of the method for implementing traceability of index relationship in the present invention, the LBA logical address is used for performing conversion and communication with the physical address of the storage medium, and the LBA logical address is used for performing storage and data search.
More specifically, the index traceability storing and reading step further comprises the following steps:
the index traceability data storage adopts a data structure of a multi-branch tree for storage, wherein the multi-branch tree is divided into a data field part and an address field part, the data field part is used for storing basic data of an index, such as an index name, an index code, an index dimension, an index value and the like, the address field part is used for storing an address set of the traceability data and directly points to the position of the traceability data of the index, and the data can be rapidly read during traceability;
the index data is stored by taking the index dimension as a file name, and the file content is stored by index basic information and address information of traceable data; simultaneously, creating an index data file according to the index dimension, and storing the LBA logical addresses of the index data in the dimension under the same dimension; when a user inquires information under a certain dimension, LBA logical addresses of all indexes under the dimension can be quickly found through the dimension index file, related index data can be quickly found according to the LBA logical addresses, and related traceable data can be quickly positioned in the index data;
And
and performing a rapid data reading step, wherein a plurality of index files are created according to different dimensions according to different service needs and requirements, and one piece of index data is in the index files of the plurality of dimensions, so that the required data can be rapidly searched under the condition of service needs.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects.
Those skilled in the art will appreciate that the methods of the present invention can be implemented in hardware, software, or a combination of hardware and software. The present invention can be realized in a centralized fashion in at least one computer system, or in a distributed fashion where different elements are spread across several interconnected computer systems. Any kind of computer system or other apparatus adapted for carrying out the methods described herein is suited. A typical combination of hardware and software could be a general purpose computer system with a computer program that, when being loaded and executed, controls the computer system such that it carries out the methods described herein.
The present invention can be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein. The computer program product is embodied in one or more computer-readable storage media having computer-readable program code embodied therein. According to another aspect of the invention, there is also provided a computer-readable storage medium having stored thereon a computer program capable, when executed by a processor, of performing the steps of the method of the invention. Computer storage media is media in computer memory for storage of some discrete physical quantity. Computer storage media includes, but is not limited to, semiconductors, magnetic disk storage, magnetic cores, magnetic drums, magnetic tape, laser disks, and the like. It will be appreciated by persons skilled in the art that computer storage media are not limited by the foregoing examples, which are intended to be illustrative only and not limiting of the invention.
Corresponding to the embodiment of the method for implementing traceability of index relationship in the present invention, according to another aspect of the present invention, a system for implementing traceability of index relationship is also provided, where the system for implementing traceability of index relationship is an application of the method for implementing traceability of index relationship in computer program improvement. The system for realizing the traceability of the index relationship comprises an index calculation subsystem, a data storage subsystem and a traceability storage and reading subsystem of the index.
The index calculation subsystem is deployed in a distributed mode, and the data storage subsystem directs the disk paths of the distributed servers to the shared disk in a shared disk mode. In the index calculation process of the index calculation subsystem, the calculated formula is firstly analyzed, the relationship between the index and the relationship between the index and the data attribute can be obtained after the analysis, and then the relationship is stored. And the traceability relation between the index traceability storage and the index traceability in the reading subsystem is stored in a multi-branch tree mode. The calculated index data result is stored in a data file mode, and in order to facilitate subsequent quick reading, the logic address of the index calculation result file is stored in an index data file mode. When in calculation or query, the logical address of the data file needs to be quickly found through the index file, and then the data file is quickly positioned through the logical address, so that the effect of quickly reading the data is achieved.
More specifically, the index calculation subsystem includes a metadata level index calculation unit, a summary level index calculation unit, and a composite index calculation unit, where the traceable content of the metadata level index calculation unit is an attribute and an attribute value of data, the traceable content of the summary level index calculation unit is an attribute of data and an attribute value of each piece of data used in aggregation, and the traceable content of the composite index calculation unit is a related index and an index value.
When the external application applies for index calculation, the traceability realization system of the index relation analyzes the calculation formula of the index, and the traceability attribute content value required by index calculation are obtained through analysis. Meanwhile, the index is sent to a calculation engine to calculate the index, so that an index result value is obtained.
The whole computing application of the traceability realization system of the index relation adopts distributed deployment, and expandability is provided for subsequent improvement of computing performance. In the whole distributed computation, the thread pools are integrated in each distributed computation application, and the multithreading parallel computation in the computation meets the requirement of large concurrency, so that the computation efficiency is improved.
The data storage subsystem adopts a shared disk mode to point disk paths of the distributed server to the shared disk, wherein the shared storage adopts NAS shared storage or distributed data storage, preferably, when the concurrency and the data volume are within a certain range, the NAS mode is adopted for data storage, and when the concurrency and the data volume reach a certain degree, the NAS disk can generate network and data reading bottlenecks, and the distributed data storage mode is adopted.
After the distributed application servers of the system for realizing traceability of the index relationship share storage, the storage media are mainly divided into high-speed storage disk media and solid-state storage media. The data storage subsystem carries out conversion and communication with a physical address of a storage medium through an LBA logical address, and when the data storage subsystem stores data, the LBA logical address is adopted for storage and data searching.
The traceability storage and reading subsystem of the index comprises a traceability storage unit of the index and a traceability reading unit of the index.
And the traceability storage unit of the index stores the data by adopting a data structure of the multi-branch tree. The whole multi-branch tree is divided into two parts. The first part is a data field part: the method is mainly used for storing basic data of the index, such as index name, index code, index dimension, index value and the like. The second part is an address realm part. The method is mainly used for storing the address set of the traceable data, directly points to the position of the traceable data of the index, and is convenient for rapidly reading the data during tracing.
The index data in the index traceability storage unit is stored by taking the index dimension as a file name, and the file content is stored by index basic information and address information of the traceability data. And simultaneously, creating an index data file according to the index dimension, and storing the LBA logical addresses of the index data in the dimension under the same dimension. When a user inquires information under a certain dimension, the LBA logical addresses of all indexes under the dimension can be quickly found through the dimension index file, related index data can be quickly found according to the LBA logical addresses, and related traceable data can be quickly positioned in the index data.
The index traceability reading unit can create a plurality of index files according to different service needs and requirements and one index data can be in the index files with multiple dimensions, so that the required data can be quickly searched under the condition of service needs.
Corresponding to the embodiment of the method for implementing traceability of index relationship in the present invention, according to another aspect of the present invention, an apparatus for implementing traceability of index relationship is further provided, where the apparatus for implementing traceability of index relationship includes: a software application, a memory for storing the software application, and a processor for executing the software application. Each program of the software application program can correspondingly execute the steps in the method for realizing the traceability of the index relationship.
A typical combination of hardware and software could be a general purpose computer system with a computer program that, when being loaded and executed, controls the computer system such that it carries out the methods described herein.
It will be appreciated by those skilled in the art that the present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, systems and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and/or block diagram block or blocks.
It will be appreciated by persons skilled in the art that the embodiments of the invention described above and shown in the drawings are given by way of example only and are not limiting of the invention. The objects of the invention have been fully and effectively accomplished. The functional and structural principles of the present invention have been shown and described in the examples, and any variations or modifications of the embodiments of the present invention may be made without departing from the principles.
Claims (10)
1. A method for realizing the traceability of an index relationship is characterized by comprising the following steps:
index calculation;
storing data;
and
traceability storage and reading of indexes;
the index calculation step adopts a distributed mode for deployment, and the data storage step adopts a shared disk mode to point disk paths of a distributed server to a shared disk;
wherein the index calculating step further comprises: performing traceable analysis on a formula of the index calculation, and acquiring the relation between the index and the relation between the index and the data attribute after performing the traceable analysis;
the data storage step further comprises the steps of storing the relation between the indexes and the data attributes;
The traceability relation of the indexes is stored in a multi-branch tree mode; the index data result of the index calculation is stored in a data file mode; the logical address of the index calculation result file is stored in a data file indexing mode, so that subsequent reading is facilitated;
when calculation or query is executed, the logic address of the data file is obtained through the index file, and then the data file is positioned through the logic address, so that data reading can be realized;
in the data storage step, the LBA logical address is converted and communicated with the physical address of the storage medium, and the LBA logical address is adopted for storage and data search.
2. The method for implementing traceability of index relationship as claimed in claim 1, wherein in the index calculating step, a step of performing traceable parsing of the index is performed, wherein the step of performing traceable parsing of the index comprises the steps of:
and responding to an application index calculation request of an external application, analyzing a calculation formula of the index, acquiring the tracing attribute content and the tracing attribute content value required by index calculation, and simultaneously sending the tracing attribute content and the tracing attribute content value to a calculation engine to calculate the index to acquire an index result value.
3. The method for implementing traceability of index relationship as claimed in claim 1, wherein said index calculating step further comprises the steps of:
a metadata level index calculation step, wherein the metadata level index is an index obtained by the operation of a plurality of attributes in a piece of data, and the traceable content of the metadata level index is the attribute and the attribute value of the data;
a summary level index calculation step, wherein the summary level index is an index obtained by aggregating the attributes of a plurality of pieces of data, and the traceable content of the summary level index is the attribute of the data and the attribute value of each piece of data used in aggregation; and
and calculating a composite index, wherein the composite index is an index obtained by calculation through the relation between the index and the index or four arithmetic operations of the index and the index, and the traceable content of the composite index is related index and index value.
4. The method for implementing traceability of index relationship as claimed in claim 1, wherein in the step of calculating the index, when one server is analyzed retrospectively by the index, obtaining result data running on another server and stored, wherein in the step of storing the data, shared storage is performed, wherein the step of shared storage is performed by NAS shared storage technology or distributed data storage technology according to the extent and degree of concurrency and data amount.
5. The method for implementing traceability of index relationship as claimed in claim 1, wherein the traceability storing and reading step of index further comprises the following steps:
the index traceability data storage adopts a data structure of a multi-branch tree for storage, wherein the multi-branch tree is divided into a data field part and an address field part, the data field part is used for storing basic data of an index, such as an index name, an index code, an index dimension, an index value and the like, the address field part is used for storing an address set of the traceability data and directly points to the position of the traceability data of the index, and the data can be rapidly read during traceability;
the index data is stored by taking the index dimension as a file name, and the file content is stored by index basic information and address information of traceable data; simultaneously, creating an index data file according to the index dimension, and storing the LBA logical addresses of the index data in the dimension under the same dimension; when a user inquires information under a certain dimension, LBA logical addresses of all indexes under the dimension can be quickly found through the dimension index file, related index data can be quickly found according to the LBA logical addresses, and related traceable data can be quickly positioned in the index data;
And
and performing a rapid data reading step, wherein a plurality of index files are created according to different dimensions according to different service needs and requirements, and one piece of index data is in the index files of the plurality of dimensions, so that the required data can be rapidly searched under the condition of service needs.
6. A traceability realization system of index relationship is characterized in that the traceability realization system of index relationship comprises an index calculation subsystem, a data storage subsystem and a traceability storage and reading subsystem of index;
wherein the index calculation subsystem is deployed in a distributed mode, the data storage subsystem directs the disk paths of the distributed servers to the shared disk in a disk sharing mode, in the index calculation process of the index calculation subsystem, a calculation formula is firstly analyzed, the relationship between an index and the relationship between the index and data attributes are obtained after the analysis, then the relationship is stored, the traceability relationship between the index in the index traceability storage and reading subsystem is stored in a multi-way tree mode, the calculated index data result is stored in a data file mode, for the convenience of subsequent reading, the logic address of the index calculation result file is stored in a data file indexing mode, when in calculation or query, the logic address of the data file is obtained through the index file, and then the data file is positioned through the logic address, thereby achieving the reading of data.
7. The system for implementing traceability of index relationship as claimed in claim 6, wherein the index calculation subsystem comprises a metadata level index calculation unit, a summary level index calculation unit and a composite index calculation unit, wherein the traceability content of the metadata level index calculation unit is the attribute and attribute value of data, the traceability content of the summary level index calculation unit is the attribute of data and attribute value of each piece of data used in aggregation, and the traceability content of the composite index calculation unit is the related index and index value; when the external application applies for index calculation, the traceability realization system of the index relation analyzes the calculation formula of the index, obtains the traceability attribute content and traceability attribute content value required by index calculation through analysis, and simultaneously sends the traceability attribute content and traceability attribute content value to the calculation engine to calculate the index so as to obtain the index result value.
8. The system for implementing traceability of index relationship as claimed in claim 6, wherein the data storage subsystem directs the disk paths of the distributed servers to the shared disk in a manner of shared disk, wherein the shared storage is NAS shared storage or distributed data storage; the data storage subsystem performs conversion and communication with a physical address of a storage medium through an LBA logical address, and the data storage subsystem performs storage and data search by using the LBA logical address when storing data.
9. The system for implementing traceability of index relationship as claimed in any one of claims 6 to 8, wherein the traceability storage and reading subsystem of the index comprises a traceability storage unit of the index and a traceability reading unit of the index, wherein the traceability storage unit of the index stores the index data in a multi-way tree data structure, the traceability storage unit of the index stores the index data in a file name of the index dimension, the file content stores the index basic information and the address information of the traceable data, and creates an index data file in the index dimension, and stores the LBA logical addresses of all the index data in the dimension in the same dimension, when the user queries the information in the dimension, the LBA logical addresses of all the indexes in the dimension are obtained through the dimension index file, and the related index data are obtained according to the LBA logical addresses, locating traceable data in the index data; the index traceability reading unit creates a plurality of index files according to different service needs and requirements and according to different dimensions, wherein one index data is in the index files of the plurality of dimensions, so that the data corresponding to the service needs is searched and obtained under the condition of the service needs.
10. An apparatus for implementing traceability of an index relationship, comprising:
the software application(s) may be,
a memory for storing the software application,
a processor for executing the software application,
wherein each program of the software application correspondingly performs the steps in the method for implementing traceability of index relationship as claimed in claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010708428.7A CN111858539B (en) | 2020-07-22 | 2020-07-22 | Method, system and equipment for realizing traceability of index relation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010708428.7A CN111858539B (en) | 2020-07-22 | 2020-07-22 | Method, system and equipment for realizing traceability of index relation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111858539A true CN111858539A (en) | 2020-10-30 |
| CN111858539B CN111858539B (en) | 2024-05-14 |
Family
ID=73001895
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010708428.7A Active CN111858539B (en) | 2020-07-22 | 2020-07-22 | Method, system and equipment for realizing traceability of index relation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858539B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101930560A (en) * | 2009-06-24 | 2010-12-29 | 株式会社日立制作所 | Cause analysis support device and cause analysis support method |
| US8578126B1 (en) * | 2009-10-29 | 2013-11-05 | Netapp, Inc. | Mapping of logical start addresses to physical start addresses in a system having misalignment between logical and physical data blocks |
| CN107066483A (en) * | 2016-12-25 | 2017-08-18 | 北京通途永久科技有限公司 | A kind of transportation industry achievement data coding and storage method |
| CN110018878A (en) * | 2018-01-09 | 2019-07-16 | 阿里巴巴集团控股有限公司 | A kind of distributed system data load method and device |
| CN110689211A (en) * | 2018-07-04 | 2020-01-14 | 北京国双科技有限公司 | Method and device for evaluating website service capability |
-
2020
- 2020-07-22 CN CN202010708428.7A patent/CN111858539B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101930560A (en) * | 2009-06-24 | 2010-12-29 | 株式会社日立制作所 | Cause analysis support device and cause analysis support method |
| US8578126B1 (en) * | 2009-10-29 | 2013-11-05 | Netapp, Inc. | Mapping of logical start addresses to physical start addresses in a system having misalignment between logical and physical data blocks |
| CN107066483A (en) * | 2016-12-25 | 2017-08-18 | 北京通途永久科技有限公司 | A kind of transportation industry achievement data coding and storage method |
| CN110018878A (en) * | 2018-01-09 | 2019-07-16 | 阿里巴巴集团控股有限公司 | A kind of distributed system data load method and device |
| CN110689211A (en) * | 2018-07-04 | 2020-01-14 | 北京国双科技有限公司 | Method and device for evaluating website service capability |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111858539B (en) | 2024-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11593037B2 (en) | File system block-level tiering and co-allocation | |
| US9710187B1 (en) | Managing data relocation in storage systems | |
| CA2846417C (en) | Shared cache used to provide zero copy memory mapped database | |
| US20110078682A1 (en) | Providing Object-Level Input/Output Requests Between Virtual Machines To Access A Storage Subsystem | |
| CN110851401B (en) | Methods, devices and computer-readable media for managing data storage | |
| US11625192B2 (en) | Peer storage compute sharing using memory buffer | |
| US11199990B2 (en) | Data reduction reporting in storage systems | |
| CN103020174A (en) | Similarity analysis method, device and system | |
| US9430503B1 (en) | Coalescing transactional same-block writes for virtual block maps | |
| WO2019034941A1 (en) | Thin provisioning virtual desktop infrastructure virtual machines in cloud environments without thin clone support | |
| US10705853B2 (en) | Methods, systems, and computer-readable media for boot acceleration in a data storage system by consolidating client-specific boot data in a consolidated boot volume | |
| CN103150225B (en) | Disk full abnormity fault tolerance method of object parallel storage system based on application level agent | |
| CN107622123B (en) | ASM file system-oriented file analysis method | |
| US20250103597A1 (en) | Data Processing System and Apparatus | |
| CN111858539B (en) | Method, system and equipment for realizing traceability of index relation | |
| US9690803B1 (en) | Auxiliary files in a container file system | |
| CN111459928B (en) | Data deduplication method and application applied to cluster-wide data backup scenarios | |
| CN118626432A (en) | Data processing method, storage system, network interface device and storage medium | |
| Mishra et al. | On-disk data processing: issues and future directions | |
| Gu et al. | Processing massive sized graphs using sector/sphere | |
| CN115309742A (en) | A method, device, electronic device and storage medium for building a table | |
| CN106227839A (en) | The expansion method of a kind of lustre file system and device | |
| Chen et al. | Performance comparison on the heterogeneous file system in cloud storage systems | |
| US12386713B2 (en) | Managing use of a shared virtual disk for accessing data in file based backups by multiple virtual machines | |
| US12306792B2 (en) | Managing access to file based backups based on storage units and workload use |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |