Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a data exporting method and system, which can save storage and transmission resources and improve the overall efficiency of the system.
In view of the above object, a first aspect of the embodiments of the present invention provides a data export method, including performing the following steps:
initializing metadata information of the metadata data in response to insertion of a general table of the database into the metadata data;
detecting whether external table description information of an external file exists in response to receiving an insert command directed to write tuple data of the normal table to the external file of the writable external table;
inserting an external file writable with the external table into the external table description information in response to determining that the external table description information exists;
scanning and acquiring metadata in the common table to determine whether the metadata needs to be exported based on the metadata information;
the metaset data is flushed to the disk file in response to determining that the metaset data needs to be exported.
In some embodiments, the metadata information includes header information and external data, the header information including an external file and an export flag bit;
the method further comprises the following steps: while initializing the metadata information of the tuple data, the external file of the header information is also reset and the flag of the export flag bit is cleared.
In some embodiments, the method further comprises: in response to determining that the external table description information does not exist, allocating memory to create the external table description information, and initializing and resetting the external file, the number of tuple attributes of the writable external table, and the converted text format data in the external table description information.
In some embodiments, inserting the external file of the writable external table into the external table description information comprises: the external file information writable with the external table is inserted into the external file reset in the external table description information.
In some implementations, determining whether the metadata data needs to be derived based on the metadata information includes:
determining that tuple data needs to be exported in response to determining that the external file of header information is reset;
determining that metadata needs to be exported in response to determining that the external file of the external table description information does not include external file information;
determining that the tuple data needs to be derived in response to determining that the derived flag bit of the header information is unmarked.
In some embodiments, the method further comprises: in response to a successful flush of the tuple data to the disk file, a flag is assigned to the derived flag bit of the header information to update the tuple data in the normal table.
In some embodiments, the method further comprises: in response to a failure to flush the tuple data to the disk file, re-waiting for receipt of the insert command.
A second aspect of an embodiment of the present invention provides a data export system, including:
a processor; and
a memory storing program code executable by the processor, the program code when executed performing the steps of:
initializing metadata information of the tuple data in response to an ordinary table of the database being inserted into the tuple data;
detecting whether external table description information of an external file exists in response to receiving an insert command directed to write tuple data of the normal table to the external file of the writable external table;
inserting an external file writable with the external table into the external table description information in response to determining that the external table description information exists;
scanning and acquiring metadata in the common table to determine whether the metadata needs to be exported based on the metadata information;
the metaset data is flushed to the disk file in response to determining that the metaset data needs to be exported.
In some embodiments, the metadata information includes header information and external data, the header information including an external file and an export flag bit;
the method also comprises the following steps: when the metadata information of the metadata group data is initialized, the external file of the header information is reset and the mark of the export mark bit is cleared; in response to determining that the external table description information does not exist, allocating a memory to create the external table description information, and initializing and resetting an external file, the number of tuple attributes writable with the external table, and the converted text format data in the external table description information;
inserting the external file of the writable external table into the external table description information includes: the external file information writable with the external table is inserted into the external file reset in the external table description information.
In some implementations, determining whether the metadata data needs to be derived based on the metadata information includes: determining that tuple data needs to be exported in response to determining that the external file of header information is reset; determining that metadata needs to be exported in response to determining that the external file of the external table description information does not include external file information; determining that the tuple data needs to be derived in response to determining that the derived flag bit of the header information is unmarked.
The invention has the following beneficial technical effects: according to the data export method and system provided by the embodiment of the invention, the metadata information of the metadata is initialized by responding to the insertion of the common table of the database into the metadata; detecting whether external table description information of an external file exists in response to receiving an insert command directed to write tuple data of the normal table to the external file of the writable external table; inserting an external file writable with the external table into the external table description information in response to determining that the external table description information exists; scanning and acquiring the metadata in the common table to determine whether the metadata needs to be exported based on the metadata information; in response to the technical scheme that the tuple data needs to be exported, the tuple data is refreshed on the disk file, so that the storage and transmission resources can be saved, and the overall efficiency of the system is improved.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
In view of the foregoing, a first aspect of the embodiments of the present invention provides an embodiment of a data export method that saves storage and transmission resources and improves overall system efficiency. Fig. 1 is a schematic flow chart of a data export method provided by the present invention.
The data export method, as shown in fig. 1, includes the following steps:
step S101, in response to the insertion of the metadata into the ordinary table of the database, the metadata information of the metadata is initialized;
step S103, in response to receiving an insertion command for writing tuple data of the ordinary table to an external file of the writable external table, detecting whether external table description information of the external file exists;
step S105, in response to determining that the external table description information exists, inserting an external file in which the external table can be written into the external table description information;
step S107, scanning and acquiring the metadata in the common table to determine whether the metadata needs to be exported based on the metadata information;
step S109, in response to determining that the tuple data needs to be exported, flushes the tuple data onto the disk file.
The invention aims to ensure the effective export of data, avoid the waste of storage space and the occupation of excessive input and output resources, save the export time of mass data, effectively reduce the unnecessary workload caused by network problems, insufficient equipment resources or artificial misoperation and improve the working efficiency.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), or the like. Embodiments of the computer program may achieve the same or similar effects as any of the preceding method embodiments to which it corresponds.
In some embodiments, the metadata information includes header information and external data, the header information including an external file and an export flag bit; the method further comprises the following steps: while initializing the metadata information of the tuple data, the external file of the header information is also reset and the flag of the export flag bit is cleared.
In some embodiments, the method further comprises: in response to determining that the external table description information does not exist, allocating memory to create the external table description information, and initializing and resetting the external file, the number of tuple attributes of the writable external table, and the converted text format data in the external table description information.
In some embodiments, inserting the external file of the writable external table into the external table description information comprises: the external file information writable with the external table is inserted into the external file reset in the external table description information.
In some implementations, determining whether the metadata data needs to be derived based on the metadata information includes:
determining that tuple data needs to be exported in response to determining that the external file of header information is reset;
determining that the meta-data needs to be exported in response to determining that the external file of the external table description information does not include the external file information;
determining that the tuple data needs to be derived in response to determining that the derived flag bit of the header information is unmarked.
In some embodiments, the method further comprises: in response to a successful flush of the tuple data to the disk file, a flag is assigned to the derived flag bit of the header information to update the tuple data in the normal table.
In some embodiments, the method further comprises: in response to a failure to flush the tuple data to the disk file, re-waiting for receipt of the insert command.
The following further illustrates embodiments of the invention in accordance with the specific example shown in fig. 2.
Referring to fig. 2, in step 1, when inserting tuple data into a normal table of a database, a record normal table information module is entered, in which information of the tuple, including header information and user data, is initialized. Here, the external file and the export flag bit in the header information of each tuple are set to 0, and then step 2 is performed;
entering a SQL command waiting module in the step 2, waiting for an INSERT command which can write an external table and input by a user in the module, writing the tuple of the common table into an external file by the command, executing the step 3 when the command is issued, and continuing to execute the step 2 and continuing to wait for the SQL command;
entering a module for detecting the external table description information in the step 3, judging whether the external table description information exists in the module, if not, entering the step 4, otherwise, entering the step 5;
entering an external table description information initialization module in step 4, wherein a memory is allocated to store external table description information, and external files, the tuple attribute number of the external table and the converted text format data in the description information are initialized to 0 (null), and then step 5 is executed;
entering an external table description information updating module in the step 5, wherein if the external file in the external table description information is null (0), the external file specified in the writable external table is opened, and the external file information is recorded in the corresponding external table description information, otherwise, the step 6 is executed without updating;
entering a module for obtaining the element groups of the common table in the step 6, sequentially scanning and obtaining each element group in the common table in the module, and then executing the step 7;
entering a module for detecting a common table tuple in the step 7, wherein in the module, firstly, whether an external file field in tuple header information is empty is detected, and if the external file field in the tuple header information is empty, entering a step 8; otherwise, detecting whether the external file in the external table description information is in the recorded external file information of the tuple, if not, entering step 8, if yes, finding the external disk file information of the current external table in the tuple header information, detecting whether the export marker bit of the tuple in the information is 0, if 0, entering step 8, and if not, returning to step 6;
entering a write-in external file module in the step 8, converting the tuple of the common table into a text format in the module, writing the tuple into a file, entering the step 9 if the tuple is successful, and returning to the step 2 if the tuple is not successful;
entering into a module for updating the ordinary table tuple in step 9, in the module, recording the external file defined in the currently writable external table into the header information of the tuple, and updating the export flag bit of the tuple to be 1, if the external file has been recorded by the tuple, finding the file to be written currently in all the external files recorded by the tuple, and setting the export flag bit of the tuple corresponding to the file to be 1.
As can be seen from the foregoing embodiments, the data export method provided by an embodiment of the present invention initializes metadata information of metadata by inserting the metadata in response to a general table of a database; detecting whether external table description information of an external file exists in response to receiving an insert command directed to write tuple data of the normal table to the external file of the writable external table; inserting an external file writable with the external table into the external table description information in response to determining that the external table description information exists; scanning and acquiring metadata in the common table to determine whether the metadata needs to be exported based on the metadata information; in response to the technical scheme that the tuple data needs to be exported, the tuple data is refreshed on the disk file, so that the storage and transmission resources can be saved, and the overall efficiency of the system is improved.
It should be particularly noted that, the steps in the embodiments of the data deriving method described above can be mutually intersected, replaced, added, or deleted, so that these reasonable permutation and combination transformations should also belong to the scope of the present invention, and should not limit the scope of the present invention to the described embodiments.
In view of the foregoing, a second aspect of the embodiments of the present invention provides an embodiment of a data export system that saves storage and transmission resources and improves overall efficiency of the system. The system comprises:
a processor; and
a memory storing program code executable by the processor, the program code when executed performing the steps of:
initializing metadata information of the metadata data in response to insertion of a general table of the database into the metadata data;
detecting whether external table description information of an external file exists in response to receiving an insert command directed to write tuple data of the normal table to the external file of the writable external table;
inserting an external file, in which the external table can be written, into the external table description information in response to determining that the external table description information exists;
scanning and acquiring the metadata in the common table to determine whether the metadata needs to be exported based on the metadata information;
the metaset data is flushed to the disk file in response to determining that the metaset data needs to be exported.
In some embodiments, the metadata information includes header information and external data, the header information including an external file and an export flag bit;
the method also comprises the following steps: when the metadata information of the metadata group data is initialized, the external file of the header information is reset and the mark of the export mark bit is cleared; in response to determining that the external table description information does not exist, allocating a memory to create the external table description information, and initializing and resetting an external file, the number of tuple attributes writable with the external table, and the converted text format data in the external table description information; inserting the external file of the writable external table into the external table description information includes: the external file information writable with the external table is inserted into the external file reset in the external table description information.
In some implementations, determining whether the metadata data needs to be derived based on the metadata information includes: determining that tuple data needs to be exported in response to determining that the external file of header information is reset; determining that metadata needs to be exported in response to determining that the external file of the external table description information does not include external file information; determining that the tuple data needs to be derived in response to determining that the derived flag bit of the header information is unmarked.
As can be seen from the above embodiments, the system provided by the embodiments of the present invention initializes metadata information of metadata by inserting the metadata in response to a general table of a database; detecting whether external table description information of an external file exists in response to receiving an insert command directed to write tuple data of the normal table to the external file of the writable external table; inserting an external file writable with the external table into the external table description information in response to determining that the external table description information exists; scanning and acquiring metadata in the common table to determine whether the metadata needs to be exported based on the metadata information; in response to the technical scheme that the tuple data needs to be exported, the tuple data is refreshed on the disk file, so that the storage and transmission resources can be saved, and the overall efficiency of the system is improved.
It should be particularly noted that the embodiment of the system described above employs the embodiment of the data export method to specifically describe the working process of each module, and those skilled in the art can easily think that these modules are applied to other embodiments of the data export method. Of course, since the steps in the data deriving method embodiment can be mutually intersected, replaced, added, and deleted, these reasonable permutation and combination transformations should also belong to the scope of the present invention for the system, and should not limit the scope of the present invention to the embodiment.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of an embodiment of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.