CN111209282A

CN111209282A - Data storage method and device, computer equipment and storage medium

Info

Publication number: CN111209282A
Application number: CN202010025819.9A
Authority: CN
Inventors: 周霞; 申志彬; 李如先
Original assignee: Shenzhen Qianhai Huanrong Lianyi Information Technology Service Co Ltd
Current assignee: Shenzhen Qianhai Huanrong Lianyi Information Technology Service Co Ltd
Priority date: 2020-01-10
Filing date: 2020-01-10
Publication date: 2020-05-29

Abstract

The embodiment of the invention discloses a data storage method, a data storage device, computer equipment and a storage medium. The method comprises the following steps: storing the data into a data base layer of a data warehouse in an avro storage format; storing the data in a textfile storage format into a base index layer of a data warehouse; the data is stored in the textfile storage format into the application layer of the data warehouse. By implementing the method of the embodiment of the invention, the data of each layer of the data warehouse can be stored in different formats according to the characteristics of the data of each layer of the data warehouse, so that the convenience of data acquisition of each layer of the data warehouse is greatly improved, the data acquisition efficiency is improved, and the power consumption of a server is reduced.

Description

Data storage method and device, computer equipment and storage medium

Technical Field

The present invention relates to the field of big data technologies, and in particular, to a data storage method and apparatus, a computer device, and a storage medium.

Background

The data warehouse of big data generally comprises three layers, namely a base layer, a base index layer and an application layer. In the existing open source community, the data in the database of each layer in the data warehouse above the big data platform are stored based on the same format. Because the applications of each layer in the data warehouse are different, the convenience for acquiring data of different layers is inconsistent, the data acquisition efficiency is low, and the power consumption of the server is increased.

Disclosure of Invention

The embodiment of the invention provides a data storage method, a data storage device, computer equipment and a storage medium, and aims to solve the problems of low data acquisition efficiency and high server power consumption caused by poor convenience in data acquisition from a data warehouse in the prior art.

In a first aspect, an embodiment of the present invention provides a data storage method, which includes:

storing the data into a data base layer of a data warehouse in an avro storage format;

storing the data in a textfile storage format into a base index layer of a data warehouse;

the data is stored in the textfile storage format into the application layer of the data warehouse.

The further technical scheme is that the method also comprises the following steps:

and setting the table structure of the database in the data base layer as an appearance.

and setting a table structure of the database in the basic index layer as an inner table.

and setting a table structure of the database in the application layer as an inner table.

The further technical scheme is that in the textfile storage format, separators between columns are \ t; in the textfile storage format, the line-to-line separators are \ n.

In a second aspect, an embodiment of the present invention further provides a data storage device, including:

the data warehouse comprises a first storage unit, a second storage unit and a data processing unit, wherein the first storage unit is used for storing data into a data base layer of the data warehouse in an avro storage format;

the second storage unit is used for storing the data into a basic index layer of the data warehouse in a textfile storage format;

and the third storage unit is used for storing the data into the application layer of the data warehouse in a textfile storage format.

The further technical scheme is that the data storage device further comprises:

the first setting unit is used for setting the table structure of the database in the data base layer as an appearance;

a second setting unit, configured to set a table structure of the database of the base index layer as an inner table;

and the third setting unit is used for setting the table structure of the database in the application layer into an inner table.

In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the above method when executing the computer program.

In a fourth aspect, the present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program can implement the above method when being executed by a processor.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic view of an application scenario of a data storage method according to an embodiment of the present invention;

FIG. 2 is a schematic block diagram of a data storage device provided by an embodiment of the present invention;

FIG. 3 is a schematic block diagram of a data storage device according to another embodiment of the present invention;

fig. 4 is a schematic block diagram of a computer device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".

Referring to fig. 1, fig. 1 is a schematic flow chart illustrating a data storage method according to an embodiment of the present invention. As shown, the method includes the following steps S1-S3.

And S1, storing the data into the data base layer of the data warehouse in the avro storage format.

In an embodiment of the invention, the data warehouse comprises a data base layer, a base index layer and an application layer. The data in the data base layer is obtained by data extraction from the service system. The data in the basic index layer is obtained by processing the data in the data basic layer. The data in the application layer is obtained by processing the data in the basic index layer.

In a specific implementation, the data is stored in the data base layer of the data warehouse in an avro storage format. The data base layer has huge data size and is suitable for being stored in an avro storage format with better compression performance.

Further, in certain embodiments, such as the present embodiment, the method further comprises: and setting the table structure of the database in the data base layer as an appearance. The appearance table building mode is adopted because if the condition except deleting the table is met, only the metadata of the table is deleted, the data cannot be deleted, and the data can be quickly recovered.

S2, storing the data in a textfile storage format into a base index layer of the data warehouse.

In a specific implementation, the data is stored in a textfile storage format into the base index layer of the data warehouse. the data in the textfile storage format can be opened and read by business personnel in various tools and modes, and the reading convenience is improved.

Further, in certain embodiments, such as the present embodiment, the method further comprises: and setting a table structure of the database in the basic index layer as an inner table. The appearance table building mode is adopted, and when the inner table is deleted, the metadata and the data are completely deleted, so that the index reprocessing and storing are facilitated.

It should be noted that in the textfile storage format, the separators between columns are \ t; in the textfile storage format, the line-to-line separators are \ n. The above separators can be adopted to facilitate the business personnel to open and read the data in the basic index layer in various tools and modes.

S3, storing the data in a textfile storage format into the application layer of the data warehouse.

In a specific implementation, the data is stored in a textfile storage format in the application layer of the data warehouse. the data in the textfile storage format can be opened and read by business personnel in various tools and modes, and the reading convenience is improved.

Further, in certain embodiments, such as the present embodiment, the method further comprises: and setting a table structure of the database in the application layer as an inner table.

The appearance table building mode is adopted, and when the inner table is deleted, the metadata and the data are completely deleted, so that the index reprocessing and storing are facilitated.

It should be noted that in the textfile storage format, the separators between columns are \ t; in the textfile storage format, the line-to-line separators are \ n. The above separators can be adopted to facilitate the business personnel to open and read the data in the application layer in various tools and modes.

By applying the technical scheme of the embodiment of the invention, the data is stored in the data base layer of the data warehouse in the avro storage format; storing the data in a textfile storage format into a base index layer of a data warehouse; the data is stored in the textfile storage format into the application layer of the data warehouse. The data of each layer of the data warehouse are stored in different formats according to the characteristics of the data of each layer of the data warehouse, so that the convenience of data acquisition of each layer of the data warehouse is greatly improved, the data acquisition efficiency is improved, and the power consumption of the server is reduced.

Fig. 2 is a schematic block diagram of a data storage device 40 provided by an embodiment of the present invention. As shown in fig. 2, the present invention also provides a data storage device 40 corresponding to the above data storage method. The data storage device 40 includes a unit for executing the above-described data storage method, and the device may be configured in a server. Specifically, referring to fig. 2, the data storage device 40 includes a first storage unit 41, a second storage unit 42, and a third storage unit 43.

The first storage unit 41 is configured to store the data in the data base layer of the data warehouse in the avro storage format.

And the second storage unit 42 is used for storing the data into the base index layer of the data warehouse in a textfile storage format.

And a third storage unit 43, configured to store the data in a textfile storage format into the application layer of the data warehouse.

In one embodiment, in the textfile storage format, the separators between columns are \ t; in the textfile storage format, the line-to-line separators are \ n.

Fig. 3 is a schematic block diagram of a data storage device 40 according to another embodiment of the present invention. As shown in fig. 3, the data storage device 40 of the present embodiment is the above-mentioned embodiment, and is added with a first setting unit 44, a second setting unit 45, and a third setting unit 46.

A first setting unit 44, configured to set a table structure of the database in the data base layer as an appearance.

A second setting unit 45, configured to set a table structure of the database of the base index layer as an inner table.

A third setting unit 46, configured to set a table structure of the database in the application layer as an inner table.

It should be noted that, as can be clearly understood by those skilled in the art, the specific implementation process of the data storage device 40 and each unit may refer to the corresponding description in the foregoing method embodiment, and for convenience and brevity of description, no further description is provided herein.

The data storage means 40 may be implemented in the form of a computer program which is executable on a computer device as shown in fig. 4.

Referring to fig. 4, fig. 4 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a terminal or a server, where the terminal may be an electronic device with a communication function, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, and a wearable device. The server may be an independent server or a server cluster composed of a plurality of servers.

Referring to fig. 4, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.

The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, causes the processor 502 to perform a data storage method.

The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.

The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 can be caused to execute a data storage method.

The network interface 505 is used for network communication with other devices. Those skilled in the art will appreciate that the configuration shown in fig. 4 is a block diagram of only a portion of the configuration associated with the present application and does not constitute a limitation of the computer device 500 to which the present application may be applied, and that a particular computer device 500 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps:

In one embodiment, processor 502 further implements the steps of:

setting a table structure of a database in the data base layer as an appearance;

setting a table structure of a database of the basic index layer as an inner table;

It should be understood that, in the embodiment of the present Application, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It will be understood by those skilled in the art that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program instructing associated hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.

Accordingly, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program. The computer program, when executed by a processor, causes the processor to perform the steps of:

In an embodiment, the processor, in executing the computer program, further implements the steps of:

The storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, which can store various computer readable storage media.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.

The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be merged, divided and deleted according to actual needs. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, while the invention has been described with respect to the above-described embodiments, it will be understood that the invention is not limited thereto but may be embodied with various modifications and changes.

While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method of storing data, comprising:

2. The data storage method of claim 1, wherein the method further comprises:

3. The data storage method of claim 1, wherein the method further comprises:

4. The data storage method of claim 1, wherein the method further comprises:

5. The data storage method of claim 1, wherein in the textfile storage format, the separators between columns are \ t; in the textfile storage format, the line-to-line separators are \ n.

6. A data storage device, comprising:

7. The data storage device of claim 6, further comprising:

8. The data storage device of claim 6, wherein in the textfile storage format, the column-to-column separators are \ t; in the textfile storage format, the line-to-line separators are \ n.

9. A computer arrangement, characterized in that the computer arrangement comprises a memory having stored thereon a computer program and a processor implementing the method according to any of claims 1-5 when executing the computer program.

10. A computer-readable storage medium, characterized in that the storage medium stores a computer program which, when being executed by a processor, is adapted to carry out the method according to any one of claims 1-5.