CN102129459A - Omnibearing enterprise data exchange high-speed engine - Google Patents

Abstract

The invention discloses an omnibearing enterprise data exchange high-speed engine. The engine is characterized in that: when data is inserted, an application system prepares a specified data object, and splices conditions for querying the data into a Key insertion engine; after obtaining storage resources by a resource allocator, the data is stored on a red black tree; a tree node points to a data storage space by a partial addressing mechanism, and stores the data object into the real storage space, thereby finishing the whole data insertion operation. When a server is powered off due to an anomaly, and the data in an index is stored into a file by a file mapping mechanism, thereby avoiding data loss under the condition of well storing a hard disk of the server. Therefore, the engine is highly efficient and safe.

Description

All-in-one enterprise data exchange high-speed engine

technical field

The invention relates to a high-speed engine for all-inhabitant enterprise data exchange.

Background technique

Data exchange is an essential mechanism for any computer application, service, or system. From a small Hello world program to a large operating system, data exchange is ubiquitous. Enterprise-level applications require frequent and large amounts of data exchange to support the entire system. This solution is a new solution for the characteristics of enterprise-level data exchange, hereinafter referred to as "engine".

At present, the means of enterprise-level data exchange applications include memory exchange, file exchange, and databases. However, these data exchange methods have disadvantages. First of all, memory exchange is the first choice for enterprise-level small data exchange. It has the advantage of fast speed, but its disadvantage is that it is difficult to manage. The exchange algorithms developed by developers of different levels vary widely. The experience of developers determines the stability and stability of the overall system. Processing efficiency, this is also the place where the system has the most frequent accidents on the production line; second, the data in the memory will be wiped out after the host computer is powered off. estimated loss. File exchange is also one of the commonly used methods. The advantage is that the stability of the data is guaranteed. After the host is powered off, the data is still in the file, but the efficiency is limited by the disk IO, and the difficulty of management is comparable to that of the memory method. Therefore, it can only be used when the efficiency requirement is very low. At present, the database is the most widely used exchange method in medium-sized systems. The data is relatively stable and has a complete maintenance mechanism. However, it is precisely because the database has a lot of considerations for the stability of the data, and the efficiency of use is often proportional to the hardware of the machine. In the face of Tens of thousands of visits per second often need to use a professional database server to meet. For small and medium-sized projects, this part of the cost will be directly deducted from the project's profit.

To sum up, it is not difficult to see that there is currently no cheap, safe and efficient means of mergers and acquisitions in enterprise-level switching solutions. Therefore, the "All-in-One Enterprise Data Exchange High-speed Engine" has emerged, which perfectly complements the strengths of all companies. It is suitable for the vacuum of medium and small application systems.

Contents of the invention

The purpose of the present invention is to provide a high-speed data exchange engine for all-environmental enterprises. The engine encapsulates addresses and resources based on the file mapping mechanism, and provides very simple methods for adding, deleting, checking, and modifying modules using the engine. The addressing algorithm adopts the red-black tree of the Linux operating system kernel to manage the memory, and the borrow and return resource allocator avoids directly allocating resources without searching. For ease of use, the way to access the engine is key-value correspondence. Simply put, the user can get an expected value after entering a specified key. The key can be a combination of various conditions, and the value can be a data structure or data object. , so the combination of keys and values can express searches of arbitrary complexity.

First of all, we need to introduce virtual memory. Virtual memory refers to the use of disk as an extension of RAM, so that the size of available memory increases accordingly. The kernel will write the contents of the temporarily unused block of memory to the hard disk, so that the memory can be used for other purposes. When the original content is needed, they are read back into memory. These operations are completely transparent to the user; programs running under the operating system just see that there is a large amount of memory available and do not notice that some of them reside on the hard disk from time to time. File mapping is the central concept of virtual memory. On the one hand, file mapping provides users with a set of measures. Users map files to a certain part of their own address space and use simple memory access instructions to read and write files; on the other hand, it also Can be used for the basic organizational mode of the kernel, in which the kernel sees the entire address space as a mapping of a set of distinct objects such as files.

As shown in Figure 2, after the application system prepares the specified data object when inserting data, it assembles the conditions for querying the data into a Key insertion engine, and the data is stored in the red-black tree after obtaining storage resources through the resource allocator. , the tree node points to the data storage space through the internal addressing mechanism, and the data object is stored in the real data storage space, and the entire action of inserting data is completed.

The whole process is completed in the virtual memory. The engine has been tested under 2CPU and 16G memory. 8 million records are inserted, each 1K of data reaches 521,580.388 records per second, 2790,373.212 records per second for search, and every 2764340.013 seconds. Forcibly power off the server during insertion, and continue processing after the server restarts.

When the server is powered off due to an abnormality, the data in the index has been saved to the file through the file mapping mechanism, so that the data will not be lost when the server hard disk is kept intact, so this engine is efficient and safe.

In summary, owing to adopting above-mentioned technical scheme, the beneficial effect of the present invention is:

The efficiency of virtual memory and real memory is almost the same, but the whole process will be completely put into the swap file, and the swap file becomes a controllable part through memory mapping, and the key part of the engine is also here. When exchanging files, the timing and efficiency of exchanging data are also guaranteed by the system (can be controlled manually). When data is exchanged, it enjoys the efficiency of memory and preserves the reliability of files. This is the core advantage of the engine. Modifying, deleting, and querying data are also consistent with the access process in the above figure, and will not be repeated here.

Description of drawings

The invention will be illustrated by way of example with reference to the accompanying drawings, in which:

Fig. 1 is a schematic diagram of the technical solution of the present invention.

Fig. 2 is a schematic diagram of the action of the application system when data is inserted in the present invention.

Detailed ways

All features disclosed in this specification, or steps in all methods or processes disclosed, may be combined in any manner, except for mutually exclusive features and/or steps.

Any feature disclosed in this specification (including any appended claims, abstract and drawings), unless expressly stated otherwise, may be replaced by alternative features which are equivalent or serve a similar purpose. That is, unless expressly stated otherwise, each feature is one example only of a series of equivalent or similar features.

As shown in Figure 2, after the application system prepares the specified data object when inserting data, it assembles the conditions for querying the data into a Key insertion engine, and the data is stored in the red-black tree after obtaining storage resources through the resource allocator. , the tree node points to the data storage space through the internal addressing mechanism, and the data object is stored in the real data storage space, and the entire action of inserting data is completed

When the server is powered off due to an abnormality, the data in the index has been saved to the file through the file mapping mechanism, so that the data will not be lost when the server hard disk is kept intact, so this engine is efficient and safe.

The present invention is not limited to the foregoing specific embodiments. The present invention extends to any new feature or any new combination disclosed in this specification, and any new method or process step or any new combination disclosed.

Claims (3)

1. A high-speed engine for all-inhabited enterprise data exchange, characterized in that: after the application system prepares the specified data object when inserting data, the conditions for querying the data are assembled into a Key insertion engine, and the data obtains storage resources through the resource allocator Finally, the data is stored in the red-black tree, and the tree node points to the data storage space through the local addressing mechanism, and the data object is stored in the real data storage space, thereby completing the entire action of inserting data. 2. The high-speed engine for all-environmental enterprise data exchange according to claim 1, characterized in that: The mode of accessing the engine is a key-value correspondence formula, and the collocation of keys and values can express searches of any complexity. 3. The high-speed engine for all-environmental enterprise data exchange according to claim 1, characterized in that: when the server is powered off due to abnormality, the data in the index has been saved in the file by the file mapping mechanism.

Images