CN117112691A - A storage method for multi-storage engine database oriented to big data - Google Patents

Abstract

The invention discloses a big data-oriented multi-storage engine database system and a storage method thereof, wherein the system comprises the following steps: an SQL compiler; a stored procedure compiler; a transaction management unit; storing in a distributed memory array; a distributed execution engine; a data source connector; a multi-tenant management component; and a middleware management unit. The method comprises the following steps: unified data storage; unified metadata management; unifying the data model; a unified SQL parsing engine; and unified safety control. The invention ensures that at least more than two domestic processors are supported, and the performance of the multi-storage engine database is improved by 10x to 100x times compared with the open source Hadoop version, thereby meeting the low-delay requirement of an online storage and online service analysis system. SQL supports integrity and performance far ahead of Cloudera Impala. The method is also superior to other Hadoop and MPP products in TPC-DS and TPC-H reference tests, and can realize deep fusion of military databases, typical domestic CPUs and military application scenes.

Description

A storage method for multi-storage engine database oriented to big data

Technical field

The present invention relates to the field of data storage technology, and in particular to a storage method for a multi-storage engine database oriented to big data.

Background technique

With the explosive growth of battlefield information, the extensive use of sensors, and the continuous improvement of information systems, traditional database storage methods can no longer meet the requirements for capturing and understanding battlefield information and a large amount of military knowledge. A large number of data types such as pictures, text, and numerical values in military applications have become increasingly complex and large. It is almost impossible to perform statistics, screening, and machine learning in massive and disorganized data. This requires data analysts and model trainers to initialize the data before analysis and training, that is, to organize and classify the data, and filter dirty data. Wrong data and fill in missing data. Due to the continuous expansion of military information data, data analysts and model training personnel need to spend a lot of energy on data sorting, need to learn a lot of additional data processing knowledge, and require certain engineering capabilities, which makes analysts and algorithms The personnel threshold has increased, and the company's labor costs have greatly increased. In order to solve the above problems, there is an urgent need for a highly available multi-storage engine database based on big data architecture to automatically organize and classify data to reduce more labor costs and allow data analysts and model training personnel to spend more time and energy. in analysis and algorithms. In order to fill this technological gap, multi-storage engine database technology for big data has shown great advantages in solving knowledge representation, knowledge storage, knowledge query, etc., and can meet the needs of military information retrieval, intelligence analysis, and situation awareness. As the physical starting point for data analysis, storage is facing the challenges of big data applications.

Taking the typical combat and support data requirements as the background, based on the domestic basic software and hardware structure and technical characteristics, we strive to break through the multi-storage engine unified management technology, unified SQL parsing and model mapping technology, and multi-engine distributed elastic expansion technology for massive military data. Key technologies such as database real-time computing engine technology and graph database have formed a typical domestic independently controllable multi-storage engine fusion database, and a prototype system of the database core engine has been developed to support the unified management and processing of complex data in typical military application scenarios.

Contents of the invention

The purpose of the present invention is to provide a multi-storage engine database system and storage method for big data, so as to solve the problems existing in the existing technology.

In order to achieve the above objectives, the present invention provides a multi-storage engine database system for big data, including:

SQL compiler supports ANSI SQL 92 and SQL 99 standards, and supports ANSI SQL 2003 OLAP core extensions to meet the SQL requirements of the data warehouse business and facilitate smooth migration of applications;

The stored procedure compiler supports complete data types, process control, packages, cursors, exception handling and dynamic SQL execution, and supports high-speed statistics, additions, deletions, modifications and distributed transaction operations in stored procedures to meet the requirements of data applications. Migration of database to Inceptor platform;

The transaction management unit implements consistency and isolation control through the two-stage blocking protocol and MVCC, and supports the Serializable Snapshot Isolation isolation level, thus ensuring transaction consistency under concurrent conditions;

Distributed memory column storage, based on memory or SSD column storage engine Holodesk, which stores data in memory or SSD in column format, supplemented by a memory-based execution engine, completely avoiding the delay caused by IO;

The distributed execution engine independently builds a distributed data layer, which separates the calculation data from the JVM memory space of the calculation engine, effectively reducing the impact of JVM GC on system performance and stability;

The data source connector connects the execution engine and various data sources, and connects data from multiple different data sources to the engine for real-time statistical analysis without the need to import the data into HDFS in advance, making it more convenient for users to build diverse business needs;

The multi-tenant management component provides complete multi-tenant management functions, including tenant resource management, tenant permission management and security control modules, which facilitates the management and allocation of multi-tenants on a unified big data platform; allows CPU and For the configuration and management of memory resources, different tenants use different CPU and memory resource pools, so they will not interfere with each other;

The middleware management unit supports JDBC 4.0 and ODBC 3.5 standards, so it can support Hibernate/Spring middleware, is fully compatible with Tableau/QlikView/Cognos reporting tools, and can be fully connected with the current data application layer of the enterprise.

Furthermore, the stored procedure compiler includes a complete optimizer, including CFG Optimizer, Parallel Optimizer, and DAG Optimizer. CFG Optimizer optimizes the code in the stored procedure, completes loop unrolling, redundant code elimination, function inlining, etc. Major optimizations.

Furthermore, the transaction management unit supports starting transactions with Begin Transaction, ending transactions with commit or rollback, achieving consistency and isolation control through the two-stage blocking protocol and MVCC, and supporting the Serializable Snapshot Isolation isolation level, thus ensuring concurrency. transactional consistency.

Furthermore, the Holodesk supports building distributed indexes for data fields, allowing users to build OLAP-Cubes for multi-field combinations, and store the cubes directly in memory or SSD, without the need for additional BI tools to build cubes.

Furthermore, the distributed execution engine's cost-based optimizer and rule-based optimizer, supplemented by 100 optimization rules, ensure that SQL applications can maximize performance without manual changes; the distributed execution engine includes Two execution modes: low-latency mode and high-throughput mode.

Further, after the execution plan starts, the data source connector extracts the required data from other data sources through pre-established connections, enters the execution engine layer to participate in SQL calculation, and after the calculation is completed, releases the relevant database connection and the corresponding resource.

The present invention also provides a storage method for a big data-oriented multi-storage engine database using the big data-oriented multi-storage engine database system, including:

Step 1. Unify data storage, integrating the physically independent and management-separated major database systems into a complete and unified data storage;

Step 2: Unify metadata management and deploy a metadata warehouse in the logical data warehouse to use an RDBMS cluster specifically to store and manage metadata; create a new table or new index in each database system, and the logical data warehouse will store the metadata in the metadata warehouse. Synchronously create a two-dimensional metadata mapping table in the data warehouse, and record the relationship between the two-dimensional metadata mapping table and the original table and/or index, laying the foundation for unified SQL access; the metadata management system stores all Database object information provides a query interface for other systems to retrieve. The queryable information includes but is not limited to the physical distribution of data, data distribution characteristics, maximum and minimum values, and permission information;

Step 3. Unify the data model and uniformly map the data model of the multi-engine database system to the two-dimensional relational data mapping table of the relational database, laying a theoretical foundation for centralized and unified management of metadata and unified SQL access;

Step 4. Unify the SQL parsing engine to provide unified SQL parsing, optimization and execution services for all data systems in the logical data warehouse. The SQL statements submitted by the user are first received by the SQL engine, converted into Spark code after parsing and optimization, and then It is executed by a high-performance distributed computing cluster. During the execution process, the native API of each system will be called to achieve data access;

Step 5. Unify security management and control, a unified authentication system based on users and roles, comply with the account and/or role RBAC model, implement permission management through roles, and batch authorization management for users; support the security protocol Kerberos, and use LDAP as the account management system , and perform unified security authentication of account information through Kerberos.

Furthermore, the unified metadata management in step 2 further includes providing metadata access and collection of Hive, HDFS, and HBase; providing a UI interface for unified metadata management and an API interface in Restful form of related services, and providing various types of microservices In the docking method, the data storage of the UI front-end page data of unified metadata management adopts MySQL database table; provides the Messaging message queue (currently using Kafka) and the metadata operation interface and data message bus mode of the API interface; provides the TypeSystem type of unified metadata System, graph computing storage query engine layer, smart label algorithm, knowledge graph model; provides a public storage encapsulation layer for the graph computing query engine, and supports the JanusGraph open source graph computing storage query engine.

Furthermore, the unified metadata management in step 2 further includes real-time and efficient indexing of heterogeneous data of graphs, key values, documents, and relationships, and supports mixed index BRPQ of numerical point attributes, numerical segment attributes, spatial locations, and text, specifically including Construction of PSPQ index, combination of PSPQ and keyword index, spatial keyword distributed query with relational attributes, and index construction based on Lucene.

Furthermore, the unified security management and control architecture in step 5 is divided into 4 layers: the system layer uses improved ApacheDS to increase reading and writing efficiency by more than 10 times, and uses the same set of users and a unified LDAP/Kerberos authentication method to avoid OpenLDAP using Kerberos authentication. , to accelerate the efficiency of LDAP authentication; the business layer implements complete ARBAC model support, providing REST API, user-friendly WebUI and password policy functional support; the plug-in layer uses plug-ins to provide authentication, authorization, group mapping and quotas for each component Management enables each component to use a unified user, group and permission management model; various platform PaaS and SaaS services connected to the service layer are protected by unified security management and control.

The method of the present invention has the following advantages:

The invention ensures that it can support at least two domestic processors. By verifying the prototype system, the multi-storage engine database has 10x-100x performance improvement compared to the open source Hadoop version, meeting the low-latency requirements of online storage and online business analysis systems. Combined with the optimization of the execution engine and data storage layer, the completeness and performance of SQL support are significantly ahead of Cloudera Impala. It also outperforms other Hadoop and MPP products in TPC-DS and TPC-H benchmark tests, and can achieve deep integration of military databases, typical domestic CPUs and military application scenarios.

Description of drawings

Figure 1 shows the overall architecture diagram of key technology implementation;

Figure 2 shows the unified metadata management architecture;

Figure 3 shows the Thrift communication protocol diagram;

Figure 4 shows the unified security management and control architecture diagram;

Figure 5 shows the Kerbeors authentication process diagram;

Figure 6 shows the PSPQ index structure;

Figure 7 shows a schematic diagram of query based on B-tree;

Figure 8 shows the BRPQ index structure.

Detailed ways

The technical solutions of the present invention will be described clearly and completely below with reference to specific embodiments. However, those skilled in the art should understand that the embodiments described below are only used to illustrate the present invention and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

Based on the domestic processor platform, the present invention innovatively proposes to build a unified data storage engine through multi-storage engine database technology for big data, establish a unified metadata management mechanism, a unified data model, and create a unified and efficient distributed SQL Engine provides powerful platform support for military business scenarios.

A multi-storage engine database system for big data of the present invention is shown in Figure 1, including:

1.SQL CompilerSQL 2003 Compiler

Applications such as enterprise-level data warehouses and data marts are mostly developed based on SQL. However, most products in the Hadoop industry have poor SQL compatibility programs or do not support modular expansion of SQL. Therefore, the cost of application migration is very high, even Not feasible. In order to reduce application migration costs, Transwarp Inceptor has developed a complete SQL compiler that supports ANSI SQL 92 and SQL 99 standards, and supports ANSI SQL 2003 OLAP core extensions, which can meet the SQL requirements of most existing data warehouse businesses. Facilitates smooth migration of applications.

In addition to a better SQL semantic analysis layer, Inceptor includes a powerful optimizer to ensure the best performance of SQL on the engine. Inceptor contains 3 levels of optimizers: first, a rule-based optimizer, which applies static optimization rules and generates a logical execution plan, and second, a cost-based optimizer, which measures the CPU, IO and network costs of multiple different execution plans. Choose a more reasonable plan and generate a physical execution plan; finally, there is the code generator, which generates more efficient execution code or JavaByte Code for some core execution logic, thereby ensuring the best performance of SQL business on a distributed platform.

2. Stored procedure compiler PL/SQL Compiler

Most of the existing data warehouse applications in China are based on SQL 2003, and a large number of stored procedures are used to build complex applications. Therefore, in addition to the SQL compiler, Transwarp Inceptor also contains a stored procedure compiler for compiling and executing stored procedures.

Inceptor supports the two mainstream SQL standards of Oracle PL/SQL and DB2 SQL PL, including complete data types, process control, Package, cursor, exception handling and dynamic SQL execution, and supports high-speed statistics, addition, deletion, modification, query and Distributed transaction operations. Therefore, with the addition of the stored procedure compiler, Inceptor can satisfy the migration of most data applications from relational databases to the Inceptor platform.

In addition to support at the SQL syntax level, the stored procedure compiler includes a complete optimizer, including CFGOptimizer, Parallel Optimizer, and DAG Optimizer. CFG Optimizer optimizes the code in the stored procedure and completes major optimizations such as loop unrolling, redundant code elimination, and function inlining. Parallel Optimizer parallelizes some originally serial logic and uses the computing power of the cluster to improve the overall execution speed. The performance improvement of some key functions such as cursors is very obvious. DAG Optimizer will perform secondary optimization based on the generated DAG graph to generate a more reasonable physical execution plan, focusing on reducing the overhead of tasks such as shuffle. In order to be effectively compatible with other databases, Inceptor supports isolating the differences between different SQL standards through different dialect settings, thereby avoiding the ambiguity of data calculation and processing standards, thus ensuring the correctness of data processing.

3.Transaction Manager

In order to better meet the needs of data warehouse business scenarios, Inceptor provides complete SQL support for additions, deletions, and modifications, allowing data to be processed from multiple data sources. At the same time, in order to effectively ensure the accuracy of data processing, Inceptor provides distributed transaction support to ensure the ACID of data during processing, that is, atomicity, consistency, isolation, and durability.

Inceptor supports starting transactions with Begin Transaction and ending transactions with commit or rollback. The transaction management unit implements consistency and isolation control through the two-stage blocking protocol and MVCC, and supports the Serializable Snapshot Isolation isolation level, thus ensuring transaction consistency under concurrent conditions.

Inceptor supports the semantic specifications of the add, delete, modify and query parts in SQL2003, supports Insert, Update, Delete, Truncate and Merge Into primitives, supports single or updating data tables from other data tables and nested queries, and has built-in consistency checking functions to Prevent illegal changes.

Through the optimization of the SQL compiler, the addition, deletion and modification of SQL execution plans are executed concurrently in the cluster through the distributed engine. The overall throughput of the system can reach several times that of the relational database, which can meet the high throughput requirements of batch processing business. In addition, through reasonable resource planning, Inceptor allows tenants to perform high-speed statistical analysis on data while adding, deleting, and modifying data.

4. Distributed memory column storage Holodesk

In order to speed up interactive analysis, Inceptor launched Holodesk, a column storage engine based on memory or SSD. Holodesk stores data in columnar format in memory or SSD, supplemented by a memory-based execution engine, which can completely avoid delays caused by IO and greatly improve data scanning speed.

In addition to columnar storage to speed up statistical analysis, Holodesk supports building distributed indexes for data fields. Using intelligent indexing technology to build the best query plan for queries, Inceptor can reduce SQL query latency to milliseconds.

Holodesk allows users to build OLAP-Cube for multi-field combinations and store the cube directly in memory or SSD. No additional BI tools are needed to build the cube. Therefore, for some complex statistical analysis and report interactive queries, Holodesk can achieve seconds-level processing. reaction.

In addition to performance advantages, Holodesk also excels in usability. Holodesk's metadata and storage natively support high availability, supporting exception handling and disaster recovery through consistency protocols and multiple versions. Under abnormal circumstances, Holodesk can automatically restore and reconstruct all table information and data without manual recovery, thereby reducing development and operation and maintenance costs and ensuring system stability.

stability. Inceptor focuses on optimizing the performance of Holodesk based on SSD, making the performance based on PCIE SSD reach more than 80% of the full memory solution. Therefore, the combination of low-cost memory/flash hybrid storage solutions can approach the analysis performance of full-memory storage, ensuring a cost-effective solution.

5. Distributed Execution Engine

Inceptor has deeply developed a dedicated distributed computing engine based on Apache Spark, which not only greatly improves computing performance, but also effectively solves many stability problems of Spark, ensuring that the computing engine can run 24/7 without interruption. In addition, the Inceptor engine independently builds a distributed data layer, which separates the calculation data from the JVM memory space of the computing engine, thus effectively reducing the impact of JVM GC on system performance and stability.

In terms of SQL execution plan optimization, Inceptor implements a cost-based optimizer and a rule-based optimizer, supplemented by more than 100 optimization rules, to ensure that SQL applications can maximize performance without manual changes. For common data processing problems such as data skew, the execution engine can also automatically identify and optimize it, which can solve most computing scenarios with data skew and eliminate the impact of data skew on system stability.

In order to better adapt to various data scenarios, Inceptor's execution engine includes two execution modes: low-latency mode and high-throughput mode. The low-latency mode is mainly used in scenarios where the amount of data is relatively small. The execution engine will generate a physical execution plan with low execution latency to ensure that the execution time of SQL is short by reducing or avoiding some high-latency tasks (such as IO, network, etc.) , reaching or approaching the performance of relational databases in these scenarios. The high-throughput mode is mainly used in big data scenarios to improve the performance of complex statistical analysis on extremely large amounts of data through reasonable distributed execution. Therefore, Inceptor's execution engine can meet data business needs on various data volumes from GB to PB.

6. Data source connector Stargate

Enterprise data may be scattered in multiple systems, and they cannot share data or perform relevant analysis with each other, resulting in the phenomenon of data islands. By building a unified big data platform, the problem of data islands in most scenarios can be effectively solved. However, there is still a phenomenon that some data cannot be migrated to the unified platform due to various relationships. In order to solve such problems, Inceptor launched the data source connector Stargate. Stargate is a connector that connects execution engines and various data sources. It can connect data from multiple different data sources to the engine for real-time statistical analysis without having to import the data into HDFS in advance, making it more convenient for users to build diverse business needs. . At the syntax level, Inceptor is compatible with the OracleDB-Link specification. By creating a database link to pre-establish a connection pool with other data sources, you can then access the data of the data source in SQL in real time through table_name@database link in Inceptor. No further action is required. After the execution plan starts, Stargate extracts the required data from other data sources through pre-established connections and inputs it into the execution engine layer to participate in SQL calculations. After the calculation is completed, the relevant database connections and corresponding resources are released.

Stargate currently supports relational databases including Oracle, DB2, Mysql, Teradata and PostgreSQL. In addition, Stargate can currently be connected to Holodesk, HDFS, Hyperbase, Elastic Search, etc., and can be connected to Redis as a data source in the future.

7.Multi-tenant management component Guardian

Guardian provides complete multi-tenant management functions, including tenant resource management, tenant permission management, security control and other modules, which can facilitate the management and distribution of multi-tenants on a unified big data platform for enterprises.

Guardian allows configuration and management of CPU and memory resources for multi-tenants. Different tenants use different CPU and memory resource pools, so they will not interfere with each other. In addition, different priorities can be set for different users to implement quality of service control (QoS). Guardian supports the configuration and management of user disk space through SQL, including the quota, change and management of data space and temporary space, to facilitate the platform's reasonable allocation, control and billing of storage resources.

Guardian supports the use of LDAP protocol for user access control and Kerberos protocol for underlying access control to ensure data security and isolation. Guardian supports a complete set of SQL-based database/table permission control. Administrators can set user permissions for table query, modification, deletion, etc., and includes a complete set of role settings, which can be conveniently implemented through role group settings. Permission control.

In addition, Guardian supports Row Level Security to perform precise row-level permission control on table data. In a multi-tenant scenario, it can be ensured that different tenants can only see the data in the table that they have permission to, and will not see the data belonging to other tenants, thus achieving precise data isolation.

8.Middleware management unit Connector

Inceptor fully supports JDBC 4.0 and ODBC 3.5 standards, so it can support middleware such as Hibernate/Spring, is fully compatible with reporting tools such as Tableau/QlikView/Cognos, and can be fully connected with the current data application layer of the enterprise.

In addition, Inceptor also supports docking with other data synchronization tools, has completed mutual certification and integration with IBM CDC, and can support Oracle Golden Gate, SAP Data Service and other tools. Therefore, enterprise users can synchronize transaction data to Inceptor in real time for interactive statistical analysis.

The technical solution of the present invention includes:

1. Unified data storage

Integrates physically independent and management-separated major database systems into a complete and unified data storage.

2. Unified metadata management mechanism

In order to achieve centralized and unified management of metadata, a metadata warehouse is deployed in the logical data warehouse, that is, an RDBMS cluster specifically used to store and manage metadata. If a new table or index is created in each database system, the logical data warehouse will simultaneously create a metadata two-dimensional mapping table in the metadata warehouse, and record the relationship between the mapping table and the original table/index, so that Lay the foundation for SQL unified access.

The metadata management system is responsible for storing information on all database objects, including schema, table, view, column, trigger, sequence and stored procedures. It also provides query interfaces for other systems to retrieve. The queryable information includes Limited to the physical distribution of data, distribution characteristics of data, maximum and minimum values, permission information, etc. Because metadata information is very critical data, and its reliability determines whether the entire database system can provide services, the prototype system needs to fully design and consider the reliability of the metadata storage system. The present invention plans to use a consistent and highly reliable storage service to record metadata information. Optional systems include Apache Zookeeper or Apache Etcd system. Both Zookeeper and Etcd store the same data in multiple instances, and use consistency algorithms (such as Raft algorithms) to ensure that the data in each instance is completely consistent. In this way, even if some instances cannot provide services due to software and hardware problems, as long as one instance is still working normally, the metadata system can still ensure that the service is not dropped.

3. Unified data model

In order for all data systems in multiple engines to support SQL access, the data model must be unified. Research has found that although the technical principles of each system are different, most of them adopt a two-dimensional table-like data model. Take HBase as an example. Although it uses a four-dimensional table model, it also organizes data by rows and introduces column family relaxation. Column restrictions are eliminated, so a four-dimensional table can be mapped to a two-dimensional table under certain constraints. Through this idea, the present invention can uniformly map the data model of the multi-engine database system into a two-dimensional table of a relational database, also known as a two-dimensional relational data mapping table. This lays a theoretical foundation for centralized and unified management of metadata and unified SQL access. .

4. Unified SQL parsing engine

Based on the high-performance computing framework Spark, a universal distributed SQL engine is built. The engine is responsible for providing unified SQL parsing, optimization and execution services for all data systems in the logical data warehouse. SQL statements submitted by users are first received by the SQL engine, converted into Spark codes after parsing and optimization, and then handed over to high-performance A distributed computing cluster is used for execution. During the execution process, the native API of each system will be called to achieve data access.

5. Unified security control

The unified authentication system based on users and roles follows the account/role RBAC (Role-BasedAccessControl) model to implement permission management through roles and batch authorization management of users. It supports the security protocol Kerberos, uses LDAP as the account management system, and performs unified security authentication of account information through Kerberos. In a complex battlefield confrontation environment, unified security management and control can prevent the enemy's information countermeasures system from eavesdropping on one's own information transmission content, interfering with one's own system signals, and injecting false and deceptive information into the information system. The information security requirements for data applications in information warfare include:

(1) Data encryption: Provides encryption protection for data during transmission and static storage, and can still be effectively protected when sensitive data is accessed without authorization. In terms of data encryption and decryption, high-performance, low-latency end-to-end and storage layer encryption and decryption can be achieved through efficient encryption and decryption solutions (non-sensitive data does not need to be encrypted, which does not affect performance). At the same time, the effective use of encryption requires secure and flexible key management. In this regard, open source solutions are still relatively weak and commercial key management products are needed. In addition, encryption and decryption are transparent to upper-layer services. Upper-layer services only need to specify sensitive data, and the business is completely unaware of the encryption and decryption process.

(2) User privacy data desensitization: Provide data desensitization and personal information de-identification functions, and provide user data encryption services that meet international cryptographic algorithms.

(3) Multi-tenant isolation: Implement multi-tenant access isolation measures, implement data security level division, support tag-based mandatory access control, provide ACL-based data access authorization model, provide global data view and private data view, and provide data view Access control.

(4) Data life cycle management: Understanding the source of data in the big data platform, as well as knowing how the data is used and who and where destroys it, is very critical to monitoring whether there is illegal data access in the big data system. This requires This is achieved through security audits. The purpose of a security audit is to capture a complete record of activity within a system that cannot be altered. A big data platform must be able to conduct comprehensive security management and control of data, so that it can be managed beforehand, controllable during the process, and auditable afterward.

(5) Log audit: Log audit is indispensable as an important measure for data management, data traceability and attack detection. The big data platform has log management and analysis capabilities.

Specific embodiments of the invention:

1) Unified metadata management implementation method

(1) Metadata management architecture

In order to achieve centralized and unified management of metadata, a metadata warehouse is deployed in the logical data warehouse, that is, an RDBMS cluster specifically used to store and manage metadata. If a new table or index is created in each database system, the logical data warehouse will simultaneously create a metadata two-dimensional mapping table in the metadata warehouse, and record the relationship between the metadata two-dimensional mapping table and the original table/index. relationship, thus laying the foundation for SQL unified access. We have done in-depth research on metadata management, and the architectural design is shown in Figure 2:

Metadata Sources Access

Provide metadata access and collection functions for Hive, HDFS, HBase, etc.

Spring Framework UI&Restful API

Provides a UI interface for unified metadata management and a Restful API interface for related services, and provides various types of microservice docking methods. The data storage of the UI front-end page data of unified metadata management uses MySQL database tables, and the page data is operated through the front-end page. Input, real-time or offline query backend platform service acquisition

Metadata Integration&Notification API

Provides Messaging message queue (currently using Kafka) and API interface (HTTP or REST mode) metadata operation interface and data message bus mode

Core Platform

Provides a unified metadata Type System, Graph computing storage query engine layer, smart label algorithm, knowledge graph model, etc.

Graph Database

Provides a public storage encapsulation layer for graph computing query engines and supports the Janus Graph open source graph computing storage query engine.

(2) Metadata storage principle

The Metadata store uses HBase to store entity information, and Index information is stored using ElasticSearch; the Metadata store client connects to the Metadata store service, and the Metadata store then connects to the MySQL database to access metadata. With the metastore service, multiple clients can connect at the same time, and these clients do not need to know the user name and password of the MySQL database, they only need to connect to the metastore service. Unified metadata starts by launching metadata as a separate service. Start a MetaStoreServer on the server side, and the client uses the Thrift protocol to access the metadata database through MetaStoreServer. The Thrift communication protocol is shown in Figure 3:

The underlying IO module is responsible for actual data transmission, including Socket, files, or compressed data streams.

TTransport is responsible for sending and receiving Messages in the form of byte streams. It is the implementation of the underlying IO module in the Thrift framework. Each underlying IO module will have a corresponding TTransport that is responsible for Thrift's byte stream (ByteStream) data on the IO module. transmission. For example, TSocket corresponds to Socket transmission, and TFileTransport corresponds to file transmission.

TProtocol is mainly responsible for assembling structured data into Message, or reading structured data from the Message structure. TProtocol converts a typed data into a byte stream and gives it to TTransport for transmission, or reads a certain length of byte data from TTransport and converts it into a specific type of data. For example, int32 will be encoded into a four-byte byte data by TBinaryProtocol, or the four-byte data taken out by TBinaryProtocol from TTransport will be decoded into int32.

TServer is responsible for receiving the Client's request and forwarding the request to the Processor for processing. The main task of TServer is to efficiently accept Client requests, especially in the case of high concurrent requests, to quickly complete the requests.

The Processor (or TProcessor) is responsible for responding to the Client's request, including RPC request forwarding, call parameter parsing and user logic invocation, return value writing back and other processing steps. Processor is the key process for the server to transfer the user logic from the Thrift framework. The Processor is also responsible for writing data to or reading data from the Message structure.

2) Unified security management and control implementation methods

In a complex battlefield confrontation environment, unified security management and control can prevent the enemy's information countermeasures system from eavesdropping on one's own information transmission content, interfering with one's own system signals, and injecting false and deceptive information into the information system. Therefore, the research on unified security management and control technology based on multi-storage engine databases is very important. This section ensures the security and unity of the entire multi-storage engine database platform by studying the unified security management and control architecture, unified service encryption and authentication services, and unified data permission access control. Control.

As shown in Figure 4, the unified security management and control architecture is divided into 4 layers:

The system layer uses improved ApacheDS to increase reading and writing efficiency by more than 10 times. It uses the same set of users and a unified LDAP/Kerberos authentication method to avoid OpenLDAP using Kerberos authentication and accelerate LDAP authentication efficiency.

The business layer implements complete ARBAC model support and provides REST API, user-friendly WebUI, password policy and other functional support. At the same time, the JWTToken mechanism is used to make basic preparations for implementing SSO. User authentication and authorization are unified, and the same set of users and authorization mechanisms are used from Web services to the bottom layer of Hadoop. At the same time, the LDAP interface, REST API and Login Service are opened for third-party applications to integrate authentication and authorization.

The plug-in layer uses plug-ins to provide authentication, authorization, group mapping and quota management for each component, so that each component can use a unified user, group and permission management model.

Various platform PaaS and SaaS services connected to the service layer are protected by unified security control.

Provides unified service encryption and authentication services through the Kerberos protocol. The specific process of Kerbeors authentication is as follows: the client sends a request to the authentication server (AS) to obtain a certificate from a certain server, and then the AS's response contains these certificates encrypted with the client key.

As shown in Figure 5, the certificate consists of: 1) server TGT; 2) a temporary encryption key (also known as the session key "sessionkey"). The client transmits the TGT (which includes the client's identity encrypted with the server's key and a copy of the session key) to the server. The session key can be used (now shared by the client and server) to authenticate the client or the server, to provide encryption services for future communications between the communicating parties, or to provide further encryption services for the communicating parties by exchanging independent sub-session keys. Communication encryption services.

3) Efficient real-time indexing method for heterogeneous data such as graphs, key values, documents, relationships, etc.

In view of the difficulty of traditional relational databases in meeting the problem of complex data types in military big data processing, including heterogeneous data such as documents, relationships, spatio-temporal locations, etc., the present invention proposes to process PSPQ (Point and Segment Preferences) based on numerical point and numerical segment preference attribute information. Query) index, and then combined with the keyword index, a hybrid index BRPQ (Boolean Range with Preferences Query index) is proposed that supports numerical point attributes, numerical segment attributes, spatial positions and text. Specifically, it includes the construction of PSPQ index, the combination of PSPQ and keyword index, spatial keyword distributed query with relational attributes, and index construction based on Lucene.

PSPQ index structure

The PSPQ index structure consists of two parts: a hash table and an inverted file. The structure is shown in Figure 6.

(1) Hash table. Used to index numerical point relationship attribute information, including key-value pairs of (key, pIF). Among them, key is the identifier of the numerical point relationship attribute information, and pIF is the pointer to the inverted file.

Hash table key value design. There are multiple numerical point attributes, and the query for each attribute is a range query. While ensuring the pruning rate, joint range query for multiple attributes is a major challenge in key value design. In response to this challenge, each attribute value interval is divided into multiple small intervals, the key value of each small interval is assigned with the minimum value of the interval, and then the Cartesian product of the small interval keys of each attribute is performed to obtain the key value of the hash table. . This has two major advantages. On the one hand, data can be quickly accessed based on the key value, and joint range queries can be performed on multiple attributes. On the other hand, during pruning, data that does not meet the query range can be quickly filtered out based on preference queries. Hash table key values, the more each attribute value interval is divided, the more hash table key values are filtered out, and the stronger the pruning ability.

Preprocessing divides each attribute value interval to obtain the hash table key value. For each numerical point attribute, the attribute values of all objects are divided into equal parts to obtain each interval. The purpose of equal parts is to make the number of objects in the corresponding interval of each part roughly equal. The data is evenly associated under each hash table key value. It becomes more stable during pruning and there will be no pruned hash table key value association. There are too many or too few objects.

Of course, since numerical point attributes are numerical attributes and different attributes are independent, B-trees can also be used to index numerical point preference attributes. However, the cost based on B-tree is much higher than that of BRPQ index, so this project does not use B-tree index. This is explained in detail below, as shown in Figure 7.

The query cost increases based on B-tree. When indexing multiple numerical point preference attributes based on B-trees, a B-tree index needs to be established for each attribute. Multiple B-tree indexes cannot be combined with the index that handles spatial keywords and the index that handles numerical segment user preference attributes. As shown in the figure above, after using B-tree, it is necessary to use the spatial keyword index to obtain the candidate result set that satisfies the spatial and keyword query, and use the numerical segment user preference attribute index to obtain the candidate result set that satisfies the numerical segment preference. In each B-tree Find the candidate result set that satisfies this attribute preference, and then find the intersection of all candidate result sets. When querying based on the BRPQ index, the result set after the spatial keyword query of the data set is then used for preference query. Assuming that the number of numerical point user preference attributes is p, based on the B-tree, the entire data set needs to be queried p+2 times, while the BRPQ index only needs to be queried once. In the query process, the cost of the two spatial keyword query processes is the same. As for preference query, BRPQ performs preference query based on the results obtained by spatial keyword query, and based on B-tree, it performs p preference queries on the data set. to find objects that satisfy each property preference. Moreover, BRPQ uses hash technology for numerical point preference query, and the time complexity is O(1), while the time complexity of B-tree is O(log2N), N is the height of B-tree, and the time complexity of p B-trees is O(plog2N). In summary, the query cost based on B-tree is significantly higher than BRPQ.

The index space increases after being based on B-tree. When indexing multiple numerical point preference attributes based on B-tree, a B-tree index needs to be established for each attribute. Assume that the number of numerical point user preference attributes is p and the number of data set objects is n. Not only do (p+2)*n pointers to data need to be saved, but also p B-tree indexes take up a large space. BRPQ only needs to increase the key value of the hash table.

The insertion cost increases after being based on B-tree. After indexing the user preference attributes based on the B-tree index, it is necessary to insert p+2 times into the entire data set. BRPQ only needs to be inserted once. During the insertion process, the cost of inserting the spatial keyword index is the same for both, but when inserting the preference attribute, BRPQ uses hash technology, the insertion cost is O(1), and the worst insertion cost based on the B-tree index is O(logn ), the time complexity of inserting p B-trees is O(plogn). Therefore, the insertion cost based on B-tree is significantly higher than BRPQ.

(2) Inverted files. Used to index numerical segment user preference attribute information, including key-value pairs of (key, pL). Among them, key is the identifier of the user preference attribute information in the numerical segment, pL is the pointer of the inverted linked list corresponding to key, and the object identifier or pointer pO is stored in the inverted linked list to uniquely identify an object.

When querying based on the inverted file index, the DAAT algorithm provides high efficiency for searching for a certain data that appears in multiple inverted lists at the same time. When querying numerical segment preferences, use key values to quickly prune a large number of data, and use the DAAT algorithm to query efficiently.

PSPQ combined with keyword index

The PSPQ index supports collaborative pruning of numerical point and numerical segment user preference attributes. In order to solve spatial keyword range queries with user preference constraints, PSPQ needs to be combined with the spatial keyword index.

At present, there has been a lot of research on spatial keyword queries. Quadtrees or R-trees are generally used to index spatial locations. As shown in Figure 8, BRPQ uses quadtrees, but R-trees can also be used. Considering the small number of intermediate results and the fact that using ordered keyword trie provides better query performance when retrieving ordered keywords, the ordered keyword trie is used to index text information and the ordered keyword tree is pruned. The path that comes out will clearly satisfy the keyword query. The combined index BRPQ structure is shown in the figure. The quadtree leaf node contains a pointer to the ordered keyword tree, and a threshold β is set for the object associated with the quadtree leaf node. When the threshold is reached, the quadtree leaf node splits. In order to ensure few intermediate results, leaf nodes (only nodes associated with ordered keyword trees will be called leaf nodes in the following text) will continue to be divided until the number of objects contained in each quadtree node is less than or equal to k. Each path in the ordered keyword tree contains a pointer to PSPQ.

Spatial keyword distributed query algorithm with relational attributes

In recent years, the amount of data has changed from gigabytes to terabytes or even petabytes. Therefore, bulk data management plays a very important role in data analysis. But how to efficiently index and query data is the bottleneck of data analysis. When the search engine obtains data, it will only try to search for data with the closest text, closest geographical location, and satisfying relationship attributes. However, due to the sudden increase in the amount of data, existing search engines cannot efficiently respond to query statements containing these three types of information because there are no corresponding efficient indexes that can process these types of data at the same time. Even existing spatial keyword-oriented queries do not take the relational attribute into account, thus reducing query efficiency. And in this era of large-scale data, traditional index structures and algorithms are implemented based on a single machine, and the processing speed can no longer meet the needs of users. In order to solve the above problems, a Baseline distributed index algorithm KLPDQ (Baseline Algorithm for Keywords and Location-aware with relational attributes Distributed Query index) is proposed that supports keywords, geographical location information and relational attributes. And on this basis, a more efficient spatial keyword query processing algorithm with relational attributes is proposed. This algorithm can not only index these three types of data at the same time, but also use distributed indexing and query processing to significantly reduce the speed of indexing and querying.

Spatial keyword object (Object) with relational attributes = {K, G, P}. O.K is the keyword set of the object, O.G is the longitude and latitude coordinates of the object, and O.P is the set of relationship attributes. O.P={P’, S}. O.P.P’ is a set of numerical points of the relational attribute and each unit represents the minimum value of the attribute. O.P.S is a set of numerical segment attributes of the relational attribute. In the present invention, only the merchant's business hours attributes are considered for the numerical segment attributes.

Spatial keyword range query (Query) with relational attributes = {K, R, P}. Q.K is the set of query keywords, and Q.R is the set of query geographical location ranges. Q.R=={T,L,B,R}, Q.R.T is the upper left corner of the geographical location range; Q.R.L is the lower left corner of the geographical location range; Q.R.B is the lower right corner of the geographical location range; Q.R.R is the lower left corner of the geographical location range upper right corner. Q.P is the query set of relational attributes of the query. Similar to the construction of objects, Q.P is also composed of queries for numerical points and numerical segments respectively.

The necessary and sufficient conditions for an object to satisfy the query are: Specifically, it means that the object must contain the keyword queried in the query statement, must be within the spatial range of the query, and its numerical point attribute must be greater than the queried numerical point attribute, and the numeric segment attribute must be within the queried numerical segment attribute range.

However, because the query and indexing processes are based on big data environment, this mechanism is based on Lucence's distributed indexing and query mechanism. Index space, text and relational attributes at the same time, convert multi-dimensional information into one-dimensional information, and use Lucence and Hadoop to store and query spatial keyword objects under big data.

Lucene index building

Some basic concepts of Lucene mainly include index, document, domain and token. An index represents a series of documents. A document represents a series of fields. A domain represents a series of tokens. A token is an array of strings. If the same string belongs to different domains, it is a different token. Therefore, the token represents a pair of strings, one of which is the domain name, and the other is a certain string within the domain.

(1) Index: In the computer field, the inverted index is a data structure that can be used to store text mapping relationships. For example, text and numbers are mapped to their locations in database files, etc.

(2) Segment: A Lucene index that is divided into smaller blocks is called a segment. Each segment is an index. During the Lucene query process, all segments are accessed sequentially.

(3) Document: Document is the smallest unit in the Lucene indexing and query process. A document represents a series of fields. Each domain has a domain name and a series of values. Because each field is stored within the document, the document needs to give each field a unique domain name to prevent conflicts.

(4) Domain: Each domain consists of two parts, domain name and value. The value can be any string or number, etc. These values are used to represent data. Values can optionally be stored in the index in case the document is selected and the information within the domain needs to be returned.

(5) Word element: The word element represents a string of characters, consisting of two elements, one is the string text information and the other is the domain name to which it belongs.

Lucene index creation process

For example, the following are three text messages:

T0＝I was a student, and I came from Nanjing.

T1＝My father is a teacher, and I am a student.

T2＝Nanjing is my hometown, but my father’s hometown is Chengdu.

First, use a tokenizer to get keywords from three texts. The processing procedure is as follows.

1) In the preprocessing stage, Lucene uses tokenizer to divide the text content into segments. For Chinese, because it involves semantic issues, you can choose to use a Chinese syntax analyzer.

2) During the analysis phase, Lucene needs to filter out any punctuation marks and connecting words, and then convert each word into uppercase letters. The indexer then calls the function addDocument(Doc), passing the input value to Lucene for indexing. The processed data is stored in index files and on disk as inverted file data.

After the preprocessing and analysis stage, the keyword information of these three pieces of text data becomes:

key0＝I am student I come from Nanjing.

key1＝My father is teacher I am student.

key2＝Nanjing is my hometown my father hometown is Chengdu.

Now you can build an inverted index based on keyword information. After the index is established, it can be updated or deleted through IndexReader.

Although the present invention has been described in detail with general descriptions and specific examples above, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection claimed by the present invention.

Claims (10)

1. A big data oriented multi-storage engine database system comprising:

the SQL compiler supports ANSIQL 92 and SQL 99 standards, supports ANSI SQL 2003OLAP core expansion, meets the requirements of data warehouse business on SQL, and is convenient for application smooth migration;

the storage process compiler supports complete data types, flow control, package, cursor, exception handling and dynamic SQL execution, and supports high-speed statistics, addition, deletion, modification and distributed transaction operation in the storage process, so that migration of data application from a relational database to an indicator platform is satisfied;

The transaction management unit realizes the control of consistency and isolation through a two-stage locking protocol and MVCC, and supports Serializable SnapshotIsolation isolation level, so that the transaction consistency under the concurrent condition can be ensured;

the distributed memory is stored in a column mode, and a memory or SSD-based column storage engine Holodesk is used for storing data in a column mode in the memory or SSD, and is assisted with a memory-based execution engine, so that delay caused by IO is completely avoided;

the distributed execution engine independently constructs a distributed data layer, and the calculation data is independently from the JVM memory space of the calculation engine, so that the influence of JVM GC on the system performance and stability is effectively reduced;

the data source connector is used for connecting the execution engine and various data sources, carrying out real-time statistical analysis on the data access engines of various different data sources, and not needing to lead the data into the HDFS in advance, thereby being more convenient for users to construct diversified demands for the service;

the multi-tenant management component provides complete multi-tenant management functions, including tenant resource management, tenant authority management and security control module, and facilitates the management and distribution of multiple tenants of enterprises on a unified big data platform; the configuration and management of CPU and memory resources are allowed for multiple tenants, and different tenants use different CPU and memory resource pools, so that the multiple tenants cannot interfere with each other;

The middleware management unit supports JDBC 4.0 and ODBC 3.5 standards, so that the middleware management unit can support a Hibernate/Spring middleware, is completely compatible with a Tableau/QlikView/Cognos report tool, and can be completely docked with the current data application layer of an enterprise.

2. The big data oriented multi-storage engine database system of claim 1, wherein the storage process compiler comprises a complete Optimizer including CFG optimizers, parallel Optimizer and DAG optimizers, the CFG optimizers optimizing code in the storage process to accomplish loop unrolling, redundant code elimination, optimization of function inlining.

3. The method for storing a big data oriented multi-storage engine database according to claim 1, wherein the transaction management unit supports starting a transaction with BeginTransaction, ending the transaction with commit or rollback, implementing control of consistency and isolation by two-phase blocking protocol and MVCC, supporting Serializable SnapshotIsolation isolation level, thus guaranteeing transaction consistency in case of concurrency.

4. The big data oriented multi-storage engine database system of claim 1, wherein the Holodesk supports building a distributed index for data fields, allowing a user to build OLAP-Cube for multi-field combinations and store the Cube directly on memory or SSD without additional BI tools to build Cube.

5. The big data oriented multi-storage engine database system according to claim 1, wherein the distributed execution engine is based on a cost optimizer and a rule optimizer, and 100 optimization rules are used to ensure that the SQL application can exert the maximum performance without manual modification; the distributed execution engine includes two execution modes: a low latency mode and a high throughput mode.

6. The big data oriented multi-storage engine database system of claim 1, wherein the data source connector extracts the needed data from other data sources through pre-established connection after the execution plan is started, enters the execution engine layer to participate in SQL calculation, and releases the related database connection and the corresponding resources after the calculation is completed.

7. A method of storing a big data oriented multi-storage engine database employing the big data oriented multi-storage engine database system of any of claims 1-6, comprising:

step 1, unified data storage, which is to integrate each large database system which is physically independent and manageably separated into a complete unified data storage;

step 2, unified metadata management, namely, a metadata warehouse is arranged in a logic number bin, and an RDBMS cluster for specially storing and managing metadata is used; creating a new table or a new index in each database system, synchronously creating a metadata two-dimensional mapping table in a metadata warehouse by the logic number bin, and simultaneously recording the relation between the metadata two-dimensional mapping table and the original table and/or the index, thereby laying a foundation for SQL unified access; the metadata management system stores information of all database objects, provides a query interface for other systems to search, and the information which can be queried by the search comprises physical distribution of data, distribution characteristics of the data, maximum minimum value and authority information;

Step 3, unifying the data model, uniformly mapping the data model of the multi-engine database system into a relational data two-dimensional mapping table of a relational database, and laying a theoretical foundation for metadata centralized unified management and SQL unified access;

step 4, unifying SQL analysis engines, providing unifying SQL analysis, optimization and execution services for all data systems in a logic number bin, wherein SQL sentences submitted by users are firstly received by the SQL engines, are converted into Spark codes after analysis and optimization, and then are executed by a high-performance distributed computing cluster, and calling the native APIs of each system in the execution process to realize data access;

step 5, unified security management and control, namely, based on an authentication unified system of the user and the role, the user and the role follow account and/or role RBAC models, authority management is carried out through the role, and batch authorization management is carried out on the user; supporting a security protocol Kerberos, using LDAP as an account management system, and carrying out unified security authentication on account information through Kerberos.

8. The method of claim 7, wherein the step 2 of unified metadata management further comprises providing Hive, HDFS, HBase metadata access collection; providing a UI interface for unified metadata management and an API interface in a Restful form of related Service, providing a micro-Service docking mode of each type, and storing the data of the UI foreground page data for unified metadata management by using a MySQL database table; providing a message queue and a metadata operation interface and data message bus mode of an API interface; providing a Type System with unified metadata, a Graph calculation and storage query engine layer, an intelligent label algorithm and a knowledge Graph model; and providing a public storage packaging layer of the Graph computation query engine, and supporting the Janus Graph open source Graph computation storage query engine.

9. The method for storing the big data oriented multi-storage engine database according to claim 8, wherein the step 2 unified metadata management further comprises real-time efficient indexing of heterogeneous data oriented to graphs, key values, documents and relations, and supporting numerical point attributes, numerical segment attributes, spatial positions and text mixed indexes BRPQ, including PSPQ index construction, PSPQ and keyword index combination, spatial keyword distributed query with relation attributes and Lucene-based index construction.

10. The method for storing big data oriented multi-storage engine database according to claim 7, wherein the unified security management architecture in step 5 is divided into 4 layers: the system layer uses an improved Apache DS, so that the read-write efficiency is improved by more than 10 times, the same set of users and a unified LDAP/Kerberos authentication mode are used, the use of Kerberos authentication by OpenLDAP is avoided, and the LDAP authentication efficiency is accelerated; the service layer realizes complete ARBAC model support, and provides functional support of REST API, user-friendly Web UI and password strategy; the plug-in layer provides authentication, authorization, group mapping and quota management for each component by using a plug-in form, so that each component uses a unified user, group and authority management model; the service layer is in butt joint with the PaaS and SaaS services of various platforms, and is protected by unified safety control.